►
From YouTube: Propagating Programming Paradigms: Lifting High Performance Compute Into K8s - Marlow Weston , Intel
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Propagating Programming Paradigms: Lifting High Performance Compute Into K8s - Marlow Weston , Intel
Users don’t care where items run, just IF they run and how long they need to wait for completion. We should be building systems where ONLY the hardware, network, storage, and security engineers worry about how to maximally leverage underlying hardware for performance. Increasingly popular AI/ML and traditional HPC workloads have many similarities. It is historically difficult for the users to deploy their workloads in HPC environments. Kubernetes, meanwhile, has focused on simplifying the cognitive load for the users at a cost to both performance and sustainability. We show how to lift paradigms from HPC to make more performant Kubernetes clusters. We give a history of where HPC has come up short in abstracting hardware away from the user. We highlight current Kubernetes projects that do aid in performance in a cloud-native fashion and will go over continuing gaps. We will show how to improve Kubernetes to optimize both performance and sustainability without added pain to the user.
A
So
the
reason
I
have
this
talk
to
begin
with
is
because
I
keep
having
the
same
sorts
of
conversations.
So
I
talk
to
HPC
people.
They
have
assumptions
about
how
their
systems
work,
I
talk
to
Cloud
people,
they
have
assumptions
how
the
systems
work
and
they're
different,
so
trying
to
design
Solutions
in
those
spaces
is
difficult
when
you're
making
different
assumptions
so
I've
been
spending
a
lot
of
my
time,
trying
to
figure
out
how
to
put
together
paradigms
that
work
that
still
give
us
the
niceties
that
we
have
in
the
cloud
today.
A
A
A
A
Users
own
the
compute
nodes.
There
are
another
other
things
scheduled
there,
so
users,
usually
there's
exclusive
access
to
all
the
resources
on
those
nodes.
Security
is
handled
through
very
strict
access
to
the
cluster
itself,
and
the
rules
within
the
Clusters
vary
widely,
so
I've
seen
very
open
systems
and
I've
seen
very
closed
systems
depending
on
what
you're
doing
the
second
one's
harder
to
run
on
and
at
the
end
of
the
run,
the
resources
are
made
available
to
other
jobs.
You
no
longer
own
them.
A
Additionally,
of
most
of
these
systems
are
homogeneous,
at
least
from
the
old
world,
so
the
HPC
applications
usually
assume
that
the
nodes
have,
at
the
very
least,
a
very
similar
setup.
So
that
means
the
same
Hardware
the
same
bios,
the
same
OS,
the
same
drivers,
the
same
firmware
and
the
same
software
infrastructure.
There
are
tools
that
look
for
inconsistencies:
for
instance,
Intel
cluster
Checker
I'm,
a
dental
I
also
used
to
work
on
this
project
and
I
will
tell
you.
A
A
A
They
know
only
about
their
specific
application.
They
can
ask
for
a
number
of
CPUs
or
a
set
of
resources
and
maybe
Define
parameters,
but
they
don't
always
know
what
they
are
getting.
So
the
cross
requests
aren't
always
helpful
and
I've
seen
entire
teams
at
various
companies
that
their
whole
purpose
in
life
is
to
figure
out
how
to
save
money
on
cloud,
because
they're
wasting
resources
so
and
I
was
on
one
of
those
teams.
At
one
point
we
you
know,
one
flaw
would
be
ten
thousand
dollars
a
month.
A
A
So
let's
talk
about
some
positive
patterns
in
kubernetes,
because
there
are
some
things
in
the
cloud
we
want
to
keep
and
one
of
them
is
things
like
device
plugins.
The
device.
Plugin
is
very
simple:
you
have
a
device
manager.
You
have
some
devices
on
the
device
plug-in
on
the
nodes.
You
have
various
devices
that
get
lifted
into
the
pods
that
you
then
are
able
to
use.
This
is
really
neat.
It's
easy
to
write.
It's
easy
to
use.
You
have
grpc,
it's
pretty
quick.
A
There
are
some
limitations
with
Device
plugin.
You
don't
have
any
specific
understanding
about
the
devices
behind
Beyond,
healthy
and
not
healthy,
and
so
because
I'm
part
of
sustainability
and
other
things
that
we
look
at
are
power,
use,
voltage,
temperature
bandwidth,
which
isn't
really
sustainability
of
its
other
items.
A
There
is
one
thing
that
I
would
really
like
to
see
fixed
in
kubernetes,
and
that
is
this
herd
of
managers,
so
Alexander,
kanevsky,
I,
don't
know.
If
you
talked
to
him.
He
goes
by
Sasha
he's
also
at
Intel.
He
calls
us
our
herd
of
managers.
So
if
you
go
inside
the
kublet,
there
are
four
managers.
You
have
topology
manager,
memory
manager,
CPU
manager
and
device
manager,
and
all
of
these
have
to
talk
to
each
other
as
far
as
scheduling
resources,
so
that
they're
in
this
right,
pneuma
Zone-
and
this
doesn't
even
work
correctly.
A
A
A
If
you
look
at
where
people
are
going
they're
going
to
AIML,
because
they
don't
have
to
necessarily
you
have
these
very
bright
mathematician
type
Minds
that
are
going
into
cases
where
they
don't
have
to
know
systems
knowledge,
you
want
to
keep
pluggable
infrastructure,
you
want
to
make
it
easy
for
the
admins
to
put
in
any
type
of
infrastructure,
it's
even
better.
If
users
can
toggle
their
own
options
according
to
their
workload
profiles
and
make
more
things
available.
A
We
still
have
scheduling
issues,
so
we
have
kubatch
which
allows
batch
scheduling.
This
is
wonderful,
similar
to
HPC.
It's
used
in
a
variety
of
systems
and
we've
already
referenced
it
today.
It
does
not
solve
the
issue
of
guaranteeing
node
consistency
or
of
the
firmware
the
OS,
the
BIOS
Etc,
so
you're
still
going
to
get
the
performance
issues
I
referenced
with
cluster.
When
we
were
using
cluster
Checker
when
we
would
find
ethernet
drivers
being
disparate
Intel,
has
a
project
called
Telemetry
where
scheduling
which
allows
scheduling
according
to
node
metrics.
A
So
you
could
actually
look
for
things
like
this
on
your
node,
but
the
metrics
are
currently
node
only
which
we're
trying
to
fix
you
can
pull
things
off
the
node
as
well,
and
it's
also
internal
to
intels
that
we
still
have
to
Upstream.
It
there's
also
Network,
aware
scheduling.
This
was
a
neat
project
put
out
by
IBM
Labs.
It
uses
a
clusters,
Network
latency
and
topological
information
to
better
schedule,
latency
and
bandwidth
sensitive
workloads,
there's
no
guarantee,
though,
if
the
co-located
pods
kills
initial
assumptions.
A
A
A
Hpc
we
also
have
checkpointing.
We
don't
want
to
get
rid
of
that,
it's
available
for
most
ml
pipelines,
but
it's
not
default
at
this
point.
Apparently
125
for
containers.
There's
now
a
new
checkpointing
thing
I
haven't
played
with
it.
I
don't
know
if
anyone
else
here
has,
but
that
looks
promising.
As
far
as
keeping
checkpoints
for
up
more
advanced
things,
we
want
native
CPU
management.
Users
can
control,
so
all
advanced
Solutions
are
currently
out
of
trade.
A
We've
done
some
tests,
interning
these
off
and
playing
with
some
CPU
management
and
we're
getting
over
a
three
times,
speed
up,
which
is
better
use
of
resources
and
with
our
CPU
management
over
current
kubernetes.
This
is
the
thing
that
we're
running
in
case
you're,
curious
and
I'm.
Happy
to
talk
about
this.
This
will
come
out
in
about
two
weeks
three
weeks,
maybe
open
source
internal
to
Intel
and
so.
A
Other
things
we
don't
have,
we
don't
have
guaranteed
networking
bandwidth
with
network
topology.
We
still
have
the
Dual
Nick
per
node.
You
either
have
to
channel
Bond,
or
you
have
to
do
weird
things
with
multis
and
we've
and
it's
very
hand
done
in
order
to
do
that,
and
we
want
something
faster
than
the
current
MPI
operator
in
kublow,
maybe
using
a
direct
fabric
between
pots,
maybe
using
lib
fabric,
which
is
an
HPC
tool
for
for
the
base.
So
you
still
have
the
fast
fabric
between
we
have
other
factors.
A
Still
there's
file
and
block
and
container
attached
storage.
Some
of
those
container
attached
Solutions
require
a
core
outside
of
kubernetes,
just
to
run
because
they're
running
clients
all
the
way
through
they're
interesting,
and
we
need
to
be
looking
at
other
one.
Runtimes
wasm
is
getting
popularity.
We're
going
to
have
to
deal
with
it
and
Singularity
is
also
an
HPC
specific
container.
I,
don't
know
if
anyone's
used
that
pretty
neat
as
a
project,
but
it's
not
really
mainstream.
A
So
anything
during
container
ID
isn't
going
to
handle
that
then
there's
two
last
things
we're
going
to
talk
about
real
quickly.
One
thing
is:
we
want
to
update
the
math
so
once
upon
a
time
people
put
together,
PS3s
because
they're
cheap,
especially
at
the
time,
compared
to
the
compute
nodes.
So
they
made
all
these
clusters
out
of
PlayStation
3s.
A
What
they
found
was
that
the
floating
Point
operations
were
different,
so
they're,
not
necessarily
following
IEEE
standards
and
instead
of
infinity
or
negative
Infinity.
Sometimes
they
would
just
Peg
the
numbers
at
either
end,
which
isn't
really
what
you're
going
for
it's
also
very
slow
on
certain
types
of
floating
point
computes,
but
if
you're
using
different
processors,
how
do
you
do
the
math?
Because
the
numbers
you're
coming
out
may
not
be
consistent
for
your
same
processors?
A
So
this
is
something
where
we
have
to
get
a
little
bit
more
robust,
and
this
is
I'm
calling
this
a
side
quest.
But
this
is
because
I
don't
think
we
have
everything
we
need
today
as
far
as
what
we
need
to
be
thinking
about
with
compute,
because
we
need
to
break
out
of
correctness
and
efficiency,
only
that's
what
we've
been
doing
with
software
for
forever.
We
want
things,
that's
efficient.
We
want
things
correct,
but
we
don't
have
any
robustness
whatsoever.
A
So
today,
reliability
is
a
hardware
problem.
Almost
exclusively
a
desirability
is
a
software
problem.
So
if
you
go
to
this
guy's
lectures,
one
of
the
this
particular
paper
he
goes
through.
This
is
one
of
my
old
professors,
so
I
have
a
place
here
in
truth
to
him,
and
he
actually
took
quick,
merge
in
bubble
sort
and
put
in
random
events
where
it
would
give
the
wrong
answer.
A
The
only
one
that
really
converged
on
a
correct
answer
was
bubble
sort
and
we
all
know
bubble
sort
is
very
slow,
but
we
need
to
start
thinking
in
terms
of
how
to
make
things
correct,
at
least
at
the
micro
level
and
start
Falling,
Towards
correctness,
and
that's
the
sum
of
this.
So
thanks
for
listening.
A
B
D
A
D
A
I
and
so
CRM
they
have,
they
have
solutions
for
CPU
within
the
runtime
and
they're
trying
to
add
in
class-based
resources
into
the
couplet.
The
the
shortcoming
of
this
is
you're
still
going
to
have
issues
where
you
have
different
runtimes.
So
as
people
try
to
introduce
different
runtimes
and
Singularity
is
a
very
nice
HPC
runtime,
it's
not
going
to
address
that.
A
E
E
A
We
were
pinning
pods,
so
we
were
pinning
particular
pause
in
particular
particular
areas.
So
we
got
and
there's
a
talk
about
this
later,
but
we
went
from
48
to
over
70
percent
use
of
the
course
instead
of
the
normal.
So
we
we
were
using
hotel
reservation
for
that
particular
one.
So
there's
particular
benchmarks
sentence
to
your
question.
F
A
F
B
A
A
So
you
know
Intel
had
optane
memory,
which
you
could
put
it
on
the
Node
and
preload,
and
that's
that's
really
the
way.
The
storage
solutions
that
I've
seen
are
currently
working.
So
if
you
look
at
weka,
they
have
different
clients,
they
have
a
client
per
Nick,
basically,
and
they
they
pushed
it
across
trying
to
put
it
co-located,
so
I
think
storage
may
need
some
more
design.
Some
of
the
cool
things
that
I've
seen
one
of
them
was
out
of
Western
Digital.
They
actually
are
doing
modifications
in
the
storage
they
have
compute
in
the
storage.
A
A
A
G
G
A
A
We
can
borrow
from
that
if
you're
doing
things,
but
it's
you're
still
going
to
have
the
piece
where
you
still
have
the
CPU
and
trying
to
get
the
memory
over
so
I
think
we
need
to
start
reimagining
to
be
honest
as
far
as
how
we
build
these
structures,
but
there's
pieces
of
kubernetes
I
don't
want
to
lose,
including
Simplicity,
because
it's
these
systems
are
very
complicated.
So
users,
just
don't
want
to
deal
with
that
level
of
complexity,
but
you're
right.
A
H
A
So
one
of
the
things
I
would
like
to
do
is
make
it
so
that
you
have
a
driver
basically
sitting
there
monitoring
your
pod
behavior
and
if
something
is
blowing
through
the
network.
You
reschedule
that
pod,
but
we
don't
really
have
anything
like
that.
So
we
have
things
that
look
at
current
state,
but
you
still
need
a
monitoring
component
behind
there.
I
C
A
I
A
Is
where
we
need
the
checkpoint,
the
checkpointing
pieces,
so
we
just
now
have
checkpointing
of
pots.
So
I
would
so
I
think
at
this
point,
because
we
have
that
now
we
can
probably
start
to
look
to
leverage.
So
we
can
start
checkpointing
the
pods
and
then
save
the
state
somewhere
so
that
if
we
have
to
deschedule
and
reschedule,
you
can
start
from
the
current
Run
state.