►
From YouTube: Resource Orchestration of HPC on Kubernetes: Where We Are Now and... Swati Sehgal & Francesco Romani
Description
Resource Orchestration of HPC on Kubernetes: Where We Are Now and the Journey Ahead! - Swati Sehgal & Francesco Romani, Red Hat
Kubernetes has become a norm for orchestrating containerized microservice applications in the domain of cloud and enterprise; it is however not yet widely adopted in HPC. HPC enablement on Kubernetes is still a challenge due to requirements like NUMA aware scheduling, advanced resource reservation/allocation capabilities and managing job dependencies and synchronization. Resource managers in Kubelet facilitate the allocation and NUMA alignment of CPU, memory, and devices. The information disconnect between kubelet and the scheduler however, is still a gap that needs to be addressed. The scheduler is oblivious to the resources availability at a more granular, NUMA-zone level which can lead to suboptimal scheduling decisions placing workloads to nodes where alignment of resources is impossible. Contributors from sig-node formed a team to address this problem and implement a numa-aware scheduler and the related infrastructure. Representing the team, the presenters will educate the attendees about the journey of this feature, challenges encountered, end to end solution, current adoption, its roadmap and cover the deployment steps for optimized performance of workloads.
A
A
Our
team
has
been
focused
on
resource
management
capabilities
like
no
more
awareness,
hyper,
threading
and
cubelet
resource
management,
and
we
we
aim
to
enable
our
customers
and
partners
to
run
performance,
sensitive
next
generation
workloads
in
terms
of
the
current
landscape.
Even
though
we've
been
focused
on
telco.
Some
of
the
problems
that
we
have
encountered
and
are
trying
to
solve
are
not
only
irrelevant
to
telco,
but
also
to
other
wide
range
of
workloads.
A
A
So
for
performance,
critical
workloads-
typically,
they
have
very
strict
resource
requirement,
so
it
is
required
that
the
resources
such
as
cpu
memory
and
devices
be
allocated
from
the
same
luma
node
for
optimal
performance
from
resource
management
perspective
within
cubelet.
We
have
cpu
manager,
device
manager
and
memory
manager
that
are
responsible
for
allocating
cpus
devices
and
memory
or
huge
pages
respectively,
and
topology
manager
gathers
hints
from
resource
managers
and
based
on
the
configured
policy,
aligns
them.
On
the
same
note
now
numa
unaware
scheduler.
A
A
The
second
one
is
that
the
cube
root
rejection
loop
can
cause
delays
in
pod
life
cycle,
especially
for
low
latency
workloads.
This
can
be
detrimental
in
the
fact
that
you
know
it
can
impact
the
sla
and
the
performance
of
the
workload
itself.
The
third
one
is
the
unbounded
amount
of
aeropods
that
need
to
be
cleaned
up.
Let's
double
click
on
each
of
these
pain
points
to
understand
them
better.
A
In
addition
to
that,
it
doesn't
consider
the
topology
manager
policy
configured
on
the
node
and
whether
or
not
those
resources
can
fit
on
the
same
new
node.
So
essentially
it
can
place
a
pod
on
a
node
where
topology
manager
can
simply
reject
it
with
the
topology
affinity
error,
but
unfortunately,
with
the
current
design,
we
can
do
much
better.
A
A
This
the
third
pain
point,
is
very
closely
related
to
the
second
one,
where
we
have
runaway
part
creation,
and
essentially
what
happens
is
because
of
the
runaway
part
creation.
We
have
a
series
of
pods
that
need
to
be
cleaned
up,
even
if
eventually
there's
a
pod
that
is
up
and
running
the
pods
that
feel
continue
to
exist
on
the
cluster
itself.
A
A
So,
let's
dive
into
no
aware
scheduling
now
before
we
dive
into
the
architecture,
it's
important
to
understand
the
use
cases
to
understand
the.
Why,
behind
all
the
work
that
we've
been
doing
so
the
first
use
case
is
workloads
that
require
specialized
hardwares
and
that's
where
hpc
comes
into
picture.
This
includes
hpc
workloads
that
require
fpga
gpus
and
want
no
more
alignment
of
the
resources.
A
There's
another
interesting
use
case
where
you
might
want
alignment
of
multiple
accelerators,
gpu
direct
scheduling
which,
which
came
into
light
by
wire,
sat.
It
basically
requires
multiple
accelerators,
like
gpu
and
x,
for
direct
gpu
to
nick
transfers
over
pci
instead
of
going
through
cpus,
and
it
was
important
that
gpu
nick
and
the
cpu
all
be
allocated
from
the
same
luminor.
A
A
A
A
A
A
We
have
the
topology
of
data
agents
and
they
use
the
power
resource
api
to
gather
information
of
the
allocated
resources
and
the
pneuma
nodes.
Those
allocated
resources
were
come
from
and
this
is
done
to
determine
the
pure
numa
available
resources.
These
this
information
is
essentially
exposed
as
crs
and
it's
available
via
the
kubernetes
api.
A
So
now,
when
a
pod
comes
to
the
kubernetes
api
server,
you
essentially
have
the
topology
aware
scheduler
plug-in
that
comes
into
picture.
It
uses
the
crs
that's
available
via
the
api
and
it
makes
a
topology
where
scheduling
decision
by
running
a
simplified
version
of
topology
manager.
Alignment
algorithm.
A
It's
important
here-
that
we
understand
that
topology
manager
still
runs
its
alignment
logic
at
an
old
level
and
it
performs
the
admission
check
for
the
part.
But
what
we
are
trying
to
do
is
being
proactive
and
imparting
the
scheduler
intelligence
and
empowering
it
to
make
the
right
scheduling
decision
now
I'll
hand
over
to
francesco
to
cover
next
part
of
the
presentation
where
we'll
double
click
on
each
of
these
components
and
cover
them.
In
more
detail
over
to
your
francesco.
B
B
B
So,
let's
start
with
an
example
of
a
real
world
example
about
another
switch
topology
so
we'll
have.
We
want
to
highlight
first
of
all,
the
node
the
name
of
the
object
which
matches
the
name
of
the
node
which
this
object
refers
to,
and
so
it's
very
easy
to
cross,
correlate
between
node
objects
or
node
nodes
in
general
and
not
resource
to
polish
objects.
And
then
we
have
the
topology
manager
policy,
because
this
allows
the
plugins
to
make
further
logic
and
about
the
what's
actually
running
at
node
level.
In
this.
B
Each
amazon
has
a
name
which
is
unique
on
the
node,
so
it's
on
some
different
node
can
have
actually
the
same
name,
and
then
we
have
the
resources.
So
for
each
restaurants
known
to
kubernetes,
we
have
the
research
name
and
the
counters
associated
with
that
resource,
and
the
first
of
them
is
the
capacity
which
represents
how
many
of
those
resources
are
actually
physically
present
on
this
zone.
B
B
So
we
have
the
lock
table,
which
is
a
subset
of
capacity,
and
we
have
available
units,
which
is
how
many
of
the
unlockable
units
are
actually
ready
to
be
assigned
versus
already
taken
by
workload
running
on
this
node
on
this
pneumation
worth
mentioning.
Cpus
are
referred
as
a
wall,
because
the
one
single
core
one
single
cpu,
considered
a
synonym
is
the
minimum
amount
that
could
be
exclusively
allocated
by
the
cpu
manager.
B
Can
we
just
extend
the
renault
object?
Well,
it's
not
that
straightforward
for
a
bunch
of
reasons,
the
most
important
ones
is.
First
of
all,
this
is
a
very
specific
information
we
may
want
to
have
separated
by
the
basic
node
object,
which
is
widespread
and
part
of
kubernetes
api.
So
we
can
also
restrict
the
access
to
this
information
and,
in
general,
only
the
clients
that
want
to
access
this
information
may
access
those
object
versus
requiring
them
to
access
the
bulkier.
B
Node
objects
and
nodes
are
bulky
indeed,
so
adding
even
more
data
is
not
straightforward,
then,
arguably
not
today,
obviously
better
approach.
However,
we
are
exploring
options
and
we
are
keeping
the
conversation
open
about
the
best
way
to
expose
this
information
in
general
and
to
make
part
of
the
core
kubernetes
api.
So
this
conversation
is
still
in
progress.
B
We
mentioned
previously
the
topology
updater
agent,
which
is
the
component
running
on
each
node.
You
are
interested
to
expose
the
pernuma
resource
allocation
and
availability,
so
we
have
one
topology
updater
agent,
as
probably
running
as
a
demonstrate
on
each
only
one
set
on
each
worker
node,
and
this
agent
needs
to
have
authoritative
information
about
the
resource
availability
and
allow
current
allocation
and
the
authoritative
source
is
the
cubelet,
because
the
cubelet
is
the
orchestrator
and
the
cubelet
is
the
one
which
allocates
resources
on
a
node
basis
thanks
to
the
quality
manager
and
the
results
manager.
B
So
the
topology
updater
agent
talks
to
the
cubenet
queries
the
cubelet
using
the
powder
switches
api,
which
gained
support
over
the
last
few
releases
to
expose
pneuma
locality
about
the
resources
associated
with
boards.
The
project
sources.
Api
is
node
local,
so
it's
very
fast
but
and
the
topology
of
data
agents
queries
periodically
and
arranges
this
information
per
suit
space
per
normal
per
numa
basis
and
updates
the
objects
they
know
their
suits
to
publish
objects,
but
we
can
also
introduce
some
notification
mechanism
to
have
data
project
detection.
Updater
agent
reacts
quicker
in
a
quicker
way.
B
The
one
example
of
this
notification
mechanism
we
are
exploring
and
trying
it
out
is,
for
example,
plugging
into
kryoke's.
So
when
a
workload
when
a
container
requiring
exclusive
resources
starts
up,
data
project,
iteration
can
be
notified
and,
in
turn,
query
again.
The
cubelet
and
get
up-to-date
state.
B
B
So
it's
very
easy
to
enable
or
disable,
but
it's
that
it's
not
enough
to
just
expose
the
initial
state
of
the
node,
because
there
is
a
tighter
coupling
and
a
tighter
need
for
for
reconciliation
between
the
nordstrom's
topology
object
and
the
scheduler
plug-in,
because
there
is
an
uncertainty
factor
which
is
unfortunately
unavoidable
about
where
the
location
is
taking
place.
Because
of
the
split
brain
thing
we
we
just
mentioned,
we
just
described,
in
other
words
the
topology
manager.
B
Then
the
cubelet
guaranteed
that
the
workload
is
I'll
get
either
all
the
results
aligned
on
anumazon's.
But
it's
not
known
beforehand,
which
number
zones
or
this
workload
is
rejected.
So
you
can,
you
cannot
predict
on
which
the
amazon,
the
workload
is
going
to
land.
So
you
cannot
really
do
accurate
accounting.
You
can
only
learn
after
the
fact-
and
this
is
caused
by
the
fact
that
the
topology
manager
does
the
cubelet
as
the
ultimate
authority
on
the
placement.
B
This
means
that
we
need
to
reconcile
more
frequently
the
scheduler
state
with
numa
pernuma
counters
and
the
node
state
represented
by
the
nordic
swiss
topology
objects.
And
of
course
this
is
a
performance
and
scalability
concern
which
we
are
actively
exploring
and
improving
and
we'll
cover
that
our
plans
about
that
in
a
little
while.
B
Last
but
not
least,
we
cannot
just
override
the
topology
manager
because
it
has
to
have
the
ultimate
word
and
about
the
placement,
because
it
has
the
most
authoritative
picture
about
the
exact
node
allocation.
So
the
scheduler
cannot
and
probably
should
not,
even
if
it
could
drive
the
topology
managed
decision.
B
Now
now
that
we
have
the
data
about
the
paranormal
research
availability
and
capacity,
and
we
have
an
agent
which
keeps
this
information
export
this
information
and
keeps
this
information
fresh.
We
can
build
the
scheduler
plugin,
which
consume
this
information,
and
it's
the
we
have
right.
Now
two
plugins,
which
already
got
merged
in
the
scheduler
plugin
main
repository
under
kubernetes
six
and
the
two
of
them
are
the
filter
plugin,
which
filters
out
from
the
later
processing
stage
filters
out
nodes,
which
we
are
very
confident.
B
B
How
does
improves
the
pain
points
we
mentioned
previously?
First
of
all,
with
the
filter,
plugin
alone,
ruling
out
the
node,
which
cannot
accommodate
the
workload
with
the
requirements
the
world
could
have
and
we
would
align
when
the
workload
needs.
We
can
minimize
the
topology
affinity
error.
We
cannot
say,
unfortunately,
removes
completely
because
the
uncertainty
factor
because
the
fat
we
need
to
reconcile
with
the
ultimate
the
scissor,
which
is
the
cubelet.
B
However,
we
can
greatly
improve
this
scenario
and
we
can
then
have
more
predictable,
behavior
marth,
mentioning
that
some
workload
type
like,
for
example,
containers,
network
functions
and
5g
telco
workloads
also
kind
of
expects
this
kind
of
number
shedling.
So
this
is
a
way
for
us
to
cover
this
gap.
B
How
can
we
try
out
these
all
the
components
we
described?
We
talked
about?
We
have
a
gita
organization.
The
links
are
in
the
slides,
which
is
which
contains
all
the
repositories
for
the
components
we
are
developed.
So,
for
example,
we
have
the
nord
resource,
crd
definition
and
the
auto
generated
go
client.
We
have
we've.
We
mentioned
that
the
shadow
blankets
have
already
been
merged.
We
have
the
not
just
roots
topology
data
agents,
not
feature
discovery,
got
support
for
updating
in
zero
10.
B
We
have
the
switzerland
exporter
on
which
we
experimented
stuff
and
when
we
are
confident
it's
good,
then
it's
ready.
We
send
changes,
so
we
propose
changes
to
another
feature
discovery.
We
have
pre-built
container
images
for
rt
in
this
dot,
io
repository
linked
in
the
slides
same
in
the
organization.
We
also
have
repository
documentation
and
we
also
have
manifest,
which
also
encodes
not
just.
How
do
you
actually
deploy
the
components
but
which
components
which
goes
with
other
components?
B
B
We
built
a
set
of
go
packages
to
work
nicely
to
work
programmatically
with
those
of
these
components
to
install
to
manage
them,
and
we
built
a
tool
called
deployer
lacking
a
bit
creativity
here
to
actually
exercise
those
packages
and
actually
deploy
those
components
in
any
kubernetes
cluster.
This
command
line
tool
is
available,
as
binary
releases
is
again
linked
in
the
slides,
and
this
this
tool
is
not
just
a
demon
but
something
we
actually
use,
and
we
we
generate
the
manifest
we
mentioned
previously
out
of
it.
B
Last,
but
but
not
least,
this
tool
can
also
distort
and
go
packages.
Toolkit
can
also
validate
the
cluster,
meaning
checking
configuration
and
ensure
it
is
compatible
with
the
not
the
new
aware.
Shadowing
settings,
for
example,
is
the
topology
manager
well
configured
in
super
manageable
configuration,
and
things
like
this.
B
We
built
so
actually
a
subset
of
us
built
an
operator
and
on
top
of
the
generic
and
reusable
go
packages
toolkit
dimensional
this
operator
is
wants
to
be
twins
compatible,
so
nothing
specific
about
any
distribution.
However,
we
depend
on
the
machine
config
operator
because
it's
much
more
convenient
to
manage
node
with
that,
but
that's
actually
the
main
and
only
requirement
and
this
operator,
however,
us
more
opinionated
the
settings
about
how
the
animal
where
shielding
would
look
like,
should
look
like,
while
the
deployer
toolkit
we
mentioned
it
are
more
generic.
B
Now
we
can
we
head
towards
the
conclusion,
so,
let's
wrap
up
what's
in
our
future,
we
want
to
actually
investigate
and
invest
and
explore
how
we
can
reduce
the
need
to
reconcile
scheduler
plugin
and
the
state
of
nodes,
and
this
goes
in
the
form
of
the
reserve
plugin.
We
are
experiment
with
and
we
aim
to
bring
forward
in
the
next
months.
B
There
is
also
a
dedicated
slack
channel
very
low
traffic
about
this
initiative.
So
if
you
want
to
ask
questions
or
propose
changes
or
discuss
very
focused
topics
about
this
initiative,
this
is
also
a
good
choice.
Again,
we
have
the
link
to
our
own
github
organization,
which
holds
the
code
all
the
code
and
the
documentation
repos
for
all
the
things
we
mentioned.
We
are
very
welcome
to
file
issues
and
cnpr's.