►
From YouTube: This is The Way: A Crash Course on the Intricacies of Managing CPUs... Swati Sehgal & Marlow Weston
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
This is The Way: A Crash Course on the Intricacies of Managing CPUs in K8s - Swati Sehgal, Red Hat & Marlow Weston, Intel
Optimizing CPU management improves cluster performance and security, but is daunting to almost everyone. CPU management may seem complex, but it can be explained in such a way that even your inner toddler will comprehend. With this talk, we will give a path to success. You may have a multi-socket node cluster where your AI/ML workloads care about the proximity of your CPUs to GPUs. You may be running scientific workloads where you want to pin in cores within containers instead of just a pod level. You may have a single-socket server where you want to save a single core outside of Kubernetes for a daemon dedicated to mining bitcoin, without affecting your other jobs (please do not do this). We will cover these and more, helping you understand the intricacies of CPU management within the kubelet and what Kuberenetes can and cannot currently do. We will also cover how you can help escalate the visibility of use cases not currently covered within Kubernetes.
B
B
B
A
So
I
want
to
take
you
back
in
time
early
2017,
free
kubernetes,
1.8
release
and
show
you
how
the
landscape
looked
like
from
resource
point
of
view.
At
this
point,
we
have
cpu
and
memory
as
only
two
native
resources,
supported
requests
and
limits
that
you
see
highlighted
here
are
mechanisms
kubernetes
uses
to
control
these
resources
requests
are
what
a
container
asks
for
and
is
guaranteed
to
get
cube.
Scheduler
uses
this
to
identify
a
suitable
node
that
can
fulfill
the
resource
requirement.
A
Limits,
on
the
other
hand,
make
sure
a
container
never
goes
above.
Certain
value,
cubelet
and
container
runtime
uses
this
to
enforce
limits
on
contain
on
the
resource.
Consumption
cfs
quota
is
what
is
used
to
enforce
cpu
limits
on
the
containers,
particularly
from
cpu
perspective.
All
the
pods
and
containers
running
on
a
compute
node
can
execute
on
any
of
the
available
cores
on
that
node.
This
level
of
resource
control
is
not
sufficient
when
pods
are
running
cpu
intensive
workloads
and
contending
for
cpu
resources
available
on
that
particular
node.
A
So
around
this
time
there
was
a
lot
of
discussion
going
on
in
the
community
to
enhance
kubernetes
in
order
to
support,
diverse
and
increasingly
complex
classes
of
applications,
discussions
about
how
to
provide
kubernetes
the
ability
to
run
workloads
that
require
specialized
hardware
or
those
that
perform
better
when
hardware
topology
is
taken
into
consideration
because
of
the
momentum
in
the
community.
In
this
area,
a
resource
management
working
group
was
formed,
which
included
representatives
from
red
hat,
google
intel,
nvidia
microsoft
and
many
more.
A
A
B
B
So
here's
at
a
very,
very
high
level,
a
high
performance,
aiml
cluster.
You
can
tell
it's
aiml
because
it
has
the
xps,
but
you
can
also
use
these
for
high
performance
compute
and
you
no
longer
have
a
single
socket.
You
now
have
you
know
memory
on
both
sides.
You
have
xp's
on
both
sides.
You
have
dual
nooks.
You
have
these
leaf
switches.
A
So
cpu
manager
is
a
beta
feature
that
is
useful
for
cpu
intensive
workloads
that
are
sensitive
to
cpu,
throttling
context
switches
and
require
hyper
threads
from
the
same
physical
core.
Cpu
manager
gives
us
the
ability
to
allocate
cpu
exclusively.
But
in
order
to
achieve
this,
we
have
to
make
sure
that
pod
belongs
to
the
guaranteed
quality
of
service
class
and
has
positive
integral
cpu
request.
A
Internally
cpu
manager
maintains
logical
set
of
cpus,
which
is
also
referred
to
as
pools.
We
have
exclusive
shared
reserved
and
assignable
pools.
The
two
most
important
ones
are
exclusive
and
shared
exclusive
pool
corresponds
to
the
cpu
sets
assigned
exclusively
to
containers
shared
is
the
one
where
burstable
best
effort
and
non-integral
guaranteed
containers
run.
A
So
a
cubelet
flag
can
be
used
to
specify
the
cpu
manager
policy.
The
default
configuration
policy
is
none
which
basically
enables
the
existing
default
cpu
affinity
scheme,
providing
no
affinity
beyond
what
the
os
scheduler
automatically
gives
us.
Even
even
though
cfs
quota
is
used
to
enforce
poor
cpu
limits,
the
workload
can
move
between
different
cpu
cores,
depending
on
the
load
on
the
pod
and
the
available
capacity
on
the
worker
node.
A
A
The
second
policy
option
is
distribute
across
pneuma.
This
is
an
alpha
option
and
it's
hidden
by
default.
Cpu
manager
by
default
is
designed
to
pack
cpus
onto
one
over
node
until
it
is
filled
and
then
any
of
the
remaining
cpus
simply
spill
over
to
the
next
node
in
parallel
code.
This
can
lead
to
undesired
behavior,
as
the
slowest
worker
can
potentially
potentially
act
as
a
bottleneck.
A
When
the
node
is
configured
with
cpu
manager's
static
policy,
we
have
integral
cpus
and
hence
the
c
container
is
allocated
exclusive
cpus.
If,
in
addition
to
that,
we
have
cpu
manager
policy
option
configured
as
full
p
cpu
only
the
part
will
be
rejected.
With
smt
alignment
error,
as
this
will
result
in
partial
allocation
of
a
core.
B
B
A
So
around
2019
time
frame,
which
is
116
release
the
policy
manager
was
introduced.
This
is
a
cupid
component
that
coordinates
topology
of
resource
allocation
of
cpu
and
devices
and
helps
to
extract
the
best
performance
out
of
the
underlying
hardware.
It
aligns
resources
of
pods
of
all
quality
of
service
classes,
and
we
have
the
ability
to
configure
policies
and
scope
for
different
resource
assignment
requirements.
A
The
pod
is
rejected,
with
topology
fmt
error
with
topology
manager,
we
were
able
to
align
cpu
and
devices,
but
guarantees
around
memory
did
not
exist
for
a
while,
and
there
was
a
major
gap.
This
was
addressed
in
kubernetes
121
with
the
introduction
of
memory
manager,
which
became
a
beta
feature
in
122..
A
One
of
the
known
issues
that
topology
manager
has
is
that,
despite
being
able
to
align
the
resources,
the
kubernetes
default
scheduler
lacks
knowledge
of
node
hardware
topology,
because
of
which
it
can
schedule
a
code
where
it
can
fail
with
topology
definite
error.
If
the
pod
is
part
of
a
deployment
or
replica
set,
it
results
in
runaway
part
creation,
because,
because
the
subsequent
pods
that
are
created
end
up
with
the
same
error
as
well
so
topology
manager
comes
into
picture.
When
the
pod
is
being
admitted
on
the
node.
A
It
works
with
other
resource
managers,
which
are
also
called
hint
providers,
as
they
provide
hints.
These
and
providers
are
used
by
topology
manager,
along
with
the
configure
policy
to
align
those
resources
and
reject
if
the
part
cannot
be
admitted.
So
in
this
diagram,
we
have
a
topology
manager
doing
its
magic
admission
time,
where
it
asks
the
hint
providers
for
hints
once
it
gets.
B
So
we've
covered
topology,
where
scheduling
and
pinning
for
the
pods,
but
there's
still
quite
a
bit
of
gaps
for
high
performance
compute
for
optimization
the
cores
are
only
pinned
by
a
container.
They
cannot
be
pinned
by
pod
right
now.
This
causes
issues
because
if
you
have
a
pod-
and
you
want
to
share
course
between
your
containers,
you
can't
affinity
is
node
level
only.
B
So
right
now
you
have
an
eight
zone
limit
for
a
new
minute
for
the
pneuma
zones
and
you
cannot
schedule
one
pod
or
container
per
memo
zone.
So
I
can't
do
a
nice
pretty
split
across
all
of
the
zones
to
make
sure
that,
from
a
scheduling
standpoint,
every
everything
is
just
one
zone
per
each.
So
they're
not
interfering
with
each
other.
B
B
In
some
cases,
some
architecture
actually
has
different
types
of
cores
on
the
same
cpu
you
have
multiple
nics
and
some
single
nicks
and
another,
and
this
last
one
you
have
excel
an
accelerator
fabric
with
xpus
combined
on
top
of
that,
so
here's
a
workload
view
of
how
you
might
use
a
center
like
this,
you
have
a
set
of
microservices
on
the
first
one.
You
have
maybe
mixed
workloads
on
that
that
middle
one,
depending
on
what
your
cpus
are,
and
then
you
have
ai
workloads
on
that
last.
B
A
There's
actually
a
proposal
being
discussed
in
the
community
spanning
across
sydnode
and
six
scheduling
that
aims
are
allowing
pods
resource
requests
and
limits
to
be
updated
in
place
without
without
the
need
to
start
the
pod
or
its
containers,
and
this
feature
is
being
targeted
to
be
merged
in
kubernetes
125
and
once
we
have
that
particular
feature
merge.
The
gap
should
be
addressed
so
yeah.
We
have
a
long
way
to
go
and
we
welcome
feedback
and
healthy
all
the
help
we
can
get
from
the
community.
A
B
So
currently,
there
are
three
different
options
that
I
know
of
well
three
and
then
some
right,
so
there's
crirm,
which
is
a
cri
resource
manager.
This
is
put
out
by
intel,
but
it
is
open
source
and
they
do
take
contributions,
which
is
a
plugable
add-on
for
controlling
resource
assignments,
to
containers
it
plugs
in
between
the
kubelet
and
the
container.
Runtime
keeps
track
of
the
states
of
all
the
containers
on
a
node
and
it
intercepts
the
cri
protocol.
B
There's
a
cpu
pooler.
This
is
a
project
put
out
by
nokia
and
this
is
a
solution
for
kubernetes
to
manage
predefined,
distinct,
pools
of
kubernetes
knowns.
It's
phys.
It
physically
separates
the
cpu
resources
of
the
containers
connecting
to
the
various
pools
and
it's
a
device.
It
has
a
device
plug-in
that
exposes
the
cpu
cores
as
consumable
devices
and
then
there's
also
cmk.
It
was
used
in
the
past.
A
So,
like
the
projects
that
mona
just
mentioned
to
some
extent,
you
you
take
resource
management
capabilities.
In
your
own
hand,
you
could
go
essentially
all
the
way
in
and
come
up
with
a
completely
customized
solution
that
perfectly
suits
your
needs
and
make
it
do
whatever
you
would
like
it
to
do,
but
this
comes
with
a
massive
disclaimer:
do
it
at
your
own
risk?
A
A
B
So,
there's
a
lot
of
community
discussion
on
how
to
address
all
the
gaps,
because
there's
a
lot
of
gaps
right
now
we
managed
to
figure
out
what
those
gaps
are
with
the
cpu
management,
kubelet
use
case
doc,
and
what
is
unaddressed
and
you're
welcome
to
go.
There
add
add
to
it.
You
know,
comment
engage
with
the
authors.
B
B
The
first
part
is:
where
do
you
schedule
your
workload,
which
nodes
and
so
you're
welcome
to
go
to
the
stock
and
add
in
commentary
we
seem
to
have
quite
a
bit
of
community
support,
so
I'm
pretty
excited
about
that.
There's
also
nri,
which
is
a
node
resource
interface,
which
is
a
cni
type
interface
for
managing
resources
unknown
for
positive
containers.
B
This
is
coming
out
of
tag
runtimes,
and
one
thing
that's
pretty
exciting
about
this.
Is
this
would
work
well
with
the
kubelet
resource
plugin
rfc,
because
currently
they're
still
trying
to
find
ways
to
get
all
the
resources
they
need
into
nri
and
then
there's
dynamic
resource
management,
which
is
an
alternative
to
the
device
plug-in
api,
and
the
primary
idea
is
that
the
resource
allocations
can
be
ephemeral
or
persistent,
and
it
allows
users
to
specify
resources
with
specific
parameters.
A
So
kubernetes
has
various
operational
areas
that
are
organized
as
sigs
and
working
groups.
We
would
highly
recommend
that
you
keep
an
eye
on
sig,
node
and
six
scheduling.
Cygnode
is
responsible
for
life
cycle
of
pods
that
are
scheduled
to
node,
and
its
scope
includes
cubelet,
pod
and
node
api
container
runtimes
various
resource
managers
and
hardware
discovery.
A
This
is
in
no
way
a
comprehensive
list
of
what
signal
does,
but
it
should
give
you
a
good
idea
of
what
to
expect.
Six
scheduling,
on
the
other
hand,
is
responsible
for
components
that
make
point
placement
decisions.
So
your
kubernetes
scheduler
and
scheduling
features
frameworks
such
as
security
scheduler
framework
that
allows
you
to
create
custom,
plugins
all
fall
in
the
realm
of
six
scheduling.
A
Working
groups
are
groups
that
are
formed
to
solve
specific
problems
and
that
typically
span
across
multiple
things.
So
topology
we're
scheduling,
is
a
project
that
spans
across
signal
and
six
scheduling.
It
is
not
an
official
kubernetes
working
group,
but
a
bunch
of
people
that
have
come
together
from
different
organizations
trying
to
solve
the
problem
of
scheduler
being
numa
unaware.
A
The
github
organization
linked
here
will
provide
you
all
the
documentation.
Artifacts
and
components
you
need
to
get
get
started
on
this.
The
next
one
is
cnf
tag
runtime,
and
the
goal
of
this
group
is
to
enable
widespread
and
successful
execution
of
wide
range
of
workloads,
such
as
latency
sensitive
workloads,
batch
workloads
within
the
tag
runtime
itself.
We
have
a
container
orchestrated
device
working
group,
and
this
group
is
focused
on
container
runtime,
related
problems
and
device.
A
Vendors
and
container
runtime
maintainers
are
coming
together
to
solve
the
support
for
devices
in
cloud
native
space
and,
lastly,
we
have
batch
working
group.
This
is
a
very
recently
formed
working
group
and
they're,
focusing
on
enhancing
support
for
batch
workloads
such
as
hpc
aiml
natively
in
kubernetes.