►
From YouTube: Cgroupv2 Is Coming Soon To a Cluster Near You - David Porter, Google & Mrunal Patel, RedHat
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cgroupv2 Is Coming Soon To a Cluster Near You - David Porter, Google & Mrunal Patel, RedHat
Speakers: David Porter, Mrunal Patel
K8S 1.25 will aim to default cgroupv2 KEP to GA and many new linux distros are already defaulting to cgroupv2 such as latest versions of Ubuntu, Debian, RHEL, etc. This talk will focus on what is cgroupv2 (new kernel APIs for resource management), what cluster administrations need to be aware of during this transition to cgroupv2, and what new features cgroupv2 in k8s will unlock. Some examples of areas we will cover will be memory throttling, disk latency targets, and pressure monitoring. We will discuss some of the active KEPS in k8s taking advantage of these new capabilities and future work in this space. cgroups underlie how resource management works in k8s and focuses on how to properly reserve, allocate, and isolate finite resources on nodes such as CPU, memory, disk, network, etc. This talk will cover the present and future of resource management in k8s. Starting from use cases, we will look under the hood at how kubelet utilizes the underlying linux kernel cgroups for resource isolation and what will change with cgroupv2. There is widespread confusion in the community - should CPU limits be used or not or do they introduce unintended throttling? We hope to provide clarity around this topic and how cgroupv2 will affect it. We will also look at future work in this area.
A
B
So,
first
off
we're
gonna
talk
about
Resource
Management.
What
is
Resource
Management
really
in
terms
of
kubernetes,
so
here's
a
10
000
view
10
000
feet
overview
of
resource
management.
Right
clusters
consist
of
nodes
and
nodes.
Have
resources
like
CPUs
memory.
Disks,
gpus
and
resource
management
is
about
management,
managing
these
resources,
so
nodes
advertise
the
availability
to
the
kubernetes
scheduler.
So
typically,
you
have
some
amount
of
memory
on
your
node
say:
32
GB.
B
You
want
to
reserve
some
for
your
system
like
the
kernel
and
the
and
the
processes
that
are
there
on
the
Node
running
natively
a
system
Services.
Then
you
want
to
reserve
some
for
the
cubelet
India
container
runtime
say
you
take
away
4
GB
each
you
have
24
remaining
and
that's
what's
advertised
to
the
scheduler
as
allocatable.
B
B
So
what
are
some
of
the
requirements
from
Resource
Management?
So
Port
should
not
be
able
to
hurt
each
other
right.
They
should
stay
within
the
limits.
They
should
get
consistent,
Performance
Based
on
what
they
requested.
We
should
be
able
to
prevent
infinite
Loops
for
bombs,
memory,
leaks,
node
lockups,
and
we
should
allocate
the
right
amount
of
resources
for
parts
and
also
we
want
to
ensure
that
doing
all
this
management
doesn't
utilize
a
lot
of
resources.
Ultimately,
we
want
to
allow
as
many
pods
to
be
run
on
a
node
as
possible
right.
B
So
how
do
we
do
this?
So
we
do
this
with
something
called
a
c
groups
and
Linux
kernel.
It's
it's
a
way
to
group
a
set
of
processes,
hierarchically
hierarchically,
and
then
you
have
a
set
of
controllers
that
allow
you
to
put
limits
on
those
processes.
So
we
have
the
CPU
memory
iOS.
These
are
some
of
the
controllers
that
can
be
used
to
put
limits
on
the
processes
and
c
groups
are
controlled
through
a
pseudo
file
system,
which
is
called
a
c
group
FS.
B
So
here's
a
history
of
c
groups,
so
the
first
version
of
c
groups
was
introduced
by
Google
and
the
Linux
kernel
in
2006.
It
was
first
called
as
process
controllers.
It
didn't
cover
everything
then
slowly
over
time.
Lot
of
different
folks
came
and
contributed
other
controllers.
Then
in
V2
development
started
in
2016.
Some
of
the
goals
of
V2
was
try
to
simplify
things
like
things
were
added
organically
right
as
and
so
the
controllers
weren't
working
well
with
each
other
and
they
were
not
unified.
B
So
V2
is
an
attempt
to
simplify
things
and
also
provide
more
features
and
more
stability,
so
Fedora
moved
to
V2
in
2019
Docker
and
run
C
group
support
for
it
in
2020
and
in
2021.
Most
of
the
destroys
are
now
enabling
c
groups
V2
by
default.
Also
like
V1,
is
considered
Legacy
at
this
point
right,
so
kernel
fixes
in
this
area
are
mainly
going
to
V2.
Also
system
D
is
planning
to
remove
V1
support
by
end
of
2023..
So
the
big
message
here
really
is
that
V2
is
real.
It's
coming.
B
If
you
aren't
already
testing
it,
you
should
be
planning
to
test
out
V2
to
make
sure
it
works
well
for
you
and
you
can
give
us
feedback
to
fix
any
issues
that
you
find
so
here's
an
overview
like.
Basically,
the
slide
shows
that
all
the
popular
distributions
that
I
use
with
kubernetes
today,
all
the
container
runtimes,
the
high
level
ones
like
containerdy,
cryo
Docker,
the
lower
level
ones
like
run
CC
run,
have
support
for
c
groups
V2
and
in
125
we
finally
went
GA
with
c
groups.
V2,
so
kubernetes
supports
it
now.
B
So
what's
new
in
V2
right.
So
the
first
thing
is
a
single
unified
hierarchy.
So
I'll
go
over
that
in
the
next
slide,
when
there
are
some
additional
improvements,
so
I
mentioned
like
how
the
controllers
were
added
one
after
the
other
right
so
with
V2.
Now,
work
has
been
done,
so
they
can
work
well
together.
So
one
example
is
Page.
Cache
right
backs
right.
B
The
memory
and
the
I
o
controllers
work
together
to
properly
account
which
processes
are
charged
for
that,
then,
on
the
memory
side,
like
user
and
memory,
TCP
socket
buffers,
kernel,
memory
such
as
inodes
and
entries
are
tracked
together
under
the
memory
memory
controls.
Also
on
the
memory
side.
We
have
way
more
knobs
compared
to
before,
so
we
have
more
control
over
when
the
kernel
starts.
B
Throttling
your
memory
allocations,
instead
of
just
like
hitting
a
limit
and
getting
boom
killed
when
there's
something
new
called
as
pressure
stall
information,
so
it
allows
us
to
monitor
how
much
resource
pressure
is
there
in
a
particular
C
group
for
CPU
memory,
or
I
o
and
finally,
like
this
better
delegation
support.
So
this
allows
us
to
run
rootless
containers
well,
like,
for
example,
podman
uses
this
for
its
rootless
support,
so
here's
an
example
of
V1
versus
V2
hierarchy.
B
A
Thank
you
manal
for
explaining
some
of
the
new
features
in
secret
V2,
so
I'm
going
to
talk
a
little
bit
about
how
kublet
and
kubernetes
actually
makes
use
of
c
groups
to
provide
Resource
Management.
So
the
important
thing
to
understand
here
is
there's
actually
two
components
that
interface
with
c
groups
in
the
context
of
a
node,
it's
the
kubelet
and
the
container
runtime.
So
the
way
it
works
is
that
Kubla
creates
a
c
group
for
each
pod.
A
When
you
start
a
pod,
the
Kubler
actually
creates
a
c
group
to
house
all
the
containers
within
that
pod.
Next,
the
container
runtime
will
actually
create
a
c
group
for
each
container,
so
the
Kubo
actually
passes
the
path
of
the
C
group
for
the
Pod
to
The,
Container
runtime
and
the
container
runtime
owns
the
C
group
within
each
pod.
A
The
other
thing
is
that
kublet
actually
manages
not
just
the
Pod
c
groups,
but
the
whole
secret
hierarchy
with
Qs
classes.
So
kubernetes
is
the
concept
of
Qs
class,
there's
burstable
best
effort
and
guaranteed
pods
and
there's
in
different
levels
of
the
secret
hierarchy.
So,
depending
on
the
Qs
class
of
the
Pod,
the
C
group
of
the
Pod
will
be
placed
under
that
qos
top
level
C
group,
so
I
want
to
talk
a
little
bit
about.
How
do
we
actually
set
these
secret
values?
A
How
do
they
get
from
your
pod
spec
into
the
into
the
kernel
and
set
into
the
C
group
FS?
So
it
all
starts
with
a
with
a
pod
spec
right,
you
create
your
pod,
the
kublet
observes
it
and
it
has
some
some
amount
of
requests
and
limits
set.
The
next
step
is
that
Kubler
will
go
ahead
and
create
the
Pod
C
group,
then
for
the
containers
it's
going
to
actually
talk
to
the
CRI.
The
container
runtime
over
the
CRI
protocol
over
grpc
and
the
CRI
protocol
actually
has
some
definitions
for
all
of
these
different
values.
A
So
we
start
to
convert
these
values
from
the
Pod
spec
into
the
CRI
kind
of
definition.
The
next
step
is
that
the
CRI
will
actually
be
responsible
to
talk
to
the
underlying
container
runtime
and
sometimes
it's
the
same
component.
Sometimes
it's
a
different
kind
of
sub
component
and
it
converts
basically
the
CRI
into
the
real
container.
So
we
create
an
oci
Json
specification.
Oci
is
the
container
kind
of
standard
and
inside
the
oci
spec
there's
this
config.json
file,
which
is
a
standard,
and
it
has
different
fields
for
the
resources.
A
So
it
has
explicit
fields
for
the
memory
and
CPU
and
it
actually
also
has
a
new
unified
field
that
was
added
for
Seagram
V2
to
add
to
set
secret
V2
properties.
So
once
we
have
the
oci
Json
spec,
the
next
step
is
to
pass
it
to
a
lower
level
container
runtime
that
can
run
oci
container
images,
so
this
is
usually
run
C.
This
is
kind
of
the
standard
and,
depending
on
something
called
the
system,
the
C
group
driver
that's
being
used
and
I'll
go
into
a
little
bit
about
what
that
means.
A
Usually,
if
you're
using
cwv2,
you
should
be
using
the
system
DC
Group
driver.
What
that
means
is
that
the
oci
container
runtime
run
C
here
will
actually
talk
to
systemd
on
the
machine
to
create
something
called
a
systemd
scope
unit,
so
it
will
be
managed
by
systemd
and
systemd
has
understanding
around
all
these
properties
around
CPU
and
memory
and
other
resource
controls.
A
A
So
let's
talk
a
little
bit
about
what
properties
are
actually
set
and
what
they
do
so
the
main
properties
we
have
right
now
are
CPU
and
memory.
That's
the
main
resources
we
kind
of
manage.
So
the
first
thing,
let's
talk
about
is
CPU
requests.
So
when
you
set
a
CPU
request,
what
are
you
actually
doing?
The
first
thing
is
that
you're
telling
what
the
minimum
amount
of
CPU
your
container
needs,
and
so
the
schedule
will
look
at
the
CPU
request
and
before
it
schedules
a
pod
to
a
node.
A
It
checks
that
that
node
has
that
CPU
available.
This
gives
you
the
guarantee
that
you'll
always
get
that
CPU
request.
Even
if
the
node
is
100
busy
everything's
using
CPU
you'll
always
get
the
CPU
request
for
the
kublet
and
the
scheduler
never
over
commit
on
CPU
to
actually
Implement
a
CPU.
We
use
something
called
CPU
shares,
which
is
a
Linux
kernel
feature.
A
So
this
is
only
used
to
be
enforced
in
c
groups.
So
the
way
to
think
about
CPU
limits
is
they
are
the
ceiling
for
CPU.
You
can
never
use
more
CPU
than
you
put
in
the
limit.
In
fact,
if
you
use
more
CPU
than
you
put
in
your
limit,
you
will
be
throttled
by
the
CFS
bandwidth
control
in
the
kernel,
and
so
the
important
thing
to
know
here
is
you'll
be
throttled,
even
if
their
spare
CPU
Cycles
available.
A
The
way
this
works
is
that
there's
two
concepts
called
CPU
quota
and
CPU
period
and
in
secret
V2
they're
called
CPU
Max,
but
the
same
properties
apply
and
the
idea
is,
you
have
a
CPU
period
and
a
period
is
a
unit
of
time,
usually
the
default
and
the
default.
Everyone
uses
100
milliseconds,
and
then
you
have
some
amount
of
quota
and
the
way
to
think
about
it.
A
Is
you
basically
get
that
amount
of
code
that
amount
of
time
that
you
can
use
for
CPU
in
each
wall,
clock
kind
of
period,
so
you
have
100
millisecond
period
and
you
get
some
amount
of
quota
and
you
can
use
that
amount
of
quota
within
100
milliseconds.
If
you
use
up
all
the
quota
within
that
first
100
milliseconds
or
before
the
100
milliseconds
are
over,
you
get
throttled
and
then
you
need
to
wait
until
that
100
milliseconds
expires
and
then
in
the
next
kind
of
100
milliseconds.
You
again
can
use
that
quota.
A
So
it's
kind
of
this
burst
Bank,
where
you
constantly
get
refilled
with
quota
and
at
each
at
each
period.
You
have
the
ability
to
use
that
quota
and
if
you
use
more
quota
than
you're
allowed
to
you'll
get
throttled.
So
that's
the
idea
behind
CPU
limits
so
with
CPU
limits,
there's
kind
of
been
something
I
want
to
address,
which
is
kind
of
what
I
call
the
CPU
limits
debate.
So
if
you
look
online,
there's
kind
of
a
debate
going
on
you'll
see
tweets
like
this.
A
That,
for
example,
say
never
set
CPU
limit
and
then
there's
other
tweets.
That
say
something
like
you
know:
debate's
raging,
should
you
set
CPU
limits?
Yes
or
no
there's
articles
say
you
should
keep
using
CPU
limits
and
kubernetes
and
then
other
articles
for
the
love
of
God
stop
using
CPU
limits
so
really
confusing,
and
you
might
be
asking
yourself
all
right
what
is
going
on
here.
What
should
I
be
doing
with
CPU
limits?
So
let
me
try
to
give
you
my
take
on
it.
A
So
the
first
thing
that
we
can
kind
of
all
agree
that
I,
don't
think.
There's
any
kind
of
any
debate
about
is
always
setting
a
CPU
request.
So
CPU
requests
are
used
for
scheduling.
You
need
to
set
a
CPU
request
to
provide
the
minimum
amount
of
CPU
you
need,
and
if
you
don't
set
it
you'll
get
you'll,
become
a
best
effort,
pod
qos
and
you
will
basically
not
be
guaranteed
any
amount
of
CPU
so
always
set
a
CPU
request
that
we
can
all
agree
on
now
about
CPU
limits.
A
So,
like
all
things,
I
can't
give
you
a
definitive
answer.
I
think
there's
trade-offs
here.
So
let
me
try
to
explain
them,
so
the
cons
of
CPU
limits
is
kind
of
the
the
feature
of
CPU
limits
at
the
same
point,
which
is
that
you
can't
use
any
spare
CPU
Cycles.
So
if
there's
spare
CPU
Cycles
on
the
Node
and
you
set
a
CPU
limit
and
you
hit
that
limit,
you
can't
use
that
spare
CPU
available
right,
and
so
this
kind
of
translates
that
you're
kind
of
throwing
away
unused
CPU.
A
You
have
CPU
available,
but
your
pods
can't
use
it.
And
you
know
if
your
pod
is
really
bursty
and
suddenly
gets
more
traffic
or
has
more
things
it
needs
to
do.
It
won't
be
able
to
use
those
spare
CPU
cycles,
and
so,
if
you
start
to
measure
this
and
kind
of
and
analyze
it,
you
and
you
start
to
graph
it
out.
You
might
see
that
you
might
introduce
some
artificial
throttling
into
your
application
and
especially
your
you
know.
P99
latency,
for
example,
may
increase.
A
So
that's
kind
of
the
cons
right,
but
the
pros
of
setting
CPU
limits
is
that
you're
not
actually
relying
on
those
spare
CPU
Cycles.
So
if
you're
constantly
relying
on
spare
CPU
cycles
and
you're
constantly
hitting
the
throttling
limit,
that
probably
means
you
set
a
low
CPU
request,
and
the
problem
with
that
is
that
those
spare
CPU
Cycles
are
unpredictable
right.
There's
no
guarantee
you
get
them.
A
If
there's
other
pods
that
are
scheduled
later,
that
use
up
a
lot
of
CPU,
you
will
not
be
able
to
use
that
CPU
that's
available
right,
because
somebody
else
is
using
it.
So
you're
kind
of
relying
on
this
unpredictable
CPU
right
and
that's
kind
of
the
issue
there.
So
if
you
do
set
a
CPU
limit,
you'll
kind
of
get
more
predictable,
Behavior
right,
you'll
get
you'll
become
in
the
guaranteed
qos
tier
and
you'll
always
ensure
that
you're
not
relying
on
this
unpredictable
CPU
cycle.
That
may
not
be
there.
A
A
So
that's
kind
of
CPU
memory
is
actually
kind
of
simpler
to
understand
memory
request.
It's
only
used
for
scheduling.
We
don't
actually
set
it
at
all
in
c
groups
in
V1
and
V2.
That
will
change
and
I'll
talk
about
that
in
a
second
and
for
memory
limit
we
kind
of
have
two
knobs
memory.max
and
C
group
V2
and
memory
Max,
limited
bytes
and
V1,
and
it's
very
simple.
You
set
that
and
it
comes
from
your
pod
spec
from
your
container
right.
A
If
you
go
over
that
limit,
you
get
killed,
really
simple,
and
so
the
recommendation
that
we
have
generally
set
your
memory
requests
equal
to
your
limit.
The
reason
for
that
is
with
CPU
you
can
kind
of
over
commit
on
it
right.
It's
a
compressible
resource,
but
with
memory
it's
not
compressible
right.
You
can't
over
commit
on
memory,
so
the
recommendations
to
always
set
memory
request
equal
limit.
That
way,
you
don't
impact
other
pods
on
the
system
and
you
might
be
using
more
resources
than
you
can
request.
A
So,
as
I
mentioned
earlier,
there's
two
components
that
interact
with
c
groups
on
the
system,
the
kubelet
and
the
container
runtime
and
the
kublet
right
owns
the
Pod
C
group
and
the
container
runtime
owns
the
container
subgroups
and
when
you
interface
with
the
c
group,
subsystem,
there's
actually
two
kind
of
apis
that
you
can
interface
with
it.
One
is
the
C
group
FS,
where
you're
just
talking
directly
to
the
kernel
and
setting
values,
and
the
second
option
is
something
called
the
systemd
driver
and
the
system.
A
So.
The
other
item
that
we
want
to
kind
of
talk
about
is
monitoring.
So
c
groups
provide
us
Resource,
Management
right
resource
throttling,
but
also
they
provide
us
the
ability
to
export
metrics,
and
so
the
way
this
works
is
that
there's
a
project
called
the
C
advisor
I'm,
the
maintainer
of
it
actually
and
C
advisor
is
responsible
for
actually
scraping
those
metrics
in
C
group
fs
and
getting
them
to
kubl
it,
and
then
other
systems
will
kind
of
actually
get
that
information
and
Export
it
out.
A
You
know
to
Prometheus
and
and
that's
how
you
can
see
them
in
all
your
kind
of
grafana
and
other
dashboards,
Etc
right,
and
so
the
way
it
actually
works
is
Kubla
depends
on
C
advisor
as
a
library
it
links
it
in
and
C
advisor
we
had
to
update
it.
We
had
to
do
some
changes
to
ensure
it
works
with
secret
V2.
That
was
done
in
v043
version
and
it's
included
in
kind
of
the
latest.
You
know
latest
kublets
there's
also
some
other
work.
A
So
the
other
big
effort
is
as
part
of
graduating,
cigaroo
V2
to
GA.
One
of
the
big
things
that
we
worked
on
is
actually
testing
it
and
making
sure
it
works
well,
and
so
Sig
node
has
a
whole
bunch
of
kind
of
tests
that
we
run
against
couplet
and
against
the
different
ecosystem
to
make
sure
it
works.
Well.
So,
as
part
of
this,
we
wanted
to
ensure
that
all
the
features
everything
works
well
with
secret
V2.
A
So
a
couple
things
about
as
a
couple
things
to
be
aware
of,
as
you
start
to
migrate
with
C
group
E2
in
a
kubernetes.
One
of
the
things
that
you
should
do
is
probably
just
use
one
of
the
latest
Linux
distros
that
enables
secret
V2
edit
by
default
and
I'll.
Have
that
slide
earlier
had
basically
every
District.
These
days
is
kind
of
defaulting
to
secret
V2.
You
also
want
to
have
a
requirement
that
you
need
the
kernel
to
be
5.8
plus.
Most
of
those
kernels
are
already
an
even
even
newer
versions.
A
You
should
use
an
up-to-date
CRI
runtime,
the
latest
CRI
runtimes
containerd
cryo.
They
both
supports
eurov2.
The
other
big
thing
I
mentioned
earlier,
make
sure
you're
using
the
systemdc
grip
driver
on
both
the
kublin
container
runtime.
That's
some
configuration
you
need
to
set
and
signode
really
doesn't
support
using
the
C
group
FS
driver
that
was
commonly
used
on
Seagram
V1.
We
don't
support
it
and
we
don't
actually
test
it.
So
please
just
don't
use
it
and
then
also
for
hosted
kubernetes
offerings.
A
A
So
the
other
thing
that
you
should
be
aware
of
is
you
know
this
is
a
big
change
right,
and
so
you
should
test
your
apps
and
make
sure
they
work
well
with
secret
V2,
from
kind
of
the
work
that
we
did.
Most
applications,
they
don't
really
have
C
group
dependencies,
it's
quite
kind
of
rare,
but
some
applications
do,
and
so
the
most
common
case
is
like
third-party
secured
third-party
monitoring
and
security
agents.
A
But
you
have
to
make
sure
that
you're
using
those
versions
that
are
supported
so
work
with
your
vendor
to
understand
what
what
versions
have
C
group
V2
support
and
make
sure
you're
using
those
versions.
The
couple
other
things
some
of
the
popular
projects
like
C
advisor.
If
you're
running
it
as
a
standalone
demon
site,
you
should
upgrade
to
ensure
it
supports
C
group
V2,
the
version
there
is
listed,
there's
another
project,
that's
kind
of
popular
called
Auto,
Max
Crocs
by
Uber
this
one
kind
of
automatically.
A
A
A
You
won't
see
too
many
big
changes,
but
the
really
cool
thing
about
secret
V2
is
some
of
the
opportunities
that
it'll
provide
on
top
and
we
have
many
opportunities
to
have
improved
resource
management
in
general,
using
secret,
V2
and
I
want
to
talk
about
that
a
little
bit.
So
one
of
the
opportunities
that
we
have
is
to
improve
kind
of
how
we
manage
memory,
and
so
this
is
actually
a
feature.
That's
already
Alpha
and
kubernetes
like
it
relies
on
secret
V2.
It's
called
memory,
qos
memory,
quality
of
service,
so
going
back
with
secret
V1.
A
The
problem
is
with
the
kernel.
We
really
only
had
one
knob
for
memory,
and
that
was
memory
limit
right.
You
hit
the
memory
limit
and
then
you're
oomed
and
that's
the
end
of
the
story,
but
with
secret
V2
we
have
much
more.
We
have
much
more
kind
of
control
over
memory.
We
have
four
knobs,
actually
Min
low,
high
and
Max,
and
these
are
the
soft
memory
limits.
So
in
the
bottom
right
here
we
have
a
little
diagram
to
explain
how
it
works,
but
basically
memory.min
is
kind
of
the
guarantee
of
the
kernel.
Please
never
reclaim.
A
This
amount
of
memory
I
this
is
the
minimum
amount
of
memory.
I
need.
Memory.Low
is
kind
of
best
effort.
If
there's
significant
memory
pressure,
the
curl
will
try
to
reclaim
it,
but
it
usually
we'll
try
not
to
memory.high
is
kind
of
the
limit,
but
it's
not
the
hard
limit.
So
as
soon
as
you
hit
that
your
application
will
be
throttled,
it'll
start
to
reclaim
memory,
but
you
won't
be
killed
and
then
memory.max
is
just
like
the
limit.
A
We
were
talking
about
earlier,
which,
if
you
go
over
that
limit,
then
you
get
killed,
and
so
the
idea
here
is
actually
we
are
already
setting
in
your
pod
spec
right
everyone's
setting
a
memory
request,
but
we're
not
actually
using
it
at
all
for
c
groups
at
all
right,
we're
not
actually
using
this
number.
So
the
idea
here
is:
let's
map
the
memory
request
to
memory.low,
and
this
will
ensure
that
you
have
some
amount
of
minimum
memory
for
your
application.
A
The
other
idea
is
you're
setting
a
memory
limit
and
we
kind
of
want
to
get
the
guarantee
that,
as
you
approach,
your
memory
limit
you'll
start
to
get
throttled
and
not
unkilled
and
set
something
to
memory.high.
So
the
way
we
did
that
is
we
take
your
memory
limit
and
we
multiply
it
by
a
throttling
Factor.
A
The
defaults
can
have
0.8
and
then
we
set
that
to
memory.high,
and
the
idea
is
that,
as
you
approach,
your
memory
limit,
you'll
start
to
get
throttled
because
you'll
hit
memory.high
and
then
you'll,
you'll
kind
of
the
kernel
will
try
to
reclaim
memory,
and
if
you
continue
to
get
to
increase
memory
usage,
then
you'll
hit
memory
on
Max
and
you'll
get
killed.
So
the
result
here
is
hopefully
you'll
get
less
frequent
and
kind
of
better
performance.
A
As
you
approach
the
memory
limit,
some
of
the
other
work
we
wanted
to
mention
is
a
PSI
pressure
metrics.
So
this
stuff
is
kind
of
coming
down
the
pipe
and
we
want
to
integrate
Kubler
with
it.
This
will
allow
us
to
kind
of
understand
what
resource
shortages
we
have
and
improve
the
eviction.
So
we
can
detect
things
like
research
shortages
for
CPU
memory
and
I
O,
and
this
will
improve
node
stability.
A
The
other
thing
we
want
to
talk
about
is
disk
throttling.
So
so
the
kublet
has
really
good
resource
control
for
CPU
and
memory
and
ephemeral
storage,
but
disk
has
really
been
a
resource
that
we
haven't
accounted
for
this
guyo,
specifically
so
secret
V2
has
a
new.
I
o
controller
that
helps
manage
I
o,
and
we
want
ability
to
limit,
I
o
of
PODS,
so
we
can
ensure
that
pods
also
get
kind
of
some
amount
of
I
o
guarantees
and
can
impact
the
node.
The
last
thing
we
want
to
talk
about
is
D.
A
So
systemd
has
this
kind
of
New
Concept
called
UMD,
which
is
a
user
space,
oom
killer.
It
uses
PSI
metrics
for
this.
So
the
way
it
works
right
now
is
the
kublet
sets
a
score,
and
then
the
kernel
actually
does
the
oom
killing,
but
the
kernel
has
really
little
visibility
into
the
pods
that
are
running
has
no
idea
about
kubernetes
pod
priority.
A
You
know
what
pods
are
anything
like
that
right,
but
if
we
can
move
the
spoon
killing
into
user
space,
the
Kublai
can
make
these
decisions
and
kubler's
a
lot
better
informed
around
taking
into
account
things
like
pod
Qs
pod
priority
Etc.
So
we
want
to
take
the
killing
and
move
it
out
of
the
kernel
and
put
it
inside
kublet
and
ensure
that
Kublai
can
do
that
in
killing,
where
it
has
a
lot
more
information,
it
can
deal
with
and
PSI
metrics
will
help
with
that.
A
So
that's
some
of
the
future
work
that
we're
planning
to
do
and
please
join
us
kind
of
in
sign
mode
if
you're
interested
in
any
of
these
areas.
So
I
want
to
do
a
quick
kind
of
demo,
video
here
around
Cedar
V2
and
some
of
those
kind
of
Concepts
I
covered.
So
let
me
kind
of
make
this
full
screen
here.
A
So
the
first
thing
we're
going
to
do
here
is
to
create
a
cluster
with
secret
V2
I'm,
using
a
cluster
on
gke
and
GK
has
a
feature
called
node
config
that
you
can
specify
that
you
want
to
Cluster
with
secret
V2
enabled
so
here
we're
specifying.
We
want
to
see
group
V2,
enabled
we're
going
to
go
ahead
and
create
a
cluster.
This
is
a
125
cluster
on
GK
with
that
node
config.
A
The
next
step
is
we're
going
to
have
a
little
workload.
Well,
first,
we're
going
to
examine
the
nodes
that
we
have
so
I
just
created
a
one
node
cluster.
This
is
using
the
latest
kind
of
125
build
of
kubernetes
it's
using
the
container
optimized
OS,
which
we
use
on
on
gke,
it's
running
container
D
and
it's
on
the
515
kernel.
So
this
is
kind
of
the
latest
cost
version
here.
A
So
the
next
step
is
we're
going
to
deploy
a
workload.
So
I
just
have
a
very
simple
kind
of
busy
box
workload.
It
doesn't
do
anything,
it
just
does
a
sleep
and
the
important
things
I'm,
specifying
the
requests
and
limits
for
CPU
and
memory
and
I'm
the
the
limits
are
higher
than
the
request.
So
this
is
going
to
be
a
bursible
pod.
So
I
just
take
this
pod
and
I'm
going
to
deploy
here
on
my
cluster.
So
just
your
standard
standard,
Coupe
cuddle
apply.
C
A
And
then
I'm
going
to
do,
get
pods
and
see
it's
running
so
cool
the
pod's
running.
So
the
next
step
is
I'm,
going
to
SSH
into
the
node
to
kind
of
examine.
What's
going
on
and
and
what's
going
on
in
the
actual
node,
so
I'm
just
going
to
SSH
into
the
node
here,
all
right,
so
we're
on
the
Node.
So
the
first
thing
I
want
to
do
is
I
ran
this
command
called
stat
and
you
can
pass
in
the
C
group
fs,
and
you
can
see
here
that
we
get
back
C
group
2fs.
A
This
is
the
way
to
check
that
the
node
is
actually
using
cdv2,
so
sticker
V2
is
being
used
on
this
node
as
we
specified.
The
next
thing
I
want
to
do
is
kind
of
show
how
this
kublet
pod
secret
hierarchy
is
set
up,
so
I'm
going
around
kind
of
the
tree
command
on
the
kubelet,
a
coupon
slice,
and
you
can
see
kind
of
how
kublet's
managing
the
different
c
groups.
A
So,
at
the
top
level
we
have
the
Pod
slice,
and
so
the
way
it
works
right
is
we're
using
the
system
DC
Group
driver,
so
systemd
has
slices
and
then
system
D
is
creating
the
c
groups
underneath
here,
and
so
we
have
at
the
top
level
of
best
effort,
slice,
versatile
slice
and
guaranteed
slice.
These
are
the
different
Qs
classes
and
within
each
kind
of
Qs
class
we
have
a
slice
for
the
Pod
C
group
right.
Each
of
these
are
the
Pod
c
groups
that
we're
seeing
here.
A
A
This
is
the
C
group,
that's
created
by
the
container
runtime
for
the
actual
container,
so
each
container
gets
its
C
group
created
underneath
that
pod
level
C
group-
and
that's
what
we
see
here
so
each
it
might,
you
might
have
you
know
one
C
group,
basically
for
each
container
and
so
because
we're
using
the
system
DC
Group
driver,
we
can
actually
ask
systemd
about
this
type
of
stuff
as
well.
So
if
we
ask
systemd,
hey
give
us
all
the
slices
that
exist
system,
D
slice
is
basically
analogous
to
a
c
group.
A
A
So
we
can
see
all
those
and
we
can
also
ask
systemd
hey,
give
us
all
the
scope
units
scope
units
will
be
created
by
the
container
runtime
for
each
container,
and
then
you
can
see
them
here.
So
if
we
do
list
units
type
scope,
you
can
see
here.
These
are
I'm
using
containerd
here.
So
we
have
a
scope
units
for
every
single
container.
A
The
next
thing
is,
you
know.
We
deploy
that
BusyBox
sleep
workload
earlier,
so
I
just
wanted
to
kind
of
see
how
the
C
group
settings
are
set
up.
So
first
thing,
I'm
going
to
kind
of
grab
just
run
PS
and
get
the
PID
that
the
Sleep
command
is
running
under.
So
you
know,
here's
the
the
PID
and
then
I'm
going
to
use
Prock
Fest
to
actually
see
hey
what
a
c
group
is
this
processing.
So
here's
the
full
path
and
it
ends
with
that
scope.
A
That's
the
container,
C
group
right
so
I'm
going
to
save
this
into
a
environment
variable
just
called
the
container
C
group.
Just
so
I
can
kind
of
play
with
it
and
it's
a
long
path.
So
anyways
I
have
it
here
and
now
we're
going
to
see
how
the
actual
resource
settings
are
set.
So
we're
going
to
take
a
look,
look
at
CPU,
so
the
first
thing
is
CPU
weight.
This
is
your
CPU
request
and
you
can
see
they're
set
here,
so
these
will
be
converted
from
the
CPU
request.
A
You
send
your
pod
spec
and
then
also
there's
a
CPU
Max
right
and
that's
the
CFS
quota
and
period
That's
set
for
by
the
CPU
limit.
So
all
those
settings
are
being
set
by
the
container
runtime
here
and
then
that's
CPU
and
so
for
memory
here.
I
also
want
to
condense
for
how
this
is
set
up,
and
so
I
also
enabled
that
memory
Qs
feature
I
mentioned
earlier.
That
sets
soft
memory
limits.
A
So
we
have
memory.max
set
that's
set
to
the
hard
memory
limit
on
the
container
right,
that's
just
like
before,
but
we
also
have
the
soft
memory
limit
set.
Now
right,
we
have
memory.highset.
This
is
computed
earlier
right
as
the
throttling
Factor
times
the
memory
Max.
This
is,
as
you
approach
the
memory
limit
and
then
we
also
have
memory.min
set.
This
is
coming
from
your
CPU
from
your
memory
request
that
you
have
so
we're
actually
setting
a
soft
memory
limit
here
as
well.
A
All
right,
so
that's
kind
of
our
presentation
I
want
to
give
a
big.
Thank
you
to
everyone
who
worked
on
this.
This
is
looking
a
big
effort
in
Sig
node,
so
big
shout
out,
and
thank
you
to
everyone
in
signode
who
helped
help
work
on
this
I
want
to
thank
the
container
runtime
Community
container
runtimes
are
super
critical
here
and
shout
out
to
Giuseppe
who
worked
on
this
early
on.
A
A
There's
some
kubernetes
docs
that
you
can
read
about
c
groups
and
some
of
the
details
of
C
group
drivers
and
so
forth
and
if
you're
more
interested,
there's
internal
docs
that
are
a
great
resource
as
well
as
a
couple
other
kubecon
talks.
There's
another
kubecon
talk
earlier
this
week
about
secret
V2
and
then
there's
another
talk
from
2020
by
Giuseppe.
That
also
goes
into
more
details
about
secret
V2.
That's
a
good
resource!
A
D
B
So
yeah
so
actually
in
C
group
system
right,
there's
a
file
that
you
can
listen
on
and
get
events
as
you're
Crossing.
These
thresholds,
so
maybe
like
the
application,
can
open
an
FD
on
that
file
and
look
for
this
notification
and
react
dynamically,
but
I'm
not
aware
of
any
language
doing
that
right
now,
but
that's
a
great
thing
to
explore.
Yeah.
A
B
B
Yeah
so
right
now
it's
more
a
static
tuning.
It
looks
at
the
values
and
decides
what
the
GC
values
are,
but
what
you're
talking
about
is
like
more
dynamic
as
you're
using
how
do
I
react
when
I'm
getting
these
notifications
from
a
kernel
that
you
were
throtted
because
you
crossed
the
low
or
men
or
nearing
the
high.
A
Yeah,
that's
also
a
definitely
interesting
area,
I
think
it's
still
pretty
early,
but
I
think
we
also
want
to
isolate
Network.
I
o
there's
been
some
work
in
the
community
around
that
and
I
think
that's
definitely
an
area
we
want
to
explore
as
well.
I.
Think
kubernetes
is
really
good
for
support
for
CPU
and
memory
right
and
some
of
the
other
resources.
We
definitely
need
to
improve
on.
So
that's
something
we
want
to
explore
for
sure.
Yeah.
B
A
E
Yeah
so
I
might
have
missed
this
conversation
in
signode,
but
I
know
a
lot
of
the
stakeholders
in
signode
like
Google
and
red
hat
use
systemd
like
heavily,
and
so
you
mentioned
that
system
D.
You
know
everyone
was
really
pushing
for
system
the
C
group
driver.
What
is
this
is?
Does
there
exist
a
story
for
distributions
that
don't
ship
system
d.
B
But
I
think
for
those
distributions
right
like
we
see
that
majority
of
the
distributions
are
using
system
D,
we
don't
see
as
much
usage.
That's
why
we
concentrated
on
like
keeping
things
simple,
but
if
folks
are
really
interested
in
using
C
group
refers
like
we
really
encourage
them
to
show
up
to
Signal
like
raise
their
hands
and
help
get
that
support
like
fully
baked
and
working
yeah.
F
I
have
a
question
about
the
soft
memory
limits
you
mentioned
before.
If
an
application
exceeds
the
soft
memory
limits,
then
you
can
throttle
it
down.
Can
you
elaborate
about
what
kind
of
applications
you're
targeting
with
that
I
mean?
If
you're
not
scheduling
it,
then
it's
not
going
to
release
the
memory.
So
in
what
situation
that
is,
is
that
best
avoided
or
best
used.
A
So
what
one
idea
can
do
is
kind
of,
if
you're
not
using
some
of
my
memories,
flush
into
disk
right,
and
so
that
also
interacts
with
swap,
which
is
also
an
ongoing
effort.
So
it
tries
to
swap
memory
if
it
can.
Some
memory
can
be
reclaimed
by
the
kernel
right
because
it's
shared
and
it's
kind
of
caching
it.
So
those
are
the
kind
of
things
that
it
tries
to
do.
B
Yeah
yeah
the
executable
file
Pages
can
be
swapped,
even
though
swap
is
not
enabled,
but
there's
also
an
effort
to
make
swap
available,
and
then
it
will
work
better
and
also
like
for
the
integration
we'll
need
swap
so
umdi
has
the
time
to
react
to
these
changes
and
actually
make
decisions.
Otherwise,
if
if
things
are
going
too
fast
without
swap
then
you'll
end
up
getting
home
killed
by
the
current
Loom
killer,.
G
A
A
C
group
that's
set
so
depending
on
that,
that's
one
memory
pressure
can
impact
two
different
pods
right,
but
if
you
do
set
kind
of
guaranteed
pod
or
you
set
a
memory
limit,
then
you
should
kind
of
get
full
isolation
between
the
the
memory
usage
between
two
pods
right
and,
of
course,
memory.
The
Qs
feature
will
help
kind
of
with
soft
memory
limits
as
well
with
that.
H
Yeah
question
here
so
I:
remember
that
if
you
have
a
single
threaded
app
like
python,
where
it
can
only
take
advantage
of
one
CPU
anyways
with
c
groups
V1,
if,
if
I
don't
have
a
limit,
the
nature
of
the
application,
automatically
limits
it
to
one
CPU.
But
if
I
add
a
limit
of
Hey
limit
yourself
to
one
CPU,
just
the
act
of
adding
a
limit
period
can
lower
performance
and
I
was
wondering
if,
like
C
group,
V2,
helps
that
scenario.
A
So
that's
kind
of
dependent
on
on
the
you
know
on
the
on
the
python
interpreter
to
kind
of
do
that.
So,
if
it
always,
it
kind
of
depends
if
it's
hitting
that
CPU
limit
in
the
first
place.
So
I
don't
know
if
python
actually
can
configure
and
tell
it
kind
of
you
know
like
jdk
and
and
with
go
Max
project
where
you
can
set
kind
of
a
CPU
that
you
have
that
you
can
allocate
underneath
for
you.
A
But
basically
you
know
if
you
set
a
CPU
limit
of
one
and
it
actually
always
will
use
less
than
one.
Then
you
shouldn't
get
throttle
in
the
first
place
right,
but
if
you're
setting
a
lower
CPU
limit-
and
it's
always
using
more
CPU
than
that,
then
you'll
get
throttled.
So
Security
will
not
necessarily
help
with
that.
Unless
the
The
Interpreter
will
actually
integrate
with
that
and
read
those
values
and
tune
itself
appropriately
and
that's
not.
C
I
appreciate
for
your
sharing,
so
in
your
slider
you
mentioned,
there
is
a
configuration.
There's,
a
fat
fatal
conversion,
convert
memory
Mass
to
memory
highlight
so,
but
that
one
is
a
global
conversion.
So
do
you
consider?
May
it
support
purpose?
Configuration
means,
okay,
if
I
don't
set
this
for,
for
my
port
in
my
postback,
I
will
be
forbid
to
use
the
global
configuration,
but
if
I
already
say
in
my
Prospect
I
would
prefer
this
very
yeah.
C
Why
I
asked
this
solution
is
actually
it
costs
out
an
incident
happening
in
our
production
when
we
load
our
SQL
V2,
because
some
application
actually
for
the
forecast
is,
is
okay
but
some
some
education.
They
won't
don't
want
to
be
throttle
yeah
because
they
may
very
latent
system
sensitive
and
when
we
lower
actually
those
applications,
latency
becomes
very
hard
and
turns
out.
We
figure
out.
Actually
we
take
at
least
one
hour
to
figure
out.
This
is
caused
by
our
SQL,
which,
because
at
the
beginning
we
didn't
see
this
issue.
C
A
I
asked
this
question
so
I
really
mostly
focused
on
memory
kind
of
then
like
if
the
Pod
is
not
setting
memory
requests
and
limits,
that's
kind
of
your
question
or
more
on
the
CPU
side.
That's
I
didn't
fully
understand
both
okay.
So
with
the
memory
so
I'll
talk
about
memory.
So
for
memory
with
the
memory
Qs
feature,
the
one
thing
I
didn't
mention:
actually
it's
not
just
setting
the
memory
Min
and
you
know,
memory
High
settings
on
the
Pod
level.
A
It
actually
also
sets
it
at
the
higher
level
at
the
qos
level
as
well.
So
it'll
also
kind
of
look
at
node
allocatable
and
set
memory
settings
there.
So
that'll
ensure
that
even
if
you're
not
setting
any
kind
of
memory
requests
or
limits,
if
you
approach
node,
allocatable
you'll,
also
kind
of
getting
that
behavior
at
the
top
top
level
C
group
for
CPU
I,
don't
think
we
set
it
at
any
kind
of
top
level.
We
set
CPU
shares
based
on
node
allocatable
at
the
top
level,
so
that
will
ensure
that
you
know
we.
A
I
A
I,
don't
believe,
that's
currently
configurable
the
feature's
still
in
Alpha
actually,
so
this
is
actually
where
it'd
be
awesome,
to
get
your
feedback
and
try
it
out
and
see
if
that
works,
for
you
I
think
that
was
kind
of
an
estimation
that
we
kind
of
said
works
for
most
folks,
because
we
didn't
want
to
ask
people
hey,
also
set
us
a
memory
high
on
the
Pod
spec,
it's
kind
of
like
additional
info.
We
don't
think
is
super
useful,
but
if
that
is
useful
to
be
configurable,
that
would
be
great
feedback.
J
B
And
that
should
work.
Similarly,
can
you
set
it
through
a
pod
spec?
No,
you
can't
specify
which
CPUs
you
want
to
use
in
the
Pod
spec.
You
can
only
specify
the
number
of
CPUs
and
then
CPU
manager,
or
something
else
will
go
and
container
work
with
the
container
runtime
to
actually
pick
the
specific
CPUs
that
your
pod
ends
up
using
okay.
K
Thank
you
for
this
I'm
curious
about
the
UMD
killer
in
user
space.
Would
it
be?
Is
it
possible
to
I
would
love
it
if
the
monitoring
has
a
lower
priority
for
being
killed
like
the
monitoring
namespace,
so
my
user,
my
other
workloads,
would
have
a
higher
probability
of
being
killed
when
my
monitoring
is
kept
up
all
the
time.
Do
you
know
if
that's
something
you're
considering
all
right.
B
I
I
think
that's
that's
what
we
are
hoping
right
like
when
we
used
to
an
umdi
like
model.
We
can
actually
look
at
the
priorities
set
for
a
pod
or
other
signals
like
the
qos
class
and
make
smarter
decisions
and
the
kernel
Loom
killer
takes
today.
So
that's
our
goal
and
like
we
still
have
to
do
all
that
work,
so
you're
happy
to
like
join
signode,
and
you
know
give
input
on
your
use
cases
what
all
right
folks
we
have
way
over
time.
So
maybe
we
can
take
one
last.
D
Yeah
I've
noticed
that
the
burstability
of
CPU
seems
to
have
an
effect
on
the
kill
score.
So
if
I
had
my
request
and
limit
for
my
memory
set,
equal
I
would
kind
of
expect
the
quality
of
service
for
the
memory
aspect
and
the
kill
score
to
reflect.
That
of
guaranteed.
Do
you
know
if
that's
being
addressed
at
all
in
V2,
and
do
you
know
why
that
it
behaves
like
that
today?.