►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cgroups V2: Before You Jump In - Tony Gosselin & Mike Tougeron, Adobe Systems
Speakers: Mike Tougeron, Tony Gosselin
Adobe jumped into upgrading to cgroups v2 head-first and hit rocks beneath the surface. Our telemetry broke, HPA ceased to function, and then we started to notice some issues with our java apps. We ended up having to drag ourselves back to shore. However, that doesn’t mean you shouldn’t take the plunge! In this talk, we will give a background on what cgroups are (and why you should care) and how this impacts cloud-native architecture. We’ll also be touching on new cgroupv2 features in development for Kubernetes, such as tools to better manage resource utilization and an intelligent OOM killer for multi-container pods. At last spring’s KubeCon + CloudNativeCon Europe, SIG-Node shared their roadmap for Cgroups and Kubernetes, letting the community know where support for v1 and v2 is heading. This talk will help guide you in your transition and provide valuable feedback as you make the jump. Learn from our bumps and bruises, the water’s great!
A
Hi
everyone:
how
are
we
doing
this
afternoon?
Excellent
thanks
for
thanks
for
showing
up
4
30
on
Wednesday
I
know
the
parties
are
going
to
get
started
here,
another
hour
or
so
so.
I
really
appreciate
you
guys,
sticking
around
for
the
late
afternoon.
Talks
we're
heading
out
this
afternoon
we're
going
to
be
talking
about
C
group
V2
and
taking
a
little
journey
of
how
Adobe
kind
of
came
across
this
this
new
development
in
kubernetes
before
we
get
started.
A
Excellent
and
before
we
start
just
a
little
bit
about
who
the
ethos
team
is,
ethos
is
how
Adobe
does
containerized
workloads
we,
it
is
basically
it's
kubadium
based
kubernetes,
install
and
Beyond
just
kubernetes.
It
is
also
a
layer
of
tools
and
processes
and
practices.
On
top
of
that,
so
container
is
a
service
platform
as
a
service,
Ingress
CI,
CD
security
controls,
you
name
it,
we
do
it.
We
we
basically
have
a
whole
Suite
of
tools
available
for
our
developers.
A
We
read
across
a
number
of
clouds
and
across
a
number
of
nodes.
We
have
the
prerequisite
brag
sheet
there,
so
you
all
can
ooh
and
awe
with
the
the
workload
sizes
that
we're
doing
and
basically
the
philosophy
of
ethos.
Is
we
try
to
make
the
container
mistakes
for
everyone
once
so
that
they
can
focus
on
deploying
really
great
applications
and
are
there
any
members
of
either
of
our
families
here,
just
in
case
there
are
just
want
to
really
emphasize.
A
So
how'd
this
talk
come
to
be
I'd,
love
to
sit
here
and
tell
you
that
you
know
tuj
and
I
got
together
and
we
did
a
careful
study
of
of
Linux
history.
Internals
and
you
know,
picked
out
a
really
great
topic
or
that
we
we
had
a
meditation
on
on
kubernetes
releases
and
neither
of
those
things
happened.
What
really
happened
was
we
knew
the
two
is
greater
than
one
we're
operators
and
we're
kubernetes
operators,
and
we
really
focused
normally
at
that
API
level.
Give
me
an
API.
A
A
So
here's
what
happened
about
gosh.
Almost
a
year
ago,
we
were
going
through
just
a
normal
upgrade
of
our
clusters.
We
were
going
ahead
and
upgrading
the
underlying
operating
system.
We
use
convoke
flat
car,
how
many
people
are
familiar
with
flat
car,
excellent,
some
black
car
fans
here
we're
big
fans.
So
we
went
ahead.
We
upgraded
to
version
2969.0.0
and
this
particular
version
had
a
a
new
note
in
the
release.
Logins
now
we're
defaulting
to
C
group
V2
that
sounds
great
2
is
greater
than
one
I
love
new
software.
A
A
We
actually
have
to
make
kublet
and
our
container
runtime
aware
that
we're
running
this
new
version
of
C
group,
okay,
great,
we
can
figure
that
our
cluster
starts
fantastic,
so
we're
going
through
with
our
testing
and
then
we
find
out
that
gosh
metric
server
is
missing
some
key
metrics
and
in
some
cases
it's
it's
not
starting.
So
that's
that's
not
good.
So
we
we
dive
in
a
little
deeper
and
find
out.
Oh
C
group
actually
is
responsible
for
some
of
the
metrics
that
come
out
and
we're
finding
things
like
horizontal
pod.
A
Auto
scaling
wasn't
working,
so
that's
okay,
C
group
or
excuse
me.
Metric
server
rather
relies
on
the
C
advisor
and
the
version
of
C
advisor.
That
came
with
our
version
of
kubernetes,
which
at
the
time
was
1.20,
it
wasn't
compatible
or
it
wasn't
aware
of
secret
V2,
not
a
problem.
We
won't
use
that
we'll
run
a
newer
version
of
C
advisor
as
a
Daemon
set,
we're
very
smart,
so
we
run
zero
four
three
and
it
works
kind
of
some
of
the
metrics.
Now
work.
A
Some
do
not
at
this
point,
our
Telemetry
team
says
guys.
No
we're!
No
we're
not
doing
this.
You've
broken
our
metrics
where
we're
Flying
Blind
here.
This
is
definitely
not
the
way
we
can
operate
and
we
were
dumbfounded.
Two
is
greater
than
one.
We,
you
know
we're
upgrading
software,
why?
Why
is
this
not
working?
We
we
looked
through
the
logs.
You
know
we
had
seen
that
c
group
V2
it
was
mentioned
in
in
the
119
release.
A
Vlog
should
work
with
kubernetes
120.,
so
we
stopped
by
signode
and
we
said
what
did
we
do
wrong
and
they
kind
of
sat
us
down
and-
and
let
us
know
that
it's
a
lot
more
than
just
setting
a
config
flag
and
starting
your
cluster.
There
are
some
considerations
you
have
to
take
before
you
go
ahead
and
enable
this
new
feature,
and
this
is
kind
of
the
the
thrust
of
what
we
wanted
to
talk
about
today.
What
what
do
you
need
to
consider
before
you
go
ahead
and
just
flip
the
switch
on
C
group
V2?
A
So
what
are
c
groups
c
groups
are
control
groups.
Control
groups
were
added
to
the
Linux
kernel,
starting
in
January
of
2008
and
they're
accessible
via
virtual
file
system.
Like
everything
else
in
the
Linux
kernel
and
and
really
c
groups,
the
the
main
function
is
to
help
you
control
and
account
for
resources.
So
this
is
where
you
can
basically
at
a
process
level,
set
limits
on
how
much
of
a
resource
a
process
can
use
and
as
well
as
find
out
what
its
current
usage
is.
A
So
we
get
that
kind
of
accounting
and
and
Resource
Management
out
of
the
system
without
c
groups.
There
are
no
containers
and
there
is
no
kubernetes
just
to
be
very
clear.
This
is
a
feature
that
we
all
of
us
use
every
day,
whether
we
know
it
or
not,
and
it's
it's
really
part
and
parcel
to
to
how
these
systems
function.
A
How
in
particular,
you
know
what
are
some
examples
of
where
you
might
see
this
in
kubernetes,
the
two
most
common
examples:
you're
setting
your
resources
and
your
limits
on
a
container,
that's
actually
flowing
back
down
through
to
the
C
group
level,
as
well
as
the
metrics
you
might
get.
For
you
know,
resource
usage
for
a
particular
pod
or
container
those
are
also
coming
from
c
groups.
A
The
system
is
basically
interrogating
down
through
the
levels
and
that's
where
it's
pulling
those
metrics
other
things
that
might
use
c
groups,
either
directly
or
indirectly,
are
things
that
rely
on
those
metrics
like
again
horizontal
pod,
Auto
scalar
is
a
is
a
good
example
of
that,
so
secret
V2.
What
changed
secret
V2
is
introduced
to
the
kernel
as
a
non-experimental
feature
in
4.5
that
was
back
in
March
of
2015.,
so
it's
it's
fair.
It's
been
stable
for
gosh
seven
years
now.
A
A
A
Excuse
me
at
the
top
of
my
hierarchy,
I
have
what
are
called
by
controllers,
so
I
have
a
controller
per
resource.
In
this
particular
example.
I
have
a
controller
for
Block.
I
o
memory
and
process
IDs
and
then
underneath
that
you
can
see,
I
have
software,
I
have
BG
and
ad
hoc
and
I
have
groups
under
there
group
a
and
Group
B
in
this
particular
example
for
one
process
I'm
actually
spread
out
across
multiple
groups.
A
I
have
a
group
controlling
my
block,
I
o
for
a
but
then
I
also
have
a
group
for
a
controlling
memory
limits.
They're
two
different
groups.
Now,
if
I
want
to
control
things
completely
from
a
resource
perspective
resources.
First,
that's
great,
that's
a
great
way
to
organize
things.
What
turned
out
in
practice
as
most
people
manage
resources
at
the
process
level,
they
want
to
be
able
to
restrict
in
a
coordinated
fashion
at
the
process,
so
they
want
to
group
their
CPU
and
their
memory
usage
together.
They
want
to
group
their
block.
A
A
A
So
when
this
was
written
in
c
groups
V2
the
model
was
flipped
on
its
head.
Now
we
have
processes
at
the
top
of
the
tree.
It's
a
single
hierarchy.
The
process
is
at
the
top.
I
have
a
subgroup
that
says
these
are
the
resources
I'm
managing
I
have
another
subgroup
where
I
actually
set
those
limits,
and
so
now
I'm
able
to
manage
things
at
the
process
level
in
a
very
tightly
coordinated
fashion.
There
are
also
some
other
rules
and
some
flexibility
that
was
taken
away
at
this
point.
A
A
All
in
one
group
make
sure
that
it's
a
lot
more
thread
safe
that
we're
you
know
basically
coordinating
threads
in
one
group.
It
basically
takes
a
lot
of
the
convoluted
programming
around
resource
management
and
just
builds
it
directly
into
the
C
group
system,
as
opposed
to
having
to
abstract
it
out
and
build
it
in
the
layer.
Above
some
other
cool
features
that
came
out
of
secret
V2,
EBP
ebpf
oriented
device
control,
so
in
secret
V1
devices
were
often
controlled
via
static
files.
A
Now
sea
group
V2
was
actually
built
with
ties
into
EBP
ebpf
to
control
devices.
There's
the
exposure
of
pressure
stall
information,
so
Beyond
just
having
metrics
around
point
in
time,
usage
for
resources.
Now
we
have
information
about
usage
over
time.
Is
your
application
or
a
process
rather
having
spiky
usage?
A
One
of
the
things
that
gets
enabled.
That's
really
awesome
is
a
resource
limitation
for
rootless
containers
in
secret
V1.
In
order
to
run
c
groups,
they
had
to
run
with
root
permissions,
so
with
secret
V1
in
kubernetes
you
could
run
a
rootless
container,
but
you
couldn't
limit
the
resources
on
it.
You
basically
had
to
allow
whatever
resources
it
wanted
with
C
group
V2
enabled
you
actually
can
set
limits,
and
this
sets
up
a
whole
world
of
security
benefits.
A
So
just
a
quick
timeline
of
what
the
you
know
how
this
access
was
added
or
accessed
how
this
feature
was
added
to
kubernetes
in
late
2019.
This
is
when
we
first
started,
seeing
all
the
major
container
runtimes
actually
add,
support
for
c
groups,
V2
and
119
initial
support
just
for
nodes
was
added
to
kubernetes
and
by
this
basically
means
nodes
will
run,
and
that
was
it
so
no
C
advisor.
Yet
we
couldn't
actually
get
the
metrics
out,
but
you
could
actually
run
nodes
using
C
group
V2.
A
After
that,
we
started
seeing
some
Alpha
features
being
filtered
into
subsequent
releases,
one
of
which
is
memory,
quality
memory,
quality
of
service
which,
again
to
usual,
talk
about
in
just
a
minute
here
and
then
probably
the
the
key
moment
when
C
group
V2
actually
became
possible
for
kubernetes
was
in
November
of
2021.
We
saw
the
release
of
C
advisor
0.43.
This
is
the
point
where
we
got
full
C
group
V2
support
in
C
advisor,
which
means
we
can
actually
get
metrics
back
out.
A
So
not
only
can
we
set
limits
now,
but
now
we
can
actually
see
what
is
my
process.
What
is
my
container
using
and
then
have
full
enforcement
over
that
I
have
a
small
asterisk
here
that
there
were
a
couple
small
metrics
bugs
in
zero
four
three.
So
really
we
got
that
support
with
zero
four
four
which
came
in
February
of
the
following
year,
but
zero.
C
B
What
does
this
all
mean
for
me
or
more
appropriately,
I
guess
for
all
of
you,
so
these
important
hierarchy
changes
the
pressure
stall
information
is
important
because
this
starts
to
bring
it
in
on
a
per
container
or
more
appropriately
per
C
group
basis.
So,
instead
of
being
tied
to
just
the
process,
that
was
much
more
at
that
top
level.
Now
it
can
now
be
brought
down
into
the
the
C
group
itself,
so
you
can
get
that
overall
picture
and
that
kind
of
can
tie
into
the
memory
qos
perspective
prior
to
C
group
V2.
B
What
would
happen
is
you
can
set
the
memory
limit
higher
than
the
memory
request
right,
but
memory
isn't
compressible.
You
can't
use
memory
that
doesn't
exist
kubernetes
before
when
scheduling
could
schedule
all
of
that
memory,
all
at
once
and
as
another
pod
came
in
and
it
wanted
to
go
a
little
bit
above
its
memory.
One
of
those
two
would
oom
right,
because
the
memory
just
doesn't
exist
now,
when
kubernetes
did
that
it
would
assign
all
the
memory
at
once.
There
was
just
nothing
it
would
do.
B
It's
a
huge
win
to
those
of
us
that
over
subscribe,
our
nodes,
I,
don't
know
how
many
of
you
out
there
have
nodes
that
you
know
have
three
four
five
hundred
percent
over
subscription
rates,
because
users
just
don't
use
what
they
request,
but
for
us,
that's
really
common.
So
this
is
going
to
be
a
huge
win.
We
don't
end
up
oomming
those
sort
of
situations,
Facebook's
doing
some
really
cool
stuff
in
user
space,
oom
Killers.
B
B
So
let's
say
all
of
a
sudden:
your
application
starts
doing
a
lot
of
logging
and
you're
running
your
sidecar
with
just
a
little
bit
of
memory,
because
usually
your
logging
isn't
that
bad
and
your
logging
side
across
any
spikes
in
its
memory
usage
right
now
today,
with
C
group,
B1
it'll
kill
the
entire
pod
because
it
o
end.
One
of
the
potential
features
now
with
C
group
V2
is
to
oom
killed,
just
that
one
container
restart
that
one
container
and
the
main
pod
stays
up.
B
So
your
application
doesn't
go
down
now
granted
that
feature
isn't
there
yet,
but
that
functionality
is
possible
with
C
group
C2,
which
is
really
cool,
really
powerful
feature
that
can
come
down
the
pipe.
So
I'm
really
excited
about
that.
Tony
mentioned
rootless
containers,
as
many
of
you
know,
there's
been
a
big
effort
inside
of
kubernetes
to
go
more
secure.
Overall,
kubernetes
historically
has
been
unsecure
by
default
as
a
way
of
saying
it.
This
moves
us
more
towards
the
secure
by
default
approach.
B
The
more
we
can
run
things
as
non-root
the
much
better
we'll
we
will
be
so
all
new
resource
management
features.
You
know
new
things
like
the
memory
qos
the
container
aware
om
killer.
These
will
all
be
built
for
nodes
and
cubelets
things
like
that
that
are
running
C
group
V2.
Only
C
group
V1
new
features
will
not
be
coming
to
that.
So
moving
forward,
that's
where
things
will
be
going.
C
group
V1
will
still
be
supported.
That's
really
important
to
know,
however,
something
to
keep
in
mind.
B
According
to
the
Linux
mailing
groups,
systemd
will
no
longer
support
C
group
E1
at
the
end
of
year
2023..
So
what
does
that
mean
for
kubernetes
I?
Don't
know
there
hasn't
been
an
announcement
yet,
but
if
I
was
a
betting
person,
I
would
say
we're
going
to
see
some
sort
of
deprecation
announcement
coming
in
the
next
few
months
about
C
group
V1
because
of
the
Linux
kernel
doesn't
support
it
in
system.
B
D
and
systemd
is
a
requirement
for
C
group
V2
we're
going
to
see
something
happening
at
some
point,
so
something
to
be
prepared
about
and
to
think
about
in
the
future.
So
with
great
power
comes
many
updates,
but
thankfully,
this
time
around
there's
not
that
many.
We
have
to
worry
about
so
cluster
operators.
What
do
you
need
to
do
to
prep
to
prepare?
Well,
the
most
obvious:
does
your
operating
system
support
it?
B
It's
kind
of
a
duh
moment,
but
when
you're
doing
your
testing,
it
may
not
be
as
obvious
as
you
think
it
is
kernel,
5.8
or
higher.
You
can
run
this
quick
little
command
to
CFC
group
V2
is
enabled
you
can
see
where
it
returns
there.
Some
operating
systems
have
it
enabled
by
default.
Some
don't
some
require
you
to
reboot
to
enable
it
some,
let
you
set
it
at
boot
time
if
you're
running
like
an
AWS,
Auto
scaler
that
replaces
nodes
on
reboot.
Keep
that
in
mind.
B
If
you
have
to
set
it,
then
reboot
your
node,
you
may
end
up
terminating
your
node
on
that
reboot
and
then
you
did
yourself.
No
favors,
kubernetes
125
is
where
it
becomes
stable.
Keep
that
in
mind
we
highly
recommend
running
that
you
can
use
it
on
an
earlier
version.
Your
mileage
may
vary.
You
may
have
to
run
C
advisor
as
a
Daemon
set
to
get
everything
you
may
not.
So
keep
that
in
mind.
B
You
need
to
be
running
seed
or
system
D,
for
both
your
container
runtime
and
for
cubelet.
Previously,
you
didn't
have
to
do
that.
You
had
to
you
could
run
well,
you
didn't
have
to
use
systemd
for,
for
both.
You
just
have
to
be
consistent
with
what
you
used.
The
good
news
is,
is
that
cubelet
will
detect,
which
C
group
version
you're
using
C
group,
V1
or
C
group
V2,
so
that
means
you
can
launch
a
node
pool.
B
That's
using
C
group
V1
and
a
no
group
or
no
node
pool
that's
using
C
group,
V,
V1
and
V2
as
separate
pools
so
that
you
can
test
your
applications
test
your
performance
with
the
different
approaches
that
way
there.
You
can
make
sure
your
automation
Works
make
sure
your
application
works.
All
that
kind
of
stuff,
as
Sonya
mentioned,
if
you're
running
see
advisor
as
a
Daemon
set,
really
make
sure
that
you're
running
the
right
version,
I
believe
they're
up
to
0.55.
But
0.44
is
your
minimum
version.
You
need
to
run
so
as
a
developer.
B
What
do
you
need
to
do
to
prepare
well
once
again
kind
of
a
duh
sort
of
moment
make
sure
your
language
of
choice
supports
C
group
V2,
especially
if
you're
using
multi-processor
multi-threaded
applications
for
Java
jdk?
You
know
version
I,
can't
read
the
screen
here:
11.0.16
or
jdk
greater
than
or
equal
to
15..
If
you're
in
go
and
you
use
the
auto
max
proc
make
sure
you're
using
greater
than
or
equal
to
version,
15
or
1.5.0
python
3.12,
you
might
get
it.
B
There's
been
a
git
issue
open
forever
about
having
any
native
C
group
support.
I
don't
bring
that
up
to
bash
on
python
because
I
use
it
a
lot
and
I'm
a
huge
fan
of
python.
The
point
of
it
is:
is
that
some
languages
and
some
tools
and
libraries
kind
of
have
their
own
c-group
support
and
the
way
that
they
do
it
is
they
look
to
the
file
systems
themselves,
to
figure
it
out.
So,
if
you're
using
an
older
Library,
an
older
tool
that
looks
to
the
c
groups,
they
may
not
support
C
group
V2.
B
You
need
to
know
what
your
tools
are
using
know
what
your
legacy
systems
are
doing
and
do
they
even
support
C
group
V2
and
do
they
need
to
be
updated
to
support
both
I
know
in
Python.
There's
lots
of
tools
and
lots
of
libraries
out
there
that
use
their
own
way
of
looking
to
the
file
systems,
to
figure
out
how
much
CPU
and
how
much
memory
a
container
has
just
because
there
is
no
native
support
at
the
time
that
it
was
written.
B
So
that
means
testing
testing
and
not
maybe
more
testing,
definitely
more
testing,
because
otherwise
things
will
break.
But
hey
don't
take
my
word
for
it.
Don't
take
Tony's
word
for
it
just
because
you
know
test
your
stuff,
so
let's
do
a
quick
demo
of
doing
it
right
versus
doing
it
wrong.
This
was
put
together
by
Jansen
Alton
Cloud
on
engineer,
also
at
Adobe.
C
Hi,
my
name
is
Jansen.
Often
I
am
a
cloud
engineer
on
the
ethos
team
at
Adobe
and
in
this
demo
we
will
see
what
could
happen
to
a
go
Application
when
moving
to
c
groups
V2
if
it
is
using
the
Uber
Go
Auto
Max
prox
package
to
automatically
set
the
go
Max
prox
variable
to
match
the
Linux
container
CPU
quota.
C
So
here
we
have
a
kubernetes
cluster
built
with
version
125
and
deployed
on
this
cluster.
We
have
two
pods
running
the
same:
go
application,
disco
application,
spins
up
a
number
of
threads
matching
Eco
Max
procs
variable
and
then
does
some
busy
work.
Both
are
running
the
same
code,
but
the
Pod
suffix
with
old,
is
using
an
older
version
of
the
Automax
prox
package
and
the
one
suffix
with
new
is
using
a
newer
version
which
includes
the
needed
patch
to
correctly
run
on
c
groups.
C
C
What
this
means
is
that
the
application
will
now
spin
up
two
threads
of
busy
work
and
hit
the
CPU
limit
quicker
in
each
CPU
period,
resulting
in
the
application
getting
throttled
for
the
remainder
of
the
CPU
period.
So
on
the
left,
we
are
showing
this
the
throttle
percentage,
which
is
the
percentage
of
CPU
periods
where
the
container
ran,
but
was
stopped
from
running
the
whole
period.
C
The
lines
here
show
when
the
Pod
is
being
throttled
and
which,
for
the
old
pod,
is
98
to
99
of
the
time
and
the
gaps
in
the
lines
show
when
the
quota
was
renewed
and
no
throttling
moves
taking
place
in
comparison.
We
see
the
new
pod
is
not
being
throttled
nearly
as
much
as
the
old
and,
as
you
can
imagine,
as
the
Delta
between
the
go
match.
Procs
variable
and
the
CPU
limit
of
the
Pod
increases
the
quicker
the
application
will
hit
its
CPU
limit
within
the
period
and
start
getting
throttled.
Thank
you.
A
Thank
you,
Jensen,
and
thank
you.
Every
month,
everyone
I
hope
this
talk
has
been
informative,
I
hope
you're
as
excited
about
C
group
V2,
as
we
are
again,
the
big
takeaways
we
had
for
this
is
that
this
is
a
major
upgrade
for
something
that's
so
core
and
important
to
kubernetes,
and
this
really
is
setting
us
up
for
a
lot
of
great
features
for
both
operators
and
developers
alike.
A
It's
also
something
to
be
very
much
aware,
for
you
know
in
the
next
12
to
18
months,
in
terms
of
upgrades
to
both
your
infrastructure,
as
well
as
your
applications.
Just
a
couple
talks
here
to
be
aware
of
this
week.
We
actually
have
another
secret
V2
talk.
This
is
such
a
popular
subject.
We
have
another
team,
actually
talking
about
it.
Our
friends
over
at
red,
hat
and
Google
will
be
talking
about
this
on
Friday
at
2
pm
and
at
4
pm.
A
If
you're
interested
in
all
things,
node
definitely
would
love
for
you
to
check
out
signode
again.
They
were
instrumental
in
how
kind
of
helping
us
get
on
the
right
path
with
this
and
they're
a
really
friendly
bunch
of
folks,
and
they
would
love
to
tell
you
more
about
their
Sig
and
what
they
do
and
how
you
can
help
contribute.
Also,
if
you're
interested
in
C
group
V2
and
how
it
actually
affects
the
stable
release,
there
is
a
release
blog
out.