►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
PodOverhead: Accounting for Greater Cluster Stability - Eric Ernst, Intel
Accounting is very important in Kubernetes. Better accounting leads to improved node stability, density, and more accurate charging users based on their actual resource utilization. Unfortunately, there are gaps in resource accounting in Kubernetes today, mostly based on the fact that running a pod is not actually free. In Kubernetes 1.16, the PodOverhead feature is introduced to fix these issues. We’ll dive into the details of a pod’s journey from client CLI to running on a node, touching on kubectl, API server, admission controllers, etcd, scheduler, kubelet, containerd/cri-o, and runtimes like Kata Containers and gVisor. Through this we will highlight the current gaps and how the PodOverhead feature addresses them. Attend to get a basic understanding of the Pod creation process, and learn what the new PodOverhead feature is and how it can be used to improve cluster stability.
https://sched.co/Uac2
A
Hello,
everybody,
nice
steaks,
I'm,
Priyanka
and
I'm
your
track
track
show
host
I
am
director
of
technical
evangelism
at
a
company
called
gitlab
and
I
also
serve
on
the
cloud
native
computing
foundation,
sport,
the
cloud
native
computing
foundation
same
as
CN
CF.
The
organizer
of
this
show.
So
thank
you
so
much
for
coming
to
this
talk
just
a
friendly
reminder
that
please
read
it
after
the
session
on
scat
comm.
A
That
will
really
help
us
with
programming
as
we
go
along
I
serve
on
the
program
committee
as
well,
and
there's
so
many
sessions
and
just
having
this
feedback
from.
You
really
really
helps
so
we're
all
here,
because
we
care
about
kubernetes
one
way
or
another,
and
when
of
this
talk
is
very
interesting
to
me,
because
it
speaks
to
the
operational
side
of
the
story
where
it's
about
costing
resource
allocation.
A
How
much
money
are
we
spending
and
the
reality
is
that
accounting
with
qu4
kubernetes
pods
was
much
harder
until
recently,
there
are
just
gaps
with
the
release
of
pod
overhead.
That's
changed
and
things
are
much
better
now
and
that's
what
our
speaker
today,
Erik
Ernst
senior
software
engineer
at
Intel
is
going
to
tell
us
about.
Erik
has
been
working
in
open
source
for
a
while,
and
he
is
a
prolific
contributor
to
various
projects,
committees
and
he
is
of
course,
a
kubernetes
contributor.
So
with
that
Eric,
please
take
it
away.
Thank.
B
You
so
I'm
gonna
be
talking
about
pod
overhead.
It's
your
last
chance
to
leave
the
room.
If
this
isn't
what
you're
interested
in
and
a
lot
of
this
talk
is
gonna
be
based
around
you
know,
how
can
we
effectively
use
sandbox
runtimes
in
kubernetes,
but
also
you
know,
if
you're,
using
the
traditional
runtime,
where
the
gaps
and
how
will
this
improve
your
situation
as
well?
So
I
am
a
software
engineer.
B
I
am
a
technical
lead
for
a
team
working
on
the
cotta
containers
project,
a
sandbox
runtime,
which
is
kind
of
why
I'm
pretty
interested
in
making
sure
this
works,
and
you
know
I
with
the
help
from
different
folks
on
sig
note,
put
together
the
cap
and
then
did
the
implementation
for
this
feature.
So,
let's
get
into
it,
I
wanted
to
take
a
bit
of
time
to
really
walk
through
what
happens
when
you
have.
B
So
basically
cube
cuddle
is
your
first
chance
of
rejection
in
this
whole
process.
Client-Side
is
gonna,
go
through
and
verify.
Is
this
gamal
formatted
correctly,
or
is
this
pods
back
correct?
So
you
know
it
has
a
view
of
what
the
pod
spec
API
is
at
this
point.
So
if
it's
not
I'm,
just
gonna
reject
it.
But
let's
say
it's:
okay,
it's
gonna
do
go
ahead
and
create
a
request
which
it
will
then
submit
to
your
communities
cluster.
So
we're
assuming
you
already
have
a
control
plane
up
and
everything
else.
B
So,
the
first
hour,
entry
point
to
the
cluster
is
going
to
be
the
API
server.
So
the
API
server
is
on
a
control
node
and
what
the
API
server
is
gonna
do
is:
first,
it's
gonna
do
authentication
it's
gonna
say
who
are
you
you
sent
me
this
request?
Do
I,
do
I,
know
you
and
if
it
does
ok,
let's
go
to
the
next
step.
That's
gonna
do
authorization
of,
are
you
even
allowed
to
run?
You
know,
submit
a
job
to
create
a
pod.
Can
you
can
you
create
this
pod
object
in
the
cluster?
B
Let's
assume
that
we
have
everything
set
up
correctly.
We
have
the
cube
config
and
everything,
and
it
goes
through
the
next
thing.
We're
gonna
hit
our
Domitian
controllers
and
admission
controller.
They
so
there's
a
whole
bunch
of
them
and
they
kind
of
have
two
different
phases
that
they
can
run
in.
They
can
do
a
mutation
which
essentially
is
a
high
C.
B
What
you're
doing
but
I
to
change
it
slightly
an
example
of
mutation
would
be
to
update
what
the
policy
is
for
pulling
the
container
image
you
could
have
in
here
and
a
mission
control
that
just
says
always
pull
latest.
So
what
it'll
do
is
it'll?
Look
at
your
pods
back,
say:
okay,
I'm
gonna
augment
what
you
have
and
I'm
gonna
mutate.
It
I'm
gonna
change
it
after
that.
It's
going
to
go
ahead
and
do
a
validation
phases
for
different
emission
controllers.
B
One
of
the
relevant
validation
ones
would
be
resource
quota,
so
you
described
your
workload
and
you
said
that
I
have
certain
limits.
I
I
need
this
much
CPU
this
much
memory
and
it's
going
to
go
through
and
say:
AHA.
Yes,
you
have
this
much
available
in
your
namespace.
Still,
yes,
you
can
go
apply
so
there'd
be
an
example
of
validation.
So
let's
say
that
everything
is
good.
The
next
step
is
to
set
it
all
in
stone.
B
We're
gonna
use,
persistent
storage,
we're
gonna,
write
it
into
Etsy
D,
so
that
city
is
kind
of
the
view
of
what
is
the
state
of
the
cluster
and
everything
else.
Everybody's
going
to
kind
of
reference,
this
to
say
you
know,
is
this
object
exists
in
this
case.
We
write
it
in
there.
Our
pot
is
called
awesome
because
I'm
not
creative,
and
it
doesn't
have
a
no
dam
associated
with
it.
So
we
are
at
that
wonderful
state
of
pending,
so
if
you've
seen
pending
this
is
this
is
where
you're
at
it's
the
objects
there.
B
We
know
it's
there,
but
we
haven't
taken
any
action
yet
so
after
that,
what
we
would
want
to
do
is
scheduling,
and
we
do
that
in
Cube
scheduler,
which
is
very
well
named,
and
basically
this
is
going
to
be
like
a
reconciliation
controller
and
pretty
much.
Everything
in
kubernetes
is
almost
like
a
reconciliation
controller.
Where
it's
going
to
go
through
it's
going
to
look
at
C
D,
it's
going
to
go
through
that
pod
list
and
see.
Are
there
pods
that
don't
have
a
pod
binding?
B
They
don't
have
a
node
binding
associated
with
it
if
there
are
start
doing
something,
let's,
let's
assign
it.
So
it's
going
to
do
that.
It's
gonna,
see
awesome
and
it's
gonna
go
and
schedule
to
it
so
scheduling
what
it
ends
up.
Doing
is
it
doesn't
really
look
at
nodes
and
say
you
know
how
many
what's
the
current
utilization
on
the
node
in
a
real
time?
What
it
does
is.
It
looks
at
the
requests
of
the
pods
that
are
running
on
those
nodes
and
it
will
you
know,
statically,
look
at
it
like
hey.
B
You
ran
this
workload
and
I
said
it
need
two
CPUs
and
you
have
ten
CPU.
So
of
course
you
have
room.
It's
not
actually
checking
in
real-time
that
your
two
CPU
workload
isn't
actually
using
anything.
So
it's
a
bit
static
in
that
way,
but
it's
going
to
go
through
two
phases
so
essentially
have
a
node
list.
It's
going
to
go
through
and
it's
going
to
do.
First,
predication
and
then
prioritization
so
predication
is
just
binary.
It's
does
it
fit
yes
or
no
and
there's
some
different
characteristics,
little
check.
B
But
essentially
do
you
have
room
to
even
take
this
workload.
You
filter
through
that
now
you
have
a
remaining
list
of
nodes
which
one
is
the
best.
You
know
kind
of
heuristic
aliy.
What
is
it
wasn't
appropriate?
No,
it
may
be
based
on
the
amount
of
resources
remaining
or
may
be
based
on
you
know.
It
already
has
an
image
on
it
or
different
things
like
this.
So
let's
say
that
it
found
it.
It
picked
a
node
and
it
assigned
it
to
worker
another
creative
name.
B
So
we
finally
now
move
on
from
pending
to
pod,
scheduled
everything
exciting.
So
we
had
like
a
10
line
go
program
and
we
did
all
these
steps
and
we're
finally
getting
closer
to
maybe
starting
to
do
work
so
we'll
jump
from
the
worker
node
over
to
our
from
the
control
plane
node
over
to
a
worker
node,
and
this
is
where
we
start
interfacing
with
cubelet,
which
also
is
kind
of
doing
some
reconciliation.
It's
looking
at
CD
and
saying
what
are
the
pods
that
belong
to
me
out
of
these
ones?
B
What
are
the
pods
that
are
not
scheduled
and
actually
running
at
this
point?
If
there
are
any
I
need
to
take
corrective
action?
So
that's
what
this
essentially,
what
this
giant
four
loop
pod
sync
is
doing,
and
it's
not
gonna.
Do
it
all
by
itself
what
cubelet
is
going
to
do?
It's
going
to
create
a
pod
c
group
that
it's
gonna
hand
off
and
say:
hey
you
over
there.
Can
you
create
this
pod
for
me
and
you
over?
There
is
the
other
side
of
the
CRI
API.
B
So
CRI
is
the
container
runtime
interface,
so
classic
implementers
default
is
docker
shrimp,
but
the
ones
that
I
really
like
our
container,
D
or
cryo,
and
basically
the
CRI
implementation,
is
the
one
who's
kind
of
in
charge
of
actually
creating
the
pod
pulling
the
image
calling
C
and
I
to
create
a
network
device
and
everything
else,
but
in
actuality
they're
not
really
doing
anything
either.
Well,
they
are
they're
doing
a
lot
of
things,
but
we'll
get
into
that.
Basically,
we
hit
this
pod
secret,
sync
and
so
stepping
back.
B
How
do
we
do
this?
Okay?
So
we
create
a
pod
c
group
and
then
its
gonna
call
into
CRI
and
say
create
a
sandbox
for
me,
which
is
essentially
here's
this
pod
infrastructure
create
this
for
me
and
that's
asking
CRI
to
do
it
and
cRIO
call
C
and
I
created
ve,
Thor
or
whatever
kind
of
device
you're
using
drop
it
in
the
network
namespace.
B
For
you
and
then
it's
going
to
call
CRI
and
say:
can
you
create
the
Samba
or
it's
gonna
call
osei
at
this
point
and
oh
yeah,
it's
going
to
do
it
so
osya
is
the
open
containers
initiative
and
there's
several
different
runtimes.
That
kind
of
meet
this
runtime
spec
the
canonical
like
default
solution,
you're,
always
gonna
use
out
of
the
box
is
gonna,
be
run
see,
and
this
is
using.
You
know:
namespaces
cgroups,
dropping
capabilities.
All
the
typical
Linux
kernel,
primitives
they
kind
of
are
the
goodness
of
what
a
container
actually
is.
B
So
that's
wonderful,
but
then
there
are
also
other
runtimes.
Why
do
we
have
them?
Some
people
like
sandboxing
some
people
say
that
namespaces
aren't
enough
for
me.
I
want
to
do
a
little
bit
extra.
So
cata
is
an
example
of
one
gee.
Visor
is
another,
so
I
have
those
two
on
there.
They
have
different
approaches
we'll
get
into
it.
So
basically,
CRI
is
gonna,
pass
down
and
say:
OC
I.
Can
you
actually
do
this?
For
me,
it's
the
one
who's
going
to
deal
with
the
actual
lifecycle
management
of
the
containers
and
up
the
button.
B
So,
if
we're
to
go
even
further
in
the
run
C
case,
what
happens
you
had
the
pod
C
group
run
C's
called
and
says
you
know,
creates
container
for
me
everything
else,
here's
the
image
it's
going
to.
Actually
you
know
again
use
the
namespaces,
create
the
container
put
it
in
there,
but
for
the
sandbox
it
actually
puts
an
infra
container
in
there,
which
is
kind
of
a
placeholder
that
nobody
is
actually
gonna
care
about
unless
you're
trying
to
do
reaping
of
processes-
and
things
like
this.
So
this
is
something
that's
kind
of
necessary
for
cleanup.
B
You
know
Ultra
Hall,
but
it's
something
running
in
there
that
nobody
cares
about.
So
keep
that
in
mind.
If
you
were
using
Java,
what
we
do
is
we
actually
launch
a
virtual
machine
and
then
using
the
guest
Linux
kernel
inside
that
virtual
machine
will
create
a
container.
So
in
this
case
you
have
a
pod,
C
group
and
there
you're
gonna
do
your
business,
which
is
you're
gonna,
have
a
VM
M
could
be
firecracker,
it
could
be
QM.
You
could
be
cloud
hypervisor
I'm,
just
several
supportive
and
then
you
have
these
vertigo
back-end
threads.
B
These
are
essentially
hey.
You
want
to
do
networking
somebody's
got
to
take
it
from
here
and
pass
it
here.
That's
come
to
the
back
end.
What
you're
going
to
do
with
your
vmm?
Those
are
gonna,
be
running
in
the
Pazzi
group
and
then
you
actually
have
a
guest,
colonel
and
then
you
finally
have
your
workloads.
B
So
there's
some
things
in
there
that
maybe
are
not
being
accounted
for
and
then,
if
you
were
to
look
at
gee
visor
what
they
do
is
they
have
like
a
century
available
which
is
kind
of
like
a
user
space
colonel
that
they
provide
and
that's
the
interface
that
they're
gonna
use
for
the
actual
container
workload.
So,
instead
of
calling
into
the
hosts
Linux
kernel,
they
say
that's
not
ideal.
What
we
would
like
to
do
is
use
this
kind
of
proxied
one
over
here
instead,
so
you
have
that
and
you
can
they
have
this
emulated
machine.
B
Sentry
run
a/c
sandbox
go
in
there
and
then
you
have
a
gopher
on
top
of
it,
which
similar
to
the
video
backend,
you
know,
you're
gonna
have
to
pass
and
proxy
some
stuff
in
and
out.
So
all
these
things
are
running
on
your
system
and,
finally,
you
know
we
came
from
a
client
node
up
above
and
went
all
the
way
down.
You
finally
have
your
10
line,
go
program
running
with
just
a
little
bit
of
overhead
and
a
little
couple
hoops
and
everything
else
to
go.
B
B
So
if
you
look
at
the
container
and
ask
each
one,
if
you're
in
the
run
C
case,
it's
easy,
you
can
just
go
into
a
subdirectory
go
into
this
container
slice,
which
is
a
child
of
the
parent,
which
is
the
pod,
and
you
can
just
add
those
up
and
say
that
should
be
the
total
up
here.
It's
not
so
when
you
just
do
the
summation
and
containers,
that's
not
really
gonna
be
reflective
and
the
way
it's
kind
of
handled
today
is
is
a
heuristic
kind
of
it's
a
minimal
overhead.
B
Don't
worry
about
it,
it
is
per
pod,
but
it's
so
small.
We
don't.
We
just
were
just
not
accounting
for
today.
So
what
we're
gonna
do
instead
is
we
kind
of
carve
up
the
node?
So
this
is
again
on
a
worker
node
and
you
have
cube
reserved,
you
have
system
reserved
and
they
may
have
allocatable.
So
in
your
I
think
it's
a
cubelet
configuration.
B
You
can
define
these
and
set
mine
where
you
said
so
you,
maybe
you
have
a
good
10,
gig
machine
or
something
you're,
just
gonna
say:
I'm
gonna
take
a
half
a
gig
out
of
the
pool,
don't
schedule
you
like
I'm
telling
cougar
Tiny's.
You
only
have
this
alligator
bull
pool,
just
ten
minus
half
a
gig,
and
so
when
you're
doing
scheduling
and
everything
else,
just
don't
pay
attention
to
that.
Only
use
this.
You
know
you're
gonna
be
full
because
it's
on
the
side,
this
kind
of
works.
B
Okay,
if
you,
if
using
just
run,
see
it
works,
okay,
you're,
not
directly
billing,
some
of
these
overheads,
but
it's
it's
it's!
Okay,
but
as
soon
as
you
do
sandbox
friend
times,
that's
not
gonna
work,
because
the
overheads
are
very
much
not
negligible.
So
what
we
did
was
create
a
feature.
The
feature
is
called
pot
overhead
and
it's
available
in
1.16
as
an
Elfa
feature.
117,
it's
gonna
be
alpha,
feature
didn't
get
to
it,
which
is
going
to
account
for
these
overheads
associated
with
running
a
pod
so
how
it
works.
B
First,
we
had
to
modify
the
pods
back
as
well
as
the
runtime
class
object
as
well
to
kind
of
add
a
couple
of
fields
to
track
pot.
Overhead
I'll
show
it
in
a
second
then,
after
that,
at
admission
time
you
know
we
don't
want.
Users
have
to
do
anything
or
know
anything
about
this,
so
they
should
not
ever
write
this
in
the
pods
back
it'll
get
rejected.
B
If
you
do
what
ends
up
happening
is
we
have
an
admission
controller
that
does
a
mutation
at
the
beginning,
and
it
says
if
you're,
using
a
runtime
class,
I'm
gonna
go
ahead
and
if
that
runtime
class
has
an
overhead
defined,
I'm
gonna
copy
it
into
the
pods
back
and
then
from
then
on
I'm
going
to
account
for
it.
So
what
does
this
look
like?
We
have
a
runtime
class
definition
up
above
I
hope
you
can
see
it.
Runtime
class
is
pretty
basic
and
it's
your
way
in
kubernetes
I.
B
Guess
I
meant
to
talk
about
this
a
bit
more
before
you
I
said
that
you
can
call
an
OC,
I,
runtime
CRI
right
CRI
calls
an
OS
yeah
that
one
say
actually
does
the
work
everyone
just
kind
of
passes
the
buck
down.
What
CRI
does
it's
very
important?
Is
it
will
look
at
the
pods
back
and
in
the
pods
back,
you
can
say:
I
want
to
use
this
runtime
class.
B
So
it's
a
way
that,
on
a
single
node,
you
can
support
multiple
OCI
runtimes,
which
is
great
because
maybe
for
most
your
workloads,
you
just
want
to
use
run
see,
but
one
of
them
super
sketchy.
So
you
want
to
use
the
sandbox
runtime,
that's
the
way
it
works,
so
runtime
class
is
the
way
that
we
do
this.
So
this
is
the
object
form
and
if
you
look
at
it,
we
added
the
last
four
lines:
very
exciting.
There's
overhead
pod
fixed,
which
I
can
talk
about
later
memory
and
CPU.
B
So
in
doing
this,
I'm
saying
that
this
is
using
katak
containers
with
the
fire.
Cracker
vmm
I
want
to
reserve
for
this
pot
and
extra
130
megabytes,
because
if
you
run
a
full
guest,
Linux
kernel,
sorry
it
costs
that
much
money
for
the
VM
m+,
the
guest
and
everything
else
and
from
a
CPU
is
kind
of
an
arbitrary
number
that
I
put
in
there.
Now
what
happens
is
again
from
a
user.
You
shouldn't
have
to
change
anything
I,
don't
want
you
to
have
to
worry
about
what
is
the
pot
overhead?
B
What
runtime
class
am
I
using
and
what
is
you
know?
They
just
shouldn't
be
involved
in
this
process,
so
all
that
it
has
is
requests
and
limits
for
your
actual
workload
itself.
So
this
is
the
traditional
what
you
should
be
doing
anyway,
just
saying
I'm
running
busy
box,
it's
gonna
take
100
Meg's,
that's
a
lot
and
100
millisievert.
That's
all,
and
you
see
that
it
has
a
runtime
class
name.
That's
defining
the
handler.
B
Interesting,
hey
I
want
this
workload
again,
don't
use
run,
see
I
want
to
use
kata
instead.
What
happens,
though,
is
you
apply
it?
It
goes
through
the
admission
controller
and
then,
if
you
go
ahead
and
pull
out
and
say,
give
me
the
description
of
this
pod.
What's
in
it,
let
me
let
me
get
some
enamel
you'll
see
that
now
the
overhead
is
actually
inside
of
there,
which
is
exactly
what
we
expect.
There's
also
a
validation
phase.
B
B
Cubelet
created
the
pod
c
group
for
me,
which
I'm
now
allowed
to
run
in
as
a
runtime
you'll
see
that
I
requested
100
millisieverts
and
what
I
have
instead
is
350
ml
of
CPUs
and
someone
math
for
me
230
Meg's
of
overhead,
so
it's
accounted
for.
That's
that's.
A
good
amount
of
space.
I
know
that
you
know
you
can't
run
the
VM
M
in
this
amount
of
RAM,
so
stepping
all
the
way
back
and
look
at
the
whole
picture.
B
So
a
lot
of
checks
of
saying,
if
feature
gates,
enabled
do
this,
so
they
had
to
be
added,
had
to
modify
the
pods
back,
which
is
part
of
the
core
API,
which
is
fun
because
then
I
had
to
go
through
an
API
review
and
everyone
was
really
nice
and
I.
It
went
well
and
then
runtime
class
had
to
be
updated
to
add
this
fields
as
well.
Api
server.
B
B
So
the
changes
in
the
code
itself,
they're
pretty
lame.
I
was
kind
of
surprised.
How
many
times
in
kubernetes
and
different
areas
of
the
project
they
do
the
same
exact
thing,
which
is
that
for
loop,
where
they
go
through
and
sum
up,
the
requests
of
all
the
containers
compared
against
any
containers,
take
the
bigger
of
the
two
and
then
return
and
say
this
is
what
a
pot
is
so
I
had
to
go
in
there
and
see.
Well,
technically,
it's
not
exactly
that.
B
We
also
want
to
take
into
account
the
overhead
of
the
pot
infrastructure
itself,
if
so
defined
in
the
feature,
is
enabled
so
really
complex
code.
If
statement
so
now
we
kind
of
get
into
the
who
cares
aspect
of
it.
So
as
a
user,
why
do
you
care
if
you
were
severely
over
constrained?
You
just
don't
run
a
workload
it
you
can't
there's
no
room
at
this
point
and
I'll
show
an
example
of
that.
B
If
you're
over
constrained,
then
it's
I
would
say
it's
even
worse,
because
at
least
in
the
first
case
you
fail
and
you're
like
okay,
that
didn't
work.
What
am
I
going
to
do
in
the
over
constrain
case?
What
you'll
have
is
inconsistent
and
kind
of
poor,
weird
performance,
because
essentially
what
you
do
is
you
have
a
workload
running,
but
at
the
same
time,
you're
starving
the
vmm
you're
starving
the
V
CPU
threads
you're
starving
the
virtio
backend
you're
starving
all
these
things.
B
B
B
I'm
not
used
to
the
pod
slash,
yet
I
still
have
to
type
it.
It
failed
kind
of
not
really
clear
a
typical
I
kind
of
failure
that
you
can
see
now
if
I
were
to
do
and
edit
oh
well.
Let's
just
look
at
the
actual
cgroups
itself,
yeah
you'll
see
this
is
the
40
Meg.
If
you
can,
I
can
do
Express
or
whatever,
but
that
that's
essentially
invites
how
much
room
is
in
the
pod.
See
group.
B
So
in
that
pod
see
group
of
40
megabytes
I'm
trying
to
run
a
be,
it's
gonna
launch
a
kernel.
It's
going
to
now
create
for
work
club.
That's
not
gonna
happen.
So
if
I
do
an
edit
and
instead
of
untying
class,
we
can
use
a
different
runtime.
This
is
just
gonna,
come
up,
I,
hope,
that's
the
goal
of
what
I
want
you
to
see.
It
takes
a
while
yeah,
okay
and
you
check
it.
C
B
B
So
that's
you
know
a
user,
but
administrators
should
really
care
too,
because
if
you
get
into
let's
say
just
like
resource
quota,
what
you're
gonna
end
up
having
is
that
a
user
is
gonna,
come
in
request
a
small
workload
and
you're
gonna
say:
that's
fine,
they're
gonna
ask
you
now
they're
going
to
say
it's
fine
and
then
they
have
another
namespace
for
their
other
use.
And
you
say:
that's
fine!
That's
fine!
Because
I
have
this
cube
reserved,
that's
totally
fine,
but
all
the
sudden,
your
actual
system
is
severely
over
constrained
on
memory.
B
You
have
memory
pressure
and
maybe
your
control
planes
is
getting
requests
and
everything
else.
So,
if
you're
not
taking
an
account,
you're
gonna
get
an
in
stable
system,
but
for
me
yeah,
you
can
work
through
this,
and
this
is
in
an
example.
So
I
talked
about
the
infer
container.
I
just
ran
a
couple
of
pods
using
a
few
different
runtimes
and
this
is
using
run
C.
B
So
what
it
is
is
the
bottom
one
is
container
1
container
2
is
stacked
on
top
of
that,
and
then
the
blue
line,
that's
the
pod,
so
I'm
measuring
the
C
groups,
I'm,
comparing
it
against
what
the
pod
shows-
and
this
is
the
difference
and
it's
about
a
megabyte-
that's
not
really
being
accounted
for
necessarily
directly,
and
that's
because
of
that
in
for
a
container
people
are
working
on
ways
to
kind
of
pull
that
back
out.
So
we
don't
need
to
do
it.
B
So
there's
almost
zero
overhead
for
run
C,
but
well
that's
the
long
going
from
a
CPU
utilization
standpoint,
there's
really
not
much
overhead.
If
we
go
over
to
look
at
qmu
there's
a
tiny
bit
which
kind
of
makes
sense,
it's
like
a
fraction
but
you're
still
almost
using
just
200
mil
CTU,
which
is
what
the
workload
is
allowed
to
do.
If
you
look
at
memory,
this
is
a
horrible
picture
and
I
thought
about
like
quickly
running
like
elastic
or
something
like
that.
B
B
Sorry,
it's
gonna
cost
you
almost
a
little
bit
over
150
megabytes,
and
you
know
this
is
this
is
what
you
think
you're
doing
is
for,
but
it's
actually
doing
a
hundred-fifty,
and
if
you
look
at
firecracker
it
saves
about
30
megabytes
footprint,
so
you'll
see
a
little
bit
less,
but
so
obviously
kind
of
too
much
to
ignore.
From
a
provider
standpoint
you
can
just
say
I'm
going
to
be
very
aggressive
and
reserved
a
lot
in
Cube
reserved
and
system
reserved
and
everything
should
be
okay.
B
B
If
you
just
lump
sum
of
these
overheads
in
to
know
where
you're
gonna
advertise
that
cost-
maybe
two
everybody
has
to
eat
it
or
you
just
eat
it
as
a
provider
versus
if
you're,
actually
looking
at
pod,
C
group
and
everything
is
sized
appropriately.
So
you
can
use
a
pod
C
group
for
what
it
is
to
host
the
infrastructure
for
running
a
pod.
Now
you
can
say
on
a
per
workload
basis.
B
This
is
the
cost
on
the
system,
so
you
don't
have
to
conservatively
over
block,
but
you
can
actually
have
it
in
there
and
start
charging
people
for
this,
which
is
better
accounting
as
well
as
stability,
so
the
status
of
the
feature
it's
in
116
again,
it's
an
alpha
feature.
Overheads
are
static,
which
is
not
realistic,
but
it's
a
really
good
standpoint.
I'd
be
happy
to
talk
more
after
about
that.
B
I
think
you
use
like
vertical
pod,
auto
scaling,
and
things
like
this
as
long
as
it
overheads
enough
to
have
a
stable
system
vpa
should
take
over
and
do
the
rest
or
horizontal
or
whatever
you
want
to
do.
But
basically,
if
you
have
a
lot
of
network,
I/o,
you're
gonna
see
more
V,
CPU,
overhead
or
CPU
overhead
on
hosts,
but
anyway
it
I
expect
it
to
move
to
beta
in
the
118
release.
So
because
of
that
again,
it's
alpha.
You
have
to
enable
it.
B
So
the
know
parts
of
it
that
we
changed
were
the
cube
scheduler,
the
API
server,
as
well
as
cubelet
itself.
So
if
you
go
through
here
like
a
human
iam,
yeah
Mille,
you
would
just
have
to
enable
this
feature:
gate
for
those
particular
paddock
components
and
that's
all
the
content.
They
have
I'm
happy
to
answer
any
questions.
Folks
have
yeah.
D
E
B
It
depends
I,
I,
didn't
really
get
into
some
of
the
CRI
I
have
like
kind
of
different
monitors.
You
would
have
running
as
well,
which
is
not
negligible.
You
can
still
get
in
this
case
where
it's
pretty
dirty,
where
essentially
the
pot
is
saying
I
get
this
much
I'm
using
all
of
it,
absolutely
all
of
it.
So
then
you
look
at
the
parent,
C
group
and
they're,
using
all
of
it
and
you're
trying
to
run
anything
else.
That's
pretty
sensitive,
then.
B
Gonna
touch
it
until
it's
at
alpha,
so
I
would
imagine
some
of
those
people
would
like.
Oh
it's
interesting,
maybe
it'd
come
118
it'd
be
utilized
I.
Imagine
they
probably
have
as
a
service
product.
They
probably
have
a
lot
of
extra
plumbing,
underneath
it
to
do
this
accounting
side
of
it
already,
but
I
don't
have
a
lot
of
visibility
and
I'm
really
curious
to
you,
though,.
F
B
B
Not
really
you
can
easily
go
through
I
set
up
that
gharana,
where
we're
essentially
scraping
what
the
containers
say:
they're
doing
versus
what
the
pods
the
adviser
is
scraping
and
you
can
compare
and
just
calculate
it,
but
you
really
would
have
to
look
at
what
is
a
characteristic
workload
as
well,
because
it
can
be
dynamic,
but
right,
so
your
mileage
kind
of
will
vary
if
you're
gonna
have
a
lot
of
I/o
dependent.
You
don't
want
to
bump
up
the
CPU
overhead
and.
F
B
B
I
think
that
what
you're
gonna
see
is
it's
gonna
depend
on
the
workload
this
running
and
in
order
to
do
that
in
my
world,
set
the
overhead
for
it's
the
right
size
and
then
use
VP,
a
and
VP
a
is
going
to
bounce
the
pod
C
group
size
up
and
down
to
meet
your
QP
s
or
what
every
metric
are
using
to
drive
vertical
pot
on
the
scaling,
I
think
with
those
two.
It's
it's
a
pretty
cool
story.