►
From YouTube: Inside Kubernetes Resource Management (QoS) – Mechanics and Lessons from the Field - Michael Gasch
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Inside Kubernetes Resource Management (QoS) – Mechanics and Lessons from the Field - Michael Gasch, VMware (Any Skill Level)
Kubernetes Quality of Service (QoS) offers powerful primitives for resource management, that is workload prioritization, fairness and efficiency. But it's also a complex topic to understand and get right in production, e.g. if you are new to this topic or running highly dynamic and distributed systems. Getting the most value out of a Kubernetes cluster requires utilizing the features provided to categorize and prioritize your workloads. We'll look at how Kubernetes provides this functionality through its QoS implementation. Both, from an end-user's perspective but also digging into the mechanics. The work doesn't stop there though. This talk will also explore techniques on how to tune your application using best practices and lessons learned in the field. Finally, we'll provide community resources and an outlook about taking QoS to the next level in your cluster.
About Michael
Michael Gasch is an Application Platform Architect and member of the CTO Ambassador program at VMware. He supports enterprises with architectural guidance, and works closely with VMware R&D. Right from the beginning of his career, he built and operated very large Linux distributed storage and vSphere virtualization platforms. Lately, in a customer and developer-focused role, he went back to his Linux roots in the field of distributed systems, container orchestration and scheduling. He´s actively tweeting about all things distributed under his Twitter handle @embano1. When time permits, he presents at large industry events (VMworld), but also community events like Kubernetes, AWS and Docker meetups, and vBrownbag Youtube series. You can find his blog, dedicated to deep-dive container internals, here: https://embano1.github.io/
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
everyone
thank
you
for
coming
to
the
session,
especially
on
a
Friday
afternoon.
After
there's,
amazing
coop,
con
I
hope
you
had
a
good
time.
I
certainly
did
I'm
Michael
working
for
VMware,
and
today's
session
is
about
kubernetes
resource
management,
mainly
from
like
the
mechanics
behind
kubernetes
resource
management,
as
well
as
some
of
the
lessons
learned
and
best
practices
that
I
learned
from
working
with
customers
in
the
field.
A
Before
we
kick
off,
I
just
wanted
to
get
a
feeling
of
the
audience
developers
in
the
room.
Please
raise
your
hands
what
it's
that's:
a--that's,
almost
50%,
good
operators,
Zuri's
architects,
this
group
of
people,
it's
a
little
bit
more;
ok,
nice
good,
so
who?
Who
has
heard
about
kubernetes
resource
management,
the
primitives
just
whose
family
I
was
this
topic?
A
A
Much
bigger
picture
of
resource
management,
mainly
what
we
can
cover
here,
is
capacity
planning,
tooling
processes,
change
management
which
all
play
into
this
huge
topic
of
resource
management.
The
agenda
is,
or
will
be
like
a
short
introduction,
then
we'll
cover
kubernetes
primitives
as
well
as
how
they
are
implemented,
and
then,
at
the
end
of
the
presentation,
we'll
have
best
practices
and
hopefully
time
for
Q&A.
A
So
this
is
the
and
I'm
sorry
if
it's
not
readable
in
the
back,
but
I
just
wanted
to
show
you.
This
is
the
kubernetes
engine,
X
ingress,
spec
deployments
back,
and
if,
since
this
is
a
talk
on
quality
of
service
and
resource
management,
can
someone
spot
what's
missing
in
this
deployment
file?
Just
shout
it
out
loud?
If
you
want
great.
Thank
you
very
much,
there's
something
missing
in
here.
So
what
if
you
deploy
the
engineer's
controller
with
this
kind
of
manifest?
A
What
what
could
happen
if
you
send
it
to
the
API
server
during
admission,
this
deployment
or
part
at
the
end
of
the
day
could
be
rejected
because
that
might
be
the
resource
for
the
admission
controller
be
activated.
It
could
be
modified
by
the
limit.
Ranger
another
admission
plug-in
and
the
API
server,
both
of
them
we'll
cover
later,
if
it
made
it
through
and
it's
running
this
part
might
not
get
enough
resources
because
it
didn't
ask
for
any
resources,
so
it
might
be
starved.
A
It
could
affect
other
parts
running
on
the
same
cubelet,
so
it
become
a
noisy
neighbor,
for
example,
of
its
bursty.
It
could
be
affected
by
the
cube
evicted
by
the
cube,
let
when
the
cubelet
comes
under
pressure
or
it
could
be
killed
by
the
Linux
kernel
as
a
last
line
of
defense.
So
basically,
this
pod
won't
give
you
any
kind
of
predictable
runtime
behavior,
but,
depending
on
the
workload
that
you
want
to
run
so
not
necessarily
the
nginx
but
maybe
batch
jobs.
A
A
For
these
people,
all
the
links
by
the
way
are
in
the
paintings
of
this
presentation.
It's
already
online,
so
just
have
a
look
at
the
appendix
so
some
operating
system-
basics.
First,
before
we
deep
dig
into
kubernetes,
if
you
run
a
container
say
you
talk
around
hello
declined,
the
docker
client
sends
a
create
command
to
the
docker
engine,
and
this
happens
in
userspace
and
after
some
initialization
in
the
docker
engine.
Eventually,
a
Cisco
X
act
will
be
called
for
the
entry
point
or
command
specify
on
it
in
the
in
the
docker
file.
What
happens,
then?
A
Is
we
switch
in
the
kernel
mode
and
I
left
out
some
some
of
the
details
here,
but
we
switch
in
the
kernel
mode
and
now
the
kernel
takes
over
and
creates
toss
struct
for
this
command
and
by
narrated
that
we
want
to
run
so.
Basically,
the
colonel
doesn't
really
know,
or
has
this
notion
of
thread
and
processes.
A
What
he
builds
in
memory
is
at
a
struct
which
has
some
properties
and
I
just
sketched
out
summary,
which
is
C
groups
in
namespaces,
some
of
you,
you
have
might
have
heard
already,
and
then
it
needs
to
find
a
resource
to
run
our
container
or
our
binary,
so
it
kicks
in
the
scheduler
the
scheduler
kicks
in
of
the
Linux
kernel.
It
finds
the
CPU
hopefully,
and
if
it
found
the
CPU
its
kind
of
schedules,
the
process
or
the
the
task
struct
on
the
CPU
and
eventually
we'll
go
running
and
print
hello
world.
A
So
Linux
kernel
is
just
concerned
mainly
around
a
task
rack
here
is
because
I'll
get
a
lot
of
question
of
ok.
This
is
the
process,
and
this
is
this
right.
So
what
are
we
dealing
was
basically
we're
dealing
with
tossed
trucks
and
also
we
are
not
dealing
with
containers
in
the
sense
of
the
Linux
kernel.
The
Linux
kernel,
as
of
now
doesn't
really
have
a
notion
of
container
it's
just
a
set
of
properties
in
the
task
struct,
so
that
might
sound
easy
right
now.
A
As
a
concrete
example,
imagine
we
have
one
CPU
and
we
want
to
run
two
containers
which
are
pretty
CPU
bound
suppose
want
to
keep
keep
two
CPU
busy.
What
the
Linux
kernel
in
the
default
scheduling
algorithm,
which
is
a
completely
fair
scheduler,
tries
to
do.
Is
it
just
wants
to
be
fair
to
all
the
task
drugs
it?
It's
asked
to
run
so
it
puts
in
a
weight
share
so
to
say
which
you
can
find
here
and
by
default.
That's
1024.
A
If
the
shares
are
equal,
that's
basically
assign
for
the
color
to
give
both
the
same
amount
of
CPU
time.
Now,
that's
easy.
If
just
our
containers
want
to
run
each
a
toss
track
or
just
consider
a
construct
of
this
one
task
right.
But
what
if
our
container
array
is
a
little
bit
smaller
and
spin
small
threats
and
user
lingo?
Basically
meaning
that
you?
It
creates
more
task
trucks
now
the
Linux
kernel.
A
If
we
don't
have
a
groovy
magnet
mechanism,
it
tries
to
be
fair
to
all
the
task
tracks
and
now,
certainly
we
don't
we're
not
very
fair
between
the
two
processes,
because
container
II
was
the
more
task.
Drugs
gets
more
CPU
time
the
end
of
the
day
so
10
years
ago,
Google
contributed
cgroups
to
the
Linux
kernel
and
basically,
what
that
means.
A
It
kind
of
implemented
the
grouping
mechanism
around
these
da
structs
that
now,
at
a
group
level,
you
can
define
your
weights
or
shares
that
specify
the
amount
of
CPU
time
those
groups
get
and
we
kind
of
back
into
fairness.
See
groups
also
can
be
used
for
accounting
and
other
types
of
resource
management.
So
it's
not
always
a
mechanism
to
or
needed
for
resource
management.
A
You
can
also
just
use
it
for
some
statistics
and
apply
some
statistics
coverage
is
there,
but
mainly
we
use
C
groups
for
a
CPU
and
memory
and
IO
control
where
you
can
specify
priorities
or
limits
on
certain
resources,
for
example,
also
on
P
IDs
and
you
can
do
accounting.
So
it's
a
pretty
sophisticated
thing
that
we
have
in
a
Linux
kernel
scheduler
to
work
with
resources.
A
Usually
it's
mounted
in
disease
FS,
so
the
hierarchy
and
the
interface
that
we
can
interact
from
userland
is
mounted
in
disease
FS
and
there's
two
versions
that
exist
in
Linux
kernel
version.
2
has
just
been
finalized
and
one
of
the
reasons
Linux
kernel
versions.
As
far
as
I
know,
most
of
the
container
runtimes
still
use
version
1
of
this
secret
interface.
A
Now,
if
you
want
to
run
a
container
just
to
switch
more
on
kano
world
now
you
got
those
flags
like
CPU
and
memory,
for
example.
Here
we
want
to
run
busybox
and
execute
top
and
busybox,
and
we
say
just
give
me
one
CPU
and
turnip
megabytes
of
memory.
What
ends
up
happening
is
that
top
in
the
busybox
container
still
sees
all
of
our
CPU,
which
are
in
anode,
and
it
sees
one
point
ago.
You
could
get
a
gigabyte
of
memory
instead
of
the
memory
that
we
specified.
A
So
first
thing
to
note
here
is
that
C
groups
are
not
named
spaced,
so
the
containers
is
kind
of
all
the
resources
on
the
host
and
not
from
a
from
its
container
isolation.
Perspective
and
second
thing
to
note
here-
is
and
I'm
going
to
read
it
for
those
people
in
the
back
us,
since
the
container
is
kind
of
grouped
in
this
secret
hierarchy,
which
is
sis
of
SC
group
CPU
docker
in
this
case,
and
to
get
some
properties
that
tell
the
Linux
kernel
how
to
regulate
resources
on
this
container.
A
In
this
case
it
gets
a
CPU
share
of
1024.
That's
the
default
value,
but
it
gets
some
limits
applied
mainly
a
CFS
period
and
quota,
which
is
some
kernel
internals
that
mean
that
everyone
had
millisecond,
evaluate
the
quota
of
this
container
and
if
it's
beyond
this
quota,
just
take
it
off
and
and
don't
give
it
more
time
now
coming
from
a
VM
centric
world,
where
we
run
one
app
VM.
We
had
this
reservations,
typically
that
we
have
in
the
VN
like
give
me
four
CPUs
and
forget
about
the
memory
here.
A
In
this
case
it's
kind
of
the
inverse.
We
specify
some
hard
limits,
some
hard
caps
on
the
on
our
process
on
our
container,
which
you
might
get,
but
imagine
you
have
just
one
CPU
and
you're
on
ten
of
these
containers.
The
one
CPU
doesn't
give
you
one
CPU.
It
means
basically
you
at
best
time
your
maximum
get
one
CPU,
but
you
could
get
less
because
if
you
have
more
processes
running
everyone
gets
a
fair
share,
also
from
memory
here.
It
will
specify
a
limit.
So
it's
not
a
guarantee.
It's
actually
a
limit.
A
That's
been
applied
was
the
flag,
and
so
this
is
some
of
the
misunderstandings
that
are
typically
seen
in
the
field
when
people
start
with
containers
that
they
come
from
this
VM
reservation
model.
Now
they
are
into
the
kernel
model,
use
those
flags,
but
actually
the
flags
are
kind
of
the
inverse.
What
they
expect
also
keep
in
mind
that
we
don't
specify
gigahertz
or
anything
else.
Here
we
do
specify
as
a
kind
of
a
time
a
quote
our
period,
which
could
vary
between
your
infrastructure
say.
For
example,
you
run
this
container
on
your
local
machine.
A
Was
this
flag
that
could
be
1.8,
gigahertz
machine
or
on
the
cloud?
Could
be
one
point
whatever
machine,
so
it's
not
really
a
megahertz
or
gigahertz
that
you
specify
here
and
the
hard
limit.
That's
what
we
already
covered.
Just
to
recap.
This
introduction
here
from
the
Linux
kernel
point
of
view,
containers
are
just
normal
processes
or
task
tracks
to
be
precise
here,
the
Linux
kernel
default
algorithm
is
completely
fair,
so
the
motels
tracks
are
running
at
recipe,
fab
beyond
them
or
amongst
them,
and
you
can
influence
this
behavior
by
tuning
those
shares
continues.
A
You
see
groups
and
namespaces
for
prioritization
and
isolation
on
the
host,
but,
for
example,
for
C
groups,
there's
not
always
isolation,
as
we've
seen
with
the
busybox.
Another
biggest
question
is:
okay.
I
got
this
for
containers
on
my
machine,
but
how
do
we
do
that
at
cluster
scale?
And
this
is
where
kubernetes
quality
of
service
comes
comes
into
game?
I
made
this
slide
for
1.10
I,
just
changed
them
for
one
to
ten
and
I
hope.
I
have
covered
everything
and
the
reason
changes
there.
A
So
in
kubernetes,
an
equivalence
class
that
typically,
what
we
want
to
achieve
is
third
level
of
isolation
between
ten
ends
or
different
namespaces
or
teams.
We
want
to
have
prioritization.
We
want
to
apply
some
quotas
so
that
some
team
cannot
use
more
than,
for
example,
eight
CPUs.
We
want
to
be
fair
between
them,
but
also
we
want
to
increase
efficiency
in
the
cluster.
That
example
of
one
team
is
not
using
its
resources,
the
other
one
can
make
use
of
the
idle
resources.
A
So
there's
a
lot
of
things
that
we
want
to
achieve
in
a
kubernetes
cluster
and
those
are
things
that
might
not
always
work
together.
Nicely
also
keep
in
mind
all
the
premise
that
primitives
that
we're
going
to
cover
it
next
are
just
at
the
cluster
boundary.
So
if
you
have
multiple
clusters,
you
have
to
think
about
how
to
make
this
kind
of
resource
management
work
across
cluster
boundaries
and
if
you
run
in
a
cloud
or
in
a
hypervisor,
typically
that's
what
the
underlying
infrastructure
does
for
you.
A
On
the
right
hand,
side
you
can
obviously
kind
of,
in
which
phase
view
are
currently
in
community
stays,
stable
resources
for
specifying
resource
requirements,
and
those
are
CPU
and
memory.
We've
also
got
some
beta
resources,
which
is
used
huge
pages,
ephemeral,
storage
and
device
plugins,
and
yesterday
there
was
a
great
talk
by
Louise
from
Intel,
and
she
covered
a
lot
of
the
very
details
and
performance
tuning
aspects
of
some
advanced
techniques
like
new
mask
railing,
NUMA
tuning,
huge
pitches,
etc
was
a
great
talk.
I
also
have
some
custom
resources.
A
For
example,
you
want
to
have
some
some
very
specific
companies,
specific
resources
that
only
you
you
have.
Those
are
extended
resources
that
you
can
use
in
a
cluster,
for
example,
for
licenses
or
o'donnell's.
So
you
can
can
extend
on
the
resource
properties
for
each
resource
in
kubernetes
in
the
spec
we
have
requests
and
limits
I
usually
see
a
requests,
are
kind
of
guarantees,
I'm
gonna
expect
from
from
kubernetes,
and
the
limits
are
the
hot
cap
that
my
process
should
not
get
over
or
cross
over.
A
There
was
a
specified
at
the
container
level,
so
they
are
not
specified
at
a
pod
level,
but
at
a
container
level,
and
currently
only
CPU
memoranda,
ephemeral,
storage,
allow
for
over-commitment,
meaning
that
you
can
specify
different
requests
and
limits.
You
can
send
certain
the
same
which
you
have
to,
for
example,
for
GPUs
which
doesn't
allow
for
overcommit,
but
you
also
can
say:
I
just
want
one
CPU
as
a
request
for
my
pod
and
maybe
four
CPUs
as
a
limit,
so
you
cannot
converse
in
between
them.
Those
fields
are
read-only.
A
So
if
you
want
to
change
them,
you
have
to
recreate
the
part
of
the
deployment.
So
it's
controls.
For
example,
all
the
things
that
you
just
didn't
found
here
in
the
spec
CPU
memory,
things
that
you
didn't
found
can
be
may
be
set
by
sis
control.
So
this
link
has
a
link,
a
list
of
all
the
things
that
you
can
set
from
the
SIS
control
perspective,
either
per
container
or
on
the
host.
A
And
then
there's
always
the
question
like
how
can
I
set
bandwidth
guarantees
or
thresholds
or
limits,
for
example,
for
disk
and
network,
and
that
turned
out
to
be
not
so
easy
and
there's
also
discussion
on
whether
it
will
eventually
this
will
ever
make
it
into
kubernetes.
So
this
link
has
a
design
guide,
a
discussion
on
why
this
is
currently
not
implemented
as
a
resource
here,
just
as
an
example,
we
have
an
engine
expert
that
does
specify
limits
in
this
case.
Keep
in
mind
if
you
only
specify
limits,
kubernetes
sets
request
equals
limit.
A
So
you
that's
just
a
shortcut
and
not
so
you
don't
have
to
specify
a
request
in
this
case,
our
container
will
get.
We
ask
for
two
CPUs
and
two
megabytes
of
memory,
meaning
as
a
request,
but
also
as
a
as
a
limit
in
this.
In
this
specification.
Okay,
we
got
our
specification.
We
have
our
pot
yellow
file
now
we're
going
to
submit
it
to
the
API
server.
A
The
API
server
has
different
admission
controls
that
you
might
be
aware
of
and
to
add
mission
controls
that
we
are
concerned
here
are
the
resource
quota
at
mission
controller
and
the
limit
arrangement
which
we
cover
later
our
source
quota.
Admission
controller
allows
you,
as
a
cluster
operator,
to
set
a
certain
quota
on
a
namespace
which
names
based
resource
right
now
that
you
can
say
this
team,
this
namespace
should
never
ever
get
more
than
8
GPUs
or
20
pods
or
whatever.
But
you
can
also
set
some
requests
as
a
upper
bound
of
boundary.
A
A
Keep
in
mind
when
you
do
those
sizing
and
calculations
or
specifications
as
a
cluster
operator,
keep
in
mind
that
there's
currently
no
namespace
resource
sharing
that,
for
example,
if
you
set
a
namespace
on
eight
CPU
and
the
other
one
on
four
CPU,
but
the
for
CPU
namespace
is
kind
of
on
its
limit
and
the
other
one
is
idle.
It
cannot
currently
make
use
of
the
idle
resources
in
another
namespace
and
there's
a
community
project
I'm
going
to
mention
later,
which
tries
to
resolve
this.
A
So
be
it
be
aware
of
how
to
size
the
quota
on
the
namespaces.
Now
we
go
into
the
scheduling
phase.
The
scheduler
looks
at
all
the
nodes
in
your
cluster
and
kind
of
builds
and
note
info
cache
and
memory.
So
it
has.
It
knows
about
each
node
and
in
its
memory,
but
it
looks
only
at
the
capacity
like
what
does
this
node
have.
A
What
is
allocatable
allocatable
is
the
at
the
end
of
the
date
is
sommore
the
rest
of
everything
that
the
note
has
minus
some
reservations
that
you
can
apply
minus
some
eviction
threshold,
which
we'll
cover
later
so
allocated,
will
typically
is
lower
than
the
whole
note
capacity,
and
this
caused
some
confusion
for
some
customers,
like
I,
do
have
capacity
in
my
cluster,
but
schedulers
nots
gathering
it
it's
just
because
the
scheduler
uses
allocatable
and
not
default.
Note
capacity
during
scheduling
the
scheduler
has
some
credit
cards
and
priority
ranking.
A
We
don't
have
time
to
look
into
the
scheduling
algorithm.
There
was
a
talk
yesterday
in
six
scheduling
which
covered
this
in
detail,
but
what
a
scheduler
does
it
looks
at
the
resource?
Requests
of
the
incoming
part
doesn't
come
is
not
concerned
about
limits
right
now.
It
just
looks
at
the
requests
coming
in
and
it
looks
at
the
allocatable
resources
that
it
tracks
in
memory
for
each
node
and
it
starts
while
scheduling
it
would
never
over
commit
a
node
based
on
the
request.
A
A
So
now
on
the
node,
which
is
where
the
magic
actually
happens,
we
are
on
the
couplet
now
see,
groups
are
used
on
the
cubelet
to
enforce
most
of
what
we've
covered
so
far,
around
CPU
and
memory
I
created
this
diagram,
because
I
found
it
very
hard
to
figure
out
what
CPU
requests
or
memory
requests
and
limits
map
to
actually
on
the
node.
I
think
this
is
there
some
room
in
the
documentation
to
improve
this
so
on
the
kind
of
the
upper
layer?
Is
your
pot
manifest
which
specifies
CPU
requests
and
limits?
A
Also
for
memory
and
internally
they
map
on
the
nodes
on
the
cubelet,
the
couplet
maps,
those
two
CPU
shares
for
West
and
CPU
quota
and
period
for
the
limits
for
the
memory.
It's
a
little
bit
different,
because
the
Linux
kernel
has
no
memory
kind
of
request
or
a
guarantee
in
there,
because
it's
also
a
shared
resource,
so
requests
map
to
something
called
out
of
memories,
go
adjust
or
cover
that
later
and
limits
map
to
memory
limits.
A
So
here
we
run
an
engine
X
container
with
limits
CPU
1
and
memory
200
megabytes,
and
we
just
want
to
dive
into
how
this
looks
how
the
couplet
translates
this
into
the
actual
actual
secrets.
The
cubelet
does
adjust
CPU
shares
in
this
case.
Actually
it's
1024,
but
if
you
would
have
specified
something
else
here,
the
couplet
does
some
adjustment
on
the
CPU
shares
and
for
the
CFS
period
and
quoted
also
applies
the
limits
in
this
case.
So
again,
here
we
get.
We
are
hot
kept
at
one
CPU.
A
This
part
won't
ever
get
more
than
one
CPU
for
memory.
These
are
also
applied
as
a
upper
binary
in
this
case.
To
have
megabytes
so
again
here
we
are
applying
capacity
limits
to
our
port
and
as
a
guarantees
of
guarantee.
We
just
have
to
use
shares
in
this
case
now.
It
can
be
a
little
bit
complicated
to
work
with
this
requests
and
limits
and
figure
out
which
kind
of
pot
has
which
resource
guarantees
or
which
one
has
a
higher
priority
or
lower
priority.
So
very
early
queue.
A
Quality
of
service
classes
were
incorporated
into
kubernetes
that
make
it
a
little
bit
easier
or
from
an
end
user
perspective
to
see
which
kind
of
resource
guarantees
or
classes
I
I
get,
but
also
for
the
couplet
to
decide
which
part
has
a
higher
priority
or
not.
Those
classes
are
derived
from
CPU
and
memory
resource
pacification
in
the
in
the
pods
back,
and
just
to
give
you
some
examples,
if
you
don't
specify
anything
for
because
limits
for
a
container,
you
are
a
best-effort
pot
classified
best
effort.
A
If
you
specified
some,
you
are
a
burst
of
all
pot
and
if
you
specify
all
request
limits
for
CPU
and
memory
for
all
containers
in
this
pot,
then
you
are
considered
to
be
a
guaranteed
pot.
Now,
there's
some
tricky
magic
happening
inside
which
we'll
cover
in
a
second.
First
of
all,
the
couplet
builds
a
hierarchy
on
locally
on
the
node,
which
you
can
see
here
on
the
right,
meaning
that,
for
example,
our
guaranteed
pot
will
be
living
directly
in
the
root
hierarchy
of
cube
pots.
A
So
the
higher
you
derive
hierarchy,
you
typically
have
a
higher
resource
guarantee
or
resource
yeah
guarantees
from
the
cubelet,
and
then
it
builds
a
sub
hierarchy
for
best
effort
and
worst
of
all,
which
it
continuously
adjusts
values
in
there.
So
the
couplet
does
a
lot
of
more
magic
than
I'm,
showing
here
and
I.
Think
on
the
next
slide,
there's
also
a
link
of
how
the
cubelet
does
this
magic
under
the
cover.
A
So
now,
for
since
we
talked
about
resources
of
a
pot
with
requests,
will
never
be
over
committed
what
if
they
actually
do
over
commit,
you
specify,
requests
and
limits,
and
you
actually
roll
out
pots
on
on
a
node
that
has
not
so
many
resources
that
you
specified
in
the
limits.
You
have
to
be
aware
of
that.
There's
kind
of
two
types
of
resources:
compressible
resources
and
uncompressible
resources
CPU.
As
a
compressible
resource,
meaning
we
can
throttle
it
in
case
of
there's
contention,
memory
and
storage.
We
cannot
shrink
it
or
compact
it.
A
So
that's
uncompressible
and
we
have
to
have
to
find
another
way
to
make
space
in
case
of
there's
contention
on
the
couplet.
So
for
CPU
throttling
the
the
kind
of
the
value
that's
taking
encountered.
Cpu
shares
the
more
shares
you
have
and
the
CPU
has
to
be
throttled.
The
more
time
you
had
at
the
end
getting
through
in
this
even
during
the
earth
rattling.
But
for
memory
and
storage,
the
cubelet
has
to
evict
start
evicting,
make
space
on
the
on
the
and
as
eviction
logic
applied
here.
A
If
the
cupid
cannot
evict
fast
enough,
because
there's
kind
of
I
think
10
sec,
10
second
polling
evolved
in
here.
Currently
the
Linux
kernel
will
just
kick
in
and
out
of
memory
kill
as
a
last
line
of
defense,
which
is
kind
of
not
kind
of
synced
this
the
cubelet,
so
the
Linux
kernel
could
just
kick
in
and
do
its
job.
Now
the
eviction
threshold
can
be
specified.
The
cubelet
has
some
flex
that
you
can
use
for
eviction
threshold.
A
So
you
actually
have
some
control
for
the
eviction
eviction,
behavior
and
also
the
queue
blood
when
under
pressure
will
signal
to
the
API
server
that
it's
under
pressure.
The
scheduler
will
take
this
into
account
and
not
schedule,
hopefully
not
schedule
pots
on
the
dis
note,
because
that
would
make
the
situation
even
worse.
A
So
it
was
that
the
quality
of
service
best
effort
was
likely
to
be
evicted
first
and
then
most
able
and
guaranteed
as
the
last
line
of
defense.
That
was
pretty
pretty
easy
to
understand
now
that
changed
in
1.9.
Where
actually
there's
the
question,
the
cupid
asks,
which
part
has
the
highest
usage
above
its
request.
Typically,
that
would
be
possibly
at
the
staff
report,
but
it
could
also
be
a
burstable
part,
so
the
QoS
classes
here
kind
of
are
not
really
involved
anymore.
Then
it
looks
for
pot
priority.
A
If
you
don't
have
pot
priority,
enabled
in
your
cluster
and
you,
it
will
just
skip
it
and
then,
as
a
last
line
it
just
you
a
usage
miners
request
and
the
pots
that
remain
are
then
ranked
so
the
eviction
logic
has
changed
slightly,
and
it
could
also
mean
that
guaranteed
pot
will
be
evicted
because,
for
example,
you
have
a
system,
demon
or
system
process
on
the
node
which
is
kind
of
struggling,
and
then
the
queue
blood
will
make
space
for
Mike
by
killing
even
guaranteed
pots.
Now
keep
in
mind.
Daemon
sets
are
also
pots.
A
The
Kuebler
doesn't
understand
that
this
is
a
demon
set.
For
example,
it
just
sees
a
pot
make
sure
that
for
demon
sets,
you
also
apply
at
least
some
sunburst
uber
or
guaranteed
resources,
because
otherwise
you
could
end
up
in
killing
B
being
evicted.
Your
demon
set,
which
could
be
your
network,
plug-in,
for
example,
there's
much
more
going
on
that.
We
can
cover
here
on
the
covers
during
this
30min
I
mean
so
I
made
this
slide
just
as
a
reference,
it's
all
it's
all
in
the
deck,
so
that
could
be
helpful
for
troubleshooting.
A
If
there's
something
that
you
have
to
log
in
to
also
the
community
has
seen
that
the
some
of
the
parameters
that
there
are
in
kubernetes
for
resource
management
might
not
be
sufficient.
So
the
created
project
or
extensions
like,
for
example,
the
kubernetes
arbitrator,
which
allows
for
namespace
sharing
that's
pretty
interesting.
The
CPU
manager,
which
kind
of
flips
from
those
shares
which
are
just
weights
which
are
relative
weights
into
CPU
pinning.
So
some
of
you
might
be
familiar
with
CP
opening
the
soup
you
CPU
manager
will
allow
for
CPU
pinning.
Then
you
actually
get
a
CPU.
A
If
you
ask
for
a
CPU
for
CPU
dedicated
what
priority
is
something
that
I
was
really
happy
to
see
coming
into
the
cube
scheduler,
which
allows
for
adding
priority
feels
to
your
workloads
to
your
pots,
saying,
there's
a
system
critical,
and
so
you
have
an
easy
way
to
specify
your
kind
of
priority
so
that
the
that
will
also
be
taking
into
account.
For
example,
during
eviction
and
the
vertical
pot
autoscaler
is
actually
starting
to
look
at
the
metrics
to
use
resource
utilization
in
your
cluster
and
trying
to
find
the
right
requests
and
limits.
A
So
you
don't
have
to
always
watch
and
specify
them
manually,
exciting
projects,
so
some
best
practices
from
the
field
before
we
wrap
up.
If
in
doubt,
if
you
don't
use
quality
of
servers-
or
there
was
the
primitives
in
kubernetes
yet
just
start
with,
with
guaranteed
resource
classes,
guaranteed
won't
allow
for
over-commitment.
So
you
might
have
end
up
in
having
a
pet
cluster
utilization.
If
you
don't
get
the
values
right,
but
it
gives
you
a
safety
net
to
learn
and
and
and
tune
the
system
also
enable
cloak
quotas
and
enforce
defaults
in
a
cluster.
A
This
sounds
easy,
but
I've
seen
so
many
clusters,
not
really
enforcing
quarters,
or
at
least
not
enforcing,
write
defaults,
which
ends
up
happening.
A
bad
experience
for
free
workloads
also
protect
a
critical
system
parts
we
covered
us
for
daemon
sets
also
keep
in
mind
that,
as
long
as
the
cube
letters
on
the
control
of
your
workload,
that's
fine,
but
you
also
have
may
have
other
jobs
running
on
your
on
your
nodes,
which
are
outside
of
control
of
the
cubelet.
A
Also,
do
some
stress
testing
benchmarking
to
find
the
right
values,
because
if
it
ends
up
happening
in
production,
that's
actually
bad.
Tim
in
this
presentation
spoke
about
the
bork,
auto
pilot,
which
is
a
very
sophisticated,
and
maybe
one
of
the
best
engineering
things
I've
ever
heard
of
which
does
this
kind
of
adjustment
and
learning
and
pecking
the
cluster
increasing
high
increasing
utilization.
A
Also,
we
cover
this
when
we
talked
about
the
underlying
infrastructure,
hypervisors
brewster
blue
cloud
instances
that
make
sure
the
whole
stack
is
aligned,
not
just
the
pot
and
the
kubernetes,
but
also
the
Linux
operating
system
or
Windows
operating
system
and
underlying
infrastructure.
If
you
run
on
a
hypervisor
under
cloud,
monitoring
is
a
very
important
aspect,
and
not
just
for
CPU
memory,
but
also
the
Linux
kernel
is
a
shared
resource.
A
Some
of
the
primitives
we
do
have
in
pot
and
the
pots
back
can
be
applied,
but
there
are
some
things
like
open
file
descriptors
to
TCP
connections,
pits
other
other
stuff
that
at
least
you
should
monitor,
because
you
cannot
express
them
in
the
pod
specification,
so
you
at
least
have
to
monitor
them.
In
order
to
avoid
that,
you
run
into
issues
there
now
remember
the
busy
box
top
example
that
we
had
where
busy
box
was
limited
to
one
CPU,
but
it's
all
all
the
CPUs
right
now.
A
All
the
links
in
there
have
more
details
that
I
can
cover
here
and
at
the
end
of
the
not
aligning
those
values
could
end
up
at
best
and
just
low
performance.
But
at
the
adverse
crashes
of
view
of
your
system,
you
can
use
the
download
API
in
kubernetes
and
other
tools
to
make
this
more
automated.
So
you
don't
have
to
do
it
manually,
there's
different
ways
of
doing
it.
Download
API
is
something
that
I
typically
use
some
advanced
tuning
for
the
cubelet.
You
can
you
know
you.
Should
you
tune
at
least
the
eviction
thresholds?
A
Hard
soft
eviction
thresholds
swept
typically
is
enforced
to
be
not
enabled
in
your
and
your
on
your
note,
because
the
cubelet
has
it
just
has
a
hard
time
and
in
identifying
what's
fab
is
or
giving
you
a
good
resource
guarantees,
because
web
is
the
disk
resource
and
on
the
memories
results.
That's
why
it's
web
is
enforced
to
be
off
on
the
node
and
also
apply
Cupra,
soft
and
cube
or
system
reserved.
A
That's
just
reducing
the
amount
of
allocatable
resources
on
node
and
making
some
buffer
for
system
critical
demons
like,
for
example,
the
couplet
and
the
container
runtime
and
other
not
parts
on
our
port
workloads
for
a
system
or
latency
critical
workloads
use
the
cpu
manager
or,
as
we
covered
yesterday
in
your
talk
during
resource
management.
There's
much
more
like
Numa,
optimization
that
you
can
apply
to
for
latency,
critical
workloads
or
telco
networking
and
use
the
best
of
all
quality
of
service
clouds.
A
If
you
are
more
advanced
without
CPU
limits,
because
then
you
can
make
use
of
spare
idle
CPU
resources
on
the
host
and
there's
some
interesting
discussions
going
on
with
the
solando
guys
on
how
to
even
tune
this
unity
behavior
that
we
currently
have.
And
lastly,
before
we
close,
the
kernel
has
a
shared
resource.
Make
sure
you
understand
that
Windows
is
not
Linux,
there's
differences.
The
link
has
all
that
kind
of
the
way,
the
internals
of
how
Windows
implements
resource
management
from
a
cubelet
perspective
and
yeah.
Those
are
just
some
lessons
learned
here.
A
A
Yeah,
so
the
question
was
this
customers
running
kubernetes
on
vm
/,
/,
/,
provisioning
on
the
vmware
side
and
now
over
provisioning
in
kubernetes.
This
is
a
good
thing
or
not.
Yes,
and
no
please
come
see
me
afterwards.
We
have
done
some
best
practices
in
lime
and
just
see
me
afterwards.
There
was
another
question
there.
Yes,
please.
A
Well,
prometheus,
for
example,
you
could
use
this
CEO
dick
sorry.
The
question
was
some
of
the
workloads
that
you
run
could
be
starved
because
you
don't
specify
requests
or
just
minimal
requests
in
different
environments.
How
to
how
to
monitor
this
I
talked
to
Timo,
maybe
he's
in
the
room
at
the
same
issue.
You
could
use
SI
group
statistics
that,
to
another
beat
pretty
interesting
you
sinner,
okay,
come
come,
see
me
afterwards.
Team
FG
moves
in
the
room,
we
figure
it
out.
Yes,
please.
A
Yeah,
okay,
so
the
question
was,
since
you
run
into
this
fixed
default
quota
enforcement
of
100
millisecond,
which
for
burstable
workloads
could
have
neck
or
negative
affect
your
workload.
Is
there
something
to
mitigators
so
mon-sol
created
like
their
own
implementation,
or
they
kind
of
tuned
kubernetes
in
there
that
they
allow
for
changing
of
this
value
chip
that
have
reduced
it
right?
Now,
it's
not
like
in
core
upstream,
so
you
cannot
change
this
and
an
alternative
would
be
the
cpu
manager,
unfortunately
dead
static.
A
A
You're
right
yeah
there
was
which
recommended
yesterday
that
totally
disabling
quota.
So
even
if
you
specify
limits
the
cubelet
will
just
not
translate
that
and
you
will
end
up
not
having
limits
applied
on
the
system
now
that
could
be
that
could
work,
but
it
could
also
mean
that
if
you
don't
have
enough
resource
requests
underneath
that
there's
too
much
pressure
on
the
node
and
you
end
up
not
not
having
good
resources.
A
So
it's
it's
currently
unsolved
and
there's
a
lot
of
discussion
going
on
there's
in
this
presentation,
there
was
the
link
to
the
solando
and
the
discussion
with
bridge
and
the
other
guys
she's
champion,
make
your
thoughts
and
comments.
Please.
Thank
you
just
see
me
afterwards.
If
there's
more
questions,
please
thank
you.