►
From YouTube: Getting The Most Out Of Kubernetes with Resource Limits and Load Testing - Harrison Harnisch, Buffer
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Getting The Most Out Of Kubernetes with Resource Limits and Load Testing - Harrison Harnisch, Buffer
You've carefully crafted your first K8s service, and you're ready to deploy it to production. Well, not quite: there are still some important unknowns to understand before your service will be ready for production. It's still unclear how the new service behaves when it's being pushed, and it's possible that K8s will kill the service before serving a single request. At Buffer, we've developed a technique to optimize K8s deployment limits using load testing to identify optimal resource limits. When the service is under heavy load there are a few key metrics to watch to identify bottlenecks. These key metrics can be used to adjust resource limits. This approach allowed us to safely and efficiently switch over more than half our production traffic to our K8s cluster and can be applied to any application. This talk will include a live demo of how to tune Etcd using methods we do at Buffer.
To Learn More: https://sched.co/GrRX
A
A
A
So
my
name
is
Harrison
and
I'm.
A
staff
software
engineer
at
site
I've
only
been
there
for
a
couple
weeks.
I
just
switched
over
from
buffer
I
was
there
for
a
couple
years
before
that
and
see
I've
been
either
an
operator
or
an
end
user
of
kubernetes
as
a
developer
for
the
past
close
to
three
years
now.
A
So,
let's
start
off
with
a
story
that
comes
from
buffer,
but
first
started
off
as
a
monolith
written
in
PHP
and
there
was
a
service
or
this
endpoint
that
was
receiving
a
lot
more
traffic
compared
to
the
rest
of
the
application.
And
basically
what
this
thing
did
was.
If
you
don't
know
buffer
is
a
social
media
management
platform
and
basically
anytime,
that
somebody
sends
a
tweet
or
or
a
Facebook,
post
or
any
other
kind
of
post.
A
A
So
when
we
needed
more
than
one
replica,
so
we
just
set
it
to
4
and
we
deployed
the
service
on
the
kubernetes.
And
then
we
manually
verified
with
curl
to
make
sure
that
things
were
operational
and
had
some
metrics
hooked
up.
So
we
could
see
what
was
going
on.
So
we
shifted
one
percent
of
the
traffic
from
our
monolith
over
to
the
service
on
kubernetes
and
things
looking
pretty
good.
A
So
then
we
scaled
up
to
ten
percent
of
the
traffic
and
shifted
some
more
the
traffic
and
we
could
see
resource
utilization
metrics
going
up
approximately
what
you'd
expect
about
10x.
So
we
moved
on
to
the
next
phase
roll
it
out
to
50
percent,
and
this
is
when
things
started
to
get
interesting
because
well,
basically
went
into
a
crash
loop
and
so
first
thing
I
did
very
new
to
kubernetes
I,
just
scaled
it
up
to
20
pods.
It's
like
yeah,
it's
crew,
natives.
A
A
So
we
dug
deeper
and
basically
I
had
copied
a
deployment
from
another
service.
It
was
another
node
service
and
it
was
something
that
we
were
just
messing
with
just
just
a
little
prototype
to
kick
the
tires
on
kubernetes
and
this
particular
deployment.
It
contains
some
resource
limits
that
were
set
on
it
and
when
we
used
coop
control
describe,
we
could
see
that
we're
getting
o
and
killed.
So
the
pod
was
running
out
of
memory
because
we
hadn't
allocated
enough
resources
so
kind
of
set
off
this
whole
journey
of
figuring
out.
A
How
do
we
actually
set
these
things?
So
it's
important
to
understand
the
difference
between
limits
and
requests,
because
these
are
kind
of
your
your
building
blocks
here.
So
thinking
about
limits,
these
are
the
upper
limit
on
container
resources.
Now,
if
you
don't
set
any
limit
at
all,
the
containers
can
be
can
run
completely
unbounded
and
they
can
take
up
all
of
the
resources
on
it.
So
if
the
limits
crossed
now,
there's
two
types
of
things
that
can
happen
here,
there's
compressible
so
CPU.
A
A
Now
the
other
building
block
is
requests,
so
this
is
kind
of
a
hint
to
the
scheduler
that
this
particular
workload
needs
a
certain
amount
of
resources
and
if
there's
capacity
available-
and
it
gets
scheduled-
that
amount
of
resources
should
always
be
available
for
that
pod
unless
it
needs
to
be
evicted
for
some
reason
by
the
scheduler
now
for
a
period
of
time.
Let's
say
you
just
set
requests
and
there's
a
capacity
available.
A
It's
going
to
your
your
container
is
going
to
be
able
to
use
more
resources
for
a
period
of
time.
So
you
get
this
kind
of
bursty
behavior
that
you
can
program
into
it
and
if
you
don't
set
requests,
but
you
set
a
limit,
it's
basically
going
to
implicitly
set
the
request
to
the
women
so
I'm
going
to
break
that
down
here
into
a
few
different
quality
of
services,
so
just
to
name
them
off.
You've
got
guaranteed
yep.
First
of
all
and
best-effort.
A
So
guaranteed
is
when
you
set
your
limits
to
the
request
or
maybe
adjust
set
the
limits
here.
These
are
the
highest
priority
pods,
so
these
are
going
to
be
the
last
thing
that
gets
killed
by
the
scheduler
or
that
the
scheduler
decides
to
kill
and
the
next
one
is
the
burst
of
all
quality
of
service,
and
this
is
where
you
set
limits
that
are
higher
than
your
requests.
A
So,
let's
say
in
this
example:
here
you
set
your
request
to
100m,
which
is
roughly
1/10
of
a
core
and
CPU,
and
then
you
set
the
limit
to
200m,
which
is
roughly
2
to
of
a
court.
The
scheduler
is
going
to
allocate
100m
CPU
to
your
container
and
for
a
period
of
time.
If
there
are
resources
available
on
that
node,
it
can
burst
up
to
twice
the
capacity
if
it's
available
now
and
in
the
case
here,
if
it
does
burst
up
to
that
200m,
that's
it's
going
to
be
compressed
in
CPU
is
compressible.
A
Go
back
here
last
one
is
the
lowest
priority,
so
this
is.
These
are
best
effort
pods
now
these
can
use
up
any
amount
of
free
resources,
so
they're
kind
of
liquid
in
that
way,
if
there's
resources
available
on
a
node
and
it
gets
scheduled
on
a
note,
it's
going
to
it-
has
the
ability
to
kind
of
fill
in
the
gaps
and
take
up
the
rest
of
the
space,
or
maybe
you
just
want
it
to
be
able
to
take
up
all
the
resources
on
a
node.
A
If
you
schedule
things
that
way,
if
you
have
highly
interruptible
processes,
this
is
where
this
can
be
useful.
Maybe
you
do
some
garbage
collect
chinna
or
some
optimization
that
you
don't
need
to
be
running
all
the
time
or
maybe
you
want
to
run
off-peak
hours
these?
These
are
great
things
to
schedule
this
way
with
best
effort
quality
of
service.
A
So
how
do
we
actually
go
about
setting
these
things?
So
it's
important
to
talk
about
and
understand
what
optimal
means
here
and
with
any
optimization
problem.
There's
it's
usually
a
balancing
act
between
a
couple
different
variables
when,
when
thinking
about
what
you're
balancing
you're
looking
at
so
each
pod
has
enough
resources
to
complete
the
task.
So
whatever
task
that
happens
to
be
pod
has
enough
room
to
run
now
on
the
on
the
other
side
of
that
all
of
the
nodes
can
run
the
maximum
number
of
pods.
A
It's
your
kind
of
trading
off
between
here
so
pods,
just
you
can't
just
allocate
tons
of
resources
to
a
pod,
because
you
know
you'll
end
up
being
wasteful,
so
there's
a
couple
of
different
ways.
You
kind
of
put
these
group
these
into
buckets.
There's
a
couple
ways
if
you
get
it
wrong
and
then
kind
of
the
optimal
is
something
you
strive
for,
but
under
allocation
just
thinking
about
limits
here,
where
the
the
couplets
going
to
intervene.
A
If
something
is,
is
acting
out
of
place
or
unexpectedly,
if
you
cross
the
memory
threshold,
the
couplets
going
to
kill
the
pod,
so
these
things
are
a
little
bit
easier
to
see
and
we're
gonna
use
this
property.
When
we
do
some
when
we
get
into
the
demo
here,
so
we
can
have
more
visible
effects
of
these
things.
So
the
other
way
that
you
can
get
this
wrong
is
you
can
over
allocate
resources
to
your
to
your
pods,
and
basically
this
means
that
you
give
so
much
resources
to
the
pod
that
it
can
never
consume
it.
A
Maybe
something
goes
wrong,
but
under
typical
conditions,
it's
not
going
to
consume
that
amount
of
resources.
This
is
a
little
bit
trickier
to
catch,
because
the
effects
aren't
immediately
obvious
and
it
really
becomes
a
problem
when
you
scale
up
replicas.
So
let's
say
you
over
allocate
just
a
few
megabytes
of
memory
to
a
pod,
but
then
you
hit
this
point
where
you're
scaling
it
up.
Thousands
and
thousands
of
pods
you're
wasting
that
little
sliver
multiplied
by
a
thousand,
so
basically
you'll
be
wasting
resources
and
and
wasting
capacity
and
waste.
A
So
in
a
simple
example,
let's
say
we
have
a
pod
that
can
only
ever
use
one-third
of
the
capacity
on
a
node,
but
we
allocate
half
of
the
resources
on
the
node
to
it.
We're
missing
out
on
a
on
the
extra
pod
that
we
could
be
running
and
imagine
thousands
of
these
same
exact
scenario
is
happening.
You're
gonna
be
wasting
basically
wasting
a
lot
of
capacity,
so
optimal
is
something
that
you
want
to
strive
towards
in
practice.
A
You're
probably
gonna
over
allocate-
and
you
probably
want
to
Leon
over
allocation
a
little
bit
anyways,
because
you
want
just
a
little
extra
capacity,
that's
kind
of
up
to
the
application
that
you're
building
and
the
domain,
but
it's
something
that
you
strive
towards
it
shouldn't
be
it's
not
a
hundred
percent
thing.
Maybe
you
get
90%
optimal
okay.
A
So
let's
switch
gears
a
little
bit.
Let's
talk
about
how
kubernetes
does
monitoring
of
these
pods,
so
you
don't
have
to
remember
everything
here,
but
I'm
kind
of
gonna
walk
through
each
one
of
these
pieces.
But
it's
important
to
note
that
there's
multiple
nodes,
each
node
has
a
couplet
and
see
advisor
and
some
of
the
nodes
are
gonna
have
aggregation,
metrics
aggregation
here.
It's
called
heaps
to
get
a
little
bit
more
a
share,
some
news
on
that
and
a
little
bit,
but
basically
there's
this
thing.
A
A
A
Okay,
so
let's
go
back
to
setting
limits
and
requests.
So
this
is
the
define
a
process
part.
Basically,
the
goal
is
to
understand
what
one
of
your
pods
can
handle
and
when
we're
trying
to
understand
what
that
is,
we're
gonna
set
the
limits
and
the
reason
what
we're
gonna
use
limits
is
because
we
want
to
see
the
immediate
effects
we
want
to
see
the
CPU
being
being
throttled.
A
We
want
to
see
the
container
get
restarted
because
across
the
memory
utilization
so
start
with
the
conservative
set
of
limits,
so
we
can
see
those
effects
and
then
we'll
run
a
test
and
then
observe
what
happened
and
then
we're
gonna
change.
Just
one
thing
at
a
time
and
we're
gonna
look
at
what
what
happened
after
we
changed
each
one
of
those
pieces.
A
Now,
after
you
kind
of
find
that
breaking
point
and
you've
set
like
the
general,
maybe
the
macro
level.
Tweaks
then,
you
want
to
run
that
service
under
load-
maybe
let's
say
90%
of
that
breaking
point
for
an
extended
period
of
time
and
what
you're
doing
is
looking
for
things
like
well,
sudden
jumps
or
maybe
like
variants
and
you're
in
your
response
times,
because
there's
things
like
memory
leaks
that
could
occur.
There's
things
like
queue
overflows
that
could
occur.
You're
kind
of
gonna
catch
those
things
with
this
type
of
this
other
type
of
testing.
A
A
A
The
way
that
I
had
to
hook
this
up
so
just
so
you
know,
there's
no
shenanigans
going
on
and
funny.
Nginx
is
sitting
in
front
of
that
CD
and
the
reason
why
I'm
doing
that
is
so
I
can
hook
up
the
loader
I/o
token.
That
way.
Loader
I/o
can
apply
these
tests
because
loader
doesn't
want
to
be
denial
of
servicing
anybody.
A
A
And
what
this
is
doing
is
this
is
this
is
hooking
up
directly
to
the
doctor,
a
socket
on
the
node
and
it's
displaying
the
metrics
at
about
one
second
interval,
which
is
a
lot
faster
than
you'll,
get
if
you're
trying
to
get
these
metrics
from
something
like
Prometheus,
because
it's
talking
directly
to
the
socket.
So
what
you
can
see
here
is
I'm
I'm,
looking
at
the
Etsy
d-pod
directly.
A
So
I'm
doing
a
ramp
up
test
and
what
you'll
see
is
that
CPU
went
up
that
looks
like
the
container
died.
Yes,
this
is
what
we
wanted.
The
test
is
still
trying
to
run
basically,
but
we'll
see
here.
Okay,
tests,
completed
and
you'll
see
not
two
things,
basically
that
that
second
row
of
Kubb
scope
was
the
container
restarting.
So
it's
hooked
back
up
it
restarted.
So
we
know
something
went
wrong
there.
Let's
take
a
look,
so
I'm
gonna
describe
the
pod.
A
Oh
bug
they
gotta
restart,
look
at
you
know
all
right,
so
I'm
gonna
kick
the
test
off
again
and
this
time
I'm
expecting
the
pod
to
stay
up.
It's
gonna
be
a
little
slow
because
it's
still
bounded
by
you
can
see
here
that
it's
still
bounded
by
memory,
so
we've
hit
a
bottleneck
or
sorry.
Cpu
we've
hit
the
bottle
and
I
can
CPU
and
coop
scope
makes
that
pretty
obvious
because
we're
basically
being
throttled
at
about
2.1
percent
approximately
at
2.5%.
A
So
I'm
gonna.
Let
this
test
finish.
So
we
can
look
at
the
average
when
we're
done,
because
then
we
can
see
we're
gonna
make
another
adjustment
after
this
based
off
of
that
test.
So
we
just
changed
the
memory
now
we've
given
the
pod
enough
memory
to
run,
and
if
you
look
at
the
memory
percent,
we're
actually
probably
over
allocated
here
because
we're
we're
not
even
getting
close
to
the
amount
utilization
at
this
at
this
traffic
that
we're
applying
to
it.
A
Okay,
so
that
test
is
completed,
you'll
see
the
CPU
drop
down
and
the
average
it's
a
little
hard
to
see,
but
it's
around
one
that's
about
it's
around
one.
Second,
so
if
if,
in
fact
the
CPU
is
the
bottleneck
here,
if
we
give
it
more
resources,
we
should
expect
to
see
that
this
that
the
average
response
time
should
go
down,
and
if
you
want
to
get
really
detailed,
you
can
also
look
at
the
requests
per
second,
so
we're
about
96
requests
per
second
loader
just
provides
that
for
you,
which
is
nice,
so.
A
And
let's
say:
I
go
crazy:
let's,
let's
do
something!
That's
definitely
not
gonna
work.
Let's
try
to
give
it
500m,
so
I've
got
a
really
small
cluster
and
I
want
to
demonstrate
what
happens
when
you
try
to
give
something
too
much
resources.
You
see
that
cube
scope,
there's
no
pods
running.
So
if
I
get
I
get
pods,
I'm
gonna
see
something
that's
in
this
pending
status
and
if
I
describe
the
pod
for
ED
CD,
we're
gonna
see
that
warning
or
the
it
couldn't
it
couldn't
scale
because
or
couldn't
schedule
it,
because
0-1
nodes
are
available.
A
So
we're
at
about
20
M
before
right
now
we're
at
about
our
25
now
we're
at
about
50.
So
we
should
expect
to
see
not
not
quite
double
the
performance
in
this
case
kind
of
depends
on
what
you're
running
so
I'm
gonna
run
this
test
again
before
our
average
was
about
one.
Second,
I've
got
a
bug
in
Cusco.
A
A
So
I'd
keep
going
on
with
this
process
until
I
kind
of
find
this
point
where
I
it
really
depends
on
what
you're
running,
but
you
could
probably
give
this
a
half
a
core
and
in
handle
a
lot
of
traffic.
So
if
you
have
any
information
about
the
capacity
that
you're
going
to
be
handling
in
the
case
where
we
had
a
monolith
and
that
we
could
look
at
the
number
of
requests,
we
knew
we
had
to
handle
that
amount.
So
any
constraints
that
you
can
get
here
gonna
be
useful.
A
You
can
see
here,
yeah
we're
running
a
lot
better,
we're
about
550
milliseconds
before
we
were
about
a
second.
So
definitely
CPU
is
the
bottleneck
in
this
case,
so
after
I
kind
of
found
a
sweet
spot
with
these.
With
this
with
the
ramp
up
test,
I
would
want
to
run
a
duration
test
and
just
kind
of
generally
thinking
about
that.
It's
probably
at
least
10
minutes
as
a
good
rule
of
thumb,
the
longer
the
better,
especially
if
you
don't
have
any
traffic
that
you
can
throw
at
it.
A
That
way,
you
have
a
have
some
confidence
that
it's
going
to
work
when
you
get
to
production
nice
and
we
go
back
to
the
slides
here.
So
another
important
thing
is
while
you're
doing
this,
like
we
saw
some
different
failure
modes,
but
keep
a
log
take
some
notes,
while
you're
doing
this,
because
these
things
are
going
to
end
up
in
your
run
books.
These
are
gonna,
be
stuff
that
other
people
are
going
to
use
and
ask
you.
A
Some
of
the
observed
failure
modes
for
a
number
of
different
service,
so
memory,
slowly,
increasing
one
that
we
just
saw
here
was
the
CPU
is
pegged
at
100%
of
what
the
resources
that
were
allocated
to
it.
You
can
also
see
service
riders,
five,
hundreds
high
response
times.
Another
interesting
one
is
variance
in
response
times.
That's
usually
some
sort
of
queuing
mechanism
is
in
place
and
also
just
flat-out
dropping
requests.
A
So
some
of
the
stuff
that
we
learned
is
that
what
it
means
to
be
production,
ready,
I
I,
think
you
need
to
know
how
your
service
is
going
to
break
before
you
can
put
it
in
production,
because
it's
a
when,
when
you're
having
real
live
customers
using
your
products,
especially
ones
that
where
people
are
dependent
on
them,
it's
it's
it's
good
to
understand
how
they
break.
So
you
can
plan
ahead
and
when
they
do
break,
you
can
act
rather
than
have
to
do
this
triage
stuff
like
in
the
cluster.
A
A
So
some
tools
that
you
can
use
after
you
understand
the
breaking
point,
so
one
of
them
is
and
there's
a
fantastic
talk.
That's
gonna
be
later
I,
think
today
or
tomorrow
on
the
horizontal
pod
autoscaler,
but
basically
what
this
thing
does
is
it
changes
the
replica
count
based
on
a
metric?
So
maybe
once
you
hit
90
percent
of
the
CPU
utilization
and
you
start
scaling
up
replicas
until
it
has
enough
capacity
to
handle
the
load,
you
can
hook
up
all
kinds
of
metrics
to
this.
A
You
cannot
get
up
to
Prometheus
as
your
adapter
stackdriver,
just
to
name
a
few.
It's
very
well
supported
this
thing's,
this
thing's,
pretty
solid.
You
can
hook
up
multiple
metrics.
A
lot
of
people
are
running
communities
on
AWS
and
have
worker
queues
implemented.
You
can
scale
up
using
external
metrics
from
sqs,
so
you
can
look
at
the
number
of
messages
that
are
in
flight,
so
another
one.
This
is
the
vertical
pod
or
a
scalar,
and
this
one
is
it's
an
alpha
feature
so
use
with
caution
this
one.
A
A
So
just
looking
ahead
a
little
bit
kind
of
tools
for
developers
right
now,
there's
really
great
tooling.
For
aggregate
metrics
we
have
Prometheus.
We
have
things
like
data
dog,
so
we
can
see
what's
going
on
at
the
cluster
level,
but
there's
still
a
gap
for
high
resolution
tools,
so
you
can
look
at
maybe
just
a
group
of
deployments,
maybe
a
few
pods
or
kind
of
what
we
did
here.
We
were
just
looking
at
one
container,
so
this
is
why
I
started
this
project,
coupe
scope,
because
these
things
didn't
exist
and
I'd
love
help
building
this.
A
A
Like
self
boosted,
instead
of
using
an
external
service,
so
the
question
was:
are
there
any
basically
open
source
tools
that
we
could
use
to
apply?
Load
testing
within
the
cluster
I
haven't
used
it
for
a
while,
but
I
believe
locust
is,
is
one
that's
pretty
popular
depending
on
the
amount
of
load
that
you
need
to
apply.
There's
a
bunch
of
like
NPM
installable
like
like
this
one
here.
A
B
A
So
the
the
question
was
when
I
scaled
up
from
I
kind
of
had
this
service
that
was
failing
and
then
I
immediately
scaled
up.
Why
didn't
that
work?
In
that
case
it
didn't
work,
because,
even
even
at
that
level
there
weren't
there
still
wasn't
enough
resources
like
it
was
just
too
constrained.
That's
a
good
question!
A
A
Your
standard
load
should
fit
inside
of
a
request
and
then
maybe
like
after
this
kind
of
takes
a
little
bit
of
context
on
the
service
that
you're
building.
But
if
you
know
it's
kind
of
got
a
bursty
behavior
you'll
want
to
you'll
want
to
set
the
limit
higher
than
that,
so
it
can
use
that
capacity,
especially
with
memory.
It's
it's
I,
don't
want
to
say
it's
less
important
with
CPU,
because
it's
compressible,
but
yet
kind
of
it
kind
of
depends
on
what
you're
building.
If
you
wanted
to.
B
A
A
A
A
Okay,
so
the
question
was:
there's
vertical
autoscaler
and
then
there's
the
horizontal
autoscaler.
Do
it
is
there
anybody?
Is
there
anybody
who's
using
a
combination
of
those
things
to
solve
some
scaling
issues?
Is
that
correct,
I'm,
not
sure
if
anybody's
combining
them
in
a
way,
that's
yeah
I
haven't
seen
a
really
good
use
case
for
that
I'm
in
the
wild,
but
yeah
there's
a
lot
of
people
here
who
might
be
doing
that
if
any
but
show
of
hands,
does
anybody
actually
using
both
vertical
and
right
there?
Yes,
you
should
talk
to
that.
Guy.