►
From YouTube: Load Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production [I]
Description
Load Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production [I] - Harrison Harnisch, Buffer
So you've carefully crafted your first Kubernetes service, and you're ready to deploy it to production. Well, not quite: there are still some important unknowns to understand before your service will be ready for production traffic. It's still unclear how the new service behaves when it's being pushed, and it's possible that Kubernetes will kill the service before serving a single request. At Buffer, we've developed a technique to optimize Kubernetes deployment limits by using load testing to identify optimal values for resource limits. When the service is under heavy load there are a few key metrics to watch to identify bottlenecks. These key metrics can be used to adjust resource limits. This real world approach allowed us to safely and efficiently switch over more than half our production traffic to our Kubernetes cluster and can be applied to any application.
This talk will include a live demo of how to tune Hashicorp's Consul application using methods we do at Buffer.
About Harrison Harnisch
Senior Software Engineer at Buffer, implementing the transition to microservices with Kubernetes and Docker.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Well,
thanks
everybody
for
coming
out
and
sticking
around
for
the
last
talk
of
the
day-
and
this
is
my
first
ever
talk
at
a
conference,
so
this
is
really
exciting
and
if
I've
learned
anything
about
talks
from
other
people,
you
always
start
off
with
a
question.
So
my
question
is:
who
here
is
setting
resource
limits
on
their
on
their
pods
or
containers?
Okay,
so
quite
a
few
people
now
who
here
is
using
load
testing
techniques
to
set
those
resource
limits?
Okay,
not
quite
as
many
hands,
but
still
a
few
people.
A
A
A
So
we
built
the
service
and
then
we
started
to
roll
it
to
roll
it
out
to
kubernetes,
and
we
had
about
four
replicas
to
start
off
with
and
we
manually
verified
that
things
were
working
with
curl,
so
we
shifted
about
1%
of
our
traffic.
Over
to
this
service,
things
were
looking
great.
We
had
monitoring
hooked
up
with
data
dog
and
you'd,
but
you
could
barely
notice
that
there
was
any
load
running
on
each
of
the
containers
same
story
at
10
percent.
A
A
So
what
had
happened
is
I
had
copied
and
pasted
a
deployment
file
from
another
service,
and
this
particular
service
was
an
old
service
that
didn't
have
as
much
traffic,
and
we
were
still
pretty
new
to
this
at
the
time,
and
this
in
this
deployment
file
contains
some
resource
limits.
So
when
we
were
doing
the
coop
kuddle
describe
that
the
pods
were
reporting
oh
and
killed.
A
So
let's
talk
a
little
bit
about
what
resource
limits
actually
are,
there's
something
that
can
be
set
on
both
CPU
and
memory
and
when
they're
not
set
these
things,
can
it
can
run
unbounded
and
they
can
actually
take
up
all
the
resources
on
the
node
that
they're
on
and
when
the
limits
are
exceeded.
Kubernetes
is
going
to
go
through
and
it's
going
to
it's
going
to
kill
those
pods.
A
So
how
do
we
go
about
actually
setting
these
things?
So
it's
it's
helpful
to
discuss
what
optimal
limits
actually
look
like
this
means
that
pods
have
enough
resources
to
complete
whatever
tasks
they're,
given
whether
that's
HTTP.
Maybe
it's
a
worker
doing
work
on
a
queue
and
also
each
of
the
nodes
can
run
the
maximum
number
of
pods.
So
there's
no
waste.
A
So
it's
helpful
to
talk
about
ways
that
things
can
go
wrong
and
also
things
can
go
right
when
you're
allocating
resources
is
the
first
one
is
under
allocation.
This
one's
really
obvious.
This
is
what
happened
to
us.
We
didn't
have
enough
resources
allocated
and
what
kubernetes
is
going
to
do
is
recognize
that
one
of
your
limits
has
been
crawled
crossed
and
that's
going
to
cook.
That's
going
to
kill
the
container.
A
The
other
one
is
over
allocation.
This
is
where
you've,
given
your
your
pod
or
your
container
enough
resources
that
it's
never
actually
going
to
be
able
to
use
those
no,
but
for
the
load
that
that
you're
trying
to
service-
and
this
is
a
tricky
problem
to
detect
because
things
things
can
work
for
a
while-
and
you
can
be
in
this
place
where
you're
running
you
you're
you're,
handling,
all
of
your
load
and
things
look
great,
but
it
becomes
a
problem
when
you
start
to
scale
up
your
replicas.
A
A
A
This
graph
there's
a
lot
going
on
here,
but
I
wanted
to
show
this
first,
because
I
think
it's
important
I
think
it
shows
that
there's
the
centralized
thing
in
the
middle
called
heap
stir
that's
orchestrating
or
that's
collecting
information
and
also
providing
an
API
to
other
parts
of
the
system.
So
each
node
is
effectively
working
through
heap
stir
and
the
kubernetes
master
is
making
that
information
available.
Also
heap
store
pushes
things
off
to
a
storage
back-end,
that's
pretty
important
too.
So,
let's
dig
a
little
bit
into
the
details
here.
A
A
So
how
do
we
go
about
setting
limits
that
the
goal
is
really
just
to
understand
what
one
pod
can
handle
here,
and
you
start
with
a
really
conservative
set
of
limits.
They
start
really
low
and
you
change
one
thing
at
a
time
and
you
observe
the
changes
you
try
to
be
scientific
about
it,
because
if
you
change
too
many
things,
it's
it's
kind
of
hard
to
understand
what
actually
happened
when
you
change
something.
A
So
there's
a
couple
of
testing
strategies
that
I'm
going
to
employ
to
actually
set
the
limits
and
the
first
one
is
a
ramp
up
test
and
what
we're
gonna
do
is
start
with
one
client
and
then
we're
gonna
scale
up
the
clients
until
and
just
watch
what
happens
to
the
response
times
for
the
for
this
particular
test.
And
the
idea
is
that
you,
you
eventually
want
to
find
a
breaking
point,
you're
looking
for
major
changes.
Maybe
you
cross
a
threshold
and
then
all
of
a
sudden
you're
getting
a
bunch
of
500s.
A
After
that,
we're
essentially
going
to
take
the
this
slice,
the
the
tallest
slice,
where
we're
right
before
we
broke,
and
then
we're
going
to
operate
just
under
the
breaking
point
for
an
extended
period
of
time,
and
what
we're
gonna
look
for
here
is.
It
is
major
changes
in
response
times.
Maybe
you
get
some
variance,
but
what
you
want
to
see
here
is
pretty
consistent
load
pretty
consistent
response
times,
and
this
is
also
where
you
make
fine-tuned
adjustments
as
well,
so
I'm
going
to
do
a
demo
I'm
going
to
set
limits
for
an
Etsy
d-pod.
A
B
A
A
A
B
A
A
All
right
so
I've
got
an
engine,
X
Server
and
that's
exposed
or
that's
proxying
requests
out
to
at
CD
and
I'm,
also
exposing
a
loader
I/o
token
I'm.
Basically
using
I'm
using
loader
I
did
IO
to
run
these
load
tests
and
I
need
to
expose
a
token
to
give
them
permission
to
to
run
to
run
load
tests.
So
that's
that's
why
this
looks
like
I
thought.
It
was
important
to
call
that
out
because
it
look
a
little
funky.
A
A
A
We're
going
to
watch
that
container,
restart
and
if
you
remember
before,
I
was
talking
about
the
monitoring
I'm
actually
going
to
tap
directly
into
the
sea
advisor
and
I'm,
going
to
use
that
to
get
the
resource
limits
and
that
container
came
up.
So
here's
the
adviser
here
it's
a
little
clunky
to
get
to
it,
but
once
I'm
there
you
can
see
that
I've
got
my
resource
limits
are
showing
up
here.
A
The
increased
memory
shows
me
CPU
and
in
this
case
I've
only
got
one
core,
so
the
total
usage
and
usage
per
core
are
going
to
look
the
same,
and
then
we've
also
got
some
some
memory
to
look
at
and
that
50
that
I
set
to
I
have
a
lot
more
room
to
breathe.
Now,
if
I
would
have
looked
before,
this
would
have
been
about
a
hundred
percent,
as
things
are
looking
good
I'm
gonna
start,
so
the
I'm
gonna
start
running
that
ramp
up
test
now.
A
A
A
A
A
So
I'm
gonna
rerun
that
test
and
what
I'm
expecting
now
is
to
see
better
results
at
something
better
than
about
500
milliseconds.
That
30
seconds
would
500
milliseconds
response
time
at
halfway
point
through
the
test.
Oh
I
need
to
go
grab
so,
unfortunately
see
advisor
change
or
the
idea
of
of
things
change
when
the
pod
goes
down,
so
I
have
to
go
grab
it
again.
A
So
we're
at
about
the
halfway
point
and
we
can
already
see
we're
at
around
133
milliseconds.
So
at
the
moment
that
is
in
fact
our
bottleneck
and
I.
Don't
have
any
more
resources
on
this
machine
to
give
it
so
that
this
is
effectively
the
most
I
can
do
with
this
setup.
You
can
also
see
here
that
we've
we've
hit
our
resource
limit,
0.15.
A
A
Your
response
time
should
remain
relatively
flat,
and
this
graph
is
looking
kind
of
interesting.
Usually
it
sticks
around
100
milliseconds
would
probably
want
to
dig
into
to
what
happened
there
and
why?
Why
that?
Why
that
increased
for
a
certain
period
of
time,
but
could
also
be
that
somebody
here
is
running
load
tests
on
this.
A
A
So,
when
you're
going
through
this
process,
it's
really
important
to
keep
a
failed
log
and
what
what
you
want
to
see
or
what
you
want
to
write
down
or
think
the
way
that
things
failed
things
are
good.
Things
are
going
to
break
when
you
do
this,
especially
when
you
get
to
the
point
where
you're
crossing
the
threshold
of
the
particular
pod
and
some
of
the
stuff
that
we've
seen
memory
slowly
increasing
what
we
saw.
There
was
CPU
as
peg
debt
at
the
resource
limit.
A
A
So
when
we
were
going
through
this
process
with
the
link
service,
really
what
we,
what
we
were
trying
to
accomplish
here
was
we
needed
to
understand
how
things
break
and
for
us
it
things,
don't
feel
production
ready
until
you
go
through
a
process
like
this,
because
you
don't
really
understand.
What's
gonna
happen
when
you're
pushing
things
towards
the
edge
and
really
it's
about
increasing
predictability,
and
this
is
important
because
when
when
things
are
predictable,
you
can
understand
how
to
scale
this
up.
A
A
Just
looking
forward
with
kubernetes
there's
so
many
great
tools
for
ops
and
things
that
are
done
on
the
cluster
wide
level,
but
I
think
developers
really
want
to
be
able
to
dig
into
one
things
times
and
get
get
their
hands
dirty
and
to
do
some
debugging
and
there's
some
tooling.
That
could
exist
that
just
doesn't
yet
and
I
think
it's
just
kind
of
the
natural
progression
of
kubernetes.
Some
of
the
talks
earlier
today
have
reflected
this
to
to
developers.
A
A
So
the
question
was:
have
we
thought
about
automating
this
process
and
the
answer
that
is
yes,
we
have
loader
Eyal
actually
exposes
some
API
endpoints,
so
we
could
actually
get
to
the
point
where
we
can
automate
this,
maybe
working
with
the
the
dais
team
and
packaging.
Some
of
those
things
might
be.
There
might
be
something
cool
that
you
could
automate
that
whole
flow,
but
yeah
we
want
to
do
that.
Yes,.