►
From YouTube: Autoscaling Kubernetes Deployments: A (Mostly) Practical Guide - Natalie Serrino, New Relic
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Autoscaling Kubernetes Deployments: A (Mostly) Practical Guide - Natalie Serrino, New Relic (Pixie team)
Sizing a Kubernetes deployment can be tricky. How many pods should it have? How much CPU/memory is needed per pod? Is it better to use a small number of large pods or a large number of small pods? What’s the best way to ensure stable performance when the load on the application changes over time? Luckily for anyone asking these questions, Kubernetes provides rich, flexible options for autoscaling deployments. This session cover the following topics: - Factors to consider when sizing your Kubernetes application - Horizontal vs Vertical autoscaling - How, when, and why to use the Kubernetes custom metrics API - Practical demo: Autoscaling with application metrics from Prometheus, Linkerd, Pixie (request throughput/latency, number of shoes purchased in my web store) - Impractical demo: A Turing-complete autoscaler!
A
A
A
A
You
have
to
know
the
right
bottleneck
in
your
application
in
order
to
make
your
auto
scaling
as
performant
as
possible.
So
that,
then,
we'll
cover
selecting
the
right,
auto
scaling
metric
for
your
application
and
then
finally,
I
love
to
push
the
boundaries
of
the
technology
I
work
in
and
it
turns
out
you
can
make
a
touring
complete,
auto
scaler
for
kubernetes,
so
we'll
be
showing
that
at
the
end.
A
A
A
For
example,
there
is
the
methodology
of
doing
a
completely
random
guess
and
hope
that
it
works.
You
copy
pasta
from
some
other
thing
that
I've
already
deployed
on
your
cluster.
This
is
one
of
the
most
common
ones.
In
my
experience,
you
might
be
one
of
those
people
that
is
always
thinking
ahead
and
you're,
always
proactively.
Iterating
you're,
monitoring
your
deployments
and
seeing
oh
shoot.
Cpu
is
a
little
bit
close
to
the
limit,
but
mostly
in
practice
we
reactively
iterate.
A
A
A
A
A
A
A
Finally,
we
have
the
horizontal
pod,
auto
scaler.
This
is
going
to
be
the
one
that
we
focus
on
most
in
this
talk,
because
it's
extremely
powerful
it
adds
additional
replicas
to
your
deployment
etc,
and
it
also
can
look
at
resource
utilization,
but
the
really
cool
thing
is:
you
can
actually
define
your
own
metrics
that
you
want
to
scale
based
off
of,
and
this
makes
it
one
of
the
most
powerful
features
of
kubernetes.
In
my
opinion,
also,
I
know
that
the
font
might
be.
A
A
A
A
A
One
thing
to
note
is
that
the
kubernetes
contributors
in
their
documentation,
they
note
that
they've
tested
this
cluster
autoscaler
for
up
to
a
thousand
nodes
with
30
pods
per
node.
So
if
you're
running
a
mega
workload,
that's
bigger
than
this,
you
should
keep
that
in
mind
when
using
the
auto
scaler.
A
Next,
let's
look
at
the
vertical
pod,
auto
scaler.
By
the
way
there
is
this
really
hilarious:
tick,
tock
that
I
saw
for
kubernetes
that
was
like
showing
the
difference
between
the
vertical
pod,
auto
scaler
and
the
horizontal
pod,
auto
scaler
in
terms
of
glasses
of
water,
so
definitely
check
that
out
if
you're,
a
tic,
toc
user.
A
A
A
And
finally,
you
can't
auto
scale
resources
like
cpu
and
pod
for
a
given
replica
and
add
replicas
with
the
horizontal
potato
skiller
at
the
same
time
for
the
same
metric
on
the
same
application.
So
whether
you're
gonna
make
your
existing
pods
bigger
or
add
more
pods
for
a
given
deployment,
you're
going
to
want
to
pick
a
lane
and
choose
either
the
horizontal
plot,
auto
scaler
or
the
vertical
plot,
auto
scaler.
A
A
Okay,
finally,
we'll
talk
about
my
favorite,
the
horizontal
pod
autoscaler,
so
this
provides
so
much
flexibility
for
metric
selection.
It's
not
even
funny,
and
every
application
is
its
own
special
snowflake,
so
you
can
give
it
its
own,
corresponding
special
snowflake
metric,
but
you're
going
to
want
to
check
your
service
client
affinity
policies
to
ensure
even
load
distribution.
A
A
Also,
you
need
to
make
sure
you
set
resource
requests
when
you're
scaling
on
cpu
and
memory,
horizontal
positive
scaler,
when
you're
scaling
on
something
like
cpu,
looks
at
the
percentage
of
your
request
that
you're
utilizing.
So
if
you
don't
have
a
request,
it's
never
going
to
auto
scale
and
finally,
as
with
the
vertical
pod
auto
scaler,
we
can't
use
these
two
in
conjunction
for
the
same
workload
on
the
same
metric
yet.
But
that
is
something
that
the
contributors
are
working
on.
A
A
A
So
we're
going
to
watch
this
and
what
we're
going
to
do
is
use
my
favorite
load,
generating
application
called
hey,
and
it
basically
is
going
to
allow
me
to
spam
a
ton
of
traffic
to
this
endpoint,
and
this
endpoint
is
an
expensive
endpoint.
It's
very
cpu
intensive.
So
that's
why
we're
going
to
scale
on
cpu,
so
I'm
spinning
up
20
concurrent
clients
each
sending
one
query
per
second,
which
is
a
lot
for
this
service
and
in
the
bottom
left.
What
we
can
see
is
that
one
replica
is
running
right
now.
For
my
echo
service.
A
And
the
beauty
of
this
is,
we
can
jump
ahead
and
let
the
auto
scaler
do
its
thing.
Oh,
look.
We
have
four
new
replicas,
it's
responded
to
the
load,
and
now
we
know
that
this
deployment
needs
more
resources
associated
with
it.
So
we
can
look
in
pixi
to
see
the
spikes
now
the
load
has
ended
on
the
application,
but
we
can
see
the
request
per
second
has
gone
up.
The
http
latency
has
gone
up
too,
because
auto
scaling
hasn't
kicked
in
yet
cpu
usage
has
gone
up.
A
So
what
we
can
see
is
that
it
added
four
pot
or
added
three
pods
in
response
to
the
latency
that
we
had
and
it
wasn't
running
for
very
long,
but
you
can
see
that
near
the
tail
end
of
it.
The
latency
went
down
a
lot,
so
it
did
its
job.
If
this
had
been
a
persistent
spike,
then
we
would
have
seen
much
better
latencies
as
a
result
of
this
auto
scaling
and
we'll
come
back
to
this
view
later
for
other
use
cases.
A
Yeah,
so
this
is
just
a
screenshot
of
what
we
saw,
so
you
might
be
wondering.
How
is
it
that
the
horizontal
pod,
auto
scaler
figures
out
how
many
replicas
it
needs
to
add
it's
very
simple
equation.
It
basically
takes
the
ratio
of
the
current
value
for
your
metric
that
you've
defined
like
cpu
or
throughput,
divided
by
the
desired
value
for
that
metric,
and
then
that's
what
it
uses
to
determine
the
right
number
of
replicas.
A
We
have
lots
of
questions
like
how
do
we
set
the
minimum
and
maximum
number
of
replicas?
How
often
do
we
look
for
changes
in
the
metric?
I
might
just
have
a
super
transient
spike
that
I
don't
need
to
auto
scale
on,
but
transient
to
me
might
be
different
than
transient
to
you.
That
might
be
something
you
want
to
auto
scale
on
how
quickly
we
want
to
add
or
remove
pods
and
each
time
we
add
or
remove
pods.
A
A
So
what
we
see
here
is
that
for
the
same
metric
you
can
put
a
you,
can
squash
down
the
maximum
number
of
pods
and
put
a
bottom
on
the
limit
as
well.
And
this
is
good
when
you
have
resource
constraints,
or
you
want
to
provide
a
certain
guarantee
that
at
least
a
certain
number
of
replicas
are
running.
A
You
also
might
add
a
stabilization
period,
which
basically
says
I
don't
want
to
make
changes
too
quickly.
I
want
to
wait
before
making
change
to
make
sure
that
it's
actually
something
I
need
to
scale
on
now.
Looking
at
this
chart,
you
might
say
hey,
but
this
looks
strictly
worse
than
the
one
on
the
left
there's
this
period
of
time
here,
where
we
actually
don't
have
enough
pods
based
on
the
metric
that
we've
defined
and
might
be
wondering.
Why
would
we
even
do
this?
A
The
reason
is
it
reduces
the
pod
turn
so,
for
example,
consider
a
waveform
like
this,
where
the
metric
is
going
up
and
down
and
up
and
down
and
up
and
down.
If
we
were
to
do
the
naive
thing
here
and
just
add
pods
as
soon
as
we
can
as
many
as
we
can
you're
going
to
be
churning
all
of
these
pots
again
and
again
and
again,
it's
not
efficient.
A
It
would
be
much
better
to
take
like
almost
like
a
low-pass
filter
approach
to
this.
You
don't
want
your
car
to
go,
telling
you
every
single
time,
there's
a
bump.
You
want
your
car
to
provide
a
smooth
experience
and
in
the
same
way,
you
want
your
auto
scaler
to
be
smooth
as
well.
So
if
you
were
to
set
a
stabilization
period
for
this,
you
could
actually
keep
this
same
number
of
pods
instead
of
having
it
go
back
and
forth.
A
It's
a
similar
thing
with
capping
the
max
pods
to
add
or
remove
per
period.
Once
again.
If
you
look
right
here,
it
would
look
as
if
this
is
a
strictly
worse
outcome
than
the
one
on
the
left,
because
you
have
a
period
of
time
where
you
don't
actually
have
enough
pause
for
the
metric
that
you're
looking
at.
A
A
You
probably
want
to
react
in
a
more
limited
way
until
you
have
more
information
that
this
is
actually
a
persistent
change
in
your
workload,
so
that
would
be
a
use
for
capping
the
max
step
size
both
up
and
down,
and
you
might
set
these
differently.
You
might
say
hey.
I
want
to
scale
up
really
quickly,
but
I
want
to
scale
down
slowly,
because
I
have
trust
issues
that
the
actual
spike
has
gone
away.
A
A
A
A
A
A
A
There
are
so
many
possible
bottlenecks
in
your
application,
cpu,
obviously
memory
duh
network
people-
don't
always
think
about
this
one
as
much,
but
I've
seen
it
happen
in
practice
the
number
of
worker
threads.
Maybe
you
only
have
two
worker
threads
at
once,
and
this
pod
can
simply
not
take
more
than
that.
It's
just
the
way.
It
is
maybe
it's
using
the
gpu
or
something
like
that,
the
number
of
outbound
connections.
It
might
not
be
the
case
that
your
deployment
is
the
bottleneck.
There
might
actually
be
a
downstream
dependency.
A
A
So,
let's
look
at
an
example.
Application
so
in
the
upper
left
is
the
same
view
as
the
one
we
showed
the
video
of
request
per.
Second,
it's
also
showing
errors
per
second
in
green.
So
what
we're
seeing
for
this
application
before
auto
scaling
is
the
request
per
second
is
going
up
to
about
25
when
it's
hitting
load
and
when
that
happens,
we're
seeing
a
five
second
latency.
This
is
probably
unacceptable.
It
might
be
fine
for
your
workload.
But
let's
just
imagine
it's
a
common
case,
and
this
is
unacceptable
spike
in
latency.
A
A
But
after
we
auto
scale
on
latency,
we
can
see
that
we've
added
a
certain
number
of
pods.
It's
reached
five,
and
this
has
had
a
huge
impact
both
on
the
request
per
second,
which
is
now
four
x
higher
and
on
the
latency,
which
starts
off
at
that
five
second
problematic
zone.
But
then
it
drops
to
two,
which
is
a
huge
improvement
and
the
cpu
usage
is
stable
as
well.
So
this
would
be
a
good
example
of
a
case
where
you
really
have
to
think
through
the
metric
that
you're
auto
scaling
on.
A
A
A
A
A
A
A
A
A
A
turing
machine
takes
an
input
program
and
it
does
the
computation
and
it
writes
an
output
value
to
a
theoretical
tape,
and
something
is
a
turing
machine
when
you
can
do
any
possible
computation
on
that
thing.
So,
for
example,
if
all
I
had
was
the
ability
to
add
one
that
wouldn't
be
turing
complete,
because
that
structure
instruction
is
not
complex
enough
to
do
any
arbitrary
computation
that
I
might
want
to
do.
A
But
there
is
something
that
is
an
instruction
that
would
be
turned
complete
on
its
own.
This
is
called
sublek,
actually,
I'm
not
sure
how
it's
pronounced,
because
I've
only
read
about
it.
So
if
that's
completely
wrong,
I
apologize,
but
anyway,
sublek
is
a
one
instruction
set
computer
and
it's
sufficient
for
touring
completeness
on
its
own.
It
may
be
very
convoluted,
but
you
can
actually
write
any
program.
You
want
using
exclusively
sublux.
A
What
does
it
do?
What
it
does
is
it
says
I'm
going
to
compare
two
values
and
subtract
one
from
the
other.
If
that
is
less
than
zero,
I'm
going
to
jump
to
my
next
instruction
provided
in
the
command.
Otherwise,
I'm
just
going
to
continue
on
with
my
program
like
normal
I'll
leave,
the
proof
of
why
this
is
true
and
complete
to
the
mathematicians.
A
So
how
are
we
going
to
make
a
horizontal
plot,
autoscaler
turing
complete?
What
we're
going
to
do
is
have
this
horizontal
pod,
auto
scaler,
evaluate
a
different
sublet
command.
Every
interval
that
it's
queried
for
its
metrics
so
we'll
take
a
sublec
program,
we'll
execute
one
instruction
at
a
time
and
then
we'll
set
the
number
of
output
plots
and
that
output
pods
is
the
result
of
the
computation.
A
A
A
But
it
could
do
anything
so
basically,
72
is
going
to
correspond
to
the
ascii
value
for
h,
capital
h,
you
can
see
next.
The
auto
scaler
has
computed
a
value
of
105..
It's
happened
really
fast
because
I've
sped
it
up.
So
you
can
see
now
it's
hitting
this
value
up
here.
Well,
it's
actually
107
because
I
add
two
to
every
result,
so
we
never
have
a
negative
result.
But
the
point
is
that
we're
seeing
the
number
of
pods
here
hit
the
result
of
this
computation,
so
it
started
out
72
or
74.
A
Okay
great,
so
this
is
kind
of
the
end
of
the
prepared
remarks,
but
I
just
wanted
to
give
some
shout
outs
to
the
kubernetes
sig
instrumentation
they've
made
it
so
easy
to
get
started
with
defining
your
own
custom
metric
for
horizontal
pod.
Auto
scaling
so
definitely
check
them
out.
If
you
want
to
do
that,
also
check
out
pixi,
which
I
use
for
a
lot
of
these
demos
and
then
finally,
thanks
to
jana
dogen,
who
made
this
awesome
load
generator.
A
C
A
D
B
F
Hi
natalie
thanks
first
for
your
great
and
structured
talk,
one
question
and
lessons
learned
so
using
the
hpa
with
a
sidecar
proxy,
like
we
have
one
pod
ending
up
having
two
containers
like
istio,
so
the
hpa
is
always
using
the
average
of
the
predefined
metrics.
So
I
could
have
istio
using
200
of
the
requested
cpu
and
my
application
only
50.
So
I
end
up
with
a
very
messed
up
state.
Do
you
have
a
lessons
learned
there
or
like
best
practices,
yeah.
A
That's
a
really
great
point.
I
think
that
it
really
depends
on
what
you
want
to
scale
on
like
if
you're
trying
to
look
at
the
joint
utilization
of
the
entire
pot
across
containers,
then
the
built-in
behavior
can
work
just
fine.
But
in
your
case,
if
you
care
a
lot
more
about
one
of
those
than
the
other,
you
might
want
to
create
a
custom
metric
for
the
one
that
you
care
about,
because
it's
your
way
of
specifying
the
exact
thing
to
behave
with.
B
A
A
A
G
A
A
A
I
think
that
probably
the
best
practice
for
that
today
is
try
to
create
a
custom
metric
that
actually
does
that
prediction
and
auto
scale.
Based
on
that,
and
that's
something
that
you
know
the
specifics
of
how
to
do.
That
would
probably
depend
on
what
you're
predicting,
but,
at
the
end
of
the
day,
a
prediction,
a
prediction
is
just
another
metric.
H
Hi
there,
sorry,
if
I
missed
this
at
the
beginning,
but
how
does
this
differ
to
using
something
like
say,
kida,
to
manage
the
auto
scaling
of
pods,
because
kida
supports
kind
of
custom
metrics
through
prometheus?
So
I
was
just
wondering
you
know
what
advantages
does
this
have
over
keita
and
vice
versa?.