►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Autoscaling at Scale: How We Manage Capacity @ Zalando - Mikkel Larsen, Zalando SE
As of October 2019 Zalando has ~140 Kubernetes clusters varying in size from 5 to 400 nodes. This talk goes over how different applications both stateful and stateless are autoscaled using the HPA(Horizontal Pod Autoscaler) and VPA(Vertical Pod Autoscaler) and also the cluster using the Cluster Autoscaler. More specifically the talk explains the limitations and workarounds to each of these scaling strategies and also the techniques used to monitor so that applications always have sufficient capacity. It also discusses common pitfalls while scaling with these controllers and finally concludes with proposed enhancements to these controllers which would make autoscaling more reliable and efficient.
https://sched.co/ZelM
A
Hello,
everyone
and
welcome
to
this
talk
on
auto
scaling
at
scale
and
how
we
manage
capacity
at
salando.
My
name
is
miguel
larsen,
I'm
a
software
engineer
at
zalando
and
I
work
in
the
cloud
infrastructure
team
which
was
responsible
for
managing
our
companies.
Infrastructure
salando
is
the
leading
online
fashion
retailer
in
europe
and
we
operate
in
17
countries.
Across
europe
we
have
11
team.
We
have
an
11
fulfillment
centers,
the
warehouses
where
we
ship
out
all
the
fashion
articles.
A
A
We
have
around
4
000
a
little
bit
more
than
4
000
services,
where
85
is
running
in
kubernetes,
and
we
aim
to
increase
this
to
95
percent
of
all
services
and
because
of
zelando
operates
only
europe.
We
also
have
a
certain
traffic
pattern
where
we
see
less
load
in
the
nights,
because
there's
less
people
active
on
this
website
and
therefore
usual
scaling
in
in
some
of
the
bigger
kubernetes
clusters
is
between
50
nodes
in
the
night
and
350
on
the
on
the
peak
hours
of
the
day.
A
A
And
how
we
manage
scaling
from
the
user
point
of
view,
so
the
users
of
the
kubernetes
infrastructure,
the
the
teams
developing
services
for
solando,
is,
through
horizontal
part,
auto
scaling.
This
is
kind
of
the
interface
that
we
give
to
users
and
we
use
the
horizontal
part
autoscaler,
the
official
one
from
kubernetes
and
the
best
way
to
explain
this
one
is
to
look
at
the
algorithm.
A
A
So
the
calculation
is
that
it
takes
the
current
number
of
replicas
and
multiplies
it
with
the
current
metric
over
the
desired
metric,
and
the
current
metric
is
what
is
currently
being
observed
of
the
running
parts.
So
it
could
be
the
cpu
usage
across
all
the
parts
and
the
desired
metric
is
what
you
have
defined
for
your
application.
This
in
the
scaling
configuration.
So
you
might
say
that
you
want
your
service
to
have
50
cpu
load,
cpu
load
per
per
instance
per
pot,
and
then
you
configure
50
average
utilization
and
then,
if
the.
A
If
the
current
metric
is
reported
to
be
higher
than
this,
then
it
means
that
it
needs
to
scale
out.
And
if
it's
lower
than
this,
then
it
needs
to
scale
in
or
scale
down.
And
one
thing
to
also
note
about
the
algorithm
is
that
it
uses
a
seal,
and
this
is
to
ensure
that
it
it.
It
calculates
a
bit
more
than
needed
in
some
cases.
So
it's
better
to
over
shoot
the
target
than
to
undershoot
the
target,
because
you
rather
run
with
a
little
bit
extra
capacity
than
to
run
with
too
few.
A
So
that's
why
it
uses
the
seal
to
to
ensure
that
the
official
hpa
supports
cpu
and
memory
out
of
the
box
as
metric
types,
and
then
it
also
supports
custom
or
external
metrics
and
because
the
cpu
memory
is
not
enough
in
our
case,
because
we
have
a
lot
of
different
applications
and
a
lot
of
different
use
cases.
We
have
developed
something.
We
call
the
cube
metrics
adapter,
which
is
a
component
that
can
interface
with
the
hpa
and
provide
custom
metrics.
A
So
the
matrix
we
provide
there
is
amazon
sqs,
so
you
can
scale
your
application
based
on
sqsq
length.
So
if
you
have
some
job
that
reads
from
a
queue
you
can
scale
out
the
number
of
workers
based
on
that.
We
also
support
ingress
requests
per
second
metric
to
scale
based
on
the
http
traffic
for
for
ecp,
apis
and
services,
and
we
also
have
a
generic
prometheus
query
interface.
So
you
can
define
any
promises,
query
and
scale
based
on
that,
you
can
also
define
json
metrics.
A
So
if
you
have
a
service
that
just
returns
metrics
in
some
json
format,
you
can
use
this
to
to
scale
on
and
then
we
also
have
segment
checks,
which
is
our
internal
monitoring
solution
where
you
can
also
get
metrics
in
some
different
formats.
We
also
support
and
we
lastly,
also
support
influx
db
queries.
We
don't
use
influx
db
in
solando,
but
this
was
contributed
to
us
from
outside
and
and
the
cube
matrix.
Adapter
is
an
open
source
project
available
and
also
has
users
outside
of
celando.
A
So
yeah,
you
can
take
a
look
at
the
link
I
put
here
in
the
slide
and
just
to
show
how
it
it
looks
like
with
an
example.
Here
is
where
we
we
scale
on
requests
per
second.
So
this
is
one
of
our
services
where
you
see
the
orange,
which
is
the
request
per
second,
and
you
see
the
the
black
line,
which
is
the
number
of
pots,
and
this
shows
clearly
that,
like
it's
useful
to
have
something
that
scales
based
on
the
amount
of
traffic.
A
Of
course,
there
are
also
some
challenges
using
hpa
and
some
limitations
that
is
either
not
supported
yet
or
supported
very
in
very
recent
kubernetes
versions.
So
one
thing
is
scaling
behavior,
which
was
unsealed
kubernetes
118,
a
cluster-wide
setting,
so
you
couldn't
do
it
per
application,
and
another
thing
is
that
parts
with
multiple
containers
inside
are
not
handled
variable
by
the
auto
scaler,
and
I
want
to
give
a
shout
out
to
arjun
nike,
who
was
in
our
team
before
and
that
doesn't
work
at
slander
anymore.
A
But
while
he
was
here,
he
did
a
lot
of
contribution
upstream
to
kubernetes
around
the
hpa
and
he's
working
on
the
on
the
first
issue
and
he's
currently
working
on
the
second
issue,
which
is
not
yet
in
kubernetes.
But
hopefully
we
will
see
it
soon
as
a
fix
for
for
multiple
containers
and
the
the
scaling
behavior
issue
or
the
the
changes
that
were
done
in
in
118
is
that
until
117
and
including
117,
you
just
had
some
settings
you
could
set
cluster
wide
so,
namely
this
downscaling
stabilization
is
interesting.
A
This
means
how
how
fast
a
service
will
scale
down
depending
on
the
metrics,
and
normally
it
was
just
five
minutes.
So,
no
matter
what
kind
of
service
you
have
you
just
have
to
rely
on
these
five
minutes,
but
from
kubernetes
118
you
can
configure
these
individually
per
auto
scaling
configuration
so
per
application.
So,
for
instance,
you
could
do
here's,
and
in
the
example
where
you
set
up
a
lower
scaling
window
from
5
minutes
to
60
seconds,
and
this
would
mean
that
it
would
scale
down
faster.
A
A
So
it's
a
way
to
to
reduce
the
scaling
in
case
it
would
have
been
too
much
so
it
could
be
that
you
have
a
not
consistent
load
on
your
system
and
you
don't
want
to
just
scale
too
aggressively
down,
and
these
behaviors
also
work
on
the
on
the
other
direction.
So
when
scaling
up,
you
can
also
tweak
this.
So
it's
very
useful
for
having
this
pre-application,
where
you
have
different
different
behaviors
that
you
need
to
handle
regarding
the
multi
container
parts
issue,
here's
an
example
of
how
it
how
it
currently
works.
A
So
you
can
understand
what
the
problem
is.
Imagine
you
have
a
pot
with
two
containers
inside
and
they
each
request
some
in
this
example.
They
request
250
millicourse
cpu,
and
they
one
uses
200
millicourse
and
the
other
uses
50
millicourse,
and
you
could
think
of
this
as
the
green
one
is
the
main
application
in
your
in
your
pot
and
the
other
one
is
a
side
car.
It
could
be
like
a
like
a
stereo
sidecar
for
a
service
mesh
or
something
like
this,
or
it
could
be
something
that
ships
locks
or
whatever.
A
The
the
result
of
the
hpa
is
that
it
sees
50
utilization
across
all
the
containers,
so
not
necessarily
wrong
the
calculation.
But
it's
not
really
helpful
for
what
you
want,
because
if
you
want
to
scale
your
main
application
or
your
main
container,
when
it's
at
50
percent
and
it's
as
at
80
percent
in
this
example,
then
it
wouldn't
scale
up.
A
So
this
is
this
is
the
problem
and
the
solution
for
this
problem
that
is
proposed
and
currently
being
implemented
is
to
extend
the
way
you
define
metrics.
So
in
the
in
the
past,
or
currently
you
have
just
you:
can
you
can
specify
a
type
resource
and
it
could
be
cpu?
And
then
you
specify
the
the
average
utilization-
and
this
is
this
across
all
the
containers,
but
in
the
future
you
would
be
able
to
specify
another
metric
type
which
is
container
resource,
and
here
you
can
specify
which
container
that
you
want
to
scale
based
on.
A
A
A
A
Just
to
explain
how
it
works
is
that
normally
you
would
deploy
auto
scaling
groups
with
your
with
your
instances.
So
this
is
running
in
aws
as
an
example.
It
also
supports
dke
and
other
cloud
providers,
but
in
aws
you
have
autoscaling
groups
and
you
place
your
nodes
or
your
instances
in
those
and
then
once
there
is
a
pending
pot,
the
autoscaler
will
first
see
if
it
can
fit
on
an
existing
node.
A
If
not,
it
will
try
to
scale
up
in
node
in
in
one
of
these
availability
zones
and
normally
for
parts
that
don't
have
so
much
requirements.
They
can
run
in
any
of
the
zones,
but
if
a
pod
needs
a
volume,
it
has
is
usually
attached
to
a
single
zone
and
then
the
auto
scaler
needs
to
pick
the
right
zone.
So
in
this
example,
it
would
be
zone
c
that
it
has
to
pick
and
to
manage
this
on
our
side.
We
have
some
abstraction.
A
On
top
of
these,
all
scaling
groups
that
we
call
node
pools
and
a
node
pool
is
basically
a
way
to
define,
like
one
type
of
instance
and
so
on,
that
is
mapping
to
multiple
auto
scaling
groups.
So
by
default
we
met
two
three
autoscaling
groups,
but
we
can
also
reduce
it
to
less.
If
we
want
to
have
a
certain
need
for
that,
but
basically
you
define
a
node
pool,
you
give
it
you,
then
you
get
all
the
scaling
groups
with
the
same
instance
types.
The
same
note,
labels
and
tens.
A
A
A
A
So
a
template
node
is
that
there's
a
pot
that
is
pending
and
the
auto
scaler
tries
to
calculate
what
kind
of
instances
need
to
create
or
what
kind
of
kubernetes
node
it
would
need
to
satisfy
the
pot
and
in
the
in
the
official
auto
scale.
It's
not
so
easy
if
you
don't
have
any
existing
nodes
because
it
relies
on
the
existing
nodes
in
the
cluster.
A
But
if
you're
scaling
from
zero,
you
need
a
way
to
predict
how
the
node
will
look
like
once
it's
created,
and
this
we
have
made
some
changes
around
that
and
we
also
added
support
for
multiple
instance
types
on
aws.
This
is
mainly
to
handle
spot
where
you
can
create
auto
scaling
groups
that
have
different
instance
types
and
then
aws
spot
can
select,
which,
which
is
the
best
for
the
time.
So
if
there's
a
certain
shortage
in
one
instance
type,
it
will
select
another
instance
type
and
give
you.
A
So
we
don't
have
parts
pending
for
too
long
time,
and
last
thing
we
did
is
we
added
some
priority
based
expander
expander
is
a
name
used
in
the
autoscaler
for
logic
that
can
kind
of
figure
out
which
node
to
pick,
which
is
the
best
one
to
pick,
and
we
added
a
simple
one
which
uses
priorities
and
the
way
it
works
is
that
we
have
here
again
the
the
node
pool
overview.
But
you
can
see
the
config
item
that
has
priorities
defined,
and
this
way
we
can
define
the
default
null
pool
with
the
priority
zero.
A
And
then
we
can
define
fallback
node
pools
of
different
instance
types
to
have
things
to
fall
back
to
in
case
we
cannot
provide.
The
aws
doesn't
have
the
the
one
that
we
want
by
default,
and
this
can
both
work
as
just
fall
back
so
going
back
to
an
m4
large
instead
of
an
m5
light.
So
going
back
to
an
older
generation
instance.
A
A
A
Of
course,
the
auto
scale
also
have
some
limits
and
are
not
the
not
the
auto
scale
itself,
but
once
you
figure
solve
all
these
problems
of
the
auto
scaler,
so
it
can
scale
out
and
pick
new
instances
and
so
on.
Then,
what
you
start
to
do
is
you
start
to
hit
limits
and
it's
we
found
that
it's
very
easy
to
hit
the
default
aws
limits
on
on
the
number
of
instances
that
you
can
have
in
your
account,
and
we
also
hit
limits
in
the
networking
layout
so
for
the
aws
limits.
A
Basically,
every
account
that
is
created
has
a
certain
default
set
of
limiters,
so
you
cannot
just
create
an
account
and
create
millions
of
instances
and
then,
in
order
to
get
this
limit
increased,
you
need
to
open
a
support
request
to
aws,
and
this
became
very
time
consuming
because
we
had
to
do
this
for
many
accounts.
Many
clusters-
and
we
had
to
do
this
for
all
of
these
different
node
pools
that
we
create.
A
So
the
first
thing
we
hit
was
this
limit
that
we
couldn't
create
more
than
256
nodes,
and
then
we
found
a
way
to.
We
basically
figured
out
that
the
way
to
deal
with
this
is
to
just
increase
the
side
or
lower
the
lower.
The
side
address
space
to
slash
15
per
node,
because
then
you
have
only
128
addresses
per
node,
but
you
get
512
nodes
and
you
can
go
further
and
select
less
addresses
per
node
and
then
get
more
nodes
in
total.
A
A
This
also
means
that
if
you
want
to
lower
this
cider
to
slash
25
or
less
26,
then
you
also
need
to
change
the
flag
on
the
cubelet
to
have
less
spots,
because
otherwise
you
can
schedule
more
than
you
actually
have
addresses
for
so
this
is
something
to
look
out
for,
and
we
have
basically
configured
this
that
we
just
have
per
cluster.
We
can
configure
this
the
node
sider
and
then
we
automatically
calculate
how
many
parts
per
node
and
yeah,
but
the
limit
we
have
right
now
is
thousand
parts
with
this
current
setup.
A
A
This
is
actually
fine,
but
it
obviously
also
puts
some
loads
on
the
control
plane
so
on
the
api,
server
and
and
so
on-
and
this
is
just
an
example
of
yeah
when
we
skipped
a
thousand,
then
our
latency
goes
from
so
little.
We
cannot
even
see
it
here
until
several
seconds
in
latency
and
yeah.
This
is
obviously
not
ideal,
because
every
time
you
do
keep
yourself
get
pots
or
any
controller
that
needs
to
list
all
the
products
in
the
cluster
will
either
time
out
or
it
will
take
much
longer
than
than
normally
so.
A
A
The
reason
we
scale
vertically
and
not
horizontally
is
that
what
you
want
to
scale
is
that
a
single
instance
can
read
all
the
parts
from
it
cd
and
and
send
it
via
the
api
server.
So
having
many
small
instances
would
not
actually
solve
the
problem
because
then
do
just
have
many
small
instances
that
need
to
read
a
lot
of
data
and
what
you
want
is
bigger
instances
that
can
lead
a
lot
of
data
faster.
A
So
we
already
run
with
two
nodes
at
minimum
to
have
highly
available,
but
we
don't
scale
it
further
horizontally.
Instead,
we
scale
it
vertically
and
we
basically
look
at
cpu
load
as
an
indicator
for
scaling.
It
could
also
be
interesting
to
look
at
memory,
but
we
found
that
cpu
is
enough
to
indicate
whether
we
need
to
pick
a
bigger
instance
and
the
way
we
scale
this
just
by
changing
the
instance
type.
A
A
Auto
scaler
will
pick
the
next
instance
type
sorted
by
by
vcpu
and
memory,
and
one
of
the
most
important
things
is
that
this
automates
a
previous
manual
task,
because
before
we
just
had
alerts
that
would
tell
us
the
api
server
is
on
the
high
load
and
then
we
would
go
and
figure
out.
Okay,
this
is
running
this
instance
type
it's
changing
to
a
bigger
one,
and
now
we
have
automated
this,
which
makes
it
much
easier
to
manage
150
clusters,
and
it's
not
only
good
for
scaling
out.
A
So
it's
getting
bigger,
it's
also
good
for
scaling
down
again
once
the
the
cluster
is
under
less
load,
because
you
don't
want
to
just
leave
running
with
the
very
big
instances
that
are
much
more
expensive.
So
this
helps
us
both
on
cost
saving
and
also
on
on
the
manual
work
that
we
would
have
to
do
in
the
past.
A
Along
with
the
vertical
scaling
of
the
control
plane,
we
also
have
vertical
pod,
auto
scaling,
and
here
we
use
the
vertical
pod,
auto
scaler,
also
from
kubernetes,
and
the
way
it
works
is
that
it
scales
parts
vertically
by
scale
by
changing
the
request
and
limits.
So,
unlike
our
control
plane
autoscaler,
it
doesn't
change
instance
types,
but
it
changes
how
much
request
limits
that
you
define
on
the
parts
it
scales
only
based
on
cpu
and
memory.
A
A
You
want
to
collect
more
more
metrics
and
maybe
execute
more
more
queries,
depending
on
how
many,
how
big
your
cluster
is
and
how
many
resources
you
have
in
the
cluster,
and
you
cannot
just
scale
prometheus
horizontally.
You
need
to
scale
the
individual
instances
to
bigger
sizes
so
vertically
another
example
is
ingress
controller.
A
A
So
we
don't
have
to
run
with
a
very
high
memory
request
for
all
of
the
clusters,
but
it
depends
on
the
cluster
size
and
also
how
much
it's
scaling,
of
course,
there's
also
failure
modes
with
the
vpa.
It's
also,
I
don't
think
it's
that
widely
used
yet
and
there's
some
things
that
are
also
a
bit
troubling
for
us.
A
A
So
for
this
we
also
have
a
fork
of
the
vertical
powder
scaler
similar
to
the
cluster
autoscaler
and
in
our
fork.
We
have
worked
a
lot
of
improving
the
oil
and
kill
handling.
So
whenever
a
pot
runs
out
of
memory,
what
we
wanted
to
do
is
to
recover
as
quickly
as
possible.
So
we
want
the
bpa
to
scale
out
the
resources
higher
and
higher,
and
we
want.
We
made
a
bunch
of
changes
to
make
it
as
quick
as
possible.
A
There's
still
work
to
do
because
there's
still
situations
where
it
can,
where
it
can
take
too
long
to
recover
a
promisius
instance
that
that
ran
out
of
memory-
and
we
also
did
some
other
small
improvements
of
reducing
the
memory
uses
of
the
vpa
components
and
also
which
way
it's
actually
upstreamed,
and
then
we
also
added
timeouts
and
so
on.
For
the
mission
record
book,
which
is
part
of
the
vpa,
so
just
small
changes.
We
have
a
link
for
the
thought
for
all
the
changes
we
made.
If
you
are
interested
in
that.