►
From YouTube: Autoscaling and Cost Optimization on Kubernetes: From 0 to 100 - Guy Templeton & Jiaxin Shan
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Autoscaling and Cost Optimization on Kubernetes: From 0 to 100 - Guy Templeton, Skyscanner & Jiaxin Shan, Amazon
One of the great promises of Kubernetes is the ability to scale your applications and infrastructure dynamically based on demand whilst optimising cost no matter where clusters run. Autoscaling workloads and clusters without compromising the performance or availability of your applications is not always simple though. At Skyscanner, use of the Cluster Autoscaler (CA), Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA) have helped meet traveller demand in a cost effective way. Guy and Jiaxin will walk through hard earned autoscaling lessons, going from the basics to advanced features. This will include using custom metrics with the HPA, the CA’s expanders, scaling from zero, labeling and balancing between node groups to handle different use cases and challenging to scale applications. They will also cover a number of other useful community tools to help with autoscaling.
https://sched.co/Zemi
A
So
you've
deployed
kits
you've
deployed
your
clusters
into
your
cloud
or
data
center
of
choice.
You've
packaged
up
your
either
shiny
new
apps
or
old
legacy,
apps
into
containers
deployed
them
onto
your
new
case
clusters
and
that's
when
the
person
responsible
for
the
compute
building
your
organization
appears
asking
you
questions
awkward
questions.
A
B
What
we're
planning
to
cover
roughly
enforced
into
four
different
use
cases
horizontal
part,
other
scaling,
vertical,
auto
scouting
plaster,
auto
scaling
and
a
few
other
scouting
related
tips
and
tricks
in
each
of
these
areas.
I'll
take
you
through
the
basic
concept.
First
and
guy
will
share
more
tips
on
those
real-world
use
cases.
B
B
The
different
part
is
hpa,
update
the
pub
graphic
cards
and
vpa
update
the
resources
allocated
to
the
existing
part,
hpa
and
vpa
are
currently
incompatible
and
the
best
practice
is
to
avoid
using
both
together
for
the
same
set
of
the
paws
on
the
right
side.
If
there
is
not
enough
capacity
to
run
pause,
the
ca
pick
up
those
past
impending
state.
The
ca
will
do
some
calculation
to
find
the
right,
node
groups
and
desired
number
of
the
nodes
and
then
request
the
resources
from
cloud
providers.
The
part
can
be
scheduled
later
on
provision
node.
B
B
Hpa
codes
live
in
coop
controller
manager,
which
is
a
little
bit
different
from
c
and
vpa,
so
they
are
located
in
kubernetes,
auto
scholar,
project
the
hpa
automatically
scale,
the
number
of
the
replicas
in
deployment
based
on
the
target
metrics
and
there's
three
different
metrics,
the
building,
metrics
custom
matrix
and
external
metrics.
We'll
cover
this
later.
Each
of
this
metrics
can
be
served
by
one
of
a
number
of
components.
However,
a
single
matrix
like
the
custom
metrics
cannot
currently
be
served
by
multiple
items.
B
A
Yeah,
so
the
first
metric
type
is
resource
metrics.
These
are
the
simplest
of
three
metrics
cpu
and
memory
based
auto
scale.
So
this
is
the
the
sort
of
traditional
auto
scaling
you
may
be
used
to
with
your
cloud
provider
of
choice,
scaling
and
auto
scaling
group
or
equivalent
based
on
the
cpu
target,
cpu
utilization
or
memory
utilization,
and
it's
provided
by
an
api
metrics
that
case
files.
A
So
this
is
the
same
api
path,
that's
providing
metrics,
that's
when
running
cube,
call
top-
and
this
was
this
api-
was
originally
served
by
a
component
called
heapster.
However,
this
was
deprecated
in
case
111.
If
you're
running
is
still
running
this
these
days,
you
really
shouldn't
be
and
should
be
moving
on
to
the
more
modern
ways
of
serving
this
app,
and
it's
now
usually
provided
by
the
metric
server,
this
scrapes
their
resource
metrics
from
cubelets
apis
and
serves
them
up
by
the
cube
api
server
via
a
via
of
api
aggregation.
A
But
if
other
other
containers
inside
the
pod
are
using
a
low
enough,
it
can
drag
the
average
utilization
down
and
therefore
you
may
not
trigger
auto
scaling
when
you
think
you
should
be
the
next
metro
type
is
custom
matrix,
so
this
is
served
on
the
api,
custom.entrans.com
and
there's
no.
Currently,
no
official
implementation,
though
the
most
widely
adopted,
is
the
prometheus
adapter.
There's
a
number
of
other
implementations
from
companies
like
zolando,
google
and
others.
However,
currently
the
prometheus
adapter
is
in
the
process
of
believing
being
adopted
by
sick
instrumentation.
A
So
by
the
time
this
talk
gets
out.
That
may
well
have
changed
and
these
metrics
still
have
to
correspond
to
kubernetes
objects,
and
so
you
can
have
two
types
here:
pod
metrics
things
like
requests
and
flight
per
pod
or
object
metrics.
So
these
are
slightly
more
complicated.
They
still
describe
the
kubernetes
object.
It
has
to
be
in
the
same
name
space
as
the
pods
being
scaled,
so
things
like
requests
per
ingress
and
in
the
same
name
space
as
the
pods
serving
those
requests.
A
So
this
is
this
is
useful,
say
where
you
have
a
service
where
you
know,
for
instance,
you
know
how
many
requests
a
given
pod
can
serve
at
any
one
time,
but
the
cpu
or
memory
may
not
be
a
good
indicator
of
that,
given
that
it
takes
an
indeterminate
amount
of
cpu
per
request,
so
some
requests
may
be
extremely
low
in
cpu,
but
are
still
blocking
an
entire
request.
Therefore,
you
want
to
actually
scale
on
your
number
of
you,
whiskey
processes
that
are
currently
running
rather
than
cpu.
A
A
And
finally,
external
metrics,
so
this
is
served
on
the
sternum
dot
metrics.case.io
api
path.
Again,
you
have
components
registering
to
serve
this
metrics
path
under
api
aggregation.
As
jackson
said
this,
this
can
only
have
one
component
serving
this
metrics
path
at
any
one
time
and
there's
again
a
number
of
implementations.
There's
your
gcp
aws
all
provide
ones
for
their
metric
system.
So
you
can
scale
your
case
services
based
on
metrics
from
your
cloud
provider.
Some
of
the
already
mentioned
custom
metrics
implementations
also
are
capable
of
serving
up
external
documents
like
outside.
A
So,
even
in
the
case
where
one
or
more
of
your
metrics
is
unavailable,
it
will
still,
if
one
or
more
of
the
metrics
says
that
it
should
scale
up,
it
will
take
the
highest
choice
out
of
all
of
those
metrics
and
and
increase
the
size.
Conversely,
if
that
metric,
if
one
or
more
metrics
is
unavailable,
it
won't
scale
down,
because
obviously
it
can't
can't
figure
out
whether
that
metric
would
say
that
it
is
safe
to
scale
down,
and
you
can
also
scale
down
to
zero.
A
So
if
you
have
certain
services,
which
are
you
want
to
scale
up
really
fast
but
scaled
out,
slowly,
and
you
can
now
tune
that
out
of
one
eighth
kubernetes
118
onwards,
with
this
new
part
of
the
horizontal
autoscaler
spec,
where
you
can
see
the
behavior
snippet
here.
So
in
this
case,
it's
saying
at
most
scale
it
down
by
five
percent
every
five
minutes
and
scale
it
down
to
a
minimum
of
zero.
A
And,
however,
if
that's
not
much
use
to
you
if
you're
running,
kubernetes,
117
or
earlier
and
all
of
those
flags
at
that
point
were
still
cluster-wide.
So
if
you
had
two
different
apps
with
two
entirely
different
scaling
behaviors,
you
would
you
would
have
to
make
a
compromise
across
the
entire
cluster.
You
can
still
achieve
that
same
sort
of
behavior,
and
here
you
can
see
that
we
are
in
skyscanner.
A
This
object,
metric
called
downscale
limit
and,
on
the
right
hand,
side
you
can
see
the
prometheus
adapter
config
for
what
that
metric,
actually
maps
to
inside
prometheus
adapter
and
that
basically
achieves
on
for
us
the
ability
to
only
downscale
at
most
10
at
a
time
without
needing
the
kubernetes
118
tunable
behavior,
now
jackson's
going
to
take
you
over.
They
introduce
you
to
the
basics
of
vertical
autoscaling.
B
B
Yeah
for
vpa
many
of
us
may
have
questions
when
we
define
new
applications
like
how
many
resources
should
we
give
to
the
deployment
initially
so
most
of
the
time
we
want
to
give
a
higher
limit
to
handle
the
peak
load,
otherwise,
there's
a
risk.
Our
applications
are
not
able
to
serve
the
big
traffic.
So
even
if
we
like
to
get
the
highest
limit
as
we
can,
we
need
to
spend
some
time
on
the
benchmark
to
get
those
values.
Besides
that
another
challenge
is,
application
is
changing
over
time.
B
The
older
settings
is
no
longer
efficient
as
the
like.
The
daily
traffic
padding
change
or
your
users
exploring
et
cetera
the
vertical
auto
scaling
aims
to
solve
these
problems
automatically
adjust
the
cpu
and
memory
resolutions
for
your
parts
to
help
right
size,
your
applications
up
and
down
to
match
the
demand
and
reduce
the
waste.
B
The
recommended
look
at
the
metrics
history,
om
events
and
the
suggest
the
fitting
values
for
request
and
install
it
in
vpn
object.
The
updater
will
decide
which
parts
should
be
restarted.
Based
on
the
recommendations.
Importantly,
the
updater
respects
the
pod
disruption
budget.
So
you
can
apply
pdb
sphere
to
limit
the
number
of
concurrent
disruptions.
B
The
mutation
admission
workbook
is
used
to
apply
the
recommendations
on
part
submissions
and
there's
also
four
kinds
of
modes
that
you
can
use
in
vpa.
So
the
difference
is
between
them.
Is
that
when
you
want
to
adjust
the
recommended
popular
sources,
if
you
are
not
confident
enough,
we
suggest
you
to
start
from
off
which
only
recommend
the
pod
size
that
doesn't
actually
change
anything.
The
initial
results
reside
the
part
only
when
pods
are
created
well,
the
recreated
and
auto
resides
the
pause
by
restarting
the
existing
ones.
A
guy
will
talk
about
the
vpa.
A
Tips
yeah,
so
vertical
pod,
auto
scaling
is
really
useful
for
singletons,
so
things
like
the
big
prometheus
instance
you're
using
to
monitor
your
cluster
and
also
services
used
by
internal
teams.
If
these
aren't
distributed
around
will
you'll
likely
have
traffic
patterns
which
scale
up
and
down
in
a
predictable
way
and
and
it's
no
use
giving
them
peak
resource
usage
and
burning
money
during
those
quiet
periods
where
no
one's
in
the
office.
A
So
in
this
case
you
can
see
graft
over
a
week
the
cpu
utilization
of
a
prometheus
instance
inside
skyscanner,
and
you
can
see,
there's
very
obvious
traffic
peaks
when
people
when
engineers
are
in
the
office
and
then
it
drops
down
significantly
to
roughly
a
third
at
the
lowest
points
of
its
peak
usage.
These
sorts
of
things
are
the
prime
candidates
for
vertical
pod,
auto
scaling
where
you
can
easily
achieve
a
huge
improvements
in
your
resource
usage.
A
So
there
are
a
few
limitations
with
vertical,
auto
scaling.
So,
as
jackson
already
mentioned,
you
shouldn't
really
use
it
in
conjunction
with
resource-based
hpas
and
currently,
as
the
two
will
conflict,
as
obviously
the
vpa
will
be
trying
to
change
the
resource
limits
for
requests
at
the
same
time
as
the
hpa
is
trying
to
scale
on
based
on
those
things.
You
can,
however,
use
them
where
you're
your
hpa
is
set
up
using
a
customer
external
metrics
and
your
vpa
is
then
modifying
the
resource
requests
and
currently
modifying
the
resource.
A
Requests
requires
recreating
the
entire
pod,
meaning
a
pod
restart,
so
that
will
result
in
disruption.
That's
why
us
you
should
make
use
of
pdbs.
As
jackson
mentioned,
there
is
work
on
going
to
try
and
mitigate
this
and
that
at
that
point
the
auto
mode
will
become
much
more
useful
where
it
should
be
able
to
modify
the
request,
resource
requests
and
limits.
A
B
Class
autoscaler
scale
your
cluster,
based
on
the
pending
cost.
You
periodically
check
whether
there's
a
pending
cost
and
increase
the
size
of
the
cluster.
If
more
resources
are
needed,
if
nodes
are
underutilized
and
all
parts
could
be
scheduled
even
on
fewer
nodes
in
the
cluster,
so
plus
the
autoscaler
will
try
to
remove
android
utilize
node
down
to
the
minimum
size
of
the
cluster,
and
the
guide
will
talk
about
the
utilization
graph
here.
A
Yep,
so
we
have
a
graph
here
which
shows
that
so
we
can
see
here.
A
bold
red
line
and
the
amber
sort
of
filled
in
section
is
the
minimum
utilization
that
we
have
on
a
set
on
our
cluster
of
90
and,
and
you
can
see
as
soon
as
the
blur
red
line
and
sits
underneath
it
for
a
while.
It
becomes
considered
as
a
candidate
for
scale
down
by
the
cluster
auto
scaler,
at
which
point
it
cordons
out
drains
it
and
starts
no
pods
start
moving
off
it.
A
B
Cool
here's
a
detail,
scaled
down
process
you
in
each
scan
loop
c
will
mark
under
utilized
nodes
and
and
native
nodes.
At
the
same
time,
ca
uses
kubernetes
scheduler
code
to
simulate
if
the
past
on
a
needed,
node
can
be
moved
to
other
nodes.
So
after,
like
a
well
empty
node
without
the
path
will
be
recycled.
First,
in
this
case,
if
those
ender
utilize
the
nodes
as
filled
and
needed
after
10
minutes,
ca
will
drain
the
parts
on
them
and
remove
the
remove
the
node.
B
To
let's
talk
about
the
expanders
in
your
clusters,
you
may
have
multiple
nodes
groups
and
when
ca
try
to
bring
bring
more
capacity
to
the
cluster,
you
need
to
decide
which
node
group
to
scale
up
you
see
it
brings
a
concept
here
called
expanders.
So
there
are
four
different
extenders
here
like
the
random
expander
priority,
expander
price
expander
and
the
list
always
expander.
It's
very
important
to
understand
these
expanders
to
optimize
your
cost
and
performance.
You
can
choose
the
right
one
based
on
your
needs
or
customize
your
own
expanders.
B
A
So
what
happens
if
you
have
batch
jobs
or
jobs,
which
you
don't
need
to
run
immediately,
so
you
can
use
combining
the
use
of
pod
priorities
with
the
cf
flag,
expendable
pods
priority
cut
off,
so
you
can
use
this
to
avoid
the
ca
scaling
up
and
purely
for
ultra
low
priority
jobs.
So
if
you
have
jobs
which
are
not
important
enough,
that
you
want
to
spend
money
on
nodes
to
scale
up
the
cluster
but
just
want
to
bin
pack
around
and
the
rest
of
the
workload
find
their
spare
capacity
in
the
cluster.
A
This
is
ideal
for
that
and,
if
you're,
using
the
priority
expander
as
jackson
mentioned,
how
do
you
fall
back
to
on-demand
instances
when
spots
or
preemptable
instances
or
your
cloud
providers
equivalent
are
out
of
capacity?
And
so
you
can
create
on-demand
node
groups
with
lower
expansion,
priorities
and
spot
instance
node
groups
with
higher
priorities.
B
So
there
are
some
common
problems
like
these.
Like
you,
you
have
multiple
spot
note
groups,
and
some
of
them
are
like
shout
out
capacity,
so
ca
will
try
to
wait
for
the
no
calming
it
takes
up
to
the
next,
no
probation
time
to
move
to
the
next
note
group
if
the
curing
notebook
is
not
available.
However,
if
your
next
note
group
is
the
spot
group
again
and
it's
very
likely,
the
capacity
is
limited
as
well.
So
there's
two
things
you
can
do
here.
B
The
first
thing
is,
you
can
turn
the
max
node
provision.
Time
reduce
the
time
for
ca
to
move
to
the
next
node
group.
So
there's
there's
another
choice,
use
the
mixed
instance
policy.
This
is
the
aws
asd
feature
and
maximum
policy
allows
you
to
have
a
different
spot
instance
in
one
asg.
The
good
thing
is
that
even
some
of
the
instances
are
short
of
capacity.
It's
very
likely.
You
get
other
instance
types
instead
and
the
using
instance.
B
The
mixed
things
policy
has
another
advantage,
so
you
can
only
have
like
one
or
two
like
asgs
and
help
reduce
the
scan
loop
interval.
Latencies
remember:
it
will
be
better
to
use
the
similar
instance
with
the
same
amount
of
memory
and
cpus
in
one
mixed
instance
policy,
because
it
helps
ca
on
the
simulations
to
calculate
the
desired
number
of
the
node.
B
Accurately
next
we'll
talk
about
the
gpu
optimizations,
so
gpu
node
is
a
little
bit
different
from
cpu
nodes,
because
the
gpu
resources
need
to
be
advertised
by
device
plugins,
which
takes
longer
time.
The
common
problem
using
gpu
node
is
c
bring
up.
The
node
gpu
is
not
ready,
then
casing
pog
cannot
be
scheduled
on
the
newcoming
node,
since
a
lack
of
the
gpu
resources.
So
if
you
apply
the
label,
the
ca
will
wait
for
gpu
to
become
ready
and
these
problems
can
be
resolved.
So
there
is
no
analysis
with
scale
ups.
B
Another
case
in
scale
down
is:
when
you
apply
the
label,
gpu
metrics
will
be
honored
in
scale
down.
One
case
is
if
gpu
is
underutilized,
even
if
the
cpu
and
memory
utilization
is
high,
this
node
still
becomes
a
skilled
scaled
down.
Candidate
and
a
lot
of
users
need
problems.
Scaling
up
from
zero
ca
actually
doesn't
have
any
like
template.
In
this
case,
it
needs
to
build
its
template
from
the
cloud
provider
for
simulations.
B
However,
cloud
provider
has
limited
information
in
order
for
ca
to
build
an
accurate
template
user
need
to
tag
the
resources
in
auto
scaling
group.
So
the
common
cases
here
is
a
custom
metrics
and
custom
metrics
and
labels
for
node
affinities
stands
for
toleration.
So
this
guarantees
ca
makes
the
right
schedule
scheduling
decision
even
without
existing
node
and
guy
will
continue
on
the
gauchos
with
the
ca
next.
A
A
A
This
prevents
runway
scaling,
controls
your
costs
and
allows
you
to
have
honest
conversations
with
your
users
about
how
much
resources
they
actually
need
for
their
workloads
and
in
addition,
you
can
set
the
maximum
size
with
the
node
groups
and
to
limit
the
scale
of
the
cluster
on
the
cluster
autos
get
other
side
to
prevent
runaway
scaling
there
and
make
use
of
pod
disruption.
Budgets,
cluster
autoscaler
respects
these
fine
draining
nodes,
and
this
is
a
way
to
ensure
that
you
don't
end
up
with
accidental
outages.
A
Whilst
the
cluster
auto
scaler
is
draining
nodes,
and
the
other
thing
to
be
aware
of
is
the
cluster
of
scale
that
doesn't
yet
support.
All
cloud
providers,
all
the
big
ones
are
covered.
However,
there's
work
on
going
to
decouple
the
cloud
providers
and
support
a
plugable
cloud
provider
over
grpc
implementation
to
make
it
easier
to
implement
new
ones.
A
This
is
a
quick
list
of
the
flags
that
we
think
are
most
valuable
to
look
at
tuning
away,
potentially
away
from
their
default
values
depending
on
your
workloads
within
the
cluster,
auto
scaler,
so
things
like
scale
down
delay
after
add
or
scale
down
utilization
threshold,
for
instance
that
defaults
to
fifty
percent.
We
think
some
guy's
going
to
rerun
that
at
80
or
90
and
depending
on
the
cluster,
so
the
cholesterol
scaler
is
far
more
likely
to
consider
no
just
candidates
for
scale
day
and
now.
B
There's
some
other
community
projects
in
other
scaling
like
areas
like
add-on,
resizer
and
the
cluster
professional,
auto
scholar.
They
both
scale
the
resources
based
on
the
cluster
size
as
the
clusters
scale
up
and
down.
So
you
can
think
add-on
resizer
a
much
less
sophisticated
vpa.
We
use
the
linear
formula
to
calculate
the
request
limit
values
of
resources
based
on
the
cluster
size.
B
Normally
it
is
used
to
scale
metrics
related
add-ons,
for
example,
when
your
class
will
grow,
you
probably
want
a
large
matrix
server
cluster
proportional,
auto
scaler
is
normally
used
for
scaling,
dns
replicas
horizontally
to
ensure
enough
pulse
to
meet
the
dns
query
needs
of
the
cluster,
but
nothing
prevents
you
from
putting
these
q
autoscaler
or
other
type
of
workflows.
Beside
the
matrix
and
dns
and
the
kubernetes
event
event
driver,
auto
scouting
was
released.
Last
group
account.
A
Those
strategies
can
vary
depending
on
your
clusters,
environments,
cloud
provider.
The
thing
that's
most
likely
to
yield
the
best
return
on
investment.
Is
your
horizontal
pod,
auto
scaling
enabling
your
service
owners
to
figure
out
what
are
the
bottlenecks
for
them
and
what
metrics
they
want
to
scale
on
and
enabling
them
to
do
that,
whether
it's
through
resource
metrics,
through
custom,
metrics
or
through
external
metrics?
That
analysis
is
likely
to
give
you
the
most
improvement
the
quickest.
A
A
That
is
a
good
question
in
sky
scanner.
We
have
our
own
blue
green
deployment.
Tooling.
We
use
that
and
we
set
the
minimum
to
be
the
current
desired
count
when
we
started
deployment,
so
we
still
allow
it
to
scale
up,
but
we
limit
how
much
it
can
scale
down
and
during
a
given
deployment,
and
that's
that's
largely
due
to
the
behavior
of
our
our
deployment
tooling.
A
Do
you
have
any
suggestions
on
how
to
effectively
auto
scale
when
not
using
a
cloud
provider
offered
cluster
at
uksgk?
That's
that
will
depend
a
lot
on
the
setup
of
your
cluster
and
what
sort
of
auto
scaling
you're
you're
asking
about.
If,
if
you
want
to
put
more
details
into
slack
channel,
I
can
give
you
a
better
answer
than
that,
which
is
the
better
approach.
A
I
think
I've
got
just
enough
time
to
answer
one
more
and
if
I
use
githubs
and
specify
resources,
this
might
like
lead
to
the
vpa
operator
conflict
which
might
result
in
constant
pod
restarts.
Am
I
right?
I
believe
that
is
the
case,
though
currently
skyscanner
don't
use
a
github
space
workflow,
so
I'm
not
actually
certain,
but
I
believe
yes,
if
it
was
trying
to
reconcile
the
pods
running
in
the
cluster,
with
their
resources
against
what
you
were
desiring
yeah.