►
From YouTube: Horizontal Pod Autoscaler Reloaded - Scale on Custom Metrics - Maciej Pytel & Solly Ross
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Horizontal Pod Autoscaler Reloaded - Scale on Custom Metrics - Maciej Pytel, Google & Solly Ross, Red Hat (Intermediate Skill Level)
Horizontal Pod Autoscaling enables any cloud based service to handle changing demand while keeping resource utilization at the optimal level. In this talk Maciej and Solly will show how the introduction of custom metrics support makes Kubernetes autoscaling more powerful. You are now able to easily express scalability needs of your application and scale more types of workloads. Maciej and Solly will show how you can set it up for your application in just a few minutes and discuss best practices for using Horizontal Pod Autoscaling.
About Solly
Solly Ross is the SIG lead for SIG Autoscaling, and works on Kubernetes and OpenShift at Red Hat, with a focus on autoscaling and metrics. When not dabbling in other parts of the Kubernetes code base, Solly enjoys a wide range of programming topics, and can be found most places online as @directxman12.
About Maciej
Maciej Pytel works on Kubernetes autoscaling at Google, focusing on Horizontal Pod Autoscaler and Cluster Autoscaler. Prior to Google, Maciej worked on multiple cloud orchestration projects (usually related to OpenStack) at Codilime.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
B
It
works.
It
delivers
data,
it's
pretty
fast,
as
you
can
see
by
the
blurry
background
and
I
didn't
like
it
and
it
turns
out
other
people
also
like
it.
So
they
started
using
it
a
lot
and
now
it
cannot
handle
all
the
load.
So
I
guess
I
need
to
scale
it
and
I
have
some
vacation
time
coming
up
so
I
need
to
do
it
quickly.
Sorry,
can
you
just
help
me
here
and
walk
me
to
my
options?
Yeah.
A
So
I
mean
you
could
create
more
replicas
of
this
application,
which
is
great,
if
you
don't
have
a
ton
of
state
and
you
can
just
create
more
and
they
can
all
handle
your
traffic.
You
could
also
scale
a
single
instance
of
that
application
up,
make
it
beefier,
give
it
more
RAM
and
CPU,
which
would
be
great
if
you
have
a
lot
of
state
and
it's
not
easy
to
scale
out.
B
B
B
B
We
have
this
concept
of
auto
scaling
and
I
mean
the
concept
is
really
simple:
it's
just
adding
all
the
moving
resources,
as
required
by
your
application
and,
as
you
may
have
guessed
from
our
study
before
we
think
about
two
kinds
of
auto
scaling,
so
horizontal
is
adding
more
instances
and
a
solely
set
that
works
pretty
well
for
any
stateless
workload.
Maybe
Excel
Inc
workload
is
a
usual
example.
Workers.
B
B
Is
it
changes
the
number
of
replicas
based
on
the
predefined
target
and
current
load,
and
it
has
been
available
for
a
while
I
think
v1
was
actually
g8
in
like
1.2
timeframe,
but
it
it's
pretty
limited.
It
only
supports
cpu-based,
auto
scaling,
which
is
critical,
but
also
not
enough
many
applications,
so
we're
now
working
on
v2,
which
supports
custom
metrics,
and
this
is
currently
in
beta
yeah.
So
how
do
you
use
HPA
as
most
things
in
kubernetes?
B
It
starts
with
a
YUM,
and
here
is
this
young
Oh
as
much
of
it
as
I
managed
to
put
on
a
slide,
but
it's
not
much
longer
and
the
important
things
that
we
need
to
define
is
really
basically
what
you're
going
to
outer
scale.
So
this
difference
to
have
deployment
or
the
replication
controller
stateful
set
whatever
you
want
out
to
scale
minimum
and
maximum
number
of
replicas,
so
you're
asking
HPA
to
set
the
number
of
replicas
based
on
the
load,
but
please
do
not
go
below
minimum
all
over
maximum.
So.
B
That's
basically
how
HPA
v1
looks
like
so,
let's
see
v2
and
it's
actually
pretty
similar
in
many
aspects,
I
mean
all
this
stuff
is
basically
the
same.
One
thing
that
has
changed
is
now:
instead
of
just
assuming
it's
CPU,
we
actually
have
an
resource
name
in
here
and
current.
You
could
use
CPU
or
memory,
but
this
is
more
a
matter
of
what
metrics
are
available,
then
actually
just
HP
implementation
and
because
it
didn't
make
sense
to
have
a
target
CPU
utilization.
B
B
I'll,
take
the
CCS
and
I
hope
our
network
is
working
as
expected.
Oh
yes,
I
have
it
so
I
have
a
crustless
setup
where
I
have
a
sample
cellular
application.
It's
not
doing
anything.
Much
I
can
actually
call
it
and
it
will
print
okay.
After
doing
some,
CPU
burning
calculation
and
I
have
some
pots
already
heating
it.
So
there
is
some
load
on
this
thing.
B
So
let's
fly
to
outer
scale
it.
This
is
basically
the
yarn
for
the
HPA.
As
I
said,
it's
not
much
larger
than
autoshow
and
I've
just
created
it,
and
now
we
can
see
how
it
works.
So
the
deployment
started
with
one
applica,
which
may
be
actually
pointed
more
explicitly.
It
actually
has
already
been
scaled
up
by
each
page
of
the
replicas
based
on
the
load.
So
this
is
the
actual
load.
It's
168
percent.
B
Let's
quickly
go
to
what
just
happened
and
how
does
HPA
work.
So
basically,
we
have
a
built-in
HBA
controllers
in
kubernetes
and
it
monitors
all
HPA
objects
that
you
may
have
created
and
for
each
of
them,
basically
in
a
loop
for
each
of
them
runs
a
simple
bit
of
logic.
Just
get
the
matrix
use
the
matrix
and
the
predefined
target
to
calculate
the.
What
would
be
the
appropriate
number
of
replicas
and
if
the
actual
number
of
replicas
is
different
than
what
it
calculated
just
update
the
number
of
replicas.
B
So
it
matches
the
calculated
value
and
that
obviously
leaves
out
the
part
of
how
is
this
desired?
Number
of
replicas
calculated,
and
it's
really
simple-
I
mean
it
looks
kind
of
like
math,
but
if
you
actually
think
about
it
all
we
do.
Is
we
just
sum
up
the
utilization
of
everything
to
get
the
total
utilization?
We
divide
that
by
target
we
get
a
number
of
replicas.
There
is
no
thresholds,
no
something
special.
B
A
A
We're
only
a
little
bit
lazy,
alright.
So
so
you
might
have
noticed
that
before
when
the
new
form
was
introduced,
that
this
is
actually
the
metrics
field
is
actually
now
a
list
and
what
that
means
is
we
can
actually
have
multiple
metrics
entries
and
each
metric
entry
can
have
a
different
type
or
this
the
same
type.
In
some
cases
and
the
horizontal
pod,
auto
control,
the
autoscaler
controller
will
calculate
the
target
replicas
count
for
each
and
then
choose
the
highest
one.
So
here
you
can
see,
we
actually
have
two
metrics.
A
A
So
all
the
pods
in
our
deployment,
for
instance,
and
so
what
we
do
is
we
go
and
we
get
this
custom
metric,
we're
gonna
say
it's
like
HTTP
requests
per
second
here
and
we
fetched
that
value
for
each
of
the
pods
currently
in
our
deployment,
and
then
we
average
that
value
up
and
we
compare
it
to
the
target
average
value
that
we
want.
So
here
we
say
one
request
per
second
and
then
we
compute
our
ratio,
like
we
discussed
before
and
use
that
as
a
multiplier.
A
C
A
B
A
No
I
see
we
have
a
problem:
let's,
let's
go
over,
how
we
figured
this
out
yeah.
So
if
you're
familiar
with
kubernetes
objects
and
the
underlying
structure
of
them,
you'll
know
that
when
you
want
to
see
what
the
current
state
of
an
object
is
you
look
at
the
status
just
like
when
you
want
to
specify
what
the
state
should
be.
You
look
at
the
spec,
so
let's
take
a
look
at
the
status
for
a
horizontal
pot.
Autoscaler
object,
so
we
have
pretty
much
what
you'd
expect
here.
A
We
have
our
desired
and
current
replica
count
look
from
the
last
time
we
calculated
them
so
at
our
regular
interval.
We
can
look
at
our
current
and
average
utilization.
So,
like
that's
the
percentage
we
discussed
before
and
we
can
look
at
our
current
average
value,
which
is
the
raw
value
that
it's
using
to
calculate
that
utilization.
But
what
we
really
care
about
here
is
conditions.
Conditions
are
a
way
of
saying
like
what
the
current
state
of
the
horizontal
pot
autoscaler
is
and
whether
or
not
something
is
blocking
it
from
working.
A
So
if
we
take
a
look
at
the
conditions
we
have
here,
we
have
two.
So
if
you
two
conditions
so
the
first
one
is
telling
us
part
where
partially
Coba
static.
Here
we
were
successfully
able
to
fetch
the
scale
of
the
target
we're
trying
to
scale,
but
then
we
got
to
trying
to
actually
fetch
the
metrics
and
we
failed.
We
see
status,
false
and
the
reason
which
is
the
reason
for
robots
is
failed,
get
pods
metric.
And
if
we
look
at
the
message,
we
can
actually
look
at
the
message
in
the
real
case,
yeah.
B
A
We'll
see
we
say
the
API
server
is
currently
unable
to
handle
the
requests
to
get
pods
custom.
Metrics
can
say,
oh
so
that
means
we
are
missing
something
we
are
missing,
something
to
serve
our
custom,
metrics
API.
So
for
for
CPU
in
memory
we
can
get
them
from
something
called
the
resource.
Metrics
API,
that's
served
by
metric
server
and.
B
B
B
You
think
for
me,
cymatics,
and
it
actually
takes
a
few
seconds
to
load
and
I,
wanted
to
use
the
time
to
maybe
go
to
one
more
or
maybe
two
common
misconceptions
that
I
have
seen
people
struggle
with,
so
the
custom
ethics
actually
have
to
come
out
of
somewhere.
So
in
this
case
we're
using
HTTP
requests.
It's
not
something
built
in
my
simple
server.
Just
has
an
endpoint
that
exposes
this
matrix
I'm
hitting
different
replicas
load
balancer,
which
is
why
I
get
different
numbers
but
yeah.
B
Basically,
that's
what
we
need
and
it
should
have
started
by
now.
So
let's
see
it
and
we
have
the
result
and
another
thing
that
people
find
confusing
is
this.
So
this
is
basically
the
desired
value
same
as
with
CPU.
This
is
the
desired
value.
We
actually
set
it
to
one
query
per
second,
and
this
is
the
measured
value
which
seems
to
be
like
over
a
thousand,
but
this
M
here
says
it's
melee
units,
so
basically
it's
a
kubernetes
way
of
representing
fractions
any
fraction,
we'll
be
written
as
like
it
will
be
given
in
millions.
B
A
For
those
of
you
who
are
familiar
with
the
difference
between
floating
point
and
fixed
point,
this
is
kind
of
a
fixed
point
representation.
So
we
go
out
to
milling
units
when
we
need
it
and
when
we
don't
need
it.
If
it's
one
on
the
nose,
kubernetes
will
just
not
display
the
extra
zeros
and
will
just
collapse
to
a
single
unit.
B
A
One
of
the
interesting
things
we
can
do
here
is
we
don't
actually
need
to
scale
on
a
metric
describing
the
pods
in
your
target
deployment.
So,
for
instance,
you
might
have
be
measuring
some
sort
of
latency
on
a
load
balancer,
that's
measuring
like
on
a
per
service
basis,
instead
of
on
a
per
pod
basis,
and
so
for
that
we
use
the
object
metric
type.
Now
the
object,
metric
type
looks
kind
of
similar.
A
We
specify
a
metric
name
like
we
did
before,
but
we
also
specify
a
target,
and
so
this
is
the
object,
that's
being
described
by
the
metric,
so
we
say,
for
instance,
here
we're
measuring
request,
latency
on
the
sample
server
service
and
we
set
a
target
value.
It's
not
an
average
because
we're
only
describing
one
object
here.
B
A
While
that's
going
through,
we
have
a
little
fake
load,
balancer
I
wrote
and
it's
just
measuring
latency
on
HTTP
requests,
and
one
of
the
neat
things
is
with
this
particular
setup.
Note
that
we've
been
describing
metrics
as
describing
a
particular
object
and
not
coming
from
a
particular
object.
So
we
can
have
this
load
balancer.
That
is
exposing
metrics
about
a
particular
object,
and
that
comes
down
to
how
things
are
stored
in
your
monitoring
pipeline
right.
B
So
when
we
describe
our
HPA
right
now,
we
can
actually
see
the
value
and
target
for
each
metric
and
I
said
the
the
target
for
the
latency
unrealistically
low,
which
is
something
that
you
should
maybe
watchful,
because
basically
my
seller
is
way
slower
than
that.
And,
however
many
replicas
we
add,
will
actually
not
go
down
to
ten
milliseconds.
So
we'll
end
up
reaching
maximum
at
some
point.
So.
A
I'd
like
to
point
out,
the
HPA
will
actually
tell
you
this.
We
can
see
we
have
a
third
status
condition
over
here
and
that's
called
scaling
limited
and
the
scaling
limited
status
condition
triggers
normally
it's
false
and
if
it's
false,
it's
everything's
copasetic.
If
it's
true,
it's
an
indication
that
you're
either
running
up
against
the
limits
or
you're
on
your
on
your
HPA
or
you're
in
a
back
off
period.
A
A
All
right
so
so
we
recently
added
this
new
type
of
metrics.
These
are
called
external
metrics
and
these
are
useful
when
you
have
metrics
that
don't
describe
an
object
in
your
kubernetes
setup.
So
this
often
happens
if
you
are
running
on
a
cloud
provider
and
you
have
integration
from
your
cloud
providers
monitoring
system
so,
for
instance,
you're
running
on
stackdriver,
and
maybe
you
have
a
pub
sub-q
and
you
have
some
application
on
kubernetes,
that's
consuming
messages
from
the
pub
sub-q
and
you
want
to
scale
on.
A
You
know
how
quickly
items
are
being
taken
out
of
the
pub
sub-q
and
how
much
backlog
there
is,
for
instance,
well
that
pub
sub-q
isn't
necessarily
associated
directly
in
the
monitoring
pipeline
with
the
kubernetes
object,
so
you'd
use
an
external
metric,
and
so
we
have
a
metric
name
and
we
have
a
label
selector,
and
so
this
is
just
similar
to
a
standard.
Kubernetes
label
selector.
You
know
you
all
know
and
love
them
hopefully,
and
that
gets
translated
into
a
query
to
the
monitoring
pipeline
to
select
this
metric
right.
A
Notice
that,
unlike
object
metrics,
we
actually
have
two
options
right
here
for
the
way
we
specify
our
value
and
that's
something
I'll
talk
about
in
a
moment,
but
we
can
see,
we
have
target
average
value
and
the
way
that
works
under
the
hood
is
that
we
just
divide
the
metric
by
the
number
of
pods
so
that
we,
you
can
set
a
value
on
a
per
pod
basis.
Even
though
the
metric
isn't
necessarily
describing
the
number
of
plots
directly
I.
B
A
One
of
the
things
like
I
just
mentioned,
the
only
thing
external
metric
has
that
little
advantage
there
a
couple
of
advantages:
the
label
selector
and
the
target
average
value,
and
so
we're
gonna
be
bringing
those
two
object.
Metrics,
so
you're
gonna
be
able
to
drill
down
further
into
a
particular
series.
A
So
if
you
have
say
like
already
have
a
metric
broken
up
by
HTTP
requests
on
your
request,
latency
and
you
actually
just
want
to
scale
on
get
requests
or
you
want
to
scale
and
get
request,
latency
different
than
post
request,
latency
because
I
know,
maybe
you
have
some
different
constraints
on
those
two.
Currently,
you
can't
really
do
that
easily,
but
we're
gonna
be
adding
the
label
selector
in.
A
So
if
you
are
curious
to
try
to
learn,
if
you
want
to
learn
more
information
about
this,
if
you're
curious-
and
you
want
to
reproduce
the
demo,
the
demo
code
is
available
at
repo
up
there.
You
can
also
take
a
look
at
the
official
horizontal
plot,
autoscaler
documentation
for
more
information
on
consuming
this
as
a
user,
and
if
you
want
to
stand
up
your
own
Prometheus
adapter
in
front
of
your
Prometheus
instance,
the
repository
is
there
alright.
C
A
Alright,
you
can
use
use
whatever
monitoring
system
you
have,
because
kubernetes
is
not
necessarily
itself
about
collecting
those
metrics
so
use
the
tool.
That's
you
know
really
good
at
collecting
metrics
and
transforming
metrics
of
your
choice
and
then
just
make
allow
it
teach
it
to
speak.
This
standard
kubernetes
api
in
the
form
of
an
adapter,
which
is
a
really
long-winded,
say
of
way
of
saying.
No,
we're
not
gonna
have
that
certification.
B
D
A
We
get
a
lot
of
requests
for
like
how
do
I
scale
on
work
queue
numbers,
fueling,
queue,
ingestion,
egress
ratio,
stuff
like
that,
we
also
get
you
know
a
lot
of.
How
do
I
scale
on
my
HTTP
requests?
Like
we've
been
getting
that
question
since
basically,
horizontal
pot
autoscaler
appeared.
So
those
are
the
two
big
ones,
but
you
know
I've
seen
a
smattering
of
other
small
ones
too.
I.
A
So
to
clarify
for
those
of
you
who
don't
know,
there's
currently
a
back
off
period
in
the
horizontal
pod
autoscaler
that
we've
had
for
a
while
now
to
work
around
some
concerns
about
thrashing,
and
that
is
currently
only
configurable
on
like
a
global
basis
across
your
kubernetes
deployment.
And
the
question
was
whether
or
not
we
could
do
that
on
a
per
horizontal
pod
autoscaler
basis.
So
that's
been
a
common
request
and
we
will
probably
get
around
to
doing
to
to
addressing
that.
A
Eventually,
one
of
the
things
that
we've
wanted
to
do
for
a
while
now
is
to
make
the
algorithm
more
a
little
bit
more
intelligent
so
that
we
didn't
actually
have
to
have
a
back
off
period
at
all,
because
the
back
off
period
is
a
little
bit
of
a
hack.
So
you
may
see
we
made
experiment
around
with
controls
on
that
in
the
short
term.
But
our
long
term
goal
is
just
to
remove
those
entirely
and
make
them
make
them
unnecessary.
A
Have
pondered
it
on
a
rainy
day
on
occasion,
yes,
which
is
a
humorous
way
of
saying
that
I
haven't
gotten
around
to
actually
writing
the
code
out
and
figuring
out
whether
the
you
know
how
much
added
complexity
that
would
add
to
our
to
our
setup
and
and
the
actual
mechanics,
but
I
would
love
to
either.
Do
that
myself
at
some
point
or
see
someone
doing
a
proof
of
concepts.
E
Hey
if
I
had
multiple
external
systems
that
I
wanted
to
query
like
you
know,
you
know,
cystic
data
dog
catch
point
and,
and
they
were
all
feeding
various
bits
of
information
that
I
wanted.
This
scale
on
would
I
be
able
to
do
that
with
with
this
current
set
up
or
what
I
need
to
provide.
Basically,
a
single
system
that
kind
of
consolidates
all
of
them
and
feeds
that
back
at
the
kubernetes.
You.
B
Basically
need
some
sort
of
a
reverse
proxy
adapter,
which
is
basically
a
single
system
from
now.
So
we
have
this
request
before
and
basically
external
Mattox
we've
just
added
in
1.10,
but
this
is
something
we
are
going
to
consider
in
the
future,
depending
on
how
much
how
many
people
want
it
today.
So.
A
For
a
little
bit
more
detail
on
like
why
that's
not
doable
currently
like
without
a
little
bit
of
extra
work,
so
the
API
is
that
we
use
both
the
resource,
metrics
API,
which
provides
CPU
and
memory
and
the
custom
metrics.
A
API
are
actually
presented
as
standard
kubernetes
api.
So
they
appear
on
the
api
server.
You
can
use
clients
to
get
them.
You
can
test
them
out
with
Carol.