►
From YouTube: How Knative Uses Concurrency and Rps (Requests per Second) For Autoscaling - Tara Gu, IBMSDCC
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Knative Uses Concurrency and Rps (Requests per Second) For Autoscaling - Tara Gu, IBMSDCC
One of the values Knative adds is automatic scaling up and down to zero. With a new scaling metric introduced, RPS (requests per second), how does Knative use this information to scale up and down your cluster in addition to concurrency? This talk will show how users can use different scaling settings for solving different scaling problems in Knative.
A
How
are
you
guys
feeling
right
now
towards
the
end
of
the
day?
Are
you
guys
ready
for
the
party
all
right
so
alright?
So
today,
I'm
going
to
talk
to
you
guys
about
Canada's,
auto
scaling
capabilities,
I'm
going
to
talk
about
it
in
the
context
of
request,
based
scaling
and
especially
with
concurrency
and
RPS,
and
kind
of
discuss,
what
kind
of
problems
you
might
have
and
what
kind
of
solutions
can
it
will
provide
for
you?
A
So,
first
of
all,
let
me
first
introduce
myself:
I
started
contributing
to
kubernetes
last
year
in
October,
when
I
was
working
for
the
IBM
product
security
instance
response
team,
so
I
was
doing
kubernetes
on
in
my
free
time
and
like
him
to
cube
Conon.
That
was
the
best
thing
ever
and
after
that,
I
kept
contributing
and
I
joined
a
new
team
this
year,
that's
ibm's,
open
technology
team,
where
I
contribute
to
caitiff
right
now,
so
I
wish
for
IBM
and
I.
Also
love
running
in
my
free
time.
A
Here
is
the
official
definition
of
Canada
I.
Think
you
guys
should
be
very
familiar
with
Canada
at
this
point,
since
it's
been
very
talked
about,
so
it
builds
on
top
of
kubernetes
and
Sto,
and
the
really
helps
developers
to
interact
with
just
the
API
and
to
focus
on
their
code,
and
this
is
kind
of
my
own
interpretation
Matt.
A
Please
correct
me
if
I'm
wrong
here,
I
think
Canada
is
like
a
system
that
really
facilitates
configuration
of
application,
that's
running
on
top
of
kubernetes
and
Heights
complexity
from
developers,
so
that
developers
don't
have
to
place
this
admins
and
it
on
a
more
practical
level.
What
developers
do
is
that
they
provide
an
application
definition
to
2k
native
and
K
native
is
going
to
create
lower
level
resources,
custom
resources
for
the
user,
and
specifically
these
three
types
of
resources.
A
And
on
top
of
that,
one
of
the
key
selling
points
of
candidate
is
auto-scaling
capabilities,
so
in
terms
of
reducing
or
increasing
the
number
of
replicas,
and
it
has
both
the
request
based
and
also
the
resource
based
all
the
scaling.
So
this
talk
is
going
to
mainly
focus
on
the
request
based
auto
scaling.
A
Both
of
them
are
adding
and
reducing
the
number
of
pots
and
when
we
don't
receive
any
requests
in
a
certain
amount
of
time
where
the
time
is
what
the
user
specifies,
which
we're
going
to
talk
about
later,
the
candidate
is
going
to
reduce
the
replica
count
to
zero,
and
today
we're
going
to
explore
some
common
problems.
You
might
have
with
your
application
and
kind
of
how
make
a
native
kPa,
which
is
Canada's
own
class
of
autoscaler,
will
solve
the
problem
for
you
and
the
first
problem
is
bursty
traffic,
aka,
inconsistent
traffic.
A
So
this
what
I'm
going
to
talk
about
can
be
applied
for
both
kpn
and
HP
a
but
as
a
disclaimer
I'm
just
going
to
talk
about
it
in
the
context
of
kPa
with
concurrency.
So
if
the
traffic
is
predictable,
there
are
two
settings
that
you
can
use.
Two
one
can
help
with
code
starts
and
the
other
one
can
kind
of
cap.
The
number
of
instances
that
gets
created
and
both
specifies
the
number
of
instances
and
we're
going
to
see
a
little
example
here.
A
Okay,
so
can
you
guys
see
the
dashboard
here,
the
small
letters?
Okay
great?
So
this
is
the
out
of
the
box,
a
dashboard
that
comes
with
your
server
installation
if
you
follow
the
guide.
So
here
on
the
top,
we
have
a
graph
of
revision,
pod
counts
and
we
can
see
that
there
are
two
lines:
there's
the
actual
pod
count
as
well
as
requested
pod
count,
and
you
only
see
the
yellow
line,
because
the
two
lines
are
overlapped.
A
So
a
few
seconds
in
we
can
see
that
yeah
one
part
is
already
created,
but
down
here,
I'll
scare,
my
Magic's
it
it
doesn't
like
you
see
if
at
this
point
it's
zero,
that's
because
we
don't
have
any
requests
coming
in
because
after
I
create
the
application.
I
have
like
a
wait
period
for
the
FIR
to
come
up
and
then,
after
a
few
seconds,
you
can
see
that
the
pod
count
out
of
scalar
metric
here
has
realized
that,
yes,
we
need
to
create
one
instance,
and
we
can
see
that
here
the
desired
part
count.
A
It
has
a
show
as
one
as
well
as
the
actual
pot
count,
and
then
we
can
see
that
after
a
60
second
duration,
as
you
can
note
that
here
is
12
10
and
then
12
11
after
that,
the
applications,
the
artists
killers
desire,
pod
count,
has
gone
back
to
zero,
but
the
actual
pod
count
of
revisionist
still
remains
at
1,
and
that's
because
we
have
specified
the
min
scale
right
yeah.
So
what
do
we
actually
do
for
unpredicted?
A
Besides
studying
Minsky?
So
there's
one
setting
that
was
introduced
called
target,
burst
capacity
and,
like
the
name
suggests,
it
is
specifying
the
size
of
bursts,
and
there
are
four
sets
of
values
that
you
can
set
here.
Negative
one
means
infinite
and
zero.
It's
like
no
user
provided
value
or
a
specific
size
of
burst.
A
A
So
what
if
you
also
want
to
set
a
heart
limit
on
how
many
requests
can
reach
the
pot
at
any
given
time?
So
you
can
be
more
deterministic
about
the
number
of
pots
that
comes
up
and
this
that's
where
container
concurrency
comes
in
and
that
basically
limits
the
amount
of
concurrent
requests
that
are
allowed
into
the
application
and
requests
that
are
passed
beyond
that
are
being
queued.
A
And
here
are
two
settings
there
are
that
autoscaler
uses
to
help
determine
what
exactly
is
the
target
value
to
set.
So
how
are
these
two
settings
or
sets
depends
on
the
current
value
of
container
concurrency.
So
if
container
concurrency
is
not
zero,
then
this
one
would
be
used
to
help
determine
the
target
value
versus
if
container
concurrency
is
zero.
A
A
So
all
of
these
values
are
actually
have
like
candidates
preset
value,
so
it
can
work
out
of
the
box.
You
don't
have
to
set
any
of
these
words
work
out
of
the
box,
so
on
a
more
internal
level
like
how
does
candidate,
simple
traffic
and
it's
using
the
concept
of
a
larger
window
and
a
smaller
window,
the
larger
window
is
the
standard
window.
A
Is
the
default
window
that
we
collect,
request
based
metrics
such
as
concurrency
or
RPS,
and
then,
when
we
exceed
a
certain
threshold,
then
we're
going
to
start
reacting,
try
to
react
quicker
on
a
smaller
window
and
that's
what
panic
window
is
and
how
that
threshold
is
defined
is
with
the
panic
threshold
percentage
and
you
can
as
a
user.
You
can
tweak
both
of
these
numbers.
A
They
can
be
the
stable
window
here,
as
well
as
the
panic
window.
Percentage
was
defined
as
a
percentage
of
the
of
the
stable
window,
but
keep
in
mind
that
the
smaller
the
window
that
you
set
up
a
stable
window
to
be
the
quicker
like
the
more
jumpy
your
the
reaction
will
be
in
terms
of
adding
and
removing
the
number
of
pots.
A
So
here
is
an
example
of
how
panic
window
and
stable
window
works.
So
here
we
have
specified
using
kPa
as
well
as
concurrency
and
the
target
the
target
value,
we're
setting
to
6
and
the
target
utilization
percentage,
which
is
what's
the
desire
level
that
we
want
to
maintain
and
then
container
concurrency
is
set
to
zero.
A
A
It
also
increases.
But
then
we
see
at
this
point-
and
we
know
that
here
we're
not
increasing
we're,
not
doing
any
more
requests,
and
we
can
see
that
here
the
average
panic
concurrency
has
dropped.
That's
because
it's
a
smaller
window
and
then
we
increase
again
both
the
panic
concurrency,
as
well
as
the
average
concurrency,
when
we're
scaling
from
3
to
4.
A
Ok-
and
the
second
problem
is
that
how
do
we
Scout
to
zero
as
fast
as
we
can?
Obviously,
the
cost
kind
of
depends
on
the
individual
cloud
country
cloud
vendors
pricing
model,
but
in
general,
II
would
like
to
delete
resources
that
we
don't
use
anymore
and
with
candidates
out
of
the
box
scale
to
zero
capability.
A
The
default
amount
of
time
is
90
seconds
to
scout
down
to
zero,
and
that's
because,
as
we
talked
about
before,
we
have
to
first
wait
for
the
entire
stable
window
to
pass,
which
is
the
standard
amount
of
time
that
we
collect
metrics,
and
so
after
the
last
year
has
finished
after
the
last
request,
then
we
have
to
wait
for
an
entire
stable
window
to
realize
that
no
more
requests
are
coming
in.
So
we
need
to
scout
down
at
that
point
and
after
that
window
has
passed.
A
We
have
given
the
system
a
grace
period
and
out
of
the
box
is
30
seconds.
So
it's
a
bit
generous
and
as
user
you
can
tweak
both
of
these
settings
but
keep
in
mind
that
the
stable
window
size
there
is
an
absolute
limit
six
seconds
and
even
with
the
absolute
minimum
is
very
small
because,
as
we
can
recall,
the
panic
window
is
only
a
fraction
of
the
stable
window.
So
with
six
seconds
the
KPA
is
autoscaler.
A
It
collects
metrics
every
two
seconds,
so
we
need
to
be
at
least
twice
amount
of
that
time
to
have
a
good
sampling
sampling
window
and
so
so
forth.
So
let's
say
it's
four
seconds
for
the
panic
stamp
sampling
window,
and
so
that's
like
80
percent
as
the
panic
window
percentage,
which
I
think
is
a
bit
too
high.
So
even
with
six
seconds
is
still
really
low
and
I.
Don't
recommend
going
down
to
that
level
and
with
scale
to
zero
grace
period.
A
A
We're
using
here
is
requests
per
second
of
reaching
the
pod
and
concurrency
is
the
number
of
in-flight
requests,
so
setting
up
RPS
auto-scaling
is
actually
pretty
similar,
so
you
can
use
the
metric
here,
obviously
specify
RPS
instead
of
concurrency
here,
and
the
target
utilization
percentage
is
used
in
a
very
similar
fashion
as
the
container
concurrency
target
percentage,
and
then
the
the
requests
per
second
target
default.
That's
very
similar
to
the
container
concurrency
target
default
that
we
have
mentioned
before
alright,
and
here
are
some
more
resources
that
are
very
helpful.
C
So
what
considerations
do
you
take
into
account
when
trying
to
decide
a
stable
window
and
the
scale
down
to
zero
window?
It
said
purely
because
you
need
the
time
to
make
sure
the
sample
of
resource
utilization
is
accurate
enough.
Were
there
some
other
considerations
that,
like
in
other
word,
why
can't
I
scale
down
to
zero
as
fast
as
possible,
I
see.
A
C
A
C
A
B
D
D
D
Does
that
make
sense
and
there's
other
knobs
to
to
effectively
control
how
over-provisioned
you
are
when
not
scaled
to
zero,
so
that
you
can
avoid
cold
starts
as
well,
that
factor
into
resource
utilization
and
it?
Ultimately
it's
all
about
balancing
cold
starts
versus
you
know.
True:
zero
resource
utilization.
B
So
two
of
the
two
of
the
reasons
mentioned
here
for
the
scale
down
are
for
the
delay
on
the
scale
down
is
that
it
takes
a
few
seconds
from
when
K
native
says
scale
down
until
all
the
HTTP
routing
layer
has
actually
finished
receiving
that
status
and
right
now,
a
number
of
the
different
routing
layers
are
eventually
consistent
and
don't
tell
you
when
they
finished,
so
that
delay
is
to
account
for
the
time
it
takes
between
telling
them.
Please
do
this
thing
and
then
having
finished
it.
B
Anything
from
I
have
to
go
and
fetch
and
unpack
six
docker
layers
in
order
to
get
this
container
started
to
hey.
That's
in
my
page
cache,
but
the
process
hasn't
been
started
recently,
so
I'm
just
going
to
need
to
start
a
new
process
and
if
you
are
interested
I
think
that
this
would
be
a
fascinating
area
for
either
academic
or
commercial
research.