►
From YouTube: How and Why We Rebuilt Auto-scaling in OpenFaaS with Prometheus - Alex Ellis, OpenFaaS Ltd
Description
How and Why We Rebuilt Auto-scaling in OpenFaaS with Prometheus - Alex Ellis, OpenFaaS Ltd
In the Six Million Dollar Man we get the quote “We can rebuild him. We have the technology. We can make him better than he was. Better, stronger, faster.” And with that in mind, prompted by customer feedback we rebuilt the subsystem responsible for scaling OpenFaaS functions. The new and improved version serves the needs of customers better - with the added ability to scale on in-flight requests and CPU (as well as RPS). This wasn’t an easy journey and we think you’ll be able to learn from some of the PromQL we wrote, how we (instrument) collect the data and the issues we ran into along the way. There’ll be PromQL samples, live demos of scaling functions linked back to end-user use-cases.
A
Good
afternoon
prometheus
day,
my
name's
alex
I'm
here
to
talk
to
you
about
open,
fast,
specifically
the
auto
scaling
and
how
we're
using
it,
along
with
prometheus
to
horizontally
scale
out
functions
for
open
source
users
and
for
customers
alike.
But
I'd
like
to
know
something
from
you
who
writes
code,
that's
mainly
what
you
do
put
your
hand
up,
maybe
40
of
the
room
and
who
runs
other
people's
code.
A
A
A
little
bit
of
something
that
you
can
take
away
so
all
of
those
that
raise
their
hands-
you
may
be
doing
this
already,
but
if
you're,
not
recording
rules,
filtering
queries
and
rewriting
labels
are
really
powerful
features
of
prometheus
and
probably
going
to
be
very
useful
for
you
at
some
point
so
serverless.
What
is
the
basic
idea?
A
Well,
the
way
I
like
to
think
of
this,
you
may
have
seen
this
graph
before
is
some
sort
of
progression
or
evolution,
and
sometimes
people
get
confused
because
I
think
well,
you
know
microservices
are
usually
better
than
monoliths,
so
everybody
should
just
use
microservices
and
if
you're
not
it's
bad,
then
we
have
serverless
and
if
you're
not
using
serverless.
Well,
that's
bad!
A
Really,
the
way
I
look
at
it
is
you
use
what
is
right
for
your
team
and
your
technology
and
actually
functions
work
really
well,
alongside
microservices
and
monoliths,
traditionally,
they'll
take
a
request,
process
some
data
and
give
you
a
response.
Just
like
a
promql
query,
takes
in
some
text
and
gives
you
a
graph
or
some
statistics.
In
fact,
you
could
have
a
function
that
ran
from
ql.
A
A
This
idea
that
you
could
write
some
code
and
just
have
it
completely
managed
I
was
using
alexa,
but
then
I
wanted
to
run
it
with
docker,
because
we
didn't
we
weren't
really
into
kubernetes
in
the
same
way
back,
then
it
wasn't
as
popular
as
it
is
now,
and
there
was
just
no
way
of
doing
it.
So
I
had
a
bit
of
an
experimentation
on
my
own.
A
A
A
A
It's
kept
as
an
oci
image,
it's
in
a
registry
and
then,
as
we
start
to
go
up
the
stacks,
where
we
give
you
a
bit
more
value
over
you
doing
it
all
yourself,
okay,
so
the
openfast
gateway
is
a
restful
api
which
allows
you
to
deploy
your
code
programmatically.
I
quite
often
get
asked
so
alex
you've
written
a
book
about
go.
I've
got
my
microservice,
but
how
do
I?
How
do
I
deploy
it
remotely
to
this
machine?
A
Even
if
they're
not
running
kubernetes,
it's
quite
hard
well,
openvas
is
an
api.
You
send
a
json
request.
You've
now
deployed
a
new
version
of
your
function.
Prometheus
will
give
you
centralized
logging.
So
we
don't
have
this
issue
that
bartek
was
talking
about
where
our
pod
has
come
up
and
then
we
scaled
down
it's
gone
and
we've
lost
the
metrics
because
they're
aggregated
in
the
gateway.
A
That
also
means
we
can
scale
with
it
and
that's
allows
us
to
do
asynchronous
messaging
who's
ever
done,
integration
with
a
queue
or
an
event,
source
kafka,
rabbitmq
we
all
have,
and
every
company
you
go
to.
If
they
don't
have
it
yet
you've
got
to
write
it
all.
Well,
you
know
we
bundle
that
for
you,
you
just
send
a
request:
http.
A
A
A
So
this
is
what
it
looked
like
at
dockacon
back
then
we
had
the
big
amount
of
red,
metrics
so
rate,
error,
duration
and
then
look
at
that
right
on
that
side
of
the
screen.
Do
you
see
what
happened?
We
went
immediately
back
to
minimum
replicas
as
soon
as
the
traffic
had
abated,
and
that
might
not
be
what
you
want
in
production
right
if
it's
more
sporadic
so
bear
that
in
mind.
The
other
thing
that
we
had
was
scale
to
xero.
A
An
internal
customer
of
openvas
at
the
time
had
a
gcp
account
that
three
of
them
they
were
using
open
files
in
their
product
and
they
just
said
look
we're
spending
so
much
money
with
all
of
these
nodes
up.
Could
we
just
scale
these
functions
back
because
we're
not
using
them
and
get
the
money
back,
and
it
never
really
been
that
interesting
for
open
fires
and
even
today
the
default
is
to
have
one
replica
all
the
time.
A
A
A
It
then
eventually
goes
off
the
query
and
we're
back
to
zero.
We
query
the
data
and
it
says
zero.
Then
we
invoke
it
and
then
we
scrape
it
and
if
we
check
in
between
then
then
we
can
get
a
gap.
So
we've
done
quite
a
lot
of
work
about
graceful,
draining
of
requests.
Now
anybody
here
done
graceful
draining
it's
a
pain
in
the
neck
and
kubernetes
doesn't
make
it
any
easier.
So
we
automate
a
lot
of
that
for
you
as
well
right,
so
you're
not
going
to
suddenly
drop
production
data.
A
Okay,
so
one
of
the
things
about
open
files
is
we're
really
trying
to
make
the
developer
experience
simple.
Compare
this
to
your
average
pod
spec,
two
fields,
the
function
name,
the
image
is
pretty
much
all
you
need
and
then
we've
got
labels
to
help.
You
tune
the
auto
scaling,
and
this
is
the
old
way,
but
we
had
some
issues
right,
so
we
only
had
that
one
generic
rule.
So
I
could
say
I
want
10
requests
per
second
or
100,
but
I
couldn't
have
both
for
two
different
functions.
A
They
just
didn't
really
work
with
this
either
and
then
hpa
v2,
you're,
probably
thinking
why
didn't
you
just
use
this
alex?
Someone
said
that
in
the
break
we
did
have
some
customers
using
it,
but
they
found
it
incredibly
hard
to
integrate
because
they
had
to
manage
the
objects
themselves
and
then,
finally,
I
did
sort
of
mention,
there's
some
edge
cases
on
scale
to
zero
that
you
have
to
be
aware
of.
A
So
we
had
a
new
go,
had
another
go
at
this
and
this
time,
unlike
in
2016,
when
I
was
in
my
bedroom,
having
a
cool
idea,
we
actually
had
customers
paying
us
who
could
tell
us
how
they
were
using
it
so
kevin
from
search
imports,
mortgage
data,
and
he
does
it
every
day.
They
have
a
huge
amount
of
mortgage
data,
and
then
they
have
a
sales
force
integration
and
they
can
give
a
credit
rating
for
customers
of
their
customers
makes
sense.
A
It
sounds
quite
useful,
doesn't
it
and
what
he
needed
was
to
scale
up
as
quickly
as
possible,
but
be
limited
so
that
a
pod
wouldn't
do
more
than
a
couple
of
requests
at
a
time
it
was
quite
intensive,
so
we
built
an
in-flight
request
mode
which
is
started
but
not
finished,
requests
almost
like
a
queue
depth
and
that
serves
his
use
case
really.
Well,
the
other
one
was
request
per
second
and
we
made
that
more
tunable,
because
clearly
one
value
for
the
entire
system
is,
you
know
it
shouldn't
be
useful.
A
So
william
is
importing
security
logs
dns
logs,
potentially
any
other
alerts
that
are
coming
from
all
kinds
of
legacy:
systems,
running
them
through
different
functions
and
the
requests
per
second
scaling
is
really
effective
for
them
at
checkpoint
software
and
then.
Finally,
the
cpu
is
one
that
I
thought
we
just
wouldn't
need,
because
red
metrics
is
supposed
to
be
the
trendy
new
way
of
doing
everything
and
use
metrics
as
like
the
old
google
way
of
doing
it.
A
A
A
The
labels
change
slightly
as
well.
We
have
three
modes
cpu
which
you
guys
can
probably
understand
quite
well
capacity,
which
has
started
not
finished
requests
and
the
last
one
is
the
rps
from
before
and
a
bit
like
hpa
to
provide
that
familiarity.
We'll
say
the
target
is
that
pod
can
only
do
500
mi,
like
half
a
core.
If
it's
doing
any
more,
we
might
need
two
pods,
and
so
that
gives
us
a
level
of
compatibility
and
people
can
just
adopt
this
without
installing
cada
or
hpa
or
whole
new
load
of
software.
A
And
then
the
scale
to
xero
now
has
a
specific
time
per
function,
because
we
had
some
functions
that
should
scale
down
after
an
hour
and
some
almost
immediately
okay.
So
if
you
can't
see
this
at
the
back,
what
we
have
is
a
single
metric
which
is
from
a
recording
rule
and
we're
able
to
say
or
a
function
name
and
the
scaling
type.
A
A
And
now
our
dashboard
is
so
much
more
useful
than
the
one
I
showed
you
from
dockercon,
because
we
can
correlate
the
cpu
usage
just
for
the
functions
with
no
noise
from
the
rest
of
the
system
with
the
duration,
any
retries,
the
delays
and
the
scaling.
And
here
what
I
did
is
I
invoked
a
function
that
was
concurrency
limited.
A
A
If
you
recognize
this,
it
comes
eventually
from
c
advisor
and
what
we'll
find
is
that
it's
got
stuff
like
alert
manager
and
even
open
version.
We
don't
care
about
that.
The
customers
only
care
about
functions,
but
there's
nothing
in
here.
There's
no
cardinality
like
a
label
that
says
it's
a
function.
So
how
do
you
fix
it?
A
Perhaps
you
can
filter
it
out,
and
so,
if
you've
got
at
least
one
label
in
there
that
you
can
match
with
a
data
set
that
lists
all
your
functions,
you
can
then
clear
that
down,
and
you
won't
have
that
kind
of
data
spilling
out
on
your
dashboard.
So
anybody
here
think
that
might
be
useful
to
them
filtering
joining
data.
A
Then
we
can
put
a
times
and
on-
and
this
is
the
important
thing
we
need
to
have
something
common
in
both
sets,
and
then
we
can
filter
out
all
of
the
stuff.
That
is
not
in
gateway
service
account,
which
is
our
replicas
per
function,
and
what
that
means
is
that
we
then
need
to
kind
of
think
about
this
data.
We've
got
from
c
advisor
and
from
open,
faz's
gateway.
How
do
we
look
at
it
in
one
view?
A
A
A
That
gives
us
the
request
per
second
metric
that
we
needed
so
record
expression
and
labels
definitely
have
a
look
if
you've
not
and
now
we've
got
two
so
we've
got
the
rps
in
there
and
we
have
the
other
strategy
now.
One
thing
I
didn't
expect
is:
by
putting
this
data
into
the
one
chart,
it
would
help
customers
immediately
get
value
because
we
all
know
about
cube
state
metrics,
we've
read
all
the
blog
posts.
We've
been
on
stack
overflow,
but
billy
that
I
showed
you
earlier.
He
forgot
to
put
body.close.
A
A
Not
only
that,
but
because
we've
got
the
data,
we
can
now
infuse
it
into
the
openfast
rest
api.
It's
the
same.
When
you
deploy
functions
with
you
can
list
functions.
We
can
put
it
in
our
ui.
I
was
a
bit
concerned
about
this
before
because
I
thought,
if
we're
going
to
query
prom
ql
every
time
we
show
this
data,
could
it
be
slow?
A
Could
it
add
overhead
linkadi,
do
it
in
their
product?
Other
products
do
the
same
thing,
so
I
thought
we'd
give
it
a
go
and
it
seems
okay
if
it
becomes
an
issue,
maybe
cache
the
data.
But
what
this
now
does
is
give
us
a
live
view
of
the
aggregate
amount
of
cpu
and
ram
for
any
of
the
functions
in
the
cluster
click
on
them
immediately.
We
know
we
didn't
have
this
data
before
now
on
to
a
few
of
the
tips,
as
we've
got
about
10
minutes
to
roll
down
who's.
A
Seen
a
query
like
this:
before
it's
a
service
discovery
node.
Obviously,
we've
got
yeah
service
discovery.
Pods
is
very
common
back
before
we're
using
kubernetes
when
we're
on
docker
swarm.
There
was
one
for
that
and
there
was
a
dns
discovery.
This
is
really
clever
and
I
think
it
came
from
cube
state
metric.
So
thank
you.
A
A
We
just
have
our
own
copy
of
prometheus.
It
only
collects
our
metrics
it
scrapes
very
frequently,
and
it
throws
data
away
very
frequently
as
well,
and
so
we
sort
of
black
box
that
in
or
package
it
in
together,
all
in
one
stack.
So
there's
just
one
helm
chart
to
install
so
we
added
this,
and
that
gives
us
that
extra
data
but,
as
I
said
before,
filtering
sometimes
can
help
you.
But
what?
A
A
So
somehow
we
had
to
throw
off
all
those
characters
at
the
end
and
then
combine
that
with
the
namespace
as
a
new
label,
and
originally
I
was
going
to
write
a
daemon
that
was
going
to
run
the
daemon.
I
was
going
to
read
the
data
emit
the
data,
scrape
the
data
and
use
that-
and
this
was
just
so
much
easier
to
do,
and
so
that
is
what
it
ends
up
looking
like.
A
If
you
think
that
would
be
useful,
you
can
get
the
slides
later
or
come
and
ask
me
for
this,
so
we
just
match
the
first
part.
We
throw
away
the
rest,
and
then
this
is
quite
cool.
We
can
emit
a
new
label
by
joining
two
together.
So
first
of
all,
we
got
roughly
the
deployment
name
and
we
emit
a
new
one,
which
has
the,
which
is
called
function
name
and
that
matches
to
everything.
We've
already
got.
A
A
A
Well,
I
made
this
mistake
and
then
I
saw
a
customer
make
the
same
mistake
when
I
was
looking
at
their
dashboard.
Basically,
we
get
the
value
three
times
and
some
is
the
wrong
thing
to
use
when
you're
going
highly
available,
because
this
is
an
exact
picture
of
how
many
replicas
we've
got.
We
end
up
counting
it
three
times.
We
need
average
instead.
A
So
watch
out
for
this,
especially
if
you're
testing
on
your
laptop
you're,
often
not
going
to
have
a
configuration
you
get
to
production.
This
is
the
kind
of
thing
you
might
see,
but
you
still
do
need
some
so
for
the
function
invocations
and
the
red
metrics,
we
do
need
to
sum
them
up
across
the
gateways.
A
A
A
A
A
I
have
a
lot
of
problems
with
this.
You
know,
and
I
think
quite
often,
people
want
something
that
they
can
relate
to
like
sql,
where
you
can
inner
join
two
tables
on
common
keys.
Again,
we
could
perhaps
do
a
little
bit
more
sort
of
explain
this
technique
to
help
people
get
the
best
out
of
their
data.
A
A
So
if
your
function
or
some
data
you're
relying
on
to
drive
a
control
system,
could
have
you
know,
stale
data,
the
things
that
you
may
need
to
do
around
that,
such
as
what
we're
doing,
which
is
a
graceful
shutdown,
a
big
dance
about
taking
the
end
point
out
of
the
connection,
pool
waiting
a
certain
amount
of
time.
Everything
you
have
to
do
so.
A
Bear
that
in
mind
now,
in
terms
of
future
work
kevin
that
I
mentioned
to
you
before
has
now
asked
you
know:
could
we
scale
based
upon
the
queue
depth
in
that's,
so
there
isn't
really
such
a
thing
as
queue
depth
in
that
or
even
the
new
nats
jet
stream
is
very
difficult
to
do.
But
I've
got
some
ideas.
A
A
A
This
isn't
something
that's
been:
that's
come
up
and
bartek
when
he
joined
the
community
call.
He
asked
the
same
question:
it's.
How
would
you?
How
are
you
scraping
the
metrics
at
the
moment
the
red
metrics
have
been
sufficient,
but
I
could
potentially
see
that
being
an
ask.
What
we're
trying
to
do
is
keep
open
fast
as
simple
as
possible
and
not
add
features
we
think
might
be
useful
but
really
drive
it
on
customer
interest.
A
And
hp
hpa
is
an
obvious
choice
when
you're
doing
this.
So
I
didn't
cover
this
in
the
talk,
but
we
wanted
to
keep
the
installation
requirements
as
simple
as
possible
and
we
were
already
using
prometheus
and
had
a
pretty
stable
system
and
we're
happy
with
it.
This
just
meant
that
we
were
able
to
keep
the
stack
very
lightweight
and
scale
on
exactly
what
we
wanted.
Without
the
prometheus
adapter
cube,
state,
metrics
and
goodness
knows
what
else.
A
Okay
right
well
last
thing
to
say:
then,
if
you
want
to
get
a
bit
deeper
with
go,
perhaps
you're
thinking
about
writing
an
operator
or
kubernetes
controller
integrating
with
prometheus,
I
have
an
ebook,
that's
available.
You
can
find
that
just
google
for
it
every
day
go.
It
might
be
helpful
all
right!
Thank
you.
Okay,
thanks.
Everybody.