►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Crack the FaaS Cold Start and Scalability Bottleneck - Cathy Zhang & Rui Zang, Intel
FaaS provides many benefits to the end-users, such as zero maintenance and on-demand auto-scaling. As each new technology brings benefits, it brings challenges. There are two major challenges: cold start latency and autoscaling speed in response to bursty traffic. Cold start latency refers to the time it takes to create a new function instance and get it ready to start execution. Autoscaling refers to the operation of automatically adjusting the number of running function instances to meet the traffic demand. This talk provides a detailed analysis of what causes the cold start latency and the autoscaling bottleneck. It then presents a new approach that reduces the cold start latency through instantiating a new function instance from a combination of its memory snapshot and its essential code chunks. The authors will share their learnings and test results. On the autoscaling part, the authors will share their insight of using an elastic function sandbox to boost the auto-scaling speed.
A
A
A
A
After
that
we
will
deep
dive
into
a
new
approach
of
creating
a
function
instance
from
a
snapshot.
We
will
also
talk
a
little
bit
about
the
auto
scaling
bottleneck
and
our
approach
of
scanning
out
new
function.
Instances
inside
of
existing
microvia
last.
We
will
show
our
test
result
and
compare
the
code
start
latency
between
existing
way
of
starting
a
function
instance
versus
our
snapshot
based
way
of
starting
a
function.
Instance.
A
So
what
is
fast
fast
is
an
event
driven
architecture-based
computing
service
which
allocates
micro,
vms
or
containers
on
the
mount
to
run
the
developer's
function
code
in
response
to
event,
requests
fast,
provides
three
key
features.
The
first
feature
is
automatic,
on-demand
instantiation
of
function,
instances
to
run
the
function
code
upon
a
trigger
event.
A
A
A
The
lambda
functions
are
evoked
3.5
times
more
often
in
2021
than
in
2019,
and
to
support
multi-tenancy
and
security
functions
are
usually
run
inside.
A
micro
vm,
like
sandbox
advance,
provides
a
lot
of
benefits
to
the
user.
For
example,
the
users
do
not
need
to
manage
any
infrastructure,
since
the
first
platform
will
take
care
of
when
and
how
to
create
a
function.
Micro
vm,
the
fast
platform
will
manage
auto
scaling
out
and
in
to
meet
the
peak
value
of
event,
traffic,
the
users
only
pay
per
usage.
A
The
fast
platform
will
not
charge
for
the
micro
vm
resource
after
the
user's
function
completes
execution
and
with
all
these
benefits,
it
also
brings
some
challenges.
One
of
the
challenges
is:
is
the
code
start?
Latency
code
start
latency
refers
to
the
time
it
takes
the
fast
platform
to
create
a
new
function:
micro,
vm
city
out,
properly
load
the
function,
language
runtime
and
get
the
function
manager
ready
to
start
the
function
execution.
A
If
your
function
has
been
scaled
down
to
zero,
it
can
take
a
few
seconds
to
get
a
new
function
instance
ready
to
serve.
The
other.
10
point
is
the
speed
of
auto
scaling
in
response
to
burst
traffic
scaling
up
many
new
function,
instances
in
seconds,
it's
really
hard
to
do,
for
the
obvious
reason
that
it
takes
time
to
co-start
all
these
new
function,
instances
and
get
them
ready
to
start
accept
traffic.
A
On
this
diagram
in
the
slide
shows
the
steps
involved
in
the
function
startup
process
in
a
container
this
this
process
is,
you
know,
is
in
the
container
d
context
and
let's
go
through
each
of
the
steps.
So
when
the
event
trigger
comes
the
container
v
will
pull
the
function
image
from
a
remote
registry.
If
the
image
is
not
at
the
nodes,
local
disk,
then
the
container
d
will
set
up
the
root
file
system
and
start
the
runtime
machine
component.
A
A
A
So
a
few
mechanisms
have
are
being
used
by
current
fast
providers
to
reduce
the
cost
start
latency.
For
example,
some
fast
platform
will
not
delete
a
function
running
instance
after
it
finishes
its
execution
instead.
Instead,
it
will
keep
the
function
run
in
idle
state
for
a
state
extended
period
of
time,
for
example,
for
15
minutes,
hopefully
to
catch
the
next
trigger
event.
A
So
when
the
next
trigger
event
comes,
the
function
that
runs
in
the
idle
state
can
start
serve
the
traffic
without
going
through
the
cold
start
cycle,
but
this
mechanism
introduces
idle
cost,
which
is
not
good.
In
this
presentation,
we
will
introduce
a
new
snapshot
based
mechanism
to
reduce
the
cold
start
latency.
A
We
can
see
that
step
1
through
step,
10
all
contribute
to
the
code
start
latency.
Among
all
these
steps,
we
found
out
that
step.
1
takes
a
lot
of
time
and
step
5
through
step.
10
also
takes
a
lot
of
time.
Of
course,
if
the
function
image
is
already
loaded
on
the
local
disk
of
the
node.
On
due
to
a
previous
run
of
the
same
function,
then
step
1
through
step.
3
can
be
skipped.
A
So,
in
a
new
mechanism
step
five
three
step
ten
will
be
replaced
by
a
single
step,
and
that
is
instead
of
going
through
step,
five
three
step:
ten
to
create
a
new
function.
Instance,
a
new
mechanism
will
create
every
new
function.
Instance
from
that
function's
snapshot
and
different
function,
code
images
will
have
different
function.
Snapshots
a
function
snapshot
was
taken
at
the
time,
the
user
registers
and
uploads
its
function
code
to
the
fast
platform's
platform's
code
registry.
A
A
Then
how
do
we
get
the
snapshot
of
a
user
function?
We
will
get
a
snapshot
by
test
running
the
function
through
a
whole
life
cycle,
and
that
is,
we
will
take
a
snapshot
of
the
function.
Micro
vm's
memory
and
virtual
hardware
states
when
the
function
finishes
the
execution
and
returns
to
the
idle
state
waiting
for
the
next
next
trigger
event.
A
A
A
A
Oh
sorry,
too
much,
oh
here,
okay,
sorry
so
here,
instead
of
you
know,
taking
the
snapshot
after
the
after
the
function
finishes
the
execution
and
the
returns
to
the
idle
state.
We
can
take
the
snapshot
right
after
starting
the
fast
platforms
function
manager,
but
before
it
invokes
the
function,
because
no
user
function
is
invoked,
the
snapshot
will
not
have
any
user
sensitive
data
or
any
unique
ids
associated
with
a
specific
instance.
A
Oh
too
much
okay
to
address
problem.
Four,
we
break
up
the
original
function
code
image
into
two
parts.
One
part
is
the
essential
code
blocks
which
are
needed
to
start
run
the
function
in
majority
scenarios
and
the
other
part
is
non-essential
code
blocks
which
are
either
not
needed
to
run
the
function
or
are
only
needed
in
very
special
scenarios.
A
You
may
have
a
question
on
how
to
identify
the
essential
code
blocks.
We
identify
the
essential
code
blocks
by
running
the
function
in
several
different
test
environments
and
marking
those
code
blocks
that
are
loaded
into
the
function.
Micro,
vm's
memory
during
those
test
runs.
We
then
group
those
code
blocks
into
a
essential
code,
block
layer.
A
A
Now
that
the
original
image
function,
image
has
been
broken
up
into
essential
code
blocks
and
non-essential
code
blocks
sorry
wrong.
Scroll
down
our
approach
can
further
reduce
the
code
start
latency
by
starting
the
function
creation
process
at
the
earlier
time.
Point,
which
is
the
time
right
after
downloading
the
essential
code
blocks
and
the
remaining
non-essential
code
blocks
can
be
downloaded
in
parallel
in
the
background
or
as
needed.
A
A
As
to
those
unique
data
associated
with
a
specific
micro
vm
instance,
we
have
developed
a
small
program
which
will
regenerate
a
new
set
of
unique
data
and
associate
them
with
the
new
function
microvia
and
by
breaking
the
original
function
code,
image
into
essential
code
blocks
and
non-essential
code
blocks.
Our
approach
can
start
the
function
creation
process
at
a
much
earlier
time,
point
which
further
reduces
the
code
start
latency.
A
So,
instead
of
creating
a
lot
of
new
function,
micro
vms,
we
can
speed
up
the
scale
out
process
by
adjusting
by
adjusting
a
existing
mic
running
micro,
vms
resource
boundary
and
creating
more
function.
Containers
inside
that
running
micro
vm,
as
long
as
the
node
that
holds
the
road
micro
vm
has
enough
resource,
and
now
I'm
going
to
hand
over
to
ray
to
talk
about
our
test
staff
and
our
test
result.
B
B
B
B
B
We
tested
three
languages:
python,
node.js
and
java
for
each
language.
We
selected
three
functions
with
their
code
complexity,
spanning
from
simple
to
intermediate
and
to
complex
the
timing.
Measured
for
both
ways
of
creating
new
instances
is
right,
after
the
runtime
shame
is
created
and
before
exacting
the
function
code.
B
The
numbers
are
an
average
of
20
times.
One
of
the
functions
the
blue
bars
are
the
latencies
of
using
the
existing
way
and
the
orange
bars
are
the
latencies
using
the
new
approach,
the
blue
bars.
The
number
for
the
blue
bars
do
not
differ
very
much
so
as
to
the
orange
bars.
This
is
inconsistent
with
our
design,
because
the
time
we
measured
do
not
include
the
functioning
execution,
we
can
see
the
existing
way
costs
a
little
bit
about
1700
milliseconds
and
the
snapshot
based
weight
costs
about
630
milliseconds.
B
B
B
B
B
So,
as
cassie
has
introduced,
one
of
the
challenges
comes
with
snapshot-based
ways
that
we,
how
to
download
the
snapshot
file
before
starting
the
container.
This
page
shows
the
sizes
of
functional
image,
the
essential
code
block
size
and
the
snapshot
size
for
each
of
the
nine
selected
functions.
We
can
see
that
the
essential
blocks
are
smaller.
A
small
portion
of
the
overall
image
says,
which
means
that
only
a
small
portion
of
the
data
are
actually
used
during
the
function
test
run.
B
B
This
is
also
inconsistent
with
our
design,
since
we
took
the
snapshot
before
running
the
function
code,
so
we
can
see
that
even
combining
the
essential
code
blocks
and
the
snapshot
file,
it
is
still
much
smaller
than
the
original
image
size.
This
will
result
in
a
shorter
downloading
time
and
a
shorter
costar
time.