►
From YouTube: How Fast is FaaS? Reducing Cold Start Times in Knative - Paul Schweigert & Carlos Santana, IBM
Description
How Fast is FaaS? Reducing Cold Start Times in Knative - Paul Schweigert & Carlos Santana, IBM
Knative is used to build serverless-style systems. One of the key features of these systems is the ability to scale a service up/down on demand, only running pods when they are needed to handle a request. When scaling up, however, users are likely to encounter the “cold start” problem, whereby the latency of when a new pod is ready to handle requests is non-negligible (2-5 seconds or more). Scheduling and making a Pod available in Kubernetes fast enough to provide a FaaS (Function as a Service) experience is a problem that we face today as it involves many components and orchestration. In this talk, Paul and Carlos will discuss how autoscaling works in Knative, the cold start problem space, and the steps taken by Knative to reduce container startup latency. One innovative solution is to pause the container CPU while maintaining state, this will allow for a fast response by having warm containers available and orchestrated. This is a practice typically used in FaaS systems.
A
Let's
get
started
without
talking
about
call
start,
so
just
an
introduction.
We
saw
that
some
people
were
new
to
k
native,
so
to
give
an
introduction
of
what
is
knf
serving.
It
simplifies
the
deployment
like
mentioned
before
of
kubernetes
abstractions
has
traffic
splitting,
which
we
saw
a
great
demo
by
bloomberg,
how
they
use
traffic,
speeding
and
then
rollouts,
and
one
of
the
my
favorite
features
of
is
auto,
auto
tls
https.
A
So
you
define
your
service
and
k
native
serving
takes
care
of
defining
the
route
and
the
traffic
aspect
of
it
and
the
configuration
as
always
in
kubernetes
crds.
You
can
compose
them
as
separate
things,
router
and
configuration,
but
we
suggest
to
use
service
or
service.
We
have
seen
many
talks
about.
That
is
a
scaling
to
zero,
but
I
think
the
most
difficult
problem
from
scanning
to
zero
is
getting
back
to
one
like.
I
always
like
to
joke
next
slide.
A
So
just
for
reminders-
and
I
I
get
this
question
a
lot-
is
people
ask
us
like
how
do
we,
how
k
native
scales
to
zero-
and
I
usually
kind
of
like
I
said
well
and
the
interesting
part-
is
how
it
scales
to
one
it's
the
interesting
thing
not
scale
to
zero
and
some
people
actually
think
we
have
joke
about
this-
that
this
there's
some
magic
and
sprinkles
in
the
cluster.
That
makes
this
happen,
but
actually
there's
engineering
involved
right
and
if
people
want
to
get
you
know
contributions
and
learn.
A
I
know
a
lot
of
people
want
to
learn
how
it
works.
This
is
a
good
opportunity
to
get
involved
and
actually
help
with
coding
testing
and
involving
a
serving.
So
the
working
group
is
always
looking
for
contributions,
so
one
simple
example
is
like
the
life
of
the
query,
wanting
to
make
this
a
little
bit
simpler
for
people
to
use.
But
when
you
start
with
a
kubernetes
cluster,
you
usually
have
a
request.
Coming
in
into
an
ingress
in
our
case
will
be
something
like
contour
courier
istio.
You
don't
have
to
use
istio.
A
We
have
a
service,
it's
a
type
of
service,
but
it
represents
the
same
abstraction.
But
the
key
aspect
is:
there's
no
pods.
So
there's
a
deployment
out
there
for
every
revision,
but
there's
zero
pods.
So
how
do
we
get
that
request
to
a
pod
that
doesn't
exist?
Usually
that's
that's
the
key
question
and
the
answer
to
that
is
the
activator
and
I
love
the
names
that
we
give
these
two
guys
activator
autoscalers.
They
sound
like
transformers.
A
Oh
I'm,
a
fan
of
transformers,
so
we
have
the
activator
and
activator
is
in
charge
in
there
of
like.
If
there's
no
pod,
to
receive
this
request,
I'm
going
to
go
into
the
middle
there.
So
the
endpoints
get
configured
to
be
able
to
capture
that
request
and
take
the
the
aspect,
but
somebody
needs
to
get
one
pot
up.
So
this
is
kubernetes
we're
managing
kubernetes.
So
that's
what
the
aspects
of
kubernetes
is
native
is
cool
k
native,
so
activator
needs
to
tell
like
I
need
the
first
pod.
So
how
does
that
happen?
A
So
that
happens
through
the
activator
communicating
with
the
auto
scaler
and
the
auto
scatter
like
normal
kubernetes?
Who
said
that
the
replica
set
to
one-
and
this
is
where
things
start
slowing
down
telling
kubernetes
give
me
one
pod
and
eventually
a
pot
would
exist,
and
when
the
pot
comes
up,
we
have
to
bring
up
minimum
two
containers
if
you're,
using
insta
with
mesh
with
three
but
in
our
simple
example,
will
be
a
the
cube
proxy,
which
is
another
component
of
k
native.
A
This
is
the
explanation
of
how
to
scale
to
one
and
then
our
application,
so
that
our
application
is
the
one
that
that
boots
up
and
and
comes
up
the
q
proxy,
usually
spins
up
very
fast.
It's
a
very
simple
low,
low
container,
but
the
applications
defines
how
long
your
pot
will
come
up
and
be
ready
and
then
the
eq
proxy
start
like
ping
in
the
app
and
you're
ready
to
see.
If
we
can
receive
that
once
it
does
that
the
activator
says.
A
Okay,
I
can
send
start
sending
requests
to
the
queue
proxy
and
the
key
proxy.
Is
there
to
measure
the
concurrency
for
the
auto
scaler
to
be
scraping.
So
the
autoscaler
now
start
getting
the
metrics
from
that
q
proxy
and
decides.
When
is
enough,
traffic
is
coming
in
that
you
need
two
replicas
and
then
the
next
one
comes
up.
You
have
you
get
the
next
deployment.
A
You
hit
your
cold
start
depending
how
warm
what
type
of
levels
of
heat
you
have
and
then
the
activator
sends
that
to
the
creep
proxy
and
eventually
it
gets
to
a
stable,
stat,
stable
state
that
we,
this
the
system
can
get
to
it.
That
then
the
activator
is
not
in
the
critical
path
and
it
goes
there.
So
this
is
kind
of
like
one
of
the
latest
examples,
and
somebody
was
saying
there
needs
to
be
a
way
to
like.
A
B
So,
just
again
talking
in
terms
of
timing,
what
are
the
things
that
have
to
happen
between
when
the
request
is
received
and
when
the
request
is
processed
so
carl's
kind
of
over
this,
but
just
to
kind
of
recap
here
let
me
speak
into
the
microphone.
The
request
comes
in.
It
gets
routed
to
the
activator,
the
activator
triggers
the
auto
scaler,
the
auto
scaler
updates,
the
replica
count.
The
pod
is
created
and
ready.
The
activator
knows
the
pod
is
ready.
B
The
activator
forwards,
the
request
to
qprox
eq
proxy
forwards
it
to
the
app
the
app
responds
to
the
request
and
the
request
is
processed.
So
this
is
just
kind
of
the
list
of
the
eight
things
that
have
to
happen.
If
we
were
to
represent
this
more
in
kind
of
a
you
know
how
long
each
one
of
them
takes,
you
would
see
that
the
pod
being
created
and
ready
is
what
takes
the
longest
amount
of
time
now.
Why
is
that?
That's
because
there's
a
whole
lot
of
things
that
are
happening
there.
B
B
Carlos
mentioned
the
levels
of
heat
that
we're
pulling
from
a
presentation
that
matt
had
made
previously,
but
we'll
call
them
different
kinds
of
latency
types,
and
what
this
means
is
it's
really
just
referring
to
the
different
types
of
scenarios
on
a
particular
node,
so
there's
kind
of
two
kind
of
main
variables
here.
One
is
the
container
running
on
that
node,
yes
or
no.
B
Two
has
that
image
been
cached
on
that
node,
already,
yes
or
no,
and
so
what
we'll
call
a
cold
start
is
when
that
node
does
not
have
the
container
cached
on
it
and
or
the
image
cached
on
it
and
the
container
is
not
running.
That's
the
cold
start.
That's
what
takes
the
longest
amount
of
time
to
happen
next
would
be.
We
call
a
warm
disk,
that's
when
the
image
has
already
been
cached
on
said
node,
but
the
container
is
not
actually
actively
running
a
warm
memory
scenario.
B
That's
when
the
container's
been
paused,
so
the
container's
up,
but
it's
been
paused,
it's
not
kind
of
actively
serving
requested
as
it's
been
paused
by
the
container
runtime
or
something
like
that
and
then.
Finally,
what
we
call
the
warm
cpu
scenario
where
that
container
is
actively
running
and
can
serve
requests
kind
of
immediately.
So
thanks
to
matt
for
kind
of
those
categories,
there
kind
of
that's
kind
of
the
problem
space
we
find
ourselves
in
now.
The
question
is:
how
can
we
make
things
faster
so
back
to
carlos
yep.
A
A
A
How
do
we
pull
the
bits
that
I
need?
It's
like
a
like,
e-star
gz,
always
don't
use,
always
pull
and
always
pulls,
is
also
something
that
some
users
cannot
avoid
because
you're
working,
maybe
in
a
multi-tenant
environment.
So
you
you
don't
want
another
pod
from
another
user
or
a
bad
actor
to
like
use
your
image
that
you
catch.
So
you
always
ask
always
pull,
but
if
you
can
avoid
that,
always
like
don't
use
latest
tag
and
catch
the
image
on
the
notes,
that's
kind
of
like
the
the
best
one.
A
So
another
one
is
like
getting
the
image
closer.
So
I'll
talk.
If
if
on
a
few
of
them,
the
other
one
is
init
optimization,
it's
like
what
what
can
you
do
in
your
code
or
your
framework
or
the
type
of
initialization
that
you're
doing
right?
What
are
the
things
that
you're
loading
at
boot
time
that
you
can
maybe
do
later?
Or
maybe
you
don't
maybe
the
type
based
on
the
type
of
request
you
never
actually
there's
no
codepath
in
there
and
the
last
one
also
is
hidden
costs.
A
Some
people
are
using
certain
components
or
frameworks
or
something
from
a
cloud
provider
that
they
don't
know
that
is
affecting
the
coaster.
So
I'll
give
two
examples
of
that
around
istio
and
and
cni,
and
then
I
like,
I
said
the
other
strategy
is
try
to
avoid
the
the
co-star
right
instead
of
like
they
they're
costly,
so
see
if
you
can
modify,
maybe
the
concurrency
and
all
the
aspects
to
just
avoid
them.
A
So
we'll
touch
on
some
of
them,
and
this
is
like
a
lot
of
users
need
kind
of
guidance
and
a
lot
of
things
are
available,
so
one
is
always
forcing
the
image
the
image
pool
at
deployment
time.
This
is
something
that
you
can
do
on
your
own,
like,
for
example,
you
can
deploy
with
the
cli
the
candidate
service
and
then
behind
the
scenes,
try
to
deploy
something
that
will
go
to
certain
nodes
right.
A
Maybe
your
your
services
run
in
certain
nodes
from
your
cluster,
so
you
create
a
deployment,
a
demon
set
that
just
pulls
the
image,
but
it
doesn't
do
anything
there's
some.
Also
that
you
can
do
things
like
deploy
the
service
but
never
make
it
run.
The
first
spot.
There's
a
flag
and
an
annotation
for
that,
and
also
we
have
in
k
native,
is
a
crd
where
cr
get
gets
created
per
revision
and
it
has
all
the
information
from.
A
I
guess,
the
parent,
the
owner,
that
created
that
revision
and
it
has
the
url
that
the
controller
does
the
tag
resolution.
So
we
always
like
try
to
match
the
tag
to
the
sha
zoom.
So
we
give
you
have
the
the
long
image
tag
and
you
can
write
your
own
controller
or
or
that
that
knows
what
are
the
images
that
need
to
be
downloaded,
and
so
there's
some
community
repos
that
have
an
implementation
of
that.
But
it
could
be
as
simple
as
watching
this
resource
that
it
comes
up.
A
That
candidate
creates
for
someone
to
create
to
implement
it.
So
that's
one
way
is
off
and
and
again,
even
if
you
have
an
image
registry
that
is
located
in
the
same
availability
zone
or
vpc
or
from
the
provider,
it
would
not.
It
would
not
solve
all
your
issues
just
by
downloading
the
image.
Sometimes
the
image
is
very
small
and
that's
not
where
the
most
of
the
time
is
spent
and
paul
is
going
to
talk
about.
Where
is
that
spent
some
languages?
I
think
somebody
asked
earlier
about
choosing
a
language.
A
I
understand
that
some
companies
or
some
projects-
they
don't-
have
a
choice,
to
select
the
language,
but
certain
functions
like
mauricio
was
explaining.
Certain
functions
are
maybe
done
in
async
way
and
they
can
choose
a
language.
Remember.
Other
functions
can
be
written
in
other
language,
so
you
can
have
a
polyglot
of
languages,
but
be
careful
because
certain
languages,
like
I
show
here
like
for
like
a
straight
java,
will
take
a
long
time.
But
if
you
use
something
like
growl
vm
and
compile
that
with
quarkus
or
just
grab
vm,
you
can
go.
A
You
have
a
faster
speed
up,
and
this
is
kind
of
like
comparing
the
the
boot
time
for
different
languages
so
that
that's
part
of
the
init
optimization
there
and
then
another
one
that
people
I
guess
is
is
for
some
of
us
that
we've
been
involved
of
trying
to
solve
this
issue
for
everyone
in
the
best
way
possible.
Some
users
are
fine
to
set
these
some
knobs
that
we
have,
but
the
thing
is
sometimes
they're
not
aware
that
these
knobs
exist.
A
So,
for
example,
on
the
last
talk
they
were
talking
about
setting
the
mean
scale
right.
Minimum
set
of
replicas
and
some
users
are
okay
at
this
time
by
paying
that
that
penalty
of
having
a
pod
running,
24,
7.
but
it'd
be
running
24
7.,
the
candidate
would
not
shut
it
down.
The
other
way
that
you
can
have
is
another
knob
is
scaled
down
delay,
which
is
once
the
q
proxy
reports
that
there's
no
more
request
coming
in
that
it's
okay
to
scale
to
zero.
It
tells
the
system
hey
wait
this
amount
of
time.
A
Maybe
you
wait
10
minutes
instead
of
being
the
default,
the
default
is
very
aggressive.
Wait
this
amount
of
time
before
you
shut
down
will
terminate
the
pod,
and
that
could
be
a
setting
that
can
avoid
the
the
co-stars
that
we
can
say
avoid
the
have
less
frequency
of
coasters
and
the
last
one
it
could
be
like.
A
Maybe
you
want
not
every
pod
to
be
like
left
longer,
but
maybe
the
last
part
of
the
revision
of
the
deployment
leave
it
alive
a
longer
amount
of
time,
because
there
might
be
a
request
coming
in
likely
coming
in
and
that
that
avoids
a
co-star.
So
it's
about
measuring
this
knob
so
always
measure
and
run
the
load
test.
That
looks
like
there,
your
your
load,
if
it's
bursty
and
measure
what
you
change,
but
there's
knobs
that
are
available,
and
sometimes
we
don't
mention
it,
because
we
want
to
get
to
that
nirvana
place.
A
That
co-stars
are
like
you
know,
physics
is
zero
right
and
that's
not
possible,
and-
and
this
one
is
one
that
I
I
think
for
people
using
istio,
not
everyone
uses
istio,
but
you
need
to
be
careful
when
you
add
istio
and
you
have
it,
and
on
top
of
that,
use
it
in
this
still
mesh
that
you're
adding
another
container
that
needs
to
boot
up.
That
needs
to
do
things
in
your
ip
tables.
That's
the
unique
container
plus
you
have
your
istio
side
card
and
plus
the
two
containers.
A
So
everything
needs
to
come
up
and
be
ready
for
that
request
to
go
and
get
processed,
so
that
will
hit
your
co-start
and
you
just
have
to
measure
how
long
that
will
be
for
you,
and
actually
we
have
a
a
tool
called
k-perf
that
can
help
you
merger
that
the
other
one
that
I
recently
found
out
about
doing
the
end-user
interviews
which,
by
the
way,
I'm
still
looking
for
end-users,
grabbed
me
to
schedule.
An
interview
is
a
type
of
cni
cloud
provider.
A
So
some
cloud
providers
would
assign
you
like
a
elastic
ip
or
real
ip,
and
that
would
take
more
time
to
talk
to
the
cloud
provider
to
get
to
the
pool
to
get
an
ip
assigned
to
a
to
a
pod.
And
this
is
some
sometimes
a.
This
came
from
a
user
that
was
moving
from
burn
metal
on
premises
to
on
cloud
and
they
haven't
modernized
all
aspects
of
their
security
model.
So
they
need
that
pod
to
have
a
certain
ip
address.
So
that's
it's
costing
them
the
code
start.
A
I
actually
say
it's
an
end
user
that
is
in
production,
which
I
was
super
happy
to
hear
about
that.
But
it's
something
that
it
came
up
in
the
conversation
that
is
is
affecting
them
in
co-stars,
so
watch
out
for
those
type
of
things
of
what
are
the
things
that
container
d
or
cubelet
are
doing
with
the
cloud
provider
that
might
hit
your
co-star.
So
sometimes
it's
in
your
hands,
and
sometimes
it's
maybe
modernizing
the
way
you
deploy
these
worker
nodes
and
then
leave
it
to
paul
to
see
what
we're
doing
with
upstream.
B
B
So
in
upstream
we've
been
starting
to
work
with
the
kubernetes
folks
to
see
you
know
what
are
ways
that
we
can
get
containers
to
run
faster,
so
marcus
went
to
sig
note
gave
a
presentation
on
what
kind
of
arcade
native
use
cases
are,
and
we've
been
working
with
them
to
try
to
get
some
of
these
things
upstreamed
and
find
those
things
out.
We
work
with
mike
brown
who's
a
container
d
maintainer
who's
got
a
number
of
great
ideas
that
we're
making
use
of.
B
So
I
want
to
mention
a
couple
of
them
here,
just
so
that
we're
aware
that
these
things
are
coming.
You
know
we
talked
about
the
the
issue
of.
If
you
have
to
always
pull
your
images,
you
never
want
to
use
the
latest
tag
because
it
takes
time
to
have
to
pull
an
image,
but
it
might
be
a
security
requirement
if
you
don't
want
tenants
to
be.
B
Whoops
not
use
the
latest
tag,
but
still
be
able
to
so
you
can
kind
of
catch
your
image,
but
it'll
do
the
security
authentication
check
in
your
image.
That's
going
to
help
speed
up
image,
pools
not
having
to
use
the
latest
tag.
There
there's
a
number
of
performance
improvements
that
are
happening
in
the
cni
layer.
There's
prs
out
there
that
have
reduced.
B
You
know
when
you're
loading,
your
cni
plug-ins,
not
loading
them
in
parallel,
as
opposed
to
in
sequence,
which
speeds
up
the
runtime
stuff
disabling
the
dad,
which
is
dropped
about
a
second
off
the
runtime
which
are
very
good
and
then
in
terms
of
cubelet
performance.
You
know
one
of
the
things
we
do
in
k
native
is.
We
run
probes
to
make
sure
things
are
up
probes
run
at
a
second
interval.
So
if
you
know
it's
not
ready
when
the
probe
first
time,
you
got
to
wait
another
second.
B
B
Just
this
is
you
know
this
is
a
similar
picture
we
saw
before
this
is
kind
of
how
k
native
works.
We've
got
the
activator,
the
auto
scaler
you've
got
a
pod,
that's
running
what
the
container
freezer
does
is.
It
runs
a
separate
freeze,
daemon
and
what
that
demon
does.
Is
it
freezes
the
user
pod?
So
when
a
request
comes
in
still
follows
the
same
process
gets
the
cube
proxy.
The
q
proxy
calls
out
to
the
to
the
demon
sends
it
an
unfreeze
request.
B
B
Then
the
request
goes
to
the
application,
as
normally
the
nice
thing
about
this
here
is
you
can
leave
your
pod
running
and
it's
not
you
know
actively
running.
You
pause
it
in
between
requests.
When
the
request
is
finished,
either
qroc
sends
a
request
back
out
frees
the
pod
pod
gets
frozen,
so
it's
kind
of
out
there
in
an
alpha
right
now.
B
People
could
use
all
right
one
of
the
nice
things
about.
This
is
because
your
this
is
the
difference
between
kind
of
that
you
know
the.
I
think
we
call
the
warm
memory
warm
container,
responding
to
a
pause
or
a
pause
pod
versus
not
knocks,
I
can't
add
in
my
head,
but
two
seconds
ish
off
the
time,
so
it
definitely
speeds
up
the
the
response
to
a
request
when
using
said
thing:
okay
and
then
kind
of
finally
k-perf.
It's
our
performance,
benchmarking,
oh
well,
oops!
B
B
B
B
B
So
you're
gonna
see
it's
ticking
and
you
can't
really
read
what
that
thing
says:
it's
tiny,
but
basically
what
happens
is
when
the
container
gets
its
request.
The
pod
runs.
You
see
it
in
the
logs.
Let's
show
that
one
more
time
real
quick,
so
it's
tick
tick.
Tick!
Then,
when
the
screen
stops,
the
thing
stops,
such
as
kind
of
a
live
demo
of
it
actually
pausing
a
container
as
it
runs.
B
K,
perfect
is
our
performance,
benchmarking
tool,
you
just
kind
of
see
here,
I'm
not
going
to
run
this
up.
Basically
kind
of
you
can
run
a
number
of
pods
gather
benchmarks
on
them
and
kind
of
compare
them
against
it.
Here
we're
just
showing
kind
of
pod
creation
times.
You
can
see
the
yellow
line
at
the
bottom.
That's
when
the
pot
is
scheduled.
B
The
orange
line
is
when
the
container
is
ready,
so
you
can
see
kind
of
the
amount
of
time
that
it
takes,
and
you
can
see
the
different
times
that
that
it
takes
for
the
cube
products
to
start
the
user
container
to
start
so
yeah,
and
that
we
can
run
that
in
the
ci.
Nato
did
some
good
work,
adding
that
into
the
ci,
so
we
can
test
kind
of
how
each
pr
that
we
add,
improves
the
performance
or
doesn't
improve
the
performance,
as
the
case
may
be.
A
So
prs
are
welcome,
so
we
have
yeah,
that's
one
of
the
one
of
the
repos
that,
if
you're
new
to
to
the
community
and
it's
something
that
that
you
want
to
get
started,
it's
a
good
one
to
small
one
and
scope
that
you
can
help
and
and
get
started.
So
with
that,
I'm
glad
that
we
got
the
slides
in
half.
I
thought
I'm
going
to
go
over
questions.
D
Hey
hello,
so
just
wanted
to
understand
exactly
like
what
just
happened
behind
the
scene.
When
you
say
the
container
a
pod
is
paused
like
I
just
wanted
to
understand
in
terms
of
like
when
it
auto
scales
down
to
zero.
We
are
saving
on
resources
with
respect
to
compute.
Right,
like
is
that
a
similar
thing
that
we
can
achieve
during
this,
because
the
core
start
definitely
guests
off
with
the
container,
pausing
and
pausing.
But
what
what
exactly
does
happen
with
respect
to
the
computer
resource
saving?
D
B
Yep,
so
the
question
is
kind
of
when
you
freeze
a
container
what
happens
with
the
resource
usage
so
right
now
it
does.
Is
it
pauses,
cpu
usage,
so
the
memory
still
is
allocated
to
the
pods
you're,
not
saving
on
memory,
but
you
are
saving
on
cpu.
So
this
is,
you
know
it
prevents
bitcoin
miners,
for
example,
if
you've
got
an
idling
pod,
they
can't
you
know,
there's
no
cpu
cycles
that
are
spinning
so
kind
of
at
the
moment.
B
C
A
It's
similar,
if
you
do
like
a
container,
have
different
states,
the
container
can
be
stopped,
can
be,
can
be
running,
but
but
also
there's
another
state
that
is
paused.
So
if
you
are
familiar
with
docker
in
always
been
there,
I
think
you
can
do
docker
pause
and
basically
that's
essentially,
what
is
what
it's
doing,
but
we're
doing
it
with
container
id
and
there's
a
pr.
Is
the
tpr
open
right
yeah
somebody
from
the
community
did
a
pr
to
add
that
to
cryo.
A
B
Just
to
mention,
because
we
were
talking
about
you-
know
contributors.
This
was
a
contributor
who
wanted
this
function
for
cryo.
It
wasn't
implemented
in
crowd,
so
went
over
to
cryo.
Did
the
pr
to
add
the
implementation
in
cryo
and
then
wrote
the
pr
for
the
container
freezer.
So
a
great
example
of
community
involvement.
B
I
think
one
of
the
big
barriers
right
now
is
around
kind
of
the
probing
interface
right
now
we
run
a
readiness
probe
that
checks
to
make
sure
the
container
is
ready
and
if
your
container
is
paused,
the
readiness
probe
isn't
like
that,
which
is
hence
part
of
the
reason
why
we
want
to
add
startup
probes
so
that
we
can,
you
know,
perhaps
make
the
readiness
probe
not
always
there
by
default.
So
I
think
that
that's,
I
think,
that's
the
biggest
thing
and
then
honestly
like
we
need
to
test
it
a
lot
more.
B
A
A
So
how
do
we
teach
kubernetes
to
be
aware
that
they're
small
parts
that
are
paused
not
consuming
cpu,
but
it's
consumer
memory,
so
you
can
squeeze
something
in
and
then
maybe
looking
into
like
the
future
about
moving
a
a
frozen
container
or
a
pot
from
one
node
to
another
pod.
That's
something
that
I
call
it
like.
The
v
motion.
Pod,
I
know
v
motion
is
a
vmware.
A
B
It's
part
part
of
the
handler
chain,
so
we
kind
of
add
that
in
there,
so
the
call
will
go
out
and
we'll
freeze
and
then,
when
the
request
comes
in
it'll
trigger
it
again
to
unfreeze.
So
so
it
should
know
when
it's
frozen
yeah.
I
think
we
could
add
a
flag
and
it
decides
this
first.
I
don't
think
there's
anything
explicitly
in
there
at
the
moment.
But
yes,
it
could
know.
B
B
Cry
stuff:
yeah:
it's
mounting
a
unit
socket
and
then
we
add
the
it
so
it
runs
per
node.
So
it's
a
daemon
set.
It
runs
per
node
that
I
think
it
makes
the
call
over
the
socket
and
we
I
have
to
go
back
and
look
it's
been
a
little
bit,
but
we
mount
the
secret
in
a
in
a
volume
that
then
the
service
account
has
access
to.