►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Supporting Long-Lived Pods Using a Simple Kubernetes Webhook - Clément Labbe, Slack
Today's applications strive to boot fast, be stateless, and handle unexpected terminations gracefully. However, some applications like distributed caches can take a while to warm up to a running state, while batch workers would rather avoid being terminated before they're done. At Slack, such applications found their home in Kubernetes thanks to a two-sided system: one one hand an admission webhook injects tolerations in pods to inform their requirement to be long-lived, and on the other hand a custom service taints nodes with their uptime. This results in pods desiring a long life to be scheduled on young nodes less likely to be terminated early. This talk will first describe how to write a simple Kubernetes admission webhook (https://github.com/slackhq/simple-kubernetes-webhook) to inject tolerations in pods, then move onto the symbiotic node tainting system, and end with gotchas and some metrics on how this long-lived pod support is used at Slack.
A
So
I
work
in
a
cloud
compute
team,
that's
like
and
I'm
based
in
melbourne.
We
have
about
half
the
team
in
australia
and
the
other
half
in
the
united
states.
So
let's
look
at
some
numbers
to
give
you
some
context
about
like
what
we
are
dealing
with
on
a
good
day.
At
slack,
we
get
about
16
million
concurrent
users.
A
A
A
Okay,
so
in
an
ideal
world
applications
are
12-factor,
they're,
stateless,
they
boot
instantly.
They
scale
out
infinitely
without
any
drain
on
infrastructure,
they're
fun
to
work
with,
but
unfortunately
we
don't
live
in
an
ideal
world.
Instead,
we
live
in
a
growth-obsessed
capitalist
society
and
we
have
to
deal
with
applications
that
are
stateful
that
can
be
slow
to
warm
up.
That
can
be
intensive
on
infrastructure
when
they
do
scale
out.
A
So,
while
at
slack
we
do
have
some
ideal
applications,
those
were
the
first
ones
to
migrate
from
our
original
chef,
compute
platform
to
our
kubernetes
platform,
and
so
today
we
are
left
with
a
long
tail
of
unruly
ducks,
which
are
applications
that
are
hard
to
move
from
one
platform
to
another.
So
this
is
peach,
apricot
and
plum
my
ducks-
and
they
are
very
cute
friends.
A
Okay,
so
back
to
work,
let's
look
at
some
examples
of
applications
that
fit
in
that
less
than
ideal
category.
So
first
I've
got
batch
jobs.
So
while
today
you
could
design
a
bad
job
that
could
gracefully
survive
a
restart
without
losing
any
of
its
work
by,
for
example,
checking
checkpointing
its
work
onto
an
external
storage.
A
A
When
you
want
new
replicas
it
need,
they
need
to
pull
some
existing
data
from
the
existing
nodes
to
warm
up
and
get
populated,
and
so
that
can
be
slow
to
warm
up,
and
that
creates
a
lot
of
internal
network
traffic.
So
if
you
lose,
if
you
lost
half
of
your
replicas
on
your
on
your
ring,
then
you
might
head
into
a
disaster
there.
So
you
need
to
thread
carefully.
A
A
Okay,
so
some
applications
want
to
not
get
killed,
but
why
are
they
getting
killed?
Well,
they're
getting
killed
because
the
pods
are
running
on
nodes
and
the
nodes
are
getting
killed,
and
here
are
the
systems
at
slack
that
are
killing
the
nodes,
so
I've
separated
those
into
two
categories.
On
the
left
side,
we've
got
the
things
that
we
control
and
on
the
right
side,
the
things
that
we
don't
control.
A
A
We
kill
some
nodes
to
make
sure
that
everything
works
continuously
and
nodes
go
away
when
we
do
that-
and
the
third
thing
is
that
we
have
a
process
that
kills
every
node
that
gets
to
two
weeks
old,
because
that
helps
us
roll
out
patches
and
updates
talking
about
patches
and
updates.
Well,
if
we
need
to
roll
out
an
emergency
security
patch,
sometimes
you
just
don't
have
the
luxury
to
wait
for
two
weeks
to
do
that,
so
we
have
to
go
and
rotate
all
the
nodes
and
that's
something
that
we
can't
do
anything
about.
A
That's
something
that
has
to
happen
and
the
last
one
I've
got
here
is
aws
terminations,
so
aws
is
usually
pretty
polite
and
they
send
you
an
update
that
some
of
your
instances
are
going
to
get
terminated.
So
you
can
act
on
that,
but
you
could
also
lose
a
node
out
of
nowhere.
If
you're
pretty
unlucky.
A
Let's
look
at
the
whole
picture
first
and
then
we'll
dive
into
the
different
components
that
makes
this
solution
so
we're
using
taints
and
tolerations
to
match
pods
to
nodes
and
then
we're
protecting
the
instances
from
getting
killed
so
tension.
Tolerations
are
built-in
kubernetes
feature
that
help
you
match
pods
to
nodes.
A
A
A
The
duration
is
called
lifespan
remaining
and
it
has
the
values
789
all
the
way
up
to
14,
which
means
that
it's
then
happy
to
get
scheduled
on
a
node
that
has
at
least
seven
days
of
life
remaining,
and
so
then
the
nodes
we
apply
tense
on
and
the
taints
represent.
How
many
days
left
there
is
of
life
and
is
based
on
the
uptime
of
the
node?
A
Okay,
so,
let's
zoom
into
how
we
get
that
taint
on
the
nodes.
So
that's
pretty
simple:
we've
got
a
small
go
service
that
we
run
in
each
of
our
clusters
and
it
loops
through
all
the
nodes
and
updates
the
contains
the
nodes
updated
from
with
the
right
value.
So
in
the
top
there
the
node's
got
seven
days
left
because
it's
seven
days
old,
the
one
on
the
right
hand,
side
is
got
zero
days
left
to
leave
because
it's
been
up
for
14
days
and
the
one
at
the
bottom
is
a
new
node.
A
So
in
the
next
two
slides
I'll
talk
into
details
about
asg
scaling
and
chaos,
engineering,
but
the
third
one
in
that
I
talked
about
where
we
just
kill
nodes
after
two
weeks,
we
realized
that
we
could
just
retire
that
because,
thanks
to
the
first
two
ones,
nodes
actually
don't
even
make
it
to
two
weeks
already.
So
that's
something
we
could
retire,
but
then
we're
left
with
the
things
that
we
don't
control,
and
so,
if
we
need
to
rule
out
a
security
patch
or
if
aws
gives
an
instance,
this
is
just
like
too
bad.
A
A
Okay,
so
let's
talk
about
auto
scaling,
put
this
aside
for
a
second
and
and
don't
think
about
kubernetes,
the
classic
way
to
scale
instances
in
an
autoscaling
group
in
aws
is
by
using
auto
scaling
rules,
auto
scaling
rules,
look
at
the
resource
consumptions
on
your
ec2
instances,
so
like
how
much
cpu
how
much
memory
is
being
used
and
once
you
reach
a
given
threshold,
adds
or
remove
nodes,
but
with
kubernetes.
This
is
not.
A
This
doesn't
work
to
do
that,
because
workloads
in
kubernetes,
like
applications,
running
request
some
resources,
so
request
cpu,
request
memory
and
it
doesn't
mean
that
they
are
using
what
they're
requesting.
So
you
can't
get
into
this
unfortunate
situation
where
all
of
the
resources
on
your
cluster
are
being
requested
are
being
claimed,
and
you
can't
do
any
new
pods.
Meanwhile,
the
resource
utilization
on
your
instances
can
still
be
pretty
low.
A
A
A
However,
any
node
that
has
attained
that
is
being
ignored
is
seen
as
unready
by
cluster
autoscaler
and
because,
like
typically
tense,
can
be
used
to
call
the
nodes
or
make
mark
them
really
for
deletions,
or
things
like
that.
So
then,
when
too
many
nodes
are
seen
as
unready
cluster
autoscaler
like
freaks
out
a
bit
and
goes
like
hey
something
looks
dodgy
here,
I'm
not
going
to
touch
anything.
A
A
A
So
this
function
looks
at
the
pods
that
are
running
on
the
given
node
and
unless
there
is
a
pod
that
has
a
lifespan
request
that
hasn't
been
reached.
Yet
then
we're
allowed
to
kill
the
node,
so
I've
got
a
diagram
on
the
side
here
to
illustrate
what
I'm
talking
about.
Let's
say
we
just
have
one
node
and
one
pod,
and
when
the
node
is
two
days
old,
we
scheduled
a
pod
that
has
a
lifespan
request
of
four
days
for
the
first
four
days
here.
A
That
function
will
return
false,
we're
not
allowed
to
kill
the
nodes
because
there's
a
pod
there
that
still
needs
to
live
for
a
few
days,
but
after
four
days.
So
when
the
node
is
six
days
old,
then
the
function
will
return
true
and
we're
able
to
kill
the
nodes
without
breaking
any
contract
on
the
pod.
And
so
we
can
kill
the
node
before
it
gets
to
14
days
old.
A
A
A
So
in
this
example,
there's
a
service
called
grumpy
service
that
has
one
stage
to
be
deployed
in
dev,
asking
for
one
replica
and
for
a
minimum
lifespan
of
seven
days
that
yamlconfig
file
gets
passed
by
our
bedrock.
Cli
and
bedrock.
Cli
creates
a
community's
deployment
on
the
targeted
cluster,
so
it
takes
that
minimum
life
span
fields
in
the
yaml,
bedrock
camel
file
and
turns
that
into
a
label
on
the
nodes
and
the
labels
called
lifespan
request
requested.
A
A
The
admission
webhook
looks
at
the
potspeg
mutates
it
by
injecting
all
the
tolerations
that
we
need
based
on
the
label
that
was
on
it
and
returns.
The
admission
review
response
to
the
cube
server
api
with
that
json
patch,
and
so
a
pod
gets
created
with
the
tolerations
all
the
way
from
7
up
to
14..
So
users
never
have
to
type
all
that
by
hand.
Instead,
they
just
punch
in
the
fields
in
the
initial
yaml
file
and
the
magic
happens.
A
So
I
went
online
and
I
tried
to
find
out
how
like,
what's
the
best
way
today,
to
get
a
an
admission
webhook
of
the
ground
and
deployed
in
communities,
and
I
found
cubebuilder,
which
I'm
sure
you're
familiar
with.
But
let's
this
is
like
the
first
line
in
their
readme
cube
builder,
is
a
framework
for
building
kubernetes
apis
using
custom
resource
definitions.
A
A
A
So,
instead
we
went
and
wrote
a
simple
admission
web
hook
in
golling,
and
that
worked
pretty
well
for
us,
so
we
went
and
open
sourced
it.
So
you
can
find
it
on
the
slide.
Slack
hq
organization
on
github
yeah
and
like
look
even
if
it's
not
something
that
you
need
or
want
in
production.
This
report
comes
with
a
make
file
that
uses
docker
and
kubernetes
in
docker,
also
known
as
kind,
and
so
you
can,
with
a
couple
commands
just
get
a
community's
clusters
running
on
your
laptop
deployed.
A
Your
mission,
webhook
deploy
pods,
look
at
the
logs
I'll
change,
the
code
change
the
validations
mutation,
redeploy
the
webhook
and
see
what's
going
on.
So
what
I
mean
is
like
I'm
hoping
that
this
might
have
some
educational
value.
If
you
are
not
super
familiar
with
how
admission
web
hooks
work
in
kubernetes.
Well,
maybe
that's
something
that
you
want
to
play
with.
A
A
A
A
And
in
the
same
way,
like
the
the
validations
and
mutations
that
this
old
admission,
webhook
does
are
stateless
too,
so
it
doesn't
do
crds
either
it
doesn't
do
any
controlling.
So
it's
the
same
requirements
as
the
simple
web
hook.
I
presented
to
you,
so
that
was
one
of
the
reasons
why
we
went
and
created
the
simple
admission
web
book
because
we
thought
oh,
this
is
great.
Then
we
can
just
migrate
the
features
from
the
old
one
to
the
new
one
and
retire
this
old
piece
of
work.
A
But,
as
you
might
know,
retiring
old
systems
is
hard
to
prioritize,
and
so,
while
some
of
the
features
have
been
migrated
to
the
new
one,
so,
for
example,
we
validate
that
our
docker
images
are
coming
from
our
trusted
registry.
We
still
have
a
lot.
Everything
that
is
sidecar
injection
is
still
on
the
old
web
hook.
So
we
inject
side
cars
to
do
pod,
to
port
encryption,
to
deal
with
secrets
and
service
discovery,
and
so
that's
still
living
in
the
old
one.
A
Okay-
and
so
I
didn't
build
all
this
by
myself-
it
was
three
of
us
working
on
the
feature.
It
was
me
and
sean
who
I
would
like
to
thank
for
the
things
he
taught
me
about
software
engineering
and
trisha,
who
has
been
a
great
source
of
inspiration,
and
I
also
want
to
thank
javier
for
his
ongoing
support
and
for
how
excited
he
was
to
have
me
be
here
today
doing
this
talk
and
those
are
the
images
I
used
and
thank
you.
B
A
C
Thank
you
and
why
or
which
is
the
reasoning
behind
having
two
weeks
life
span
for
all
the
notes
and
probably
related
another
question,
and
can
you
guarantee
that,
within
that
two
weeks,
life
span
and
because
of
a
security
rollout,
every
node
will
have
a
14
days
day
span
so
and
if
eight
days
and
goes
let's
say
eight
days
passes
after
that?
If
someone
asked
for
seven
days
span
and
you
will,
then
you
wouldn't
be
able
to
to
schedule
because
there
is
no
node
available.
So
how
do
you
deal
with
that.
A
Yeah
right
so
like
the
the
two
x's
is
historical
to
start
with,
so
this
has
been
like
the
currency
as
to
how
often
we
rotate
nodes
and
how
often
we
roll
out
patches-
and
this
is
why
the
maximum
we
set
is
seven
days
because
then
it
means
we
have
seven
days
to
kill
the
nodes
and
get
everything
done.
Because
if
we
let
people
set
two
weeks,
then
at
the
end,
then
we
have
to
rotate
all
the
nodes
at
once
and
that's
not
a
position
we
want
to
be
in.
D
Thanks
thanks
for
the
presentation,
just
a
question
services
do:
do
they
know
internally
that
they
have
this
many
days
to
live.
For
example,
if
I
start
my
service
at
slack,
do
I
know
that
my
service
has
seven
days
and
then
do
your
engineers
do
something
about
it
like
the
exit,
gracefully
at
seven
days
and
then
they're,
just
not
consuming
resources
or,
and
then
you
end
up
with
empty
nodes.
Or
how
do
you
deal
with
that.
A
A
E
A
Not
seriously,
that
could
be
an
option.
We
should
talk
about
after
that
and
that'd
be
interesting
to
know
like
how
you
might
go
about
and
and
using
a
custom
resource
definition
or
a
custom
scheduler.
To
do
that,
we
had
a
bit
of
time
pressure
to
release
this
and
we
tried
to
find
like
sort
of
like
the
easiest
way
to
get
that
feature
delivered
and
that's
what
we
came
up
with,
but
definitely
a
custom
scheduler
would
work.
B
F
A
Like
so
we
well
at
first,
we
had
debug
logging
enabled
on
the
admission
web
hook,
and
so
we
were
getting
a
lot
of
like
energetical
data
from
the
logs
and
we
saw
how
many
applications
used
the
pods
and
what
how
all
the
pods
were
and
when
they
were
getting
killed.
And
from
that
like
little
investigation
like
that
window
of
data
we
had.