►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
10 Weird Ways to Blow Up Your Kubernetes - Melanie Cebula & Bruce Sherrod, Airbnb
It’s a brand new world in infrastructure with the advent of microservices, containerization, Kubernetes, and service mesh. And all is well. Or is it? Find out how easy it is to break container runtimes, abuse your service mesh, and take all of your production services down-- the results will surprise you! In the last year Airbnb scaled up to over 700 services in Kubernetes, running on all types of workloads across 1000s of nodes and dozens of clusters. We’ve learned a lot along the way and have some of our favorite stories to share-- from weird bugs, to hacky workarounds, to serious downtime. Favorites include: - “Just what is the autoscaler doing”? - “Knock knock, It’s Kube-DNS” - “Whose PID is it anyway”? and more!
https://sched.co/UaVY
A
And
yeah,
so
we're
gonna
talk
about
kubernetes
at
Airbnb,
very
briefly
in
one
slide
started
from
the
bottom
and
now
we're
here,
but
it's
not
all
successes
so
like
we're
here
to
talk
about
the
spectacular
pitfalls,
mistakes
and
challenges
we've
run
into
in
the
past
few
years,
Airbnb,
and
by
the
way,
this
is
10
weird
ways
to
blow
up
your
kubernetes,
not
10,
ways
to
fix
your
kubernetes.
So
please
no
judgement.
These
are
not
all
perfect
solutions.
Some
of
these
are
like
ugly
hacks
and
other
things
or
just
stuff
that
didn't
work
so
enjoy.
A
B
Up
zombie
jobs,
so
community's
jobs
and
cron
jobs
are
really
great.
We
and
we
use
them
a
lot
at
Airbnb
they're,
a
nice
distributed
highly
available
cron
job
solution.
We
use
them
for
a
lot
of
things,
but
we
have
this
problem,
and
many
of
you
have
probably
seen
this:
when
does
a
cron
job
end
or
community's
job
end?
You
have
a
typical
job
that
might
have
some
side
cars
that
do
support
services
like
a
network,
proxy
or
logging
agent.
Those
run
forever
and
kubernetes
jobs
don't
end
until
all
of
the
containers
exit.
B
So
this
is
really
a
struggle.
Kubernetes
provides
you
this
feature
to
try
to
deal
with
this.
You
can
set
active
deadline
seconds
in
order
to
get
that
to
work
properly.
You
need
to
also
set
the
concurrency
policy
and
the
restart
policy,
but
suppose
you
do
this,
you
have
you
still
have
a
few
problems.
B
First
is
it
will
appear
as
if
your
cron
jobs
are
always
running
because
kubernetes
hasn't
killed
it
yet
until
the
act
of
deadlines
seconds
expires,
so
you
don't
really
know
how
long
they're
actually
running
and
secondly,
when
kubernetes
finally
kills
a
job
based
on
it,
reaching
it
reaching
acti
active
deadlines
seconds.
It
marks
the
job
as
failed.
So
if
you
go
to
say,
cube,
CTL
get
and
take
a
look
at.
B
What's
going
on
your
cluster
and
looking
at
one
of
your
jobs,
you'll
see
the
current
one
always
running
and
all
of
the
previous
ones
always
failed.
You
don't
really
have
good
insight
into
what's
going
on
with
your
cron
jobs.
So
there's
a
simple
solution
to
this
right.
Just
use
touch
files
have
your
main
container
file
to
indicate
that
it's
finished
and
have
the
sidecars
exit
when
that
file
appears
so
I
suggested.
This
gave
it
to
Melanie
as
a
project
to
work
on
on.
A
It
so
I'm
a
lazy
engineer,
and
the
first
thing
I
do
is
Google
to
see
if
anyone
is
running
the
same
problem
so
that
I
can
steal
their
solution.
So
I
found
a
gap
issue,
better
support
for
sidecar
caters
in
batch
jobs,
same
thing,
jobs,
not
exiting
not
really
a
great
solution
here,
great
scroll
down
the
github
issue
looks
like
Jamie
Milliken
stripe
is
the
winner
pasting
code
directly
into
the
issue,
seems
like
I
can
just
introduce
some
of
that
and
see
if
it
works.
A
So
the
way
this
kind
of
the
way
the
solutions
outlined
is
that
you
have
a
main
container
in
a
sidecar
container.
The
main
container
rights
touch
file
when
it
exits
and
to
the
sidecar
container
runs
a
wrapper
script
that
looks
for
that
file
to
be
written
and
then
Alex
exits
itself
when
it
sees
that
it
has
been
written.
So
I
actually
just
took
the
bash
script
from
the
Sacramento
post
and
put
it
into
the
basket
to
see
if
it
worked
hmmm.
A
Well,
it
didn't
quite
work
which
I
kind
of
expected
so
take
them
or
we
needed
to
support
dumb
init.
Have
it
exit
properly
with
the
correct
exit
code?
Well
that
didn't
quite
work
either.
So
there
was
a
few
other
PRS
needed
to
try
to
tweak
this
behavior
and
actually
got
into
like
the
space
of
complicated
signal
handling.
Each
of
these
changes
were
actually
quite
scary
because
we're
not
running
production
batch
jobs.
This
way,
and
so
at
least
one
of
these
cause
the
production
incident
by
terminating
the
wrong
PID
yeah.
A
So
again,
I
like
look
this
up,
because
that's
what
I
do
and
OH
sidecar
containers
that
seems
legit,
there's
like
a
ton
of
people
that
want
to
support
this
and
it's
going
to
be
in
communities,
but
it's
gonna
be
in
117
and
it's
an
API
change.
I
can't
really
back
port
it
safely
to
previous
versions,
so
I
guess:
I'm
gonna,
try
to
for
kubernetes
and
I
asked
the
internet
what
they
thought.
A
I
got
the
most
horrified
response
possible,
but
I
was
like
okay,
okay,
so
I'm,
not
14,
kubernetes
I'm,
just
patching
it
yeah,
very
different
and
actually
I
didn't
have
to
set
up
this
whole
thing
where
we
pull
like
the
latest
coming
stable
version
that
we're
using
and
then
we
run
it
as
a
CI
CD
pipeline
that
like
applies
github
patches
to
this
type
of
release,
so
not
forking.
Okay,
it's
like
a
very
sophisticated
process,
but
you
will
find
that
the
kate's
codebase
evolves
quickly.
A
So
of
course,
I
looked
this
up.
Has
anyone
else
had
this
problem
and
it
turns
out?
Lyft
has
a
problem,
so
they
actually
had
their
patch
just
hanging
around
in
their
open-source
Cabrini's
fork
and
so
yeah.
Let's
see
how
their
patch
works.
Basically,
you
have
a
pod
with
a
bunch
of
containers.
You
have
your
main
service
container,
a
bunch
of
sidecar
helper
containers
and
the
patch
is
in
cubelet.
A
So
if
you
annotate
the
containers
as
standard
or
sidecar,
then
the
logic
in
cubelet
has
been
modified
to
enforce
that
sidecar
containers
start
before
the
standard
container
and
shutdown
after
them,
which
have
exactly
the
behavior
we
want
here,
and
it
supports
many
other
use
cases.
Some
one
other
thing
we
use
this
for
was
just
service
discovery
containers,
so
it
works
really
well
there
actually,
and
so
the
takeaway
here
is
to
just
solve
your
palm
at
the
appropriate,
appropriate
abstraction
level,
and
that
may
even
be
patching
it
behavior
into
communities
itself.
B
Unfortunately,
restarting
H
a
proxy
is
memory
intensive.
The
way
that
works
is
agent,
proxy
Forks,
it
starts
a
new
copy
of
itself.
Taking
up
all
intent
of
memory
is
first
copy.
The
original
copy
will
finish
any
outstanding
connections
that
are
running
and
then
terminate.
So
your
memory
usage
can
double
when
HR
proxy
is
running
or
if
your
connections
are
long-lived
and
your
service
mesh
is
changing
frequently,
you
might
have
a
lot
of
a
cheap
proxies
running
and
run
into
you
and
out
of
memory
issue.
B
B
Kubernetes
has
this
great
feature
where
it
can
deploy
really
really
fast
and
developers
don't
like
waiting,
so
they
cranked
up
max
surge
for
the
deployment
strategy
and
you
can
actually
deploy
your
whole
service
in
less
than
30
seconds
and
the
service
mesh
can't
keep
up
so
each
time
a
service
would
deploy.
We'd
see
the
spike
of
errors
calling
into
that
service,
and
so
you
see
this
error
spike
with
every
deploy.
Our
fix
for
that.
B
So
far,
is
this
kind
of
hacky
solution,
which
is
basically
to
just
tell
kubernetes
to
slow
down
the
termination
of
pods
on
deploy.
You
add
a
pre
stop
hook
to
sleep
and
wait
long
enough
for
your
service
mesh
to
catch
up
and
figure
out
what
happened
kubernetes.
Will
this
will
put
the
pot
of
determinating
state?
B
The
kubernetes
will
actually
delete
it
in
that
state
by
default
after
30
seconds,
so
you
also
need
to
set
termination
grace
period
seconds
and
once
you
have
those
two
settings
and
you
kind
of
tweak
it
a
little
bit,
you
can
get
the
pods
to
survive
long
enough
to
keep
your
to
stay
around
until
your
service
mesh
catches
up
with
what
happened.
So
our
take
away.
Kubernetes
deploys,
can
cycle
pods,
superfast,
weather
the
rest
of
your
infrastructure
and
keep
up
or
not.
A
Next
up
monster
daemon
sets.
So
why
do
you
want
a
daemon
set
anyway?
Suppose
you
have
a
pod
that
needs
to
download
a
large
amount
of
data
every
time
it
starts,
it'd
be
convenient
if
you
could
use
a
daemon
set
to
just
make
sure
the
data.
Is
there
every
time
your
products
schedule
on
to
that
node?
For
example,
we
have
gigs
of
translation
data
or
search
index
data
and
many
pods
that
need
that
data,
so
Damien
said
sure
I'll
solve
that
for
you.
A
The
reality
is
that
the
daemon
starts
kind
of
flaky,
so
he
kind
of
just
steps
outside
when
he's
deploying
and
the
pods
like,
but
wait,
I
kind
of
am
still
running.
I
still
need
you
to
be
around
also
oops
I
died,
the
daemon
psycho
is
unhealthy
or
it
just
dies
and
doesn't
come
back
that
pod
again,
it's
still
running
and
expecting
the
daemon
set
to
also
be
running.
A
So
one
thing
we
tried:
okay.
Well,
maybe
a
daemon
set.
Isn't
the
right
solution
here.
What
if
we
try
a
deployment
and
use?
This
neat
feature
called
pond
affinity
to
try
to
schedule
the
service
pods
and
the
translation
data
pods
on
the
same
node,
but
there's
some
doctors
with
pod
affinity,
for
example.
If
you
want
like
preferred
during
execution
or
required
during
execution,
you're
making
a
trade-off
of
well
they're,
not
always
scheduled
together
or
they
are
always
scheduled
together,
and
if
your
cluster
can't
keep
up
you're
like
full
of
scheduler
errors.
A
Now,
so
we
didn't
only
like
that.
Okay,
so
we
thought
of
another
idea,
which
is:
let's
use
the
daemon
set
and
then
there's
another
kubernetes
features
called
paints
and
Toleration.
So
tanks
is
a
way
to
say
you
know
don't
schedule
on
this
node
during
these
conditions.
So
maybe
we
can
paint
on
pod
readiness.
But
one
thing
we
were
thinking
about
was
you
know,
supporting
a
very
complex
cluster.
A
A
Okay,
and
in
fact
maybe
we
should
use
a
mission
controller
to
introduce
some
friction
around
creating
daemon
sets,
because
it
seems
pretty
easy
to
just
take
everything
down
and
of
course,
there
are
use
cases
for
daemon
sets
another
special
node
behavior,
so
we'll
just
create
dedicated
smaller
special
clusters
for
services
that
need
that
special
behavior
and
actually
work
pretty
well
for
us.
So
the
main
thing
right
here
was
just
that:
David
sets.
You
know
they
can
actually
just
take
down
your
cluster
in
a
way,
that's
harder
for
other
workloads.
B
B
We
came
up
with
a
simple,
relatively
simple
idea:
we'll
run
a
write,
a
program
we
call
it
ECR
cleaner.
It
works
basically
like
this.
It
runs
once
a
day
as
a
cron
job.
It
finds
all
the
images
that
are
in
use
by
calling
something
like
cube:
CT,
I'll
get
deployments
and
looking
for
all
the
image
IDs
in
use
and
then
for
each
ECR
repository,
delete
and
delete
images
down
to
some
threshold,
but
but
skipping
the
ones
that
are
in
use,
so
that
worked
pretty
well
for
a
while
till
a
couple
things
went
wrong.
B
Almost
every
line
of
this
went
wrong.
First,
one
was
CI
failures
due
to
some
sort
of
docker
image
error.
We
dig
into
this
and
discover
ultimately,
the
root
cause
is
that
first
part
of
this
ECR
cleaner.
It
only
runs
once
a
day
and
amazingly,
we
were
generating
more
than
10,000
images
a
day
in
CI.
Now
that
I
say
it
out
loud,
it
doesn't
actually
sound
like
that
much,
but
it
was
pretty
surprising
when
we
crossed
that
threshold.
B
Romney's
er,
cleaner,
even
more
frequently
and
eventually
we're
gonna,
probably
generate
more
than
10,000
images
per
hour
and
then
we're
gonna
have
to
come
up
with
something
else,
but
this
is
holding
for
right
now.
The
next
problem
we
ran
into
manifested
like
this.
This
is
how
these
things
manifest
they're
kind
of
mysterious
and
takes
some
tracking
down.
B
You
rotate
a
kubernetes,
Minya,
node
and
suddenly
the
service
is
down
and
the
reason
why
is
because
the
last
remaining
pods
of
that
service
that
had
workable
images
were
on
that
one
node
and
the
images
are
gone
from
ECR.
Why
did
that
happen?
Since
we
have
this
cleaner
well
find?
All
images
in
use
was
only
checking
the
cube
deployments
in
the
cluster
where
the
ECR
cleaner
was
running,
but
now
we're
a
multi.
We
have
a
multi
cluster
solution.
B
In
fact
we
have
over
30
clusters
and
it
wasn't
checking
all
the
other
clusters
so
that
so
we
need
to
experience
and
use
to
actually
find
all
images
in
all
clusters
that
are
in
use,
but
that
presents
a
problem
to
where
do
you
run
the
ECR
cleaner?
Because
you
want
to
have
isolation
between
your
various
kubernetes
clusters,
VCR
cleaner
really
needs
to
be
a
global
singleton.
It
needs
to
know
about
all
images
in
use
in
all
clusters.
So
where
does
it
run?
We
solve
this
by
adding
a
new
cluster,
so
we
create
a
cluster.
B
We
call
the
management
cluster,
which
has
permission
to
access
the
API
servers
of
every
other
cluster,
and
then
we
have
a
more
strict
review
process
for
services
that
run
in
the
management
cluster,
so
the
takeaway.
Why
can't
I
hold
all
these
docker
images,
make
sure
to
keep
track
of
your
docker
images?.
A
A
A
So
let's
split
the
logic
and
do
an
initial
update
in
an
init
container
and
then
have
the
runtime
configuration
pole
container
silly
for
more
updates,
oh,
but
to
pull
in
the
latest
dynamic
configuration.
We
actually
need
to
communicate
with
runtime
config
service,
which
requires
service
discovery
which
definitely
doesn't
exist
during
the
initialization
phase.
And
so
therefore
you
know
the
first
approach
of
this
is:
let's
have
the
init
container
set
up
its
own
service
discovery?
That
seems
like
great
idea:
yeah,
it's
not
so
it
was
like
yeah.
A
So
the
way
that
air
worked
is
it
started
up
the
network
proxy
and
then
ran
its
logic
to
pull
the
data
and
then
like
shut
down
the
network
proxy.
This
was
complex,
pretty
brittle,
slow
and
then
obviously
had
a
significant
footprint
on
our
service
discovery
info,
so
not
ideal,
but
we
have
sidecar
ordering
now.
So
let's
just
slow
this
at
that
problem.
Surely
this
will
solve
it?
Let's
just
make
runtime
configuration
and
network
proxy
sidecars,
but
then
you
know
we
still
aren't
guaranteeing
that
the
network
proxy
is
up
before
the
runtime
config.
A
Is
it
so
it's
possible
that
the
runtime
config
in
itself
can
still
not
connect
to
the
network
proxy
if
it's
ready?
First,
okay?
So
what
do
we
do?
We
can
add
logic
to
the
runtime
configure
like
our
to
just
wait
for
the
network
proxy
site
card
to
be
ready
with
the
rapper
script,
so
some
kind
of
logic.
It
says
you
know
continuously
a
pole
for
runtime
config
service.
A
If
it
can
now
successfully
connect
to
runtime
config
service,
then
let's
get
that
initial
update
and
then
begin
polling,
so
takeaway
you
might
need
ordering
between
sidecars
to
and
you
might
say,
hey
Melanie.
You
didn't
solve
that
directly
communities
you
used
in
other
batch
scripts
to
which
I
challenge
you
to
please
write
a
topological
sort
patch
for
sidecar
ordering.
Thank
you.
Please.
B
Let
us
know
when
you've
got
and
you've
got
that
next
up,
where's
my
custom
resource,
so
custom
resources
are
great
and
at
Airbnb
we
use
them
a
lot,
especially
to
keep
track
of
resources
external
to
the
communities
cluster,
but
that
are
associated
with
a
service
such
as
its
storage,
its
dashboard,
its
Alerts,
its
AWS
resources
like
I,
am
roles
and
so
forth.
One
challenge
with
using
custom
resources
is
knowing.
When
the
deploy
has
completed.
B
We
want
to
be
able
to
determine
whether
a
deploy
is
completed
or
not,
and
whether
the
custom
resource
changes
were
applied
successfully
or
not.
So
we
came
up
with
this
kind
of
simple
idea:
we'll
make
use
of
the
status
field
in
the
resource
right.
Simple
keeps
ETL,
cube,
CTL
apply,
we'll
set
the
status
depending
the
the
controller
will
wake
up
when
it
receives
the
new
custom
resource,
it
will
apply
the
changes
and
then
it'll
change
the
status
to
either
ready
or
error,
and
this
is
really
convenient.
B
It's
easy
for
the
custom
for
the
controller
to
know
when
it
needs
to
run
it
just
checks
to
see
if
the
status
is
pending
and
it's
easy
for
an
operator
manually
to
check
and
see
what
happened.
You
can
just
do,
keep
CTL
get
and
the
status
is
at
the
bottom
and
is
kind
of
human
readable.
So
that's
great,
except
that
it
doesn't
work
or
it
didn't
work.
It
worked
up
until
kubernetes
version
112
after
1.12
only
controllers
can
modify
the
status
field
of
resource
objects.
B
B
It
was
a
kind
of
it
was
an
insidious
nasty
outage
where
what
happened
is
that
new
resources
would
apply
and
work
fine
right,
but
any
changes
to
existing
custom
resources
silently
succeed,
because
they
would
still
look
like
they
were
in
ready
state,
but
the
controllers
would
never
run
on
them
because
they
never
got
reset
back
to
pending
state
by
cube,
see
tail
apply,
which,
incidentally,
succeeds
but
doesn't
make
the
change.
So
this
was
a
pretty
gnarly
outage
caused
by
this
change,
in
that
we
determined
as
part
of
the
kubernetes
upgrade
when
they
upgraded
past
12.
B
So
when
you
update
a
resource,
kubernetes
will
change
that
generation
from
say
one
to
two:
when
you
do
your
cube,
CTL
apply
the
status
still
stays
the
same.
You
can't
change
that
part,
and
this
at
least
gives
you
some
facility
to
figure
out
what's
going
on
with
your
custom
resources,
but
it's
it's
not
enough
right.
How
does
your
controller
know
whether
it's
operated
on
generation
2
or
not?
So
here's
our
Airbnb
solution.
B
We
add
in
the
status
field
and
observe
generation,
which
is
the
last
generation
that
the
controller
has
run
on
now.
The
controller
updates
when
it
when
it
sees
a
new
version
of
the
resource
it'll.
If
the
generation
doesn't
match
the
observe
generation,
it'll
reapply
the
changes
and
update
the
status
field-
and
this
works
for
now,
which
is
good,
but
the
user
interface
for
manual
inspection
is
still
quite
a
bit
worse.
When
you
do
keep
CTL
get
well
on
the
slide.
B
I've
put
these
two
lines
pretty
much
right
next
to
each
other,
but
there
might
be
hundreds
of
lines
of
spec
in
between
and
you
kinda
have
to
manually
eyeball
to
see
that
those
things
are
are
updated,
so
not
as
good
as
it
used
to
be,
but
still
works.
That's
our
that's
our
stopgap,
so
our
takeaway,
knowing
when
it
deploy,
is
complete
and
if
it
succeeded
or
not,
is
tricky.
A
Cool
next
up,
I
can't
believe
I
have
all
the
notes,
resources.
You
probably
see
where
this
one
is
going.
So
before
kubernetes
there
was
a
simple
time:
simple
place:
I
got
one
service
on
one
instance,
instance
is
pretty
big:
it's
got
72
gigs
and
the
JVM
allocates
made
72
gigs,
so
that
works
out
pretty
well
in
kubernetes.
You
have
multiple
co-located
services
running
in
pods
and
the
JVM
gives
you
something
to
gigs
and
use
72
gigs
and
use
72,
gigs
and
now
suddenly
you're
out
of
memory,
and
it's
the
same
thing
for
CPU.
A
So
all
kinds
of
problems
and
the
problem
is
that
all
the
versions
of
the
JDK
are
not
aware
of
cgroups
and
so
okay.
Well,
I
guess
we're
running
a
really
old
JDK.
We
should
probably
upgrade
it
it's
solved
in
newer
versions,
and
it
knows
how
to
allocate
resources
based
on
the
container
or
not
the
node.
A
So
we
kind
of
like
did
investigation
just
to
make
sure
this
was
the
case.
So
here's
a
real
graph
of
a
service
tracking,
it's
p95
latency
after
the
upgrade
the
latency
went
down
and
for
its
worth
we
tried
a
few
other
flags
and
things
that
didn't
really
seem
to
help
us,
so
that
seemed
to
work
pretty
well.
A
So
now
a
GVM
gives
your
pod
the
correct
amount
of
CPU,
but
this
service
has
a
concurrency
bug
that
gets
worse.
The
less
CPU
that
was
allocated
to
you,
based
on
the
third
Pole,
oh
okay,
so
basically
downgrade
to
the
correct
amount
of
CPUs,
actually
expose
an
application
race
condition
which
led
to
an
outage
so
yeah
and
here's
the
here's
the
real
evidence
of
the
outage.
Basically,
it
was
like
really
slow
requests,
as
I
started
to
hit
the
race
condition
more
and
therefore,
when
upgrading
you
know
the
JDK
or
any
other
language
framework.
A
You
know
you
have
to
just
pay
attention
to
these
sorts
of
problems.
It's
actually
another
major
upgrade
that
you're
taking
on
so
cool
application
owner.
Please
use
canary
and
test
it
in
24
hours
before
you
know
rolling
out
to
the
rest
of
production,
and
so
this
is
not
specific
to
Java
services.
So
you're,
like
yeah,
Melanie
yeah,
this
is
like
cube,
Con
2016.
Everyone
already
knows
about
that,
but
I
guess
the
reason
I
kept
it
in
here
is
kind
of
it.
A
Just
it
just
keeps
happening
so
like
with
other
Lego,
train
works
and
side,
cars
and
stuff,
and
so
I
think
it's
just
something
that
you
should
just
always
having
your
back
pocket
of.
Maybe
maybe
that's
what's
happening
here
so
like
another
example
is
when
we
first
started
experimenting
with
envoy.
We
noticed
that
it's
that
concurrency
didn't
have
CPUs
on
the
underlying
host
by
default.
That
sounds
familiar,
like
probably
that's
gonna
call
its
contention
on
the
host
so
like
we
had
to.
A
We
had
to
like
set
the
concurrency
to
be
way
lower,
so
the
takeaway
there
it's
just
it
like
be
aware
of
language
framework
since
I
cars
they're
not
aware
of
these
container
abstractions,
and
you
know
that
you
might
be
signing
yourself
up
for
more
upgrades.
You
know
when
you,
when
you
upgrade
to
containers
we're
nice.
B
Alright
next
autoscale
akka
limbs
so
auto
scaling
is
great,
probably
hearing
me
using
this
a
lot.
I
love
all
these
Cuban
ease
features
till
they
don't
work
so
auto
scaling
for
those
of
you
that
aren't
familiar
the
way
auto
scaling
works
is
it
creates
a
new
resource
called
a
horizontal
pot
autoscaler
which
monitors
your
deployment
and
whenever
the
average
CPU
of
your
deployment
deviates
from
a
set
value,
it'll,
add
or
remove
pods,
so
keep
your
utilization
approximately
flat.
Now
suppose
you
have
a
service
like
this.
B
You
have
a
service
that
consumes
a
lot
of
CPU
on
start
say
for
initialization.
In
our
case,
we
have
some
services
that
go
through
a
startup
phase,
that
we
call
JVM
warm-up
where
they
replay
synthetic
traffic
in
order
to
populate
their
in-memory,
cache
to
establish
connections
and
their
connection,
pools
to
fill
up,
thread,
pools,
etc.
This
process
takes
a
few
minutes
and
uses
essentially
a
hundred
percent
CPU,
while
it's
happening
so
what
happens
with
the
autoscaler,
so
here's
an
example
of
a
deploy
of
a
service.
B
Just
what
is
the
autoscaler
doing
this
service
typically
uses
about
30
pods?
It's
autoscaler
ranges
from
20
to
200
and
you
deploy
which
surges
and
adds
a
few
extra
pods.
They
come
in
at
a
hundred
percent
CPU
and
raise
the
CPU
average,
which
causes
the
autoscaler
to
add
some
pods,
which
raises
the
cpu
average,
which
causes
the
autoscaler
to
add
some
more
pods
and,
in
fact,
will
spike
the
number
of
pods
almost
all
the
way
to
the
maximum
allowed
in
this
auto
skill
group.
In
this
in
this
autoscaler.
B
It's
the
trajectory
is
that
the
autoscaler
is
adding
pods
so
fast
that
that
all
the
ones
that
are
coming
in
are
still
in
their
warm-up,
bays
and
still
burning,
more
cpu,
so
a
service
that
typically
runs
in
30
pods
will
spike
to
190
pods
on
every
deploy.
That's
not
great!
How
do
you
fix
this?
It's
not
obvious
like!
Surely
there
must
be
a
setting
from
it
for
this
I
hear
you
cry
well,
there
used
to
be
a
setting
for
this,
but
it
was
removed
with
this
cryptic
comment
in
the
documentation
starting
from
1.12.
B
A
new
algorithmic
update
removes
the
need
for
upscale
delay,
so
here's
our
embarrassing
hack
how
we
work
around
this.
Incidentally,
there
is
a
setting
called
horizontal
pod,
autoscaler
initial
readiness
delay,
which
sure
seems
like
it's
meant
to
address
this
problem,
but
it
did
not
work
for
us.
It
did
not
help
this
problem
at
all
and
we
tried
even
setting
it
as
long
as
high
as
15
minutes.
So
our
really
dumb
hack
is
to
set
the
pod
autoscaler
sync
period.
B
This
essentially
means
we
run
the
autoscaler
only
once
every
5
minutes,
and
you
really
can't
run
it
any
less
frequently
than
that
or
you
really
risk
the
autoscaler
not
reacting
to
real
traffic
spikes,
and
even
with
that
setting
it
doesn't
fix
the
problem.
It
only
mitigates
the
problem,
so
here's
the
same
service
deploying
now
typically
runs
at
30,
pods,
still
spikes
to
around
85
or
90
pods,
due
to
runaway
auto
scaling
on
every
deploy.
A
Okay,
next
up,
hey,
my
scheduled
operation,
took
down
all
services,
so
here's
an
excerpt
from
an
actual
operation
fYI.
We
are
running
an
operation
to
upgrade
stats
the
agent
in
Cabrini's.
In
a
few
minutes
there
may
be
a
few
minutes,
we're
symmetric,
SAR
not
available
to
services
and
then
a
few
minutes
later.
A
I
think
all
of
my
kubernetes
sources
are
down
right
now
and
they
also
have
metrics
but
mostly
they're
down,
and
so
to
explain
this
one
just
a
brief
overview
of
kubernetes
health
checks,
so
health
tech,
synchronize,
there's,
readiness,
probes
and
liveness
probes,
vinius
probes,
don't
send
traffic
to
me.
If
it
fails,
life
is
probes,
replace
me
if
it
fails,
and
so
you
know
when
you
think
about
a
service
container
for
us
at
least
most
of
the
services
just
said
both
of
these
the
flash
health.
A
So
if
it
responds
with
flash
health,
it's
probably
healthy
and
you
can
set,
you
know
more
complex
ones
if
you
want,
but
in
Cabana
health
checking
is
per
container,
so
you'd
be
there
to
think
about
all
of
these
containers
and
how
they
contribute
to
your
health
check.
And
so
what
should
their
health
text
be
and
like?
How
should
they
affect?
You
know
whether
your
whole
pod
is
receiving
traffic
or
not?
A
So
you
know
a
while
ago,
before
this
incident,
we
had
already
decided
that
we
didn't
want
stats,
see
agent
failing
to
take
down
our
service,
so
our
initial
try
was
okay,
just
don't
set
probes
for
non-critical
containers.
Containers
like
logging
tracing
metrics
should
not
affect
service
health,
so
pretty
easy
right.
Just
some
set
the
probes.
A
Well,
we
didn't
know
that
a
crashing
container
fails
its
readiness
check,
regardless
of
whether
you
delete
all
the
probes
or
not,
it
will
still
fail
the
readiness
check,
and
so
with
sexy
agent
crashed.
It
failed
its
readiness
check
and
at
the
time,
our
service
much
logic
basically
had
a
loop
like
this.
So
it's
pause
pot
ready
true
for
each
container
size
in
the
pod.
If
it's
not
ready,
the
pod
is
not
ready
and
then,
at
the
end
of
this
whole
loop
with
the
pods
ready
publisher,
that's
ready
so
that
it
can
receive
traffic.
A
So
if
any
one
of
the
containers
or
ready
the
whole
pot
isn't
ready
just
receive
traffic,
so
in
this
case
our
crashing
starts
the
agent
container,
which
became
unready
caused
all
of
the
pods
in
our
cluster,
to
become
on
ready,
not
great.
So
the
next
solution
was
basically
well,
let's
just
like
hack
this
logic
into
our
service
mesh.
So
we
mark
containers
that
we
don't
want
to
be
considered
for
pod
readiness
as
non
critical
and
here's
the
example
again.
A
B
Next
scheduling
is
easy
and
fun,
so
this
is
an
example
of
a
deploy
of
a
service
that
runs
in
three
availability
zones
turns
out.
The
kubernetes
scheduler
is
not
very
good
at
a-z
balancing.
In
fact
it's
really
bad
at
agency
balancing
it
appears-
and
this
is
speculation.
I
haven't
really
looked
too
closely
the
code,
but
it
appears
that
in
each
sort
of
round
of
scheduling
it
will
assign
all
of
the
pods
it's
trying
to
create
to
a
single
AZ.
B
So
if
you
have
a
fairly
large
max
urge
you
Serge
a
set
of
pods
at
once,
they
seem
to
all
land
in
one
AZ
at
once,
so
deploys
will
tend
to
knock
services
completely
out
of
balance,
and
we
do
a
lot
of
AZ
aware
routing.
So
in
our
services
are
out
of
balance
like
this,
it
can
cause.
You
know
significant
problems
in
our
infrastructure,
so
scheduler
doesn't
do
what
we
want
so
we'll
write
it
can
how
we
solve.
This
will
write
a
controller,
so
we
implement
what
we
call
our
deployment
pruner
controller.
B
The
way
this
controller
works
in
our
first
attempt
to
fix
this
we're
ready.
We
have
this
controller,
it
checks
for
deployments
that
are
out
of
AZ
balance,
and
then
it
tries
to
cautiously
delete
pods
until
the
deployment
gets
back
into
balance.
Now
this
has
a
couple
of
problems
still
right.
First
of
all,
it's
fighting
with
the
scheduler,
so
it's
deleting
pods
the
schedulers
putting
the
pods
back,
but
the
schedule
are
still
not
putting
them
back
in
AZ
balance.
B
In
this
example,
this
went
from
out
of
balance
and
one
AZ
to
out
of
balance
and
a
different
AZ,
as
the
scheduler
kind
of
fought
with
the
pruner
and
the
whole
process
is
pretty
slow
in
this
example
of
a
deploy
took
over
four
hours
for
the
service
to
get
back
into
AZ
balance
within
one
cluster.
So
this
is
kind
of
this
was
a
stopgap
solution,
but
really
not
a
very
good
one.
B
So,
as
Melanie
pointed
out
earlier,
some
of
the
time
you
just
need
to
fix
this
in
kubernetes,
so
we've
created
a
couple
of
patches
to
fix
this
one's
been
merged,
one
is
still
open.
So
if
some
of
you
wanna
someone
here
can
help
us
get
that
accept
it
that'd
be
really
nice.
These
fix
the
problems,
fix
the
ultimately
fix
the
bug
in
the
scheduler
itself,
so
that
it
distributes
pods
correctly
to
maintain
AZ
balance.
B
Fortunately,
in
the
case
of
the
scheduler,
you
don't
have
to
fork.
Excuse
me
patch,
kubernetes,
there's
a
workaround.
You
can
just
change
the
docker
image
of
the
scheduler,
so
there's
a
silver
lining.
You
don't
have
to
run
a
completely
patch
version
of
kubernetes.
You
can
just
kind
of
poke
in
this
change
and
get
an
upgraded
scheduler.
So
the
takeaway
you
may
need
to
make
fixes
to
the
community
scheduler
itself,
but
you
can
upload
at
least
you
can
easily
upload
them
as
custom
image.
A
A
A
Are
these
various
Twitter
handles
and
just
a
quick
plug
directly
after
this
talk
been
huge,
we'll
be
in
the
room
next
door
talking
about
scaling
kubernetes
to
thousands
of
nodes
across
multiple
clusters
calmly,
and
if
you
want
to
hear
more
about
basically
ways
to
blow
up
your
kubernetes,
but
specifically
focusing
on
latency
tomorrow,
our
co-workers,
Jen
and
Steven
will
be
talking
about
the
Kuban
Eddie's
make
my
p90
5s
worse.
Thank
you.
Everyone
thank.