►
From YouTube: Keynote: How Spotify Accidentally Deleted All its Kube Clusters with No User Impact - David Xia
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Keynote: How Spotify Accidentally Deleted All its Kube Clusters with No User Impact - David Xia, Infrastructure Engineer, Spotify
During Spotify's Kubernetes migration, David's team deleted most of their production Kubernetes clusters. Accidentally. Twice. With little to no user impact. David shares how they recovered and learned to operate many clusters automatically and safely.
In 2017, Spotify planned the migration of hundreds of teams, thousands of services, and tens of thousands of hosts to Google Kubernetes Engine (GKE). In the last half of 2018, Spotify migrated 50 teams and hundreds of services, including critical ones, onto multiple production clusters.
David describes what led to the cluster deletions and how they barely affected users. Since the postmortem, Spotify has minimized downtime and human error by declaratively defining clusters in code with Terraform, backing up and restoring clusters with Ark, and increasing scalability and availability by running many more clusters.
https://sched.co/MQbb
A
Hello
good
morning,
welcome
to
day
two
of
koukaon
I'm
really
excited
to
be
here
in
beautiful
Barcelona
and
I'm,
grateful
to
the
CN
CF
for
giving
me
this
chance
to
tell
you
about
how
we
had
Spotify,
like
Brian,
said,
deleted
our
production
kubernetes
clusters
accidentally
twice,
but
with
little
to
no
end
user
impact.
Now
I
actually
just
am
curious.
Raise
your
hand
if
you
have
accidentally
deleted
a
production
cluster
just
want
to
get
a
sense,
cool
yeah.
Look
at
that
elite
club
out
there.
Yes,
I,
see
you
alright.
A
My
team
builds
infrastructure
related
to
google
kubernetes
engine
and
for
the
purposes
of
this
talk
again
we'll
call
my
team,
the
cluster
operators
and
we'll
call
the
feature
teams,
the
cluster
users
so
currently,
as
Spotify
we're
helping
all
stateless
back-end
services
migrated
over
to
gke
and
at
the
time
of
these
incidents,
we
were
only
running
a
subset
of
services
and
partially
each
on
gke
I'll.
Tell
you
a
little
bit
more
about
what
that
means
later.
A
I'm
going
to
tell
you
two
of
them
and
the
first
one
is
about
how
I
accidentally
deleted
this
cluster,
and
the
other
is
about
how
my
co-workers,
when
trying
to
prevent
that
from
ever
happening
again,
they
accidentally
deleted
this
cluster
and
it
gets
better
because
in
the
process
of
trying
to
recreate
this
cluster
in
Asia,
that
happened.
Okay.
So
how
exactly
did
this
happen
and
how
did
we
prevent
any
end
user
impacts
so
that
people
could
keep
listening
to
the
music
that
they
loved?
A
So
it
was
November,
2018
and
I
wanted
to
test
the
gke
feature
so
I
create
a
new
GK
test
cluster
in
a
test
project
and
I
wanted
to
have
the
same
configuration
as
our
production
clusters.
So
I
opened
two
browser
tabs
one
for
the
production,
one
one
for
the
testing
one
I
finished
my
test
and
I
wanted
to
delete
that
test.
Cluster
and
I
had
the
wrong
tab
open,
so
I
was
freaking
out
and
on
slacker
I.
Remember
asking
my
colleague:
how
do
you?
How
do
you
stop
this?
A
Is
there
any
way
to
stop
this
I
had
deleted
a
50
node
production
cluster
running
dozens
of
workloads,
so
with
GK
it's
actually
really
easy
to
create
a
cluster.
That's
what's
really
awesome
about
it.
You
click
one
button.
You
get
everything
masters
nodes,
you
get
networking
all
set
up.
You
get
authorization,
unfortunately,
with
GK.
It's
also
really
easy
to
delete
an
entire
cluster.
So
with
one
click
of
a
button,
everything
goes
away.
A
Masters
nodes,
authorization,
networking,
and
so
is
there
a
way
to
stop
it
and
I
learned
the
hard
way
that
no
there's
there's
just
no
way
to
stop
it.
You
just
wait
for
it
to
clean
up
all
of
the
compute
instances,
and
then
you
kind
of
like
hold
your
breath
and
hopefully
nothing
too
bad
happened.
You
try
to
recreate
it.
So,
let's
talk
about
how
we
recreated
this
cluster.
It
took
us
three
hours
and
twenty
five
three
hours
and
15
minutes
to
restore
it
fix
all
of
our
integrations
with
it.
A
The
restoration
time
took
this
long
because
we
found
bugs
in
our
cluster
creation
scripts
that
we
hadn't
exercised
a
lot.
Our
documentation
was
incomplete
and
sometimes
just
had
mistakes
in
it
and
our
cluster
creation
process
wasn't
resumable,
so
sometimes
would
fail
halfway
and
we
had
to
restart
from
the
very
top
we
didn't
design
it
to
be
resumable
from
the
middle.
So
this
wasted
a
lot
of
time.
A
Okay,
so
we
thought
about.
How
would
we
you
know
human
mistakes
happen?
How
are
we
going
to
prevent
this
from
happening
again
and
we
wanted
to?
We
decided
to
use
an
open-source
tool
for
putting
your
infrastructure
in
code.
This
was
called
terraform
just
one
of
the
many
out
there
and
basically
allows
you
to
codify
your
infrastructure.
You
can
write
down
your
clusters,
declaratively,
you
can
version
it.
You
can
review
changes
and,
while
trying
to
adopt
terraform,
unfortunately,
we
we
accidentally
deleted,
not
one
but
two
clusters.
A
So
how
did
this
happen?
This
is
the
second
story
and
I'll
give
you
some
context
about
how
we
configured
our
clusters
and
how
terraform
works.
So
you
understand
the
details
of
it,
so
we
have
a
shared
git
repository
where
we
define
all
of
our
cluster
level
configuration
and
resources
on
for
every
cluster,
and
this
repository
is
used
by
both
cluster
users
and
by
my
team,
the
cluster
operators.
A
So
it's
all
mixed
together
when
we
started
using
terraform
we
we
realized
that
there
was
like
a
state
file
that
terraform
wrote
to
the
kind
of
like
know
about
the
state
of
your
infrastructure,
but
we
weren't
very
familiar
with
it.
Terraform
actually
uses
the
state
file
to
decide
what
it's
going
to
do
so.
First,
my
team.
A
When
trying
to
get
terraform
and
manage
these
clusters,
we
created
a
pull
request
to
import
the
asia
cluster
into
the
state
file
and
this
state
file
actually
effects
what's
going
to
happen
in
production,
but
we
ran
review,
builds
that
actually
modified
the
state
file
during
a
during
the
view
bill.
So,
let's
think
about
that,
we
modified
a
global
state
file
that
affects
what
happens
in
production
during
a
review
build.
A
We
didn't
merge
this
PR
yet
though,
and
then
a
few
minutes
later,
a
cluster
user
who
knows
nothing
about
terraform
and
shouldn't
need
to
know
anything
about
it.
They
made
a
pull
request
to
add
three
more
users
to
an
hour
back
file,
an
existing
rule,
binding
file,
and
they
they
asked
us
to
review
it
and
it
looked
very
innocuous.
It
was
just
three
extra
users
on
an
existing
game
space,
but
what
was
important?
What
wasn't?
What
was
in
the
PR
or
the
change
file?
A
It
was
what
was
missing
and
it
was
missing
the
definition
for
the
Asia
cluster
and
these
two
pull
requests
were
merged
out
of
order,
so
that
one
was
merged
to
master
first.
Can
you
guess
what
happens
next
and
that's
what
happened
so
terraform
looked
at
this
emerged
it
in
it's
like,
oh
I,
don't
see
the
Asia
cluster
defined
I'm,
just
gonna
this.
This
shouldn't
be
here:
I'm,
just
gonna
get
rid
of
it.
So
this
way
we
thought
okay!
Well,
it
can't.
You
can't
get
any
worse
right.
A
We
still
have
that
unmerged
PR,
oh
and
when
you
think
that
it
can't
get
any
worse,
it
can
always
get
worse.
It's
always
like
I
know
it's
like
one
of
those
phrases
like
I'll.
Do
the
dishes
later.
You
know
it's
not
gonna
happen,
so
we
still
have
that
emerged
PR
from
the
previous
slide.
That
declared
the
age
of
cluster.
So
we
thought
okay,
we'll
just
merge
that
one
in
and
it
will
recreate
it.
So.
A
Our
cluster
creation
script
actually
fails
halfway
because
it
didn't
have
the
right
permissions.
So
before
we
were
using
terraform,
each
of
us
were
just
running
a
few
terraform
commands
locally
and
our
on
our
work
laptops
with
our
personal
GCP
accounts
and
those
personal
accounts
had
they
were
owners
of
the
project.
So
we
had
all
the
permissions
we
needed,
but
now
we're
using
a
service
account
to
run
these
terraform
commands
and
they
didn't
have
all
the
permissions.
A
So
we
gave
it
we
kind
of
just
like
gave
it
all
the
permissions
it
needed,
but
these
actual
were
different.
Permissions
in
terraform
originally
used
to
import
the
clusters
and
when
you
have
different
permissions
and
you
call
the
Google
API
s,
you
get
different
attributes.
So
now,
when
terraform
gets
a
list
of
clusters
from
GCP,
they
have
different
properties
because
it
can
see
more,
and
so
these
properties
have
changed
depending
on
their
permissions.
A
So
we
use
one
set
of
permissions
to
import
the
clusters
and
now
we're
using
another
set
of
permissions
to
kind
of
manage
them.
And
now
the
clusters
look
different
to
terraform.
So
that
happened,
because
if
I
don't
know
about
this
cluster,
this
must
be
different.
So
at
this
point
we
have
deleted
2/3
of
our
production
clusters
from
the
face
of
the
earth.
So
this
was
pretty
bad.
Thankfully
didn't
get
any
worse
than
this.
We
didn't
delete
like
all
three
this
time,
but
this
was
pretty
bad
still.
A
A
My
team
realized
all
the
places
we
had
hard-coded
the
old
clusters
master
IPS,
so
we
had
to
go
in
and
like
update
all
of
those
and
kind
of
minor
annoyance,
but
everyone
using
coop
CTL
had
to
refresh
their
cluster
credentials
when
we
recreated
the
clusters
that
had
new
certificates,
but
otherwise
our
user
support
slack
channel
internally
was
quiet.
It
was
like
you're
really
quiet
and
no
one
paged
us.
A
So
let's
talk
about
the
end
user
impact
like
the
sky
who's
listening
to
Spotify.
While
you
know
on
his
jog
he
kept
running,
we
had
no
reports
of
either
cluster
deletion
affecting
our
end
users,
so
first
a
summary
of
how
we
did
this
and
what
we
did
right,
that
prevented
any
end
user
impact.
So
from
day,
one
of
doing
this
complex
migration,
we
plan
for
failure,
because
you
know
things
happen.
A
So
how
did
we
plan
for
failure?
We
originally,
while
migrating,
we
told
each
team
to
migrate
their
services
just
partially
to
kubernetes,
don't
put
100%
of
it
on
do
half
and
half
gradually
ramp
up.
We
were
still
building
confidence
in
our
ability
to
manage
these
clusters
and
to
scale
them
up
and
to
test
out
all
of
our
custom
integrations
with
it.
A
The
second
way
that
saved
us.
The
second
thing
that
saved
us
was
how
we
registered
these
services
running
on
kubernetes,
and
the
results
of
these
two
is
that
the
the
failover
to
non
communities
instances
happened
as
we
expected
when
the
clusters
were
deleted.
So
let
me
be
clear:
these
three
things
were
not
accidents,
but
they
were
deliberate
choices.
We
made
to
make
our
infrastructure
more
reliable
and
our
migration
process
reversible
and
now
going
to
go
into
a
little
more
details
on
each
of
these
three
and
how
we
did
that.
A
So
for
the
partial
migration
on
a
per
service
level,
kubernetes
usage
is
Spotify
at
the
time
was
marked
as
beta,
and
what
this
means
is
that
we
recommended
teams
only
migrate,
some,
but
not
all,
of
each
of
their
services.
Capacity
to
kubernetes
and
meanwhile
we
were
continue
working
on
integrations
reliability
and
managing
multiple
clusters
at
once,
and
we
actually
registered
our
our
services
on
kubernetes
in
a
kind
of
different
way.
The
non
Kerber
Nettie's
way
due
to
interoperable
interoperability
reasons.
A
So
our
internals
service
discovery
uses
pot
IPs,
it
doesn't
use
the
service
at
all,
and
this
internal
service
discovery
pulls
the
services
endpoints
and
then
it
updates
it's
it's
pretty
much.
It's
updates
its
internal
state
with
the
pot
IPS
and
then
when
it
noticed
that
it
couldn't
reach
a
cluster,
our
team
was
paged,
and
so
we
would
have
to
go
and
actually
remove
that
cluster
from
the
configuration
so
that
that
service
discovery
mechanism
would
no
longer
pull
that
non-existent
cluster
anymore.
And
then
what
happened
was
that
we
failed
over
to
the
non
kurby
Nettie's
instances.
A
We
could
always
have
a
chance
to
reverse
it
if
we
needed
to
so
number
one
thing
to
do,
and
also
a
huge
part
of
what
saved
us
was
we
backed
up
these
clusters
from
day
one
we
codify
that
infrastructure
or
started
to
after
the
first
cluster
was
lead
'add.
We
started
performing
disaster
recovery
tests
and
we
made
sure
that
we
practiced
these
drills.
We
practice
these
scenarios
and
actually
ran
through
them,
so
our
cluster
backups
they
were
essential.
A
We
had
already
tested
restoring
from
these,
because
if
you
have
backups
but
you've
never
tested
restoring
from
your
backups
I'm,
sorry,
you
don't
have
backups
right.
You
just
never
tested
them,
codifying
your
infrastructure,
so
we
were
trying
to
introduce
terraform
gradually
and
it
was
you
better
than
what
we
had
even
though
it
you
know
there
were
some
hiccups
along
the
way.
A
Terraform
and
tools
like
this
help
us
standardize
our
workflow
and
the
change
management
of
infrastructure
code.
We
added
linters
and
validators
to
make
sure
that
the
configs
that
we're
writing
were
correct.
We
also
added
other
things
like
adding
the
output
of
the
dry
run
to
the
comments
in
the
pull
requests,
so
that
it
was
very
obvious
what
it
was
going
to
do
and
we
use
github.
So
we
can
have
status
checks
that
are
required
to
pass
before
merging.
One
of
these
status
checks
is
that
the
branch
must
be
up
to
date.
A
A
Of
course
we
require
approving
pull
request
reviews
and
we
would
just
immediately
fail
reviews
if
we
detected
certain
keywords
like
destroy.
You
probably
don't
want
to
destroy
something.
We
still
do
those
destructions
of
clusters
manually,
but
not
not
in
this
repository,
so
disaster
recovery
tests
disasters
will
happen,
whether
you
plan
for
them
or
not.
So
it's
always
good
to
just
plan
for
them.
We
schedule
them
in
advance
and
we
announce
widely
to
operators
and
users
when
we're
gonna
do
them.
A
But
now
we
can
actually
restore
clusters
that
are
way
bigger
than
the
ones
we
deleted
before
in
one
hour,
so
I
think
that's
a
huge
improvement
and
most
very
grateful
and
thankful
that
Spotify
has
engineering
teams
have
a
great
culture
of
learning,
not
blame,
so
I
was
freaking
out
when
I
deleted
that
cluster.
My
team,
though,
was
very
supportive,
and
you
know
they
said
things
like
it's
better
now
than
never.
You
know
this
would
have
happened.
Anyways
and
Ashley
said
we
learned
so
much
from
doing
this.
A
They
stopped
short
of
like
thank
you
for
deleting
it
on
a
Friday,
but
you
know
it
was
still
good
that
that
happened
before
we
got
too
many
users
on
so
we're
all
human
and
we've
all
made
mistakes.
What
is
important
isn't
who
made
the
mistake,
but
what
you
learned
and
how
you
can
prevent,
making
that
same
type
of
mistake
in
the
future.
A
So
the
next
steps
for
us
using
kubernetes
as
Spotify
I'm,
very
happy
to
say
that,
as
of
last
Monday,
we
kind
of
flipped
the
switch
kubernetes
is
available
for
everyone
to
use,
and
it's
marked
as
generally
available
at
Spotify,
which
means
that
we've
kind
of
lifted
that
recommendation
against
running
your
service
entirely
on
kubernetes.
We
feel
confident
enough
that
you
can
take
your
mission-critical
service
and
run
it
a
hundred
percent
on
kubernetes,
and
we
now
have.
We
are
in
the
process
of
figuring
out
or
have
already
kind
of
made
good
progress
on
managing
configuration
around.
A
A
So
that's
my
talk.
I
want
to
again
thank
the
CN,
CF
and
Spotify
for
this
opportunity.
I
want
to
thank
my
team
for
all
their
hard
work
for
making
kubernetes
a
reality
in
production
and
I.
Want
to
thank
all
of
you
here
for
attending.
Spotify
is
a
great
place
to
work
if
you're
looking
we're
hiring
we're.
Also
using
the
latest
cloud
native
technologies
to
make
the
entire
music
ecosystem
of
work
better
for
everyone,
so
my
name
is
David.
You
can
find
me
on
Twitter
here.
Thank
you.
So
much.