►
From YouTube: Keynote: Anatomy of a Production Kubernetes Outage - Oliver Beattie, Head of Engineering, Monzo Bank
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Keynote: Anatomy of a Production Kubernetes Outage - Oliver Beattie, Head of Engineering, Monzo Bank
This talk will dive into a production Kubernetes outage that Monzo experienced a few months ago, its causes and effects, and the architectural and operational lessons learned.
About Oliver
Oliver Beattie is Head of Engineering at Monzo, leading the development of the distributed systems to power a new kind of bank. He previously worked on Hailo's global micro-services platform.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
everyone,
so
my
name's
Oliver
I
am
the
head
of
engineering
at
Mons
Oh,
which
is
as
Liz
mentioned.
A
bank
Mons
Oh
is
a
start-up
Bank
based
in
London.
We
are
built
for
the
20th
century
and
also
built
in
the
24
21st
century.
Pardon
me
and
we
are
built
with
and
for
our
community
currently,
which
is
over
600,000
customers.
A
We
we
basically
want
to
build
the
best
Bank
in
the
world
and
kind
of
one
of
our
one
of
our
principles
is
to
move
mountains
behind
the
scenes
to
make
everything
we
do
seem
simple
for
the
customer,
and
this
talk
is
about
some
of
those
mountains.
The
difference
from
legacy
banks,
hopefully,
is
apparent.
When
you
first
open
our
app.
A
If
you
look
at
a
transaction
in
our
app
you'll,
see
there's
kind
of
no
none
as
a
cramped
or
caps
statement
description,
so
you
might
be
used
to
as
rich
information
about
the
transaction.
We
show
you
the
location,
we
show
you
an
automatically
categorized
category
for
it
and
we
let
you
upload
receipts,
split
the
transaction
with
friends
and
we
also
perform
spending
analysis.
This
is
my
lunch
I
think
from
last
week
and,
as
you
can
see,
I've
spent
a
lot
on
lunch
over
time.
A
Whenever
you
spend
money,
you
get
a
real-time
notification
from
onso
everything
about
it's
real
time.
Your
balance
updates
in
real
time
and
you
get
emoji
with
every
transaction.
One
of
my
favorite
features
over
the
last
three
years.
We've
built
a
banking
system
from
scratch.
We
believe
that
this
is
the
only
way
that
we
can
build
the
experience
that
we
want
and
we
also
want
to
own
our
own
up
time.
A
You
can
probably
guess
that
it
runs
on
kubernetes,
given
that
I'm
here,
that's
obviously
one
of
the
components
that's
involved
in
this
outage
kubernetes,
as
you
may
know,
stores
all
of
the
state
about
the
cluster,
the
desired
state
in
the
actual
state
in
sed,
which
is
a
strongly
consistent,
key
value
store
and,
as
I
mentioned,
we
run
hundreds
of
micro
services,
the
thousands
running
at
any
one
time,
and
they
like
to
talk
to
each
other.
We
use
a
linker
D
to
do
that.
A
A
The
last
thing
that's
crucial
here
are
humans
in
any
incident,
something's,
obviously
gone
wrong
kind
of
by
definition
or
human
is
getting
involved.
So
the
story
starts
around
two
weeks
before
the
actual
outage,
so
we
had
upgraded
our
XE
d
cluster.
As
you
may
know,
SED
last
year
released
I
think
it
was
last
year
at
CD
version
3
in
your
previously
on
version
2,
and
we
were
upgrading
to
xt
version
3
as
well
as
taking
the
cluster
from
3
to
9
notes.
This
involved
migrating
to
a
totally
new
cluster.
This
went
totally
as
planned.
A
The
reason
I'm
mentioning
it
here
is:
it
comes
back
to
bite
later
sort
of
fast-forward
a
little
bit
to
the
day
before
the
outage
and
an
engineer
who
worked
I
think
I'm,
one
of
our
product
teams
was
working
on
a
new
feature
and
was
shipping.
This
new
feature
to
production.
It
wasn't
yet
available
for
users,
but
when
they
shipped
it
to
production,
they
saw
some
strange
behavior,
so
they
scaled
it
to
zero
replicas.
So
this
is
important,
because
this
left
a
service
in
the
kubernetes
sense
in
kubernetes
that
had
no
endpoints
again.
A
The
outage
itself
starts
when
an
engineer
deploys
a
change
to
our
ledger,
so
the
ledger
is
kind
of
like
any
other
service
in
our
back
end,
in
the
sense
that
it's
deployed
the
same
way,
it's
built
the
same
way
and
deploying
services
is
very
routine.
It's
not
unusual
for
us
to
deploy
a
hundred
times
a
day,
but
the
ledger
is
quite
important.
If
you
know
anything
about
how
a
bank
works,
it's
it's
the
truth
like
it
tells
us
how
much
money
every
customer
has.
A
A
Few
minutes
later,
it's
identified
that
link
Adi,
which,
as
I
mentioned,
is
necessary
so
that
our
services
can
talk
to
one
another
is
sending
traffic
was
meant
to
go
to
the
ledger
somewhere
else.
So
the
way
that
link
Adi
works
is
it
watches
the
kubernetes
api
and
gets
sets
of
endpoints
from
the
API
and
then
load
balancers
traffic
across
those.
In
this
case
it
was
sending
traffic
to
IP
addresses
that
didn't
exist.
It
was
something
I,
I
peered.
A
A
So
we'd
actually
seen
some
much
smaller
instances
of
this
happening
before
so
on
on
a
small
subset
of
lincolni
pods
we'd,
seen
it
routing
traffic
to
the
wrong
place,
and
we
assumed
this
was
probably
the
same
buck.
So
we
run
some
internal
tools
we
developed
to
tell
us
which
link
ADIZ
were
misbehaving.
In
this
case
it
tells
us
all
of
them
are
so
they're
all
sending
traffic
for
The
Legend
of
the
wrong
place,
which
we've
never
seen
before,
but
sort
of
it's
reasonable
to
assume.
It's
probably
the
same
problem
on
a
larger
scale.
A
Do
we
don't
do
it
all
at
once,
a
little
while
later,
we
notice
that
the
ones
that
have
been
restarted
it
can't
start
and
a
little
bit
investigation
shows
that
the
cubelets
on
every
node,
which
of
the
things
that
are
responsible
for
ensuring
that
containers
are
running
properly,
is
refusing
to
start
the
pod,
because
it
can't
get
the
configuration
that's
needed
to
start
link
a
tea
from
the
API
server.
The
the
logs
indicate
that,
like
the
API
server,
is
failing
to
respond
to
that
request.
A
So
again,
standard
thing,
restart
the
API
service
here
that
actually,
in
this
case,
did
fix
that
problem.
So
after
the
API
server
restart
the
link
ad
pods
started
and
we
continued
rolling
the
link
ad
pods,
just
after
we'd
finished
rolling
the
link
ad
pods.
This
happened.
All
of
the
kind
of
alarms
start
to
go
off,
and
some
of
them
are
very
simple.
Some
of
them
are
just
kind
of
pings
against
our
API
and
the
fact
that
we
can't
respond
to
those
is
pretty
dire.
Like
we've.
Basically,
our
total
outage
on
our
hands
here.
A
A
All
in
all
the
incident
lasted
one
hour
and
21
minutes,
which
is
a
very
long
time
not
ideal.
However,
throughout
this
and
for
reasons,
I'll
come
on
to
talk
about,
the
majority
of
payments
did
continue
to
succeed,
and
this
is
kind
of
one
of
those
core
business.
Metrics
is
one
of
the
things
we
care
the
most
about
like
and
one
of
the
things
our
customers
care,
the
most
about
that
they
can
access
their
money.
So
while
this
was
a
really
hairy
outage
for
our
engineers,
we
only
actually
had
two
customer
complaints
from
this.
A
A
The
first
one
was
a
bug
that
eventually
came
to
light
that
existed
in
the
G,
RPC,
client
library,
so
SCD
in
version
3
uses
G
RPC
to
talk
between
clients
and
servers,
and
this
was
a
bug
in
a
client,
so
the
client
is
embedded
in
the
API
server.
The
API
server
has
this
book
and
it
happened
when
there
are
connection
resets,
because,
as
I
mentioned
earlier,
we'd
two
weeks
earlier
rolled
the
entire
cluster
they'd
all
experience
these
connection,
resets
so
horrible
look.
A
The
second
one
was
the
incompatibility
between
kubernetes
and
linker
D,
as
I
mentioned,
this
only
existed
between
very
specific
versions
of
kubernetes
and
linker
D,
and
it
was
because
of
a
breaking
API
change
that
had
been
introduced
in
kubernetes
1.6
prior
to
1.6.
This
is
how
an
empty
endpoint
set
was
represented
after
1.6.
This
is
how
an
empty
endpoints
that
was
represented
so
tiny
little
change
really,
but
because
linka
D
wasn't
expecting
that
field
to
be
nullable,
it
couldn't
pass
the
JSON
from
its
perspective.
A
It
was
invalid
and
when
he
tried
to
access
that
it's
got
a
nullpointerexception
so
unfortunate.
The
last
thing
that
was
kind
of
a
root
cause
in
a
sense
of
why
this
outage
lasted
so
long
was
human
error.
So,
while
everyone
obviously
acted
with
their
best
intentions,
we
did
make
some
bad
decisions
and
by
we
include
myself
in
that
specifically
restarting
those
linka
deep
odds
wasn't
necessary
and
made
things
far
worse.
The
reason
we
did
that
was
because
we
didn't
have
enough
easily
accessible
info
about
what
to
do
so.
A
A
The
first
one
I
think
is
really
important
and
actually
one
of
the
things
that
worked
well
here
is
this
concept
of
defense-in-depth,
which
is
traditionally
a
security
thing,
which
basically,
is
that
you
have
multiple
layers
of
redundant
security
such
that
they
compensate
for
one
another.
If
one
fails
and
I
think
this
can
apply
to
reliability
as
well
and
did
in
this
case
quite
well.
So
if
I
explain
a
little
bit
about
how
the
MasterCard
Network
works,
it
doesn't
work
on
the
Internet.
A
It's
a
network,
in
the
literal
sense
that
you
connect
a
physical
wire
to
MasterCard
and
that's
kind
of
fundamentally
incompatible
with
AWS,
which
is
where
most
of
our
most
of
our
systems
run.
So
we
have
this
edge
layer
of
data
centers,
which
connect
to
AWS
and
basically
proxy
traffic
between
MasterCard
on
AWS.
A
What
we
did
in
this
case
is
we
have
a
fallback
system
that
exists
in
the
edge
layer,
that
kind
of
in
the
event
of
a
really
catastrophic
failure.
It
can
continue
to
authorize
payments,
even
though
the
everything
else
is
down
subject
to
fraud
controls
and
then,
when
the
backend
comes
back,
it
can
replay
those
transactions.
A
So
customers
didn't
see
transactions
in
their
out
immediately,
but
crucially,
they
could
spend
their
money
and
it
really
paid
off
to
have
built
that
in
this
case
and
I
think
this
is
a
lesson
that
can
apply
it
much
more
broadly
than
just
with
payment.
I
know.
This
is
something
this
is
one
of
the
main
reasons.
I
think
Netflix
built
the
historic
library.
A
The
second
thing
I
want
to
talk
about.
Another
thing:
popularized
by
Netflix,
actually
is
chaos
engineering.
This
is
something
we
were
not
doing,
however,
I
think
if
we
were,
this
would
have
caught
the
issue
before
it
became
an
issue.
So
chaos
engineering
is
defined
as
the
discipline
of
experimenting
on
a
distribution
system
in
order
to
build
confidence
in
the
system's
capability
to
withstand
turbulent
conditions
in
production.
A
A
Netflix
has
this
toolkit
for
doing
this
called
the
simian
army,
and
one
of
the
primary
constituents
of
this
army
is
called
the
chaos
monkey,
which
is
this
thing
that
you
would
run
in
an
environment
in
this
case,
if
we'd
been
running
it
in
production,
which
kills
nodes
in
a
controlled
way.
To
give
you
confidence
that,
if
that
happens,
in
an
uncontrolled
way,
which
we
acknowledge
is
inevitable,
then
you
know
how
your
system
will
behave
and
I
think
if
we'd
been
doing
this
in
production,
we
would
have
seen
that
link.
A
Adiz
cannot
restart
safely
and
we
would
have
caught
that
long
before.
That
was
an
actual
problem.
I
think
this
is
particularly
interesting
in
a
cloud
native
and
kubernetes
sense,
and
there
are
some
projects
starting
to
emerge
around
this
I
think
there's
one
called
cube.
Monkey
I
know
bloomberg,
has
built
a
tool
to
do
this
as
well.
A
The
next
things,
I
think,
are
more
operational,
so
in
this
outage
our
monitoring
did
a
very
good
job
of
telling
us
that
this
deployment
of
the
ledger
had
broken
everything,
but
it
did
not
do
a
very
good
job
of
telling
engineers
that
this
gradual
process
of
restarting
these
lincolni
pods
was
slowly
killing
the
platform
until
it
was
too
late
like
there
were
metrics
for
that,
but
they're,
just
sort
of
buried
too
deeply
in
places
and
not
kind
of
in-your-face
enough
and
I.
Think
putting
this
stuff
in
people's
faces
is
definitely
a
good
practice
and
I.
A
A
The
last
thing
I
want
to
talk
about
is
one
of
the
one
of
the
really
good
aspects:
I
think
of
Manzo's
culture,
one
of
the
main
aspects
of
our
culture.
Most
important
is
transparency.
So,
as
I
mentioned,
we're
building
this
bank
with
our
community
and
what
that
means
is
internally.
We
have
this
concept
of
defaulting
to
something
being
transparent,
and
this
means
that
an
argument
has
to
be
made
to
keep
something
private
rather
than
an
argument
needing
to
be
made
to
make
it
public.
A
A
This
was
read
by
thousands,
probably
tens
of
thousands
of
people,
including
crucially
people
in
the
communities,
community
and
I
thought.
The
response
was
fantastic
to
this,
that
the
community
really
sort
of
took
it
upon
themselves
to
help
us
understand
these
issues
and
fix
them,
and
if
we'd
not
been
forthcoming
about
what
happened,
I
don't
think
we
would
have
benefited
in
this
way
before
you
wouldn't
understand
the
root
cause
of
some
of
these
even
now,
because
some
of
them
are
so
low
level
and
I.
A
Think
lots
of
companies
are
very
good
about
talking
about
their
successors,
but
not
their
failures,
but
I
think
we
could
all
learn
from
each
other's
experiences.
Good
and
bad,
so
I
think
embrace
the
community
in
every
way
that
you
can
so
that's
about
it.
I
would
like
to
have
a
shameless
plug
if
you
think
these,
if
you
think
these
challenges
are
interesting
and
these
technologies
are
interesting,
please
come
and
find
me
tweet
at
me.
Whatever
we
would
love
to
have
you
on
board
thanks
very
much.