►
Description
Casey Rosenthal, of Netflix, describes their approach to Chaos Engineering. Chaos Engineering is a best practice for building confidence in distributed systems that run at scale. It is a logical consequence of loosely coupled, reactive, and microservice architectures. Netflix handles about a third of the Internet’s bits at peak, all controlled by a microservice architecture. They use tools like Chaos Monkey and Chaos Kong to induce failures in their production system, but breaking things is just a small part of what Chaos can and will be. Casey surveys the current state of Chaos best practices, and provides a preview of the future of Chaos.
About CodeConf
CodeConf improves the software community by providing a forum for thought-provoking talks and forging social connections. The third installment of the CodeConf series took place in Nashville in 2015. Attendees came together to discuss open source, best practices, documentation, and community.
For more information on this year's CodeConf, go to:
https://codeconf.com/
B
My
name
is
Casey
Rosenthal
I
work
for
a
small
internet
company
called
Netflix.
If
you
haven't
heard
of
us
were
kind
of
like
blockbuster,
and
the
topic
I'm
talking
about
today
is
chaos,
engineering
which
is
really
exciting,
because
I'm
basically
introducing
a
whole
new
discipline
into
software
engineering
into
comput
ology.
If
you
l
chaos,
engineering
is
basically
a
system
for
increasing
your
confidence
in
a
distributed
software
environment
Netflix
the
company
worked
for
really
big
into
open
source
software.
A
good
portion
of
our
platform
is
released,
open
source.
B
One
of
the
components
that
you
may
have
heard
of
is
chaos.
Monkey
I'll
give
you
a
for
those
of
you
who
don't
know
what
chaos
monkey
does.
Netflix
at
peak
is
about
thirty-seven
percent
of
North
America's
internet
traffic
by
bytes.
So
that's
a
lot
of
data
most
of
that
comes
out
of
our
CDN,
but
some
of
it
comes
out
of
control
plane
that
we
deploy
on
AWS
in
multiple
regions.
B
Architects
are
cool,
nothing,
yes,
anybody
who's
an
architect,
but
in
traditional
software
and
architect,
generally
maintains
a
mental
model
of
an
entire
system
and
then
helps
guide
to
engineers
right,
that's
kind
of
the
job
description
for
software
engineering,
architect
and
what
we've
noticed.
What
I've
noticed
is
that,
in
the
industry,
we're
moving
away
from
having
this
role,
in
fact
that
Netflix
we
don't
have
architects,
we
just
have
senior
software
engineers
and
by
working
to
make
their
component
code
the
best
that
it
can
be.
B
They
don't
need
architects
and
the
development
of
the
product
is
controlled
by
like
an
invisible
hand
right.
So
we
don't
we're
making
a
couple
of
assumptions.
One
is
that
we're
getting
to
the
point
where
software
so
complex
that
no
human
can
hold
the
entire
mental
model
in
their
head
at
a
given
time
anyway.
B
So
it's
really
hard,
if
not
impossible
at
this
stage
in
software
development,
for
us
to
have
architects,
but
we're
also
making
a
conscious
decision,
which
is
that
we're
treating
off
on
the
comfort
of
having
an
architect
to
instead
have
more
velocity
and
flexibility
in
what
we
build
by
empowering
the
engineers
to
build
the
things
that
they
want
to
without
hat,
without
going
through
an
architect.
Okay,
hope
that
makes
sense.
That's
that's
a
trade-off
that
we're
again
consciously
making,
but
it
does
take
away
some
of
the
order.
B
Some
of
the
comfort
of
having
an
architect
that
combined
with
other
industry
trends,
leads
to
an
exploit
us
are
not
that
kind
of
explosion.
This
kind
of
explosion,
an
explosion
of
complexity
and
I'm
just
going
to
tacit
lee
n,
plus
explicitly
assume
that
all
of
you
agree
with
me
that,
as
a
whole,
our
industry
is
getting
more
complex
from
microservice
architectures
to
any
loosely
coupled
design
paradigm
reactive
systems
message
based
distributed
systems
containerisation.
So
now
anybody
can
deploy
directly
to
production
in
a
matter
of
seconds
instead
of
minutes
or
hours
or
whatever
your
it's
like.
B
What's
previously
continuous
deployment
stuff
is
getting
more
complex
and
that's
really
scary.
So
it's
not
scary.
It's
flowers
right,
it's
beautiful,
it's
not
scary,
but
it
does
create
some
uncertainty.
So
chaos
engineering
is
designed
to
reduce
that
journey.
To
add
confidence
in
these
systems
that
we
built
I'm
going
to
quickly
give
you
a
taste
of
the
kind
of
complexity
that
that
I'm
talking
about
here
by
taking
you
through.
B
What's
called
the
beer
game,
there's
actual
beer
tonight,
so
you're
not
going
to
get
drunk
during
his
presentation,
I'm,
not
well,
maybe
so
the
beer
game,
if
you've
pursued
an
MBA
and
you've
taken
a
systems.
Thinking
class,
maybe
you've
been
through
the
beer
game
before,
if
you're
thinking
of
getting
an
MBA
I'm
ruining
a
semester
for
you,
so
we're
not
actually
going
to
play
this
I'm
going
to
take
you
through
what
a
typical
game
of
the
bear
game.
A
typical
game
usually
looks
like
so
usually.
B
Class
is
divided
into
three
teams:
customers
who
like
to
drink
beer,
retailers
they
get
to
sell
beer
and
manufacturers
microbrewers
they
may
get
to
make
the
beer,
and
these
three
teams
can
only
communicate
to
each
other
through
slips
of
paper.
That
say
what
what
their
intention
to
purchases
from
the
next
person
of
the
chain-
okay,
I'm
gonna
have
to
this
is
a
live
presentation
of
chaos
in
action.
B
The
font
got
screwed
up,
so
this
table
is
all
out
of
whack,
but
I
will
just
fill
in
the
numbers
for
you
and
you're
going
to
have
to
use
your
imaginations.
So
the
yell
calm
is
the
customers
of
green
columns
or
the
retailers
and
the
columns
for
the
manufacturers.
So
a
customer,
a
steady
state
is
customer,
wants
one
case
of
beer
a
week,
so
they
go
to
the
retailer
who
has
two
in
stock.
They
buy
one
that
retailers
left
with
one.
The
retailer
orders
one
case
of
beer
from
the
brewery.
B
Okay,
that's
steady
state,
but
in
so
that's
a
week,
zero.
So
in
week,
1
something
weird
happens:
Elvis
comes
out
of
retirement
and
he
wants
to
celebrate
so
he
goes
to
get
some
beer
and
he
orders
for,
but
there's
only
two
in
stock,
so
he
can
only
purchase
two
and
that
gets
past
the
retailer
to
the
microbrewery
and
so
on,
and
then
he
throws
a
bigger
party
because
he
decides
he
likes
this
beer
so
much.
B
Let's
call
it
Graceland
lager
that
he
writes
a
song
about
it
and
he
invites
his
friends
over
and
they
want
to
order
16,
but
they
can
only
purchase
for
because
there's
only
four
in
stock
at
the
retailer
and
you
get
the
retailer
passed,
the
order
on
to
the
microbrewery
and
then
the
song
starts
to
take
off
and
into
town
right
and
presumably
be
here
in
Nashville,
and
so
now
64
people
want
to
buy
cases,
but
the
going
by
eight
is
a
retailer.
Only
has
eight
so
there's
this
missing
demand.
B
This
remainder
demand
of
56
and
the
retailer
is
completely
out,
but
they
see
hockey
stick
rose
so
they're
like
oh
we're,
going
to
start
manufacturing
250
cases
a
week,
then
somebody
realizes
hey
Elvis.
If
you
were
alive
today,
he'd
be
like
80
years
old,
and
this
guy's,
like
in
his
30s,
so
clearly
he's
not
Elvis
so
well.
That
news
is
breaking.
People
still
want
to
order
this,
but
there's
none
left
in
the
store.
B
The
retailer
has
more
cases
of
beer
in
stock
than
they
could
possibly
ever
sell
in
a
year,
a
lot
of
missing
domain
remaining
stock
and
the
manufacturers
also
left
with
a
bunch.
So
we
end
up
with
these
results:
customers
who
were
not
particularly
happy
retailers
who
are
way
over
stocked
and
missed
a
huge
amount
of
potential
profit
and
a
manufacturer
who
is
also
way
overstocked
and
that
has
an
extra
factory.
What
the
heck
are
they
going
to
do
with
that?
B
B
The
analogy
here
is:
if
we're
in
a
micro
service,
micro
service
architecture
or
learn
a
decentralized
development
environment,
all
of
our
libraries,
all
of
our
services
might
be,
might
have
full
test
coverage
right
from
unit
test
antenna
test.
Everything
is
is
golden
and
passing
they
are
behaving
exactly
as
intended.
However,
in
the
aggregate
behavior
of
the
system,
we
get
a
really
undesirable
result.
B
That's
the
kind
of
thing
that
chaos
engineering
is
trying
to
attack.
Heads
on
that
uncertainty
that
potential
for
a
systemic
failure.
Testing
does
not
address
that
so
I'm
going
to
propose
a
five
principles
that
cast
engineering
is,
is
founded
on
and
hopefully,
from
this
enthusiastic
open
source.
Community
I
can
get
some
feedback
on
so
KS
engineering
first
principle
keep
it
real.
I
almost
said
that,
with
a
straight
face
up
here,
Cass
engineering
is
a
series
of
experiments
and
the
variables
are
real-world
variables.
So
the
things
that
can
happen
in
real
life.
B
Thank
you.
Perfect
example
of
chaos
in
action
right.
So
this
whole
presentation
has
been
an
experiment
and
thank
you
casey
and
at
any
time
she
could
have
thrown
this
at
me,
and
I
had
no
idea
what
she
was
going
to
do
it,
but
that's
exactly
kind
of
real-world
effect
that
I'm
talking
about
it
doesn't
matter
what
the
integrity
of
your
engine
is.
If
we
throw
a
pothole
in
front
of
you,
okay,
examples
that
that
we
want
to
use
servers,
disappearing,
backhoes,
cutting
cables,
hard
disk
going
bad,
not
always
bad
things.
B
B
All
of
these
are
outside
of
the
scope
of
the
logic
of
that
algorithm
that
the
engineers
are
writing.
Okay.
So
this
is
not
testing.
Testing
is
the
domain
of
correctness.
Where
recovers
the
domain
of
crack
correctness,
where
you
have
an
environment,
and
you
insert
some
conditions
and
assert
that
it
does
what
you
want.
Ks
engineering
is
about
running
an
experiment
where
you,
you
have
an
environment,
you
change
some
variables
and
you
don't
necessarily
know
what
you're
going
to
get.
But
you
know
what
to
look
for
it's
a
hypothesis
that
you
want
to
refute.
B
B
We're
making
an
explicit
assumption
that
the
system
is
getting
too
complex
for
an
architect,
a
hold
a
mental
model
of
the
whole
thing
in
their
head.
So
by
the
same
token,
we
assume
that
QA
testing
the
engineers
writing
this.
They
also
cannot
predict
accurately
what
the
user
will
do
with
that
system.
B
B
Third
principle:
measure
steady-state,
so
steady
state
is
like
your
guiding
star
for
a
chaos
experiment
and
it's
it's.
This
can
be
complicated
fearing
out
what
steady-state
means
for
your
business
case,
looking
for
something
that
sometimes
in
some
cases,
isn't
there,
but
it's
an
explicit
trade-off
that
we're
making
it's
a
conscious
trade-off
for
making
to
say,
okay,
the
number
of
transactions
per
second,
that
the
site
is
supposed
to
handle
or
the
number
of
movies
being
washed
per
second,
or
something
that
steady
state.
B
We
have
to
fingerprint
that
as
the
help
of
the
system
and
the
experiments
have
to
focus
on
measuring
that
as
the
metric
of
import,
this
keeps
us
from
focusing
on
things
that
are
outside
of
the
realm
of
chaos
like
how
does
this
model
work
if
you're
investigating
something?
That's
an
investment
in
your
time
and
how
the
model
works
might
be
really
interesting
and
it
might
be
necessary
if
you're
fixing
bugs,
but
that's
outside,
of
the
realm
of
chaos,
engineer
so
for
chaos.
B
Engineering
we
have
to
focus
on
measuring
the
steady
state
of
the
system
run
continuously,
I'm,
not
going
to
say
why,
again,
all
these
leave
this
as
an
exercise
but
cancel
exercises
have
to
be
running
all
the
time
and
the
fifth
principle
is
machine
intelligence.
So
the
analogy
I
like
to
use
here
is
a
cast
engineer
and
can't
simply
build
a
better
broom
to
sweep
up
the
floor
and
then
ask
people
to
sweep.
We
need
to
build
something
like
a
Roomba
ignore
the
animals
I
swear.
B
This
is
the
only
picture
of
a
room,
but
I
could
find
on
the
internet,
but
you
know
between
fingerprinting
the
health
of
the
system
and
knowing
having
the
automation
to
run.
Ts
exercises
constantly
have
it
correctly
parameterised
and
learning
from
its
own
previous
experiments.
You're
gonna
need
machine
intelligence
to
pull
off
the
ecosystem
of
chaos
successfully.
You
can't
just
do
these
as
one-off
manual
things.
So
to
summarize,
the
55
principles
variables
reflect
real
world
world
affects
not
algorithmic
or
logical
correctness.
We
want
to
identify
the
requests,
interaction
in
production.
B
We
want
to
measure
steady-state,
not
not,
models,
not
other
effects.
We
want
to
run
experiments
continually
and
machine
intelligence
is
required
and
I'm
arguing
that
that
makes
a
foundation
for
chaos.
Engineering
that
I
hope
is
an
industry
phenomena
in
2015
2016,
that's
chaos,
engineering,
I'm
case
you
Rosenthal,
Casey,
the
lessor
and
again
I
would
love
your
feedback
I'm,
not
above
bribing
you.
So
I
have
netflix
gift
certificates.
The
best
question
comment
or
criticism
that
comes
from
a
human
or
robot
I'll.
Give
you
a
netflix
gift
certificate
thanks.
A
B
Aside
from
chaos,
Netflix
constantly
ones
AP
tests
to
see
to
try
to
constantly
continuously
improve
the
user
interface,
so
that
might
be
more
like
what
you're
talking
about,
but
in
chaos.
There's,
no
reason
why
we
shouldn't
also
do
things
like
threshold
testing,
where
we
say
hey
what,
if
there's
you
know,
we
release
the
season
of
aren't
just
a
new
black
and
all
of
a
sudden.
We
have
a
huge
surge
in
traffic.
Let's
has
that
kind
of
thing
which
isn't
a
failure,
its
that's
a
good
condition
right,
can't
let
daredevil
ever
stop.
B
So
you
don't
accidentally
take
our
production
who
says
that's
a
bad
thing,
so
I
think
that's
I!
Think
that
the
experiment
you
would
have
to
look
at
it
on
a
case-by-case
basis.
So,
for
example,
we
have
what
we
called
chaos
Khan,
where
we
feel
out
an
entire
region
in
non
amazon
terms.
That's
like
an
entire
datacenter.
We
move
all
internet
traffic
over
to
another
region
and
and
without
telling
amazon.
They
don't
like
that
fun
fact,
and
in
that
case
we
know
things
are
going
to
break
the
first
time.
B
We
did
it
and
we
just
have
to
go
okay,
okay,
it's
where
it's
going.
It's
going
up,
something's
broke
and
roll
back
roll
back
roll
that
move
all
the
traffic
back
to
that
Dave's
then,
and
then
fix
those,
and
then
it
took
a
couple
tries
before
now
we
run
that
kind
of
exercise
routine.
So
it's
it's
definitely
there's
definitely
something
outside
of
a
principle
that
is
in
the
context
of
the
kind
of
experiment
that
you're
running.