►
From YouTube: Deep Dive: Chaos Engineering BoF - Sylvain Hellegouarch, ChaosIQ & Julien Bisconti, 56k.cloud
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Chaos Engineering BoF - Sylvain Hellegouarch, ChaosIQ & Julien Bisconti, 56k.cloud
Automating Chaos Engineering with a Service Mesh and the Chaos Toolkit In this talk, Julien and Sylvain will take you on the journey of performing Chaos Engineering exploration and automation of a simple set of services backed by Istio, a service mesh provider. Julien will first demo how you can degrade conditions in your system through the Istio fault injection API, with a very small blast radius to not impact users. Sylvain will then demonstrate how you can automate Julien's work with the Chaos Toolkit to run Chaos Engineering experiments in a continuous manner while surfacing these executions for greater observability. This presentation will be a good introduction on practicing Chaos Engineering.
To learn more: https://sched.co/Gren
A
Just
kidding
so
we're
gonna
start.
The
plan
for
today
is
to
introduce
the
concept
of
a
service
mesh
and
see
how
the
evolution
of
from
from
the
code
perspective
to
the
infrastructure
and
with
that
it's
going
to
set
the
base
for
gas
engineering,
which
is
a
fault
injection
demo
and
Silvan,
is
going
to
continue
with
the
automation
of
that
experiment
with
the
cows
toolkit
so
see
one
you
might
want
to
introduce
yourself.
B
A
A
B
If
you
are
that
yesterday,
like
Jillian
said,
you
probably
know
me
otherwise,
so
my
name
is
Silvana
GWACs
I'm
the
chaos
IQ
CTO
and
chaos.
Iq
does
few
projects
run,
chaos
enduring
like
chaos,
toolkit
chaos,
platform
and
a
Kelso,
so
I'll
be
demoing
a
kill
circuit,
a
bit
of
the
host
platform
today
and
I'll.
Let
you
continue
thank.
A
A
For
real,
so
so
it
to
get
more
more
serious,
we
there
is
this
idea
that
doing
microservices
is
fast
and
easy,
but
actually
I
wouldn't
like
to
ask
I,
don't
know,
I
think
it
takes
two
good
from
a
monolid
to
a
micro
service
architecture
years.
Okay,
and
still
there
you're
not
fully.
You
cannot
fully
get
rid
of
the
of
the
Manolis.
You
still
have
that
that
core
that's
there,
but
you
build
a
whole
ecosystem
of
micro
services
around
it
and
that
drinks
too.
A
You
know
now
that
you
split
everything
you
you
fall
into
your
problem
with
the
network,
because
if
you
look
at
the
eight
fallacy
of
distributed
system,
the
first
one
is
the
network
is
reliable
and
that's
the
biggest
lie
that
people
tell
you.
You
know
it's
not
going
to
happen,
and
the
proof
is
that,
twenty
years
of
twenty
two
years
ago
an
RFC
came
which
is
called
at
the
bottom
twelfth
networking
truth,
it's
basically
those
those
prove
that
you
know
that
you're
not
going
to
be
faster
than
the
speed
of
light.
A
That
is,
you
know,
even
if
you
manager
say,
but
maybe
if
you
split
proton,
they
good
twice
as
fast.
No,
it's
going
to
be
the
same.
So
with
those
micro
services
we
came
and
the
container
helped
to
for
the
deployment
and
the
repeatability,
and
once
you
have
the
container
you
you
if
they
brought
a
whole
new
set
of
question
that
you
need
to
make
decision
to,
and
one
of
them
is
scheduling.
So
how
do
you
deal
with
secret?
A
How
do
you
make
them
resilient
and
how
do
you
make
those
service
talk
to
each
other,
and
this
is
where
we
have
kubernetes
communities.
If
you
have
communities
doesn't
mean
that
those
problem
are
trivial,
but
at
least
they
provide
you
a
standard
API
to
solve
the
problem,
you
don't
need
to
reinvent
the
wheel,
but
also
once
you
have
communities,
this
is
not
the
Silver
Bullet
I
know
you
can
already
see
people
like.
A
So
don't
worry,
I'm,
not
gonna
trash
Kuban,
which
is
a
brilliant,
brilliant
tool,
and
so
many
people
agree
on
it
that
it
makes
it
super
convenient
to
build
on
top
of
it.
So
if
you,
if
you
want
to
do
for
instant
racing
or
if
you
want
to
know
your
metrics
or
how
to
do
canary
release
and
service
authentication
communities,
doesn't
tell
you
anything,
you
have
to
figure
out
yourself.
So
what
people
do
is
usually
the
use
code
to
solve
those
problems.
But,
with
course,
came
common.
A
A
lot
of
problems
as
well,
like
you,
have
a
ton
of
language
ton
of
libraries
framework
and
they
don't
match,
and
you
never
update
them,
because,
even
if,
if
it
doesn't
force
you
to
upgrade
it,
it's
basically
it
works
and
let's
not
fix
something.
That's
not
broken
and
all
the
little
thing
that
lets
you,
for
instance,
draining
we
request.
A
You
know
if
you,
if,
when
you
deploy,
you
need
to
make
sure
that
all
the
network
is
drained
and
that
the
the
previous
requests
are
not
cut
so
that
you
don't
send
a
500
to
your
customer
and
that
can
impact
your
error
budget.
So
you
need
to
take
care
of
it.
And
even
if
you
go
to
someone
and
say
okay,
can
you
help
me
fix
because
between
deployment
we
see
something
going
on?
And
you
see
for
me,
it
works
fine
or
it
works
on
my
machine.
A
Feel
free
I
have
plenty
of
those,
and
so
the
idea
of
a
service
Nash
is
also
that
you,
you
want
to
extract
the
code
and
make
it
to
to
remove
this
concern
from
the
application,
and
for
that
we
move
it
to
name
trust
structure.
So,
instead
of
having
code
that
you
need
to
import
and
deal
with,
it
becomes
a
separate
piece
that
you
deploy
separately,
and
that
is
the
pattern
that
emerges
the
sidecar
proxy.
A
So
you
see
instead
of
service
a
talking
directly
service
B,
you
go
through
a
proxy
and
that
allows
for
a
lot
of
observability.
You
know
L
all
the
traffic
that
goes
in
and
out.
You
can
do
interesting
things
like
and
mutual
TLS
and
so
on
and
so
forth.
So
that's
where
envoi
came,
and
so
envoi
is
the
state
that
the
network
should
be
transparent
to
the
application.
A
But
of
course,
once
you
have
a
tons
of
microservice
and
each
of
them
have
a
psychic
approxi.
How
do
you
manage
the
fleet
of
them
because
you
need
to
schedule
them?
You
need
to
organize
them,
synchronize
them,
and
so
that's
the
service
mesh.
If
you
want
to
know
how
to
describe
a
service
mesh,
you
need
to
answer
what
problem
does
it
solve,
and
that
is
the
communication
between
services.
A
A
Gonna
show
you
and
voi,
and
it's
just
a
simple
HTTP
bin
application
that
when
you
query
it
it
sends
you
back
the
the
result
depending
on
the
route,
so
here
status,
code
200
is
gonna,
send
me
a
200
code
and
if
I
curl,
that
you
see
that
I
curl
to
receive
the
header
and
if
I
go
and
query
if
I
start
envoy
boom
it
crash
because-
and
it
says
here,
value
required,
so
envoi
doesn't
work
out
of
the
box.
You
need
to
configure
it.
A
I
we
tried
three
times
so
now,
instead
of
having
tons
of
code
that
are
really
hard
to
test,
you
have
just
declarative
language
that
allows
you
to
to
specify
those
things
and
the
cluster
is
basically
the
destination
where,
when
you
query
this
board,
it
send
you
to
that
route.
So,
let's
start
envoy
now
it
starts
and
we're
gonna
run
up.
Sorry,
no
one,
and
now
you
see
that
if
you
query
and
voice
added
header,
so
it
rewrites
things
on
the
fly
and
that
was
making
a
really
powerful
proxy
because
you
don't
need
to
care
about.
A
You
know
doing
that
in
the
code
you
can
just
specify-
and
it's
separate
from
the
business
logic
and
so
now
I'm
querying
for
getting
a
500
code,
and
you
can
see
that
if
I
check
for
the
metrics,
you
see
that
all
that
it
all
those
observability
is
stored
in
Envoy
that
we
can
query
by
just
specifying
the
slash
tags.
So
if
you
see
here
I,
query
and
void
before
I
was
querying
here.
I
query:
the
HTTP
beam
directly,
so
I'm
talking
to
HTTP
been
through
and
void,
and
that's
what
allows
for
different
routing
condition.
A
So
now
that
we
see
invoice
hto
is
the
control
plane
that
cone
and
control
all
the
invoice?
And
there
is
this
application
called
bootcamp?
Oh,
and
you
can
see
that
for
every
version
it's
load
balancing
between
three
versions.
This
is
the
actually
the
schema
of
the
the
calls.
So
if
you
see
you
can
see
that
some
25%
of
the
traffic
is
going
to
those
routes,
but
let's
generate
some
more.
A
A
A
So
before
I
had
some
virtual
service
and
now
with
Sto
I,
define
more
virtual
service
that
spin
all
the
traffic
to
a
version.
So
now,
if
I
redo
the
test,
you
can
see
that
very
quickly.
All
the
traffic
is
going
to
get
pinned
to
the
version.
1
see
already,
you
see
0%
and
50%
here.
So
that
does
give
you
the
control
over
the
network,
but
with
great
power
comes
great
responsibility.
So
now
we
are
in
charge
of
the
network.
You
can
also
shoot
yourself
in
the
foot.
A
A
A
Json
is
doing
to
another
version,
so
now
you
can
specify
and
have
a
user
that
has
another
version
which
leave
you
very
much
in
control
of
what
you
want
to
test
into
the
system.
You
can
actually
reproduce
and
have
all
the
the
containers
in
the
same
cluster
with
different
versions.
So
basically
having
different
environment
is
completely
like.
You
have
to
rethink
the
concept
because
you
can
have
everything
in
communities
just
have
those
rules
saying
that
staging
and
and
production
are
are
different
depending
on
the
user
or
the
traffic.
A
Yes,
indeed,
but
so,
and
the
nice
thing
with
that,
is
it's
not
a
super
advanced
concept.
You
see
that
I'm
gonna
take
the
cookie
session
and
paste
it
into
the
JSON
web
token
page,
and
if
you
see-
and
you
can
see
that
here,
the
JSON
is
here,
so
you
can
very
easily
encrypt
and
decrypt
those
headers
with
GWT.
A
That's
what
unvoyage
support,
but
you
can
write
your
extension
in
envoi
and
create
your
own
authentication
mechanism,
which
is
not
what
I
recommend
by
the
way-
and
let's
say
that
now
you
are
testing
with
this
user
JSON
and
security
patch
comes
out,
so
you
need
to
migrate
your
user,
but
G's
only
still
pin
to
version
two.
So
what
you
can
do
is
sorry.
A
So
this
is
where
the
the
cows
engineering
part
is
a
little
bit
comes
into
play
and
why
a
service
mesh
is
a
great
tool
for
that,
because
it
comes
with
the
the
primitive
define
for
the
iv
injection.
So
let's
say
that
now
I
want
I,
want
Jason
to
have
a
seven
second
delay
for
every
of
his
requests.
I
know
it's
really
mean,
but
it's
for
the
purpose
of
this
demo.
A
Sacrifice
himself,
so
you
can
see
here,
I'm
not
able
to
to
refresh,
and
the
reason
is
that
there
is
a
timer
that's
going
to
get
triggered,
and
all
of
that
is
also
you
can
visualize
very
well
all
the
traffic
that's
going
through
and
you
see
here
there
is
an
error,
that's
appeared
and
you
can
see
that
the
developer
of
the
application
made
the
architecture
in
such
a
way
that
an
error
doesn't
crush
the
side.
So
you
get
an
error,
but
you
still
display
something
to
your
user,
which
is
a
completely
new
level
as
opposed
to.
A
Oh,
you
have
the
front
end.
You
have
the
back
end
now
you
see
the
whole
system
at
once
and
you
can
make
informed
decision
about
what
you
user
see
and
that
leads
to
a
lot
of
reliability,
because
now
your
engineers
can
go
and
figure
out
like
is
that
the
user
experience
that
you
want
to
give
to
user
as
soon
as
they
can
see
that
it's
easy
for
them
to
go
and
pinpoint
where?
Where
is
the
bottleneck
and.
A
A
This
is
something
that
you
have
controller
so
here
I
have
all
the
help,
and
if
you
look
at
the
stats,
that's
basically
all
the
metrics
that
are
stored
in
envoi,
so
you
can
make
some
really
interesting,
auto
scaling
metric,
because
now
you
know
how
many
requests
you
have
in
any
given
time
and
those
requests
are
stored
in
prometheus.
You
can
tell
kubernetes
to
say
scale
depending
on
how
many
requests
you
have.
That
is
the
completely
new,
because
those
things
are
really
hard
to
get
right.
A
Now,
it's
not
that
you
cannot
do
it
is
that
it
it
takes
a
lot
to
get
it
right.
There
is
a
lot
of
little
new
ones
that
you
need
to
fine-tune
and
discover
for
in
order
to
to
make
it
reliable,
and
the
last
thing
I'm
going
to
show
before
giving
the
hands
to
Sylva
is
that
when
you
ssh
into
them
the
box
by
itself-
and
you
want
to
see
what
and
voice
is,
do
a
actually
sto
is
doing
under
the
hood.
A
You
see
that
you
have
some
kubernetes
IP
tables
here,
but
envoi
use
the
nut
table,
and
that
is
a
monster.
So
that's
about
hundred
rules.
That
is
your
inject
and
you
know
it
depends
on
which
level
of
abstraction
you
want
to
work
with
which,
which
level
of
abstraction
you
want
to
create
chaos.
Usually
the
idea
is
to
create
chaos
in
a
lower,
lower
level
of
abstraction,
see
the
result
in
the
upper
layer
that
we
did.
We
create
a
timeout
into
the
system
and
we
see
the
result
in
the
application
layer.
A
So
that's
really
powerful
in
order
to
you
know,
break
down
the
system,
because
what
I
did
here
was
not
very
I
would
say
impressive
or
I
did
was
add
a
timeout
somewhere,
but
the
problem
is
that
in
the
code
most
of
the
libraries
that
we
use
have
a
default,
timeout
I'm,
pretty
sure
nobody
knows
what
they
are
and
those
you
know,
I
think
it's
a
github
that
two
months
ago
had
an
outage
because
of
seven
seven
second
delay
that
create
a
replication
of
MySQL
database
and
everything
blew
up.
So
those
things
seem
so
insignificant.
A
But
it's
like
the
butterfly
effect.
Everything
can
blow
up
from
there
and
it's
kind
of
it
might
seem
daunting
at
first,
but
little
by
little
you
can
easily
get
to
to
providing
a
better
service
to
your
customer.
So,
let's
zoom
out
a
little
and
also
if
you
get
started,
don't
don't
worry,
because
even
if
you
do
callus
engineering
doesn't
mean
that
from
now
on,
everything
is
going
to
be
perfect
and
you're.
Never
gonna
fail,
because
it's
not
true
and
I
can
show
you
this.
A
You
know
I,
don't
know
if
you
can
read,
but
basically
a
cue
engineer
walks
into
a
bar
order,
a
beer
order,
zero
beer
order,
nine
and
a
nine
beers
of
the
releaser
order.
One
beer
order,
a
random
string;
first,
real
customer
walks
in
and
ask
where
the
bathroom
is
the
bar
burst
into
flames
and
killing
everyone.
A
So
that's
that's
the
point
you
didn't
prove
like
work
on
that
case.
So,
of
course
something
unexpected
happened
and
that's
the
point.
You
have
something
that
you
don't
know
and
you
don't
have
do
you
don't
even
know
how
to
ask
the
question
about
what
you
don't
know.
That's
why
we
call
it
an
experiment.
You
try
out
and
figure
out
wow
what
happened
if
I
mess
up
one
of
the
up
a
table
of
Sto
with
something
like
we
still
work
or
what
happened
and
so
is
the
is
the
discovery
phase.
A
A
So
so
you
know
we
come
up
with
the
sli
SLO
and
SLA
and
in
basically
what
it
means
is.
You
can
state
your
experiment,
your
hypothesis,
that
you
want
to
test
and
those
things
translate
to
business
need
and-
and
your
SLA
is
basically
money
that
you're
going
to
give
back.
So
those
impact
the
business
as
well
and
I
hope
that
it
will
bring
a
new
way
of
our
organizational
change,
because
it's
not
about
technical
change.
A
B
B
Yeah
usually,
but
you
know
this
is
Linux,
so
yeah
I've
got
ten
fingers
yeah,
actually
you're
right.
Thank
you
very
much.
All
right
I've
got
the
don't
yet
so
I've
got
no
slides,
because
yesterday
too
many
slides
I've
got
the
terrible
job
of
making
it
boring.
Now
through
automation,
everything
Julian
has
done
we're
going
to
try
to
automate,
because
it's
no
point
in
trying
to
do
that.
B
If
you're
not
going
to
think
in
automation,
the
whole
point
of
care
centering
is
try
to
have
a
protocol
of
sort,
an
experimental
protocol
to
actually
make
sense
of
what
you're.
Seeing
if
you
do
everything
manually
this
a
lot
of
room
for
mistakes,
a
lot
of
room
for
all
that
primate
or
did
change.
Oh
you
know,
that's
sort
of
you
know.
What
do
you
do?
That's
time
and
you're
going
to
try
to
look
at
your
story
and
you
won't
make
sense
any
of
it.
B
B
So
so
the
idea
here
is
to
say,
like
I
said
we
have
an
experimental
protocol,
so
the
point
of
the
experimental
protocol
is
to
say:
we've
got
a
system
right,
our
system
that
looks
normal
for
whatever
normal
means
to
you
and
we're
going
to
decorate
or
change
some
condition
in
our
system,
usually
one
condition
only
because
of
why's.
You
won't
be
able
to
actually
make
sense
of
what
happened
so
we're
going
to
change
something
and
that's
what
we're
going
to
do,
and
then
we
run
the
the
hypothesis
you
made
initially
and
see
if
you've
changed.
B
If
it's
still
normal,
then
either
you
ask
your
own
questions,
so
you
actually
attempted
to
change
something
that
has
nothing
to
do
with
what
you're
trying
to
understand,
or
you
ask
the
right
question
and
you
learn
that
your
system
can
can
so
survive
or
cope
with
with
that
error
or
that
degradation.
If
the
system
has
deviated,
then
you
may
have
actually
found
a
weakness
in
your
system
and
then
that's
where
you
start
making
analysis.
So
that's
what
we're
trying
to
do
here
with
the
tool
called
chaos
toolkit
it's
a
declarative
format.
B
It
has
prongs
and
calls
cons.
The
pros
is
that
you
can
focus
really
on
the
domain
itself
or
calendaring.
The
currency
is,
you
know
it's.
It
is
limited
in
some
fashion.
You
really
people.
People
prefer
writing
code,
but
it
actually
has
worked
well
so
far.
So
we
carry
on
with
that.
So
it's
a
zania.
You
can
do
yeah
Mel.
If
that's
your
price
poison,
that's
exactly
the
same.
So
the
idea
here
is
to
declare
and
I'm
going
to
highlight
some
some
some
sections.
B
We
are
going
to
actually
hide
those
for
now
write
that
one
as
well.
So
here
we've
got
what
we
call
the
steady
state
hypothesis.
So
it's
a
bit
verbose,
but
basically
it's
your
baseline.
What
your
system
looks
like
when
it's
normal
and
you
make
the
decision
of
looking
at
specific
aspect
of
your
system
to
make
the
decision
it
looks
normal.
It
doesn't
mean
that
your
entire
system
needs
to
be
normal.
It
may
be
just
an
angle
of
your
system.
It's
really
it's!
B
It's
nice
to
actually
recoup
that
with
your
SLI
here
to
say
as
a
business.
This
is
what
matters
to
us,
but
it
can
also
be
very
technical
issue.
You
know
doing
something,
and
only
internally,
so
in
this
case
we
saw
earlier
that
when
we
introduced
a
delay.
Obviously
you
know
bad
things
happen.
As
a
business,
we
made
a
promise
to
our
users
that
they
will
get
the
the
the
product
page
and
the
reviews
in
one
under
one.
Second,
that's
what
we
tell
them.
We
are
performant.
So
what
happens
if
W
is?
B
Obviously
if
we
send
a
delay,
we
know
exactly
what's
going
to
happen,
but
that's
a
trivial
example
for
the
sake
of
the
demo
here.
Basically,
what
I'm
doing
in
this
example
is
simply
taking
what
Julian
has
showed
you
and
transforming
that
into
something
that
you
can
automate.
So
in
this
case
we
have
what
we
call
a
probe
that
simply
says
fetch
the
product
page
for
JSON
and
that
one
is
implemented
in
or
that
one
is
not
the
one
I
wanted
to
show
you
so
different
one.
B
Well,
that's
basically,
the
same
sorry
coming
back
there
you
go
so
here.
This
one
says
simply
calls
a
Python
function
yeah.
Why
are
we
using
Python?
Sometimes
we
can
use
simply
HTTP
if
you
know
you're
making
a
call
or
you
can
call
a
process
as
well.
But
if
you
want
something
a
bit,
you
know
more
and
more
more
evolved,
you're
going
to
call
a
Python
function
and
it's
really
just
a
function.
There
is
no
class,
there's
no
state,
it's
just
a
function.
B
You
implement
to
say
this
is
what
I'm
doing
to
actually
collect
data
here
and
you
can
have
as
many
as
you
need.
What
happens
is
if
that
is
false,
the
first
time
it
runs
it
base.
There
is
no
way
we
can
learn
from
our
system
if
it's
already
not
normal,
so
the
first
time
it
runs,
it
either
passes
and
goes
on
or
it
fails
and
simply
say,
I
can't
I
can't
learn
anything
from
that
all
right.
So
once
you've
done
that
we
go
to
the
method
which
is
basically
forget
about
that.
B
For
now
we
have
another
set
of
activity,
which
is
an
action
I'm,
sorry
I'm
pretty
good
with
laptop,
and
this
is
what
what
basically
here,
we're
doing
exactly
what
Julian
has
done.
It's
just
written
in
Python.
In
this
case,
we
call
the
API
of
ECL,
but
that's
exactly
the
same.
We
introducing
in
the
delay
for
chasing
a
5-second
in
this
case
and
that's
basically
it
on
a
rollback.
We
simply
remove
that
rule
that
we
created
rollback
is
a
terrible
term.
Here
we
should
have
called
that
remediation
or
something
else,
because
it
makes
a
promise.
B
You
won't
be
able
to
cope
with.
If
your
system
one
went
away,
we're
not
going
to
roll
it
back.
What
we
do
is
we
try
to
n
do
what
we
started
to
do
so
be
carefully.
A
rollback
is
not
like
a
crazy
tool.
You
can
do
a
lot
of
things
with
all
right.
So
what
does
it
look
like
when
we
actually
run
that
all
right,
so
the
kale,
the
the
command,
is
called
chaos?
That's,
that's
that's
pretty
explicit
I!
B
Guess
we're
going
to
do
that
and
we
just
run
it,
and
hopefully
it
will
actually
go
through
rot
on
the
top
right.
You
see
that
we
send
a
message
to
snack,
that's
good
for
visibility,
and
it
runs
that
fine.
We
saw
you
know
incrementally
and
basically
sequentially,
so
everything
you
saw
in
the
fan
is
what
is
run
there,
there's
nothing
new
here.
What
is
interesting
here
is,
it
does
tell
you
at
the
end,
so
the
first
thing
it
says,
or
what
the
system
looked
normal,
so
indeed
we
could
actually
return
under
one.
B
Second,
then
we
introduce
that.
That's
that
error
here,
which
one
is
it
inject
fault
this
one,
and
we
see
that
when
we
run
again
the
the
baseline
now
it
fails.
So
obviously
we
knew
that
would
happened,
but
that
tells
you
something
that
when
that
condition
happens,
you
know
this
is
how
your
system
is
impacted.
Now
we've
got
the
same.
B
Second,
you
know
the
same
error
with
instead
of
sending
a
creating
a
delay,
we're
going
to
actually
raise
it
return,
an
HTTP
error
and
we'll
see
a
difference
here,
because
what
we
say
is
we
still
need
the
response
under
one.
Second,
we're
not
introducing
a
delay
here
we
introducing
a14
into
one
of
the
sub
service.
In
this
case,
we
show
that
our
system
can
cope
with
that
because,
like
Julian's
showed
the
way
it's
been
built
means
that
you
always
get
a
response
as
a
user,
it's
just
you
going
to
get
less
data
shown.
B
You
won't
see
the
reviews,
but
you
still
get
the
page
and
you
still
get
that
under
one
second.
So
that's
interesting
because
here
with
we
know
now
we
don't
have
a
hunch.
We
know
that
we
can
cope
from
a
user
perspective.
We
can
cope
with
at
least
the
review,
so
is
sending
back
a
for,
for
that
was
okayed.
I.
Think
I
was
sitting
up
here.
B
So
we
learned
that
our
system
can
sort
of
cope
to
a
certain
degree
and
we
can
go
back
to
the
business
and
say:
is
that
okay
with
you,
if
your
users
get
part
of
the
results
they
expected?
If
that's
okay,
then
we're
good?
With
that
we
know
we
can
sustain
that
error
and
that's
basically,
how
you
you,
you
automate
with
a
kill
circuit.
It's
exactly
all
the
way,
all
the
time
that
way,
so
you,
you
simply
define
your
state.
B
B
Right
so
at
this
stage
we
good
we
know
you
know,
we've
got
something
on
a
machine,
but
it'd
be
nice.
If
we
could,
you
know
communicate
that
at
one
work.
That's
not
the
one
I
want
yeah,
so
we
can
create
reports
here.
Pdf
reports
or
other
sort
of
reports
happened
in
that
J,
but
when
we
do
that,
I
don't
know
if
it's
going
to
open
the
right
window,
so
they're
with
me
yeah.
B
Right
so
in
the
in
the
in
the
specification
of
the
the
experiment,
what
you
can
add
as
well,
it's
just
a
blog
saying
what
do
I
contribute
with
this
experiment
and
what
do
I
contribute
to.
So
those
are
things
that
you
care
about
as
a
company
as
as
a
team,
so
it
could
be
reliability.
Things
like
that
and
you
simply,
you
know,
create
try
to
zoom
again
you're,
simply
saying
I'm
interested
in
those
reliability,
performance
and
and-
and
you
know
other
things
and
you
give
them
away.
B
That's
interesting,
because
once
you
start
having
a
lot
of
experiments,
you
can
have
through
those
punch
cards
as
through
those
reports,
a
fairly
quick
understanding
of
where
you
put
in
your
effort
on.
So
if
you
define
that
for
ample
security
is
important
and
you
see
very
little
done,
you
can
start
going
back
to
say
well,
we
need
to
put
more
effort
into
that
sort
of
you
know
contribution
here
when
we're
doing
a
lot
on
reliability,
so
we're
doing
a
lot
of
chaos
and
experiments
on
that.
B
But
we
don't
do
enough
Onix
on
security,
that
sort
of
thing
and
after
that
is
basically
the
logs
of
your
experiments.
So
it
shows
you
when
it
started.
You
know
it's
failure
and
it
shows
you
if
you
collect
it.
You
know
data
as
you
run,
so
you
can
see
from
poor
in
this
case
that
we
collected
the
state
of
the
history
of
the
service
itself.
So
that's
basically
humanity's
you
know
resource
here
and
you
can
see
whether
or
not
it
worked.
B
So
in
this
case
we
have
no
fault
injected,
then
we
inject
the
fault
and
we
come
back
and
we
have
that
dealer
fault.
We
were
talking
about
it's
just
a
nice
way
when
you
I've
got
that
report
and
you
send
it
out.
People
can
figure
out.
What
did
you
do?
You
know
and
start
making
sense
and
start
the
analysis,
which
is
the
human
side
that
is
interesting
here
and
before
I.
Leave
you
to
it
with
that,
once
you've
got
that
there
is
a
one.
B
B
So
what
what
we
have
is
a
chaos
platform,
which
is
you
know
the
big
sister
of
of
chaos
toolkit
if
we
will-
and
it's
that's
one-
it
allows
you
to
simply
it
has
it's
a
it's
a
control
plane
with
an
API
for
calcium
showing
so
what
it
gives
you
it's
it's
it's
a
way
to
upload
your
experiments
and
share
that
with
your,
you
know
your
team's
organization.
However,
you
want
to
see
it
and
then
schedule
them
so
from
Paul.
B
If
I'm
running
that
here,
what
I'm
saying
is
I'm
going
to
those
are
IDs
of
you
ski,
but
they
are
exactly
that.
The
experiments
I
showed
you
I've,
uploaded
them
and
they're
going
to
be
run
that
way,
so
here
I'm
saying
you're
going
to
run
that
experiment
every
minute
of
every
day
of
every
week
or
every
year.
You
know
but
you're
going
to
do
that
three
times.
B
B
That's
a
ready,
skew,
that's
what
I'm
using
to
schedule
so
I'm,
putting
the
schedule
into
a
queue
and
let's
get
distributed
between
those
workers,
that's
all
it
does,
and
the
experiment
is
going
to
be
run
within
one
of
those
workers
which
can
be
distributed
across
your
you
know
your
system.
In
this
case
it
will
have
to
wait
because
because
I
say
every
minute,
it
needs
to
need
to
go
back
to
20
seconds.
B
Please
bear
with
me,
but
the
idea
here
is
it's
going
to
show
you
that,
as
you
want
and
going
to
run,
that
and
report
back
the
results
to
the
platform,
and
obviously,
after
that
you've
got
a
VI
to
art
weeks.
You
express
all
those
results
with
and
share
those
results.
So
it's
really
what
you
want,
if
you
do
think,
is
an
isolation.
Only
you
you're
talking
to
yourself
ridiculous
engineering
is
a
collaborative
thing.
You
want
to
do
with
everyone
else
right.
So
that's!
B
What
we
see
here
on
the
right
is
a
log
of
the
kill
toolkit
that
we
didn't
serial
because
it
was
on
the
debug
mode,
but
that's
exactly
what
happens.
So
it's
simply
running
that
for
you
now
and
you
can
leave
it
it's
going
to
do
that
continuously
and
that's
what
you
want
to
make
sure
that
it
always
works
that
way.
So
obviously
you
might
think.
Oh,
it's
a
bit
like
testing.
B
It's
really,
not
in
a
sense
where,
with
couch'
engineering,
you
asking
a
question
about
something
you
don't
know
and
you're
trying
to
explore
your
system
right,
you're,
not
making
an
assumption
necessarily
and
that's
it
really
once
you've
got
that
and
for
those
who
were
here
yesterday
and
we
talked
about
the
idea
of
sharing
on
knowledge.
So
if
you
want
here
basically
the
idea
is
let's
say:
I've
got
that
experiments
on
East
you
and
I've
become
a
master
of
East.
You
you're,
not
it
be!
It's
been!
B
So
cool,
if
I
could
actually
upload
that
somewhere
and
you
could
just
take
it
and
run
it
because
it
means
my
knowledge
of
SEO,
of
its
failure
of
its
you
know.
Weak
points
could
actually
become
something
you
can
action
and
run
for
yourself.
So
that's
something
you
want
to
do
after
that.
Once
you've
done,
you
know
the
whole
thing
you
want
to
share
those
your
source
experiences,
those
findings
to
actually
contribute
back
to
the
world
or
at
least
within
your
organization.
B
You're.
Not
you
know
keen
on
the
world
itself
right,
I!
Think!
That's!
That's
enough
for
me!
I!
You
know.
If
you
want
to
know
more
about
the
kill,
circuit
and
kills
prom
dress,
come
talk
to
me
or
come
on
or
slack,
but
basically
that's
the
idea.
We
start
with
what
you
done
express
all
the
problem.
You
know
that
we
need
from
our
platform
that
off
and
that
all
platform
have,
you
know,
become
more
and
more
abstracted
more
and
more
complex.