►
From YouTube: Chaos Engineering WG Deep Dive – Sylvain Hellegouarch, ChaosIQ (Intermediate Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Chaos Engineering WG Deep Dive – Sylvain Hellegouarch, ChaosIQ (Intermediate Skill Level)
This session is dedicated to using the Open Source Chaos Toolkit project, as a unified driver, to explore your system weaknesses through Chaos Engineering.The talk will walk you through a practical scenario of an application living a Kubernetes cluster, and how using the Chaos Engineering experimental approach, using various chaos tools driven by the Chaos Toolkit, you can learn about your system in order to improve it.
About Sylvain
Sylvain Hellegouarch is the CTO and lead engineer at ChaosIQ. He has been passionnate about Open Source for as long as he can remember working on various successful Python projects (CherryPy, WebSocket 4 Python and more recently, the Chaos Toolkit). He has participated to IETF working groups such as the WebSocket (RFC 6455) and Atom Publishing protocols (RFC 4287). He has worked on companies of various size and loves a great team to learn from.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right,
hello,
everyone,
sorry
to
interrupt
you,
while
you
tweeting
or
something
I
promise.
I
won't
do
that
like
more
than
half
an
hour,
so
you
can
come
back
to
it,
yeah
exactly
to
change
your
password,
so
who
has
actually
come
to
the
other
tool.
Conclusion
hang
on
on
Wednesday,
just
to
aware:
okay,
good,
so
I'll
try
to
not
rephrase
everything
I
said
would
be
a
bit.
You
know,
use
less,
but
I'll
have
to
rehash
a
few
things
we
saw
the
other
day.
A
So
today
we're
going-
and
my
name
is
Silvia
and
egg
wash
I'm
only
saying
my
name,
because
it's
not
easy
to
pronounce,
not
sure
so
just
you
know
call
me
Silva
that
would
be
here.
Yep,
don't
go!
Go
away!
We're
going
to
talk
today
about
calendaring
with
the
kill
toolkit,
so
I'll
focus
the
first.
You
know
part
of
the
discussion
on
chaos
enjoying
in
general.
Why?
A
A
The
subtitle
of
this
talk
is
for
active
reliability,
validation
because
we're
trying
to
remain
reliable,
I've,
reliable
system.
You
know
in
our
life
right.
That's
that
the
goal
resiliency
is
part
of
it,
availability,
safety
and
all
of
those
things,
security,
but
really
what
we
want
is
reliability,
because
our
system
system,
Eve's
reliable,
means
that
our
users
trust
us
and
will
use
our
system
I'm,
not
sure
the
password
thing
is
going
to
impact
Twitter
or
not,
but
it's
actually
not
a
good
thing
right,
a
quick
reminder,
so
Kent
calendaring
starts
with
our
complex
system.
A
I
bet
for
those
who
have
been
in
the
game
for
a
long
time.
You
know
years
ago
you
were
able
to
actually
go
to
the
mainframe
of
the
machine,
whatever
install
your
system
and
your
system
was
basically
one
binary
and
a
few
rules
on
Linux
I'm
singing
like
that
today.
It's
not
that
at
all,
your
machines
are
so
far
away,
even
even
if
it's
not
virtualized,
even
if
it's
at
your,
you
have
your
data
center.
A
Only
by
collecting
data
you'll
get
the
full
picture
of
what's
happening
at
least
a
partial
one,
not
just
based
on
intuitions
and
angeas,
but
real
data
practicing
edges
edge
cases
in
control
manner
is
the
goal
of
chaos,
engineering
right:
it's
not
going
and
breaking
stuff
randomly
without
people.
Knowing
this
is
not
it.
If
you
do
that,
your
team
is
going
to
be
pissed
off,
you
will
never
learn
anything.
So
it's
not
the
point.
Breaking
stuff
is
one
thing
and
you
don't
really
have
to
actually
break
anything.
A
If
you
want
to
learn
from
chaos
engineering
anyway,
it's
just
one
one
aspect
of
it,
but
you
need
to
do
that
in
a
controlled
manner.
That
means
you
tell
your
colleagues
you're
going
to
control
a
customer
and
experiment
that
will
break
something
or
that
will
tell
them
something
or
change
the
system
in
some
manner,
so
that
your
your
team
mates,
first
of
all
on
surprised
and
pissed
off
at
you
and
second
of
all,
they
will
learn
because
they'll
see
that
didn't
catch
something
in
the
first
place.
A
Well,
it's
not
a
rock
star,
I'm
I,
don't
know
in
teams.
Sometimes
we
we
get
caught
up
in.
You
know
the
rocks
attitude
of.
We
need
that
person.
No,
it's
not
that
okay
Angela
needs
to
be.
You
know
quite
far,
it's
it's
not
it's!
Not
the
person
who
is
going
to
save
the
day,
but
it's
going
to
help
the
team
save
the
day
so
be
open-minded.
Try
to
not
forget
your
experience,
but
remove
the
prejudice.
You
have
will
do
that.
You
know
we
have
egos.
We
have
you
know
experience
and
we
come
and
said
man.
A
You
don't
know
what
you're
doing.
If
you
approach
this
this
way,
people
will
not
learn
because
it
should
shut
down
immediately.
So
don't
do
that.
Nurture
empathy.
Empathy
is
really
really
at
the
core
of,
of
course,
engineering.
If
you,
if
you
have
empathy
for
the
system
for
your
users,
for
your
teammates
they're,
trying
all
to
do
right
thing,
your
teammates
are
not
trying
to
do
the
bad
thing,
but
sometimes
a
struggle
so
you're
here
to
actually
help
them
get
there.
A
Yeah,
don't
blame
it
or
be
snarky
that
never
helps
anyone,
love
an
experimental
approach
and
that's
a
third
really
characteristic
I'm
not
going
to
list
many
of
them.
Two
or
three
don't
be
scared
to
of
not
doing
the
outcome
again.
Allow
yourself
not
knowing
it's
probably
not
something
we
do
very
well
in
our
industry.
We
always
think
we
know
what's
going
to
happen
or
the
nullity
response.
Well
forget
about
that.
Try
to
accept
that
you
don't
know
and
you're
going
to
try
something
to
learn
from
it.
That's
really
the
fun
of
the
game.
A
Actually,
if
you
allowed
yourself
to
learn
you
it's
going
to
be
much
more
fun
so
that
those
three
things
to
me
are
the
core
of
a
good
chaos
engineer.
If
you
have
open
minded,
if
you
are
in
the
empathy
of
your
teammates
users-
and
you
know
everything
surrounding
you
and
and
basically
and
you
love
the
experimental
approach,
you're
going
to
have
a
fun
as
a
chaos,
Andrea
and
your
teammates,
will
you
know
really?
Thank
you.
If
you
forget
these
people
will
see
you
as
a
the
only
annoying
guy.
A
You
know,
I
mean
in
person
the
one
who's
breaking
stuff,
all
the
time
that
really
doesn't
help
anyone.
So
try
to
be.
You
know
in
those
positive
mindset,
if
you
will
so
now.
Let's
talk
about
the
the
chaos
engineering,
workflow
again,
I
know
I'm
going
very
fast,
so
don't
hesitate
to
come
back
after
you
know
at
the
end
and
come
and
just
have
the
discussion,
so
it's
very
simplified
I'm
terrible
at
diagrams,
so
I
drew
what
I
could,
but
that
that's
the
gist
of
it.
We
need
an
iPod
thesis
right.
A
If
we
don't
have
a
baseline,
we
can't
learn
anything
because
we
don't
see
a
deviation
from
the
baseline,
so
we
need
that
baseline
now,
once
we
agree
that
we
have
that
baseline
and
it's
there,
because
there
is
no
point
in
in
trench
learning
from
a
system
that
is
already
broken
you're,
not
helping
yourself
you're
going
to
apply
method.
Now
it
sounds
very
scientific.
It's
not
really
a
scientific,
but
we
like
the
idea
of
the
philosophy
of
the
scientific
approach.
So
the
method
here
is
just
a
bunch
of
actions,
you're
going
to
run
against
your
system.
A
Now,
if,
at
the
end
you
replay
the
hypothesis
and
the
steady-state
is
not
normal,
that
means
your
system
has
changed
based
on
the
conditions
you've
imposed.
Then
you
probably
have
discovered
something
either.
Your
steady
state
as
by
processes
is
wrong.
Good.
You
asked
on
one
question
or
you
has
the
right
question,
but
you
discovered
something:
that's
something
now
needs
to
be
discussed
within
the
team.
Is
it
a
real
weakness?
It's
something
the
business
should
care
for.
A
Maybe
you
know
having
two
passwords
for
Twitter
in
the
in
log
files:
it's
not
actually
a
business
problem
or
is
it
now?
That's
the
kind
of
question
you
would
ask
once
you
have
discovered
the
weakness,
you
just
don't
go,
run
and
say:
I
find
a
weakness.
No,
usually
I.
Wouldn't
let
anyone
you
prioritize
your
order,
your
weaknesses,
some
of
them
you
just
you,
will
agree
that
it's
an
old
system
that
is
going
to
be
replaced
because
we're
working
on
a
different
one.
A
Now
so
don't
forget,
you
know,
don't
worry
about
it
now,
if
it's
a
security
breach,
probably
you
you
want
to
pay
more
attention.
So
basically
you
do
that
in
a
loop
continuously
and
that's
an
important
one.
You
you
continuously
do
that
and
that's
where
it
made
a
difference
for
it.
If
it's
a
different,
it's
different,
sorry
from
disaster
recovery,
which
usually
digital
recovery,
is
done
once
every
blue
moon
and
basically
it's
done
whenever
it
can
be
done.
It's
heavy
and
it's
it's
very
imposing
on
the
team
and
the
older
structure.
A
Calendaring
is
more,
you
know
it's
a
flow
ready.
You
do
that,
like
you,
do
testing,
which
I
hope
you
do,
and
if
you
do
this
thing,
then
you
can
move
to
calendaring
otherwise
start
with
testing
and
Cal
State
engineering
is
fairly
different
from
testing
testing.
You
know
where
you're
going
it's
a
fairly
binary
thing.
Is
it
there
is
not
those
enjoying
it's
an
open
question,
you're
trying
to
learn
something
you.
A
You
forgot
your
prejudice
right,
so
the
mindset
is
different,
even
if
the
tooling
will
probably
look
fairly
the
same
to
you,
studies
that
so,
let's
come
back
to
that.
It's
a
normal
it
in
your
system.
What
your
system
is
when
it's
normal
for
ample.
It's
it's
a
you
know
made-up
city-state
here,
but
customers
spent
eight
X
amount
of
dollars,
so
you
know
euros
or
whatever
at
8
a.m.
8
a.m.
on
average.
Now
that
we
think
the
business
cares
for
it
doesn't.
No
one
cares
for
your
service
going
down
your
resiliency.
All
of
that.
A
What
the
business
guess
is
that
this
remains
true.
Whatever
you
roll
out
whatever
you
do,
you
don't
want
to
change
that
at
least
not
in
a
negative
way.
If
you
do,
then
you've
learned
that
something
went
wrong.
So
that's
where
you
start,
but
let's
all
back,
do
you
can
you
make?
Because
we
now
have
a
normal
position.
We
have
a
baseline,
but
can
we
make
an
ID,
but
this
it's
not
actually
easy.
You
don't
know
you
don't
know
what
you
don't
know
right.
A
So
if
you
first
ask
questions,
sometimes
you
have
an
intuition
that
this
thing
will
actually
impact
the
system
in
some
fashion.
Right,
so
that's
the
question.
You
ask
as
a
team.
What
could
go
wrong?
That
would
impact
that
baseline
and
even
before
that
you
may
not
even
have
a
question
so
collect
data,
I've
matrix
and
start
analyzing
your
matrix
and
make
sense
of
them
and
and
figure
out
things
you
don't
know.
A
You
probably
have
dashboards
fantastic
dashboards
that
you
may
or
may
not
look
on
a
daily
basis,
but
I'm
sure
you
look
at
them
and
you're
like
they.
Don't
actually
tell
me
what's
up
with
my
system,
I
will
know
when
it
happens,
I
will
probably
see
that
the
response
time
is
going
down
when
it
goes
down,
but
it's
too
late
at
that
time.
So,
even
if
you
have
dashboards
sometimes
they
don't
actually
tell
you
in
advance
what
you're,
looking
for
and
by
practicing
chaos
engineering.
A
You
force
that
to
happen
before
it
happens,
in
real
life
with
your
users
and
you
learn
and
you
fix
you
know
it's
positive.
So
your
hypothesis,
it's
a
statement
that
should
be
proved
or
disproved
usually
to
detect
a
weakness
in
a
stem,
but
also
you
could
use
it
to
predict
an
outcome.
So
a
fine
Apple.
Let's
say
we
have
you
know
services.
You
know
that
pay
that
manage
the
payments
and
I'm
using
that
based
on
on
Mon,
so
the
the
other
end,
the
keynote
which
was
fantastic.
A
So
let's
say
you
have
like
10
services
and
you're
going
to
do
a
rollout
and
that
we
did
not
roll
out
you.
You
say
I'm
going
to
kill
five
of
them,
which
will
be
replaced
and
I'll
keep
five
ordeals
and
I'll.
Do
it
if
roll
out
like
that,
so
does
this
decrease
the
average
spending
and
that's
a
question
you
want
to
prove
or
disprove
its.
It
doesn't
have
to
be
business
oriented,
it's
obviously
more
interesting
and
more
forceful,
and
you
can
go
back
to
your.
A
You
know
managers
because
they
understand
that
not
necessarily
your
matrix,
but
you
know
because
chaos
engine
can
be
at
various
level.
You
can
also
have
obviously
very
engineering
hypotheses
that
you
keep
for
yourself
or
your
team,
the
experimental
mater.
Now
that
you
know
you're
looking
for
you
know
where
you're
starting,
you
know
what
the
question
you
have.
A
What
are
you
going
to
change
to
actually
make
that
happen
to
see
if
you
can
prove
out
this
proof
that
you're
going
to
vary
things
so
basically
you're
going
to
perform
a
route
rule
out
or
you're
going
to
kill,
pods
or
whatever
you
know,
is
necessary
to
sort
of
prove
this
proof,
or
at
least
getting
some
knowledge
of
your
of
your
system.
Sometimes
you'll
have
to
do
that
many
times,
because
what
you
very
doesn't
actually
impact
the
thing
you're
trying
to
prove
now
it
doesn't
mean
that
it's
safe.
A
It
means
maybe
that
you're
not
changing
the
right
thing
scale
down
the
payment
service
by
Aleph
and
query
Prometheus
for
number
of
payment
endowment
during
that
period.
That
would
be
an
experimental
method
to
actually
see
if
that
proves
or
not
the
system
is,
is
sane,
but
don't
worry
too
many
things
because
you'll
make
you
won't
be
able
to
make
sense
of
any
of
it.
So
I
have
discrete
experiments
as
best
as
you
can.
A
In
summary,
a
chaos
engine
demonstrates
great
empathy
again
very
important
for
users
teams
and
the
system
and
the
system
I'm
I'm
inclined
ear
to
insist
on
the
system.
You
may
come
at
on
the
legacy
system
that
you
hate.
Do
you
dislike,
because
it's
in
your
mind
of
our
good
engineers,
you
think
that
it
should
be
done
this
way
or
this
way
that's
not
a
problem
here.
You
have
a
system
in
front
of
you
and
you
need
to
make
sense
so
it
and
by
observing
system.
A
We
saw
that
the
probing
is
important
here,
because
you
collect
data
as
you
go,
you
may
have
monitoring,
but
for
the
purpose
of
the
experiment,
you
may
also
collect
data
on
the
side
that
are
not
necessarily
based
on
your
monitoring
and
you
look
if
the
system
deviated
and
if
it
did
again
you
may
have
starting.
You
know
you
may
start
the
analysis
with
your
teammate.
So
now
that
was
a
very,
very,
very
short
and
and
all-too-brief
you
know
overview
of
calendaring
and
how
you
could
develop.
A
That
mindset
again
come
back
to
me
after
that,
and
we
can
chitchat
on
this
and
now
we're
going
to
see
one
tool
that
could
actually
help
you
on
doing
that.
It's
called
here's
toolkit,
it's
an
open
source
one
and
it
implements
a
flow
that
we've
just
described.
So
what
you
do
is
you
describe
in
that
in
a
coastal
kit
in
experiment?
You
have
your
steady
state,
the
hypothesis
is
the
method
and
that
just
replace
it.
A
It
was
born
in
recently.
So
it's
a
fairly
small
project.
Really,
you
know
dynamic
in
the
sense
where
things
are
happening,
but
it's
not
well
established
yet,
but
he
accepts
contributors,
it
has
nice
community.
We
value
very
much
the
friendliness
of
the
community,
it
may
be
small,
but
we
we
all
very.
We
try
to
be
friendly
together.
A
You
know
we
don't
want
to
be
snarky
at
each
other
saying
you
know
you
should
be
doing
this
way
or
in
that
way
or
thing
like
that,
if
a
user
arriving
on
the
project
fails
to
understand
something,
it's
because
the
documentation
is
wrong
or
something
is
not
well
explained.
So
that's
that's
where
we're
starting
from
and
it's
implement
in
Python
yay
I'm,
sorry
for
all
the
go
along
users,
but
you
know
you
know
you
got
a
succeed
so.
A
The
core
objectives
are
the
toolkits.
We
try
to
be
the
entry
point
of
the
case,
engineering
experiment.
What
that
means
is
really
when
we
started
it.
We
realized
there
are
many
tools
out
there,
but
aren't
easy.
If
you
tried
chaos,
monkey
just
even
setting
up
requires
quite
a
bit
right,
that's
something
you
can
grab
and
just
run.
So
if
you
haven't
got
the
time
the
mindset,
then
it's
it's
quite
involved
because
toolkit
tries
to
be
the
opposite.
Try
it
awfully
to
be
as
simple
as
you
can,
but
still
learn
quickly
from
it.
A
So
simplicity
is
a
key
for
durability
in
software.
In
my
opinion,
if
your
simple,
then
things
can
change
fairly,
you
know.
Naturally,
if
you
become
complicated
or
complex,
then
your
software
is
doomed
to
have
many
problems
on
the
road
we
interface
with
platforms
and
system
natively.
So
what
that
means
is
the
are
plenty
of
tools
like
I
said
doing
kills.
You
know
injecting
force
in
your
system
early
like
the
chaos
monkey,
gremlin
Pumbaa,
all
power
seals,
many
tools,
but
so
we
thought
well.
We
we're
not
going
to
rewrite
that.
A
That
would
be
foolish
we're
going
to
drive
them,
because
that
really
what
matters
is
simplicity
of
getting
them
started
so
and
we
quickly
realized
you
can
do
a
lot
already
by
just
talking
to
actually
the
system
terminating
a
part.
You
don't
have
to
go
and
do
fancy
things
to
actually
terminate
a
pod.
You
call
terminate
put
the
delete,
you
know,
function,
delete
method
on
the
pod
at
the
HTTP
level
and
you're
done
you've
a
treat
them
into
the
pod.
So
there's
not
much.
A
You
know
you
need
to
do
and
fancy
you
know
engineering
here,
so
we
prefer
extending
as
much
as
we
can.
We
are
declarative
and
I'll
show
that
later
on,
that
means
you
don't
necessary,
have
to
actually
code
anything.
You
can
just
declare
your
ex
a
bit
like
some
tries
to
do
Domitian,
first,
less
student,
so
it's
a
zero
for
now.
You
won't
have
any
UI,
you
don't
have
to
click
buttons.
What
you
have
is
you
describe
that
in
a
your
experiment
somewhere
in
fine
github,
so
you
can
share
that.
A
You
know
review
blah
blah
blah
and
then
you
can
run
that
from
any
CI.
You
want,
or
anywhere
basically
and
like
I,
said
before
any
project
community,
so
roughly
because
again,
I'm
not
very
good
at
diagrams,
but
this
is
where
we
stand.
The
chaos
Orkut
is
in
front
of
all
those
systems
or
all
those
providers,
if
you
will-
and
we
just
use
them
to
you-
know
to
inject
some
problems
in
your
system.
So
there
are
others
right
there
and
the
plugins
are
merely
you
know
extending
the
toolkit
for
new
functions
Wow.
A
A
You
know
we
will
like
a
spec
and
we're
going
to
try
to
do
that
before
we
could
anything
and
we're
going
to
implement
the
spec
first
write
the
spec
first
and
implement
that,
after
that,
the
spec
is
is
declared
in
using
JSON,
but
we
tolerate
ml
people
now
I'm
kidding.
Basically
it's
a
you
can
declare
in
any
of
those.
It
doesn't
really
matter
to
the
toolkit
itself,
just
load
that
into
it.
You
know
a
dictionary
of
some
sort.
A
Drivers,
like
I,
said
we
drive
API,
so
I
went
for
that.
So
usually,
drivers
are
implemented
in
their
own
repository
projects,
as
in
in
Python
again
like
like
I,
said
later
on
we're
trying
to
work
on
whether
or
not
you
can
actually
call
like
a
go
or
rest
or
whatever
you
want
so
call
call
them
from
Python.
Instead
of
you
know
just
bison,
because
we
understand
that
communities
are
in
various.
Oh,
you
can
even
call
processes.
So
sometimes,
instead
of
writing
code,
you
have
a
command
line
like
we
use
Vegeta
for
load,
testing.
A
A
We
don't
try
to
be
clever
with
the
with
with
the
extensions
when
I
say
that
is
we
don't
try
to
create
too
much
intelligence
in
one
extension,
we
give
the
tools
and
you
do
as
you
please
with
them.
We
drivers
like
you
so
for
them.
Plugins
such
as
you
know,
adding
you
know,
functionalities
or
for
notification,
and
things
like
that,
not
very
interesting.
A
A
We
are
going
to
use
we're
going
to
run
a
demo
against
communities
yeah,
of
course,
and
we're
going
to
use
communities
on
gke
could
work
on
me
if
it's
just
calling
cabinet
alum
communities
at
that
stage
there
will
be
the
most
specific
to
GK
if
I
have
time
so
we'll
see
that
on
the
right
on
right
hand,
side
here,
you've
got
we've,
we've
scoped,
that's
one
of
the
product
of
weave,
so
I,
don't
know
if
you're
familiar
with
that
product.
But
we've
scope
is
very
neat.
A
If
you
need
to
see
your
system,
because
it
shows
you
real
time,
the
topology
of
your
system,
your
kubernetes
perd
services.
So
it's
really
helpful
when
you
do
something
like
that
to
actually
have
a
view
of
what's
going
on
in
the
system
and
that's
just
the
default,
you
know
a
lot
dashboard
for
for
primitives.
Now.
A
A
In
in
enamel,
so
we
we're
going
I'm
going
to
go
through
the
the
that
spec
quickly,
so
that
you
understand
what
we
do:
I'm,
sorry
for
those
who
hate
DSL,
but
it's
either
to
your
code
in
one
language
in
language
and
makes
much
easier
with
ESL
to
have
extensions
from
various
languages.
So
hopefully
that
don't
put
you
put
you
off,
so
we
have
piece
of
metadata
not
interesting
way.
A
Here.
We've
we've
got
the
configuration
and
secrets
now
secret,
sound,
very
secrets
when
there
are
like
that
right,
but
that's
a
demo
and
that's
demo
as
a
password,
so
that
you
know
it.
But
again,
that's
a
demo
and
I
didn't
want
to
actually
build
anything.
But
the
idea
here
is,
you
have
configuration
secret
secrets
could
be
loaded
from
the
environment
or
volt,
and
if
the
community
wants
to
contribute
something
like
a
mess
that
will
come
as
well.
A
So
the
idea
is,
you
describe
your
secrets
and
and
because
again
Carol
kid
always
trying
to
reuse
what
already
exists.
We
don't
have
a
secret
management
system.
I
would
be
foolish
so
useful
to
use
something
else
or
environment.
That's
fine
but
we're
just
saying
what
what
you
can
load
a
configuration
are
not
secrets
that
it's
similar
to
communities
right
at
this
stage.
Now
we
have
the
Osterley
state,
so
we
have
a
title
and
then
we
have
those
probes.
A
Probes
are
just
predicting
data,
but
at
the
level
of-
and
this
one
is
not
necessarily
very
powerful
to
in
that
case,
but
anyway
we
have
what
we
call
tolerance,
a
steady-state
with
bail.
When
we
start
the
experiment
will
bail
immediately
if
the
system
is
not
normal.
There's
no
point
like
I,
said
to
even
run
this
thing.
If
it
doesn't
look
normal,
so
that's
a
gatekeeper.
If
you
will
and
basically
here
what
we
say
is
we
using
a
different
kind
of
provider?
It's
just
such
TTP.
A
It's
going
to
run
that
and
see
if
that
you
know
comes
back
with
a
200,
but
you
could
have
in
torrance
a
reg
X
JSON
path.
Things
like
that,
so
you
can
go
pretty
wide
with
it
blah
blah
blah.
So
we're
going
to
them.
You
know
one
by
one
and
then
here
it's
an
interesting
one,
because
it's
a
different
one.
It's
it's
a
Python
one.
A
So
this
one
says
I'm
going
to
use
that
module
and
I'm
going
to
use
that
function
that
we
just
you
know
that
is
implementing
that
module
I'm
going
to
use
those
arguments.
So
we
pick
up
that
function
here
through
the
label
selector
and
we're
going
to
say
we
want
one
only
if
there
are
two
and
no
system
in
that
experiment
is
not
normal.
We
wouldn't
learn
the
right
thing.
If
that
was
a
case
so
more
see,
we
have
a
JSON
pass
here,
so
you
can
do
something.
A
You
know
more
more
on
the
the
payload
that
is
returned
right
and
the
action
is
more
interesting
here.
What
we're
going
to
use
here
in
that
experiment?
The
idea
is
to
say,
I
want
to
see
if
open
files
automatically
scale
up
my
scale
out
my
my
function,
I
basically
might
put
under
the
hood
when
the
load
is
increasing.
That's
a
promise.
It
max
and
I
just
want
to
see
that
a
case
now
here
that
demo
is
using
Bare
open,
fast
I'm,
not
judging
open
fast,
and
actually
it
does
the
right
thing
anyway.
A
So
that's
fine,
but
you
would
have
to
tune
up
and
fast
I
want
to
make
that
clear,
yeah.
But
the
idea
is
we're
going
to
inject
some
load,
so
we're
going
to
use
vegetative
say
attack
the
system.
They
all
have
to
be
called
attack.
I,
don't
know
why
you
want
to
attack
a
system,
but
it
feels
very
I,
don't
know,
but
we're
going
to
use
that
to
send
some
load
and
we're
going
to
say
we
want
that
rate
of
450
seconds.
A
I
want
30
requests,
you
know
per
second
or
things
like
that
and
you
you
spread
that
across
eight
workers
doesn't
really
you
know
matter
what
how
it
works
and
we're
going
to
that
in
background,
because,
obviously,
if
we,
if
we
do
that
and
wait
for
it
to
finish,
we
wouldn't
learn
either.
So
at
that
age
a
stage
in
the
method,
we
also
have
a
probe
that
is
going
to
simply
fetch
the
earth.
A
Here
we
don't
have
a
torrance
within
a
method:
you're,
not
testing
things
you're,
not
bailing
at
that
stage,
you're
looking
objectively
what
the
system
does
and
that's
the
thing
you're,
not
bailing
immediately
you
you're
trying
to
be
as
objective
about
the
system
reacting
badly
or
well,
as
you
can,
and
then
we
have.
You
know
more
probes
here
and
this
one
is
just
going
to
look
for
just
to
have
basically
one.
We
do
reporting
we'll
see
how
many
you
know
pods
were
triggered
so
with
without
further
ado,
and
here
we
could
query
primitives
again.
A
Let's
run
this,
let's
run
race
and
we'll
try
to
have
visibility,
observability
as
well.
Won't
we
so
back
to
weave
and
I
really
hope
this
going
to
work
come
on
alright,
so
the
weight
works
is
fairly
simple.
It's
called
chaos
so
that
you
know
exactly
what
you're
doing
and
run,
and
you
basically
take
that
JSON
/ml
file,
alert
and
call
the
experiment.
Alright,
so
I
would
like
you
to
pay
attention.
A
A
Could
you
yeah
so
the
experiment
will
check
the
validity
of
the
the
file
itself
as
base
is
good,
so
it
fails
immediately
if
it
doesn't
work
and
then
it
runs
the
experiment.
I
just
gave
you,
so
it
does
exactly
what
we
saw.
We
see
that
the
load
is
arriving
to
the
function
and
basically
we're
still
waiting
for
open
files.
So
what
because
open
fast
at
this
stage
are
come
on.
A
It's
triggering
on
that
and
because
I
was
too
slow,
it
should
have
been
yellow
and
then
red.
But
it's
it's
easy,
the
alert
so
it
triggers
and
creates
new
new
pods
new
function,
instances
and
we're
just
still
running,
and
we
because
what
we're
interested
in
is,
did
we
get
the
alert?
Did
it
work
things
like
that?
That's
a
kind
of
things
that
we're
interested
in
that
experiment
so
where
it's
still
red
and
the
Lord
is
it
spread
across,
as
you
know
nicely,
so
we
continue
doing
that.
A
Like
I
said
earlier,
it
looks
like
testing
tools.
The
tooling
is
fairly
similar
in
that
in
that
regards,
but
the
mindset
again
is
different.
You're
not
trying
to
be
binary
here,
you're,
just
exploring
what
happens,
and
in
this
case,
because
I
already
have
done
the
work.
It's
going
to
be
green,
and
this
demo
gods
on
gets
against
me,
but
it
should
be
green,
but
it
would
be
red
saying
that
something
was
fishy
and
and
was
deviated,
which
I'll
show
you
how
it
looks
after
that.
A
So
here
you
see
clearly
that
the
idea
of
the
thing
we
at
the
very
bottom,
we
see
steady
states
that
steady
state
hypothesis
is
met
as
we
see
at
the
beginning.
Those
are
exactly
that's
one
that
is
run
before
and
after
and
you
see
if
there
was
a
deviation
so
far,
it
would
be.
It
would
be
read
immediately
if
it
had
deviated.
It
would
tell
you
this
is
not
what
you
expected
right
and
then
you
learn
by
by
looking
at
report.
A
All
right,
so,
basically
what
it
does
if
the
report
is
is
fairly,
you
know
dumped
in
a
sense
where
it
it
doesn't
magically
tell
you
what
you
need
to
do
right.
It
gives
you
something
you
can
share
across
the
team
and
start
learning
from
it.
Ideally
down
the
line,
you
would
generate
a
temporary
report.
You
know
add
your
comments,
write
your
you
know,
analysis
and
then
set
send
that
across
because
we're
using
pen
doc
it
can
generate
HTML
all
those
things
like
modern
or
whatever.
Now
the
interesting
bit
here
is,
if
that
worked.
A
Sometimes
it's
a
bit
flaky
on
that
one
we
see
generates
also
graphs
from
data
that
was
collected
now
it's
interesting
because
our
steady
state
was
just
about
whether
or
not
we
have
one
function
there,
one
functioning
instance,
but
we
see
that
from
the
run
three,
some
connections
failed,
I,
guess
during
the
time
that
we
we
couldn't
actually
under
the
load
before
it
triggered
right.
So
you
still
have
an
analysis
after
the
steady-state
right.
You
see
that
basically
what
what
happened
still
impact
the
user.
A
So
either
you
come
back
to
steady-state
and
make
it
more
knowledgeable
or
you
simply.
You
know
making
a
statement
confirmation
with
your
team,
what
happened,
and
sometimes
you
have
to
rerun
that,
because
you
know
it's
not
necessarily
easy
to
make
three
tries
blah
blah
blah
here,
but
the
interesting
one
I
think
is:
is
it
this
one?
A
So
that's
querying
the
number
of
function,
pods
that
we
have,
and
we
do
see
it's
easy
to
see
that
we
went
from
one
to
five,
because
open
files
will
add
five
of
them
dynamically,
at
least
in
the
default
configuration
for
now.
So
that's
what
we
expected
and
it
came
back
to
one
again:
that's
what
I
expected?
A
What
time
is
it
I?
Think
I'm
done
right,
I'm
soon
done
three
minutes,
so
I
want
to
show
you
the
next
demo,
the
other
day
mode.
I'll
come
back
to
the
talk
which
is
almost
finished,
the
the
other
demo.
You
could
see,
find
an
N
come
back
to
me
after
that
is
using
jke,
where,
let's
assume
your
nodes,
have
your
virtual
machine
nodes,
not
the
kubernetes
one's
exactly
where,
but
the
machines
where
they
leave
the
nodes
have
a
secretary
security
breach.
A
You
need
to
change
something
in
J
cloud,
it's
easy
to
actually
create
and
you
not
pull
bring
it
up,
join
the
cluster
and
swept
to
it.
But
have
you
done
that?
Have
you
ever
done
that
in
production
in
a
manner
where
you
feel
confident
of
doing
it,
you
may
have
put
it
early,
it's
more
like
I
have
to
do
that
now
is
easy
here
to
do
that,
or
is
she
here?
A
You
know
my
my
users
while
I
do
that
and
you
run
that
and
that
will
basically
create
and
you
not
pull
switch
to
it
and
see
if
that,
if
the
users
are
still
receiving
the
load
and
that's
I,
think
that's
pretty
powerful,
so
I'll
finish
quickly
with,
if
I
can't,
if
I
can
find
them,
why
you
yay
I
will
quick,
let's
read
yeah
it
does
that
again
so
for
now,
you'll
see
yeah
exactly
it's
chaos.
So
let's,
let's,
let's
you
know
it's
just
one
slide
anyway,
I
think
so.
A
What's
next,
because
chaos
toolkit
is
a
small
and
a
newer
project.
We
have
plenty
of
things
we
can
look
forward
to.
One
of
the
things
is
for
now.
You
run
that
from
an
outside
machine,
which
I
think
is
right
thing
to
do,
but
sometimes
you
want
to
run
that
from
communities
itself,
so
we're
going
to
look
at
co,
d--'s
and
jobs
and
sing
like
that.
We
need
to
improve
the
documentation
a
bit,
because
it's
a
bit
messy
right
now
again
will
try
to
implement
drivers
so
that
you
can
do
that
and
go
in
register.
A
However,
you
want
we
want
to
in
work
on
discoverable
experiments,
because
it's
nice,
if
you
could
share
experiments
between
yours,
you
know
between
each
other,
say:
oh
you've
got
that
system.
You
already
that
kind
of
system.
Can
you
send
me
your
experiments
that
would
be
very
awesome
and
obviously
trying
to
reach
a110
by
by
the
end
of
the
year,
and
I
think
yeah.
Thank
you.
Thank
you
very
much.