►
From YouTube: Cloudy With a Chance Of Chaos: Verifying the Resiliency Of Cloud-Native Applications - Bella Wiseman
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cloudy With a Chance Of Chaos: Verifying the Resiliency Of Cloud-Native Applications - Bella Wiseman, Goldman Sachs
Speakers: Bella Wiseman
Interest in chaos engineering has exploded over the last few years, with more and more organizations looking to adopt the practice. But as those same organizations shift to using managed services in the cloud, traditional chaos engineering techniques are often no longer viable. Powering down a machine is a simple, powerful, and versatile way to uniformly inject failure across all types of applications. But today, when we build cloud native apps, we often choose to use managed services that provide a layer of abstraction on top of the underlying machines. How can we inject realistic chaos when we have no access to the underlying machines? Join Bella Wiseman of Goldman Sachs, as she discusses chaos engineering essentials, chaos on the cloud, and a real-life case study of a chaos engineering experiment at Goldman Sachs.
A
A
A
Foreign
you'll
find
official
definitions
online.
This
is
my
definition,
find
your
next
production
incident
before
it
finds
you
right.
So
that
means
kind
of
deliberately
doing
things
that
cause
production
incidents.
Okay,
so
that
you
can
determine
what
the
impact
will
be
right.
It's
going
to
be
an
environment,
but
essentially
you
know
doing
something
like
doing
a
bad
Trigger
or
something
like
cause
a
production
incident.
And
then
you
know,
hopefully
the
impact
won't
be
a
prod
incident.
But
if
it
is,
you
know
you're
able
to
actually
see
what
that
impact
is.
A
So
the
classical
example,
which
is
how
this
whole
thing
started,
is
bringing
down
a
machine
by
bringing
down
the
machine
that
one
of
the
machines
that
you're
that
your
application
is
running
on
so
I
look
at
there
as
being
five
parts
to
running
a
chaos
test
right.
First
of
all,
defining
your
success
criteria,
understanding
what
you're
trying
to
achieve
right,
two
being
defining
and
measuring
the
steady
state
of
your
system,
three
injecting
the
failure.
A
Yeah,
so
what
does
defining
your
success
criteria
mean
in
this
context
right
it's.
What
does
success
mean
for
this
test?
Are
you
expecting
no
impact
to
availability?
Everything
should
just
continue
to
run
as
expected,
a
minimal
of
impact
to
availability,
in
which
case
the
question
is:
how
much
impact
is
acceptable
right?
A
Three,
a
self-healing
system
right
there
will
be
impact,
but
you
expect
the
system
to
be
able
to
recover
on
its
own
without
outside
intervention,
in
which
case
the
question
is,
after
how
long
and
then
four
perhaps
you're
looking
for
you
know
there
will
be
impact,
maybe
even
annual
intervention
required,
and
you
want
an
alert
to
be
triggered.
You
want
your
support
person
or
on
call
to
be
paged
right
and
then
five.
You
also
might
want
to
check
that
your
dashboards
are
actually
reflecting
the
system
State,
showing
that
you
have
an
issue.
A
So,
let's
go
into
defining
and
measuring
your
steady
state
right.
So
when
we
talk
about
study
State,
it's
like
how
your
system
is
supposed
to
behave.
If
everything
is
going
well
and
typically,
when
we
talk
about
observability
and
resilience,
we
talk
about
an
SLO,
a
service
level
objective
and
that's
usually
measured
actually
over
a
longer
period
of
time.
So
something
like
a
30
days.
A
You
might
say
we
want
99.9
percent
of
requests
to
return
a
successful
response
over
a
30-day
or
90
day
period
and
that's
extremely
important
because
you
don't
want
to
be
impacted
by
little
blips.
You
know
for
five
minutes.
Something
happened
for
three
minutes:
something
happened.
You
want
to
be
able
to
maintain
that
SLO
consistently
over
the
long
term,
but
for
chaos,
tasks
that
actually
doesn't
work
very
well
right,
because
in
your
chaos
test
for
a
30-day
period
at
least
I,
you
know
I
hope
that
you
want
to
get
results
much
quicker
than
that.
A
A
Then
there's
measuring
availability
so
always
try
to
do
it
from
your
customers.
Viewpoint,
that's
really
important,
and
then
you
know,
while
this
may
be
complex
and
you
might
not
have
all
the
answers,
especially
if
it's
a
new
system,
it's
okay,
to
run
a
chaos
experiment
in
non-prod
before
you
have
all
the
answers,
because
it
will
help
you
learn
about
your
system,
help.
You
discover
some
of
these
things
know
what
the
right
questions
are
to
ask
and
it
can
actually
be
a
really
useful
exercise.
A
So
next
we'll
talk
about
some
best
practices
when
injecting
failure.
So,
first
of
all,
you
start
off
by
identifying
points
of
failure.
You
look
at
your
architecture,
diagram.
Your
system
figure
out
what
could
fail
right
and
then,
if
you
do
a
good
job
of
that,
you'll
probably
end
up
with
quite
a
long
list
of
things
that
could
fail.
So
then
you
need
to
prioritize
because
you're
not
going
to
start
with
testing
everything.
So
when
you're
prioritizing,
you
want
to
prioritize
based
on
impact
right.
A
What's
the
worst
thing
that
could
happen
if
something
goes
wrong
here,
as
well
as
the
probability
of
it
happening,
and
when
you
talk
about
probability,
you
know,
there's
you
want
to
avoid
both
sides
of
the
extreme.
So
something
happens.
Every
30
minutes
there's
not
really
much
use
in
doing
a
chaos
test
on
it,
because
you
might
not
even
call
that
chaos
right.
It's
happening
already
all
the
time.
A
On
the
other
hand,
if
it's
likely
to
happen,
maybe
once
in
2000
years
I
mean
I'm,
not
saying
it's
not
valuable,
to
run
that,
but
you'll
probably
find
something
with
a
higher
Roi
somewhere
in
the
middle,
like
something
that
happens
every
six
months,
something
that
you
know
will
happen
in.
You
know
30
days
from
now,
like
that's,
usually
your
sweet
spot
in
terms
of
priority
and
then
just
current
confidence
level
like
how
much
do
you
know
about
what
will
happen
if
this
thing
goes
wrong?
A
Third
thing,
I
would
say
start
simple,
even
manual
like
there
are
great
tools
out
there.
You
know
for
chaos,
testing
and
definitely
feel
free
to
leverage
them.
But
when
you're,
just
starting
out
like
do
something
really
small
and
easy
and
simple,
because
you'll
learn
a
lot
from
that
as
well
and
we'll
get
into
that
soon.
A
So
you
don't
want
to
rely
exclusively
on
whatever
you
know,
system
dashboards.
You
already
have
you
want
to
consider
those
dashboards
as
part
of
your
system
under
task
part
of
what
you're
trying
to
verify
with
your
chaos
test.
So
you
want
to
have
a
different
way
of
also
verifying
what
how
your
system
is
behaving
and
then
just
like
in
a
real
incident.
You
would
be
double
checking
everything
like.
Are
you
sure
this
is
up?
You
know,
make
sure
the
website
is
up.
Look.
A
A
Finally,
you
need
to
restore
the
steady
state
might
not
be
required
at
all
if
everything
went
well
and
your
system
is
self-healing,
but
if
your
chaos
has
expected
outcome,
you
may
need
to
do
that
and
since
you
started
a
non-prod
and
you
need
to
panic,
but
you
do
need
to
get
things
back
up
and
running
or
two,
you
know
right
away:
the
system
wasn't
self-healing.
There
would
be
some
run
book.
A
A
So
even
though
it's
non-prod,
if
you
all
developers,
are
blocked
for
three
days,
because
you
ran
a
chaos
experiment,
the
chances
of
you
you
know
being
able
to
do
more
of
this
is
not
that
high
right,
so
be
careful
and
non-pride
as
well.
After
you
test
a
non-profit,
then
you
can
have
the
information
to
decide
whether
you
want
to
move
to
prod
and
do
this
in
prod
or
not
right.
You
have
to
consider
the
risks
of
doing
it
in
prod
and
then
also
the
risks
of
not
doing
an
improv
right.
A
Next,
we'll
talk
about
chaos,
ready
teams
right
so,
first
of
all,
you
need
a
commitment
to
resiliency
and
operational
excellence
if,
for
whatever
reason,
the
system
that
you're
talking
about
actually
has
known
resiliency
issues
that
no
one
is
fixing.
There
is
no
point
in
running
a
chaos
experiment
to
discover
more
operational
resiliency
issues
that
will
just
get
added
to
the
backlog
and
no
one's
going
to
fix
right.
The
one
exception
the
rmit,
if
like
there,
are
resiliency
issues
that
are
coming
out
in
prod,
but
so
far
they've
been
small.
A
No
major
incidents
sometimes
running
a
chaos.
Experiment
can
actually
help
you
bring
proof
to
management
that
yeah.
So
far
the
issues
have
been
small,
but
if
one
of
them
would
happen
during
this
high
volume
day
or
if
one
of
them
would
happen-
and
this
thing
would
go
wrong,
you
could
have
this
major
incident
and
then
that
can
help
you
get
the
attention
to
resiliency
that
you're
trying
to
get
and
then
as
well
like
when
it
comes
to
culture
right.
A
Engineering,
because
you're
just
bringing
more
bad
news
that
people
would
rather
not
know
about
right
and
then
finally,
psychological
safety,
blameless,
postmortems
right,
you
know
when
you're
doing
cast
tests
if
you're
running
them
in
production,
there
is
always
a
possibility
that
something
will
go
wrong
right.
Hopefully,
you've
thought
about
it.
You've
decided
that
the
benefits
of
running
the
chaos
task
outweigh
the
risks,
because
the
risk
of
not
running
the
chaos
test
is
also
very
high.
But
knowing
that
you
know
the
finger
won't
get
pointed
at
you
that
people
understand
that
everyone's
doing.
A
You
know
what
they
think
is
right
for
the
system
and
is
you
know
doing
things
appropriately
will
help
you
be
able
to
do
chaos
tests
repeatedly,
and
you
know,
move
things
forward.
Okay,
great!
So
now
you
have
a
chance,
ready,
environment
chaos
ready
team.
Now
what
you
need
to
do
is
convince
your
boss.
A
So
here
are
some
things
you
shouldn't
say
it's
fun,
I
just
read
a
really
great
blog
post
about
chaos.
Engineering,
it's
trending
on
my
Twitter
feed
I
just
saw
Bella,
really
great
talk
at
Cloud,
nativecon,
Nevada
and
I
want
to
try
it
out
right.
I
want
to
write
a
really
cool
blog,
post
or
I
want
to
get
promoted
right.
A
These
are
probably
not
the
things
that
are
going
to
convince
your
boss
to
let
you
do
chaos
engineering,
but
there
are
what
I
find
like
two
categories
of
times,
one
of
those
actually
commercial
value
to
doing
Chaos
engineering-
and
this
is
a
great
time
to
raise
your
hand
and
say:
hey:
let's
try
it.
The
first
one
is
before
a
release
right,
whether
it's
the
first
release
of
your
system,
you're
about
you're,
going
to
go,
live
soon
or
it's
the
release
of
a
major
feature
right
at
that
point.
A
People
are
nervous
right,
they're,
looking
for
confidence,
they're,
looking
for
a
way
to
make
them
comfortable
that
when
they
go
live,
something
bad
won't
happen,
that's
a
great
time
to
actually
figure
out
what
the
points
of
failure
are
trigger.
Those
points
of
failure
ensure
that
your
system
can
be
resilient
to
them.
That's
one
category!
Second,
one
is
after
a
major
incident
right.
You
have
all
your
post-mortem
follow-ups,
but
if
you're
actually
recreate
the
trigger
that
happens
in
production,
how
do
you
know
that
those
are
the
right
follow-ups
right?
A
How
do
you
know
that
those
follow-ups
were
actually
done
appropriately?
So
that's
another
time
when
actually
saying
a
chaos
task
like
let's
actually
re-trigger,
that
make
that
bad
thing
happen
again
and
then
ensure
that
we
have
an
incident
or
at
least
that
that
incident
is
much
smaller
than
it
was
previously.
A
That's
also
a
recommendation
that
I
often
give
you
know
post
incidents.
So
then
you
think
is
KS
engineering
easier.
Is
it
hard
and
I
kind
of
came
to
the
conclusion
that
it's
full?
So
here's
some
reasons
why
chaos
engineering
is
easy
right
because,
first
of
all,
even
if
you
do
a
manual
chaos
test
and
I'll
discuss
what
that
means
soon,
you
can
get
a
lot
of
value
with
very,
very
minimal
investment,
also,
there's
lots
of
Open
Source
and
vendor
software
available
basic
use
cases.
A
Similarly,
like
tons
of
blog
posts
out
there
tons
of
talks,
it's
easy
to,
like
you
know,
read
up
on
what
you
should
do
and
then
four,
if
you're
on
the
cloud
you
know
Cloud
providers
like
because
you
know
infrastructure
is
much
more.
You
know
democratized
right
like
it
could
be
actually
really
easy
to
bring
something
down.
A
You
know
with
apis
and
just
like
easy
tools.
On
the
other
hand,
chaos
engineering
is
also
really
hard
right.
Some
of
the
technical
challenges.
First
of
all,
chaos
tests
by
definition
are
system
tests
right
and
system
tests
are
notoriously
a
flaky
right.
You
have
all
these
false
positives.
You
know
things
that
are
like
all
these
moving
Parts.
How
do
you
make
sure
everything's
up
and
running
the
way
it's
supposed
to
be?
A
So
that's
one
challenge
and
then
you
know
running
in
a
production,
parallel
environment
which
could
be
an
ongoing
investment.
If
you
have
one
already,
then
great
you'll
use
it,
but
then
you
might
be
you
know
with
other
developers
like
kind
of
computing
for
resources
or
you
might
have
to
invest
in
creating
a
dedicated
environment
for
your
chaos
test
and
that
can
require
investment.
A
Alternatively,
you
can
use
canaries
right,
even
though
that
is
already
in
prod
if
you
have
a
mature
Canary
thing,
but
there
are
some
like
large-scale
issues
that
can
be
difficult
to
simulate
canary
second
manage
services
on
the
cloud,
so
not
just
Cloud,
but
actually
using
like
the
higher
level
of
abstraction
of
managed
Services.
Very
often
they
abstract
away
the
details
right,
serverless,
right,
they're
they're,
obviously
are
machines,
but
you
don't
have
access
to
them.
You
can't
manipulate
them
and
that
can
actually
make
it
really
hard
to
inject
a
failure
and
quote
something
to
happen.
A
Another
reason
why
KS
engineering
is
hard:
it's
a
human
one
right,
so
there
are
three
virtues
or
programmers
right,
laziness
and
patience.
Hubris
right,
so
you
think
about
laziness.
Everyone's,
like
okay,
I,
already
have
enough
chaos.
Without
this
all
right,
if
it
ain't
broke,
don't
break
it
like
leave
me
alone.
I
want
to
go
home
and
then
there's
impatience,
right,
I
want
to
just
code,
Netflix
feature
I.
This
is
going
to
take
some
of
my
time
and
then
hubris
right.
My
code
won't
break
right.
I
already
wrote
code
to
handle
this.
A
This
scenario
you
know
two
years
ago,
all
right
it
might
be
true.
Have
you
tested
it
recently
right?
You
know,
as
we
all
know,
it's
always
a
moving
Target
and
things
change
and
the
environment
changes.
Okay.
So
now
we'll
go
actually
into
a
case
study.
So
this
is
a
simplified.
You
know
slightly
modified
architecture
of
a
real
system
in
pre-prod
at
Goldman
Sachs.
So
if
we
go
through,
you
know
the
architecture.
On
the
right
hand,
side
you
have
an
API
Gateway,
which
is
proxying
and
validating
authenticating.
A
You
have
a
service
running
an
ETS
Fargo
right,
three
different
tasks,
running
on
different
availability
zones,
we're
using
oppa,
open
policy,
agent
attacks
to
check
entitlements,
and
then
we
have
an
elastic
surgeon
since
it
was
searches
across
different
documents
and
then
as
well,
there's
redis,
which
is
a
dependency
for
some
types
of
requests.
So
when
we
were
looking
at
doing
cath
engineering,
we
went
through
this
diagram
figured
out
points
of
failure
across
all
the
different
components
and
then
proceeded
from
there.
A
So
some
points
of
failure
right-
the
redist,
for
example,
was
supposed
to
be
a
soft
dependency
for
most
requests.
So
you
know
well
what
happens?
Is
it
really
is
soft
dependency?
Perhaps
it
accidentally
turned
into
a
hard
dependency.
That
would
be
one
thing
to
look
at
right:
elasticsearch,
what
if
it
goes
down
what
if
some
of
the
nodes
go
down
right,
Opa
right
the
entitlement
system?
What
if
you
know
what
effect
gets
disconnected
like?
We
know
that
we
can't
operate
without
entitlements,
but
can
we
like
recover
from
that?
A
How
quickly
can
we
recover
once
it
comes
back
up
and
then,
finally,
you
know
on
the
far
gate
side,
our
tasks
are
going
down.
How
are
we
going
to
be
able
to
handle
that
now?
These
are
all
like
real
points
of
failure
and
things
that
can
go
wrong,
but
again
right
challenges
on
the
modern
Cloud.
How
are
you
going
to
simulate
these
things?
You
don't
have
any
machines
to
bring
down
right.
I
mean
there
are
machines,
but
you
don't
have
access
to
them.
A
A
A
It
says
that
are
best
practices
and
you
do
the
opposite
right,
so
it
says:
don't
run
this.
You
know
resource
intensive
query,
make
sure
never
to
run
of
query
against
your
your
elastic
search
instance,
and
you
go
okay,
write
up
that
query,
run
it
right
and
we're
not
right
and
see
what
actually
happens
right.
You
might
find
the
entire
cluster
goes
down.
How
can
you?
How
can
you
recover
from
that
right?
That
can
be
another
really
useful,
real-life
way
to
enter
us
without
access
to
the
machines.
A
Third
of
the
law,
disrupting
network
connectivity.
So
you
have
multiple
managed
services
that
are
talking
to
each
other
right.
You
can
actually
is
an
ISP
to
actually
you
know,
create
network
black
holes,
so
the
services
can't
talk
to
each
other
right.
That
can
also
be
a
pretty
generic
and
Powerful
way
to
inject
chaos
into
your
system.
A
So
now
we'll
dig
into
the
ECS
fargate
use
case.
That
was
the
thing
we
decided
to
start
with,
because
it's
you
know
relatively
simple.
So
what
is
AWS
ECS?
It's
elastic
container
service,
fully
managed
container
orchestration
on
AWS.
You
can
either
run
on
ec2
instances,
in
which
case
it's
not
fully
fully
managed
and
you
have
access
to
the
underlying
machines
or
you
can
go
server
list
with
ECS
fargate.
A
A
Well,
I
mean
well.
The
good
things
should
happen
right
automatically
bring
up
a
new
one
to
replace
the
one
that
stopped,
but
that
opens
up
a
bunch
of
questions.
How
long
is
that
going
to
take
what
if
multiple
tasks
stop
right?
How
is
your
application
going
to
behave
in
the
interim
and
if
there
is
like
impact,
how
long
will
it
take
for
you
to
recover?
A
So
you
can
read
the
documentation
you
can
or
you
can
try
it
and
that's
what
we
did.
So
when
I
talk
about
starting
simple,
this
is
what
I
mean
right.
There
might
be
amazing
tools
out
there
that
you
might
want
to
use
soon
or
eventually
to
do
this
in
an
automated
fashion.
But
you
when
you're
starting
like
start
simple,
even
a
non-prodite,
just
starts
simple.
A
You
can
actually
go
to
the
console
or
run
a
simple
API
command
or
something
on
the
AWS
CLI
and
just
stop
a
task
right
and
what
you
learn
from
that
is
actually
huge
and
you'll
see.
We
learned
a
lot
just
from
doing
that
once
so.
Simple
experiment,
valuable
findings
and
we
found
was
that
there
was
actually
a
20
error
rate
for
between
3
and
10
minutes.
A
So
what
happened
right?
Well,
obviously,
we
we
know
what
happened.
You
know
we
stopped
an
ECS
task
right,
so
one
of
those
three
tasks
was
down.
The
network
load
balancer,
though,
was
still
sending
traffic
to
the
bad
task
right
for
a
few
minutes
until
the
health
check
for
that
task
finally
failed
and
it
was
brought
out
of
rotation.
A
So
how
would
you
solve
for
something
like
that
right?
So,
first
of
all,
you
can
increase
the
number
of
tasks
right.
Three
tasks
is
not
very
much.
You
can
scale
it
up
to
100
and
then
the
the
risk
or
the
impact
from
a
single
task
going
down
is
negligible
right,
but
that's
a
cost
versus
resiliency
trade-off.
Depending
on
how
much
resilience
you
need
you,
you
might
choose
to
pay
for
more
tasks
or
not
two
is
you
can
tune
the
timeout
of
your
load
balancer
so
that
you
remove
tasks
that
are
not
responding
more
quickly?
A
That's
also
a
trade-off
because
you
might
end
up
with
churn
right.
Like
you
know,
you
remove
the
task,
it
really
would
have
come
up
on
its
own
things
like
that.
So
there's
always
trade-offs
and
tuning
that
goes
on
here,
but
the
second
really
interesting
thing
that
we
found
was
the
dashboards.
So,
on
the
left
hand
side
you
can
see
a
big
green
smiley
face
because
the
system
dashboard
said
everything
is
good.
A
All
the
requests
that
are
seen
by
this
dashboard
are
returning
successful
responses
right,
but
that's
not
what
our
customers
were
seeing
right
on
the
right
you
see.
Actually,
we
were
using
Locus
the
load
testing
tool
to
simulate
the
production
traffic
and
there
again
testing
from
the
customer's
perspective.
We
were
seeing
errors.
So
what
was
happening
there
was
that
the
dashboards
were
pulling
data
from
an
agent
that
was
running
alongside
the
ECS
task.
A
When
the
task
went
down,
the
agent
went
down
and
it
start
stopped
reporting
those
metrics
right
and
therefore
you
didn't
see
those
errors
at
all,
so
Lessons
Learned
one
try
to
make
sure
your
monitoring
is
decoupled
from
your
service
two,
and
this
is
exactly
what
we
did
test
your
dashboards
and
alerting
regularly
with
real.
You
know,
production
like
incident-like
scenarios
and
then
three
monitor
from
your
customers
perspective
using
things
like
synthetic
Pros.
A
And
then,
finally,
a
fundamental
question:
is
this
an
incident
right
and
the
answer
is
it
really
depends
right?
What
are
your
slos?
What
are
your
service?
What
are
your
customers?
Expectations
is
three
minutes.
Eighty
percent
like
it
really
depends,
and
this
actually
having
this
graph
in
front
of
you
having
a
real
thing
that
happened.
You
know
maybe
inviting
your
customers
to
come
test
with
you
in
non-prod
can
actually
help.
You
clarify
some
of
these
things
before
you
know.
Fingers
are
being
pointed
before.
There's
actually,
a
real
incident
happening
live
in
production.
A
So
that
concludes
the
case
study
and
now
we'll
go
to
takeaways.
So,
first
of
all,
getting
started
with
chaos.
Engineering
does
not
need
to
be
difficult.
It
doesn't
require
fancy
tools
if
you
want
to
be
successful,
ensure
that
your
chaos
tests
are
aligned
with
real
business
needs
and
requirements,
and
three
after
major
incidents
strongly
consider
including
a
chaos
test
as
a
post-mortem
follow-up
to
prevent
a
recurrence-
and
this
is
just
about
Goldman
Sachs
engineering.
A
A
A
B
B
A
Sure
so
I
think
the
first
and
like
the
high
level
answer,
is
that
to
do
chaos,
engineering,
you
don't
necessarily
need
the
visibility
into
the
infrastructure
right.
You
need
a
visibility
into
how
your
customers
are
viewing
your
system
right.
So
what
you
do
need
is
access
to
do
something
bad
right.
So,
however,
you
actually
choose
to
achieve
that.
A
Typically
right
previously,
it
was
not
on
the
infrastructure
level,
because
that's
like
really
generic
you
can
you
bring
that
machine
and
impacts
anything
right,
but
now
that
we're
moving
to
the
higher
up
attractions
that
becomes
harder
to
do
so
with
ECS
tests.
I.
Think
the
example
there
is
clear
for
Lambda.
There
are
ways
that
you
can
actually
inject.
You
know
latency
into
your
Lambda.
A
You
can
actually
like
put
things
in
front
of
it
to
like
simulate
failures,
but
the
truth
is
that
it
is
more
challenging
right
with
serverless
to
actually
inject
these
failures
and
to
make
things
fail.
Also,
realistically
and
I
wish
I
had
like
all
the
answers
and
I
don't,
but
but
yeah
like
there
are.
A
You
know
there
are
ways
that
you
can
actually
go,
and
you
know
whether
it's
Network
right
putting
something
like
disrupting
the
network,
so
we
can't
reach
the
Lambda,
but
your
customer
can't
reach
the
Lambda,
even
though
it's
up
and
running
or
whether
it's
you
know
putting
something
in
the
Lambda
that
like
lets
you
for
certain
requests
fail
or
for
certain
requests.
You
know
inject
latency
right.
Those
are
some
of
the
strategies
that
we
use.
I,
hope
that
helps.
Thank
you.
C
A
A
Operate
according
but
that's
that's
definitely
a
really
valuable
thing
to
do.
For
chaotic
engineering,
it's
one
of
the
most
dangerous
things
because
you
can't
recover
like
recovering
the
data
is
hard
and
definitely
you
know
definitely
a
scenario
worth
testing.
I
can't
give
you
personal
a
personal
fun
attitude
here,
though,
thank
you.
D
Hey
so
thank
you
for
this.
So
in
terms
of
chaos,
testing
pre-prod,
you
need
to
simulate
stuff,
which
means
you
need
to
do
load
testing,
because
not
everything
happens
just
because
it's
in
a
browser
right,
one
person
at
a
time,
so
in
terms
of
stimulating
load
tests,
how
have
you
found
that
just
capturing
Network
traffic,
you
know
especially
on
new
features
that
doesn't
exist
yet
you
know
if
something's
already
in
production,
you
can
kind
of
capture
a
sample
and
then
and
then
sanitize
it
and
stuff,
but
like
how?
A
A
You
can
record
what's
actually
there
in
prod
put
that
down
and
then
replay
it
in
non-prod
right
like
it's
a
little
bit
hard
to
to
know
what
your
traffic
will
look
like
once
you're
in
prod,
and
it's
really
just
about
best
guesses,
which
is
why,
like
in
general,
you
want
to
move
incrementally
and
release
to
prod,
even
in
small
increments
right
as
quickly
as
you
can.
So
you
can
get
real
life
feedback,
but
yeah
I
don't
have
a
magic
magic
answer.
There.
E
A
A
great
question,
so
it
actually
is
a
very,
very
large
part
of
the
chaos
testing
and
that's
why
I
said
before,
like
if
there
isn't
a
business
use
case
attached
to
it
like
it'll,
often
just
like
fizzle
out
right,
you
did
the
chaos
test.
You
found
some
things
but
like
nobody's
like
really
interested
enough
to
to
dig
into
it.
If
it's
not
Material
right,
if
it's
not
something
that
people
actually
care
about,
but
yeah
I
mean
it's
just
like
a
post
like.
If
you
had
a
real
production
incident,
you
would
spend
right.
A
F
A
Sure
so,
depending
on
the
structure
of
the
team
right
often,
the
Ops
team
will
be
involved
in
the
non-prod
environment
as
well.
So,
if
that's
the
case,
then
that's
great,
but
definitely
like
you
need
to
involve
your
apps
team
right
because
they're,
the
ones
who
know
what's
going
on
from
an
auth
perspective
and
that's
actually
what
I
find
is
a
very
good
outcome
of
chaos.
Tests
is
that
you
bring
everyone
together
right
and
like
it
kind
of
under
a
less
much
less
stressful
than
a
product.
A
Everyone
comes
together
for
prod
incidents
right,
but
it's
not
the
most
Pleasant
time
for
everyone
to
get
together.
You
get
everybody
together
for
a
chaos
task.
You
know
everyone
starts
talking
to
each
other
and
you
often
uncover
like
disconnects
and
understanding
right
where
Ops
thinks
that,
like
you
know,
you're
doing
X
and
Dev
thinks
you're
doing
why,
and
then
they
realize
that
oh
well,
actually,
we've
been
right
like
completely
misunderstanding,
each
other
the
whole
time.
A
So
definitely
you
want
to
involve
apps,
often
they're
involved
in
non-prot
anyways,
and
even
if
they're
not
get
them
involved
right
like
make
it,
and
hopefully
I,
find
that
off.
People
are
actually
often
more
even
more
excited
about
chaos,
engineering
than
the
devs,
sometimes
because
it's
like
you
know
more
off
their
alley
and
they're
very
often
really
happy
and
excited
to
have
somebody
proactively.
Looking
at
Ops
type
things
so
yeah.
G
So
my
question
just
adding
to
what
he
has
asked.
One
of
the
often
challenge
we
face
is
that
we
require
access
or
admin
access
for
carrying
out
this
thing,
I
understand
that
pre-products,
lower
environment
is
easy,
but
Goldman
Sach
how
you
are
planning
to
do
in
abroad,
because
that
require
probably
Beyond
VP
level
approval
to
touch
anything
right.
Sure.
A
So
it's
definitely
a
challenge.
There's
two
processes
that
there's
an
access
side
like
the
the
technical
part
of
like
does
somebody
have
access
and
typically
somebody
will
have
access
right
there
like
there
will
be
somebody
who
has
that
access
so
then
getting
that
person
to
actually
participate
and
then
there's
the
flip
side,
which
is
not
the
technical
part
of
getting
access,
but
the
like
organizational
part,
which
is
like,
if
you're
going
to
do
a
chaos
test
in
production.
You
want
to
have
like
a
go
no-go
type
of
meeting
with
everybody
in
the
room
right.
A
You
don't
don't
cast
alone
for
sure,
not
in
Pride
right.
Just
like
you
before
you
release,
you
have
a
go,
no
go
like.
Should
we
do
this?
Should
we
not
do
this
same
thing?
Write
down
your
plans,
get
everybody
in
a
room.
Have
everybody
sitting
there,
you
know
virtually
or
in
reality
and
then
have
the
person
with
permissions
press
the
button
right
and
then
the
other
Pro
of
that
is
that
if
something
goes
wrong,
anybody
who's
needed
to
actually
resolve.
A
A
A
A
What
I
find
is
interesting,
honestly
is
like
a
lot
of
tools,
start
with
the
simple
things
right
and
then
like
kind
of
stuff,
because
the
other
stuff
is
really
hard
right,
so
I
think
like
what
would
be
great
is
you
know,
kind
of
getting
together
and
and
slowly
getting
a
library
where
we
actually
also
solve
the
more
complex
problems
right
rather
than
everyone
kind
of
writing
their
own
new
chaos
tool?
That
starts
and
does
bring
down
ec2
instance
stop
the
ECS
test
and
then
kind
of
Fizzles
out,
because
the
other
stuff
is
too
hard.
A
I
Thank
you
so
a
lot
of
the
so
we
were
talking
about
like
manual
chaos,
tests
and
stuff
like
that.
What
about
moving
to
like
automating
the
chaos
test,
like
what
are
some
of
the
things
that
you
have
to
keep
in
mind
and
I
guess
like
how
do
you
ensure
that
you're
always
watching
the
automated
chaos
tests
that
are
running
sure.
A
That's
a
great
question,
so
automating
is
definitely
very
valuable.
It
is
a
system
test
right.
So
let's
say
a
lot
of
what
suggested
is
running
as
part
of
your
your
CI
CD
pipeline
right.
You
have
a
chaos,
testing
step
some
of
the
challenges.
There
are
again
the
fourth
positive
right.
So
when
you're
running
it
once
you
can
set
things
up,
you
can
make
sure
everything's
running
what
we
found
to
be
a
challenge.
A
lot
is
you're
automating
chaos.
Tests
is
like
you
know.
A
A
The
other
challenge
is
like
you
know
determining,
and
that's
why
running
manually
first
helps
you
need
to
have
when
you're
automating
a
pass
fail
threshold
right.
You
need
to
have
like
that
that
binary,
like
yes
or
no,
we
other
Pastor
failed
and
when
you're
running
manually,
you
can
look
at
it
and
kind
of
tune
that
and
figure
out
where
that
should
be.
A
But
if
you
set
that
too
aggressive
you're
going
to
have
a
lot
of
false
positives,
also
right,
you
say:
oh,
we
only
want
99.9
percent
of
the
requests
every
single
time
that
we
run
our
chaos
test
and
our
cicd
pipeline
you're,
probably
going
to
end
up
with
lots
of
build
failures
and
everyone's
just
going
to
start
ignoring
that
right.
So
you
want
to
set
it.
You
want
to
make
sure
you
have
a
really
stable
environment
to
run
it
in.