►
From YouTube: Lightning Talk: A Culture of Resiliency through Chaos - Scott Glaser & Prakash Ramesh, Salesforce
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Lightning Talk: A Culture of Resiliency through Chaos - Scott Glaser & Prakash Ramesh, Salesforce
Releasing software changes at a high velocity is one thing. How do you ensure your applications continues to run release after release with high resilience and reliability? In this talk we will share how any organization can develop a culture of reliability, identify pitfalls and common mistakes that can be made, and how to build your own roadmap towards developing key reliability and resilience capabilities in your own team and organization. Drawing from practices and tools such as Chaos Engineering and Observability we will share how you can collaborate and continuously learn to improve your application reliability.
A
A
A
So
they
reacted
very
quickly
and
through
the
experience
of
the
pilots
and
through
the
procedures
they
followed,
they
were
able
to
land
the
plane
in
the
Hudson
River
and
that
whole
situation
of
the
155
passengers,
no
one
was
hurt.
In
fact,
the
national
transport
safety
board
said
it
was
the
most
successful
ditching
of
an
aviation
plane
in
aviation
history.
A
So
how
does
all
this
story
relate
to
resiliency
and
and
Chaos?
So
what
this
means
is
that
things
are
going
to
happen,
unpredictable
things
that
you're,
not
gonna,
know
are
going
to
happen,
and
it
really
comes
to
how
you
react
and
how
you
recover
and
learn
from
that.
So
in
the
situation
of
the
story,
I
told
you
about
the
plane
flight.
Is
you
can't
predict
that
every
time
you
take
off
from
an
airport,
you
know
you're
in
a
flock
of
birds
and
how
what's
going
to
happen?
A
So
that's
very
chaotic,
you
don't
know
it's.
Those
kind
of
things
can
happen.
So
when
those
things
happen,
how
do
you
react
to
that?
And
how
do
you
have
some
sort
of
way
of
dealing
with
it
and
learning
and
recovering
from
it
very
quickly?
So
that's
how
the
story
kind
of
relates
to
what
we're
going
to
talk
to
you
about
next
in
a
technical
perspective.
So
what
ended
up
happening
when
you
go
through
these
things?
A
You
tend
to
fall
back
on
what
you
know
and
how
you
work
in
your
workplace,
which
is
really
an
evolution
of
your
culture.
So
what
we're
going
to
talk
about
and
continue
in
more
detail,
is
how
this
chaos
unpredictability
in
chaos.
Engineering
affects
what
you
deliver
and
how
you
improve
your
products,
we're
going
to
look
at
how
to
use
observability
and
how
you
become
more
resilient
by
looking
at
these
things.
A
So
in
the
process
we're
going
to
introduce
some
random
unpredictable
behaviors
and
identify
for
you
to
identify
some
weaknesses
in
your
product
and
how
would
you
recover
from
those
and
how
and
the
main
goal
here?
One
of
the
main
goals
is
not
only
learning
from
that,
but
also
how
quickly
can
you
recover
from
those
kind
of
things,
because
one
of
the
things
you
want
to
really
practice
on
is
your
mean
time
to
recover
all
right,
so
next
I'm
going
to
hand
it
off
to
prakash
and
he'll.
Take
you
on
to
the
next
steps.
B
Thank
you,
Scott,
hello,
everyone.
My
name
is
prakash.
Who
will
be
talking
for
the
quick
chaos
demo?
Now,
let's
see
the
methodology
we
adopt
to
conduct
the
chaos
first,
we
Define
a
steady
state
and
plan
for
chaos,
orchestrate
chaos,
which
can
be
as
simple
as
a
failure
which
we
will
see
later
in
a
demo
and
or
it
can
be
as
complex
as
a
multi-step
failure
scenario,
where
more
than
one
component
can
fail.
B
Once
we
have
identified
and
defined
the
chaos
experiment,
we
plan
it
as
a
game
day
and
constantly
look
for
opportunities
where
it
can,
where
we
can
make
it
as
a
game
session
and
move
it
ahead
in
the
devops
lifecycle
during
a
game
day
or
a
game
session.
We
create
a
chaos
and
observe
the
impact
using
the
standard,
observability
tools.
B
We
have
we
primarily
look
at
Golden
signals
like
error
rate,
saturation
of
resources,
response
times,
latency
of
the
services
and
the
system,
so
on
and
so
forth,
along
with
the
alerts
emitted
by
those
tools,
once
we
have
identified
any
resiliency
failure,
we
fix
and
expand
the
scope
in
the
next
cycle
of
the
chaos
experiment.
Also
before
we
ship
any
service
or
product
it's
tested
for
standard
set
of
resiliency
experiments,
that's
how
we
match
and
build
a
culture
of
resiliency.
B
Now
we
will
see
we
test
test
for
chaos.
It
can
be
done
during
any
part
of
the
devops
life
cycle
during
a
cact,
build
performance
or
load
testing,
functional
or
integration
testing
in
pre-prod
or
broad
environment.
The
key
thing
we
have
identified,
based
on
our
experience
during
our
experiments,
is
system
Behavior
under
load
with
chaos,
gives
much
more
insights
and
helps
us
to
identify
the
unknowns
than
a
chaos
test
just
conducted
without
any
load.
So
real
and
challenging
resiliency
issues
are
observed
under
production
like
load.
B
Second
step
is
to
create
a
chaos
using
standard
set
of
tools
which
are
available
in
cncf
landscape
or
as
simple
as
a
script
which
you
can
in-house
develop.
One
thing
we
take
care
when
we
conduct
a
or
choose
a
tool
is
it
is
multi
substrate
which
allows
us
to
conduct
chaos
across
different
Cloud
deployments.
We
have
third
step.
Is
we
use
the
observability
tools
across
all
the
platform
and
the
clouds
which
we
have
and
look
at
the
key
golden
signals
based
on
the
service
and
the
system
under
test,
along
with
alerts
and
infer
resiliency
improvements?
B
B
That's
how
we
plan
and
conduct
a
chaos
and
observe
the
results
of
the
chaos
with
that,
let's
jump
into
a
quick
demo.
We
we
would
like
to
close
this
talk
with
a
simple
demo.
In
this
demo.
We
are
going
to
build
a
hypothesis
around
a
simple
scenario
using
elasticsearch
which
has
an
functional
index
and
a
step
Lloyd
and
kubernetes
cluster.
As
a
stateful
set,
we
are
going
to
query
an
index
that
returns
some
results
using
a
load
testing
tool.
B
While
this
happens,
we
are
going
to
create
a
chaos
by
killing
one
of
the
pod
in
a
three
replica
stateful
set
and
see
how
it
impacts
the
index
and
check.
If
the
index
is
designed
to
be
highly
available,
let's
go
ahead
and
start
the
test.
Now
the
green
line
here
indicates
success.
The
red
indicates
failures,
so
I'm
going
to
kick
off
a
new
test
which
is
going
to
create
a
load
and
start
querying
the
index
now
I'll.
B
B
So
the
same
API
can
be
extended
and
integrated
to
any
collaboration
tools
like
slack
where
the
game
day
can
be
really
converted
into
a
game
session
on
Demand
by
the
developer
themselves.
Now
I
have
kicked
off
the
API,
which
is
going
to
shut
down
one
of
the
or
go
ahead
and
kill
one
of
the
pods,
which
is
deployed
as
a
stateful
set
and
I'm,
using
the
elastic
monitoring
to
see
the
index
state.
So
we
can
see
that
the
index
is
getting
queried
and
it's
just
having
one
replica.
B
So
that's
the
purpose
of
this
demo
to
see
how
it
impacts
and
what
is
the
action
item
you
need
to
take
in
order
to
improve
the
higher
availability
in
a
bit?
We
are
going
to
see
one
of
these
nodes
go
down
while
this
query
is
successfully
being
run.
So
let's
wait
for
some
time
and
see
how
this
impacts
yeah.
Now
we
can
see
that
the
failures
have
started
to
claim
climb,
which
indicates
that
there
is
a
node
failure
as
per
our
chaos
created.
B
B
As
per
this
simple
demo,
we
can
definitely
go
ahead
and
increase
the
replica
count
which
will
make
the
index
highly
available.
We
can
see
that
the
successful
success
has
started
and
the
failures
have
dropped
right
now.
This
is
because
of
the
nature
of
the
stateful
sets
which
is
designed
to
be
you
know,
self-reli
resilient
and
also
the
kubernetes
controller
kicks
in
and
make
sure
that
the
specific
part
which
got
killed
is
restored,
which
also
restores
the
specific
index
available.