►
From YouTube: The Narrative of Reliability: A Chaos Engineering Case Study with LitmusChaos - Prithvi Raj, Harness
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Narrative of Reliability: A Chaos Engineering Case Study with LitmusChaos - Prithvi Raj, Harness
How can Cloud-Native environments continue to run smoothly as they grow and include more services? The only constant is change. We live in a world where technology is progressing at a rapid speed in every direction. With technologies like Kubernetes, applications are developing rapidly. We don't have outages as regularly as we did five years ago. But everything comes at a cost, greater innovation brings more complexity, making it a never-ending game to stay on top of application dependability concerns. There are business-critical services operating on Cloud-Native infrastructures today, and the need of the hour is a compelling technique for improving the millions of applications in production. Breaking things on purpose became a reality and was coined the term 'Chaos Engineering', a theory that turned into practice to ensure resiliency through constant testing. This session helps you understand the basic building blocks of Chaos Engineering for your Cloud-Native systems and covers the various aspects that need to be adhered to while moving in that direction such as basic principles, benefits, the LitmusChaos tool in the CNCF ecosystem, the history, the roadmap for the future, and much more to pave your path towards reliability
A
So
good
morning,
everyone,
as
I
said
my
name-
is
prithvi
Raj
and
I.
Welcome
you
all
it's
nice
to
see
so
many
folks
joining
in
for
a
chaos
engineering
session
we'll
be
talking
about
the
litmus
chaos
project.
It's
a
very
popular
cncf
project
and
Chaos
engineering
or
chaos.
Testing
has
been
a
very
popular
aspect
that
has
been
coming
up,
not
in
just
in
the
cncf
space,
but
also
overall,
a
lot
of
people
are
adopting
chaos,
engineering,
so
I
hope.
This
talk
is
helpful
to
you
all
and
again,
I'll
just
start
by
introducing
myself.
A
So,
as
you
can
see,
my
name
is
prithviraj
I've
been
working
in
the
chaos
engineering
space
since
the
last
two
and
a
half
years
we
started
off
with
Maya
data,
and
then
chaos
native
became
a
primary
sponsor
to
litmus,
and
now
harness
is
a
primary
sponsor
to
litmus
chaos.
I've
been
working
in
the
communities
for
some
time
now,
I've
been
organizing
kubernetes,
Community,
Days,
Bangalore,
Chennai
and,
and
so
many
more
events,
chaos
carnivals.
So
it's
all
been
about
chaos,
engineering
and
feel
free
to
reach
out
to
me
on
on
these
socials.
A
If,
if
you
are
interested
in
chaos
or
or
something
around
community,
so
moving
on,
let's,
let's
take
a
look
at
the
agenda,
we'll
be
talking
about
outages,
of
course,
and
then
we
will
be
taking
a
closer
look
at
chaos.
Engineering.
Why?
Why
chaos
engineering
and
how
how
you
can
get
started
with
chaos?
A
Engineering
by
why
chaos
engineering
is
important
for
you
we'll
also
take
a
closer
look
at
running
a
chaos
engineering
game
day
and
then,
following
by
that,
we
will
just
talk
about
a
short
case
study
on
ifold,
which
is
one
of
the
adopters
of
litmus
and
lastly,
how
you
can
be
a
part
of
the
community,
how
how
you
can
contribute
to
the
community
as
well
so
moving
on.
Let's
just
get
started,
so
here
is
One
Outage
that
I
have
presented
that
that
recently
happened
with
meta
where
Facebook
Instagram
WhatsApp
was
done
and
reported
thousands
of
outages.
A
But
then
you
know
this
is
not
just
one
outage.
If
you
see
my
screen
here
are
examples
of
outages
happening
day
in
and
day
out,
where
you
know
large
companies,
large
Enterprises
are
having
outages
here
and
there
every
day
now
and
then-
and
that
is
where
you
know
scaling-
is
not
just
the
only
issue
there.
Are
you
know
that
there's
you
know
hardware
issues,
There's
issues
with
you
know
some
sort
of
environment
which
they
are
moving
to.
Let's
say
we
are
moving
from
monoliths
to
microservices
architectures
and
their
issues
are
supposed
to
happen.
A
And
if,
if
you
talk
about
kubernetes
itself,
I
mean
if
I
take
the
example
of
kubernetes,
then
kubernetes
a
kubernetes
application
itself
is
in
the
form
of
a
pyramid
where
your
kubernetes
architecture
is
on
the
top.
But
there
are
other
dependencies
as
well.
Where
you
can
see,
or
let
us
say,
your
monitoring.
A
Or
let's
say
you
can
you
can
see?
I
mean
your
platform
layer
or
there
are
your
other
services,
your
mongodb
Kafka
running
and
and
each
and
every
layer
can
possibly
have
an
outage.
Each
and
every
layer
can
go
down,
and
you
know
it's
it's
just
like
the
Murphy's
Law.
What
what
is
supposed
to
go
down
will
go
down
and-
and
why
wait
for
an
outage
to
happen?
A
Why
wait
for
something
like
this
to
come
out
as
news
or
you
know
down
times
to
cause
losses
in
in
various
forms
we
will
be
talking
about
them
to
to
just
you
know,
follow
follow
some
sort
of
case
where,
where
a
lot
of
companies
are
actually
not
moving
to
chaos
testing
because
they
are
skeptical,
people
are
skeptical
how
to
run
such
production
failures.
Maybe
they
don't
have
the
budgets,
and
these
are
the
things
we
are
facing
day
in
and
day
out
being
part
of
the
part
of
the
community
and
running
the
community.
A
For
so
long
so
moving
on,
we
will
see
I
mean,
as
I
said,
these
outages
are
avoidable,
and
these
brownouts
are
avoidable.
Brownouts,
as
I
as
I
spoke
about
I
mean
scaling
is,
is
another
issue:
I
mean
Netflix
started
this
concept
around
2011-12,
where
they
started
off
with
a
single
production
level.
Failure
where
it
was
more
about
deleting
your
infrastructure
in
production
or
deleting
some
some
pods
in
the
kubernetes
aspect,
and
then
it
moved
on
with
more
production
level,
failures
and
more
failures.
So
this
is
something
that
outages
can
cause
and
or
not.
A
Adopting
chaos
can
cause
loss
of
customer
confidence.
There
is
loss
of
employee
confidence
and
reduce
stock
price,
as
you
can
see,
if,
if
you
have
seen
the
stock
prices
of
Facebook
over
last
one
year
due
to
multiple
outages,
I
mean
that
can
be
one
possible
reason.
There
may
be
other
reasons
as
well,
but
here
are
such
possible
reasons
and
if,
if
I
take
examples
from
real
life,
there
are
so
many
news
applications.
A
Other
applications
which
have
faced
such
outages
in
in
the
recent
past,
causing
various
sort
of
losses
for
them,
so
obviously
outages
as
I
mentioned,
will
still
happen,
and
these
are
some
I
mean
companies
facing
frequent
outages
are
not
looking
at
the
chaos
aspect.
They
they
have
seen
I
mean
2x
higher,
mean
time
to
recovery.
There
are
more
team
members
who
are
required
to
avoid
these
outages.
The
operation
costs
are
increasing
and
obviously
there
are
many
more
damages
with
the
major
aspect.
A
So
if
those
were
not,
those
have
decided
not
to
make
a
choice.
You
are
still
making
a
choice
right.
If
you
have
decided
not
to
adopt
chaos,
you
are
still
making
the
choice.
You
are
still
deciding
not
to
go
ahead
with
this
practice.
So
obviously
this
this
choice
is
something
which
I
mean
by
the
end
of
this
talk.
I
hope
you,
you
rethink
and
let's
take
a
closer
look
at
chaos.
Engineering
now,
as
as
I
think
you
might
have
got
a
clear
idea.
A
Chaos
engineering
is
nothing
but
inducing
a
fault
deliberately
into
a
system.
It
can
be
I
mean
a
lot
of
people.
Think
it's
just
production
level
failures,
but
I'll
tell
you
in.
If
you
are
in
your
developer
environment
in
your
pre-staging
staging
environment,
in
your
CI
CD
pipelines,
you
can
induce
a
fault
anywhere.
You
can
Target
your
infrastructure
in
a
certain
way
to
identify
what
sort
of
a
possible
outage
might
happen
when
when
the
system
goes
into
production
or
when
a
real
life
scenario
happens
and
a
simple
example
I'll
take
here,
I
mean
in
India.
A
We
have
I'm
coming
from
India
by
the
way
in
India.
We
have
these
great
Indian
festival,
sales
or
Black
Friday
sales,
as
you
can
call
them
globally
and
and
there's
a
requirement
for
scaling.
There
are
a
lot
of
users
accessing
these
e-commerce
applications
at
a
particular
point
in
time
and
the
systems
go
down.
There's
a
payment
failure,
there's
a
failure
with
the
catalog
there's,
a
failure
with
the
overall
card,
and
here
it's
a
very
simple
example
where
chaos
engineering
can
come
into
play.
Where
you
know
such
scenarios
can
be
predicted
beforehand.
A
By
inducing
such
faults,
if
you
talk
about
the
kubernetes
ecosystem,
then
a
simple
fault
can
be
a
part,
delete
experiment
or
a
node
KL
or
a
container
kill.
These
are
some
very
simple
experiments
that
are
used
by
that
is
used
by
the
community
in
day
to
day
life.
But
as
I
mentioned
again,
it's
basically
it's
not
breaking
things
on
purpose,
as
as
a
lot
of
people
say,
but
it's
breaking
things
to
mitigate
the
risks
or
to
safeguard
your
systems.
B
A
Listed
them
down
in
a
few
steps,
so
obviously
you
understand
the
steady
state
of
your
system
how
your
system
is
behaving
while
it's
in
the
normal
condition
or
what's
the
normal
behavior
of
your
system,
and
then
you
induce
the
fault
as
I
took
in
a
very
simple
example:
apart,
delete
experiment
and
what
happens
when
the
fault
is
induced.
If
you
talk
about
the
SRE
terms
or
SRE
practices,
I
mean
chaos.
A
Engineering
was
seen
as
an
SRE
practice,
but
being
part
of
the
community
for
sometime
now,
it's
also
being
shifted
to
developers
we'll
talk
about
that
later
on.
But
if
your
SLO
service
level
objectives
are
continued
to
be
met,
if
your
system
is
still
resilient
or
the
way
you
apply,
I
mean
expected
the
system
to
behave
after
the
fault
was
induced.
A
If
the
system
continues
to
behave
in
in
in
a
certain
way,
then
of
course
the
systems
are
resilient,
but
if
no
there's
a
weakness
found,
there's
a
possible
I
mean
vulnerability
or
requirement
to
mitigate
that
sort
of
an
outage.
Then
obviously
you
move
back.
You
mitigate
that
and
then
again
you
continue
this
process
where
you
again
identify
the
steady
state
conditions
and
introduce
a
new
fault
and
I
have
you
know.
I
have
created
this,
or
this
has
been
taken
from
the
principles
of
chaos.
A
Engineering,
if
you
have
already
read
them,
and
you
see
what
I
explained
is
explained
properly
here,
how
a
chaos
experiment
or
how
chaos
engineering
is
run.
You
select
a
set
of
experiments.
You
hypothesize
around
it.
What's
your
hypothesis,
what
are
the
faults
that
have
are
that
are
supposed
to
be
run
in
your
system
and
how
will
the
system
behave?
According
to
your
hypothesis,
you
create
an
experimental
group
and
then
you
obviously
experiment
with
those
systems.
I
mean
that
running
your
chaos.
A
Experiments
is
just
not
about
running
it
on
your
target
applications,
but
it's
also
about
setting
particular
intervals.
Creating
the
right
chaos
scenario.
A
chaos
scenario
is
nothing
but
multiple
chaos
experiments
running
according
to
your
test
suit.
We
will
be
talking
about
that
later
on
as
well,
and
then
you
use
your
learnings
to
basically
identify
or
how
your
system
is
behaving
and
then
you
you
continue
the
process.
It
goes
on
in
a
loop
so
that
you
know,
because
you
you
are
you
have
to
understand
it's
just
not
a
single
point.
A
As
you
grow
as
you
move
ahead,
you
have
to
understand
what
are
the
points
or
a
failure
is
possible
and-
and
you
continue
to
create
new
scenarios,
new
chaos
tests
so
moving
on.
Where
can
chaos
be
used?
I
mean,
as
I
mentioned,
it's
not
just
about
running
it
in
production.
Obviously
our
eventual
goal
is
to
run
chaos,
engineering
and
production.
Ensuring
a
production
systems
are
resilient,
but
you
can
run
them
anywhere.
Your
Dev
CI
pipelines
your
staging
environment,
your
pre-production
again,
it
depends
on
the
chaos
tools.
I
mean
there
are
a
lot
of
tools.
A
It's
all
started
off
with
chaos,
monkey
which
is
again
an
open
source
tool.
A
lot
of
people
use
it
for
simple
production
failures,
and
then
there
was
the
Simeon
Army
there
are.
There
were
new
tools
that
came
up,
Gremlin
chaos,
mesh
and
a
lot
of
tools,
but
the
right
tool,
for
you
helps
you
run
chaos
Engineering
in
all
all
these
environments,
so
moving
on
we'll
we'll
introduce
the
right
tool.
According
to
my
perception,
but
again
there
are
so
many
tools
out
there
being
part
of
the
open
source.
A
Community
I
would
urge
you
to
take
a
look
at
all
the
tools
that
are
out
there.
Litmus
chaos
is
one
of
them.
It's
a
cncf
incubating
project.
It
started
off.
It
started
off
as
a
project
to
test
another
cncf
project
by
the
way
open
ABS,
it's
a
it's
a
cloud
native
data
project
and
it
started
off
to
just
curate.
A
Some
simple
I
mean
database
chaos
and
then
it
you
know
we
started
writing
it
and
eventually
it
became
a
project
of
its
own
and
it's
it's
today,
one
of
the
most
popular
chaos
tools
out
there
open
source
tools
out
there
and
it's
having
around
50
to
60
chaos.
Experiments
I
could
have
shown
you
the
chaos
Hub,
maybe
we'll
we'll
move
on
to
that.
But
it's
it's
driven
by
the
principles
of
cloud
native
chaos,
where
it's
open
source.
A
We
have
a
functionality
called
git
Ops,
where
you
know
you
can
scale
your
experiments,
you
can
contribute.
You
can
write
your
own
experiments,
there's
there's
so
much
we
can
do.
We
will
be
talking
about
that.
But
just
taking
a
look
at
Stats
and
it's
it's
adopted
by
the
world
today,
a
lot
of
I
mean
even
in
Japan,
a
lot
of
folks
have
been
using
litmus.
We,
we
have
seen
a
lot
of
usage.
Aws
FIS
has
electrical
integration,
there's
there's
so
much
more
being
being
used
by
the
community.
A
As
you
can
see,
end
users
to
Cloud
native
projects
have
been
using
it
and
Chaos
engineering
has
exponentially
grown,
I
mean
over
the
years
yeah.
When
you
know
people
were
skeptical
today,
large,
not
just
large
Enterprises.
Even
the
small
Enterprises
are
moving
towards
adoption
of
chaos
and,
as
you
can
see,
that's
that's
the
exponential
growth,
and
maybe
this
is
just
a
small
map
where
the
light
purple
ones
are
not
using
I.
A
I
won't
say
the
whole
world
is
using
chaos,
but
the
Deep
Purple
ones
are
are
the
ones
the
countries
which
have
people
or
Enterprises
using
chaos,
engineering.
So
moving
on
this
is
the
dashboard.
I
mean
I.
Don't
have
a
lot
of
time
to
explain
a
lot
of
things
run
a
demo,
but
this
is
the
litmus
keros
dashboard,
as
you
can
see.
So
this
is
the
home
page
where
you
can
see
your
chaos,
I
mean
what
are
the
projects
that
are
running,
who
you
can
invite
your
team
members
chaos.
A
Engineering
is
not
just
a
process
for
a
for
a
single
SRE
or
a
team
member,
but
it's
a
team-wide
process
where
SRE
is
QA.
Engineers
developers
all
come
together
to
ensure
your
systems
are
resilient
and
then
there's
the
chaos
Hub
I
I'll.
Take
you
through
the
chaos
Hub.
Perhaps
excuse
me
a
second
yeah.
A
So
if,
if
you
see
this,
this
is
the
Open
Marketplace
there,
there
is
only
one
project,
the
call
witness
that
has
the
Open
Marketplace,
where
you
can
see
the
experiments
that
are
contributed
by
the
community
I
mean
it's
all
started
off
with
kubernetes
Centric
experiments,
and
then
it
I
mean
chaos.
Engineering
is
required
not
just
for
kubernetes
but
non-kubernetes
environments
as
well.
There's
application,
Level
chaos
in
form
of
core
DNS,
Cassandra
Kafka
and
then
there's
your
Cloud
architectures
AWS
gcphjr
as
well,
which
which
might
require
some
form
of
chaos
or
which
might
require
mitigation.
A
So
this
is
something
which
the
community
has
developed
as
of
now.
But
again,
we
we
are
moving
ahead
there.
There
is
a
litmus
SDK
which
helps
you
write
your
own
experiment.
So
if
there's
sub
chaos,
engineering
experiments
specific
to
your
application,
then
you
can
write
them
yourself
and
use
them
with
with
your
application.
A
So,
moving
back
to
my
presentation,
we
will
talk
about
a
chaos
engineering
game
day
and
eventually
we'll
talk
about
a
small
case
study.
So
a
game
day
is
nothing
but
gamifying
the
process.
It's
it's
nothing,
but
you
know
you
running
chaos,
experiments
according
to
your
application,
understanding
who
the
stakeholders
are
understanding,
how
your
workloads
are
going
to
behave.
You
need
to
create
a
report
and
kiosk
engineering
starts
I
mean.
Obviously
it
should
start
with
building
a
culture.
You
need
to
understand
the
practices.
A
You
need
to
understand
how
chaos
engineering
works,
but
once
the
the
practices
and
the
culture
is
adopted,
then
you
move
on
to
running
real
experiments
running
a
game
day
right,
and
these
are
some
simple
steps.
In
initially
you
introduce
the
participants,
it
depends
upon
who
is
running
chaos
or
where
do
you
want
to
run
chaos?
Let's
say
it's
in
a
developer
environment,
then?
Maybe
your
queue
engine
is
coming
to
play.
A
If
it's
being
run
on
a
production
or
pre-production
environment,
then
maybe
the
sres
or
the
devops
engineers
come
into
play
and
then,
if
you
are
eventually
wanting
to
run
it
into
production,
then
your
customer
success
engineers
and
and
so
many
more
people
come
into
to
play,
to
run
chaos,
experiments.
And
then
you
describe
your
environment,
let's
say:
you're
turning
it
in
a
gke
or
your
AKs
eks
clusters.
You,
you
formalize,
that
your
workloads
are
pretty
important.
A
I
mean
for
kubernetes
you're,
running
Chaos
on
your
pods
or
you're
running
it
cluster-wide
or
you're,
running
it
on
on
a
specific
workload,
but
then
again
non
kubernetes
as
well.
They
are,
you
are
having
a
PVS
and
your
stateful
sets
and
so
much
more-
and
these
are
some
steps.
I
I
mean
I
I.
Don't
have
a
lot
of
time
to
explain
everything,
but
I
I
hope
everyone
can
take
a
picture
of
this
so
that
they
understand
the
real
steps
of
running
a
game
day.
A
Everyone
doesn't
really
know
how
to
run
a
game
day,
but
this
is
what
you,
as
I
have
mentioned.
Observability
AIDS
I
mean
you
are
running,
you
have
to
monitor
where
the
chaos
is
happening
or
where
the
systems
are
going
down
and
dinitrace
Prometheus
grafana
people
folks
are
using
and
eventually
you
the
last
step
is
obviously
to
make
the
fixes
so
make
sure
that
post
creating
a
report
or
once
you
have
run
the
game
day.
You
constantly
run
these
game
days.
A
According
to
the
reports
that
you
are
creating
so
quickly,
I
mean
the
typical
way
of
doing
Chaos
engineering.
As
I
said,
the
eventual
goal
is
to
automate
these
game
days
or
automate
the
experiments
it.
It
usually
was
a
manual
process
and
I
mean
developers,
run
code,
gated
code,
merge
chaos,
gated
code
merges
or
integrated
tests,
but
the
SRE
eventually
runs
an
automated
or
a
manual
game
day.
So,
let's
check
out
the
iFood
story.
That's
the
the
case.
Study
iFood
is
a
food
delivery.
App
just
like
Uber
Eats
in
Japan.
A
Ifood
is
a
popular
delivery,
app
in
in
Brazil,
and
it's
been
taking
60
million
orders
per
day
and
it's
similar
to
any
food
delivery,
app
that
you
will
see
out
there
and
the
problem
statement
was
that
they
were
moving
to
that
microservices
architecture
and
they
saw
servers
going
down.
I
mean
their
messaging
Brokers
were
crashing
and
the
main
issue
arised
on
a
Brazilian
Valentine's
Day
in
2020
when
they
saw
a
major
outage
and
what
did
they
try?
A
I
mean
they
started
off
with
circuit
breakers
and
fallbacks,
where
you
know
their
major
system
went
down
and
then
their
fallback
system
started
running
or
you
know
they
stopped
using
the
circuit
breaker
methodology.
They
stopped
these
sort
of
outages
or
they
curated
those
sort
of
tests.
But
again
they
had
to
understand
that
this
is
a
larger
problem,
and
this
is
not
just
about
mitigating
that
fault
at
a
particular
moment,
and
that
is
where
they
they
moved
on
to
chaos.
Engineering
I
mean
they
started
off
with
manually
testing
again
again.
A
A
So
this
is
how
they
chose
litmus
I
mean
there's
so
much
more,
which
is
available
on
a
Blog
you
can
you
can
check
out
the
the
cncf
blog
and,
as
next
steps,
they
decided
to
create
more
experiments,
create
more
permissions
and
work
on
the
security
and
and
so
much
more
that
that
can
be
done
in
the
chaos,
space,
serverless
chaos
and
application.
Level
chaos.
So
what's
next
for
litmus
it's
3.0,
we
are
developing
a
new,
a
new
UI,
it's
more
robust,
clean
and
developer
Centric.
A
We
are
working
on
on
creating
something
special
for
the
community
and
you
can
wait
perhaps
for
next
kubecon
and
you
will
see
what's
what's
coming
up
and
I
mean
you
can
contribute
to
it
as
well.
There's
there's
new
features,
bug
fixes,
charts,
workflows
that
you
can
contribute
to.
You
can
be
a
part
of
the
community
and
get
involved
in
the
community.
There's
the
code
docs
slack
Twitter,
which
which
you
can
join
so
make
sure.
Let's,
let's
connect
after
my
talk,
maybe
feel
free
to
reach
out
to
me.
A
I
might
not
have
time
for
Q
a
because
I'll
have
to
stop,
but
thank
you
so
much
once
again,
you
can
get
involved
in
the
community
and
thank
you
I'm
part
of
harness.
Again.
We
are
building
harness
chaos.
Engineering
over
litmus
chaos
itself,
so
feel
free
to
reach
out.
If
you,
if
you
have
any
questions
or
want
to
get
started
with
the
chaos
engineering
practice.
Thank
you
so
much
everyone.
Thank
you.
C
A
So
it's
it's
chaos.
Engineering
integrated
with
observability
products,
so
observability
products
are
not
inducing
any
sort
of
chaos.
They
are
just
helping.
You
observe
how
your
systems
are
behaving.
What's
the
normal
behavior
of
your
system
and
when
a
fault
is
induced,
what
sort
of
you
know
Behavior
will
change
happen.
Change
happens
to
your
system,
so
you
are
pulling
out
the
metrics.
Basically,
you
are
running
Chaos
on
on
these
tools
itself,
litmus
chaos,
chaos,
mesh
or
any
other
tool,
and
then
you
are
using
your
observability
tool
to
observe,
what's
happening.
A
Let's
say
a
diner
trace
or
your
grafana
dashboards,
where
you
are
actually
observing
how
the
network
is
slowing
down
or
how
the
system
is
going
down
and
when
there's
a
spike
in
the
system.
What
exactly
happens,
how
much
time
it's
taking?
That's!
That's
just
the
metrics
that
that
are
important
as
part
of
your
okay.
So.
A
A
A
So,
with
the
UI
there
there
is
some
obserability
already,
which
you
can
monitor,
but
other
than
that
I
mean
that's,
that's
just
an
inbuilt
UI,
but
people
usually
the
community
use
that
uses
other
projects.
There
I
mean
for
logging
elastics
plan
and
for
monitoring
again
they
pull
out
the
Prometheus
Matrix.
They
visualize
it
on
the
grafana
dashboards,
dynatrace
or
data
dog,
or
there
are
so
many
platforms
out
there
which,
which
you
can
use
to
analyze
I
mean
that's
just
expanding.
What
what
you
are
already
using.
This
chaos
comes
into
play.
There.
B
B
B
A
Cts
engineering
is
not
just
about
software
faults.
It's
about
mult,
as
I
spoke
about
this
service,
less
chaos
that
application
Level
chaos,
there's
there's.
There
are
multiple
levels
of
chaos:
there's
VM,
VMware
chaos
bare
metal
chaos,
so
chaos
engineering
is
expanding
today,
I
mean
that
we
haven't
worked
on
Hardware
chaos,
specifically
as
a
community,
because
the
the
cloud
native
World
itself
hasn't
moved
on
to
a
lot
of
Hardware
level
faults
but
but
again
chaos,
engineering
as
it
expands.
A
Maybe
in
five
years
from
now
you
will
see
a
lot
of
Hardware
Giants
I
mean
it's
essential
for
everyone.
As
I
spoke
about
from
Airlines
to
banking,
to
e-commerce,
to
food
delivery.
There
is
chaos
required
everywhere.
Everyone
is
moving
to
chaos,
not
just
Netflix
today,
but
this
practice
has
moved
on
over
the
world,
but
yeah
Hardware
chaos
may
be
hopefully
we'll
we'll
see
that
soon
as
well.
C
Thank
you
for
your
presentation.
I
have
a
question
for
the
the
process
on
cloud
engineering.
We
are
a
beginner
waterfall
model
development
development
team,
but
you
are
introduced
at
house
engineering
and
end
the
game
so
and
the
game
is
a
Ubi
okay.
During
the
end
of
that
process
or
on
the
agile
environment.
Maybe
we
make
that
endogame
Sprint
would
be
happen.
So
my
question
is
that
how
often
put
the
end
game
during
the
development
a
time
frame
so
I
want
to
know
that
that
timing
introduces
the
cost
engineering.
A
So
so,
as
I
had
mentioned
before,
a
game
day
is
not
usually
run
at
the
beginning
of
the
process.
I
mean,
as
you
mentioned,
that
you
are
beginning
with
chaos
engineering,
so
the
first
step
is
bringing
the
culture
in
understanding
the
practice,
differentiating
it
with
the
existing
testing
purposes
that
you
already
have
and
then
moving
on
to
the
practice
of
chaos,
where
you
initially
understand
your
systems
where
chaos
engineering
is
required.
A
Where,
where
are
this,
you
know
the
vulnerabilities
involved
in
scaling
or
other
failures
that
are
usually
happening,
and
then
once
this
culture
is
brought
in,
you
start
with
chaos.
Testing
itself
not
running
a
game
day,
because
game
day
is,
is
a
bigger
process
which
is
run
not
at
the
end
of
it.
Obviously,
at
the
end
of
it,
the
goal
is
to
test
the.
The
goal
is
not
just
to
run
a
game
day,
because
game
day
is
more
or
less
to
understand
your
systems,
but
it
it's.
A
It
starts
from
the
middle
of
your
process
where,
once
this
culture
is
brought
in
once
you
have
started
running
some
tests
in
your
staging
or
developer
environments
and
understood
the
impact
of
chaos.
Understood
what
the
journey
is
ahead,
then
you
start
off
with
a
game
day
where
initially
a
game
day
is
just
trying
to
understand
how
the
how
the
game
day
works
and
then
slowly
slowly.
You
run
multiple
game
days
to
basically
create
multiple
reports
on
application.
Behavior.
What
are
the
risks?
C
A
See
as
I
mentioned,
get
started
with
the
resources.
So
all
the
resources
that
are
out
there
created
by
the
community
are
really
helpful
for
you
to
get
started,
and
then
you
just
pull
up
this
platform,
the
litmus
platform,
and
you
run
a
basic
chaos
test
on
a
normal
kubernetes
cluster,
not
even
in
your
environment.
You
just
pull
up
a
kubernetes
cluster
and
you
just
run
a
simple
pod,
delete
or
or
certain
experiments.
A
Then
you
run
a
chaos
scenario
where
multiple
chaos
experiments
are
run
on
the
application,
and
then
you
move
on
to
your
application,
where
you
or
where
you
understand
that
these
chaos
experiments
are
specific
to
my
architecture.
Let's
say
you
are
part
of
Vodafone,
so
how
your
I
mean
the
telecommunication
architecture
or
mobile
application
architecture
requires
chaos,
and
then
you
start
running
a
game
day.
According
according
to
your
application,
yeah.