►
From YouTube: Workshop4 : Chaos Engineering 101 with LitmusChaos
Description
Chaos Engineering has evolved in the past few years with the development of the Cloud-Native paradigm. Chaos testing is seen as a must for all Cloud-Native practitioners emphasizing reliability as a container challenge. With the introduction of the LitmusChaos tool which helps curate Cloud-Native Chaos Engineering in a controlled way, practicing Chaos has become easier. Belonging to the CNCF ecosystem, Litmus stresses upon applying Chaos Engineering to modern architectures and building resilient applications simultaneously. The features and ideas majorly surround Community Collaboration, Observability, GitOps, Open APIs & much more. This talk stresses on the evolution of Chaos Engineering with the introduction of the LitmusChaos 2.0
A
All
right
folks,
we
are
recording
now
so
good
morning,
good
evening,
good
afternoon.
Everyone
from
wherever
you
are
joining
in,
I
believe
everyone's
from
india
so
good
morning,
folks
and
my
name
is
prithwi
raj
I've,
I'm
also
part
of
the
co-organizing
team
of
the
kcd
bangalore
2022
happening
on
the
23rd
of
april.
I
hope
everyone
is
registered
for
that
as
well,
and
we
are
glad
to
welcome
you
all
to
do
this.
A
Chaos,
engineering,
101
workshop
with
with
the
litmus
chaos
team
and
we'll
be
going
through
a
lot,
a
lot
of
things
I
mean
we'll,
not
keep
it
till
one.
Of
course
we
won't
want
to
stretch
it
a
lot,
but
we'll
be
covering
a
lot
in
chaos.
Engineering.
I
think
it's
a
new
topic
for
everyone.
A
lot
of
people
wants
a
hands-on
demo
on
what
exactly
litmus
chaos
is
or
what
exactly?
A
How
exactly
you
run
your
chaos
experiments,
but
this
is
going
to
be
an
interactive
session
and
I
hope
you
all
ask
your
questions
feel
free
to
post
your
doubts
and
you
have
a
learning
session
along
with
us,
so
moving
on,
let's
just
start
with
a
quick
introduction
about
us.
As
I
said,
my
name
is
I've
been
leading
the
community
efforts
for
litmus
chaos
since
the
last
two
years.
It
all
started
off
at
maya
data
back
in
2020
and
to
give
you
a
history
of
this
project.
A
What
this
project
it
is,
it
was
not
was
not
even
a
project.
It
just
started
off
as
a
project
to
test
another
another
tool
set
altogether,
which
is
based
on
cloud
native
data
management
that
was
open,
ebs
and
and
soon
in
a
year
or
so.
We
believe
that
chaos
engineering
can
become
a
project.
A
lot
of
people
just
believe
that
why
adopt
chaos?
Engineering,
is
it
worth
it?
So
we
will
be
talking
about
that.
But
yes,
of
course
it
is
worth
it,
and
that
is
where
it
became
project.
A
B
Thank
you
prithvi,
so
hello,
everyone,
my
name
is
I'm
currently
working
as
a
software
engineer
and
harness,
and
before
that
I
was
also
working
at
chaos
native,
so
basically
at
chaos
native
my
role
as
a
software
engineer
was
to
develop
new
chaos
experiments
so
basically
in
my
day-to-day
life,
what
basically
goes
on
is
I
come
forward
with
different
chaos.
Engineering
scenarios,
I
inspect
the
requirement
of
these
chaos,
workflows
that
we
can
create
in
order
to
simulate
different
kind
of
outage
scenarios,
and
then,
basically,
I
integrate
them
within
the
latest
chaos
environment.
B
A
Thanks
a
lot
in
london,
so
let's,
let's
move
ahead
with
the
agenda
for
this
workshop,
what
we'll
be
talking
about
and
what
all
we'll
cover
for
you
if
there's
something
we
do
not
cover,
then
feel
free
to.
Let
us
know
any
aspect
that
you
want
to
talk
about
and
we
will
open
the
floor
for
questions
in
in
between
as
well.
A
So,
first
of
all,
obviously
we
will
be
talking
about
what
chaos
engineering
is,
how
how
resiliency
drove
the
idea
of
chaos,
engineering
and
what
exactly
are
the
principles
of
chaos
and
then
moving
on
we'll
just
introduce
a
little
bit
of
litmus
chaos
to
you.
What's
the
journey
been
like
how
how
the
community
is,
what
sort
of
help
you
can
get
from
the
community
and
then
second
and
then
then?
Thirdly,
we
will
talk
about
the
architecture.
A
What
how
is
litmus
cures
built?
What
are
the
core
components
of
fitness?
How
how
you
can
understand
them
and
then
advantages
and
disadvantages,
of
course,
and
there
are
not
a
lot
of
disadvantages,
but
let's
say
if
anyone
is
interested
in
any
sort
of
comparison,
or
you
want
to
know
why
litmus
chaos
should
be
a
tool
that
you
should
look
upon,
then
feel
free
to
ask
that
question
the
roadmap.
A
A
So,
let's
get
started
folks
to
for
this
workshop
at
number
one
chaos,
engineering,
a
closer
look.
So
before
I
talk
about
chaos,
engineering,
let's,
let's
just
talk
about
what
resiliency
is
and
how
chaos
engineering
was
brought
into
place.
I
think
a
lot
of
you
might
have
heard
about
it
already,
and
a
lot
of
you
might
not
have
heard
about
it
and
being
part
of
this
kubernetes
ecosystem
or
kubernetes
in
itself
has
a
lot
of
possible
vulnerabilities.
A
So
there
are
so
many
possible
outages
that
can
happen
in
the
day-to-day
world
when
you
are
working
with
kubernetes
and
it
being
dynamic
already
there
are.
There
are
so
many
you
know
dynamic
changes
that
are
happening
enhancements
that
are
happening
day
in
and
day
out
and
even
with
the
load
I
mean
there
are
so
many
factors
that
lead
to
an
outage.
A
So
so
what
happens
when,
when
there's
you
know
a
possibility
of
vulnerability
or
an
outage,
then
there
are
two
possibilities
right:
either
the
system
stays
resilient,
that
is,
the
pod
is
rescheduled
and
the
dependent
service
is
healthy
or
the
node
is
again
in
spite
of
a
not
ready
state
that
the
system
is
working
or
even,
if
there's
a
memory
leak
in
a
container,
you
don't
face
a
possible
outage,
but
but
let's
say
that
the
dependent
services
get
slow
or
you
you
are
not
able
to
work
around
it.
Then.
A
Obviously
there
is
a
possibility
that
that's
an
example
of
a
weakness
right,
and
that
is
where
we
differentiate
between
a
weakness
and
and
how
the
systems
are
resilient.
Obviously,
resilience
is
the
ability
of
a
system
to
stay
afloat.
Let's
say
any
fault
happening.
The
system
still
stands
up
then
that's
obviously
an
example
of
the
system
being
resilient.
But
if,
if
it
goes
down,
then
you
have
to
identify
that
there
is
a
weakness.
A
There
is
a
possibility
that
your
system
might
might
just
lead
to
a
downtime
or
an
outage,
and
that
leads
to
you
know:
companies
losing
millions
and
billions
of
dollars.
So,
as
you
can
see,
there
are
some
expensive
outages
as
examples
that
we
have
taken.
Facebook,
instagram
whatsapp
just
went
down
a
few
days
ago
and
they
they
lost.
A
You
know
millions
in
in
the
stock
market
and
as
well
as
because
because
of
this
outage
in
internally
as
well-
and
there
have
been
so
many
outages
over
the
past
few
years,
slack
slack
has
seen
some
outages.
We
you've
been
using
all
the
social
media
platforms,
twitter,
amazon,
reddit,
massive
internet
outage,
brought
it
down.
There
was
a
dns
error
or
doordash
and
psn
again
became
the
backup
because
of
aws
going
through
an
outage
and.
A
I
mean
financial
companies,
banking
sectors,
you
talk
about
banks,
they
they
are
looking
at
solutions
to
possible
outages.
I
think
american
airlines
went
through
an
outage
delta.
Airlines
went
through
an
outage
a
few
years
ago.
Aws
has
gone
down,
and
these
these
are
the
possible
outages
that
that
have
literally
caused
down
times.
They
have
led
to
these
companies
losing
billions
in
in
revenue
and
and
outages
are
expensive.
A
A
Engineering
was
just
the
process
of
inducing
a
fault
deliberately
into
the
system
or
or
a
real
world
outage
into
the
system,
to
identify
what
will
happen
when
the
fault
actually
happens
or
whether,
when
the
outage
actually
happens,
and
then
then
you
know
create
a
system
which
can
withstand
these
unexpected
disruptions
or
which
can
basically
withstand
any
sort
of
outage.
That
is
happening
in
in
the
system,
and
that
is
what
chaos
engineering
is.
A
Is
it's
just
a
process
of
testing
a
a
computing
system,
a
complex
system
to
ensure
that
the
resiliency
of
the
system
is
maintained
when,
when
such
real-world
scenarios
happen?
So
why
chaos
engineering?
Why?
Why
do
you
need
chaos,
engineering
that
there
are
a
lot
of
folks
who
want
to
ask
or
who
want
to
know
that
what
is
chaos
engineering
or
why
is
it
required?
So
this
is
an
example
of
an
application
we
have.
We
have
taken
this
example
lot
many
times
that
an
application
is
in
form
of
a
pyramid.
What
how?
A
How
does
an
application
behave?
It's
it's
just,
not
your
application,
but
there
are
other
dependencies
as
well,
and
these
are
the
dependencies
that
are
out
there.
I
mean
there
are
your
application
dependencies,
for
example,
your
mongodb,
I
mean
the
database,
your
kafka,
there
are
postgres,
I
mean
there
are
so
many
dependencies
that
your
application
has
under
the
layer.
So
the
first
layer
is
your
application,
of
course,
the
topmost
layer
of
the
pyramid,
and
then
there
are
your
application
that
there's
your
application
layer
of
the
dependencies
and
then
yeah.
There
are
your
cloud
native
services.
A
There
are
so
many
open
source
projects
as
well
code,
dns
onboard
prometheus
for
your
monitoring,
open
ebs
again
for
your
data.
So
that's
the
third
layer
and
then
there
are
the
kubernetes
services
running
and
then
there
is
your
infrastructure
layer.
So
so
I
mean
your
application
itself
is
divided
in
in
five
layers
or
more
than
that
who
knows,
and
each
and
every
layer
has
its
resiliency
dependencies
every
layer.
You
know,
there's
a
possible
of
an
outage
in
every
layer
and
there's
a
there's
a
need
to
test
each
and
every
layer.
A
I
mean
your
there's
a
pod
that
can
go
down,
there's
something
that
can
go
down
in
your
kubernetes
service
or
that,
let's
see
or
any
other
service,
there
are
possible
outages
happening
because
of
enhancements
happening
every
day
I
mean.
Let's
say
your
opening
open
ebs
is
a
cloud
native
data
service
was
enhanced
and
there
were
some
new
features
that
was
added
and
then
suddenly
it
went
down
which
which
actually
led
to
the
whole
application
going
down.
So
so
I
mean
how?
How
can
you
even
depend
on
one
one
particular
factor
right?
A
You
have
to
depend
on
each
and
every
layer,
and
that
is
where
chaos
engineering
comes
into
play
it.
It
predicts
each
and
every
possible
scenario
that
can
happen
each
on
each
and
every
layer.
People
are
testing
their
ui.
People
are
testing
their
data
management
services.
People
are
testing
their
vms,
so
so
chaos
engineering
was
was
necessary
because
the
applications
are
dynamic.
They
are
changing
every
day.
Each
and
every
dependency
requires
testing,
and-
and
that
is
where
chaos
engineering
comes
into
play
and
tests
your
application
from
top
to
bottom,
because
that
is
how
the
dependencies
increase.
A
That
is
how
the
requirement
for
testing
increase
in
in
a
top
to
bottom
way,
so
I
mean
too
many
applications,
as
I
suggest
it
can
affect
the
resiliency
of
your
system,
because
it's
a
dynamic
world,
they
are
changing
and
you
need
to
constantly
validate
the
resiliency
of
the
entire
system,
with
with
not
just
kubernetes
applications
but
with
non-kubernetes
applications
as
well.
The
ones
who
are
using
legacy
systems
bare
metal
vms.
A
They
are
also
requiring
a
lot
of
validation
and
testing
which
chaos
engineering
brings
in
so
where
in
the
devops
loop
comes
chaos
engineering,
I
mean
a
lot
of
people
believe
that
it's
on
the
dev
side,
because
testing
is
part
of
the
dev
site,
but
we
believe
that
chaos
engineering
is
more
on
the
ops
side
of
things
where
you
deploy
when
you
monitor
when
you
operate
so
the
feedback
loop
in
the
in
the
devops
side
gets
activated
by
testing
in
production.
The
eventual
goal
is
to
test
in
production
with
chaos.
Engineering
you
just
don't
want
to.
A
I
mean
a
lot
of
folks
talk
about
pre-staging,
environment,
staging
environments
and,
of
course,
chaos.
Engineering
can
be
used
anywhere
in
the
devops
cycle.
They
can
be
used,
I
mean
when
you're
building
when
you're
testing,
when
it's
in
the
staging
phase
or
or
when
it's
it's
in
production,
but
the
eventual
goal
is
to
ensure
that
the
system
stay
resilient
in
in
production,
so
the
feedback
loop
gets
activated
by
testing
in
production.
A
So
how?
How
do
you
do
chaos?
Engineering?
What
exactly
are
the
steps?
How
do
you
follow
the
exact
way
of
chaos
engineering,
but
before
that
I
would
just
like
to
cover
a
little
bit
on
the
principles
of
chaos,
engineering,
so
there's
a
website
out
there.
You
might
be
aware
of
the
same.
That
is
principles
of
chaos.
Engineering.Org,
I
think
that's
a
very,
very
famous
website.
A
But
how
is
it
practiced
or
how
is
chaos,
engineering
practice,
and
these
are
the
principles
of
chaos
engineering,
so
the
first
step
is
and
the
first
and
foremost
step
is
by
defining
the
steady
state
of
your
system
or
by
by
identifying
how
the
system
is
when
it's
it's
normal
or
when
it
indicates
normal
behavior.
So
you
need
to
define
that
define
the
output
of
your
system
when,
when
you
know
it's
in
steady
state
and
then
you
need
to
hypothesize
around
it,
then
you
need
to
identify
that
how
the
steady
state
will
continue.
A
I
mean
that
you
divide
it
in
two
groups:
one
is
the
control
group
and
one
is
the
experimental
group.
The
control
group
is
is
the
group
with
that
which
basically
indicates
the
the
normal
behavior
of
the
system
or
the
the
normal
experiments
that
happen
here
and
there.
But
the
experimental
group
is
where
you
experiment
with
your
system.
You
try
to
introduce
various
faults
in
your
system.
You
try
to
induce
any
sort
of
real
world
outage,
and
that
is
where
the
hypothesis
comes
into
play
to
create
the
various
real
world
events.
That's
the
step
three.
A
Where
you,
then,
you
know,
there's,
let's
say
a
network
connection
problem
or
there's
a
service
crash
down
or
there's
issue
with
scaling.
I
mean,
for
example,
the
biggest
example
that
we
can
take
here
in
india
is
an
example
of
a
big
billion
day,
sale
or
or
let's
say
the
great
indian
amazon
festivals
that
happen
during
the
festivities,
where
websites
like
amazon,
flipkart
and
other
e-commerce
platforms
just
give
out
sales
and
there's
a
huge.
A
I
mean,
there's
a
huge
demand
for
of
users
who
come
in
and
and,
and
you
know,
the
the
applications
need
scaling.
They
need
to
act
according
to
the
number
of
users
or
even
if
there
are
a
lot
of
transactions
happening
at
that
point
of
time,
it
needs
to
act
properly,
but
that
is
introducing
a
real
world
event
or
a
possibility
of
an
outage
right.
A
And
then
you
know
you,
you
keep
on
testing
the
system
and
let's
say
if,
if
the
systems
still
are
resilient
after
that
fault,
then
you
go
on
to
a
new
experiment,
a
new
sort
of
a
real
world
in
a
scenario,
and
you
keep
on
testing
until
you
understand
the
systems
are
resilient
and
it
goes
on
in
a
continuous
loop.
But
let's
say
this:
the
system
is
not
resilient
and
there
is
a
fault
as
an
outage.
The
system
goes
down.
A
So
again
you
you
move
on.
You
fix
that
system,
you
fix
that
weakness
and
then
you
introduce
that
fault
again.
So
it
goes
on
a
loop,
and
that
is
where
the
step
four
comes
on,
where
you
try
to
disprove
of
the
hypothesis
by
looking
for
a
difference
in
the
steady
state
between
the
initial
stage
and
and
the
later
stage,
that
is
the
real
world
scenario.
So
these
are
the
steps,
as
I
mentioned
they
are,
they
are
just
briefed
out.
You
vary
real
world
events,
you
run
your
experiments
in
production
and
then
you
automate
them.
A
I
mean
obviously
chaos
engineering
today,
the
the
the
ecosystem
of
chaos.
Engineering
is
such
that
people
are
running
them
manually,
chaos
engineering
is
being
run
manually
and
folks
are
still
focusing
on
that,
but
the
eventual
goal
is
obviously
automating
them.
You
don't
want
to
run
them
at
a
particular
time
or
you
don't
want
to
run
a
particular
experiment
according
to
your
need
and
run
them
manually.
Rather,
you
want
to
automate
them
and
the
experiments
should
run
according
to
the
need
of
your
application.
They
should
be
scheduled
through
the
interface
itself.
A
So
obviously,
chaos
engineering
helps
build
automation
into
the
system
and
the
last
goal
is
obviously
minimizing
your
blaster.
It
is
because
experimenting
in
production
has
its
own.
You
know
demerits
cons.
There
is
a
possibility
that
it
can
cause
you
unnecessary
pain
unless
a
negative
impact,
and
that
is
one
of
the
reasons
why
chaos
engineering
people
are
skeptical
to
adopt
chaos.
They
are
scared
of
chaos,
engineering
and
they
are
not
ready
to
go
ahead
and
ensure
that
you
know
chaos.
A
Engineering
comes
in
as
a
practice,
but
it
is
the
responsibility
of
an
obligation
of
of
the
engineer,
practicing
chaos
to
ensure
that
you
minimize
the
brass
radius.
You
contain
the
experiments
and
they
are
ex.
I
mean
experimented
in
a
careful
environment
so
that
chaos
engineering
in
spite
of
being
a
powerful
practice
because
becomes
a
boon
rather
than
a
vein,
so
going
back
to
the
slide
deck
that
we
have
we,
as
I
mentioned,
these,
are
the
steps
in
which
chaos,
engineering
or
cloud
native
chaos.
Engineering
is
practice.
A
You
identify
the
steady
state
of
the
system,
you
introduce
a
fault
if
the
slos
I
mean
sres
were
the
initial
persona
that
started
introducing
chaos
in
the
system.
So
as
an
sre,
you
need
to
ensure
that
your
slas
or
your
I
mean
slos
slis-
are-
are
managed.
They
are
continued
to
be
met.
So
if
they
are
continued
to
be
met,
then
obviously
your
systems
are
resilient,
but
let's
say
there's
a
possibility
or
there's
a
weakness
that
is
found.
Then
your
your
systems
are
going
to
be.
A
B
B
As
part
of
chaos
engineering,
we
basically
abstract
the
entire
chaos
engineering
process
into
four
simple
steps,
so
these
are
basically
those
four
steps
that
we
are
trying
to
cover.
First
of
all,
you
basically
identify
your
steady
state
condition.
This
is
only
on
a
logical
understanding
basis.
I
mean
this
is
not
part
of
the
actual
litmus
chaos
tool.
B
That
is
something
we
would
not
want,
because
that
makes
us
that
makes
it
more
likely
for
the
cluster
to
go
down
in
the
production,
so
we
basically
create
that
kind
of
a
fault.
We
inject
that
fault
within
our
let's
say
our
kubernetes
cluster,
and
then
we
basically
match
the
outcomes
of
the
chaos
injection
that,
whether
even
after
the
chaos
injection
our
cluster
is
basically
let's
say,
is
able
to
return
to
its
original
form
or
not
like.
B
Let's
say
if
we
are
deleting
the
two
parts,
then
throughout
those
spots
like
is
there,
let's
say
we
have
some
kind
of
a
deployment
here.
So
is
that
deployment
still
reachable
while
we
are
deleting
a
certain
percentage
of
the
plots?
This
is
one
of
the
experiments
that
we'll
be
actually
doing
today.
So
that's
how
you
basically
go
on
testing
by
introducing
the
faults.
That
is
the
idea
behind
chaos
and
generating
so
think
of
it
as
a
vaccine.
B
What
we
do
in
case
of
a
vaccine
is
basically
we
inject
some
kind
of
a
fault
to
make
our
bodies
more
resilient
against
certain
kind
of
a
virus
or
a
certain
kind
of
weakness
that
we
might
be
affected
in
the
future.
So
here
we
introduce
a
fault
in
order
to
make
in
order
to
understand
what
can
basically
go
wrong
in
the
production
if
there
is
some
faulty
condition
or
any
kind
of
a
weakness
that
is
in
our
cluster,
so
that
is
the
idea.
C
Yeah
yeah
yeah,
which
I
just
I
wanted
another
question
I
was
having
is
that
is
there
been
so
see?
I
know
that
I
can
reduce
the
number
of
pod
or
some
other
thing,
and
we
see
that
how
the
stable
the
system
is,
but
either
the
tool
or
is
there
any
guidelines?
Actually?
Okay,
these
are
the
things
we
do
supposed
to
do
actually,
so
that
to
check
the
resiliency
of
the
system
or
is
there
anything
like
that
or.
B
B
Okay,
so
this
is
actually
the
expertise
of
a
devops
engineer
or
even
a
chaos
engineer
you
would
say
who
is
on
the
forefront
of
designing
these
chaos
workflows,
which
can
actually
reveal
the
actual
kind
of
weaknesses
that
a
system
might
have
what
litmus
kiosk
provides
are
experiments.
You
can
think
of
experiments
as
the
building
blocks
of
the
chaos
workflows.
B
You
can
combine
different
kind
of
experiments
over
a
chaos
workflow
that
can
actually
expose
these
weaknesses,
but
as
far
as
it
goes
that
which
experiments
that
you
should
be
actually
practicing
and
which
not,
that
is
something
dependent
on
a
case-by-case
approach.
In
fact,
it
is
very
much
recommended
that
we
should
always
take
each
chaos
engineering
scenario
on
a
case-by-case
approach
to
get
the
most
out
of
a
to
get
the
most
results
out
of
chaos.
B
Experimentations
see
the
thing
is
that,
like
whenever
we
are
performing
these
chaos
experiments
it's
like
we
are
exposing
a
new
set
of
weaknesses
every
time,
because
the
conditions
are
always
changing
and
it's
like
always
making
sure
that
we
are
hitting
the
right
kind
of
weaknesses
that
we
are
trying
to
find.
What
I
mean
to
say
is
that
let's
say
you
have
a
certain
number
of
pots
in
your
application.
B
Now,
if
you
have
that
many
parts
as
part
of
a
deployment,
if
you
decide
to
kill
all
of
them
at
once,
then
certainly
your
service
is
going
to
go
down
or
basically
your
let's
say
your
end
point
will
not
be
reachable.
So
that
is
something
you
would
not
want,
of
course,
but
you
need
to
basically
be
judgmental
of
what
is
a
real
life
scenario
which
can
actually
affect
our
systems.
It
can
be
anything
from
a
simple
dns
to
something
like
a
network
chaos.
A
Sure
nilanjin
thanks
a
lot
krishna
for
your
questions
I
think
moving
on.
We
will
cover
them
as
well
with
the
project
and
with
the
demo
and
and
talking
about
the
project
as
well.
So
I
think,
with
with
this,
we
we
talk
about
how
chaos
engineering
is
practiced.
Today
I
mean
folks
who
are
practicing
already,
might
get
an
overview
of
of
how
chaos
engineering
is
typically
done,
but
this
is
not
how
kiosking
is
supposed
to
be
done.
All
this,
while
I
mean
game
days
is,
is
I
think,
one
of
the
most
important
ways
of
practicing
chaos.
A
This
is
a
must
be
folks
today
are
conducting
game
days.
That
is
nothing,
but
you
know
running
a
chaos,
engineering
test
suit
or
or
some
multiple
tests
workflows
according
to
the
application
to
just
understand
application,
behavior
or
just
to
understand
how
how
the
system
will
run
when
chaos
is
induced
in
in
a
system
in
production,
and
that
is
where
the
findings
are
noted
down.
A
The
findings
are
are
taken
in
our
consideration
to
to
basically
identify
if
the
chaos
engineering
tool
or
a
particular
project
is
is
something
that
that
can
be
used
or
not,
but
but
other
than
that
it's
it's
still
rarely
integrated
into
the
ci
cd.
Although
chaos
engineering
is
must
for
the
ci
cdfs
system,
let's
say
there
are
adopters
in
form
of
talent
who
are
using
chaos,
engineering
in
the
ci
pipelines.
A
They
are
using
here,
chaos
litmus
cures
with
jenkins,
but
but
as
of
now,
it's
still
rarely
integrated
into
the
cics
of
the
system
and
there's
a
belief
that
only
the
sre
persona
is
is
something
is,
is
the
one
who
can
practice?
Chaos,
engineering,
but
that's
not
the
case,
I
mean
being
part
of
the
community
since
the
last
two
years.
The
developer
persona
has
also
been
engaging
in
chaos.
Engineering.
A
A
For
for
the
testing
purposes,
everyone
is,
for
I
mean
qa
engineers.
There
are
so
many
different
personas
that
are
slowly
slowly,
starting
to
use
chaos,
engineering,
but
that's
the
current
scenario
and
obviously
there's
manual
planning
and
execution
behind
it.
I
mean
a
man,
as
I
said,
it's
a
lot
of
manual
labor
that
is
involved,
but
slowly
slowly,
that
the
the
steps
are
being
taken
towards
automating
this,
and
you
know
everything
becomes
part
of
an
automated
process.
A
I
mean
observability
is
not
a
commodity.
You
you
need
to
observe
your
experiments.
You
need
to
find
out
how
your
experiments
are
run.
What
is
going
down
how
your
systems
are
behaving?
You
need
to
check
the
charts,
the
graphs
which
which
show
in
where
is
your
just
there,
where
there
is
a
spike
when
there
is
a
possible
outage
where
there
is
a
down
time,
there
is
an
uptime.
What
what
exactly
is
happening
so
each
and
every
enterprise
obviously
has
a
customer
built
stack
and
any
of
someone
are
using.
A
Dynatrace
someone's
using
data
dog
but
observability
is,
is
still
key
and
and
people
want
to
check
out
their
logs.
They
want
to
manage
and
monitor
what
exactly
is
happening.
They
want
to
manage
them
all
together
and
slowly
slowly
it
that
that
is
becoming
the
part
of
the
test.
So
it's
a
chaos.
Engineering
doesn't
just
come
with
running
these
tests
and
finding
out
the
fault,
but
it
also
comes
along
with
monitoring
them.
A
Checking
out
the
logs
checking
out
what
exactly
is
happening
in
the
system
and
obviously
there
was
a
custom
measurement
process
to
manage
and
see
how
reliability
is
functioning,
but,
with
you
know,
scaling
and
githubs
coming
in,
as,
as
you
know,
helping
to
scale
these
systems,
I
think
the
measurement
process
has
has
absolutely
changed,
and
now
chaos
engineering
is
coming
out
of
all
these
practices,
and
rather
these
practices
are
being
reserved
reversed.
Where
I
mean
it's
integrated
into
the
cicd
of
the
system,
developers
are
also
jumping
in
to
practice.
Chaos.
Automation
is
taking
place.
A
The
the
uis
of
various
chaos
engineering
tools
out
there
are
helping
automate
these
these
tests,
observability
is
becoming
a
must.
Everyone
is
integrating
their
chaos,
engineering
tools
with,
let's
say
prometheus,
or
I
mean
I
mean
they
are
visualizing
them
on
the
grafana
dashboards
or
on
the
data
dog
dyna
trace.
These
platforms
are
using
chaos
in
a
huge
way
at
the
measurement
process
to
find
the
resiliency
scores
to
basically
find
out
the
reliability.
Scalia
systems
have
also
also
become
great,
with
with
kit,
ops
and
all
these
practices
coming
in.
So
these
are
the
foundation.
A
I
mean
founding
principles
of
kiosk
or
cloud
native
chaos.
Engineering,
as
we
talk
about,
we
believe
in
open
source.
I
mean
the
open
source
is:
is
the
future?
It's
the
key
to
develop
chaos,
engineering.
We
need
to
identify
inputs
coming
in
from
from
the
community.
We
need
to
add
experiments
that
come
in
from
the
community.
Let's
say
from
the
banking
sector,
we
have
learned
a
lot
from
the
use
cases
from
the
e-commerce
sector
from
the
automobile
sector.
A
Every
other
sector
or
enterprise
has
a
different
use
case
has
a
different
input
that
can
come
into
chaos,
engineering
or
cloud
native
chaos,
engineering,
and
that
is
what
we
have
believed
in
with
that.
We
have
built
the
chaos
hub.
So
let
me
just
take
you
through
the
chaos
hub,
I
mean
we
could
have
introduced
that
before
after
introducing
litmus,
but
just
understand
what
chaos
hub
is
it's
it's
the
marketplace
where,
where
you
know
you
can
find
all
the
different
experiments
that
you
want
to
run.
I
mean
your
infrastructure
is
based
on
kubernetes.
A
That
is
what
litmus
started
with
it
was
based
on
a
kubernetes.
It
was
a
kubernetes
community
chaos,
engineering
platform
and
slowly
slowly,
we
shifted
to
other
non-kubernetes
scenarios
as
well
aws.
You
can
test
your
applications,
kafka,
kodi
and
s
cassandra.
You
can
test
on
cloud.
You
can
test
your
other
cloud
native
applications
just
as
open
abs
and
there's
so
much
more
coming
out,
based
on
cross,
plane,
witness
and
and
open
policy.
A
So
there
are
so
many
applications
that
need
to
be
tested,
but
as
of
now,
you
have
got
tests
for
your
vms,
your
azure
that
so
so.
That
is
what
a
chaos
hub
is,
and
that
is
something
that
is
contributed
by
the
community.
The
community
comes
in
tells
us
what
sort
of
experiments
do
they
need?
They
contribute
themselves.
They
make
these
experiments
better.
A
What
can
be
done
in
terms
of
documentation
or
in
terms
of
the
experiment
run
itself,
so
that
is
what
the
chaos
have
became:
a
marketplace
for
all
these
experiments,
where
you
can
pull
one
of
these
experiments
to
run
them
in
your
application
or
create
a
chaos
workflow
and,
as
the
launcher
mentioned,
the
workflow
is
nothing
but
different
chaos,
experiments
running
in
parallel
or
series.
According
it's
a
chaos,
engineering
test
suit
done
according
to
to
your
needs
or
the
needs
of
your
application
and
observability.
A
As
I
mentioned
it's
key
you,
you
pull
in
your
metrics
for
your
prometheus
metrics
and
visualize
them
on
your
dashboard.
It
can
be
a
grafana
dashboard
or
any
other
dashboard.
When
you
visualize
and
again,
you
need
to
visualize.
So
that's
that's
important
and
then
there's
git
ops.
So
you
need
to
you
need
to
scale
your
experiments.
You
need
to
identify
how
I
mean
from
where
you
can
pull
in
your
experiments,
how
you
can
pull
in
experiments
from
different
scenarios.
A
A
A
It
has
an
amazing
ui,
which
is
called
the
chaos
center,
which
helps
you
visualize
your
experiments,
schedule
them
run
a
chaos
workflow
identify
the
reliability
of
the
system,
run
the
chaos
probes.
There
is
so
much
to
it
and
it's
it's
an
amazing.
It's
a
popular
project
which,
which
is
really
popular
in
the
community.
There
are
1200,
plus
plus
slack
community
members
who
have
been
running
litmus.
It
started
off
way
back
in
2018
and
since
then,
in
this
four
year
journey
it
has
become
a
project
which
has
been
adopted
by
several
organizations
from
eco
commerce
giants.
A
As
its
open
source,
so
where
in
the
devops
sys
pipeline
or
where
in
the
devops
ecosystem,
you
can
use
litmus
cures,
a
lot
of
folks
believe
that
it
can
just
be
used
in
the
production
environment,
but,
as
I
mentioned,
it
can
be
used
everywhere.
You
can
use
it
in
your
ci
pipelines
in
a
pre-staging
environment,
for
your
long-running
tests
in
your
pre-prod
and
as
well
as
your
production
environment.
A
So
everywhere
you
can
induce
chaos
in
some
form
or
for
faults
in
some
form,
and
most
of
the
folks
are
actually
starting
off
with
the
pre-staging
or
staging
environments.
That
is
what
we
have
learned
from
the
community
and
then
moving
on
to
production.
So
these
are
some
project
stats.
That's
just
for
your
reference,
how
it
is
grown
and
you
can
see
the
chart
it's
grown
exponentially.
There
are
50
plus
experiments
that
are
out
there
available
on
the
kia
sub
almost
reaching
500
000
operator
installs
and
more
than
850
000
experiment
runs.
A
Then
there
must
be
more
who
who
knows?
But
this
is
the
project
growth.
As
you
can
see,
I
mean
by
2020
it
was
just
starting
to
grow
and
we
hadn't
even
reached
100.
000
experiment
runs,
but
today
it
has
just
grown
to
reach
somewhere
around
900
000
experiment
runs,
and
that
is
what
has
helped
this.
This
project
become
a
go-to
chaos,
engineering
tool,
something
that
can
be
talked
about
in
workshops
and
other
other
places
as
well
in
the
community.
A
So
we
we
are
now
in
the
2.7
version
of
litmus
chaos
and
litmus
kiosks
I
mean
nilanjin
will
be
covering
the
features
and
everything
about
litmus.
But
this
is
what
is
exciting
about
litmus
chaos
or
the
current
space
of
litmus
chaos,
where
the?
What
are
the
characteristics
or
what
are
the
the
functionalities
that
are
provided
to
you?
I
mean
there
are
a
lot
of
features
in
one
dot
x.
That
is
the
chaos
hub
which
which
which
allows
you
to
I
mean
bring
in
predefined
experiments.
A
You
don't
need
to
write
them,
but
litmus
one
dot
x
also
had
an
sdk
which
helped
you
write
your
own
experiment.
So
let's
say
there
is
some
experiment
which
is
specific
to
your
application
or
which
you
want
to
induce
to
your
application,
and
that
is
not
available
on
the
marketplace
or
the
chaos
hub.
Then,
using
the
sdk
which
is
available
in
go
and
python.
A
You
can
write
your
own
experiment,
you
can
you
can
just
write
something
specific,
you
can
contribute
it
back
to
the
community
and
you
can
use
it
by
yourself
and
let's
say
there
are
other
chaos.
Engineering
tools
out
there
like,
for
example,
chaos
toolkit,
and
there
is
some
experiment
that
is
available
with
chaos
toolkit,
but
not
with.
Even
you
can
use
that
with
the
litmus
framework,
so
you
can
basically
bring
in
that
experiment
and
use
it
using
litmus
chaos.
A
So
there
were
some
amazing
features
which
one.x
provided
in
terms
of
security
resiliency
in
terms
of
user
being
user
friendly.
But
here
are
some
other
exciting
features.
I
mean
you,
you
get
a
user
management
and
teaming
process.
You
just
don't
have
to
use
it
for
one
member,
but
your
whole
team,
as
in
I
mean
if
there
are
five
folks
in
your
team
or
there's.
You
know
a
company
full
of
folks
which
want
to
use
chaos,
then
there's
a
there's.
An
ability
for
for
teams
to
use
the
same.
A
The
chaos
center
is
the
ui
which
helps
make
it
easy.
You
don't
need
to.
You,
know,
run
or
type
out
a
lot
of
experiments,
just
copy
paste,
those
xaml
files
or
run
them
one
by
one,
but
the
chaos
center
helps
you
create
the
workflows
easily
and
there's
a
resilience
score
calculation
which
helps
you
calculate
the
resiliency
score.
How
resilient
your
systems
are?
How
well
the
probes
run.
The
probes
are
nothing
but
the
custom
health
checks
which
help
identify
the
reliability
of
the
resiliency
of
your
systems.
A
There
is
monitoring
and
observability
which
is
available
in
built
in
the
chaos
center.
You
can,
where
you
can
just
pull
out
your
metrics
and
then
you
can
observe
what
is
exactly
happening
in
the
system.
The
workflow
editor
helps
you
edit,
your
workflows,
you
can
create
them
in
series
or
parallel.
The
githubs
functionality
helps
you
scale,
your
systems,
bring
in
your
own
experiments
or
run
them
accordingly,
and-
and
there
are
so
so
many-
I
mean
abilities
that
that
nilanjin
is
gonna
cover
moving
ahead.
A
But
this
is
an
a
quick
overview
of
what
chaos
engineering
was
and
what
exactly
lit
mischios
is
in
in
in
layman's
definition,
and
with
this
I
think,
we'll
cover
more
on
the
architecture,
we'll
cover
more
on
what
exactly
litmus
is,
and
how
do
you
define
everything?
How
do
you
get
started
with
it
and
I'll
I'll
perhaps
ask
nilanjin
to
take
the
floor
from
here,
but
if
there
are
any
questions,
perhaps
we
can
just
give
a
couple
of
minutes
to
any
sort
of
questions
as
of
now
and
then
the
luncheon
can
move
ahead.
D
A
So
I
mean
there
are
a
lot
of
chaos
tool
sets
out
there
I'll
just
take
a
few
names.
One
of
them
is
gremlins
mentioned
chaos
mesh
it's
another
cncf
project.
There
is,
I
mean,
chaos,
blade
chaos,
monkey
chaos,
tool,
kit,
steady
bait.
I
think
these
are
the
mostly
the
cure
and
anal
atmospheres,
of
course.
So
these
are
mostly
the
famous
tool
sets
out
there
available.
So
I'll
start,
one
by
one
chaos.
Monkey
was
a
tool
that
was
developed
by
netflix
back
in
2012.
A
It
was
just
to
run
some
basic
production,
related
tests,
a
cpu
attack
or
something,
and
since
then,
the
development
of
this
project
or
according
to
the
world,
how
it
has
changed,
how
kubernetes
has
come
up
has
relevantly
been
slow,
so
chaos
monkey
is
something
which
folks
did
not
adopt
with
with
time
or
which,
which
became
a
little
bit
redundant
with
time,
and
then
the
other
tools
are
pretty
much
out
there.
They
are
pretty
much
unique
in
their
own
ways,
as
sandeep
mentioned
about
a
couple
of
tools,
chaos
smash
and
gremlin.
A
So
I'll
talk
about
gremlin
first
gremlin
again
is
an
amazing
tool.
I'm
sure
they
are
doing
a
great
job,
but
the
I
think,
the
the
few
limitations
that
it
has
is
number
one.
It's
a
closed
source
tool,
so
you
you
don't
get
the
benefits
of
being
open
source.
The
I
mean
community
contributions
do
not
come
here,
so
there
doesn't
come
a
lot
of
understanding
with
what
the
community
understands.
Then
there
are
a
lot
of
functionalities
which
are
which
gremlin
might
not
provide
you.
A
Let's
say
let's
say
the
option
of
I
mean
probes
or
let's
say
the
chaos
hub.
There
is
no
marketplace
that
gremlin
has,
or
there
is
no
custom
health
checks
that
gremlin
provides,
but
I'm
sure
gremlin
also
has
certain
things
better.
I
mean
in
terms
of
experimentation,
maybe
learnings
from
the
community
that
that
you
know
close
source
project
takes
from
take
from
their
customers,
and
you
know
certain
attacks,
let's
say
black
hole,
attacks
or
some
attacks
and
similarly
kiosk
mesh.
I
think
it
provides
the
benefit
of
being
open
source.
A
It's
it's
a
project
which,
which
runs
which
it
runs,
as
I
mean
simultaneously,
which
is
litmus
chaos.
It's
also
part
of
the
cncf
space
and-
and
there
are
so
many
things
which
differentiate
litmus
and
cures
mesh
altogether.
Again,
as
I
said
probes,
the
the
marketplace,
chaos
hub-
I
mean
git,
ops
and-
and
these
are
a
few
things
which,
which
the
kiosk
mesh
might
not
have
but
again,
chaos
mesh
might
have
certain
experiments,
let's
say
jvm
chaos
or
http
chaos,
which
which
are
yet
to
be
a
part
of
the
litmus
chaos.
A
I
mean
chaos,
which
are
yet
to
be
part
of
the
litmus
cures
test
suit,
but
I
mean
every
chaos
tool
set
has
its
differences.
Some
are
some
tools
are
great
with
vms
I
mean,
if
you
run
litmus
with
kubernetes,
then
I
don't
think
you'll
have
a
complaint
here
and
there,
if
you're
in
chaos
mesh
with
with
a
particular
application,
then
it
is
great,
but
I
think
it's
it's
all
about
identifying
what
suits
best
to
you
and
it's
also
about
the
community.
I
think
I
mean
being
part
of
the
community.
A
I
can
say
more
than
the
tool.
I
think
what
is
important
is
community
support.
I
mean
you
can
run
a
tool,
but
chaos.
Engineering
is
new
for
everyone,
everyone
is
just
getting
started
with
chaos,
and
a
lot
of
help
is
necessary
in
in
terms
of
community
support
for
for
someone
to
get
started
with
chaos,
if,
if
they
are
not
opting
for
you
know
an
immediate
enterprise
support
or
something
of
that
sort
which
enterprise
support
providers
are
already
giving
you
so
so,
in
terms
of
that,
I
think
litmus
chaos
becomes.
A
standout
of
the
community.
A
Support
is
something
which
comes
into
play.
The
various
use
cases
are
something
that
everyone
in
the
community
learns
from,
and
if
you
are
a
part
of
it,
you
will
get
to
know
that
how
how
the
community
drives
this
project
to
to
help
you
understand
the
various
functionalities,
get
you
started
quickly
and
and
make
it
easier
for
you
to.
You
know,
induce
chaos
or
bring
in
chaos
as
a
practice,
and
these
are,
I
think,
a
rough.
A
I
mean
difference
between
these
two
is,
if
you,
if
you
go
into
the
depth,
I
think
there's
a
lot
out
there
in
the
community,
but
I
think
you'll
have
to
use
each
and
every
tool
a
little
bit
deeply
or
have
a
poc
on
on
all
of
them
to
to
basically
identify
which
is
the
best
tool
for
you.
I
hope
this
answers.
Your
questions.
Okay,.
D
Thanks
yeah,
that's
that's
thanks
for
the
update
yeah
and
one
more
thing
I
need
to
shout
since
I'm
using
litmus
from
past
two
months
right.
I've
been
very
impressive
with
the
community
and
the
slack
response
how
I
get
so
in
that
case,
I'm
quite
happy
at
least
to
start
exploring
and
taking
it
to
my
organization
as
such.
D
And
one
more
thing,
since
we
have
a
very
strong
isd
team,
the
recent
one
from
incubation
state
to
the
sorry
sandbox
for
the
incubation
state,
which
was
helped
us
a
lot
for
getting
approval
for
isg
to
go
to
the
stage
and
card
environment.
For
sure.
That
was
a
very
good
from
the
litmus
and
inducer's
perspective.
A
Thanks
a
lot
sandeep,
I
think,
with
the
incubation
we
obviously
are
focusing
on
graduation
now,
but
but
obviously
it's
a
long
journey,
but
I
think
with
incubation
there
will
be
a
new
road
map
which
will
be
coming
into
place
and
open
source
I
mean
litmus.
Chaos
is,
is
going
to
see
a
lot
of
contributions
coming
in
and
I
hope
sandeep
from
your
end
as
well.
We
will
see
some
amazing
contributions
both
on
the
technical
side,
the
toolset
side,
as
well
as
on
the
community
side
with
blogs,
videos
and
anything
that
is
possible.
A
B
B
Yeah,
thank
you
so
much
ritu
all
right
folks.
So
let's
try
to
deep
dive
a
bit
into
what
enables
litmus
chaos
to
become
such
a
awesome
chaos,
engineering
tool
set.
What
I'll
do
is
basically
lead
you
through
this
architecture
of
the
electric
skiers.
I
will
try
to
keep
it
less
verbose
so
that
you
just
give
a
get
a
gist
of
what's
under
the
hold
of
litmus
cures,
so
the
entire
applica
application,
or
like
the
entire
architecture
of
the
literacy
chaos,
can
be
divided
into
two
sections.
B
First,
is
the
chaos
control
plane
and
the
second
is
the
execution
plane.
So
what
you
interact
with
is
the
chaos
control
plane.
That
is
the
front
facing
user
component
in
which,
with
which
the
user
usually
interacts
in
order
to
either
schedule
their
own
chaos,
experiments
or
chaos,
workflows
or
configure
their
observability
dashboards,
so
on
and
so
forth.
So
every
usually
everyone
has
one
chaos
control
plane
using
which
they
can
control
multiple
chaos
execution
planes.
B
What
I
mean
is
say
that
you
are
installing
little
scars
into
your
own
kubernetes
cluster.
Of
course
you
have
a
self
agent
over
there.
That
is,
you
can
inject
chaos
into
within
that
particular
cluster,
but
at
the
same
time
you
can
also
inject
chaos
into
other
kubernetes
clusters
as
well.
That
is
something
is
enabled
using
the
litmus
agent.
B
So
the
led
bus
agent
enables
the
installation
of
liters
execution
plane
components
within
your
external
kubernetes
clusters,
and
then
we
can
connect
those
agents
using
our
chaos
control
plane
in
order
to
inject
chaos
within
them
as
well.
So
moving
ahead
within
the
chaos
control
plane,
you
can
see
that
we
have
first
of
all
our
kiosk
center.
The
chaos
center
is
a
combination
of
three
components
that
we
have.
First
is
the
the
web,
ui,
of
course,
which
we
use
in
order
for
basically
injecting
chaos
into
our
kubernetes
components.
B
Then
you
have
the
litmus
api.
The
electronics
api
also
does
the
same
thing.
You
can
use
the
litmus
api,
the
graphql
api
in
order
to
inject
chaos
into
your
various
kubernetes
components
using
the
api,
and
then
you
have
liquid
ctl
that
was
ctl
is
a
command
line,
interface
for
injecting
chaos
in
a
declarative
manner
or
basically,
com
or
basically
controlling
few
aspects
of
your
chaos
center.
Using
your
cli
so
the
way
it
does
all
this
is
using,
of
course,
an
odd
server
as
well
as
a
graphql
server.
B
So
the
graphql
server,
then,
is
in
communication
with
the
different
chaos
execution
planes
and
if
you
go
inside
the
chaos
execution
place.
The
very
first
thing
that
we
see
is
still
it
was
agent
infra.
So
what
is
the
litmus
agent
in
travel,
as
I
mentioned,
that
litmus
agent
is
responsible
for
setting
up
the
chaos
execution
plane
and
within
this
agent
we
get.
First
of
all,
the
most
important
part
is
the
subscriber.
B
The
subscriber
is
what
sets
up
the
entire
agent.
It
is
responsible
for
communicating
the
messages.
The
logs
the
error,
messages,
etc
back
to
the
graphql
server
and
ensuring
that
the
server
is
in
a
healthy
state
whenever
you
are
restarting
your
kubernetes
cluster
or
basically,
whenever
you
are
setting
up
your
kubernetes
cluster.
B
So
while
this
subscriber
is
there,
there
is
also
something
known
as
a
workflow
controller,
the
workflow
controller
or
the
argo
workflow
controller
is
basically
the
it
is
basically
a
controller
loop,
a
kubernetes
controller
which
looks
out
for
the
workflow
custom
resource.
Now,
what
is
the
workflow
custom
resource?
B
Well,
what
happens
when
you
create
a
kiosk
workflow
from
your
chaos
center
is
that
it
is
nothing
but
a
long
manifest,
which
is
a
argo
workflow
custom
resource
so
that
workflow
custom
resource
manifest
is
simply
passed
on
by
the
graphql
server
to
your
chaos,
execution
plane,
where
the
workflow
controller
reconciles
the
state
of
your
kubernetes
cluster
according
to
what
has
been
defined
inside
your
workflow
custom
resource.
So
let's
say
that
you
have
your
different
chaos
execution
steps
inside
it.
You
have
two
experiments
or
three
experiments
in
serial
or
parallel.
B
All
that
can
be
actually
be
mediated
by
the
workflow
controller.
Don't
worry
if
you're
not
completely
able
to
grasp
it
we'll
be
seeing
a
work,
a
hands-on
demo
of
the
same
in
a
short
while
so
after
we
are
basically
getting
our
workflow
custom
resource,
we
are
having
the
different
litmus
chaos.
Custom
resources,
such
as
the
chaos
experiment
and
the
kiosk
engine
chaos.
Engine
is
responsible
for
remediating
the
step
of
the
kubernetes
cluster,
which
is
being
actually
done
by
the
kiosk
operator.
B
That
is
how
to
basically
perform
that
experiment
and
that
is
being
reconciled
by
our
chaos
operator
and
the
chaos
experiments
here
contains
the
low
level
details
of
these
experiments,
including
what
is
the
image
that
is,
we
are
using
with
our
experiment
or
like
what
is
the
environment
and
variables
that
we
are
using
in
order
to
tune
our
experiment
so
on
and
so
forth,
so
yeah.
This
is
a
basically
a
brief
overview
of
the
litmus
backend
execution
infra.
Where
the
actual
experiment
execution
happens.
B
We
left
out
on
two
important
components
in
the
agent
infra
as
well.
First
is
the
event
tracker
okay.
So
what
is
an
event
tracker?
Well,
the
event
tracker
helps
you
to
declaratively,
define
your
ideal
state
of
a
kubernetes
cluster,
in
which
you
would
like
to
trigger
a
chaos
workflow.
So
it
works.
Something
like
this.
You
basically
write
a
event
tracker
manifest
this
event.
Tracker
manifest
will
continuously
look
out
for
the
current
state
of
the
kubernetes
cluster.
B
So
let's
say
that
at
any
point
of
time
your
kubernetes
clusters,
conditions
or
basically
specifications
matches
that
what
has
been
defined
in
our
even
tracker
like
for
an
example
in
a
particular
name
space
for
a
particular
deployment.
We
should
have
a
certain
amount
of
parts.
Only
then
execute
this
workflow,
so
this
kind
of
a
condition
can
be
defined
using
the
event
tracker.
B
Finally,
we
have
the
kiosk
exporter
and
the
kiosk
exporter
is
being
used
by
the
prometheus
in
order
to
fetch
the
different
metrics
of
chaos
of
the
applications
or,
basically,
the
chaos
of
the
letters
chaos.
Basically,
while
our
chaos
injection
is
taking
place,
so
don't
worry
I'll,
be
basically
showing
you
how
you
can
use
chaos
exporter
in
order
to
create
the
different
observability
dashboards
for
your
kubernetes
cluster
moving
forward,
we
have
let
us
try
to
deep
dive
a
bit
into
the
different
components
of
retrospect,
what
we
understood
so
far.
B
So,
first
of
all,
you
have
the
chaos
center.
That
is
nothing
but
the
request,
control,
plane
and
the
chaos
agent,
which
is
comparable
to
the
execution
plane.
While
I've
already
explained
the
significance
of
both
of
them
again,
the
chaos
center
is
the
front-facing
user
component,
from
which
you
control
all
the
activities
happening
inside
your
litmus
chaos.
In
order
to
inject
chaos
workflows
into
your
kubernetes
cluster.
B
Well,
chaos
agent
is
responsible
for
setting
up
your
your
execution,
infra
and
within
your
kubernetes
clusters
moving
forward,
we
have
the
little
ctl
so,
as
I
explained
it
now,
ctl
is
nothing
but
the
cli
for
atmospheres
2.0.
It
is
used
for
setting
up
the
various
aspects
of
that.
If
you
are
setting
up
your
litmus
care
center,
it
can
help
you
by
helping
you
set
up
the
agent
and
do
a
few
user
management
stuff
so
on
and
so
forth.
B
Moving
forward,
we
have
the
multi-cloud
support
with
litmus
chaos
2.0
as
well.
So
what
it
means
is
that
you
can
basically
install
atmospheres
into
any
of
these
clouds.
Basically,
any
cloud
that
has
kubernetes
you
can
install
litmus
into
it
using
your
little
ctl
as
with
a
chaos,
control
pin
or
even
as
a
external
agent
using
the
electrical
ctl.
B
So
let's
take
it.
Let's
take
a
deep
look
into.
What
are
the
core
features
that
are
being
offered
by
the
litmus?
So
these
are
again
the
same
features
that
we
have
been
discussing
right
now,
but
I'll
shortly
go
over
each
one
of
them.
Give
you
a
brief
of
what
they
look
like
and
what
is
the
particular
use
of
that
feature
within
the
litmus
starting
with
the
chaos
center.
You
have
cure
chaos
workflows.
B
So
when
I
say
chaos
workflows,
think
of
chaos,
experiments
first,
the
chaos
experiments
are
nothing
but
the
building
blocks
for
chaos,
workflows
and
chaos.
Experiments
by
themselves
represent
a
certain
type
of
a
failure
which
you
can
inject
in
your
target
resource.
So
you
can
use
all
these
different
blocks,
in
sequence,
in
a
particular
sequence,
whether
that
is
serial
or
parallely
running
experiments
to
create
complex,
real-world,
real-world
chaos,
scenarios
which
are
known
as
chaos
workflows.
So,
although
it
sounds
complex
but
believe
me,
it
is
a
very
simple
process
that
can
be
actually
be
achieved.
B
Using
your
chaos
center
as
I'll
show
you
in
a
while,
then
you
have
the
option
for
creating
these
chaos
workflows
from
templates
or
from
using
custom
workflows
directly
from
scratch.
By
using
the
experiments
from
chaos,
herbs
or
even
pre-created
emails,
you
can
directly
upload
next,
you
have
the
different
control,
that
is,
the
kiosk
experiment.
Sequence
control.
As
I
mentioned,
you
can
have
the
experience
in
serial
or
parallel.
You
have
the
choice
of
creating
your
own
defined
chaos
scenario
using
this
electroscopes.
B
Finally,
you
have
the
schedules,
so
you
can
also
run
singular
workflows
or
you
can
set
them
as
clone
workflows
so
that
they
will
keep
on
running
based
on
a
particular
schedule.
Finally,
you
have
the
experiment
priority
that
is,
while
calculating
the
resiliency
score
as
per
the
outcome
of
every
single
experiment.
In
your
chaos
workflow,
you
can
attach
weightage
to
a
particular
experiment
so
that
let's
say
you
want
to
provide
more
weightage,
more
importance
to
a
particular
experiment
as
compared
to
another.
You
can
set
up
a
priority
or
chaos,
basically
wait
for
it
resiliency
score.
B
Moving
forward.
We
have
the
workflow
management,
so
you
have
githubs
so
get.
Ops
is
basically
rolling
out
automated
changes
using
githubs.
There
is
actually
two
types
of
get
offs
that
we
have
here
in
litmus.
That
is,
first,
you
see
front-end
githubs
and
the
back-end
get-offs
the
back-end
get-offs,
is
what
I
explained
you
earlier.
B
So
what
it
does
is
that
git
becomes
a
single
source
of
truth
for
all
your
workflow
runs,
and
there
is,
if
you
make
any
of
the
changes
inside
your
get
inside
the
manifest
that
is
stored
in
the
get
it
will
help
to
run.
The
automated
changes
run
a
workflow
using
those
automatic
changes
here
in
litmus
as
well.
B
Then
you
can
also
use
your
custom
images.
That
is
let's
say
you
have
a
custom
image
registry
from
where
you
want
to
pull
all
the
different
images
required
for
the
litmus
geos.
You
can
certainly
do
that.
You
also
have
the
option
of
setting
a
resiliency
score
in
order
to
understand
how
resilience
exactly
is
your
kubernetes
cluster
for
a
chaos
workflow.
B
Then
you
have
options
for
multi-tenancy.
As
prithvi
had
explained,
we
had
introduced
multi-tenancy
as
part
of
litmus
geos
2.0.
You
have
scope
support.
That
is.
You
have
option
for
the
control,
plane
and
agents
can
be
run
in
both
in
a
cluster
scope,
as
well
as
name
scope,
name,
space
scope
mode.
So
what
happens
is
in
cluster
scope
mode?
You
can
inject
yours
into
each
and
every
name
space
inside
your
kubernetes
cluster,
while
in
name
squeeze
scope
mode.
Basically,
you
can
inject
chaos
into
only
one
name
namespace.
B
So
this
helps
in
reducing
the
blast
radius
so
that
you
don't
accidentally
affect
your
other
applications.
While
doing
some
chaos,
then
you
have
strong
authentication
support.
Well,
basically,
you
can
using
you,
can
authenticate
your
electric
skills
using
oauth
by
using
dex
support
from
by
you
can
authenticate
with
google
or
github
for
your
teams.
You
can
also
choose
basic
email
password.
That
is
of
course
there.
B
Then
you
also
have
options
for
creating
teams
so
that
let's
say
that
you
want
to
run
a
particular
set
of
chaos
workflows
as
a
team,
you
want
to
explore
the
causes
of
failures
as
a
team,
so
you
can
certainly
do
that.
You
can
create
your
own
team,
invite
your
team
members
accept
the
invitations
and
collaborate
over
your
creating
your
chaos,
workflows
and
exploring
chaos
engineering
with
them.
B
Then
you
have
find
game
our
back
controls.
Well,
what
happens
is
like
flexible.
Our
backs
are
used
to
drill
down
and
grant
correct
privileges
to
the
users,
so
you
have
full
control
over
what
you're
trying
to
do
inside
your
kubernetes
cluster.
Let's
say
that
you
want
your
experiments
to
have
a
certain
permission
or,
let's
say
not,
have
a
certain
permission.
You
can
go
to
the
our
back
for
that
experiment
and
then
change
your
privileges.
B
To
basically
only
have
that
sort
of
permission
and
don't
let
any
additional
permission
apart
from
it
then
comes
three
monitoring
and
absolutely
so.
Observability
is
something
that
is
very
much
important
in
chaos.
Engineering,
because
only
injecting
faults
is
not
enough,
while
you're
doing
chaos
engineering,
you
need
to
be
always
able
to
gain
feedback
from
what
you're
doing
and
understand
that
what
is
making
your
system
slow
or
like
what
is
making
your
system
less
resilient.
B
So
therefore,
absolutely
is
not
a
separate
commodity
in
here
in
litmus.
We
treat
it
as
a
first
class
citizen
and
it
is
inbuilt
as
a
feature
within
the
electric
chaos.
So,
starting
with
what
are
the
different
features
that
you
get
out
of
observability.
First
of
all,
you
can
connect
your
own
data
source.
Currently,
we
only
support
prometheus,
but
in
other,
but
in
the
future,
builds
you'll
be
certainly
seeing
data
dog,
as
well
as
more
data
sources
for
your
for
connecting
different
data
sources.
Then
you
have
the
visualization
capabilities.
B
That
is,
whenever
you're
running
your
workflow,
you
can
see
them
as
aggregated
modules
or
schedules,
basically,
so
that
you
can
make
a
sense
out
of
what
is
working
and
which
step
is
not
working.
Then
you
can
compare
different
workflows.
Of
course,
what
went
wrong?
You
can
find
out
that
using
that,
then
you
can
upload
your
own
dashboards,
like
let's
say
when
I'll
show
you
within
litmuscus
that
how
we
are
creating
interleave
dashboard,
so
you'll
be
able
to
see
that
you
can
create
a
work,
create
your
application,
dashboards
store
them.
B
Then
you
can
again
upload
those
dashboards,
basically,
in
order
to
create
a
seamless
experience
within
your
different
litmus
chaos.
Centers.
If
you
have
multiple
them,
then
you
can
tune
your
dashboard.
So
what
we
mean
by
tuning
your
dashboards
is
that
you
can
have
different
sorts
of
graphs
as
well
as
different
sorts
of
metrics,
that
you
can
monitor
for
using
these
dashboards.
So
monitoring
everything
doesn't
make
sense.
You
can,
therefore
you
can
only
select
those
dashboards
that
mean
the
most
to
you
and
to
monitor
your
chaos
in
the
real
time.
B
Next
is
get
off.
So,
as
I
explained,
there
are
two
types
of
githubs.
First
is
the
kit
as
the
db,
that
is
the
front-end
githubs,
and
then
you
have
the
event
trackers,
which
is
the
back-end
get-offs,
which
can
basically
run
and
if
get
up,
electroscopes
workflow
automatically,
when
a
certain
condition
is
validated
within
your
kubernetes
cluster,
then
you
have
your
non-kubernetes
skills.
Well,
this
is
something
that
we
have
been
working
with
some
vega
since
last
few
versions.
That
is
whenever,
since
our
litmus
chaos
2.0
has
released.
B
B
So
yeah,
basically,
that's
non-kubernetes
chaos
in
a
nutshell
which
will
be
further
exploring
in
today's
demos.
Finally,
here's
a
bit
of
a
diversity
of
the
different
experiments
that
we
have
as
part
of
electric
chaos.
You
have
pod
based
experience,
you
have
node-based
experiments.
You
have
different
network
experiments
within
the
kubernetes
sphere.
You
have
different
kind
of
stress:
chaos,
experiments
that
spans
across
the
pod
and
node
blast
radiuses,
and
then
we
have
different
cloud
service
experiments
such
as
aws
waste
experiment
for
ec2
stoppage
or
like
aws
evs,
detach
disk
detach.
B
Then
you
have
application
bios
based
chaos
as
well,
like
kafka,
based
experiments
or
open
abs,
experiments
and
latest
through
this
addition
is
a
machine
based
experience
as
well,
that
is,
we
go
within
a
vm,
try
to
interact
with
the
os
based
resources,
such
as
processes
or
cpu
of
a
virtual
machine,
so
on
so
yeah
with
that,
let's
talk
about
a
bit
of
the
future
roadmap.
So
of
course
we
have.
B
We
are
committed
to
providing
more
non-kubernetes
infrastructure
components
because,
as
we
understood
non-kubernetes,
it
is
not
always
the
case
that
your
kubernetes
cluster
is
perfectly
cloud
native.
You
might
be
trying
for
an
ideal
cloud
native
solution,
but
you
might
have
certain
legacy
systems
certain
legacy,
bare
metal
clusters
or
your
own
data
centers
that
yeah
that
are
also
right
now
hosting
your
legacy
applications.
So
in
this
case
you
cannot
just
move
to
kubernetes
and
cloud
native
ecosystem.
B
You
still
need
to
validate
your
chaos,
engineering
use
cases
for
your
legacy
applications,
and
that
is
something
non-kubernetes
application
non-community
chaos.
Experiments
will
enable
you
to
do
then.
We
are
also
looking
forward
to
more
application,
specific
kiosk
experiments
with
native
faults
and
health
check,
and
definitely
it's
in
the
road
map.
For,
of
course,
as
we
mentioned,
we
are
looking
forward
to
various
cloud
native
applications
such
as
etc,
in
which
we
are
planning
to
inject
faults
and
bring
them
out
as
chaos
experiments
for
litmus.
B
Next,
we
have
the
kiosk
stk.
Well,
the
kiosk
sjk
is
what
helps
developers
to
write
new
experiments,
so
we
are
certainly
planning
on
to
improve
the
support
for
it.
Just
just
to
mention
like
what
chaos
sdk
is.
Currently,
we
have
it
for
go
python
as
well
as
sensible,
and
it
helps
you
bootstrap
a
new
chaos
experience
so
that,
as
a
developer,
you
are
only
responsible
for
defining
the
core
logic
of
the
experiment
and
nothing
more
than
that.
B
Next,
we
are
also
planning
to
have
more
probe
types.
So
if
you
are
unaware
of
what
litmus
chaos
probes
are
sticks
with
me,
I'll,
be
showing
you
inside
the
network's
portal
that
how
you
can
define
troops
and
what
it
does,
but
in
a
nutshell,
troops
are
nothing
but
a
declarative
way
of
validating
your
chaos
hypothesis
in
an
automated
fashion.
So
let's
say
you
have
an
hypothesis
as
part
of
your
kiosk
workflow.
B
You
can
automate
the
process
of
its
validation
using
these
probes,
so
we
have
different
types
of
probes
already,
such
as
cmd
probe,
http
pro
prom,
probe,
katus
probes
and
we'll
be
adding
more
in
the
future
moving
forward.
We
also
have
the
we
are
also
pledging
for
improved
observability
suit
for
the
kiosk
experiments.
With
the
current
auxiliary
suit,
you
can
definitely
have
into
leave
dashboards
for
your
different
experiments,
but
we
feel
that
that
is
not
enough.
We
need
also
support
for
different,
more
types
of
data
sources.
B
Moving
forward
from
prometheus.
We
also
are
looking
forward
to
integrating
data,
dog
or
other
types
of
data
sources
which
will
be
beneficial
for
us
and
finally,
we
also
have.
We
are
also
pledging
for
more
chaos,
types
that
is
more
chaos
experiments
while
it
is
a
continuous
process,
but
we
still
feel
the
requirement
at
every
meetup
or
at
every
fixed
interval
to
mention
the
different
types
of
chaos,
experiments
that
we
are
working
on.
B
So
that
is
something
that
you
can
certainly
look
forward
to,
so
how
you
can
contribute
well,
there
are
a
host
of
features
that
you
can
contribute
to
when
it
comes
to
litmus
skills,
just
visit
our
cater
slack
and
we
can
set
it
set
you
up
with
what
kind
of
contributions
you
can
do.
Otherwise
you
can
also
visit
to
the
github
pages
or
sorry,
the
github
repository
of
litmus
chaos,
where
you'll
find
all
the
different
kinds
of
issues
that
you
can
get
your
hands
dirty
with.
B
B
Let
me
first
go
to
my
chaos
center.
So
for
those
of
you
who
don't
know
what
chaos
center
is
well
chaos
center
is
your
front
facing
user
component,
where
you
can
inject
chaos.
As
I
explained
in
the
presentation
and
as
part
of
this
demo,
we
are
going
to
run
three
chaos
workflows
today.
So
what
are
these
chaos
workflows
or
basically,
what
are
we
going
to
do?
B
Well,
first
of
all,
I'm
going
to
show
you
a
kubernetes
based
chaos,
experiment,
and
it
is
one
of
the
most
popular
kiosk
experiments
that
we
have
here
known
as
a
pod
delete
experiment.
The
name
itself
should
be
enough
to
explain
what
it
what
we
are
going
to
do.
We
are
going
to
basically
delete
a
few
parts
of
an
application
and
see
how
it
affects
our
application.
B
B
So
with
that,
let
me
just
show
you
how
you
can
first
of
all
schedule
a
workflow
here,
or
maybe
I
can
just
give
you
a
brief
tour
of
what
the
chaos
center
is
and
what
are
these
different
features.
So
this
is
the
home
dashboard
you,
where
you
can
see
the
number
of
agents
you
have.
What
are
the
projects
that
you
have,
then?
You
can
see
the
different
past
workflow
runs.
If
you
have
run
past
workflows,
then
you
can
see
the
litmus
workflows
panel
over
here.
B
It
also
shows
the
past
workflow
runs
as
well
as
any
schedules
that
you
might
have.
So,
as
I
had
explained
you
that
you
can
run
the
different
chaos
workflows
in
a
schedule
using
atmospheres,
then
you
have
the
let
us
chaos
agents.
Well,
the
chaos
agents
is
what
enables
you
to
inject
yours,
either
into
your
own
cluster,
using
a
self
agent
or
into
your
external
kubernetes
cluster,
using
an
external
agent.
So
I
currently
don't
have
any
external
agents.
All
I
have
is
self
agent
over
here.
B
As
you
can
see,
so
you
can
connect
n
number
of
agents
using
atmospheres
which
effectively
means
that
you
can
inject
chaos
into
n
number
of
kubernetes
clusters
using
a
single
control
plane
moving
forward.
We
also
have
chaos
hubs.
So
chaos
herbs
as
prithvi
had
explained,
is
a
marketplace:
an
open
source
marketplace
for
different
chaos.
Experiments
that
you
have
you
have
the
standard
chaos
up
provided
by
litmus,
that
is
the
electroscale
sub,
which
is
like
the
standard
chaos
of
with
every
like.
You
will
be
always
having
access
to
this.
B
B
You
can
have
your
own
chaos
up
where
you
can
add
the
different
experiments
and
maintain
it
in
order
to
basically
in
order
to
separate
them
from
the
actual,
like
open
source
chaos,
since
this
will
be
your
private
experiments
for
say
next
up
we
have
the
observed.
So,
as
I
told
you
absolutely
is
a
common
commodity
in
the
litmus
kiosk
world,
we
treat
it
as
a
first-class
citizen,
because
auxiliary
is
deep
rooted
within
the
chaos
engineering
practice
itself.
B
So
what
it
can
do
is
it
can
basically
show
you
the
different
workflow
statistics
like
what
are
the
different
workflow
runs
that
you
have
over
a
duration
of
time.
What
are
the
failed
and
the
past
workflows
and
you
can
basically
filter
them
based
on
the
different
criterias,
you
can
see
the
different
periods
over
which
you
are.
You
are
basically
going
to
like.
Have
your
workflows,
then,
according
to
the
agent
itself,
there
are
so
many
options
actually
to
configure.
B
Then
you
also
have
your
data
sources.
So
as
part
of
this
demo,
I've
already
set
up
a
prometheus
data
source.
I've
set
it
up
and
connected
it
with
litmus.
So
what
it
does
is
it
helps
you
create
different
application
dashboards.
So,
for
example,
we
have
a
sockshop
dashboard
over
here
I'll
show
you
how
it
will
be
helping
us,
but
before
that,
let
me
just
give
you
a
sneak
peek
into
what
it
is
doing.
B
So,
as
you
can
see,
we
have
different
application,
metrics
observable
using
this
auxiliary
tool
and
these
all
this
data
is
actually
directly
being
fresh
from
your
prometheus
over
here.
So
you
can
create
interleave
dashboards
using
different
options
here
moving
forward.
Basically,
this
is
it
in
a
nutshell
like
what
litmus
it
is.
You
can
again
play
with
different
settings,
so
there
is
option
for
custom
for
github
setup
for
custom
image
registry,
but
I'll
leave
you
up
to
it
to
explore
some
time.
B
Okay,
so
the
idea
is
that
the
dashboard,
the
monitoring
dashboards,
are
supported
by
the
primitives,
because
you
need
something
kind
of
a
time
series
database
or
some
type
of
a
data
source
from
where
you
will
fetch
in
your
different
metrics.
So
the
way
it
works
is,
if
you
remember
I'll,
just
go
back
to
my
slide
deck.
B
We
have
something
known
as
a
kiosk
exporter
over
here
right,
so
what
kiosk
exporter
does
is,
while
your
experiments
are
running
in
your
kubernetes
cluster,
it
fetches
all
those
different
metrics
while,
while
your
chaos
is
running
and
these
metrics
are
basically
being
fetched
into
a
prometheus
database,
the
prometheus
database
is
then
storing
these
different
metrics,
which
becomes
your
single
source
of
truth
for
basically
setting
up
your
monitoring
dashboard.
So
that
is
the
idea
behind
it.
You
are
having
different
applications
with
yourself
and
you
are
having
your
own
kiosk
exporter.
B
You
can
have
your
own,
like
you,
can
have
a
litmus
chaos
exporter,
of
course,
but
even
if
you
have
let's
say
node
exporter
or
kubernetes,
export
or
kubernetes
state
exporter,
you
can
also
integrate
them
within
this
dashboard
and
create
an
interleave
dashboard,
so
that
basically
will
help
you
to
understand
that
what
is
the
impact
of
chaos
while
you
are
running
it
like,
let's
say
node
exporter,
is
for
seeing
like
what
is
the
status
of
a
node
different
metrics
like
cpu,
io,
latency,
etc.
Kubernetes
exporter
is
something
of
a
similar
line
for
kubernetes.
B
So
all
of
this
information
will
help.
You
understand
in
a
better
way
that,
while
you're
injecting
chaos,
what
is
going
in
with
my
system,
why
some
of
some
of
like
what?
Why
is
some
queries
having
a
greater
latency
than
other?
So
this
kind
of
information
will
help.
You
gather
information
about
your
system
and
make
your
system
more
risky.
D
B
Okay,
so
yeah
we
are
capturing
different
metrics
as
well,
but
you
can
use
your
same
prometheus
database
here
as
well.
The
idea
is
that
the
kiosk
exporter
gets
installed
as
part
of
your
kiosk
execution
plan.
Let
me
just
hop
into
my
terminal
to
give
you
a
better
idea
of
what
I'm
saying
so,
if
I
say
cube,
ctl
get.
B
B
So
the
idea
actually
is
while
we
are
waiting
for
it
like
the
idea.
Actually
is
that
you
have
your
chaos
exporter,
the
chaos
exporter
is
nothing
but
a
prometheus
exporter
which
exports
the
different
metrics
from
prometheus
to.
Basically,
your
not
from
the
prometheus
excuse
me
from
your
kubernetes
regarding
your
litmus
chaos
execution
to
your
prometheus.
So
these
execution
metrics
include
chaos,
results,
chaos,
different
chaos,
workflow
summary,
basically,
just
a
quick
confirmation,
prithvi,
audible.
D
The
only
reason
why
I'm
asking
this
is
take
take
an
example.
Right
we
have
one
cluster
is
where
all
our
devops
components
like
chaos
is
also
there
and
in
other
components
where
applications
are
getting
deployed.
So
this
exporter,
I
wanted
to
check
if
it's
something
which
we
need
to
deploy
even
in
the
application
time,
or
it
will
only
try
to
do
it
because
it
has
to
someone-
has
to
pump
that
metrics
to
this
particular
expo
proteus
right
so
yeah,
which
we
get
installed,
along
with
the
chaos
ui.
B
B
Okay,
so
as
I
understood
that
you
are
asking
that
who
is
basically
responsible
for
pulling
all
the
different
metrics
for
from
like
your
chaos
metrics
from
the
kubernetes
cluster,
am
I
understanding
it
correctly?
Yes
to
this
kiosk
exporter
through
this
chaos,
exporter?
Okay,
so,
as
you
can
see,
I
just
listed
down
all
the
parts
in
my
leftmost
name
says:
I'm
not
sure
why
getall
is
not
working,
but
anything
can
get
it.
So,
as
you
can
see,
kiosk
exporter
is
actually
installed
within
your
list
namespace
by
default.
B
What
it
is
doing
is,
it
is
actually
fetching
all
the
different
information
that
is
getting
really.
That
is
actually
related
to
your
litmus
chaos
and
it
is
being
stored
right
now
in
inside
my
prometheus
db.
So
you
as
a
user,
don't
need
to
do
anything.
In
fact,
you
just
need
to
install
it
first
gears
and
that's
it
if
you
already
have
prometheus
installed,
that
integration
will
be
taken
care
of
directly
while
you're
integrating
chaos,
exporter
and
setting
up
your
dashboard
over
here,
like
what
you
need
to
do
is
before
setting
up
your
dashboard.
B
It's
basically
connecting
your
data
sources.
So
in
overview
section,
you
will
have
actually
an
option
to
create
your
dashboard
like
sorry,
adding
your
new
data
source.
So
right
now,
as
you
can
see,
only
one
data
source
is
added.
So,
as
in
your
case
like
if
you'll
have
a
prometheus
data
source,
you
can
simply
add
it
over
here,
and
you
already
have
chaos
exporter
over
here.
So
you
just
need
to
set
your
data
source
up
with
this
chaos,
exporter
and
you're
good
to
go.
Nothing.
B
Yeah
sure,
okay,
I'm
not
sure
what
is
happening.
Some
real
life
chaos,
it
seems
my
kubernetes
ui
is
sorry.
The
terminal
is
lagging
a
bit,
but
I'll
try
to
show
you
what's
happening
over
here,
all
right
with
that,
let's
just
jump
into
our
first
chaos
workflow.
So
with
this
first
chaos
workflow,
it
will
be
a
simple
experiment.
It
will
be
a
part
delete
experiment.
B
So
actually,
these
are
all
the
previous
experiment
runs
that
we
have
different
parts
related
to
that
I've
just
forgot
to
clear
my
dashboard.
So
let
me
just
delete
all
this
before
we
actually
go
at
so
I
hope
we
can
meanwhile
get
started
with
creating
our
workflow
while
the
deletion
is
actually
taking
place.
What
are
we
going
to
do?
Well,
let
me
just
introduce
to
an
application
that
we
have.
B
It
is
known
as
sock
shop
so
who
doesn't
love
love
socks?
Well,
even
if
you
don't-
let's
just
pretend
for
this
demo
that
you
do
so
what
is
sockshop
well,
it
is
an
e-commerce
application,
an
e-commerce
website
for
selling
swaps.
So
the
special
thing
about
this
application
is,
it
is
basically
a
completely
micro
service
application.
B
It
is
managed
by
a
host
of
different
host
of
different
microservices,
and
each
part
of
this
application
is
actually
a
microservice,
including
this
catalog
or
including
this
home
page.
That
you're,
seeing
or
including
the
cart
that
you
have
everything
is
a
microservice.
Even
the
users
like
you
can
do
login
and
stuff.
That
is
also
a
microservice,
so
what
we
are
going
to
do
is
basically
it
has
a
catalog
section
as
well.
We
are
actually
going
to
delete
that
catalog.
B
So
what
I
mean
by
deleting
the
catalog
is,
let
me
just
show
you
basically
that
what
the
catalog
looks
like
in
the
kubernetes.
So
if
I
say
cube,
ctl
get
deployments
thousand
sock
shop.
My
application
is
actually
deployed
in
this
sockshop
namespace.
So
let
me
just
fetch
the
different
deployments
for
you.
B
As
you
can
see,
we
have
the
different
deployments.
Catalog
is
one
of
them.
It
has
two
ports
currently
that
are
running
as
part
of
this
deployment,
so
what
we
are
going
to
do
for
this
experiment
is,
of
course
it
is
not
representative
of
a
real
baby,
like
scenario
that
you
might
have,
but
we
are
going
to
do.
Is
we
are
going
to
delete
all
of
both
of
these
parts?
B
B
Right,
it
should
automatically
come
back,
but
here's
the
deal
with
this
chaos
experiment
will
ensure
for
a
certain
duration
of
time.
Will
not
let
these
parts
come
back
up
and
that's
how
the
power
delete
experiment
works.
We
basically
choose
some
kind
of
a
resource,
kubernetes
resource
like
a
deployment,
a
replica
set,
something
like
that.
We
go
ahead
and
delete
the
parts
from
that
resource
and
for
a
certain
duration
of
time.
They
stay
deleted,
and
in
that
duration
we
basically
try
to
validate
a
few
test
conditions.
That
is
your
hypothesis.
B
So
we
are
trying
to
say
that
hey,
if
I
say,
delete
five
parts
or
ten
parts
out
of
my
deployment,
my
endpoint
or
like
my
website,
should
still
be
usable.
That
is
my
hypothesis
validation.
But
how
will
you
validate
that?
Whether
it
is
happening
or
not?
That
is
where
kubern
like?
Let
first
chaos
comes
into
play.
It
actually
goes
ahead
and
create
that
scenario
and
that's
whether
the
condition
is
still
holding
valid
or
not.
B
B
Sorry,
like
let
me
just
create
an
experiment
here:
okay,
so
as
part
of
this
like
how
you
can
basically
create
your
chaos
workflows
well,
in
order
to
create
your
own
chaos
workflows,
it
is
very
simple.
You
can
basically
just
go
ahead
and
add
a
new
experiment
over
here.
So
what
is
the
experiment
that
we
are
trying
to
do
over
here?
Well,
it
is
a
port
delete
experiment,
so
you'd
search
for
a
pod
delete
here
and,
as
you
can
see,
under
generic,
we
have
the
pod
delete.
Experiment.
You'll
simply
add
it
over.
B
B
So
let
me
just
go
ahead
and
tune
this
kiosk
experiment
by
tuning
I
mean
that
what
we
are
actually
setting
up
over
here
is
we
are
defining
our
target
application.
The
target
application
in
this
case
is
basically
the
stock
shock
application
that
we
have
so
I'll
use
the
application
namespace
as
sockshop.
It
is
a
kind
of
deployment,
as
we
saw
here,
and
then
we
are
going
to
specify
its
label,
so
the
label
actually
is
name
is
equal
to
catalog.
B
Once
we
have
defined
our
target
application,
we
need
to
right
now
specify
the
probes.
So
if
you
remember
probes
are
nothing
but
declarative
hypothesis
checks
for
our
applications.
So
what
I
mean
by
that,
let
me
just
define
a
probe.
It
will
be
kind
of
an
http
probe
and
I'll
say,
let's
say
catalog
group,
and
this
will
be
a
non-chaos
type
of
probe.
We
have
different
probe
modes
like
you
have
start
of
the
test
end
of
the
test
edge.
That
is
both
start
and
end.
B
B
Let
us
set
our
timeout
of
let's
say
three
seconds
and
we
must
retry
for
two
times
at
an
interval
of
three
seconds.
So
what
are
we
setting
this
timeout
retry
and
interval
fall?
Well,
we
have
a
url
over
here,
so
this
category,
like
whatever
information
you
are
seeing
over
here,
is
actually
getting
served
by
this
api.
B
What
you're
seeing
is
the
catalog
api?
Okay,
so
the
catalog
api
is
basically
responsible
for
serving
this
entire
page.
The
parts
that
you're
seeing
over
here
is
part
of
deployment
are
responsible
for
serving
this
api.
Sorry,
this
api
endpoint,
which
is
providing
this
json
response,
which
is
causing
this
page
to
come
up.
B
So
what
I'll
do
is
if
we
are
deleting
the
part,
the
idea
should
be
that
this
url
is
not
supposed
to
be
reachable,
so
I'm
just
putting
this
url
over
here,
and
what
are
we
trying
to
validate
over
here
is
that,
while
we
are
deleting
these
spots,
is
this
url
still
reachable
or
not
or
more
specifically,
we
are
defining
a
condition
over
here.
Is
that
when
we
are
doing
when
we
are
performing
a
get
request
on
this
endpoint,
are
we
getting
a
response
code
of
200
or
not?
B
B
But
let's
see
what
we
can
do
over
here
right
now,
we
are
trying
to
tune
the
experiment.
So
here's
some
metrics
like
what
is
the
total
chaos,
duration
or
the
chaos
interval
for
which
the
chaos
should
continue,
and
then
you
have
the
pods
affected
percentage.
So,
as
you
saw
like
in
this
deployment
the
catalog
deployment,
we
have
two
parts
right.
B
So
if
we
are
looking
forward
to
deleting
both
of
these,
we
just
need
to
save
100,
because
100
is
referring
to
two
parts,
but
let's
say
we
are
giving
50
so
that
would
have
referred
to
one
part,
so
one
would
have
been
related,
so
let's
just
delete
both
the
parts
and
see
how
it
does
I'll.
Let's
just
go
ahead
and
create
the
reverse
schedule
as
false
just
a
second.
B
B
B
Okay,
so
let
us
start
with
the
experiment:
oh
yeah,
before
we
go
into
it
like
what
we
were
explaining
earlier
for
the
resiliency
score,
you
can
attach
a
weight
to
each
of
your
experiments.
So
since
we
have
only
one
experiment
right
now,
10
is
fine,
but
let's
say
you
have
more
than
one
experiment.
You
can
attach
a
particular
weight
to
it
so
that,
while
calculating
the
resiliency
score,
that
particular
weightage
will
be
taken
into
consideration.
B
So
we
are
at
schedule
right
now.
Schedule
is
basically
responsible,
for
it
is
basically
responsible
for
ensuring
that
what
you
want
to
do
with
your
the
workflow
that
you
are
creating
like
you
want
to
run
it
only
once
where
you
want
to
run
it
as
a
recurring
schedule
so
on.
So
I
guess
we
are
good
with
that.
Let
us
just
go
ahead
and
schedule
our
chaos
experiment.
B
B
Things
so
our
workflow
has
actually
started.
Let
me
guide
you
through,
like
what
is
happening
over
here
as
the
workflow
has
already
started.
We
have
the
install
kiosk
experience
step
right
now
going
on,
and
this
is
the
visual
representation
of
all
these
steps
that
will
be
having
as
part
of
this
workflow
right
now.
We
are
actually
installing
the
body
rate
experiment
within
our
kubernetes
cluster.
So
we
have
this
kubernetes
cluster
with
us.
B
B
Okay,
so
right
now
we
are
actively
watching
for
all
the
deployments
in
this
sock
shop.
Namespace
up
until
this
point
like
we
can
see,
the
catalog
pod
is
still
here,
but
as
we
proceed
on
with
the
experiment,
as
you
can
see
right
now,
the
pods
are
actually
gone.
Both
of
these
parts
are
deleted
and,
as
you
can
see,
the
parts
are
not
coming
up
by
themselves.
There
are
zero
available
ports
to
confirm
this.
Let's
just
simply
reload
this
application.
B
B
B
Okay,
so
the
grafana
dashboard
is
apparently
working,
not
sure
what
happened
to
litmus.
I
guess
it
is
because
I
have
chosen
a
bit
small
cluster
like
I
guess
we
are
not
able
to
sustain
the
load
of
the
ex
of
all
the
things
that
are
working
over
here.
So
what
I
was
actually
trying
to
show
you
is
this
catalog
qps.
B
Since
we
have
deleted
the
catalog
pod,
we
are
actually
not
able
to
see
which
we
are
able
to
see
that
the
queries
per
second
for
the
catalog
part
is
actually
going
down.
You
can
see
that
there
is
a
sharp
decline
over
here
and
we
are
actually
not
able
to
see
anything
beyond
this
point.
So
this
actually
goes
on
to
show
that
while
orders
is
fine,
everything
is
else
is
fine.
B
So
this
is
the
annotation.
Basically,
the
chaos
exporter
has
enabled
this,
like
whatever
the
orange
annotation
you
saw.
That
is
because
of
the
kiosk
that
is
in
indicating
the
chaos
interval.
Okay,
the
matrix
is
not
working,
I'm
not
sure
why,
but
let's
okay
anyway,
we
are
able
to
see
it
over
here.
So
it's
fine.
B
B
So
let
us
wait
for
a
while
for
this
experiment
to
stop
or
basically
go
to
an
end.
And
meanwhile,
if
you
have
any
questions,
please
feel
free
fine.
So
if
you
have
no
questions
I'll
lead
you
through
the
logs
of
this
experiment,
so
let's
say
that
right
now
you
have
a
problem
with
your
application.
Resiliency
you
saw
that
it
failed,
but
it's
also
important
to
understand
that
why
your
application
feel
or
basically
why
this
particular
experiment
feels.
B
So
if
we
take
a
look
at
the
logs,
we'll
be
able
to
understand
more
about
what's
going
on
during
the
chaos
beyond
just
the
observability,
so
as
you
can
see
in
the
logs,
we
have
defined
our
probes
and
the
probe
validation
is
taking
place
over
here.
We
are
waiting
for
chaos
interval
of
30
seconds
after
we
have
deleted
the
two
parts,
so
the
status
of
the
pods
like
we
are
able
to
see
the
different
parts
that
we
had
earlier
from
the
catalog.
This
is
the
first
part.
B
This
is
the
second
pod
we
have
deleted
both
of
them
and
we
are
waiting
for
30
seconds.
During
that
time
we
are
actually
trying
to
get
the
response
code
for
that
particular
url
that
we
had
set
up
and,
as
you
can
see,
the
actual
value
is
500
and
not
200.
So
since
it
is
500
well,
the
pro
validation
is
actually
failing,
because
we
are
expecting
a
response
code
of
200.
B
So
that
is
how
it
goes
like.
We
are
then
again
restarting
our
parts
and
they
prob.
The
pro
validation
has
actually
failed.
That
is
what
has
been
indicated
over
here,
because
of
which
the
experiment
is
failing,
and
we
also
know
now.
We
have
something
known
as
a
kiosk
result,
which
is
a
custom
resource
inside
our
kubernetes,
for
let
us
so
here
it
is
an
aggregation
of.
B
Cool-
let's
just
move
to
our
second
workflow,
so
the
second
workflow
that
I'm
going
to
show
you
is
a
multi-cloud
chaos
workflow.
So
as
part
of
this,
what
we
are
going
to
do
well,
we
have
a
virtual
machine,
an
ec2
instance
running
in
aws,
and
we
have
a
virtual
machine
running
in
gcp
as
well
the
one
that
I've
ticked
over
here
as
well
as
we
have
a
gcba
disk
over
here,
like
a
persistent
disk.
B
Let's
just
go
ahead
with
the
experiment,
so,
let's
just
what
I'm
going
to
do
is
I
have
actually
already
created
a
multi-cloud
chaos
workflow
as
a
template
like
I
had
run
this
experiment
before,
and
I've
created
a
template
for
you,
you
can
basically
create
these
templates.
That
will
be
handy
with
you.
If
you
are
running
a
chaos
workflow
multiple
times
anyway,
you
can
create
a
schedule
out
of
them,
but
you
can
also
save
them
as
a
template.
B
So
if
I
just
go
ahead,
you
can
see
that
this
template
is
actually
saving
all
the
information
that
we
want
to
have
over
here,
as
part
of
this
experiment,
you
can
see,
will
be
first
of
all,
installing
these
three
experiments
easy
to
terminate
y
id
gcp
vms,
to
stop
and
vm
disk
loss
and
then
we'll
be
running
the
ec2
experiment.
First
and
after
that,
both
the
gcp
experiments,
but
what's
special
about
this
gcp
experiments
execution,
is
that
they
are
running
in
parallel.
B
B
So
how
will
you
simulate
such
an
such
a
scenario
here
in
litmus?
Well,
this
is
the
answer
to
it.
Basically,
you
can
tune
your
own
workflows
to
do
that,
so
how
you
can
create
this
workflows.
Well,
if
you
go
here
in
edit
sequence,
as
you
can
see,
you
can
either
pull
these
experiments
below
like
take
them
to
different
tiers,
or
you
can
basically
like
create
a
like
a
serial
workflow
out
of
them.
B
So
in
parallel
right
now
we
have
gcp2
experiments
in
parallel,
so
they
are
appearing
as
such
over
here.
So
I'll
just
save
the
changes
over
here
and
I
have
also
configured
these
experiments,
basically
so
for
the
ec2
experiment.
Let
me
just
quickly
run
you
through
that
what
we
are
going
to
configure
over
here,
I
haven't
defined
any
probes,
but
what
I've
defined
is
actually
the
ec2
instance
id.
B
B
We
are
actually
using
an
an
iam
role,
basically
as
part
of
a
secret,
so
we
basically
create
a
kubernetes
secret
containing
the
credentials
that
we
need
in
order
to
authenticate
with
our
aws
cloud
and
mount
it
as
such,
so
always
just
like
to
show
you
that
how
you
are
able
to
mount
all
that
like
if
I
go
to
edit
yaml
by
the
way.
What
is
this
document?
Let
me
just
go
through
that
as
well.
B
So,
as
you
can
see,
this
is
the
this
is
of
kind
workflow,
so
it
is
actually
the
workflow
customer
resource
the
argo
workflow
custom
resource
that
is
actually
mitigated
by
our
workflow
controller
in
the
execution
plate.
So
while
we
are
creating
all
these
information
in
the
kiosk
center,
it
leads
to
the
creation
of
this
manifest
the
this
entire
manifest.
B
B
B
In
order
to
mount
the
different
credentials,
you
would
need
to
authenticate
with
your
gcp
cloud,
but
apart
from
that,
based
on
the
experiment
like
if
you're
doing
a
vm
instance
job
experiment,
you
need
to
provide
your
vm
instance
name,
which
I've
already
done
over
here,
the
project
id
of
your
gcp
and
then
you
need
to
provide
your
instance
zone
as
well
and
then
also
provide
like
it
is
a
part
of
an
auto
scaling
group
or
not.
So
these
are
all
all
the
things
that
you
need
and
I've
also
provided
mounted
a
secret
for
it.
B
B
As
you
can
see
here
in
the
resiliency
score
adjustment,
you
can
set
up
different
weights
for
each
one
of
them
so
that
if
some
experiment
is
having
like
more
importance,
then
the
resiliency
score
for
it
will
be
calculated
accordingly.
So
for
the
timing,
let's
just
keep
everything
to
full:
let's
just
go
next
and
yeah,
we
are
actually
ready
for
the
experiment.
So,
let's
just
start
the
experiment.
B
B
B
The
experiment
has
already
started
and
the
only
way
to
confirm
that
whether
this
experiment
is
actually
working
or
not
is
by
actually
going
to
the
aws
console
and
see
that
whether
our
instance
is
actually
like,
still
running
or
not
like
before
the
experiment.
Of
course
it
was
running.
If
I
refresh
this,
as
you
can
see,
it
is
right
now
already
stopping
so
this
actually
confirms
that
we
are
able
to
successfully
inject
kiosks
into
our
aws
cloud
resources
using
wetman
skills.
So
what
is
the
significance
of
this?
B
B
It
is
not
affected
by
any
means,
so
it
is
an
excellent
way
to
test
test
this
out
by
the
way,
as
part
of
this
experiment,
I'm
just
targeting
one
vm
over
here,
but
you
can
do
this
for
a
number
of
vms.
This
is
something
that
led
bus.
Chaos
is
very
well
able
to
do.
That
is
scale
up
or
scale
down
in
a
very
efficient
way.
It
can
efficiently
scale
up
for
hundreds
of
vms
at
at
ago
as
well.
It
can
handle
them.
B
So
we
have
to
wait
for
some
time
like
I've
set
the
chaos
interval
as
30
seconds,
so
we
would
have
to
wait
until
the
chaos
interval
has.
Okay.
Okay
interval
has
already
passed
as
we
were
talking,
so
it
is
back
to
our
running
state,
and
this
should
actually
mark
our
experiment
as
fast,
because
our
even
after
stopping
the
ex
our
our
awscc
to
instance,
it
came
back
to
our
running
instance
and
since
we
didn't
had
any
probe
validation,
so
everything
everything
basically
got
cleared
and
we
are
good
over
here.
B
B
So
you
must
be
wondering
like
what's
taking
it
so
long
to
finish,
it's
actually
like.
We
are
currently
updating
our
chaos
result
and
stuff,
because
it's
not
only
about
stopping
or
starting
the
ec2
instances.
It's
also
about
maintaining
the
kiosk
result
where
we
are
aggregating.
The
results
of
all
the
chaos
runs,
so
we
need
to
be
able
to
make
sure
that
we
are
recording
the
outcomes
of
this
experiment.
B
So
yeah
that's
done
now.
Let's
just
start
with
our
gcp
chaos
experiments,
so
we
can
see
that
already
our
gcp
kiosk
expense
has
started.
We
first
of
all
have
our
vm
instance
top
experiment.
So
as
part
of
this
experiment,
we
are
actually
seeing
that
our
vm
instance
is
still
running
and
if
I
go
to
the
disk,
we
are
actually
targeting
this
disk
and
it
is
right
now
attached
to
a
vm
vm
instance
over
here.
So
right
now,
since
our
experiment
is
running,
let
us
refresh
and
see
if
this
instance
is
still
running
or
not.
B
So
if
I
refresh
over
here,
as
you
can
see,
the
instance
is
right
now
stopped,
and
it
is
the
experiment
that
caused
it
to
go
from
a
running
to
stop
state
and
same
here.
If
I
refresh
this,
just
keep
an
eye
out
for
on
this
test
disk.
If
I
refresh
this,
as
you
can
see
right
now,
it's
not
attached
to
our
vm
any
four.
So
what
are
the
implications
of
these
experiments?
Well,
with
the
gcp
vm
instance
top
experiment,
it's
the
same
as
aws
experiment,
but
here
in
gcp
and
with
the
disk
detached
experiment.
B
B
How
can
you
be
sure
that,
basically,
all
your
data
is
still
secure
and
backed
up
and
no
harm
is
inflicted
on
your
application,
so
these
two
experiments
can
basically
help
you
in
validating
those
scenarios
by
creating
the
actual
circumstances
where
you
are
effectively
creating
the
outages
at
the
same
time.
For
everything
like,
as
you
can
see,
at
the
same
time,
we
are
stopping
the
instances
and
the
disks
as
well
so
yeah.
Basically,
we
should
again
right
now
see
the
instances
come
back
up.
B
Okay,
the
instance
is
actually
running
right
now,
like
it
has
already
started
again,
so
our
disks
should
be
also
good,
like
only
a
disk
test
disk.
If
I
refresh
it
yeah.
As
you
can
see,
it
is
right
now
in
connected
to
our
instance,
so
this
these
two
experiments
have
also
passed
with
flying
colors.
Since
we
were
able
to
revert
everything
after
we
stopped
our
after
we
stopped
our
vm
and
detached
our
disk.
We
were
again
able
to
start
backup
instance
and
attach
our
disk
so
yeah
with
that.
B
Basically,
we
can
see
that
all
the
all
the
steps
of
the
experiment
is
successful,
so
we
achieved
a
100
score.
I
guess,
if
we
go
to
a
table
view
yeah,
we
actually
have
to
100
resiliency
score
cool,
so
we
came
a
lot
across
the
last
few
minutes.
We
saw
the
different
capabilities
of
litmus
chaos
if
you
have
any
questions
anything
at
all
for
me
before
we
proceed
on
to
our
last
chaos
workflow,
I
would
like
to
take
them.
D
B
Okay,
let
me
just
show
you
that
part
again
I'll
just
go
from
the
scratch
like
I
used
a
template
right
now,
I'll
be
not
doing
that
just
to
show
you
that
how
that
part
can
be
done.
So
you
have
a
setup
over
here
right
now
and
let
me
set
up
into
it
nothing
more
than
that,
like
we,
don't
have
anything
more
so
I'll,
just
go
to
the
schedule.
Workflow
choose
self
agent
choose
next
and
then
we'll
just
choose
the
chaos
hub.
B
So
chaos
hub,
if
you
remember,
is
the
open
source
marketplace
and
inside
that
we
have
our
default
hub.
The
electricity
also
we'll
go
next.
We
can,
you
can
name
it
anything
like,
let's
just
say,
gcp
from
now,
and
if
we
go
next,
you
can
add
new
experiments
from
here,
so
I'll,
just
search
for
the
gcp
experiment.
Let's
just
say
that
we
want
to
create
use
the
gcp
vm
instance:
topics
when
so
I'll,
add
it
and
upon
adding
it
you'll
be
able
to
tune
this
experiment
how
to
tune
it.
B
You
can
basically
click
on
this
pencil,
icon
or
basically
click
here
on
the
experiment
name
and
then
you
can
have
different
options
over
here,
like
first
of
all,
the
name
of
the
herb
and
the
experiment,
name,
etc.
Then
you
can
specify
the
target
application,
so
this
sentence
is
not
a
key.
It
is
like
experiment,
so
it's
not
really
relevant
for
us,
but
you
can
still
enforce
an
annotation
check
which
is
not
really
important
in
this
case.
B
B
While
your
vm
will
go
down,
you'll
be
able
to
see
that
whether
that
url
is
still
applicable
or
not,
or
even
you
can
plan
for
even
complex
scenarios,
where
you
have
a
load
balancer
serving
some
url,
which
is
dividing
the
load
between
some
vms
and
basically,
you
can
disable
a
percentage
of
the
vms
or
a
particular
vms
out
of
that
load
balancer
and
see
whether
they
are
still
working
or
not.
Something
like
that,
but
anyway
I'll
not
go
into
the
probes.
B
Right
now
and
lastly,
comes:
we
come
to
your
question
that
how
you
can
specify
the
chaos
injection
on
my
cloud
resources,
so
for
so,
for
example,
here
is
my
project
network
c2e,
and
here
is
my
vm
instance-
nail
test
instance
so
I'll
just
copy
the
name
of
the
vm
instance
over
here
and,
as
you
can
see
over
here,
while
tuning
the
experiments
you
have
a
few
enables
over
here.
That
is
the
options
that
you
need
to
configure.
B
You
have
your
chaos,
duration,
the
chaos
interval.
That
is
the
time
duration
for
which
you
want
to
you
want
to
like
stop
these
gcp
instances.
Then
you
can
specify
the
name
of
your
vm
instance,
so
here
I'll
put
in
the
name
of
the
vm
instance
that
I
just
copied,
then
the
gcp
project
id.
That
is
basically
the
project
that
I
have
like.
This
is
the
id
of
my
project
and
you
need
to
put
it
there
then
just
protein,
the
instance
zone.
That
is
your
central
1a
and
that's
it
basically,
mostly.
B
You
can
also
make
this
enable
if
you
have
an
auto
scaling
group,
but
that
vm
is
not
auto
scaling
group.
So
I'll
not
do
that
I'll
not
run
this
experiment
right
now,
but
here
is
how
you
can
specify
and
one
last
piece
of
information.
That
is
how
you
will
authenticate
with
your
gcp
account.
Well
for
that,
basically,
you
need
to
create
a
gcp
service
account
what
this
gcp
service
account
will
do.
Is
it
will
let
your
this
kubernetes
experiment?
Sorry,
this
litmus
chaos
experiment
to
impersonate
a
an
iam
identity.
B
Basically,
and
then
it
can
make
requests
on
behalf
of
basically
that
iem
rule
that
has
been
assigned
to
it
like
it
will
have
all
those
permissions.
So
I
have
actually
already
created
all
that,
like
I
had
listed
my
secrets
a
while
ago,
so
yeah.
So
basically
that's
how
you
provide
the
permissions
cool.
B
B
B
So,
in
order
to
do
so,
what
we
can
do
well
think
from
a
technical
standpoint
that
we
have
kubernetes
right
now
and
we
are
trying
to
orchestrate
chaos
within
a
virtual
machine.
That
is,
we
are
trying
to
access
the
internals
of
the
virtual
machine,
the
the
operating
system,
the
the
let's
say,
the
disk
or
io,
whatever
components
there
are.
So
how
would
you
do
that?
Basically,
in
order
to
mitigate
the
same
issue
in
order
to
have
a
superior
observability
and
have
different
features
that
we
can
integrate
during
machine
chaos?
B
B
So
while
we
are
ss
etching
into
it,
let
me
just
go
ahead
and
create
our
kiosks
experiment.
So
I'll
again
choose
this
self
agent
and
in
this
case
I'll
be
actually
using
my
own
hub.
So
the
thing
is
that
the
experiment
that
I'm
going
to
show
you
is
still
not
ga
like
we
are
still
not
making
the
charts
available
just
yet,
but
you
can
be
sure
that
in
the
further
iterations
we'll
be
definitely
going
ahead
with
it
like
you
can
still
use
this
experiment
from
my
hub
like
from
my
owner.
B
But
again,
please
don't
do
that,
because
if
there
are
any
inconsistencies
right
now
that
will
that
might
affect
your
system.
So
let
me
just
still
show
you
that
how
it
can
be
used.
So
if
you
have
like,
I
have
selected
my
own
hub
and
what
it
will
do
is
basically
it
will.
Let
us
it
will.
Let
us
basically
choose
the
process,
skill
experiment.
B
So
if
I
say
process
skill
over
here,
it
lets
me
choose
it.
Meanwhile,
as
you
can
see
before
I
start
and
going
ahead
with
the
configuration
we
are
into
my
virtual
machine
right
now,
so
I
was
talking
about
m
agent
so
well.
What
is
amazing?
First
of
all,
let
me
talk
about
it.
M
agent
is
a
machine,
scoped
or
basically
vm
scoped
agent,
a
daemon
agent,
which
helps
you
remotely
inject
faults
within
within
your
virtual
machines,
bare
metal
nodes
or
even
physic
physical
machines.
B
So
it
is
basically
a
demon
agent,
that
is,
it
is
managed
completely
by
your
own
operating
system,
and
it
helps
you
to
remotely
inject
faults
in
background
using
litmus
chaos
experiments.
So
what
I'm
trying
to
say
is
that
m
agent
is
basically
installed
within
this
virtual
machine
as
part
of
a
binary,
and
we
can
interact
with
the
m
agent
in
order
to
inject
chaos
like
process
skill
kiosks,
for
example,
into
our
machine.
B
So
if
I
want
to
right
now,
specify
m
agent
to
let's
say,
perform
the
process,
skill
experiment,
so
we
can
like
the
process,
is
very
similar
to
what
we
are
doing
so
far
like
I'm
coming
to
chaos
center
over
here,
I'm
choosing
the
process,
skill
experiment,
but
there
are
a
few
more
steps.
What
are
those
steps?
Well,
first
of
all,
the
problem
is
like.
We
need
to
specify
that
we
need
to
yeah
sorry,
so
we
need
to
basically
specify
that
which
process
ids,
that
we
are
trying
to
kill.
B
B
You
can
see
that
the
apache
to
like
is
a
process
is
already
here.
It
has
a
pid
of
390.,
so
let
me
copy
it
and
provide
it
over
here
now.
The
issue
is:
how
will
we
actually
communicate
with
our
agent?
So
the
agent,
as
the
name
suggests,
is
nothing
but
a
entity
which
performs
chaos
on
behalf
of
our
litmus
chaos,
experiments
that
is
litmus
chaos
or
our
experiment.
Pod
will
be
actually
talking
to
the
m
agent.
B
That
is
sitting
right
inside
our
virtual
machine
and
we
are
supposed
to
basically
tell
m
agent
that
hey
just
go
ahead
and
delete
this
process,
the
process
with
the
process
id
of
390..
So
in
order
to
do
that,
basically,
we
need
two
things.
First
of
all,
you
need
an
agent
endpoint
as
well
as
an
auth
token.
B
The
agent
end
point
is
nothing
but
basically
the
url
of
your
of
your
agent,
because
you
need
to
specify
basically
at
that,
at
which
point
like
you
want
to
communicate
like
at
which
end
point
you
want
to
communicate
and
the
auth
token
so
that
you
can
basically
authenticate
your
machine
for
all
the
requests
that
are
being
sent
from
this
experiment
part.
While
we
are
running
the
experiment.
So
the
question
is:
how
will
you
generate
this
agent
environment
auth
token
in
a
safe
way?
B
Well,
that's
where
emission
test
first
comes
in
because
it
can
be
used.
We
can
use
it
as
a
cli
tool
over
here.
So
if
I
say
m
agent
and
then
issue
the
flag
get
token,
it
prompts
me
that
the
token
should
expire
after
what
time.
Since
it's
not
a
good
practice
to
generate
long
running
tokens
that
don't
expire
since
they
can
potentially
be
compromised.
Let
us
take
a
token
for
let's
do
30
minutes
only
since
it's
a
short
demo.
B
So
this
token
that
we
are
getting
over
here
is
our
authentication
token,
which
will
be
actually
valid
for
a
duration
of
30
minutes
only
which
you
can
use
to
authenticate
your
chaos
experiment,
and,
apart
from
that,
it
is
also
providing
us
with
an
asian
endpoint.
Well,
this
is
nothing
but
basically
your
the
public
url
of
the
vm
that
we
are
using
right
now.
So
if
I
go
over
here,
here's
the
public
url,
like
as
you
can
see,
34
134
7828.
B
This
is
the
same
value
over
here.
Add
it
with
the
port,
so
m
agent
is
actually
utilizing
a
port
at
which
our
our
experiment,
pod
will
try
to
connect.
So
both
of
these
pieces
of
information
make
up
our
agent
endpoint.
So
what
I'm
doing
right
now
is
I'm
just
copying
over
my
agent
endpoint
as
well
as
authentication
token
into
my
into
my
basically
the
secret
that
I'll
be
mounting
right
now,
so
what
I'm
going
to
do
is
basically,
I
have
set
up
this
secret.
B
It
has
this
agent
endpoint
as
well
as
this
auth
token,
and
it
is
a
simple
secret,
so
I'll
just
apply
the
secret
now
cube.
Ctl
apply
agent
secret
in
the
lettuce
name
space:
let's
wait
for
it
to
get
applied,
and
while
it
is
applying,
let's
just
move
ahead
since
we
are
providing
both
of
these
information
as
part
of
a
secret,
we
can
basically
just
move
ahead.
Basically,
that's
all
you
need
to
set
up
like
if
you
are
using
amazon.
B
The
first
thing
that
you
need
to
do
is
basically
set
up
your
image
and
within
the
vm
that
you
want
to
injecting
the
kiosk
into
it's,
a
simple
installer
script
that
you
can
basically
access
at
litmus
k
or
slash
agent.
There.
You
will
have
the
entire
guide
to
how
to
set
up
amazon
and
once
it
is
set
up,
you
just
need
to
say
image
and
get
token
and
you'll
get
the
both
pieces
of
information.
B
You
need
to
create
a
secret
out
of
it
so
that
in
subsequent
runs
you
are
safe
against
any
security
compromise
and
then
basically,
you
just
need
to
set
up
your
own
kubernetes
experiment
over
here.
So
I'll
just
go
ahead
and
yeah
we
can
set
the
weight
as
10.
We
can
simply
schedule
the
experiment
now
with
that.
Let's
just
finish:
yeah
go
to
workflow,
so
okay,
we
have
a
warning
over
here
seems
like
there
has
been
some
issue
in
workflow
runnings
from
oh,
I
don't
know
what
is
this
since
we
already
started
the
workflow
right
now.
B
B
Apache
2,
so
this
will
let
us
watch
like
the
status
of
apache
2.
So,
as
you
can
see,
it
is
running
right
now
like
it
is
still
running
and
we
are
just
about
to
get
started
with
our
process.
Skill
experiment,
so
we'll
just
keep
an
eye
out
for
here
like
the
status
is
active.
It
is
still
running,
but
the
process
scale
experiment
has
actually
started.
B
It
also
goes
on
to
say,
like
code
kill
signal
is
equal
to
kill,
and
we
can
confirm
that
using
this
message
so
yeah
this
goes
on
to
prove
that
the
experiment
was
actually
successful
or
basically
the
chaos
injection
was
successful.
We
have
successfully
actually
killed
our
apache
process
and
yeah.
Basically,
the
possibilities
for
imagent
is
limitless
like
this
is
only
the
beginning
of
our
interaction
with
them
agent
and
we
are
excited
about
creating
more
such
experiments
with
image,
and
we
can
basically
create
faults
for
processes
cpus.
B
We
can
also
do
it
for
application
types
of
chaos
like
jvmkos,
etc.
That
is
definitely
in
our
roadmap,
so
yeah
I'll,
just
we'll
just
give
it
some
time
to
complete
this
experiment.
Meanwhile,
if
you
have
any
questions
about
m
agent
or
anything
I'll
be
happy
to
take
them,
okay,
great,
so
basically,
this
experiment
will
go
on
for
more
30
seconds.
I
guess
we
can
wait
up
till
up
until
that
point,
but
yeah,
mostly
that's
it
for
this
demonstration.
B
This
was
the
last
demonstration
of
the
electroscopes
workflow
that
I
had
for
you.
It
has
actually
completed
and
yeah
as
part
of
the
revert
chaos.
We
are
basically
removing
the
experiment
board
or
the
chaos
workflow,
port,
etc,
so
that
process
is
also
completed
so
yeah
guys.
That
is
it
basically
for
this
demo.
We
had
a
lot
of
learning
today,
starting
from
the
basics
of
chaos,
engineering.
B
Why
do
we
need
chaos,
engineering
or
what
it
seeks
to
achieve?
What
are
the
processes
that
we
are
using
inside
chaos,
engineering
to
the
very
core
of
litmus
skills?
We
explored
litmus
skills
in
a
very
comprehensive
way
in
depth,
and
we
saw
different
workflows
that
we
did
using
aws,
gcp
kubernetes
and
even
machine
scope
resources
so
yeah
guys.
I
hope
it
was
a
learning
experience
for
you,
a
great
experience
for
you,
because
I
know
it
was
a
great
experience
for
me
having
you
guys
with
me
over
here.