►
From YouTube: Constructing Chaos Workflows with Argo and LitmusChaos - Umasankar Mukkara, MayaData & Sumit Nagal
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Constructing Chaos Workflows with Argo and LitmusChaos - Umasankar Mukkara, MayaData & Sumit Nagal, Intuit
LitmusChaos is an open-source cloud-native Chaos Engineering framework for Kubernetes. It provides custom APIs (via CRDs) to orchestrate Chaos on your clusters while providing readily usable, off-the-shelf Chaos experiments via the ChaosHub. However, Litmus is also highly extensible and can orchestrate chaos experiments implemented using other tools in the ecosystem in its BYOC (Bring-Your-Own-Chaos) mode. In this talk, the Litmus maintainers, Sumit (Intuit) & Uma (MayaData), will discuss how Intuit, a CNCF end-user, uses Litmus to manage & orchestrate experiments implemented using ChaosToolkit in a completely cloud-native way. They will also discuss a powerful DevOps pattern from Intuit that combines Litmus with Argo for the creation of "chaos workflows" that uses the Litmus Operator to execute chaos while simulating other workload behavior (percentage load, for example) in parallel.
https://sched.co/ekDC
A
A
A
It
is
accepted
as
a
sandbox
project
early
this
year.
Our
mission
is
to
help
kubernetes,
srs
and
developers
in
practicing
kiosks
engineering
in
a
cloud
native
way.
My
data
integrate
and
amazon
or
the
current
maintenance
of
this
project.
Chaos
hub
is
an
asset
to
the
project
where
all
the
chaos
experiments
are
hosted
together
for
the
community.
A
In
fact,
90
percent
of
the
resilience
of
your
application
depends
on
some
other
cloud
native
service
or
kubernetes
itself.
How
do
you
achieve
resiliency?
The
answer
is
practice.
Chaos,
engineering
right.
So
what
is
chaos
engineering?
It
is
a
practice
of
introducing
a
random
fault
into
a
system
that
is
running
at
steady
state
and
then
observing.
If
the
steady
state
has
been
regained
or
not,
if
s,
then
the
system
is
resilient.
If
not,
you
found
a
weakness.
A
How
do
you
run
this
chaos
and
practice
chaos,
engineering
at
scale,
so
for
that
litmus
has
introduced,
chaos,
workflows,
chaos,
workflow
is
built
on
top
of
an
argo
workflow
or
it
uses
an
argo
workflow
to
run
multiple
experiments.
You
can
configure
to
run
them
in
sequence
or
in
parallel,
and
then
this
litmus
chaos
workflow
consolidates
the
results
of
such
experiments
to
give
a
meaningful
result
to
the
user,
because
this
entire
workflow
is
run
declaratively
or
configured
declaratively.
A
You
can
practice
chaos,
engineering
using
github's
practice.
Let's
look
at
litmus
chaos.
Architecture
litmus
provides
a
helm
chart
with
which
you
can
install
a
litmus
portal,
and
then
you
can
run
agents
litmus
agents
on
different
kubernetes
clusters,
where
you
need
to
practice,
chaos,
engineering
or
you
need
to
run
kiosk
experiments.
This
can
be
the
same
cluster
on
which
you
install
portal.
A
For
your
analysis
from
litmus
portal,
you
can
run
chaos
on
different
kubernetes
clusters,
a
multi-cloud
environment
or
a
hybrid
cloud
environment
as
well.
So
this
really
means
that
with
litmus,
you
can
practice
chaos
engineering
across
your
enterprise
and
it's
not
just
a
set
of
experiments
that
you're
putting
together,
but
it's
a
tool
set
that
provides
the
entire
infrastructure
required
for
running
chaos,
engineering
at
scale
across
your
clusters
in
your
enterprise.
A
So
you
can
practice
your
chaos
engineering
most
of
the
time
with
already
available
experiments,
70
or
more
than
70
percent
of
the
experiments
are
already
made
available
for
you.
You
construct
the
remaining
experiments
using
bitmus
sdk
and
if
you
think
this
new
experiment
is
going
to
be
useful
for
your
users,
you
can
also
upstream
such
an
experiment
to
back
to
the
hub.
A
You
will
see
that
there
are
currently
there
are
about
22
generic
chaos,
experiments
the
most
famous
one
being
port
elite,
that's
the
one
that
everybody
starts
using
whenever
they
try
litmus,
but
one
of
the
other
important
one
is
also
cubelet
service
skill,
which
is
an
important
function
to
test
when
your
kubernetes
is
very
much
in
production
and
surveying
at
scale.
This
experiments
covered
a
lot
of
variety
across
network
cpu
memory,
disk
and
services
io
and
node
in
general.
A
A
A
A
This
is
how
a
litmus
portal
looks
like,
and
once
you
log
in
you
will
have
a
set
of
predefined
experiments
here,
I'm
showing
a
set
of
workflows
that
are
already
scheduled
and
run,
and
you
can
go
and
schedule
your
new
workflow,
and
that
gives
you
a
set
of
predefined
workflows,
which
you
can
choose,
and
then
you
can
tune
the
weightage
of
experiments
that
are
part
of
that
workflow,
and
then
you
schedule
it.
When
it
is
run,
you
can
go
and
see
how
a
given
chaos
workflow
was
executed.
A
Did
everything
go
right
or
not,
and
it
gives
a
good
picture
of
the
sequence
of
experiments
that
were
run
and
also
litmus
portal
provides
a
good
amount
of
analytics
around
this
chaos
experiments
and,
as
stated
earlier,
the
mission
is
to
help
developers
and
sres
to
practice
chaos,
engineering
on
kubernetes
in
a
kubernetes
native
way.
With
that
introduction,
let
me
switch
this
session
to
sumit.
B
B
B
I
play
kubernetes
lot
of
open
source
being
in
java
and
python
for
a
long
time
doing
a
lot
of
testing
and
performance
work
and
leveraging
cloud
on
observability
platform.
I
am
leading
a
reliability
engineering
team.
As
part
of
that
team.
We
are
building
a
paved
road
which
help,
which
is
on
open
source
tools,
providing
infrastructure
and
reliability
for
any
specific
service.
B
It
has
a
three
solution:
chaos,
engineering,
performance
and
infrastructure.
Now
this
team
is
actually
helping
the
intuit
developer
platform,
which
is
building
next-gen
kubernetes
platform
for
into
it.
We
have
thousands
of
developers
and
hundreds
of
clusters
with
the
more
than
two
thousand
plus
services
already
onboarded,
and
this
one
is
going
lips
and
bounds.
B
Little
bit
about
my
chaos.
Journey.
I've
been
working
on
chaos
from
last
couple
of
years,
initially
been
working
mainly
on
the
application
side,
where
we
were
building
the
hystic
circuit
breaker,
as
well
as
a
point
two
point
solution
like
aws
chaos,
monkey
and
simian
army
kind
of
stuff.
We
did
some
game
day
so
that
bring
a
lot
of
awareness
about
chaos.
B
B
We
added
many
extensions
when
hence
a
money
extension
and
later
we
wrap
it
up
in
service
and
created
a
node-based
use
case
during
that
time
there
were
many
incident
happen
not
to
our
platform,
but
a
few
of
our
products
which,
being
there
in
fraud
that
create
a
bit
awareness
about
that.
How
someone
will
go
and
build
the
resiliency
for
this?
So
what
we
started
doing
is
that
we
try
to
see
that
if
we
can
put
this
work
to
the
kubernetes
and
then
we
figure
it
out
that
it
is
a
little
challenging.
B
We
cannot
put
that
because
of
security
compliance
and
it's
not
a
native
kubernetes.
So
we
did
the
initial
poc
with
the
litmus
and
we
have
many
use
cases
more
than
70
plus
use
cases
specifically
from
application
cloud
and
platform
site.
We
we
don't
wanted
to
rebuild
everything
from
scratch,
so
what
we
have
use
and
we
build
with
a
small
pc
and
help
from
the
community.
B
We
build
a
plug-in
infrastructure
where
all
our
work
has
been
being
executed
by
a
custom
resource
and
later
point
of
time.
We
figure
it
out
that
how
this
work
can
be
added
as
part
of
the
ci
cd
and
a
workflow.
We
work
closely
with
our
workflow
team
and
build
the
solution,
and
after
that
we
have
introduced
the
performance
as
well
as
the
chaos
side
by
side
and
executed
via
jenkins,
speculative
pipeline.
B
This
custom
resources
are
the
one
which
is
actually
being
invoking
our
framework
and
with
the
the
right
kind
of
our
back,
we
are
targeting
specific
application,
as
well
as
a
the
cube
system,
specific
namespaces
and
all
the
data.
We
are
right
now
pushing
to
various
monitoring
and
observability
solution,
and
this
has
been
executed
by
jenkins
pipeline,
taking
one
level
down
how
this
magic
is
happening.
B
B
B
This
custom
resource
is
using
an
image
which
is
a
python
based
container
image
and
in
this
python,
based
container
image,
we
are
using
chaos
toolkit
framework,
which
has
many
extension.
We
use
existing
as
well
as
we
build
our,
and
then
it
has
a
specific
logic
written
which
is
using
with
the
steady
state
hypothesis.
B
B
B
This
workflow
will
look
that
whatever
instruction
has
been
provided
as
part
of
this
workflow,
that
experiment
exists.
So
this
experiment,
which
already
being
set
it
up
on
this
specific
namespace
it
pick
that
specific
scenario
it
launched
the
chaos
runner.
This
chaos
runner
will
be
be
using
and
our
existing
framework
image
and
launch
the
experiment.
B
Now
why
ergo
workflow?
So
we
we
have
seen
that
you
can
execute,
keep
cuddles
and
yammers,
and
then
you
can
execute.
But,
logically
speaking,
if
you
really
wanted
to
execute
everything
as
part
of
pipeline
many
scenarios,
it
become
very
challenging.
So
automation
was
one
of
the
thing
now
with
ergo
workflow.
Everything
is
coming
as
a
one
yaml,
where
we
can
just
use
one
of
the
parameter
to
the
argo
submit,
and
we
can
invoke
that
specific
thing
and
everything
is
a
code.
B
So
you
don't
need
to
maintain
a
various
different
kind
of
yaml
and
we
have
a
scenario
of
a
hundreds
of
clusters.
It
become
very,
very
challenging
how
we
roll
it
out.
As
we
are
using
ergo,
there
are
a
lot
of
already
cost
optimization
and
resource
utilization,
optimization
already
done
that
which
is
actually
helping
in
the
cost,
which
is,
I
think,
one
of
the
more
important
thing
in
the
current
pandemic
going
and
it
as
we
have
introduced
the
chaos
with
the
performance.
We
are
getting
more
reliability.
B
We
are
not
only
looking
for
statefulness
of
the
chaos,
we
are
getting
the
statelessness
of
the
performance,
and
then
we
merge
both
of
them.
It
help
us
to
build
lot
of
complex
scenarios,
which
is
in
practical
writing,
equips
catalyamal
is
not
possible
and,
as
this
whole
execution
is
happening
in
a
manner
that
it
is
a
very
predictable,
it
bring
lot
of
trust
and
confidence
in
the
overall
setup.
B
Last
but
not
least
it
because
it's
a
code,
self-service
and
onboarding
is
very,
very
easy
and
then,
as
we
are
using
workflow
it,
it
become
the
complete
life
cycle.
Whatever
you
have
set
the
base
state,
it
will
finish
the
same
way
state
again
now.
This
is
the
final
chaos
ergo
workflow
setup,
how
it
will
look
like
where
you
could
see
that
we
are
running
the
chaos
as
well
as
the
performance
execution.
B
B
Once
I
apply
the
r
back,
I
can
go
the
execution,
which
is
the
chaos
experiment
using
curos
toolkit
latest
python.
By
the
time
it
started,
the
runner
pod
has
come
up
and
the
runner
part
will
initiate
the
container,
which
has
a
framework
which
is
here.
This
container
will
try
to
do
certain
action
based
on
that,
how
we
have
configured
that
so
I
have
kept
one
experiment
one
so
here
you
could
see
that
we
are
using
one
of
the
image
from
litmus
chaos,
kiosk
toolkit
image.
B
B
So
let's
see
here
now
here
it
actually
deleted
that
nginx
demo,
which
is
another
app
and
it
bring
down
the
delete
and
then
the
runner
part.
So
with
this
we
are
able
to
do
the
execution,
and
I
have
this
demo
script
available
at
your
fingertip.
Now,
let's
go
to
the
another
scenario
where
we
wanted
to
do
this,
but
we
wanted
to
use
the
ergo
to
do
that.
So
now.
B
B
B
B
So
this
is
the
way
it
is
executing.
Now
everything
is
done
now
I
will
go
and
I
will
say
that
I
wanted
to
execute
the
chaos
test
with
the
performance.
That's
the
give
me
the
value.
Add
off
that
if,
during
chaotic
operation,
something
is
happening
to
my
endpoint,
I'm
going
and
this
time
I'm
executing
the
test
and
it
will
impact
the
application
part,
and
here
I'm
side
by
side
running
the
performance
test
as
well
as
I'm
running
the
chaos
test.
B
B
I
can
go
and
look
for
workflow,
so
here
the
pdb
create
happen,
and
then
I'm
execution,
as
well
as
the
performance
just
both
are
happening
side
by
side.
Now,
one
chaos
has
been
finished
and
you
could
see
that
the
test
is
still
going
and
the
application
has
been
brought
down.
I
can
go
and
look
for
my
logs
here,
which
is
the
performance
test
during
this
time.
B
Here
you
could
see
that
during
certain
time
there
is
a
pod
has
been
impact
impacted.
You
could
clearly
see
that
and
if
I
go
and
look
for
other
one,
I'm
right
now
running
two
parts.
So
now
with
this,
you
could
see
that
we
can
go
and
identify
all
the
problem
on
your
application
and
we
probably
need
to
put
the
resiliency
build
on
the
tap
application.
A
Thank
you
sumit
for
that
wonderful
demo,
with
that
we
have
come
to
the
end
of
this
session
on
creating
chaos
workflows
using
litmus
and
argo.
Please
do
take
a
look
at
the
github
projects
of
both
litmus
and
argo.
Thank
you
again
for
watching
our
wonderful
conference
till
next
time
see
you
bye.
Thank.