►
From YouTube: Lightning Talk: Using Argo Workflows to Curate Chaos Engineering wi... Amit Kumar Das & Saranya Jena
Description
Lightning Talk: Using Argo Workflows to Curate Chaos Engineering with LitmusChaos - Amit Kumar Das & Saranya Jena, Harness.io
As resilience use-cases proliferate, Chaos Engineering has become a compelling practice for enhancing your application resilience in production. As a large number of organizations and individuals are moving towards the Kubernetes-Native approach, an important problem arises, testing these applications the right way to prevent outages. In this talk, we will talk about how the LitmusChaos CNCF project leveraged the Argo project to curate Chaos Engineering Workflows as set of actions strung together to achieve desired chaos impact on a Kubernetes cluster. While simulating real-world failures via chaos experiment injection on developer/staging environments as part of a left-shifted, continuous validation strategy, it was essential to create a sequence instead executing standalone chaos injection actions. They are an effective mechanism to simulate real world conditions & gauge application behaviour in an effective manner. Further, we will cover the role of Argo in the evolution LitmusChaos 1.x to 2.x by constructing and executing ChaosWorkflows with the help of Argo and observe the changes which are introduced when the system/application is under Chaos.
A
Hey
everyone
welcome
to
this
lightning
talk
on
using
orgo
workflows
to
curate
kiosk
engineering
with
litmus
skills.
This
is
sarna
and
I
work
at
harness
at
senior
software
engineer
and
I'm.
Also
one
of
the
core
contributors
to
litmus
kills.
So
to
start
off
with.
First
of
all,
we
need
to
know
what
is
resilience
so
resilience
is
basically
a
system's
ability
to
sustain
a
fault
and
bring
itself
back
up.
Let's
say,
a
port
gets
evicted
from
the
node.
So
what
is
its
state?
Is
it
healthy
or
not?
So
it
is
healthy.
A
A
Then
moving
ahead
down
times
are
expensive,
not
just
in
terms
of
money,
but
also
there
are
other
aspects
as
well.
The
secondary
factors
which
are
equally
important
as
money,
that
is
some
of
them-
are
loss
of
customer
confidence,
damage
to
Brand,
Integrity,
loss
of
employee
morale
and
a
lot
others.
So
keeping
all
these
factors
in
mind,
we
definitely
don't
want
to.
We
definitely
want
to
avoid
down
times
at
any
cost.
So
one
of
the
way
to
achieve
this
is
adopting
the
practice
of
chaos.
A
Engineering,
so
with
that
I
would
like
to
introduce
kios
engineering,
which
is
the
process
of
testing
a
distributed
computing
system
by
injecting
fault
intentionally.
So
the
goal
here
is
to
identify
the
weaknesses
in
the
application
through
control,
control,
experiments
and
like
and
check
and
check
if
it
can
sustain
it,
can
withstand
an
unexpected
Behavior
or
not
so
yeah,
so
how
it
is
done.
First
of
all,
you
need
to
identify
the
steady
state
conditions.
A
A
The
chaos
injection
was
done
only
one
experiment
at
a
time.
So
keeping
all
these
facts
in
mind,
we
decided
to
integrate
all
go
workflows
with
litmus
cures,
and
these
are
some
of
the
features.
The
main
reasons
why
we
decided
why
we
chose
or
go
workflows.
We
can
defined
workflows
where
each
step
in
the
workflow
is
a
container.
So,
as
I
said
earlier,
a
chaos
injection
was
done
only
one
experiment.
A
At
a
time
we
can
execute
only
one
chaos
experiment,
but
with
this
we
can
have
multiple
chaos,
experiments
spun
together
as
a
one
single
workflow,
which
is
considered
as
a
chaos
scenario.
So
each
chaos
experiment
can
be
a
single
container.
Then.
Second,
second
Point,
moving
on
to
the
second
Point
proper
visualization
of
workflow
or
the
chaos
scenario,
can
be
done
using
a
graph,
so
that
is
also
provided
by
orgo.
Then
next
is
individual
template
in
the
chaos.
A
Workflow
can
be
tuned
independently,
like
we
can
tune
each
and
each
individual
template
without
affecting
the
whole
chaos
scenario
or
the
whole
workflow,
and
then
sequence
of
steps
can
be
changed
as
per
the
requirements
such
as,
like,
let's
say
you
added
experiment,
one.
First
and
then
you
added
experiment
two,
but
later
you
decided,
you
want
to
alter
the
sequence,
so
you
can
like.
We
can
also
do
that
with
orgo
or
you
can
put
move
the
experiment
to
in
the
first,
so
that
can
be
done.
A
And
lastly,
this
is
also
one
of
the
important
features
that
is,
the
experiments
can
be
run
in
both
parallel
or
in
sequential
manner,
which
was
not
there
in
litmus
1.0,
so
yeah.
So
these
are
some
of
the
important
features
and
reasons
why
we
decided
to
shift
to
our
goal.
Workflows
and
that's
it
and
now
Amit
will
be
giving
a
small
and
a
brief
demo
of
how
the
chaos
engineering
can
be
done
with
orbo
workflows
and
lit
muscures.
Thank
you.
Thanks.
B
So
if
I
see
like
we
have
installed
the
back
of
anthos
application
in
the
bank
namespace-
and
we
have
a
few
Services
running
here
so
from
the
QR
Center-
we'll
go
to
create
a
new
scenario,
and
here
we
can
select
our
agent
and
and
in
the
next
step
we
can
select
the
chaos
Hub.
So
QR
sub
is
basically
a
marketplace
where
we
can
find
a
lot
of
experiments
which
can
be
used
as
per
a
user's
requirement.
So
we
have
selected
the
kiosk
option.
B
So
yeah
so
I
have
added
the
Pod
delete,
experiment
and
I'll
try
to
Target
the
bank
name
space
along
with
the
balance
reader
label,
so
this
will
actually
Target
the
balance
reader
deployment,
which
will
basically
delete
the
balance
reader
pod,
and
we
should
see
that
the
balance
should
not
be
displaying
here.
So,
let's,
let's
go
ahead
and
click
next,
so
here
we
can
add
different
types
of
probes,
so
I'll
just
go
with
the
default
configuration
so
I'll
click.
B
Next,
and
in
this
step
we
can
tune
our
chaos
experiment,
so
I'll
just
make
some
changes
and
I'll
just
increase
the
total
chaos
duration
to
60
and
keep
the
other
environment
variables
as
it
is,
and
I'll
just
click.
Next,
and
in
this
step
we
can
make
some
Advanced
chaos
engine
engine
configurations,
so
I'll
just
keep
everything
as
default.
I'll
just
finish
this
and
yeah,
so
I
have
made
some
changes
in
my
chaos,
experiments
that
should
be
visible
when
I
inject
chaos
and
if
I
want
to
see
the
Manifest.
B
B
So
for
this
we
are
using
the
Chrome
workflows,
which
is
also
provided
by
Argo,
and
we
have
several
options
like
we
can
tune
the
recurring
flow
of
the
Chrome
workflow
and
for
the
for
the
demo.
I'll
just
create
a
current
schedule,
so
I'll
be
scheduling.
It
now.
I'll
click
next-
and
this
is
the
final
step
where
we
can
see
all
the
details
that
we
have
provided
in
the
previous
steps.
B
So
I'll
just
click
finish
and
yeah,
so
a
new
scenario
is
created
and
then,
if
I
click
here,
so
we
can
see
that
the
Run
has
started
and
I
can
actually
visualize
the
overall
workflow
so
in
this
workflow
we'll
be
targeting
this
balance
reader
service.
So
if
I
go
here,
I
can
see
that
the
balance
Reader
Service
is
up
and
running
and
if
I
go
back
to
my
litmus
namespace,
where
we
can
see
that
a
pod
delete
trainer
has
been
created
which
will
actually
Target
this
balanced
Reader
Service.
B
So
we
should
see
that
this
balance
reader
Port
getting
terminated
and
yes,
it
is
happening
now
and
if
I
refresh
this
page,
I
should
see
that
this
balance
is
not
visible,
so
yeah.
So
we
have
success
fully
induced
Chaos
on
this
Bank
of
anthos
application
on
this
balanced
Reader
Service.
If
I
try
to
add
some
amount
like
and
I
deposit
it,
so
this
will
be
happening,
but
we
will
not
be
able
to
see
the
balance.
So
we
from
an
end
user
side.
B
You
don't
want
to
see
transactions
going
on,
but
you
are
not
able
to
visualize
or
you.
You
are
not
able
to
see
the
overall
balance,
so
this
is
how
chaos
engineering
is
performed
using
the
litmus
kios
and
QR
Center,
and
we
are
using
the
basic
Argo
workflows
to
curate
chaos,
engineering,
so
yeah.
So
let's
wait
for
a
while
to
let
this
experiment
complete
and
we
can
see
all
the
logs
over
here.
So
these
are
the
logs
which
are
available
with
the
kiosk
experiment
yeah.
B
So
we
can
see
that
the
workflow
is
completed
and
we
have
the
overall
logs
here.
We
also
got
the
chaos
results.
Like
the
phase
is
completed
and
the
vertical
is
passed.
We
got
the
proof
success
percentage
and
if
you
go
back,
we
can
see
that
the
balance
reader
pod
is
back
into
its
ready
and
running
State.
And
if
you
go
back
to
the
application,
if
I
do
a
quick,
refresh
I
should
see
the
balance
coming
up
yeah,
so
the
balance
is
coming
up.
B
So
we
have
successfully
injected
the
chaos
on
one
of
its
services,
which
is
the
balance
reader
services
and
yeah.
We
can
also
try
to
inject
Chaos
on
some
other
services
as
well,
so
yeah
these
these
all
can
be
performed
with
the
chaos
and
of
litmus
chaos,
and
so,
as
part
of
the
demo
like,
we
have
seen
how
we
are
using
Argo
workflows,
as
well
as
the
Crone
workflows
to
inject
Chaos
on
different
microservices,
and
that
was
pretty
much
it
from
my
side.
So
the
thank
you.