►
From YouTube: Making Kubernetes Resilient with LitmusChaos
Description
Kubernetes is the most widely adopted cloud-native container orchestration tool. While companies are widely adopting Kubernetes for their microservice architecture solutions for the much-required competitive edge, it is paramount to ensure the resiliency of these systems for ensuring reliable services for everyone. But how can we ensure the reliability of these systems while they change at the pace of hours and minutes for the much-required competitive edge?
LitmusChaos is an end-to-end chaos orchestration framework that can assess the reliability of your cloud-native systems against the unforeseen weaknesses that lead to its failure. In this talk, we'll talk about why resiliency is the cornerstone of every cloud-native application architecture and how LitmusChaos can help DevOps engineers and SREs to validate the reliability of an entire business use case.
A
B
Hello:
everyone,
I'm
nilanjan,
a
software
engineer
working
at
harness
where
I'm
building
kiosk
engineering
at
scale
for
every
organization.
I
had
started
my
chaos
engineering
journey
at
chaos
native
the
company
behind
the
litmus
chaos
framework,
which
is
an
open
source
cnc
of
incubating
project
for
performing
chaos,
engineering
in
cloud
native
environments.
B
In
my
day-to-day
work,
I'm
responsible
for
assessing
the
chaos
engineering
requirements
for
any
given
scenario
and
authoring
the
chaos
experiments
that
can
suitably
validate
the
resiliency
of
an
application.
For
that
scenario,
from
my
experience,
I
can
tell
you
that
the
successful
chaos
engineering
practitioners
come
to
master
the
skill
of
fabricating
every
chaos
experiment
on
a
case-by-case
basis
to
leverage
the
most
results
out
of
each
experiment.
A
So
today's
agenda
will
be
first
of
all,
we'll
be
talking
about
chaos,
engineering.
What
is
it
and
why
do
we
need
chaos
engineering?
Then
we
will
be
introducing
letting
litmus
chaos
talk
about
its
core
components
and
features,
and
then
lastly,
nilanjin
will
be
giving
a
short
demo
of
how
cures
can
be
injected
in
an
application,
so
without
any
further
delay.
Let's
get
started
talking
about
the
era
of
cloud
computing.
A
According
to
the
recent
reports
of
o'reilly
cloud,
adoption
has
been
rising
steadily
with
around
90
percent
of
organizations
already
using
it,
and
75
percent
of
respondents
are
working
in
an
organization
which
has
cloud
infrastructure,
e-commerce
and
retail,
finance
and
banking
software
are
the
proactive
ones.
However,
with
the
increasing
adoption
of
cloud
for
all
its
benefits,
there
also
comes
some
challenges
as
well,
and
down
times
are
one
of
them.
A
So
the
need
of
the
r
is
to
mitigate,
if
not
eliminate,
the
predicament
of
the
downtimes.
By
breaking
things
on
purpose,
we
get
to
see
the
behavior
of
the
application
in
a
whole
new
light,
and
this
diagram
is
a
very
basic
flow
of
how
chaos
engineering
can
be
practiced.
First
of
all,
you
identify
the
steady
state
conditions
of
the
application.
A
Then
we
introduce
we
intentionally
introduce
a
fault
and
then
we
check
we
continue
to
check
if
the
slos
are
met
or
not.
If,
yes,
then
the
application
is
resilient
and
if
no,
we
identify
the
weakness,
work
upon
it,
and
then
chaos
engineering
can
be
practiced
again
as
a
part
of
the
cycle
so
yeah.
Now
it's
time
to
introduce
litmus
scales,
which
is
an
open
source
cloud
native
curve
engineering
framework.
It
is
also
a
cnc
incubating
project
with
adoption
of
across
several
organizations.
A
Here
are
some
of
the
stats.
Litmuscus
has
around
52
experiments
with
over
2.5
k,
plus
github
stars.
More
than
88k
experiment
runs,
so
yeah
are
talking
about
the
core
components
of
litmus
scales.
Litmus
scales
at
its
core
has
two
components
there.
First,
one
is
the
cure
center,
which
is
the
single
source
of
truth
of
all
the
cures
activities,
be
it
the
creation
of
chaos,
workflows,
scheduling
them
managing
the
team.
All
these
skills
activities
are
managed
through
care
center
and
the
second
one
is
cures
agent,
which
is
nothing
but
a
target
cluster.
A
Where
the
cures-
okay,
against
which
the
cures
is
injected
coming
to
cure
center,
they
will
talk
about
some
major
features
that
it
provides.
First
of
all,
we
have
the
cures
workflows.
Litmus
skills
provides
like
has
the
concept
of
chaos
workflow,
which
is
nothing
but
a,
but
an
orgo
workflow,
where
more
than
one
experiments
can
be
can
be
added
either
in
in
serial
manner
or
parallely,
and
litmus
also
provides
multiple
options
to
create
these
workflows.
A
Users
can
do
it
from
templates
or
they
can
create
their
own
custom
workflow
by
from
scratch,
or
can
use
the
predefined
pre-created
gambles
once
the
cures.
Once
these
workflows
are
created
it.
These
can
be
scheduled
either
as
cron
workflow
or
as
a
singular
run,
and
let
us
also
provides
the
feature
of
attaching
priorities
to
the
chaos
experiments
in
each
workflow,
based
upon
your
own
use
case,
then
coming
to
workflow
management.
First
of
all,
it
has
githubs.
I
will
talking
about
it
a
bit
later.
A
Secondly,
it
also
provides
like
it
allows
additional
for
image
from
your
own
custom
image
server,
which
can
be
either
public
or
private.
Then,
once
the
chaos
workflows
are
executed,
the
like
users
can
measure
and
analyze
the
resilience
score
of
of
the
workflow,
then
litmus
also
provides.
It
also
provides
the
multi-tenancy
support
where
users
can
create
their
own
team,
then
run
workflows
and
each
of
the
resources
resources
have
comes
with
a
very
flexible
role,
based
access
controls
and
yeah.
A
So
it
has
also
scoped
support,
which
can
be
either
like
a
name,
space
scope
or
the
cluster
wide
scope
as
well.
Then
it
has
it
has.
It
also
supports
monitoring
and
observability,
where
users
can
connect
their
own
data
source
visualize,
the
workflow
runs
and
start
their
workflow
runs,
and
the
schedules,
the
their
statistics
and
users
can
also
compare
to
our
one
or
like
two
or
more
workflows
and
upload
their
upload
and
fine-tune
their
own
dashboards.
A
And,
lastly,
they
can
also
monitor
the
cures
in
real
time.
Then,
coming
to
the
githubs
for
cures,
get
offs
once
enabled
provides
a
single
source
of
truth
for
the
cures
workflows.
That
means
that
these
cures
workflows
will
be
synchronized
bi-directionally
between
the
cure
center
and
the
git
source.
Secondly,
it
also
supports
tracking
which
provides
even
tracker,
which
is
a
like
a
micro
service
that
is
used
to
automatically
launch
the
subscribed
workflow
in
case
there.
Any
changes
happen
to
the
target
application,
such
as
app
upgrades,
etc.
A
B
Hey
there,
let's
get
down
to
creating
some
chaos
with
litmus
chaos,
but
before
that,
do
you
love
socks?
Oh,
come
on.
Everyone
loves
socks,
so
let
me
bring
you
sockshot
an
online
e-commerce
marketplace
where
everyone
loves
to
buy
their
socks.
What's
so
special
about
this
application
is
that
it's
completely
made
out
of
kubernetes
microservices?
B
So
as
part
of
this
experiment,
we
are
going
to
actually
delete
a
few
parts
from
this
catalog
deployment.
So
let
us
hop
into
our
kubernetes
terminal
and
let's
see
that
what
are
these
deployments
look
like
so
over
here?
You
can
see
that
we
have
our
sock
shop
name,
space
in
which
we
have
a
bunch
of
deployments,
and
one
such
deployment
is
catalog,
as
you
can
see
as
part
of
this
deployment,
we
have
exactly
two
parts
and
both
of
them
are
in
running
condition
right
now.
B
So
let
me
ask
you
that
what
do
you
think
will
happen
if
we
just
go
ahead
and
delete
both
of
these
parts
for
a
certain
chaos
interval?
What
do
you
think
would
be
able
to
access
our
catalog?
Even
if
our
parts
are
gone?
Well,
let
us
find
out
using
litmus
scales.
So
let
me
just
write
hop
right
into
our
chaos
center
and
with
that
I'll
go
ahead
and
schedule
a
workflow.
B
Let
me
just
select
this
self
agent,
since
my
application
is
also
deployed
in
the
same
cluster
in
which
litmus
scale
slice
will
go
next
and
I'll
choose
the
creation
of
a
new
workflow
using
chaosa.
So
what
is
chaos
up?
Well,
it
is
an
open
source
marketplace
for
all
the
different
kiosk
experiments
you
have
available
as
part
of
litmus
chaos,
so
I'll,
select
the
chaos
hub
and
just
go
next.
B
So,
first
of
all,
we
need
to
specify
our
namespace,
so
this
will
be
the
soft
shop,
since
our
application
is
in
stock
shop
namespace,
then
the
kind
of
resource
that
we
want
to
target
so,
as
we
saw
the
catalog,
is
a
deployment
that
we
have
in
the
soft
shop
namespace.
So
we'll
select
this
as
deployment,
then
we'll
be
specifying
the
label
that
we
want
to
target.
So
we
are
going
to
select
name
is
equal
to
catalog,
since
that
is
the
label
of
our
deployment
that
we
want
to
target.
B
B
Probes,
help
you
to
automate
the
process
of
hypothesis
checking
as
part
of
your
litmus
chaos
experiment.
So,
as
part
of
this
part
delete
experiment,
we
can
definitely
define
some
probes,
which
can
help
us
to
automate
the
process
of
checking
that
whether
our
body
experiment
is
resulting
in
a
successful
hypothesis
condition
or
not.
So
what
is
the
hypothesis
condition
that
we
want
to
define?
Well,
we
just
want
to
be
ensured
that,
while
we
are
deleting
our
parts,
while
we
are
performing
our
body
lead
experiment,
our
catalog
should
still
be
available.
B
So
how
can
we
ensure
that
well
I'll
like
to
show
you
an
api
endpoint
that
we
have
over
here?
So
it
is
the
catalog
api
endpoint,
which
is
actually
enabling
the
display
of
all
these
items
that
are
over
here.
So,
as
you
can
see
right
now
that
this
catalog
api
endpoint
is
still
available.
If
I
refresh
this,
we
are
still
getting
all
the
information
of
the
songs
that
are
that
are
available
to
us.
B
But
the
question
is:
is
this
catalog
endpoint
will
still
be
available
once
we
are
deleting
our
parts.
So
let
us
try
to
validate
this
exact
condition
using
the
litmus
probes
in
an
automated
fashion,
so
back
into
my
chaos
center.
Let
me
just
add
a
new
probe
I'll
name,
this
I'll
name
this
catalog
pro,
and
it
is
a
type
of
an
http
group.
B
Why
an
http
pro
well,
let
me
just
define
the
probe
and
I'll
explain
to
you
that
why
are
we
doing
an
http
pro,
so
we
need
to
add
some
time
out
as
part
of
this
pro.
That
is,
after
which
time
we
can
say
that
well,
our
http
probe
has
timed
out.
So
we
can
see
a
timeout
of
three
seconds
and
let
us
just
retry
twice
with
an
interval
of
three
seconds
and
this
http
probe
expects
a
url.
B
So
here's
where
we
are
going
to
actually
put
the
url
that
we
copied
from
our
catalog
api
endpoint,
I'm
putting
this
over
here
and
what
we
are
going
to
do.
Is
we
are
actually
going
to
send
some
get
requests
at
this
api
endpoint
and
the
criteria
that
we
are
going
to
check
for
upon?
Each
of
this
request
is
whether
we
are
getting
a
response
code
of
200
or
not.
So,
as
you
know,
if
we
are
getting
a
200
response
code,
it
means
that
everything
is
fine.
B
We
are
still
able
to
access
our
api
endpoint
for
the
catalog
and
therefore
we
should
be
able
to
still
access
our
catalog
in
our
application.
But
if
we
are
not
getting
a
200
response
code,
then
it
certainly
means
that
something
is
off.
Something
is
not
quite
right
and
therefore
we
can
say
that
well,
our
hypothesis
is
not
validated.
It
is
therefore
right
now
invalid
and
therefore
we
can
flag
the
experiment
as
fail.
B
So
let
me
actually
define
this
and,
as
you
can
see,
the
catalog
group
has
been
added
to
this
experiment.
Let's
go
next
and
finally,
we
are
in
the
part
we
need
to
tune
the
experiments,
so
we
need
to
provide
a
chaos
interval
as
well
as
a
chaos,
duration,
I'll
select,
30
for
both
of
them,
and
then
we
have
the
parts
affected
percentage.
B
B
And
we
are
here
where
we
can
specify
the
reliability
score
of
this
experiment.
Well,
what
does
this
does
is?
Basically,
you
need
to
set
a
score
that
you
need
as
part
of
your
final
resiliency
score
calculation.
So
let
us
set
the
weight
directly
to
100.
That
is
10,
which
will
cause
us
to
have
full
weightage
of
this
experiment.
B
B
So,
as
you
can
see
over
here,
the
bar
delete
experiment
is
just
about
to
start,
and
here
we
can
see
that
the
port
delete
experiment
has
actually
started.
So
how
can
we
confirm
that?
Well,
whether
our
bots
are
still
available
to
us
or
not,
since
the
quantitative
experiment
has
actually
started
right?
B
So,
as
you
can
see,
the
pod
delete
experiment
has
actually
caused
both
of
our
parts
to
go
missing
and
therefore
we
cannot
see
anything
here
in
our
catalog,
but
the
most
important
part
of
your
chaos
experiments
is
obviously
you
need
to
be
aware
that
something
is
not
quite
right.
While
you
are
running
these
experiments,
which
is
paramount
to
understanding
where
your
application
is
weak.
Litmus
chaos
provides
an
observability
panel
right
as
part
of
your
auxiliary
stack
and
it
uses
external
data
sources
such
as
prometheus
in
order
to
generate
your
application
dashboards.
B
B
Since
the
beginning
of
the
chaos
experiment,
as
you
can
see,
the
catalog
is
not
the
same.
The
queries
per
second
from
catalog
is
progressively
going
down
since
we
have
injected
this
chaos
and
at
the
same
time,
the
catalog
latency
is
also
going
very
high.
Since
there
are
no
parts
at
the
moment
to
serve
the
request,
we
can
validate
the
same
using
a
grafana
dashboard
that
which
we
can
set
up
as
part
of
our
kubernetes.
B
So
we
already
have
a
prometheus.
We
have
integrated
grafana
with
it
and,
as
you
can
see
during
the
chaos
interval
the
annotated
portion,
you
can
see
that
our
queries
per
second
for
catalog
metrics
is
going
down,
as
we
had
seen
earlier
in
our
chaos
center
observability
stack
and
in
a
similar
fashion.
The
catalog
latency
is
going
way
above
for
the
same
reason
that
the
parts
are
not
there
anymore
and
therefore
our
queries
are
not
being
served.
B
All
right,
so
our
application,
our
experiment,
is
right
now
completed
and,
as
we
can
see,
the
experiment
was
unsuccessful,
but
why
is
it
unsuccessful
before
we
get
into
that?
Let's
just
dive
right
into
our
kubernetes
console
to
see
that
whether
the
experiment
effect
is
reverted
or
not.
That
is
whether
our
ports
are
back
here
or
not,
and
as
you
can
see
that
well,
the
pods
are
indeed
back.
B
The
effects
of
this
chaos
has
been
reverted
and
we
can
confirm
the
safe
if
we
go
to
our
application
and
if
we
try
to
reload,
as
we
can
see,
the
catalog
is
again
back.
We
are
right
on
track,
but
it
is
paramount
to
understand
that
what
caused
our
chaos
experiment
to
fail,
why
has
it
failed
and
to
do
that?
We
can
definitely
view
our
logs
the
experiment
logs
right
from
our
chaos
center.
B
So
let
us
try
to
inspect
the
logs
in
order
to
understand
that
why
exactly
our
chaos
experiment
has
been
marked
field,
so
we
can
see
the
different
stages
of
the
experiments
demarcated
over
here
in
in
our
logs
and
as
part
of
this
experiment,
you
can
see
that
we
are
deleting
two
parts.
First
of
it
is
this
catalog
port,
and
this
second
part
as
we
are
going
into
our
experiment.
B
It
is
not
what
we
had
earlier
expected
that
is
200,
and
therefore
this
has
been
the
main
reason
that
why
our
chaos
experiment
failed,
because
the
probe
that
we
had
defined
for
the
validation
of
whether
our
catalog
endpoint
is
available
or
not.
Well,
that
failed,
because
we
didn't
get
an
actual
value
of
500,
sorry
200
and
the
actual
value
that
we
received
was
500.
B
So
that's
how
you
will
be
assured
that
why
exactly
your
up
your
kiosk
experiment
field
and
you
can
therefore
take
measures
to
improve
the
application
resiliency.
As
a
final
step,
you
can
also
verify
the
kiosk
results.
The
chaos
result
basically
displays
that
a
summary
of
your
chaos
experiment.
So,
as
you
can
see,
the
experiment
status
is
completed,
but
the
verdict
is
failed
and
the
reason
for
that
is
basically
the
group
success
percentage
is
zero.
B
The
catalog
probe
that
we
had
defined,
which
is
an
http
type
of
probe,
it
failed,
and
therefore
we
come
to
an
end
of
our
demo.
We
saw
that
how
we
can
use
litmus
chaos
in
order
to
make
our
kubernetes
cluster
more
resilient
by
testing
out
different
chaos
scenarios
using
different
chaos,
experiments.
We
saw
how
the
body
elite
experiment
helped
us
in
order
to
validate
the
resiliency
of
our
application
base
workshop
application,
and
we
saw
that
how
we
can
observe
the
chaos
in
real
time
using
our
auxiliary
stack.
So
that's
it
for
this
demo.