►
From YouTube: Chaos Engineering For Hybrid Targets With LitmusChaos - Uma Mukkara, Karthik S & Prithvi Raj
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Chaos Engineering For Hybrid Targets With LitmusChaos - Uma Mukkara, Karthik S & Prithvi Raj, Harness
Speakers: Uma Mukkara, Karthik S, Prithvi Raj
Downtime amongst platform resources is very common today and is one of the reasons why SRE teams are heavily invested in Chaos Engineering. Another upcoming phenomenon is that of multi-cloud deployments wherein organizations attempt to achieve redundancy through cross-cloud instances of their services. These requirements together warrant a vendor-agnostic approach to chaos experimentation, preferably from a common control plane (read: Kubernetes) with the flexibility to add custom constraints around the fault injection process as well as the expectations (hypothesis) around them. There is also a need to marry this process with the standard CI/CD pipelines to derive maximum benefit via continuous chaos testing. In this session, the maintainers of the LitmusChaos project explain how this can be achieved.
A
B
A
B
A
B
A
So
this
is
just
a
quick
refresher
just
to
set
the
context
before
we
start
off
so
chaos
engineering
is
essentially
injecting
controlled
failures
into
your
environment
and
the
idea
is
to
introduce
these
parts
to
find
out
weaknesses.
If
there
is
something
that's
unknown,
something
that's
you've
saw
probably
not
accounted
for.
You
will
find
those
things
out
using
chaos,
engineering,
so
there's
a
lot
of
material
on
the
internet
around
what
it
is.
It's
especially
used
for
distributed
systems
right
as
far
as
litmus
chaos
goes.
It's
a
project
that
started
around
2018
19
time.
A
We
are
an
incubation
project.
The
litmus
tool
or
framework
is
kubernetes
native.
It's
just
a
kubernetes
application.
You
can
install
it
via
command,
Helm
command.
You
can
do
a
Helm
install
and
you
can
just
go.
Do
whatever
configuration
you
need
in
the
values,
so
you
can
do
the
installation
of
request
framework
as
a
limited
scope,
one
just
for
yourself
or
you
can
do
it
at
a
larger
scale.
For
your
organization
there
is
multiple
modes,
supported,
namespaced
mode
and
clustered
mode,
so
in
the
namespace
mode.
A
Litmus
will
allow
you
to
orchestrate
chaos
in
a
specific
namespace
that
might
have
been
allotted
to
you
as
a
developer.
That's
where
you're
experimenting
or
you
could
be
the
SRE,
and
you
would
like
to
set
up
one
control
plane
for
all
the
developers
you're
trying
trying
to
create
some
kind
of
a
self-service
environment.
A
So
you
could
go
install
it
in
a
cluster
mode,
once
you've
got
the
control
plane,
running
and
I'll
probably
show
a
quick
demo
before
we
get
into
the
release
details
and
all
that,
once
you
set
up
your
control
plane,
that's
where
all
the
chaos
management
happens,
you're,
basically
connecting
different
Target
environments
into
that
control
plane.
These
Target
clusters
could
be
residing
anywhere
across
any
Cloud
platform,
or
it
could
be
your
internal
Data
Center.
A
You
could
be
connecting
clusters
or
you
could
be
connecting
namespaces
as
well,
so
you
could
have
litmus
control
plane
in
one
cluster
you're
connecting
either
another
cluster
or
one
namespace
in
another.
Cluster
right
depends
upon
how
much
autonomy
you
have
on
your
infrastructure
and
when
you
connect
that
you
will
be
basically
running
an
agent
in
the
cluster
or
namespace
that
you've
connected,
and
that
agent
is
what
we
call
as
the
chaos
delegate
here.
A
That
carries
out
the
business
logic
of
your
chaos
experiment
and
when
we
talk
about
the
business
logic
of
chaos,
experiment
we
are
talking
here.
The
fault
injection,
the
pre
and
post
fault
checks
that
we
would
like
to
do
as
well
as
any
other
hypothesis
validation
that
you
might
want
to
done.
You
might
be
querying
some
metrics.
You
might
be
looking
at
the
availability
of
some
Downstream
services,
so
any
hypothesis
that
you
have
around
how
your
application
should
behave
that
can
be
bond
in
into
your
experiment
spec.
A
So
in
the
litmus
control
plane,
you
will
find
certain
chaos,
artifact
sources
embedded.
We
call
them
the
chaos
Hub,
so
chaos
Hub
can
be
sort
of
spoken
as
an
Open
Marketplace
of
chaos,
experiments
that's
where
the
community
goes
and
pushes
their
experiments
into.
So
you
can
pull
those
experiments
from
their
construct,
complex
scenarios
using
them
and
go
ahead
once
you've
created
something
that
you
feel
is
worth
sharing
with
others.
You
can
push
the
scenario
back
into
the
chaos
Hub
as
well.
A
So
what
you
see
here
are
just
a
couple
of
chaos:
hubs
on
on
the
demo
environment
that
I
have
so
once
you've
gone
ahead
and
logged
in
into
the
control
plane
once
you've
set
it
up.
You've
logged
in
you've
configured
your
chaos
hubs
now
you're
ready
to
build
your
chaos
scenario
as
part
of
the
scenario
creation.
You
pick
a
fault
and
then
you
go
ahead
and
say:
what
is
it
that
I
want
measured
during
the
fault?
Execution
fault
injection
right?
We
use,
we
use
something
called
probes.
We
built
something
called
probes
to
do
that.
A
You
can
actually
see.
There
are
two
probe
types.
There
are
many
others
you
can
find
them
in
the
documentation.
I've
just
got
a
couple
of
screenshots
here
for
illustration
purposes:
there's
something
called
as
HTTP
probe
where
you
can
go
and
query
the
status
of
your
services.
You
could
be
testing
this
service
status
for
the
application
that
you're
subjecting
to
chaos
or
you
could
be
testing
the
status
of
some
service,
that's
dependent
on
the
one
that
you're
impacting
right.
A
Called
a
Prometheus
probe,
you
could
run
some
prom
ql
queries
and
you
could
basically
check
what
is
the
deviation
in
The.
Matrix
right
is
that
as
per
expectation
or
no
Etc,
so
these
are
probes.
We
use
that
for
steady
state
hypothesis,
validation
and
once
you've
constructed
your
scenario
by
picking
the
right
faults
by
setting
the
right
probes.
Now
you
go
ahead
and
run
it
and
then
you'd
like
to
bring
more
users
onto
the
control
plane
into
your
workspace.
So
in
the
litmus
control
plane,
each
user
is
allocated
dedicated
workspace.
A
What
we
call
as
project
you
go
inside
the
project
and
you
invite
other
members
on
the
platform
into
that
project
to
collaborate
with
you
and
you
assign
a
certain
role
to
them.
Are
they
only
going
to
be
able
to
view
the
chaos
scenarios
that
you've
already
run?
Are
they
going
to
be
able
to
run
it?
Are
they
able
to
construct
new
scenarios
Etc?
So
that's
the
next
step
that
you
would
generally
do
then,
when
you've
got
that
system
working
you've
got
your
experiments
working
at
as
desired.
It
is
proven
to
be
effective.
A
Now
you
would
like
to
run
this
again
and
again
right.
You
might
want
to
run
it
as
some
kind
of
a
background
service.
You
can
schedule
a
like
a
scenario
for
recurring
execution.
There
are
some
ground
schedules
that
you
can
do
or
you
could
be
invoking
your
chaos
scenario
as
part
of
a
CD
pipeline.
You
go
ahead
and
deploy
your
application
and
you
want
to
check
the
sanity.
So
you
go
ahead
and
run
the
chaos
scenario
or
you
could
be
triggering
the
chaos
scenarios
part
of
github's
deployment.
A
If
you're
not
running
the
pipeline
way,
if
you're
using
githubs,
then
you
have
flux
or
Argo
CD
or
any
of
these
tools
go
and
upgrade
your
application
and
the
application
change
can
go
ahead
and
trigger
a
chaos
scenario
run
right,
so
that
can
be
done.
So
what
you
see
here
are
some
screenshots
that
explain
how
you
can
do
it
there's
a
recording
schedule.
A
You
have
the
written
as
API
which
you
can
use
and
you
can
invoke
the
litmus
API
to
run
this
chaos
scenarios
from
your
pipelines
or
you
could
be
setting
up
the
event
tracker
policies.
This
is
a
crd
where
you
can
see
in
the
conditions
you're,
basically
going
and
looking
for
certain
checks
on
your
deployments.
What
is
the
replica
account
that
I
have
here?
What
is
the
image
that
I
have
here?
A
Any
changes
to
this
configuration
is
actually
going
to
trigger
a
chaos
in
our
neuron
and
finally,
once
you've
got
all
these
scenarios
running
and
you're
comfortable.
Now,
you
would
actually
go
ahead
and
check.
What
is
the
resilience
Trend
right
so
you're
running
these
scenarios?
That's
great,
but
are
they
succeeding?
Are
they
not?
What
is
the
state
of
your
application's
resilience?
So
let
us
provides
something
called
as
a
resilience
score,
so
this
is
a
metric
that
essentially
Bridges
your
chaos
scenario
and
your
application
Target
right.
So
this
application
is
this
resilient
to
this
scenario.
A
The
scenario
can
actually
be
an
outage
that
you
saw
in
the
past.
So
that's
why
you've
actually
created
it
so
you're
going
and
mapping
that
against
your
application
and
you're
saying
this
is
how
resilient
I
am
and
there's
a
specific
way
to
calculate
this
resilience
score.
We'll
probably
talk
about
it
once
I
do
the
demo,
you
can
actually
go
ahead
and
add
a
lot
of
probes
to
your
faults.
A
Each
fault
is
going
to
return
a
probe
success
percentage
and
each
scenario
is
going
to
have
multiple
such
faults
right
and
each
part
is
associated
with
certain
constraints.
What
we
call
as
probes,
so
each
fault
is
associated
with
a
weightage
or
criticality,
and
that
criticality,
along
with
the
probe
success
percentage
together,
decides
your
resilient
score
on
a
scale
of
0
to
100
and
using
this
metric
you
can
compare
scenario
runs.
A
Maybe
you
run
the
scenario
on
build
X,
and
then
you
run
it
on
build
y
you're,
comparing
how
the
residents
varied,
or
you
might
have
run
the
scenario
in
your
Dev
environment
and
QA
and
then
pre-prod
and
prod.
You
might
be
looking
at
how
the
residence
Trends
have
been
changing
across
environments
or
maybe
there
is
a
change
that
you
made
within
the
experiment.
Spec
your
your
scenario,
spec.
You
have
gone
ahead
and
changed.
Retunable
you've
said
instead
of
2000
milliseconds
of
latency
I'm,
injecting
2500.
A
Now
we
want
to
see
what
happened
across
runs
because
of
the
change
in
the
amount
of
latency
right,
so
you
can
still
go
ahead
and
do
the
comparison
based
on
the
residence
course.
So
that's
about
most
of
the
features
and
I'll
just
introduce
you
to
this
concept
of
single
control,
plane
on
hybrid
infrastructure.
I
think
you
might
have
already
got
it,
but
I
just
want
to
stress
upon
it
a
little
bit
so
with
litmus.
A
You
have
one
control
plane
which
is
comprising
of
a
dashboard,
a
chaos,
Center,
a
chaos
server
and
a
mongodb
to
support
the
state,
and
you
could
be
connecting
other
kubernetes
clusters
or
namespaces
which
are
running
anywhere.
It
could
be
running
on
on-prem
cases.
You
could
be
running
on
AWS.
You
could
be
running
on
Azure
or
VMware.
What
have
you
as
long
as
you
have
the
network
connectivity?
You
will
be
able
to
connect.
B
A
And
run
chaos,
so
why
is
this
powerful?
The
one
of
the
recent
ways
of
deploying
applications
is
to
have
it
spread
across
different
Cloud
providers
for
redundancy
purposes,
so
you
would
like
to
have
one
homogeneous
control
plane
from
where
you
can
actually
Target
your
infrastructure
components
residing
in
different
platforms.
A
So
that's
what
is
enabled
by
this
model.
You
can
actually
see
this
in
this
illustration.
We
have
the
portal
here,
which
is
the
chaos
Center,
and
then
you
have
different
clusters.
You
can
see
three
clusters
here.
Each
of
them
is
connected
to
the
same
portal
and
they
could
actually
be
residing
on
different
environments.
A
I
could
have
one
cluster
which
is
actually
core
residing
with
my
control
plane,
and
then
there
could
be
others.
For
example,
you
see
something
called
virtual
machines
which
could
be
either
VMware
or
openstack,
or
it
could
be
maybe
bare
metal,
physical
machines,
and
then
you
have
your
your
Cloud
environments
and
you
could.
You
will
have
the
Agents
of
retina
sitting
there
and
that
is
going
to
carry
out
the
execution
of
your
chaos
workflow.
A
What
we've
been
talking
as
chaos
scenarios
all
this
file
and
then
it
emits
some
metrics
and
events
which
you
can
bring
back
to
your
control
plane
and
make
a
decision
on
how
your
resilience
looks
right.
So
that's
what
it
is
the
feature
somebody
is
is
there
here
this
diagram
is
sort
of
having
a
lot
of
circles.
A
You
can
see
the
chaos
Center
at
the
center
of
it
all.
You
can
use
that
for
user
management
and
teaming
you
can
connect
agents
to
it
once
you've
connected
an
agent,
you
do
some
amount
of
asset
Discovery
or
find
out
what
services
run
there.
What
microservices
run
there
the
eventual
subjects
of
your
chaos,
then
you
pull
the
faults
that
you
would
like
to
do
against
those
micro
services
that
you've
discovered,
and
then
you
add
in
the
hypothesis,
validation
using
probes.
That's
the
observability
part,
let's
say
you're
working
off
a
disconnected
environment.
A
You
might
want
to
change
the
images
you
might
might
have
pulled
the
litmus
images
and
pushed
it
to
your
own
registry.
So
you
can
go
ahead
and
use
that
make
that
setting
in
in
the
control
plane
and
say
all
the
images
that
I'm
pulling
for
the
experiment.
Execution
should
come
from
this
registry
and
then
the
chaos
waffle
actually
runs
you
could
you
could
schedule
that
once
or
you
could
schedule
that
repeatedly
and
then
this
Loop
just
the
circle,
just
continues
right
and
you
can
see
a
few
other
circles
here.
A
We
basically
say
that
the
experiments
that
are
there
support
different
runtimes,
containerdy
Docker
CR,
a
o.
You
have
the
github's
way
of
doing
things.
You
can
enable
a
switch
in
the
control
plane
and
all
the
chaos
scenarios
that
you
build
in
the
platform
can
get
committed
back
into
a
git
repository
so
that
you
can
share
it
with
your
teammates
right.
So
that's
that's
pretty
much.
The
the
summary
before
I
go
to
the
release.
Section
I'll
show
you
a
very
quick
demonstration,
foreign.
A
A
It's
called
self-agent
and
I've
run
some
scenarios
before
I'm
going
to
run
a
new
one,
and
you
can
see
the
chaos
hubs
here.
So
this
is
the
chaos
artifact
source
that
I
talked
about,
and
you
can
see
experiments
of
different
categories.
There
are
a
multitude
of
experiments
which
mostly
solve
your
resilience
needs.
You
can
write
new
ones
as
well
and
it
must
allows
for
creating
new
experiments
using
an
SDK.
So
let
me
schedule
a
chaos
scenario
and
I'm
trying
to
follow
the
sequence
that
I
showed
you
in
the
circle.
A
A
A
And
I'm
interested
in
subjecting
specific
application
to
podkin,
let's
say
I
have
a
micro,
Services
application.
You
can
see.
There
is
a
online
boutique
here,
Simple
microservices,
app
I've
selected
that
namespace
I'm
going
to
select
deployment
as
the
workload
time
I'm
going
to
select
one
of
the
services
the
cards
right.
So
this
is
the
asset
display.
A
I
was
speaking
about,
so
we
are
going
to
Target
a
deployment
which
is
identified
by
this
label
and
which
resides
in
this
particular
namespace
right
and
then
I
I
have
the
option
of
adding
probes
I'll,
probably
not
do
that
in
the
interest
of
time
and
I
can
provide
some
tunables
here.
How
long
do
I
want
the
chaos
to
run
at
what
intervals?
Do
I
need
the
particles
to
happen?
A
What
is
the
nature
of
the
kill
Etc
and
that
differs
from
experiment
to
experiment,
and
there
are
some
Advanced
configuration
that
you
can
do
to
Define
topology
and
additional
filtering
of
workloads.
So
with
this,
let
me
go
ahead
and
provide
some
weightage.
This
is
the
this
is
the
part
of
the
residence
score
calculation
I
was
talking
about,
so
let
let's
say
give
all
points,
because
I've
just
selected
one
fault
and
I
have
the
option
of
scheduling
it
repeatedly.
I'll
schedule
it
just
once.
A
This
is
a
summary
and
I
just
go
ahead
and
finish
it
and
you
can
see
the
experiment.
Execution
can
be
tracked
here.
Typically,
each
experiment
within
a
scenario:
I've
just
got
one
fault
or
one
experiment.
Within
this
scenario,
you
can
pull
multiple
ones.
Each
experiment
is
associated
with
a
couple
of
steps.
One
is
just
putting
the
template
from
The
Hub
and
installing
it
on
your
cluster,
and
the
next
step
is
the
actual
invocation
of
the
fault
and
the
hypothesis
validation,
etcetera.
The
business
logic
runs
in
the
next
step
right.
A
So,
while
we
are
at
this
I'd
like
to
show
you
the
grafana
dashboard
that
I'm
using
to
monitor
the
boutique
applications,
a
very
simple
dashboard
just
got
QPS
and
some
latency
being
measured
here
and
I've
annotated
this
dashboard
with
a
metric,
that's
coming
from
the
writteness
framework,
so
you
can
actually
see
when
the
fault
is
live.
You
will
have
some
detail
here
that
actually
tells
you
the
fault
is
active,
then
you
can
see
how
the
application
Behavior
has
changed
during
such
time
right.
A
So
let's
wait
for
that
to
happen,
and
meanwhile
you
can
also
do
all
that
I
did
on
the
chaos
Center
using
the
litmus
API.
So
you
can
use
that
to
automate
all
these
proceedings
and
you
could
invoke
that
in
a
con
in
your
CI
CD
pipelines,
or
maybe
you
have
your
own
test
execution
framework
that
you
would
like
to
plug
into
and
once
you
have
this
runs
executed.
I
said
you
can
compare
it,
you
can
actually
see
some
runs
here.
A
Maybe
I
have
got
better
resilience
across
builds
right.
So
that's
improving
here
and
this
comparison
space
made
on
the
basis
of
the
resilience
score
and
you
could
download
some
reports
on
how
your
experiment
execution
has
gone.
What
Pro,
what
are
the
probe
results?
What
are
the
verdicts?
What
is
the
residence
score?
Other
information
is
available
and
you
can
actually
go
ahead
and
add
members
to
your
team,
like
I,
said
so.
I've
got
a
couple
of
members.
A
Apart
from
the
admin
here,
I've
got
Uma,
who
is
editor
and
this
Karthik
host
viewer
on
this
platform
right,
so
you
can
create
new
users
and,
and
once
they
are
available
on
the
platform,
you
can
pull
them
into
the
team.
So
that's
a
quick
demonstration
in
this
particular
deletion
is
happening.
The
particle
is
happening.
You
can
see.
What's
happened
here
as
far
as
the
front
end
and
the
cards
go,
the
QPS
nose,
dived,
the
error
count
increased.
A
So
this
is
reflecting
an
inefficient
deployment
or
something
that's
gone
wrong
with
my
application
right
in
this
case,
I
just
have
a
single
replica
of
cards,
which
is
why
the
failure
is
seen
But.
The
same
could
be
caused
due
to
an
application
failure,
an
application
bug
a
software
bug
in
your
core
as
well.
So
this
was
a
quick
demonstration
of
what
I
was
talking
about.
A
I
have
not
added
any
probes,
so
you
can
see
without
any
additional
constraints.
The
experiment
succeeds,
but
there
are
several
runs
that
have
happened
prior,
for
example,
this
one
where
you
can
see
the
failure
has
happened
because
of
a
probe
failure.
The
experiment
verdict
has
failed
because
one
of
the
probe
has
failed.
You
can
see
that
the
chaos
result
here,
so
that
was
a
very
quick
demonstration
of
what
you
can
do
with
litmus
chaos.
Let
me
go
back
to
the
presentation
and
talk
about
what
we
got
done
between
previous
Valencia
you've
gone
to
now.
A
A
So
across
the
last
few
months,
we've
had
about
three
to
four
about
four
releases,
and
this
is
what
we
managed
to
do
in
that
time.
We've
added
new
fault
types:
there's
a
HTTP
chaos,
experiment,
Suite
that
we've
added,
which
allows
you
to
simulate
erroneous
status,
codes,
corrupted
response
body
Etc
to
see
how
your
application
behaves.
You
can
inject
latency,
you
could.
We
also
have
support
now
added
for
newer
versions
of
kubernetes
and
openshift.
We've
upgraded
our
operators
Etc
to
be
able
to
do
that.
A
The
support
for
many
chaos,
experiments
of
the
network
category,
the
HTTP
experiments,
the
DNS
ones,
latency
packet
drops
Etc,
are
Now
supported
on
environments
which
also
have
service
mesh
running
there,
and
we've
got
support
for
randomization
of
fault
inputs.
What
we
mean
by
this
is
let's
say:
I
provide
a
range
of
latency
0
to
2000.
You
could
actually
be
injecting
faults
with
latency
anywhere
between
0
to
2000
right
in.
A
Iterations
of
the
experiment,
you'll
have
different
value
of
fault
getting
injected
and
you
can
also
do
the
same
for
intervals.
You
can
do
the
same
for
any
other
tunable
that
is
native
to
a
fault.
You
can
randomize
the
input.
We've
got
the
ability
to
do
all
the
operations
that
I
just
showed
you
most
of
the
operations.
At
least
that
I
showed
you
on
KR
Center,
using
the
CLI
tool
called
it
the
CTL
you
can
list
workflows,
you
can
trigger
new
workflow
runs.
A
You
can
delete
them,
basically
do
some
thread
operations
using
the
CTL
tool,
we've
added
support
for
the
runtimes
container,
D
and
crio.
We
started
off
with
Docker,
but
obviously
that's
replaced
with.
Mostly
continuity
is
the
de
facto
and
openshift
use
crio.
So
we
have
first
class
support
for
these
runtimes
and
we've
also
introduced
improvements
to
our
SDK,
which
basically
helps
developers
to
bootstrap
their
chaos.
Experiments
and
we've
also
got
some
newer
category
of
experiments
here.
As
you
can
see,
we've
got
applications
that
we've
got
chaos
that
does
faults
on
Spring
boot
applications.
A
We've
improved
the
scope
of
our
of
our
probes.
So
some
of
what
you
saw
in
the
spec,
we
had
the
Prometheus
probe
and
the
HTTP
Pro,
but
there's
a
command
probe
which
we've
gone
ahead
and
made
it
very
flexible,
so
command
probes
help
you
to
run
any
validation
of
your
choice.
It
need
not
be
in
HTTP
request.
It
need
not
be
a
promql
query,
but
it
could
be
doing
anything
you
could
be
having
your
own
CLI.
Anything
could
be
done
and
then
we've
made
our
apis
better,
more
user
friendly.
A
B
I
think
I
got
the
mic.
Thank
you,
Karthik.
There
was
a
great
run
of
what
we
got
done
in
the
last
six
months,
but
just
to
go
back,
we've
been
adding
newer
features
for
almost
three
years
now,
right
so
with
a
release
coming
out
every
month
on
the
15th
right,
I
think
we
have
done
more
than
30
releases
without
missing
the
15th
date
right.
So
what
that
did
really
is
it's
kind
of
feature
complete
to
do
the
basic
chaos
experimentation
right?
B
So
litmus
1.0
was
more
about
getting
the
chaos
operator.
All
the
crds
done,
making
sure
that
you
do
the
chaos
experimentation
in
a
cloud
native
way
right,
so
chaos.
Let
me
scale:
2.0
was
getting
the
multi-cluster
support,
introducing
the
chaos
Center,
where
multiple
clusters
and
namespaces
can
connect
to
the
chaos
interest
so
and
introduce
the
team
mean
right,
so
that
enabled
Enterprises
to
think
of
chaos,
engineering
in
a
practical
way
right,
so
more
teams
can
come
and
practice
chaos,
engineering.
B
B
What
do
you
really
want
to
focus
on
making
litmus
more
robust
right
and
because
you
know
whatever
the
bug
fixes
that
the
community
reports
that
we
find
through
our
automated
test
will
focus
on
making
sure
that,
even
if
it
means
that
you
know
re-architecting
few
things
here
and
there
will
focus
primarily
on
making
it
rigid,
and
we
also
want
to
make
litmus
leaner.
So
what
that
means
is
today
to
run
chaos
experiment
it
runs
under
our
go
workflow,
so
many
people
want
to
run
litmus
experiments
in
pipelines
right.
So
maybe
it
is.
B
If
you're
running
just
one
chaos
experiment,
it
may
be
having
little
bit
more
footprint
than
what
you
need
right,
because
you
have
our
go
workflow
controller.
It
can
do
a
lot
of
things,
but
just
for
one
experiment:
it
may
be
a
little
bit
more
than
what
you
want
right.
So
we
are
going
to
start
work
on
providing
a
native
orchestration
engine
where
you
can
run
the
exact
chaos
experiment
without
the
Argo
workflow.
So
that's
that's
going
to
make
a
litmus
leaner.
Probably
you
know
on
low
footprint,
kubernetes
on
edge
and
other
places.
B
You
can
start
running
litmus
in
a
more
easier
way,
and
the
next
Focus
for
us
also
is
how
to
take
chaos.
Engineering
for
Developers,
so
one
thing
that
we
have
observed
is
we
started
originally
for
sres
running
chaos,
experiment
in
pre-prod
right,
you
know
just
make
it
totally
declarative
three
MLS
and
also
make
sure
that
you
know
it's
got
all
the
API
support.
Chaos,
controllers,
operators,
everything
and
then
we
realized
that
chaos
has
found
its
need
in
the
pipelines
right.
B
So
more
and
more
Community
users
are
you
coming
and
using
it
in
their
CD
pipelines
right.
So
that's
when
this
chaos,
the
workflow
engine,
getting
started
more
and
more
adoption
now
I
think
you
know
it's
time
to
do
even
more
left
shifting
where
developers
can
come
and
think
of
introducing
chaos,
experiments
even
before
the
code
gets
merged
right.
B
Think
of
the
scenarios,
as
you
know,
I'm
writing
a
lot
of
cloud
native
code
and
it's
before
it
goes
to
you
know,
build
I
mean
so
it
goes
to
a
pipeline
and
then
finally
gets
pushed.
Is
it
possible
for
me
to
run
a
experiment
on
the
build
possible
build
code
right?
What
happens
if
you
delay
it
is,
you
know
it
goes
to
a
pipeline
and
then
gets
deployed,
and
then
rollback
has
to
be
done.
So
there's
a
little
bit
of
a
cost
involved
in
that.
Can
the
developers
really
avoid
that
right?
B
B
So
if
you
go
to
the
litmus
repo
there's,
a
discussion
thread
created
and
what's
the
focus
of
the
roadmap
for
the
next
one
year
and
we're
going
to
work
through
this
monthly
maintainer
or
contributor
sessions
a
little
bit
openly
right,
whoever
is
interested,
they
can
actually
come,
not
only
contribute
the
code,
but
you
can
contribute
your
requirements
and
ideas.
Hey.
You
know.
This
is
what
I'm
thinking
of
for
My
chaos
entering
use
case.
B
So
you
know
come
on
openly
and
then
discuss
and
we'll
say
you
know
how
to
get
that
done
right.
So
that's
that's
one
announcement
that
we
as
maintenance
want
to
do
that.
You
know
we
are
opening
up
the
board
for
the
next
phase
of
treatments,
which
is
3.0
right
so
and
finally,
I
want
to
conclude
this
session
by
showing
how
to
get
in
touch
with
the
community
right.
B
So
we've
been
fortunate
to
get
contributions
from
various
different
people
right
over
a
period
of
last
three
to
four
years,
so
we
have
200
plus
quantity,
different
contributors
from
various
companies
right.
So
we
would
like
that
to
increase
more
so
we
always
encourage
contributing
new
ideas,
new
experiment
types,
not
necessarily
that
you
have
to
develop.
But
you
know
hey
here
is
my
use
case,
and
then
you
know
somebody
will
pick
it
up
right.
If
you
can,
then
we
certainly
would
like
to
add
more
as
a
core
team
right.
B
So
the
other
contribution
that
we
think
can
be
accelerated
is
the
list
of
experiment
types
into
the
chaos
Hub.
So
chaos
Hub
has
grown
from
30
plus
so
experiments
last
year
to
about
45.
Now
or
40
plus
experiments,
so
now
that
we
have
a
stable
litmus
platform,
it's
time
to.
You
know
Focus
contributions
on
the
experiments
right,
so
the
community
has
actually
created
a
lot
of
new
experiments
because
it's
got
a
good
SDK,
but
we
have
not
concentrated
in
requesting.
You
know
here
is
how
you
can
Upstream
it.
B
It
can
receive
more
contributions
and
litmus
agents
can
be
made.
You
know
a
linear,
Helm
charts.
There
are
already
some
contributions
there,
so
there
are
many
many
areas
which
the
project
can.
You
know,
take
contributions
and
to
do
all
this,
you
know
we're
always
hanging
out
at
the
litmus
channel
on
kubernetes
lag.
So
please
do
join
us
and
we
also
run
a
monthly
Community
meetups
virtual
meetups,
where
you
know,
users
come
and
talk
about
their
use
cases
and
also
ask
questions
and
also
maintenance,
announce
any
new
features
that
were
introduced,
etcetera,
Etc.
B
So
because
of
this
beta
that
we
are
announcing,
we
are
also
run
one
more
meet
up
every
month,
that
is
for
contributors
and
maintenance
right.
It's
a
maintain
myth.
So
you
know,
if
you
are
looking
at
developing
yourself
into
a
more
active
cloud
native,
you
know,
contributor
is
an
opportunity
to
directly
work
with
the
maintainers
right.
So
that's
all
I
have,
and
you
know
we
obviously
want
to
know.
If
you
like
the
project,
we
would
want
you
to
like
the
projects
or
give
us
a
start.
That'll
always
help
you.