►
From YouTube: Power Level 9000! Improving Application Performance with Chaos Engineering - S. Pathak & K. Gaekwad
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Power Level 9000! Improving Application Performance with Chaos Engineering - Saiyam Pathak, Civo & Karthik Gaekwad, Verica
Chaos engineering is becoming a standard to test the resiliency and performance of cloud native applications. It allows you to validate assumptions, catch loopholes, and generally improve resiliency in your cluster. This talk will focus on the specific experiments that will improve cluster and application performance and cover tools in the ecosystem, including Litmuschaos, Kraken, and Chaos Mesh. At the end of the talk, the audience will understand the basics of experiments, apply them in their orgs, and code to run in their infrastructures. Breakdown: The idea of an experiment Application Performance Ideas Identifying Cluster Latency Issues (k8s, k3s) Improving App SLO Fixing issues with app latency Detecting latency in Service Meshes Rightsizing cluster auto-scaling issues Ways we use Chaos Experiments in our companies Demo: Running a latency Chaos Experiment continuously in a CI pipeline on k3s Chaos Tooling review FAQs about Chaos Engineering Conclusions
B
B
A
And
I'm
karthik
wadd,
I'm
the
head
of
cloud
native
engineering
over
here
at
verica
and
lately
in
the
last
probably
eight
months
or
so.
I've
been
doing
a
lot
of
work
in
the
chaos,
engineering,
space
building,
chaos,
engineering
verifications,
I've
written
the
learning,
kubernetes
and
other
cloud
native
courses
on
linkedin
learning.
A
A
So
let's
talk
about
the
storyline
today,
so
we'll
do
a
quick
overview
of
the
state
of
the
cloud
native
universe
figure
out
where
chaos
engineering
fits
in
and
then
kind
of
talk
about
what
we
mean
by
kiosk
engineering,
its
intersection
with
cloud
native
and
then
the
most
important
part,
the
demos,
of
course,
and
finally,
we'll
go
into
some
speaker
recommendations
at
the
end,
so
state
of
the
universe.
So
I'm
a
big
fan
and
I'm
kind
of
a
nerd
when
it
comes
to
looking
at
reports
and
things
like
that.
A
So
this
the
state
of
our
world
is
we're
kind
of
going
to
a
multi-cloud
world.
We've
known
this
for
a
while,
but
you
know
in
the
research
report
from
451
research.
This
is
kind
it's
a
little
bit
older
came
out
in
2018
2019
that
time
frame,
but
it
said
67
percent
of
enterprises
will
have
multi-cloud
or
hybrid
it
environments
by
2019.
A
But
you
know,
with
greater
choice,
brings
more
complexity.
We
saw
this
again
in
the
cncf
report
that
came
out
last
year,
where
26
of
usage
is
actually
multi-cloud.
So
that's
grown
a
lot
from
years
past,
which,
if
you
notice
in
the
graph
you
know
there
was
nothing
in
20
2019
but
2020.
That's
that's
a
brand
new
thing
there's
also
increased
container
usage.
So
this
is
no
surprise
for
to
anybody.
A
The
use
of
containers
and
production
is
that
has
become
the
norm
today
for
the
server
survey,
and
so
you
know,
there's
steady
increase
in
use
of
containers
in
production
today,
as
well
as
you
know,
everybody
and
their
parents
and
sisters
and
brothers
and
co-workers
are
using
kubernetes.
It's
kind
of
become
a
ubiquitous
thing,
so
you
know
that's
kind
of
where
we
stand
from
a
cloud
native
standpoint,
but
also
in
the
2020
cncf
survey.
A
They
talked
about
cultural
challenges
where
cultural
challenges
and
complexity
were
the
biggest
challenges
with
respect
to
using
containers.
Actually,
let's
talk
about
complexity
for
just
one.
Second,
so
on
the
complex
complexity
scale
you
have
the
simple
systems
and
then
you
have
the
complex
systems,
so
simple
systems
are
defined.
You
know
where,
where
the
process
flow
is
very
linear,
you
have
predictable
output
and
everything
is
comprehensible
versus
more
complex
systems.
Are
you
know
everything
doesn't
really
happen
in
a
linear
fashion,
so
things
are
non-linear.
A
You
have
a
lot
of
unpredictable
behavior,
so
you
don't
really
know
what
might
happen
and
it's
also
impossible
to
build
like
one
complete
mental
model.
One
of
the
really
interesting
things
that
I
found
from
this
was
you
know
in
the
past.
You
know
10
15
years
ago,
you'd
have
one
architect
that
was
able
to
understand
the
whole
system.
So
when
something
went
wrong,
you
know
you
could
go
to
that
one
person
and
ask
him
him
him
or
her
hey.
A
What's
going
on
with
the
system-
and
you
know
you
might
get
a
response
of
like
oh
there's-
maybe
the
messaging
service
or
something
like
that
might
be
offline,
but
in
current
world
today,
when
everything's
a
little
bit
more
distributed,
it's
hard
for
one
single
person
to
have
like
a
complete
mental
model
of
the
entire
system.
So
it's
become
a
lot
more
complex.
A
You
know,
and
and
testing
is
hard,
so
this
wouldn't
be
a
conference
I'm
guessing.
This
slide
is
going
to
be
in
every
presentation,
but
you
know
you
have
your
productionized
distributed
application
on
the
right,
large
container
ship
and
then
you're
trying
to
test
this,
and
you
know
we're
all
kind
of
like
using
probably
using
jmeter
to
like
load
test
our
applications.
So
you
have
the
little
like
forklift
on
the
on
the
left
hand
side
of
this
image.
So
you
know,
testing
is
just
a
hard
problem
in
a
distributed
world
today,
so
enter
chaos
engineering.
A
A
So
let's
talk
about
testing
versus
experimentation.
Real
quick
testing
is
the
idea
of
making
an
assertion
on
a
property
of
a
system
based
on
existing
knowledge
and
then
validating
that
specific
property.
Experimentation,
on
the
other
hand,
is
proposing
a
hypothesis
which
can
either
be
proven
or
disproven.
A
A
Okay,
so
you
were
trying
to
practice
chaos,
engineering
in
the
real
world.
So
what?
How
do
we
actually
make
this
happen
so
for
the
principles
of
chaos,
you're,
first
trying
to
define
a
steady
state?
So
you
know
what
normal
behavior
might
look
like
in
normal
conditions
and
then
you
take
your
hypothesis
and
you
hypothesize
on
the
steady
state
which
will
continue
for
both
your
control
group
and
your
experimental
group.
A
Now
that
you
have
two
groups,
you
start
introducing
variables
to
reflect
real-world
events
in
the
system.
What
is
this?
What
does
this
mean?
So
you
know
you
might
like
in
the
kubernetes
space
you
might
start
to
delete
pods
or
you
know,
take
a
note
away,
etc.
On
the
on
the
control
and
your
experimental
group,
and
you
start
to
inject
real
world
chaos
to
it,
and
then
you
try
and
disprove
the
hypothesis
by
either
looking
at
the
difference
in
the
steady
state
between
the
control
and
the
experimental
group.
A
A
So
you
want
to
look
at
if
you
have
an
application
running,
you
want
to
look
at
things
from
the
outside
like
traffic
coming
in,
you
know,
response
rates,
and
things
like
that
versus
you
know
looking
at
something
inside
of
your
application,
maybe
like
cpu
or
memory,
for
example,
you
really
want
to
vary
real
world
events,
so
you
want
to
you
know,
turn
things
off
slow
things
down,
send
invalid
responses.
Things
like
that
so
once
again
from
an
outside
in
perspective,
but
you
want
to
change
things
that
might
happen
in
the
real
world.
A
So
I
just
said
that
you
are
running
experiments
in
production,
so
you
want
to
remember
that
you're
doing
these
things
in
production,
so
you
want
to
minimize
your
blast
radius.
So,
in
order
to
not
take
everything
down
and
last
but
not
least,
continues
we'll
talk
about
this
as
we
go
along,
but
really
what
I
mean
by
this
is
you
want
to
keep
doing
this
from
a
continuous
kind
of
perspective?
A
A
Some
of
them
are,
you
know,
kraken
litmus
and
chaos
mesh
and
there's
a
whole
bunch
of
others
as
well,
including
enterprise,
tooling.
So,
for
the
purposes
of
this
presentation,
we'll
focus
on
the
first
three,
because
realistically
we
only
have
25
minutes.
Probably
you
know
22
by
the
time
I'm
like
done
talking
about
this
stuff,
so
we'll
we'll
cover
the
first
three,
but
definitely
take
a
look
at
all
the
others
as
well,
so
first
up
kraken.
So
what
is
kraken
kraken
helps
us
inject
failure
into
either
openshift
or
kubernetes.
A
It's
kind
of
built
from
two
components.
One
of
them
is
powerful
seal
that
injects
failures
onto
kubernetes
clusters,
so
powerful,
powerful
seal
is
the
actual
thing
that
you
know
injects
failures
or
does
something
to
your
actual
cluster
and
then
there's
cerebrus.
That
watches
and
reports
on
cluster
components.
A
So
cerebrus
is
actually
the
watcher
component,
so
when
powerful
seal
goes
and
maybe
kills
a
pod
or
you
know,
slows
down
a
road,
for
example,
cerebrus
actually
sits
there
and
watches
for
changes
in
your
in
your
infrastructure.
So
it'll
tell
you
what's
actually
going
on
so
kraken
is
actually
used
for
a
bunch
of
scenarios.
You
can
use
it
for
part
chaos.
A
Here's,
a
quick
architecture
diagram,
so
kraken
you
have.
We
have
it
on
the
left
hand
side
here,
this
injects
failures
and
openshift.
These
are
the
different
components
that
it
can
kind
of
work
with.
It
also
works
with.
You
know
different
cloud
apis.
So
if
you
want
to
use
the
aws
azure,
for
example,
you
can,
you
know
call
out
to
those
as
well.
On
the
left
hand,
side
we
have
cerebrus
that
actually
watches
all
of
these
different
components
as
well,
but
kraken
itself.
B
Thank
you
karthik
for
the
awesome
introduction
to
chaos,
engineering
and
why
it
is
needed
next
up
on
the
tooling.
Is
chaos
mich,
so
chaos
mesh
is
a
powerful
chaos.
Engineering
platform
for
kubernetes,
it
is
extremely
simple
to
use,
can
be
installed
on
any
humanities
cluster
and
it
is
designed
for
communities.
It
has
a
interactive
dashboards
which
you
can
use
for,
creating
the
chaos,
experiments
and
also
can
be
used
for
analytics
as
well
like
which
all
points
your
chaos
actually
ran.
B
Some
of
the
disruptions
that
chaos
mesh
causes
are
killing
the
pod
latency
network
system,
io
chaos
so,
and
there
are
others
as
well.
So
all
these
chaos
can
be
introduced
in
your
within
your
communities.
Cluster
chaos,
mesh
architecture,
so
these
are
the
set
of
experiments
that
are
there
with
which
you
can
create
as
a
cr
and
submit
it
to
the
kubernetes
cluster.
So
you
will
be
writing
a
yaml
file
which
is
a
custom
resource,
and
then
you
will
be
submitting
to
the
kubernetes
cluster.
B
So
after
the
control
manager
takes
up
the
cr,
it
will
then
hand
it
over
to
the
daemon
and
the
demon
is
actually
the
chaos.
Demon
is
actually
running
on
all
the
nodes,
which
will
internally
run
your
chaos,
experiments,
and
these
experiments
then
are
based
on
certain
selectors
that
selects
the
particular
workload
to
which
this
particular
chaos
experiment
has
to
be
run.
So
that's
how,
in
a
short
summary
chaos,
mesh
works.
B
Next,
up
on
the
tooling
is
litmus
chaos,
so
litmus
is
again
an
open
source
tool,
it's
also
a
cncf
project
and
it
is
basically
chaos
workflows
for
communities.
So
litmus
features
are
team
collaboration,
so
you
can
integrate
it
with
your
github
and
you
can
add
your
team
members.
You
can
add
public
private
chaos,
hubs
which
will
contain
the
chaos
experiments,
so
you
can
enable
chaos
detox
as
well,
which
can
act
as
a
single
source
of
truth.
B
So
whatever
workflow
you
create
in
the
litmus
will
be
stored
in
your
github
repository
whichever
is
connected,
and
then
you
can
have
the
chaos
observability
as
well.
You
can
view
the
duration
of
the
experiment
when
it
ran
on
in
the
grafana
dashboard,
and
you
can
also
have
the
provitues
as
well
chaos,
apis
that
can
be
used
and
chaos
workflows
for
creating
the
complete
workflow
of
the
chaos
experiment
that
has
to
be
run
on
the
cluster
litmus
architecture.
So
this
is
litmus
2.0
architecture.
B
B
B
So
this
is
a
complete
runtime
workflow
for
litmus.
So
you
can
see
here
the
crds,
so
there
are
three
crds
the
chaos
engine.
So
we
write
the
chaos
engine
file,
the
chaos,
experiments,
the
chaos,
experiments
are
installed
onto
the
cluster
and
then
the
chaos
results.
Okay
also
result
will
tell
the
the
success
failures
and
what
are
the
metrics
for
that.
So
you
have
the
kiosk
operator
installed
onto
the
cluster.
B
You
write
the
engine,
dot,
yaml,
kiosk,
engine.yaml
file
and
so
user
creates
a
chaos,
engine
cr
and
maps
it
to
an
application
to
which
the
chaos
experiment
has
to
be
run.
So
then,
the
chaos
operator
would
read
that
file
and
create
a
chaos
runner
pod,
which
is
actually
reading
all
the
configuration
mapping
it
all
to
the
workload
which
it
has
to
run
the
chaos
experiment
on
and
after
that
it
triggers
a
chaos
job
now
job
is
actual
experiment
that
runs
and
performs
the
chaos
checks
and
do
the
experiment
and
then
hands
over
the
result.
B
Chaos
hub
is
the
place
where
you
have
all
your
experiments
stored,
so
you
can
use
the
public
kiosk
or
you
can
create
your
own
chaos
hub
privately
and
link
that
to
the
litmus
portal.
You
can
also
attach
multiple
target
clusters
to
litmus.
So
you
it's
not
only
limited
to
single
cluster,
you
can
have
multiple
clusters
and
select
targets
for
different
workflows.
So
this
is
how
all
in
all
litmus
works.
B
B
B
B
B
B
B
B
B
B
B
B
Will
be
slash,
run,
k3s,
container
d,
container,
dot
sock.
So,
let's
add
this
experiment
finishing
heading
and
construct
the
workflow.
Now
this
is
the
yaml
file
that
you
can
edit
as
well,
so
we
edit
the
monitoring
section.
So
we
want
to
enable
the
monitoring
so
that
it
can
flow
and
tell
us
what
the
exact
chaos
duration
was
when
the
experiment
ran.
B
B
B
B
B
So
our
workflow
has
run
successfully
and
all
the
network
loss
pods
have
done
their
jobs
and
we
can
see
the
workflow
succeeded
now
how
to
visualize
this.
Now,
let's
go
to
our
grafana
dashboard,
so
I
have
already
imported
grafana
and
connected
into
prometheus
and
loaded
the
json.
So
this
is
how
it
is
showing
the
probe
duration
seconds
and
you
can
see
the
graph
which
is
having
the
network
loss.
B
So
the
red
area
that
we
are
seeing
is
coming
because
we
said
the
monitoring
true
in
the
yaml
file,
so
monitoring
true
in
the
yaml
file,
will
give
the
matrix
the
chaos
proper
chaos
duration,
where
actually
the
chaos
ran
for
the
particular
workload.
So
this
is
that
particular
duration
for
which
the
chaos
ran,
and
it
helps
to
visualize
when
you
have
a
lot
of
chaos
running
and
your
cluster,
so
that's
how
you
can
run
chaos,
workflow
and
visualize
it.
B
B
B
B
So
we
can
see
in
the
kubecon
repository
we
have
the
litmus
folder
created.
So
when
we
did
the
workflow
and
we
did
the
github
solid
much
so
it
gave
the
workflow
it
created
the
workflow
inside
the
repository
it
pushed
it
and
we
have
the
workflow
id
and
the
customer
id.
So
we
need
to
have
these
two
so
that
we
can
subscribe
the
deployment
for
this
particular
workflow,
so
we'll
copy
the
workflow
id.
B
So
the
bootstrapping
of
flux
is
finished
and
we
can
see
everything
is
there.
So
the
flux
has
been
bootstrapped
cube,
cdl
get
parts
hyphen
and
flux
system,
and
we
can
see
all
the
pods
are
running.
So
what
we
have
done
is
we
have
bootstrap
flux
with
some
of
the
components
on
the
repository
cube,
con
branch
main
and
the
path
deploy,
so
the
path
deploy
will
be
taken
and
anything
that
gets
deployed
or
edited
for
the
deploy
folder.
B
It
will
apply
the
the
new
version
of
the
application
on
the
cluster
and
in
our
application
we
have
added
few
annotations.
So
in
the
annotations
we
have
bitops
as
true
and
the
workflow
id
we
just
post
to
github.
So
if
I
do
a
get
pull,
it
will
give
me
all
the
latest
details
so
for
the
deployment
we
have
added,
some
annotations,
which
is
the
git
ops,
enabled
and
subscribe
to
this
particular
workflow.
So
anytime,
the
new
version
of
this
application
deploys.
It
will
trigger
this
workflow,
which
is
our
network
loss,
workflow.
B
B
So
that's
how
you
can
use
litmus
chaos
using
your
github
actions
and
continuous
deployment,
continuous
integration.
So
all
these
things
will
be
helpful
when
you
are
running
for
a
production
environment,
so
you
will
be
having
on
every
commit
or
on
a
specific
commit
on
specific
action.
You
can
run
various
different
workflows
and
your
github
repository
acts
as
a
single
source
of
truth,
which
has
all
the
workflows
that
you
created.
B
A
That
was
awesome,
it
did
look
like.
We
went
through
a
lot
of
things
in
that
demo,
so
let
me
see
if
I
can,
you
know,
break
it
down
and
see.
If
I
can
get
this
right,
so
we
had
an
application
that
we
deployed
and
then
we
we
ran
litmus
2.0
on
the
cluster
and
we
decided
to
run
the
pod
network
network
gloss
experiment.
A
A
A
For
your
application,
but
then
we're
also
using
flux
as
well
to
you
know,
do
that
in
a
continuous
kind
of
format.
That's
great
all
right!
Well,
let's,
let's
wrap
up,
you
know
we,
we
talked
through
a
bunch
of
things
in
this
specific
talk,
so
let's
give
our
recommendations
for
just
chaos,
engineering
in
general.
A
A
A
This
is
like
the
best
time
to
talk
to
the
maintainers
and
the
practitioners
in
the
in
the
space
that
are,
you
know,
have
the
ideas-
and
you
know,
are
actually
doing
a
lot
of
these
things
in
the
in
the
space.
So
come
come
talk
to
us
talk
to
the
folks.
The
litmus
folks
are
all
here
at
kucon.
You
know
it
would
be
great
to
kind
of
understand.
You
know
their
thinking
behind
the
scenes.
A
Probably
the
my
most
important
recommendation
right
now
would
be
to
communicate
your
your
experiments,
because
a
lot
of
this
is
you
know,
it's
recommended
to
run
these
tests
in
production,
but
it's
also
important
from
an
organizational
perspective
from
a
team
perspective
to
be
able
to
communicate
that
hey,
we
are
running.
You
know,
chaos,
engineering
tests
in
your
cluster
on
on
production,
for
example,
so
you
don't
get
into
a
weird
corporate
or
weird
organizational
kind
of
trouble
or
whatever.
A
So
that's
that's
my
big
recommendation,
there's
a
bunch
of
like
community
stuff
and
community
groups
and
meetups
that
you
can
join
put
a
couple
of
links
in
here
and
also
I
talked
about
theory.
If
you
want
to
get
the
kiosk
engineering
group
book
from
casey
rosenthal,
you
can
get
that
at
verica,
dot,
io,
slash
books
or
slash
book.
B
So,
just
to
add
on
to
kartik's
recommendation,
I
have
some
of
the
things
like
chaos.
Engineering
is
expected
to
grow
even
more,
so
it
is
new,
but
it
is
growing
rapidly
with
the
number
of
tools
with
everything
that
is
growing
in
the
cloud
native
ecosystem,
and
we
should
work
towards
continuous
chaos.
B
There
are
a
lot
of
new
tools
coming
up
for
chaos,
engineering
and
the
current
tools
are
maturing
enough
to
match
the
needs
of
the
production
grade
clusters
check
out
for
the
integrations
that
will
help
you
in
your
developer,
workflows
and
ci.
Cd
workflows,
like
captain,
is
a
very
good
project
and
captain
plus
litmus
works
very
well
octato
plus
litmus
works
very
well.
So
you
can
try
these
out.
They
have
a
lot
of
blogs.
A
lot
of
meetups
talks
that
they
have
done
and
others
as
well.
A
B
Yeah,
definitely
the
try
out
the
repository,
it's
public
and
you
can
try
out
the
demo
and
you
can
try
out
on
any
kubernetes
cluster
as
well.
So
try
it
out.
Let
us
know
what
you
feel
about
chaos,
engineering,
any
questions.
We
are
happy
to
help
and
we
are
there
in
the
slack
channels,
kubernetes
and
cf.
You
can
reach
us
out
anytime,
all
right.