►
Description
Chaos Engineering is the new entrant into the delivery pipeline of cloud-native applications, and a proven means of ascertaining SLO compliance and evaluating the resilience of applications. This assumes even more importance given the en-masse migration that organizations are making to Kubernetes as the development & deployment platform of choice. LitmusChaos is a cloud-native chaos engineering framework that provides a rich suite of chaos experiments and the ability to orchestrate chaos as defined by the principles of chaos. In this webinar, we will view how Keptn's litmus service triggers chaos on applications under load as part of its powerful automated CD pipeline, thereby evaluating application resilience and strengthening the quality gate evaluation process.
Slides: https://docs.google.com/presentation/d/1ZFEwXqFIkpicM5-aRkLinWh8AWjombkg1d0Y-Z1DC68/edit#slide=id.p1
Learn more: https://keptn.sh
Get started with tutorials: https://tutorials.keptn.sh
Join us in Slack: https://slack.keptn.sh
Star us on Github: https://github.com/keptn/keptn
Follow us on Twitter: https://twitter.com/keptnProject
A
Hi
and
welcome
everyone
to
this
captain
webinar
on
evaluating
the
resiliency
of
your
microservices,
with
litmus
chaos,
tests
and
captain.
I
have
a
very
special
guest
today:
hi
kartik,
hello,
everyone
kartik
is
a
maintainer
and
architect
of
litmus
chaos
and
he
will
join
us
for
this
webinar
and
give
us
more
background
information
in
all
the
details
on
litmus.
B
A
And
we
have
also
prepared
a
demo
that
we
will
show
you
how
you
can
actually
use
litmus
chaos
and
how
you
can
use
the
integration
that
we've
built
together.
We
have
a
chat,
feature
and
a
q,
a
feature
in
this
zoom
webinar.
So
please
make
use
of
these
features.
If
you
want
to
ask
any
questions,
we
will
be
actively
taking
a
look
at
the
chat
and
the
q
a
feature.
A
So
we
will
try
to
ask
these
questions
I'll,
try
to
add
to
forward
the
questions
to
kartik,
or
we
will
just
answer
these
questions
during
the
webinar
or
at
latest
at
the
end
of
the
webinar,
where
we
also
reserve
some
time
for
the
q
a,
but
let's
just
jump
into
this
webinar
and
explain
a
little
bit
what
we
actually
want
to
do.
So
we
were
talking
a
little
bit
about
how
to
actually
evaluate
the
resistance,
the
resiliency
of
microservices
and
how
you
can
do
this
with
litmus
chaos.
A
We
will
talk
about
also
how
and
why
functional
tests
are
not
enough,
because
outages
might
happen
or
will
happen
in
your
production,
environments
and
chaos.
Engineering
is
a
way
to
already,
let's
say,
simulate
these
outages
or
actually
trigger
these
outages
and
then
react
on
this
and
with
chaos,
engineering
and
captain.
We
can
actually
evaluate
them.
A
The
resiliency
of
your
microservices
or
with
the
slo-based
evaluations
of
the
captain
quality
gates
that
you
might
already
know,
if
you're
not
so
much
familiar
with
either
litmus
chaos
or
captain,
don't
worry.
We
will
cover
everything
in
this
webinar
and
at
the
end
or
the
second
part
of
the
webinar.
We
also
have
a
demo
and
we
will
come
back
after
the
demo
for
a
q,
a
session
so
stay
tuned.
A
B
It's
also
a
corporate
development
platform
today,
so
it's
both
it's
both
the
development
platform,
as
well
as
the
eventual
deployment
platform
on
which
your
application
is
going
to
run.
So
during
the
course
of
time,
during
this
migration,
you
might
have
made
a
lot
of
changes
to
your
application.
You
might
have.
B
You
have
made
it
closely
coupled
now,
you're
ready
to
get
on
to
kubernetes,
but
there's
a
lot
of
apprehension,
because
kubernetes
is
a
really
complex
and
dense
system,
and
whenever
you
are
running
your
applications
on
top
of
kubernetes,
it
essentially
means
that
you
have
a
lot
of
layers
playing
at
one
point
of
time.
You
have.
B
The
infrastructure
upon
which
you
created
your
cluster,
it
could
be
on
premise
or
it
could
be
on
the
cloud.
Then
you
have
the
actual
communities
components,
the
various
services,
the
control
plane
of
communities
along
with
ncd
and
all
the
services,
and
most
often
you're
not
going
to
deploy
your
application
in
isolation.
It's
going
to
be
working
along
with
a
lot
of
other
tooling
that
you
might
have
sourced
from
the
cnc
of
the
landscape.
B
Prometheus
grafana,
you
might
be
using
surface
discovery.
Mechanisms
might
be
using
stateful
storage
providers,
who
might
be
using
container
storage
solutions
like
open
abs,
and
you
have
so
many
of
these
and
then
to
run
your
application.
You
have
dependent
framework
components
like
mongodb,
or
you
might
be
running
message
queues
like
kafka.
You
might
be
running
postgres.
B
B
B
There
can
be
failures
at
any
of
these
layers
and
that's
why
we
are
saying
in
this
slide
that
more
than
90
percent,
figuratively
speaking,
the
resiliency
of
your
application,
depends
upon
other
components,
most
of
them
cloud
native
and
your
a
physical
infrastructure,
on
top
of
which
you
are
learning
that
and
outrageous
cost
a
lot
of
money.
We've
heard
of
famous
instruments
on
slack
or
github
or
aws,
where
you
could
lose
just
a
lot
of
money
by
being
down
for
just
a
brief
period
of
time.
B
So
when
you
migrate
into
kubernetes
you're
really
interested
in
verifying,
if
your
application
is
going
to
take
these
failures
and
still
be
operational,
and
when
we
say
operation,
then
there
are
meanings
to
it.
We
talk
about
the
five
nine
six
time.
Kind
of
a
big
availability
availability
is
one
facet
of
your
operational,
the
matrix
of
your
operational
excellence.
B
There
will
be
other
things
how
quickly
you
recover
from
a
failure.
For
example,
these
are
all
some
things
you
want
to
measure
beforehand.
So
that's
where
tears
engineering
comes
in
chaos.
Engineering
is
something
that
has
been
popularized
and
it's
really
become
important
over
the
last
10
11
years,
especially
from
the
time
there
has
been
a
distributed.
B
Micro
services
based
architecture
of
deploying
applications,
netflix
coined
the
term
chaos
engineering
clearly,
but
it
has
been
practiced
in
some
form
or
the
other
by
a
lot
of
organizations
since,
and
there
is
a
certain
difference
between
testing
and
chaos
experimentation.
You
could
argue
that
failure
testing
is
very
similar
to
chaos,
injecting
failures,
but
chaos.
Engineering
is
not
only
about
injecting
failures.
It's
a
mode
of
experimentation
and
there's
always
this
hypothesis
that
is
attached
to
your
chaos.
Engineering
in
failure.
B
Is
on
the
hull?
That
is
how
the
application
is
going
to
function
and
you're,
really
interested
in
finding
out
the
nuts
and
bolts
of
your
application
as
to
how
it
is
performing,
but
with
chaos,
engineering.
You
are
mostly
interested
in
the
when
an
application
is
deployed
in
production.
There
are
some
metrics
that
you
are
going
to
track
saying
this
application
is
working
the
way
the
way
it
is
supposed
to.
B
B
In
application,
intent
is
being
expressed
in
in
custom
resources,
and
the
state
of
these
custom
resources
generally
indicates
how
your
application
is
performing
or
what
state
it
is
in.
So
you
might
have
a
lot
of
different
kind
of
ways
to
identify
the
way
your
application
is
performing
or
your
infrastructure
is
performing,
whether
you're
meeting
your
solos
or
no.
B
So
that's
where
chaos
engineering
really
comes
in
it
tries
to
indicate
it
tries
to
execute
some
failures,
inject
some
failures,
but
at
the
same
time,
you're
also
going
to
as
part
of
that
evaluate,
what's
happening
to
results,
even
if
you
were
missing
the
mark,
how
quickly
you've
recovered.
That's
where
all
these
metrics,
like
the
mean
time
to
failure,
mean
time
to
recovery,
mean
time
between
failures,
etcetera,
coming,
so
chaos.
Engineering
is
a
lot
about
resiliency.
In
all
these
aspects,
it
is
much
like
vaccination.
B
It's
about
injecting
failures
beforehand
in
an
environment.
That's
going
to
mimic
your
production
environment
to
see
how
things
are
going
to
work
and
oftentimes
scarce
engineering
is
performed
as
game
days.
Where
you
have
a
lot
of
folks
working
on
the
product,
come
together,
different
stakeholders
come
together
and
you
take
a
sign
off
from
each
of
them.
Set
expectations
beforehand,
have
mitigation
plans
or
roll
back
after
you,
injection
of
after
you
inject
chaos.
B
Is
all
about
dml
it's
about
dealing
with
different
kind
of
resources,
native
resources,
custom
resources,
everything
is
declarative.
These
configurations
are
maintained
in
your
git
source.
The
infrastructure
is
available
as
code,
so
you
bring
up
your
clusters
using
some
infrastructure.
As
code
modules,
you
could
be
using
terraform,
you
could
be
using
ansible
or
you
could
be
using
anything
like
that
and
then
the
applications
that
you're
running
on
top
of
kubernetes
is
also
stored.
B
As
yamls
you
could
be
running
them
as
deployments
so
stay
processed.
What
not
so
that's
where
all
the
githubs
comes
in
all
the
githubs
controllers,
or
deployment
managers,
as
they
are
called
popularly,
ensure
that
the
state
of
your
cluster
is
what
is
reflected
in
the
state
of
your
source,
your
git
source.
B
So
when
dealing
with
different
aspects
of
communities,
you
could
talk
about
infrastructure.
You
could
talk
about
application,
lifecycle
management,
resource
management,
everything
is
policy
management.
Everything
is
controlled
by
yamula
in
a
declarative
fashion.
So
when
it
comes
to
resiliency
tests
or
experimentation,
you
also
want
to
adopt
the
same
kind
of
an
approach
to
doing
this
as
well.
B
B
Litmus
comes
in
and
litmus
was
born
as
a
project.
It
was
born
out
of
the
need
or
the
chaos
engineering
needs
of
the
content
based
storage
project
openly
base,
which
is
also
a
sandbox
project
in
cncf.
Today
and
as
a
community,
we
figured
out
that
a
lot
of
folks
that
are
running
open
ebs
needed
a
community
selected
way
to
run
experiments
as
well.
B
You
have
resources
that
and
the
corresponding
controllers
and
the
controllers
do
reconciliation
operation
and
we
wanted
to
enhance
or
if
we
wanted,
to
take
the
tears
engineering,
toolset
or
systems
available.
At
that
point
of
time,
we
wanted
to
make
it
cloud
native
and
define
a
sub
category
called
as
cloud
native
chaos,
engineering
and
cloud
native
chaos.
Engineering
typically
does
this.
B
We
identify
the
steady
state
conditions
and
the
entity
is
a
fault
and
we
check
whether
the
steady
state
conditions
are
maintained
or
whether
they
are
regained
within
a
desired
period
of
time.
If
so,
they
are
recalling
that
infrastructure
or
application,
or
your
deployment
environment
as
a
whole
resilient
to
that
scenario,
or
that
outage
or
that
fault.
If
not,
then
that
means
there's
a
weakness
found
and
you
go
back
to
your
drawing
boards
and
you
fix
either
the
business
logic
of
the
applications
or
they
go
and
set
something
right
in
your
infrastructure
or
your
deployment
patterns.
B
B
Of
the
chaos
engine
it
acts
upon
the
crud
operations
that
you
perform
to
the
chaos
engine
runs
the
experiment
using
the
help
of
a
certain
number
of
chaos,
resources
parts
we
are
representing
them
by
the
kiosk
runners,
and
then
it
generates
a
result
depending
upon
whether
your
steady
state
checks
where
met
or
know
whether
your
hypothesis
was
achieved
or
met,
or
no
so
just
to
look
at
these
chaos
crds.
Once
again,
these
are
the
heart
or
the
crux
of
the
litmus
chaos
project.
B
B
What
are
the
images
that
are
necessary
to
run
your
experiments,
so
the
experiment
run
as
kubernetes
jobs,
the
experiments
themselves,
the
fault
injection
methods
are
themselves
containerized,
and
this
is
where
litmus
has
this
approach
of
bring
your
own
chaos
b
by
oc.
That
is,
you
might
be
doing
some
kind
of
failure
testing
already,
and
you
might
wish
to
do
that
in
production,
to
see
how
things
behave.
It
must
helps
you
to
orchestrate
those
exp
those
tests.
Also,
it
gives
it
an
experiment,
paradigm
and.
B
Permissions
to
execute
that
particular
experiment
on
your
cluster,
so
different
experiments
have
different
blast
radius
and
thereby
different
set
of
permissions
associated
with
them.
So
the
qs
experiment
defines
that
and
it's
pre-built
and
available
on
the
chaos
hub,
which
is
an
open
hub.
You
can
call
it
as
some
kind
of
a
marketplace
for
gears,
experiments
which
is
open
and
free,
and
you
can
pull
your
chaos
experiment
crs
manifest
and
install
them
on
your
cluster.
So
they
are
previous.
A
B
On
or
turned
off,
where
do
you
want
the
chaos
resources
to
run
sometimes
folks
designate
nodes
to
run
utilities
like
litmus?
They
want
all
the
control
plane
operators
to
run
there,
whereas
you
want
other
microservices
of
the
application
from
elsewhere
on
the
other
nodes,
so
you
can
define
the
scheduling
policies
where
you
want
the
chaos
resources
to
run.
You
can
define
your
steady
state
defined
as
the
state
of
a
certain
downstream
service
endpoint,
whether
it
is.
B
B
Clear
schedule
is
another
cr
that
helps
to
repeat
your
chaos
in
different
ways.
You
can
execute
it
once
at
a
particular
time
or
run
it
across
time
ranges
with
a
specific
interval.
You
can
do
it
randomly
or
you
can
strictly
schedule.
Your
chaos,
etcetera
and
result
is
another
cr
which
is
generated
at
runtime
by
the
experiment
itself,
and
it
holds
information
about
the
experiment
stage.
What
is
the
verdict?
What
is
the
probe
status
and
any
failure
steps
if,
at
all,
there
has
been
an
experiment
failure?
B
So
these
are
the
important
artifacts
in
litmus
that
you
would
be
dealing
with
on
a
day-to-day
basis,
now
just
a
schematic
diagram
to
tell
what
happens
with
litmus
during
runtime.
So
you
have
a
cluster
in
which
you
install
the
litmus
infrastructure
components
by
helm
or
by
a
manifest
which
installs
the
operator
and
some
associated
deployments.
Like
the
event
of
metric
exporters
and.
B
Then
we
pull
the
chaos
experiment
crs
from
the
hub.
We
create
the
kiosk
engine,
which
then
is
watched
by
the
operator,
and
it
in
turns
create,
creates
the
chaos
runner
parts,
which
are
the
ones
that
actually
execute
the
chaos,
jobs
and
the
results
are
stored
in
a
chaos
result
which
is
available
for
consumption,
and
you
can
create
your
decisions
based
upon
this
result.
So
this
is
what
happens
with
runtime
now
I
just
want.
B
A
B
They're
also
planning
to
run
kiosks
experiments
during
as
part
of
their
release
pipelines,
so
they
bring
it
into
the
continuous
delivery
process,
not
something
that's
left
to
the
sre
ops.
So
people
do
some
kind
of
case.
Experiments
in
ci
pipelines
or
ephemeral
target
clusters
then
go
ahead
and
use
the
images
or
application
artifacts
that
are
qualified
in
long-run
long-running
clusters.
B
They
then
bring
it
on
to
staging
pre-prod
and
then
eventually
deploy
the
application
in
production
and
then
do
chaos
there
as
part
of
a
game
day.
So
this
is
the
left
shift.
That's
happened
in
recent
times
and
one
of
the
important
facets
or
requirements
of
being
able
to
run
chaos
as
part
of
your
continuous
delivery
pipeline
is
to
be
able
to
validate
your
slos
as
the
experiments.
B
A
A
Tackling
this
chaos,
that's
actually
a
very
good
idea
and
a
very
good
approach.
This
will
be
something
that
we
are
also
already
thinking
about,
but
for
the
first
use
case,
we've
really
thought:
how
can
we
introduce
chaos
as
part
of
the
cd
pipeline
triggered
by
captain
and
then
also
evaluated
by
captain.
A
A
A
Captain
comes
with
its
own
git
repository
that
lives
in
your
cluster,
but
you
can
back
it
to
a
public
or
to
a
cloud-based
git
repository.
You
can
link
your
container
registry.
You
can
link
different
deployment
tools,
test
automation,
tools
like
chainmeter,
and
that's
also
where
we
see
the
litmus
integration.
B
A
Demo
for
today-
and
I
will
briefly
explain
what
we
will
see
in
the
demo
and
then
kartik
will
explain
our
demo
environment
and
then
we
will
jump
into
the
demo.
So,
let's
imagine
you
have
a
pipeline.
We
want
to
deploy
something
into
our
chaos
stage.
We
are
using
the
demo,
a
new
cncf
demo
project,
it's
called
potato
head
and
we
are
deploying
this
using
helm.
So
we
are.
We
are
deploying
this
as
a
next
step.
Once
the
deployment
is
finished,
captive
will
go
ahead
automatically
and
trigger
the
tests.
A
In
our
case,
we
will
trigger
some
load
testing
with
chemita,
but
also
at
the
same
time
we
are
triggering
litmus
chaos
experiments.
So
this
is
the
benefit
also
from
captain
that
being
event-based.
We
can
trigger
different
integrations
at
the
same
time
since
they
just
listen
for
these
events
that
they
want
to
start
the
tests,
so
both
chamina
tests
and
litmus
chaos
will
be
executed
on
our
demo
application
once
the
tests
are
finished,
capture
will
be
informed
that
the
tests
are
finished
and
will
trigger
the
quality
gate.
A
The
quantity
gate
basically
reaches
out
to
a
data
provider
in
our
demo.
It's
pro-voice
and
from
the
data
provider
it
will
query
the
slis
and
evaluate
the
slos
and
will
then
come
up
with
a
score
and
based
on
this
score,
you
will
get
let's
say
a
thumbs
up.
You
will
get
high
score
it
will
you
will
pass
the
quality
gate.
That
also
means
the
resiliency
that
you
that
we
are
checking
in
our
slos
is
satisfying
or
if
we
cannot
reach
the
score,
that's
needed.
A
That
means
our
resiliency
is
not
satisfying
and
we
need
to
rework
or
we
we
have
to
work
on
increasing
the
resiliency.
I
think.
Actually,
I
was
not
sure
if
I
do
have
an
example
of
the
captain
quality
gates.
If
I
have
a
slide
here,
but
if
not
it's
not
a
big
of
a
deal,
because
I
would
just
explain
this
then
in
the
demo
so
kartik
you
can
go
ahead
and
explain
the
demo
environment
and
then
we'll
jump
into
the
demo.
B
B
We
are
running
it
in
the
litmus
chaos
name
space
and
its
deployment
is
being
controlled
by
the
captain
control
plane
as
part
of
a
deploy
event.
We
are
going
to
have
the
departure
head,
hello,
service,
app
deployed
on
the
cluster,
and
we
also
are
going
to
launch
the
chaos
experiments.
The
pod
lead
chaos.
B
You
can
find
the
instructions
in
the
litmus
service
repository
in
the
capital
sandbox
or
which
you
can
just
going
to
point
out
shortly.
So
we
have
the
ripples,
infra
components
already
available.
The
experiments
are
going
to
be
triggered
immediately
after
the
deployment
of
the
hello
service
is
successful.
B
B
Regarding
this
there's
a
pro
success
percentage,
which
is
essentially
about
how
much
times
it
is
successfully
able
to
query
the
hello
service-
and
we
are
representing
that
as
a
percentage
and
another
one
which
is
the
duration
in
microseconds,
the
amount
of
time
taken
to
receive
a
success
upon
trying
to
access
the
whole
service.
These
metrics
are
being
scraped
by
the
prometheus
instance
and.
B
A
So
I'm
taking
over
here
screen
yeah
thanks
for
explaining
the
environment
and,
let's
just
take
a
look
at
what
we
currently
have.
Currently
we
have
that's
the
captain's
ui.
We
call
it
the
captain's
bridge
and
we
currently
have
application
running.
It's
our
hello
service
and
we
just
deployed
in
one
stage.
So
it's
in
this
case.
It's
not
a
multi-stage
deployment,
it's
just
a
single
stage
deployment
because
we
are
interested
in
evaluating
the
resiliency.
A
A
So
there
is
no
need
to
rewrite
hundreds
of
pipelines
if
you
have
hundreds
of
microservices,
but
it's
just
you
just
define
one
additional
service
and
the
service
will
listen
for
events
that
are
sent
through
the
through
the
delivery
of
the
artifact,
and
so
that's
really
really
easy
to
include
this.
We've
already
prepared
this
and
we
will
go
ahead
and
start
a
new
deployment.
So
what
we'll
do.
B
A
Will
deploy
my
demo
application
or
it's
actually
not
my
demo
application,
but
I
hosted
it
here
under
my
github
account,
but
we
are
taking
the
the
official
cncf
potato
head
application
and
we've
made
only
some
introductions
with
prometheus
instrumentation.
This
is
the
reason
why
we
pushed
it
into
our
private
docker
hub
account
and
we
just
want
to
deploy
it
with
a
replica
count
of
one
and
we
will
go
ahead
and
we
will
do
this
with
the
captain
control
plane
and
with
the
captain
cli.
A
So
in
this
case,
I'm
just
sending
this
cloud
event
by
a
captain
to
my
captain,
control,
plane
and
captain
will
take
over
and
will
take
some
action
based
on
this
cloud
event.
So
this
cloud
event
is
of
the
type
configuration
change.
So
captain
now
knows:
there's
some
configuration
change
and
we'll
figure
out
it's
this
version
of
the
microservice
with
a
replica
count
of
one.
That's
our
basically,
that's
that's
the
basic
configuration
and
we
can
take
a
look
in
our
captain's
bridge.
So
we
can
see
there
was
a
configuration
change
just
received.
A
We
can
also
see
the
deployment
was
already
finished
and
once
the
deployment
is
finished,
what
captain
does
is
it
will
trigger
the
tests
and
in
this
case
it's
not
only
the
chemita
tests,
but
it's
also
the
chaos
tests
and
we
can
actually
take
a
look
in
our
lens.
That's
an
ide
for
kubernetes,
let's
say,
and
we
can
take
a
look.
I
just
reloaded
and
we
can
take
a
look
that
there
is
a
couple
of
things
going
on.
So
we
can
see
there
is
a
chaos
runner.
A
There
is
a
pot
delete
part
and
there's
also
the
hello
service.
It
was
actually
when
I
was
reloading.
We
can
see
it's
not
yet
up
there
and
we
can
see
it's
very
new,
so
the
initial
deployment
was
already
finished
earlier,
but
the
chaos
experiment
was
already
triggering
a
deletion
of
our
pot.
So
we
just
wait
for
the
pot
to
come
up
again.
It
will
take
a
couple
of
more
seconds
and
we
can
see
it
here.
So
the
pod
was
coming
up
again
and
I
think
I'm
fighting
with
the
mouse
against
kartik
here.
A
So
we
can
see
it's
it's
up
and
running
again.
So
what
we
will
do
is
we
will
come
back
to
the
captain's
bridge
and
we'll
take
a
look
if
the
tests
have
already
been
finished
and
what
is
evaluation
of
the
tests
and
if
we
have
a
high
resiliency
of
our
microservice
or
not
just
checking
up
here,
the
tests
have
not
been
finished.
A
So
what
is
going
on
in
the
background
captain
was
triggering
both
testing
integrations,
let's
say
the
the
chain
meter,
service
and
litmus
chaos.
The
litmus
chaos
was
finished
a
little
bit
earlier
than
the
chamina
service,
and
what
we
are
really
waiting
for
here
is
for
the
chimita
service
to
finish,
and
we
can
already
see
meter
tests
have
now
been
finished.
A
That
will
provide
us
with
the
data
that
we
can
then
use
in
the
evaluation
so
captain
reaches
out
to
in
this
case
it's
promethoids
will
fetch
the
data
and
will
come
up
with
the
score.
So
we
have
evaluation
that
was
executed
and
we
can
see
our
result
was
actually
zero
percent
and
we
will
need
at
least
75
percent
for
it
to
pass.
A
A
So
we
see
the
resiliency
was
not
high
enough.
Actually,
we
failed
this
evaluation.
What
we
will
try
now
in
the
second
round
of
the
demo,
we
will
try
to
increase
the
replica
count.
Captain
would
automatically
trigger
the
tests
again
the
chaos
again
and
we'll
come
up
with
a
new
evaluation,
and
we
will
check
if
the
new
evaluation
will
successfully
pass
our
criteria
regarding
the
resiliency.
A
So
let
me
go
back
to
visual
studio.
I
will
just
increase
the
replica
account
here.
We
take
the
same
version.
The
exact
same
application-
and
I
will
just
send
another
cloudinet
to
captain
again
captain
will
take
over.
We
will
just
take
a
look
in
the
bridge
when
captain
is
picking
up
this
event.
It
should
just
take
a
second,
so
captain
is
already
picking
up
this
event,
and
now
we
will
switch
back
to
to
lens.
A
Why
I'm
doing
this
too
quickly
is
because
right
now
we
can
see
that
there
are
two
additional
instances
of
the
hello
service
coming
up.
This
is
we
increase
the
replica
count
we
can
see.
Now
there
are
two
additional
services
coming
up,
so
we
will
see
just
in
a
second
that
there's
also
an
event
sent
to
captain
that
the
deployment
is
finished
usually
takes
about
30
seconds
for
it
to
be
ready.
We
introduced
some.
We
introduced
the
readiness
probe
and
we
just
wait
for
a
couple
of
seconds
for
it
to
be
ready.
A
B
Yeah,
I
think
we
can
sort
of
look
at
the
events,
so
you
can
see
that
the
the
kept
in
control
plane
has
actually
launched
custom
resources.
The
chaos
engine
has
been
created
which
is
led
to
the
creation
of
these
parts.
There
are
parts
of
the
body
pod,
I
think,
on
the
lenses.
We
have
a
category
for
custom
resources
here
that
you
can
take
a
look
and
we
have
the
ritmus
kiosk
custom
resources.
Here
you
can
see
that
the
chaos
engine
has
been
created.
B
You
can
see
that
there
are
lot
of
events
generated
when
the
kiosk
engine
has
been
in
operation,
and
these
these
events
are
helpful
for
observability
purposes
and
to
know
what's
actually
happening,
as
the
experiment
goes
about
performing
its
thing
at
each
stage
of
the
experiment.
You
have
an
event
that
signifying
what
went
well
or
know
you
can
see
that
the
kiosk
resources
have
been
created
successfully.
A
B
Experiments
have
dependencies
like
conflict
maps
or
secrets,
which
you
might
push
so
as
part
of
the
experiment
from
the
operator
validates
whether
they
are
available.
Then
we
create
the
experiment
job.
Then
we
check
as
part
of
a
pre
chaos
entry
check
whether
the
application
under
test
is
running
after
the
chaos
is
completed.
B
We
go
ahead
and
check
once
again
for
the
application
as
part
of
a
post
chaos
and
exit
check.
Then
we
go
ahead
and
remove
these
resources,
so
this
is
what
happens.
I
just
wanted
to
show
this.
What
happens
in
the
background
when
the
resources
are
applied?
Yeah.
I
think
we
can
now
take
a
look
at
our
captain
bridge.
A
A
A
In
the
previous
run,
the
traffic
could
not
be
served,
so
we
got
from
the
blackbox
exporter.
We
could
not
successfully
do
our
probes,
and
this
was
the
reason
in
the
first
run.
We
got
some.
We
got
not
a
very
high
success
percentage
because
for
a
couple
of
requests
the
service
was
not
rated
was
not
ready
to
respond
with
a
higher
replica
set.
We
can
now
satisfy
our
quality
criteria.
A
So
with
this
integration
between
captain
and
litmus
case,
it's
very
easy
to
evaluate
the
resiliency
of
microservices,
because
you
can
just
play
with
the
values,
for
example,
of
a
replica
set,
but
you
can
also
tweak
other
things.
You
can
write
your
own
slis
and
slos,
and
you
can
just
run
a
new
delivery
workflow
with
captain
and
have
chaos,
engineering
and
also
some
performance
tests
being
executed,
and
then
all
and
you
have
an
automated
evaluation
of
this
test
run.
A
So
you
can
automate
a
lot
of
things
and
you
can
even
think
of
building
a
multi-stage
pipeline
with
this.
So
every
time
that
you
successfully
pass
a
quality
gate,
you
can
move
it
to
the
next
stage,
for
example,
to
a
pre-production
stage
or
from
a
development
stage
to
a
chaos
stage
to
a
pre-production
stage
to
production
stage.
So
all
of
this
is
possible
with
a
lot
of
automation
and
with
the
help
of
litmus
ksc
captain.
A
We
already
had
a
couple
of
questions
in
the
chat.
I
would
just
take
a
look
at
them.
I
know
kartik
is
already
answering
them,
I'm
not
sure
if
everyone
can
see
the
questions
and
the
answers.
So
the
last
question,
which
is
really
much
related
to
this
scenario,
is
why
are
we
triggering
load
with
both
litmus
and
chamber?
It's
a
very
good
questions.
Question.
The
reason
is
when
we
only
execute
the
litmus
scales
on
our
service
on
the
test.
A
We
don't
have
a
lot
of
traffic
on
it,
so
it's
difficult
to
make
good
evaluations
in
the
quantity
gate
if
there
is
no
traffic
on
the
service.
So
for
this
reason
we
are
executing
both
performance
tests,
but
also
chaos,
because
we
want
to
actually
identify
how
does
chaos
affect
our
application
if
it's
under
load,
because
in
production
you
hopefully
always
have
some
load
on
your
system?
A
And
for
this
we
want
to
evaluate
litmus
chaos,
experiments
with
some
load
testing.
It
doesn't
have
to
be
cheap
leader.
You
can
use
a
new
load,
you
can
use
place
media,
you
can
use
other
tools,
but
we
think
it's
a
good
good
practice
to
have
load
on
and
chaos
at
the
same
time,
because
otherwise
it
would
just
kill
a
port
and
if
it
doesn't
receive
any
traffic,
you
would
not
get
any
valuable
insights
into.
B
The
questions
thanks,
I
think,
just
to
add
on
to
that.
I
think
the
deepak
had
an
interesting
perspective
about
considering
load
as
chaos.
That's
that's
a
very
nice
perspective
and
there
are
ways
of
triggering
load
waveness
as
well.
I
saw
the
question
on
chat.
There
are
experiments
which
help
you
to
trigger
io
stress
on
your
applications.
B
B
So
I
try
to
answer
that
by
saying
captain
allows
you
a
graded
kind
of
slo,
so
you
saw
right
now
the
bridge
that
you
have
a
certain
percentage
below
which
you're
going
to
be
called
as
failure,
after
which
you
want
and
then
you'll
actually
fail.
So
it's
basically
or
it's
a
success.
So
it's
a
success,
there's
a
warning
and
then
there's
a
failure.
A
B
You
can
set
these
slos
against
different
slis
that
you
might
have
configured.
You
might
have.
Metrics
that
you
are
creating
sli
against,
you
might
have
a
list
of
them
is
from
just
one.
In
this
case
we
have
couple
of
them
probe
success,
percentage
and
population
miss
against,
but
you
could
have
more
and
those
metrics.
B
Be
indicative
of
the
critical
path
or
non-critical
path,
and
you
can
set
these
gradations,
you
can
adjust
these
warning
failure
and
success
against
them
in
order
to
get
that
overall,
experience
of
you
know
this
is
what
messelo
is
for
a
non-critical
service,
and
this
is
what
it
is
for
a
critical
service.
That's
just
my
perspective.
I'd
like
to
hear
from
you
again
as
well.
A
A
It
was
a
perfect
answer
already,
but
I
just
wanted
to
point
out.
We
also
have
this
here
in
the
ui.
We
have
a
a
warning
and
a
fail.
We
also,
if
you
just
want
to
be
notified
about
something
you
don't
have
to
define
any
qualitative
criteria.
You
don't
have
to
define
an
slo
for
the
sli.
You
can
just
include
it
if
you
just
want
to
retrieve
the
values,
so
captain
is
anyway
reaching
out
to
the
data
provider
and
capturing
all
the
values.
A
A
That
means
the
complete
quality
gate
will
be
failing,
for
example,
something
like
uptime
or
error
rate.
You
would
not
want
a
higher
error
rate
of
let's
say:
five
percent,
ten
percent
or
whatever
is
crucial
for
you,
but
if
you
have
a
lot
of
slos
defined
or
you
might
also
want
to
say-
and
it's
it's
that
critical-
that
you
want
to
define
it
as
a
key
sli.
A
It's
just
that
in
this
demo
kartik
and
I
we
decided
that
we
use
the
probe
success
percentage
and
the
probe
duration
in
milliseconds.
What
you
usually
get
out
of
the
box
when
you're
using
captain
is
the
red
metrics.
It's
the
error
rate,
the
throughput
and
the
response
time.
These
metrics,
you
will
get
out
of
the
box
for
services
for
for
microservices
or
applications
that
you
manage
with
captain,
and
these
come
out
of
the
box.
A
B
A
Yes,
absolutely
so
the
slos
they
are
not
predefined.
By
captain
you
can
have
the
snis.
These
are,
let's
say,
pre-built
and
they
are
already
available
inside
captain,
but
the
slos
you
would
just
define
them
by
yourselves.
It's
the
slo
definition
is
a
plain
yaml
file.
We
can
also
take
a
look
at
our
slos
here,
so
this
is
actually
the
yaml
file
that
we
defined,
and
it
just
says
this
probe
success
percentage-
has
to
be
higher
than
95
percent
to
pass
this
evaluation.
A
If
it
does
not
satisfy
this
criteria,
we
are
allow
it
with
half
of
the
score,
because
we
get
a
warning
if
it's
still
higher
than
90.
So
this
is
the
complete
all
these
lines.
This
is
the
complete
slo
file
and
you
can
just
add
more
and
more
slis
and
slows
here
and
just
increase
the
the
robustness
and,
let's
say,
of
your
quality
gate.
If
you
only
have
two
slis
and
leslos
that
you
want
to
check,
it's
not
a
lot,
but
you.
A
More
and
more,
we
have
seen
installations
with
captain
that
check
against
quite
a
lot
of
of
slos,
because
they
want
to
check
each
and
every
aspect
of
of
a
microservice.
That's
totally
fine.
Just
for
the
demo.
We
decided
to
just
go
for
actually
for
some
kind
of
uptime
criteria
so
that
we
can
see
when
we
kill
a
part
how
it
does
how
it
will
affect
our
availability.
A
B
I
think
it's
it's
about.
I
think
the
deepak
is
making
a
comment
that
we
could
start
setting
the
slos
right
during
the
requirement
gathering
stages
of
an
application
or
project
so
that
we
can
go
ahead
and
see
how
we
are
adhering
to
it
right
from
the
beginning
and
probably
improve
our
applications
along
the
way
yeah.
I
think
he
said
yes,
so
yeah,
that's
a
great
thought.
B
I
would
probably
just
recommend
you
to
read
up
on
the
philosophy
that's
being
adopted
nowadays
called
chaos
first
and
something
that
we
are
trying
to
help
advocate
as
a
proud
native,
curious
engineering
group
as
well
deeper.
So
cures
first
is
an
approach
where,
right
from
the
day,
one
of
your
application
journey,
you
start
with
chaos,
experimentation
and
we
use
tools
like
captain
to
do
automated
delivery
and
include
chaos
as
part
of
that.
So
you
know
you
have
a
tab
on
how
your
applications
are
behaving.
B
The
slos
can
be
recalibrated
as
you
go
along
or
you.
You
can
even
use
the
reference
portal,
which
is
another
tool
which
we
will
look
to
integrate
with
captain,
which
is
giving
you
even
more
information
about
how
your
resiliency
scores
can
be
set.
Rates
can
be
set
against
experiments,
so
yeah
secures
first
is
is
what
we
are
calling
that
approach.
What
we
just
mentioned,
as
I
think
that's
really
important
thanks
for
bringing
that
up.
B
There
was
another
interesting
question
about
servant
is
chaos
which
I
think
is
an
altogether
new
topic
and,
I
think,
kept
him
being
an
event
triggered
framework
and
we're
also
working
towards
experiments
litmus
framework
being
event
triggered,
so
it
should
sort
of
integrate
at
some
point
with
serverless,
though
that
capability
might
not
be
aware
today.
Do
you
have
any
thoughts
on
this.
A
I
I
think
it's
a
it's
a
very
valid
use
case.
What
might
not
be
like
our
experiment
won't
be
easy
to
replicate
in
a
serverless
world
because
you
can
just
not
kill
a
killer
part
but
having
some
some
downtime
or
network
issues
simulating.
This
is
very
valid
for
captain.
It
does
not
really
matter
where
the
application
is
running
if
it's
a
monolithic
application,
if
it's
a
microservice
application,
if
it's
a
serverless
application
as
long
as
there
is
a
way
to
query
metrics
for
this
application.
A
So
if
you're
monitoring
this
with
promethos
or
dyna
trace
or
some
other
monitoring
and
captain
is
able
to
fetch
the
data,
then
we
can
perfectly
do
this
quality,
gait
evaluation
and
if
it's
possible
to
inject
chaos
into
serverless
applications.
That
would
be
the
question
to
kartik,
because
I'm
not
sure
if
that's
possible
right
now,.
B
Right,
I
think
one
of
the
things
that
sort
of
helps
us
in
that
aspect
is
the
litmus
experiments
run
as
a
one-time
job
that
sort
of
fits
well
within
the
overall
paradigm
of
serverless,
but
it
it
depends
right
now
I
mean
it's
a
serverless,
if
you're
doing
it
on
some
infra
component,
that
litmus
is
not
able
to
talk
to
today
and
that's
something
that
we
need
to
build.
That
capability,
for
example,
you're
running
up
an
instance
and
you're
doing
something
against
that
or
if
it's
on
communities,
then
you
have
it.
B
A
Are
more
questions,
please
also
put
them
in
the
chat
or
in
the
q
a,
and
we
still
try
to
answer
them.
I
just
want
to
go
back
to
the
slides
and
just
highlight
how
you
can
actually
use
this
it's
available
of
as
of
today,
so
this
integration
in
the
demo
that
we
showed
you
is
already
available
on
github
to
install
the
integration.
A
It's
basically
just
one
cube
ctl
apply
and
you
would
just
install
the
integration
between
litmus
and
captain
just
make
sure
you
first
have
to
install
captain.
You
first
have
to
install
litmus,
but
the
integration
for
those
two,
it's
basically
one
qc
they'll
apply,
and
then
you
will
just
add
the
chaos
experiment
to
captain
captain
will
add
it
to
its
internal
git
repository
we'll
manage
it
from
there
in
this
way.
A
A
A
Please
take
a
look
if
you
think
of
a
new
use
case.
If
you
want
to
help
us
improving
it
file,
an
issue
there
or
raise
a
pr,
we
are
happy
to
include
this.
I've
already
talked
about
this
a
little
bit
in
the
beginning.
What
we
are
thinking
about,
as
the
next
step
is
actually
also
to
consider
the
results
of
the
case,
experiment
that
are
already
part
of
litmus
to
consider
them
and
take
them
into
account
as
part
of
their
evaluation.
B
A
Want
to
achieve
is
to
have
litmus
chaos
as
the
tests
for
auto
remediation,
so
not
only
in
shifting
left
into
your
cd
pipeline,
but
really
having
this
as
a
way
to
test
your
operational
scripts.
If
something
goes
wrong
in
your
environment,
in
your
kubernetes
cluster
or
in
your
application
running
on
kubernetes,
how
do
you
usually
react
on
this?
Are
there
any
scripts
that
you
would
trigger
as
a
human
as
a
counteraction?
B
A
A
I
can
also
post
it
in
here
really
quick
and
hopefully
you
have
access
and
take
a
look
and
can
take
a
look
I'll,
just
post
it
to
the
in
the
chat,
but
I
will
also
attach
it
to
the
youtube
video
that
will
go
live
tomorrow,
yeah.
I
think
there
are
no
more
questions
so
with
this
we
will
be
closing
this
webinar
thanks
again.
Kartik
for
joining
and
yeah
have
fun
everyone
introducing
chaos
to
your
applications
and
see
you
soon.