►
From YouTube: Putting Chaos Into Continuous Delivery to Increase App... Juergen Etzlstorfer & Karthik Satchitanand
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Putting Chaos Into Continuous Delivery to Increase Application Resiliency - Juergen Etzlstorfer, Dynatrace & Karthik Satchitanand, Mayadata
Continuous Delivery practices have evolved significantly with the cloud-native paradigm. GitOps & Chaos Engineering are at the forefront of this new CD approach, with an ever-increasing pattern involving Git-backed pipeline definitions that implement “chaos stages” in pre-prod environments to gauge SLO compliance. In this talk, maintainers of the Keptn (Juergen) & LitmusChaos (Karthik) CNCF sandbox projects will discuss how you can construct pipelines that include chaos experimentation (mapped to declarative hypothesis around application steady-state) while simulating real-world load, and implement quality gates to ensure resilient applications are deployed into production. All this - in a GitOps native manner. They will also demonstrate how you can include chaos tests to your existing CD pipelines without the need of rewriting them.
A
B
A
A
B
Let's
look
at
why
keras
engineering
is
important
and
what
are
some
practices
we
can
follow
to
improve
the
resiliency
of
our
application
infrastructure
load
tests
are
great,
functional
tests
are
great,
but
they
need
to
be
augmented
with
failure
scenarios,
especially
so
in
a
cloud
native
world
where
everything
is
in
the
form
of
a
micro
service.
Everything
is
loosely
coupled.
There
is
so
much
points
of
failure,
the
surface
attack,
surface
area
for
attacks
or
failures
rather
more,
and
it
is
important
for
us
to
test
what's
happening
when
the
components
surrounding
our
applications
and
business
apps
fail.
B
Then
you
have
a
host
of
microservices
that
you
have
from
the
cloud
native
ecosystem,
the
cncf
landscape
for
service
discovery,
the
storage
for
observability,
etc,
and
then
you
have
your
application
stack.
There
can
be
so
many
things
that
fail
and
it's
important
for
us
to
recognize,
what's
happening
in
these
components
and
underground
failures,
so
failure
testing
is
important.
In
other
words,
chaos
is
really
important
for
us
to
inject
and
find
out
what's
happening.
B
Chaos.
Engineering
is
a
discipline
every
scientific
one,
and
there
are
a
lot
of
assumptions
that
we
go
ahead
with
when
we
write
our
applications
or
when
we
go
ahead
and
deploy
them.
We
assume
that
networks
are
reliable.
The
latency
is
always
very
less
or
nearly
zero.
We
have
infinite
bandwidth,
abundant
storage
and
compute
resources,
but
it's
not
always
the
case.
We
really
want
to
simulate
these
conditions
and
find
out
what
happens
there.
B
Failures
that
you
know
what's
going
to
result,
we
call
them
as
known
knowns
and
it's
important
to
do
some
kind
of
regression
around
that
some
kind
of
chaos
experimentation
repeatedly
to
find
out
if
that
still
holds
true.
But
there
are
a
lot
of
unknowns
and
lot
of
assumptions
that
you
want
to
validate,
especially
so
in
environment
that
mimics
production,
where
things
are
really
very
dynamic,
and
this
is
a
lot
of
churn
coming
to
the
word
churn.
Why
is
it
important
for
us
to
do
chaos
testing
continuously?
B
B
B
Of
chaos
is
really
important.
That's
not
to
be
done
away
with.
That's
really
the
nirvana
of
a
mature
chaos
engineering
practice,
especially
when
done
on
production,
but
it's
important
for
us
to
automate
this
and
keep
running
this
continuously
on
pre-prior
environments
to
find
out
whether
really
our
services
behave
the
way
they
do
and
we
are
confident
and
whenever
we
do
chaos
engineering.
Whenever
we
do
chaos
experimentation,
it
is
important
for
us
to
carry
some
hypothesis
around.
What's
the
failure
going
to
result
in
how
are
my
services
going
to
behave?
What's
the
impact
on
downstream
applications?
B
A
B
And
these
are
going
to
be
very
close
and
going
to
con
going
to
form
what
will
be
the
service
level
agreements
that
you
might
have
with
your
end
users,
it's
important
for
us
to
accompany
or
marry
these
checks,
along
with
chaos,
experiments
every
time
now
coming
to.
We
spoke
about
why?
Let's
talk
about
how
in
chaos,
we
want
to
go
the
declarative
way
now,
because
in
the
cloud
native
world,
everything
is
declarative
right
from
the
way
you
define
your
infrastructure.
B
The
way
you
define
your
applications,
the
way
you
manage
their
life
cycle,
the
way
you
define
resources
on
your
policies,
everything
is
done
in
a
declarative
way
as
a
mls.
We
want
to
do
that
with
resilience
checks
as
well,
in
other
words,
with
chaos,
experiments
as
well,
so
it
is
important
that
we
can
define
chaos
into
our
custom
resources
and
use
the
same
paradigm.
A
A
B
Use
the
traditional
etops
controllers
in
your
chaos
flow
as
well.
Let
me
introduce
the
litmus
chaos
project.
It
has
been
growing
for
some
time
now.
It's
a
sandbox
project
right
now.
It's
getting
contribution
from
other
organizations
and
the
chaos
intent
here
is
defined
as
custom
resources.
There
are
multiple
custom
resources,
each
serving
a
different
function,
and
there
is
a
chaos
operator
which
acts
on
these
and
actually
executes
your
chaos
and
this
tool,
or
this
platform
rather
has
been
built
with
certain
principles
in
mind.
You
can
see
those
principles
on
this
screen.
B
B
Experiment
you
actually
find
out
whether
you
met
your
hypothesis
or
around
your
study
state,
whether
you
met
your
solos
or
no,
all
that
information
is
captured
in
an
experiment
because
of
this
model
it
fits
well
into
ci.
Cd
pipelines
in
the
subsequent
portion
of
this
stock
urine
will
talk
about
what
the
captain
project
is
about
and
how
it
leverages
litmus
to
introduce
chaos
stages
as
part
of
continuous
delivery.
A
A
Everything
is
based
on
declarative
descriptions
very
well
aligned
also
to
other
projects
such
as
littles
and
stored
everything
in
git,
so
we're
following
github's
approach
here
as
well.
So
how
can
we
actually
use
this?
And
how
can
we
set
up
now
a
project
where
we
want
to
integrate
litmus
chaos
into
a
city
workflow?
A
A
You
were
putting
in
repository
and
kept,
will
act
upon
this
definition
and
in
another
concept,
which
is
called
the
captain's
uniform.
You
will
then
add
the
tooling
that
is
responsible
for
each
task.
For
example,
it
has
to
be
executed
as
part
of
the
shipyard
definition,
this
separation
of
concerns
by
the
how
and
the
what
is
basically
done
with
cloud
events.
So
it's
an
event-based
approach
and
captain
will
make
sure
to
send
the
cloud
events
with
all
the
information
that
is
needed
for
other
tools
to
to
act
upon
this.
A
So,
for
example,
the
deployment
itself
is
not
done
by
captain
itself,
but
captain
can
use
helm,
for
example,
to
send
all
the
deployment
information
to
help
and
get
the
help.
Integration
will
act
upon
this
and
will
trigger
them
on
execute
the
deployment.
The
same
is
true
for
the
testing,
so
it's
not
only.
B
A
Can
support
litmus
tests
but
also
chimita
tests,
local
tests
other
test
integrations.
They
will
wait
for
captain
to
trigger
them
and
then
once
triggered
they
will
execute
the
tests.
The
test
instructions
are
stored
in
the
git
repository
managed
by
captain
will
be
provided
to
the
test
integrations
and
then
they
can
execute
the
tests
in.
A
Of
litmus
chaos,
the
the
chaos
experiment,
the
chaos
test
itself
is
a
gamble
file.
It
will
be
provided
by
captain
to
litmus
and
litmus
will
then
act
upon.
This
will
execute
its
chaos
test
and
will
then
come
back
with
a
result
to
captain.
So
captain
can,
for
example,
in
the
evaluation
phase,
then
trigger
develop
the
tool
that
is
responsible
for
the
evaluation
we're
using
here.
It
builds
in
functionality
of
captain
the
captain
lighthouse
service
doing
now
the
evaluation.
A
So
how
does
an
evaluation
look
like
again
here?
It's
a
declarative
description,
it's
based
on
service
level
objectives,
and
let
me
start
here
with
the
with
the
second
picture
here,
on
the
left
hand,
side
where
we
already
have
an
objective
defined
based
on
an
sli.
Let's
say
the
probe
success
percentage.
A
That
means
what
is
the
success
percentage
of
all
the
probes
we
are
sending
within
the
given
time
frame
or
that
we
are
doing
within
that
given
time
frame
and
we
needed
higher
than
95
percent
of
success.
If
we
want
this
objective
to
fully
pass,
if
we
cannot
meet
this
criteria,
capital
will
evaluate
the
boarding
criteria
it
has
to
still
be.
B
A
Actually,
retrieved
is
then
defined
in
the
service
level
indicator
file.
This
is
basically
a
mapping
between
the
name
of
the
circuit
indicator
and
you
can
think
of
as
a
promql
with
placeholders
so
that
you
can
easily
reuse
this
for
different
services
for
different
time
frames,
these
kind
of
things.
So,
with
these
two
files,
we
can
go
ahead
and
do
evaluation
either
within
the
execution
of
a
captain,
shipyard
definition
or
also
triggered
via
the
api
or
by
the
cli.
If
you're,
just
interested
in
the
captain
quality
evaluation
ad,
let's
say
either
way
once.
A
A
B
B
B
B
It
is
deployed
using
help,
that's
the
deployment
tool
of
choice,
and
once
the
deployment
is
completed
there
is
a
deployment
finished
event
which
actually
triggers
the
next
set
of
tasks.
One
a
load
generator,
that's
a
locust
starting
to
put
load
on
the
hello
service
app.
So
we
want
to
do
chaos
when
the
application
is
actually
serving
requests,
not
in
idle
conditions.
That's
why
we
have
locust.
B
This
triggers
a
quality
gate
evaluation
and
as
part
of
this
evaluation,
we
are
going
to
use
metrics
provided
to
us
from
prometheus.
We
are
going
to
have
defined
some
service
level
indicators
which
are
essentially
prometheus
functions
on
top
of
some
metrics
that
is
exposed
by
the
applications
and
the
litmus
and
the
tools
that
we
have,
and
there
are
some
slos
defined
as
cutoffs
on
top
of
the
values
provided
by
this
sli,
and
we
are
going
to
evaluate
whether
those
cutoffs
are
met
as
part
of
the
quality
gate,
evaluation
and
once
it
is
met.
B
B
So
in
this
particular
demonstration
we
are
actually
going
to
highlight
an
inefficient
deployment
approach
and
uruguay
is
going
to
inform
and
take
you
through
that
demo
before
we
actually
get
into
it.
Let
us
spend
a
couple
of
minutes
to
detail
the
flow.
That's
going
to
happen
as
part
of
this
use
case.
B
First,
we
are
going
to
have
the
potato
head
hello
service,
app
deployed.
It's
going
to
be
deployed
with
the
readiness
pro
that
has
a
neat
delay
sequence
of
around
30
seconds.
It's
actually
going
to
take
some
time
for
this
particular
application
to
be
ready
and
come
up
with
an
end
point
which
can
be
queried,
and
then
we
have
a
black
box
exporter
which
is
consistently
trying
to
access
this
application
and
give
us
some
accessibility
information,
and
it's
going
to
give
us
these
two
metrics.
That
is
probe
success
and
probe
duration
seconds
probe.
B
Success
is
an
accessibility
factor
and
duration
seconds
is
an
indicator
of
how
long
it
takes
for
us
to
access
the
application
successfully.
Then
we
have
the
request
operator
and
the
dependencies
along
with
the
litmus
service
integration
service
in
deciding
in
the
kitchen
control
plane,
which
is
actually
going
to
help
trigger
this
chaos
experiment.
B
The
experiment
itself
is
going
to
be
a
simple
graceful
deletion
of
a
single
replica
of
the
hello
service,
app
we're
just
going
to
do.
One
instance
of
this
part
delete,
and
initially
we
are
going
to
do
this
against
a
single
replica
deployment
and
see
that
the
slos
are
not
really
met,
and
these
slos
are
essentially
being
built
on
top
of
these
slides
that
are
averages
against
the
probe
success
and
probability.
B
A
Thank
you
carter.
So,
for
the
sake
of
efficiency,
we
took
all
the
screenshots
of
the
demo
and
put
it
on
the
slides
to
skip
all
the
waiting
times
whenever
the
tests
execute.
So
the
demo
starts
with
triggering
a
new
delivery
based
on
the
shipyard
file
that
we've
seen
earlier,
and
we
are
just
going
to
deploy
one
image.
It's
the
hello
server
application,
which
is
part
of
the
potato
head
application
once
the
deployment
is
finished.
A
We
are
doing
this
with
hell
and
with
a
replica
set
of
one
in
the
first
round
of
the
demo,
captain
will
make
sure
to
trigger
the
tests,
the
test,
definitions
and
the
chaos
definitions.
These
are
all
part
all
stored
in
the
git
repository
of
captain
and
basically
provided
to
the
tool
integrations
here.
So
both
the
linkedin
service
and
the
local
service
will
start
the
work.
They
don't
even
have
to
know
that
the
other
services
running
as
well,
so
you
can
just
reuse
all
the
performance
tests.
A
You
already
have
and
add
the
chaos
on
top
of
this
once
the
services
are
finished
and
capturing
waits
for
both
services.
To
finish
the
evaluation
started.
So
we
can
see
here,
the
litmus
service
was
finished
successfully.
Also,
the
local
service
was
finished
successfully.
That
means
they
both
did
their
job
and
indicated
that
during
their
execution
there
was
no
problem.
So
in
the
evaluation
captain
is
now
reaching
out
to
me
voice
and
collecting
the
data
we
can
see.
The
evaluation
is
failing
in
this
case
and
taking
a
look
at
a
detailed
evaluation
overview.
A
A
So
what
we
can
do
to
improve
the
resilience
of
our
application
in
this
case
is
to
come
up
with
the
idea
of
a
more
high
availability
setup,
increasing
the
replica
account
of
this
application.
That
would
be
one
first
good
approach
to
do
this
and
we
just
rewrite
the
cloud
event
that
we
are
sending
to
captain
and
it
will
be
then
used
as
the
instruction
for
the
deployment
in
hell.
A
So
we
are
just
re
writing
this
file
and
adding
the
replica
count
of
three
to
this,
sending
it
to
the
captain
control
plane,
which
will
then
forward
it
to
hell
helm
will
do
the
deployment
once
it's
finished
again,
the
tests
will
be
automatically
triggered.
It
will
be
the
same
tests.
It
will
be
the
same
integrations
once
the
tests
are
finished
again.
They
will
indicate
back
to
captain
it's
finished
captain.
Please
go
ahead
and
do
the
evaluation,
and
this
time
the
evaluation
is
successful.
A
A
So
the
traffic
was
only
served
by
those
two
other
instances
that
were
not
affected
by
the
populate
test
that
we
triggered
with
littles.
So
with
this,
we
can
evaluate
already
how
to
increase
the
resilience
of
applications.
We
have
not
changed
anything
in
our
application
code,
but
we
actually
increase
the
resilience
by
tweaking
the
deployment
instructions
and
having
a
higher
replica
set.
This,
of
course,
might
be
different
for
your
applications,
but
this
is
just
a
validation
of
the
approach
that
and
where
we
also
have
a
continuous
evaluation
of
chaos
and
the
impact
of
our
chaos.
A
So
with
this,
we
want
to
leave
you
with
three
key
takeaways,
the
first
one.
Is
we
really
want
to
encourage
you
to
establish
a
process
of
continuously
evaluating
the
resiliency,
not
only
to
do
it
once
or
twice
by
a
so-called
game
day,
but
really
putting
this
idea
of
a
continuous
resiliency
evaluation
into
your
cd
pipelines
and
having
chaos
tests
in
addition
to
performance
tests.
I
think
with
this,
you
can
really
increase
the
resilience
of
the
applications
and
having
this
in
a
continuous
way.
It
really
gives
the
highest
value.
A
The
evaluation
should
be
based
on
service
level
objectives.
They
have
proven
to
be
a
very
efficient
way
to
evaluate
performance
criteria,
but
also
resilience
criteria.
You
can
even
have
memory,
consumption
or
other
parts
as
part
of
your
slos
and
having
a
combination
between
more
than
one
or
between
a
huge
amount
of
service
level.
Objectives
gives
you
a
very
strong
quality
indication
of
your
applications
and
we've
already
seen
what
we
are
talking
about
today,
adopted
by
a
company
called
ketoki.
A
They
are
using
exactly
the
stack
that
we
also
used
for
the
demo
they're
using
locus
for
the
performance
tests
they're
using
captain
for
orchestrating
them
for
doing
the
evaluation,
and
they
have
added
litmus
chaos
as
part
of
their
quality
evaluations,
and
they
have
already
run
those
experiments
in
those
tests
hundreds
of
times,
and
it
has
already
proven
to
increase
the
resilience
of
their
applications
if
you're
interested
in
more.
We
have
one
resource
slide
here,
so
please
visit
us
on
litmus
chaos,
dot,
io,
that's
the
project
website
of
the
litmus
case
project
on
captain.sh.
A
You
will
find
everything
on
the
captain
project
and
how
to
use
it.
If
you
want
to
use
the
litmus
integration,
you
will
find
it
on
github.
There
is
even
a
tutorial
how
to
use
it.
The
litmus
chaos
team
and
the
captain
team
have
even
teamed
up
to
write
the
blog
two
two-part
blog
series
on
this
whole
topic.
So
there's
a
lot
of
resources
around
this.
We
really
encourage
you
to
make
use
of
them
and
you
can
also
reach
out
to
us.
A
Our
respective
twitter
handles
are
here,
and
we
are
really
happy
also
to
follow
up
with
you
on
these
topics.
With
this,
I
think
we
already
can
open
up
the
question
section
thanks
so
much
so
much
kartik.
It
was
really
a
pleasure
working
with
you
on
this
and
thanks
to
the
whole
open
source
and
cncf
community
for
taking
part
in
this
talk.