►
Description
Cloud native DevOps has emerged as a prominent sub segment of the legacy DevOps. CNCF and Kubernetes have been at the center of this transition. This transition has helped find many new best practices for cloud native developers and SREs. One such best practice is to take service reliability into consideration while the applications are designed and developed that leads to more reliable cloud native services and reduces the overall TCO. In this session, Uma Mukkara will discuss the evolution of chaos engineering in cloud native DevOps, its current state and how all the personas from CIOs to developers should be looking at taking advantage of the recent advancements in this chaos engineering space in cloud native.
A
I
am
omar
mukhera
head
of
chaos,
engineering
at
honest,
I'm
also
co-creator
and
maintainer
of
litmus
project,
which
is
an
incubating
project
at
the
cncf.
At
the
moment
before
we
begin.
Let's,
let
me
talk
a
little
bit
about
where
I
work
and
what
I
do
harness
is
a
modern
software
delivery
platform.
It
is
various
features
like
continuous
integration,
continuous
deployment,
future
flags
cloud,
cost
management,
security,
test,
orchestrations,
securities
service,
reliability,
management
and,
of
course,
chaos,
engineering.
A
A
So
to
grow
within
cloud
native
very
efficiently
and
confidently.
You
need
to
have
a
reliability
in
cloud
native,
so
we'll
talk
about
cloud
native
a
little
bit
and
then
what
is
reliability
then
what
it
means
to
have
reliability
in
cloud
native
which
will
help
you
grow
within
cloud
native
very
confidently.
A
So,
as
we
all
know,
cloud
native
will
usually
have
lots
and
lots
of
micro
services
you're,
basically
moving
away
from
either
bare
metal
or
virtualized
environments
into
containers,
where
your
focus
is
really
the
small
amount
of
software
and
you're
writing
a
good
api
around
it
and
they
can
run
anywhere
right.
So
eventually,
your
application
is
broken
up
into
multiple
microservices
and
then
in
cloud
native
you're.
Seeing
these
micro
services
are
being
shipped
out
very
fast
right,
that's
one
of
the
advantages
of
our
benefits
of
moving
to
cloud
native
right.
A
So
your
software
ships
fast
and,
of
course,
this
micro
services
are
now
run
under
on
any
cloud.
Wherever
there
is
kubernetes
abstraction.
Many
of
the
cloud
providers
are
providing
kubernetes
abstraction
layer
or
community
service.
So
you
can
expect
that
your
microservice
runs
everywhere,
almost
all
on
clouds
or
on
other
enterprise
boundaries
wherever
kubernetes
distributions
are
being
deployed.
A
So
this
is,
in
summary,
what
the
cloud
native
is.
So
you
have
heterogeneous
environment.
You
are
having
multiple
micro
services
containers
which
are
being
shipped
out
faster
and
what's
also
new
in
cloud
native
compared
to
your
legacy
system
is
you
will
generally
see
more
and
more
dynamism
right,
so
basically
they
are
being
shipped
out
faster.
A
There
are
too
many
components,
and
also
you
have
lots
of
new
personas
or
people
who
are
bringing
in
you
know
new
capabilities,
not
like
legacy
system
capabilities,
and
you
know
these
people
are
needed
because
you
are
bringing
in
new
technologies.
You
are
building
new
pipelines
for
your
software
delivery
and
operations
and
management,
and
so
on
and
so
forth
right.
So
eventually,
this
is
also
leading
to
creation
of
new
personas
in
your
software
delivery
management
lifecycle
systems,
for
example,
the
the
concept
of
sre,
is
becoming
a
site.
A
Reliability
engineer
is
becoming
more
and
more
common
nowadays
in
any
cloud
native
system.
Cloud
native
developers
are
a
class
of
you
know,
a
new
software
developers
who
have
to
assume
that
your
software
is
configurable
declaratively
and
it
can
run
anywhere.
It
has
to
have
a
good
api.
So
you
need
to
be
using
new
set
of
tools
to
to
make
sure
that
you
know
all
these
are
being
satisfied.
A
So
basically
cloud
native
environment,
development,
ecosystems,
devops
ecosystem
is
definitely
a
bit
different
and
we
sometimes
call
this
as
cloud
native
devops,
and
so
that's
that's
also
one
thing
to
note
about
cloud
native.
As
you
can
see
here,
there
is
a
big
change
that
is
underway
and
we
call
it.
As
you
know,
moving
to
cloud
native
and
almost
you
see
product
services,
vendors,
you
know
participating
in
every
area
to
to
make
the
journey
of
cloud
native
moment
very,
very
successful.
A
So
it's
it's
it's
happening
and
cncf
is
leading
the
entire
change
from
the
plant.
So
that's
about
cloud
native.
Let's
also
talk
about
what
is
reliability
before
we
talk
about
reliability
in
cloud
native.
Let's
really
touch
upon
what
do
we
really
mean
when
we
say
reliability,
the
reliability
is
actually
a
kind
of
a
perspective.
A
Everybody
says
that
you
know
we
are
reliable.
Our
products
are
reliable,
our
services
are
reliable,
but
then
why
are
we
talking
about
reliability
with
such
an
importance?
Reliability
is
a
perspective.
It's
always
about
how
many
nines
do
you
have
in
your
service
when
you
say
your
service
is
reliable,
is
it
like
four
nines,
five,
nine
six
names
etc?
A
So
it's
always
measured,
as
you
know,
a
way
of
how
many
outages
that
you
had
in
the
last
year
or
so
right.
So
when
the
outages
are
increasing,
you're,
less
reliable,
your
reliability,
you
know
decreases
so
how
to
actually
increase
the
reliability
of
your
services,
really
by
reducing
the
outages
right.
So
outages
are
the
point
to
be
noted
here.
A
So
why
is
reliability
so
relevant
now
in
this
modern
digital
era?
Is
the
digital
services
are
growing
very
fast,
we're
of
course,
seeing
a
lot
of
businesses
moving
to
internet
based
traffic
because
of
all
this
coverage?
We
have
seen
this
digital
transformation.
You
know
happening
past
already,
and
the
traffic
is
probably
10
times
more
than
what
we
had
been
seeing
in
a
few
years
earlier
right.
A
The
reliability
is
really
important
because
there's
so
much
of
new
traffic
coming
on
on
to
your
digital
system
and
when
things
are
happening
at
such
rate,
the
outages,
if
what
all
they
happen,
they're
going
to
be
very
expensive,
as
you
can
see
some
examples
here
right
how
much
ever
you
try
outages
are
kind
of
you
know
a
common
thing.
It
only
matters
how
often
they
happen
or
how
rarely
they
happen,
but
they
do
happen
and
when
they
happen
there
are
going
to
be.
A
You
know
either
financial
issues,
losses
or
reputational
losses,
as
we
can
see
that
you
know
the
popular
products
or
services
that
we
do
day
in
and
day
out
have
gone
under
this
outages,
and
they,
of
course,
are
you
know
expensive
to
the
owner
right,
so
the
impact
of
outages
can
be
of
various
types.
There
can
be
reputational
outages.
Just
like
you
know
we're
seeing
here
on
slack.
A
You
know
a
huge
challenge
right
so
sometimes
so,
impact
of
outages
is
very
difficult
to
quantify,
they're,
usually
bigger
than
what
you
would
really
assume
for
short
term
and
long
term
right,
and
they
are
very
important.
And
now,
let's
try
to
understand
what
causes
these
outages
right.
There's
not
really
one
thing
that
causes
this
outages,
but
a
set
of
things,
and
usually
it's
you
know
there
can
be
application
failures
right.
So
you
have
something
gets
filled
up
because
of
logging
and
something
you
know
when
you
try
retry.
A
Sometimes
things
can
go
bad
and
certainly
stomach
can
happens.
These
are
categorized
into
application
values
and
we
have
also
seen
infrastructure
failures,
causing
these
outages
right
most
of
the
times
this
infrastructure
there
is
redundancy,
but
still
you
know
you
never
know.
Whenever
there
is
a
failure
in
infrastructure,
things
start
to
fail
somewhere
else
right.
A
Sometimes
big
infrastructure
values
like
you
know
the
whole
region
becoming
unavailable,
it's
not
very
common,
but
they
do
happen
and
can
happen
and
the
other
types
of
values
that
can
cause
outages
are
operational
failures
right,
so
capacity
issues
here,
not
populated
your
service
with
enough
capacity
and
then
under
high
traffic
things
can,
you
know,
go
very
bad
and
you're
not
scaling
the
way
you
are
expected
to
scale
under
that
that
conditions
or
that
traffic,
and
whenever
there
is
an
incident,
happen.
How
well
you
manage
that
incident.
A
You
know,
are
you
recovering
fast
or
is
there
an
auto
recovery
for
such
a
failure?
These
are
all
you
know,
part
of
operational
values,
and
then
you
know
a
good
incident.
Management
and
monitoring
is
very
important
when
you
don't
have
very
good
monitoring,
and
you
know,
ops
monitoring
systems
that
itself
could
be
a
major
factor
that
causes
a
longer
outage
right.
So
these
are
some
of
the
generally
observed
things,
and
if
these
are
very
the
general
reasons,
why
outages
can
happen?
A
Let's
talk
about
in
cloud
native
environment,
what
can
cause
an
outage
right
so
before
we
go
there?
Let's
just
remind
ourselves
here,
the
typical
you
know
nature
of
a
cloud
native
stack.
It's
a
it's
a
kind
of
a
pyramid
here,
where
your
app
as
a
developer,
what
you
concentrate
is
really
on
the
top
of
the
pyramid
and
it's
a
cloud
native
app,
which
really
means
that
you
know
it's
a
container
and
then
there
are
a
bunch
of
services
that
your
app
requires
to
function
right.
They
can
be.
A
You
know
the
message,
bus
database
basis
or
the
cloud
native
services
or
kubernetes
itself,
and
then
there
is
underlying
big
platform
on
which
kubernetes
runs
right.
So
important
thing
to
note
here
is
a
bunch
of
them
are
also
micro
services.
It's
not
just
your
application
or
your
container,
the
below
layers,
all
the
way
down
to
kubernetes
they're,
all
of
microservices
based
so
in
microservices.
A
What
can
happen
is
especially
on
kubernetes.
A
deletion
of
a
pod
is
generally
not
seen
as
a
real
failure
right
that
can
happen
whenever
some
pressure
is
put
and
kubernetes
can
just
delete
your
pod
and
then
relaunch
it
right.
So
the
architecture
itself
is
such
right,
so
this
is
actually
a
kind
of
value,
but
you
know
this
is
your
kubernetes
is
designed
to
withstand
that
value,
and
is
your
app
or
service
designed
to
withstand
such
a
failure,
and
will
it
be
providing
continuous
service?
A
A
So
other
important
thing
to
consider
in
cloud
native
is
how
fast
they
are
getting
shipped
out.
So
the
changes
into
your
pyramid
are
happening
10
times
faster
than
they
would
usually
happen
in
legacy
system.
So
you
have
a
continuous
change
of.
You
know
stack
happening
all
the
time
in
cloud
native
and
port
deletes
can
happen
anytime
in
your
stack,
including
your
application.
A
It's
very
difficult
to
define
the
state
of
system
at
a
whole,
a
complete
system
at
a
given
point
of
time
right
as
the
traffic
increases
as
the
network
traffic
increases
system,
state
changes
and
load,
a
lot
of
things
can
be
different
from
time
to
time
and
then,
when
something
goes
wrong,
your
redundancy
comes
into
the
picture
that
expect
most
of
the
time.
Yes,
it
works,
but
you
know
it
may
not
be
the
case
all
the
time
right.
So
that's
the
criticality
of
service
that
we
are
talking
about.
A
What
it
means
is
that,
most
of
the
time
they
are
reactive
when
problems
happen,
you
go
and
try
to
recover
them
right
and
they're,
not
collaborative
you.
Don't
have
systems
in
place
to
collaborate
of
failure
and
recovery
value
recovery.
Basically-
and
you
don't
have
this
failure
testings-
that
they
do,
they
may
be
ad
hoc
in
nature-
they're
not
well
integrated
into
your
cse
systems
right.
A
The
right
approach
is
what
we
call
as
take
the
chaos
first
approach
in
your
dev
of
devops
right.
What
is
chaos
first
approach?
Let's
talk
that
chaos
first
approach
is
to
introduce
chaos
engineering
as
a
tool
to
systematically
build
the
culture
of
you
know
improving
reliability
into
your
devops
right.
Reliability
is
not
one
person's
job;
it
is
everybody
in
everybody's
job.
A
Making
sure
that
reliability
is
always
there,
so
what
is
chaos?
Engineering?
Let's
talk
about
that
chaos.
Engineering
is
the
practice
of
breaking
things
on
purpose
before
they
happen,
you
break
things
and
then
verify
that
there
is
no
system
weakness.
Even
if
there
is
a
system
weakness
now,
you're
going
and
fixing
it
right.
So
you
start
small.
You
start
with
small
faults
on
the
system
in
various
different
layers
onto
various
different
systems
and
keep
proactively.
Introducing
these
faults.
A
All
right,
introduce
this
fault,
compare
it
with
the
result
with
the
expected
behavior
which,
if
it
is
same
then
you're
good
test.
Your
system
is
resilient
at
that
time
against
such
failure.
If
it
is
not,
then
you
have
got
something
to
learn
and
because
you
are
introducing
this
value,
you
are
well
prepared
to
deal
with
the
outcome
of
such
an
introduction
of
failure
and
you
can
either
reverse
it
quickly
or
automatically
recover.
A
It's
always
better
than
you
know,
a
fault
happening
without
your
knowledge
right
chaos,
engineering
is,
is
done
always
in
a
controlled
environment
and
you
will
have
the
ability
to
control
the
blast,
radius
and
you're
proactively,
verifying
your
system
against
such
faults,
one
by
one.
So
that's
chaos,
engineering
and
chaos.
Engineering
is
a
practice
that
is
now
coming
as
a
you
know,
effective
tool
in
cloud
native
to
improve
reliability
in
all
areas
of
devops
right.
A
So,
in
other
words,
chaos
engineering
has
to
become
a
kind
of
a
culture
in
devops
and
you
go
and
introduce
chaos,
tests
in
your
development,
ci
pipelines
or
in
your
delivery
mechanisms
in
cd
or
in
your
ops,
where
you
continuously
validate
your
slos
service
level,
objectives
again
such
failures,
and
then
you
keep
creating
some
random
tests
like
game
days
and
even
before
you
actually
get
into
production,
you
could
be
doing
a
failure.
Testing
chaos
testing,
you
know
very
effectively
before.
Actually
your
code
can
go
into
the
production
environment.
A
So
if
you
approach
the
failure,
testing
from
all
angles
with
good
tools,
then
chaos
engineering
becomes
very
effective
and
it
actually
aids.
You
know
retaining
the
reliability
even
when
your
systems
also
dynamic,
and
you
know,
and
running
at
high
scale,
so
who
can
use
chaos,
engineering
right
chaos.
Engineering
is
typically
thought
of
it's
a
supporting
practice
or
a
tool
for
sres
or
operation
systems,
but
it
is
now
increasingly
being
used
by
both.
A
You
know:
qa
teams,
as
well
as
the
developers
right
so
developers,
are
using
chaos,
engineering
practices
to
test
their
microservices
code.
You
know
in
in
various
different
environments
right
at
the
development
time.
Similarly,
qa
teams,
they
will
have
their
large
test
setups
and
they
can
now
test
against
various
failures
before
actually
qualifying
completely
right.
A
So
and
of
course,
sres
can
actually
use
chaos,
engineering
to
perform
annual
game
days
and
once
they're
comfortable,
they
go
to
the
automated
game
days
where
you
are
basically
introducing
calls
randomly
and
then
checking
your
system
if
it's
reliable
or
not.
So,
basically,
all
personas
can
be
involved
in
chaos,
engineering,
and
we
are
seeing
that
all
this
person
are
slowly
taking
else
engineering
as
a
as
a
common
practice
in
their
day-to-day
devops
job
all
right.
So
let
me
talk
a
little
bit
about
one
such
tool:
the
tool
that
we
work
on.
A
It's
litmus
chaos,
it's
a
complete
chaos,
engineering
platform
to
practice,
end-to-end
chaos,
experiments
and
the
post-effects
of
the
experiments
and
effectively
manage
the
entire
chaos
experience
on
kubernetes
right.
So
it
is
a
cncf
project
now
and
it's
been
there
in
use
and
development
for
about
five
years
now
very
stable.
We
have
done
more
than
50
enterprise
releases
and
it's
being
used
by
many
enterprise
users,
and
you
can
see
that
you
know
its
growth
is
increasing
according
by
quarter
basis.
A
So
it's
a
very
stable
project
and
feature
complete
for
your
basic
chaos.
Engineering
needs
in
cloud
native
right,
so
it
has
at
the
core
what
we
call
as
chaos
center,
which
is
a
centralized
platform
where
different
personnels
can
come
and
collaborate
on
chaos
and
that's
what
we
call
as
clear
as
control
plane
and
you
have
a
bunch
of
chaos,
experiments
that
are
stored
in
a
public
or
your
own
private
chaos.
A
You
know
infrastructure
or
systems
such
as
kubernetes
or
various
cloud
platforms
or
bare
metal,
or
you
know
your
on-prem,
vmware,
etc,
etc.
That's
that's,
how
litmus
structures
and
in
litmus,
as
I
just
said,
your
teams,
collaborate
around
chaos
tests
and
they
can
be
launched
in
various
ways
right.
You
can
send
them
as
a
trigger
to
a
certain
event
or
you
can
just
schedule
them
every
weekly
or
you
can
just
put
them
into
your
ci
or
cd
pipelines.
A
The
developers
use
litmus
chaos
workflows,
as
you
know,
as
a
way
they
do
the
other
program,
executions
using
cube,
ctr.
So
basically
a
developer
can
do
a
cube.
Ctl
apply
a
chaos
workflow
or
wherever
they
want
right.
So
that
gives
a
bigger
and
effective
capability
to
include
this
in
various
cacd
pipelines
or
in
the
unit
test
or
integration
test,
etc,
etc.
A
A
Results
are
pushed
back
into
chaos
center.
So
the
moment
you
introduce
this
chaos
stage
into
pipeline,
they
get
automatically
run
and
you
have
your
chaos
analytics
already
available
in
chaos
center,
and
you
can
compare
them
over
a
period
of
time
how
my
application
has
been
improving,
or
you
know
you
can
continuously
verify
that
there
is
no
actually
new
bug
that
you're
introducing
before
you
merge
your
code
into
your
repository.
A
A
So
where
do
you
start?
Is
another
question?
There's
no
specific
way
to
start
chaos
test.
You
can
start
in
pipelines
and
move
towards
right
right,
reproduction
production,
sometimes
people
start
in
reproduction.
You
know
if
there
is
an
sre
that
is
interested
in
chaos,
engineering.
They
will
bring
chaos
engineering
into
that
organization
and
then
start
introducing
failures
or
automating
those
failures,
tests
in
the
cd
and
then
ca
pipelines.
A
A
So
this
is
how
a
good
chaos
maturity
model
looks
like
in
reality
right.
So
you
always
start
on
basic
infrastructure,
introduce
failures
in
infrastructure
and
you
verify
your
code
or
your
service
is
reliable
against
such
values
and
it
could
be
a
various
combinations
of
failure,
even
at
the
infrastructure
and
slowly,
you
will
move
up
introducing
values
that
are
related
to
middle
layer,
which
has
message
queues
or
api
servers,
then
go
to
your
data
services
right
or
your
databases.
A
And
finally,
you
start
writing
chaos,
tests
that
are
specific
to
your
application
right
and
all
these
things
happen
over
a
period
of
time.
It
could
take.
You
know
a
few
quarters
to
sometimes
few
years
by
the
time
that
you
have
a
full-fledged
chaos
engineering
practice
where
you
have
all
sorts
of
chaos,
experiments
or
chaos
tests,
and
they
are
fully
automated
into
your
divorce
practices.
A
So
chaos
is
not
like
a
magic
bullet.
You
have
a
good
tool
to
start
off
with
and
it
has
to
primarily
become
a
culture.
That
is
when
you
have
a
good
way
to
measure
reliability
and
your
measuring
the
choice
of
our
chance
of
not
introducing
new
failures
into
your
software
before
they
get
shipped.
So,
just
to
summarize,
the
benefits
of
chaos.
Engineering
in
your
devops
introducing
chaos.
Engineering
in
your
devops
will
lead
to
three
major
benefits.
A
First,
you
will
see
the
ability
to
introduce
you
know
a
failure
quickly.
It's
called
mpta
mean
time
to
identify.
This
is
a
common
problem
in
most
of
the
systems.
You
know
that
a
failure
is
happening,
but
it's
very
difficult
to
reproduce.
When
you
have
chaos
engineering,
your
developers
are
well
versed
in
the
failure
practices.
There
are
a
bunch
of
chaos,
tests
that
are
already
available,
so
this
new
scenario
can
be
quickly
reconstructed
right.
A
So
your
time
to
identify
a
bug
or
a
scenario,
alternately
reduce
and
because
you
are
able
to
you
are
actually
reproducing
this
faults
are
introducing
these
faults
all
the
time
your
capability
to
recover
from
a
potential
outage
will
be
very,
very
high
and
the
recovery
times
will
become
slower
and
because
you
are
now
going
deep
and
debugging
them.
Your
development
systems
are
also
practicing
chaos.
Engineering
values
in
general
will
reduce,
and
that
increases
the
distance
between
the
failures
or
entities.
A
When
you
do
this
as
a
practice
are
guaranteeing
you
know
to
your
customers
that
the
service
is
reliable,
that's
because
you're
already
testing
all
the
time
against
potential
failures
in
various
areas.
Right
so
just
to
summarize,
reliability
should
be
a
devops
focus
in
cloud
native
and
introducing
chaos.
Engineering
into
devops,
your
devops
cloud
native
devops
will
result
in
high
roi
and
it
yields
structural
improvements.
A
It's
not
a
magic
bullet,
it
heals
structural
improvements
to
your
product
and
to
your
devops
practices,
and
also
your
teams
will
be
better
because
they
are
aware
of
the
product
working
whenever
values
do
happen,
so
they
have
debugged
the
system
many
many
times
whenever
these
values
are
introduced
and
weaknesses
are
found,
so
you're,
basically
having
a
much
better
engineering
team.
That
knows
your
product
very,
very
well
right.
A
So
the
popular
resilience
is
a
kind
of
a
structural
ladder
here
when
you
follow
chaos
and
streaming
as
a
practice
in
devops,
that's
mostly
about
what
I
want
to
talk
right
and
you
can
use
litmus
the
open
source
version
from
github.
It's
very
easy
to
start,
and
you
also
have
a
freely
available,
hosted
version
at
elect
musculosk.cloud.