►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Using GitOps for Kubernetes Reliability at Scale - Uma Mukkara, ChaosNative
GitOps is typically seen as a practice to get your things in order for infrastructure changes and infrastructure configuration management. However, there is another interesting use case for GitOps where it is used to chaos test the large scale deployments when changes happen to the application. Complete reliability as such is a complex subject and it requires carefully designed chaos engineering practices and SRE focus. In large scale systems where Chaos Engineering is scaled up, the chaos scenarios also become big enough to be maintained and applied. One solution is to maintain them and apply them through GitOps. In this lightning talk, I discuss a case study of how GitOps is used in a large customer environment to automate Chaos Engineering using LitmusChaos and FluxCD. The challenge of managing chaos experiments is presented when there are multiple team members and teams are involved. Then I discuss the schematic representation of how GitOps was structured between the application change and chaos experiments. Finally I discuss and summarise the GitOps best practices used to overcome these challenges.
A
Hello
and
welcome
to
github's
con
north
america
2021,
I'm
umam
mukhera
maintainer
of
litmus
chaos,
cncf
project
and
I'm
also
the
ceo
at
chaos
native
in
this
session.
The
goal
is
to
talk
about
the
role
of
gitobs
in
chaos,
engineering
and
how
we
can
use
gitops
to
scale
the
chaos
engineering
itself
in
production
or
tech
production
in
general.
The
idea
of
gitops
is
to
do
a
set
of
operations
where
now
there's
a
change
to
an
application
right.
You
do
some
set
of
id
operations.
A
You
can
use
the
same
principles
to
extend
whatever
you're
doing
to
do
the
validation,
testing
of
your
change
as
well,
and
you
can
do
chaos
engineering
whenever
there's
an
application
change
on
your
existing
deployments.
That's
the
idea
of
extending
it
ops
to
do
some
validation
testing.
More
importantly,
chaos
testing,
so
that
you
retain
the
reliability
of
your
system
whenever
there
is
a
change
or
you
may
even
improve
the
reliability
whenever
there's
a
change
to
your
application
or
the
service.
That's
the
idea
of
your
change
anyways.
A
A
Uncertainty
is
definitely
related
to
reliability,
so
chaos
engineering
is
helping
in
improving
your
reliability
or
reducing
the
uncertainty
it's
really
about.
If
there
is
a
fault,
is
there
uncertainty
about
the
service
being
continuously
available?
If
it
does,
if
it
continues
to
available,
then
there
is
no
uncertainty
and
it
is
reliable.
A
So
the
faults
do
happen
because
of
many
reasons
in
various
dependent
components.
So
your
end
service
really
depends
on
a
lot
of
other
things
and
a
fault
anywhere
can
cause
uncertainty
to
your
service.
So
if
the
uncertainty
increases
your
reliability
reduces,
if
you
need
to
increase
the
reliability,
you
need
to
reduce
uncertainty.
A
That
means
your
system
has
to
stay
up
against
a
lot
of
potential
failure
right.
So
it's
really
about
increasing
your
chances
of
uptime
against
certain
potential
and
use
chaos,
engineering
to
willfully
inject
these
values
and
keep
correcting
them.
So
that's
chaos,
engineering.
So,
in
summary,
why
will
you
invest
in
chaos?
Engineering
is
to
reduce
the
values
or
increase
the
time
between
the
values
and
to
decrease
the
recovery
time
or
mttr
mean
time
to
recover,
also
to
reduce
the
mean
time
to
identify
or
mean
time
to
inject
a
failure.
A
So,
in
practice,
what
you
realize
is
reduction
of
mtti
to
begin
with
and
the
more
capability
you
have
to
quickly
inject
the
fault.
You
will
be
able
to
do
something
about
recovery
rates
right,
so
you
increase
your
automated
scripts
or
do
something
new
to
recover
the
system.
Quick,
because
you're
able
to
recreate
the
fault
very
very
quickly
and
at
will
and
as
you
keep
fixing
more
bugs
because
of
this
false
inducing
down
times,
you
will
be
able
to
increase
the
time
between
the
failures
or
increase
the
uptime
that
is
empty
here.
A
So
this
is
the
end
goal
to
reduce
the
failures
or
increase
of
time
chaos.
Engineering
has
been
there
for
quite
some
time,
but
now
why
is
it
more
important
in
the
era
of
cloud
native
devops?
So
it's
primarily
because
your
code
is
shipping
faster,
maybe
10
times
faster,
and
there
are
many
more
components
to
deal
with.
A
So,
together
you
are
getting
more
dynamism
into
your
devops,
maybe
100
times
more
dynamism,
sometimes,
and
that
really
increases
the
chance
of
some
fault
happening
elsewhere
and
you
becoming
a
victim
of
that
vault.
So
the
burden
is
to
make
sure
that
your
service
is
up
and
running
or
uptime
continues
to
be
higher
right.
So
there
is
more
complexity
in
handling
the
situation
of
service
of
time.
A
Other
way
of
saying
this
is
your,
even
though
your
application
is
small,
the
uptime
of
your
service
depends
on
90
of
the
time
on
other
components,
so
you
have
to
be
worried
about
fault
scenarios
in
90
of
the
components
that
are
not
owned
by
you,
so
that
really
increases
the
dynamic
complexity
of
how
you
can
keep
your
uptime
high
or
reliability
high
right
chaos.
Engineering
is
being
used
as
a
way
to
deal
with
this
complex
situation,
and
definitely
it
is
on
the
rise
for
the
last
couple
of
years.
A
A
A
So
the
whole
idea
is
you
increase
the
chances
of
your
uptime
by
giving
shocks
wherever
there
is
a
possibility
shock
of
bringing
down
a
dependent
component
or
a
resource
within
your
application
right,
and
also
the
one
of
the
reasons
for
this
adoption
is.
There
are
more
tools
available
for
doing
chaos,
engineering
in
a
cloud
native
way
more
easier
way.
There
are
more
experiments
available.
A
You
can
get
started
with
chaos,
engineering
super
quick
right,
so
you
can
do
chaos
engineering
by
simply
picking
up
a
tool,
putting
the
experiments
together.
Sometimes
even
the
workflows
are
also
available
for
validating
your
service
against
a
failure
of
complex
systems
such
as
kubernetes
right,
and
you
will
continue
to
validate
the
hypothesis
of
your
system
against
such
failures
and
you
find
issues
and
then
improve
reliability.
A
Let
me
introduce
one
such
tool
to
do
easy
chaos,
engineering
in
cloud
native
that
is
litmus
I'm
a
maintainer
of
litmus.
We've
been
developing
this
project
for
four
years,
almost
four
years
now
and
it's
in
it's
got
a
great
community
of
users
around
it
contributors
around
it.
There
are
about
thousand
plus
possible
users
who
are
using
litmus
in
some
form,
and
it's
achieved
quite
a
bit
of
stability.
A
Recently
the
project
has
achieved
2.0
ga
and
that
really
tells
that
there
is
a
lot
of
usage,
validation
and
feedback
that
is
put
back
into
the
project
and
it
is
a
cncf
sandbox
project
on
its
way
to
get
incubation
status
sometime
in
the
near
future.
So,
let's
see
a
little
bit
more
detail
about
what
litmus
is
so
with
litmus
you
install
it
using
helm.
A
You
get
chaos
center,
it's
a
central
place
where
you
co-ordinate
or
collaborate
the
development
of
chaos,
workflows,
which
are
nothing
but
a
set
of
experiments
put
together
to
be
executed
in
certain
fashion.
So
your
team
members
need
not
be
about
ops
team.
It
can
also
be
about
qa
or
developer
team.
All
of
your
devops
personnel
can
interact
together
in
the
development
of
chaos,
experiments,
development
of
ks
workflows
and
put
them
in
chaos
hubs
and,
finally,
you
can
use
them
against
various
different
targets.
It
could
be.
A
It
need
not
be
just
about
kubernetes
targets,
it
could
be
about
bare
metal,
physical
resources
or
any
other
cloud
platform
such
as
aws,
azure
gcp,
and
you
can
also
target
virtual
machine
resources
such
as
vmware
resources.
So
litmus
is
end-to-end
capable
in
helping
you
practice,
chaos,
engineering
and
it
is
developed
to
be
used
by
teams.
A
Where
do
you
typically
use
litmus
and
what
are
some
of
the
use
cases?
As
I
said,
there
are
some
experiments
that
are
already
available
for
various
resources,
such
as
kubernetes
and
its
platform,
vmware
and
all
other
cloud
platforms,
and
you
use
these
experiments
to
create
litmus
workflows,
and
you
put
a
steady
state
hypothesis,
validation
scenario
using
litmus
probes.
This
is
one
of
a
powerful
feature
that
you
get
in
litmus
to
declaratively.
A
A
So
we
have
developed
litmus
probes
to
be
able
to
declaratively
manage
this
hypothesis
and,
as
far
as
the
use
case
is
concerned,
once
this
litmus
workflows
are
created,
you
can
use
them
either
to
do
slo
management
or
you
can
use
them
in
qa
for
doing
continuous
chaos,
testing
or
you
can
start
game
days
quickly
to
introduce
chaos,
culture
into
your
organization
or
even
to
enable
your
some
efficiency
in
your
observability
systems.
Right
against
failures.
A
A
How
do
you
use
chaos?
Engineering
in
general,
in
your
organization
or
in
your
devops,
is
first
by
introducing
chaos
itself.
There
is
a
lot
of
inertia
that
you
can
expect
when
you
introduce
chaos.
Engineering
in
your
arc,
primarily
because
system
will
be
going
down
initially
much
more
than
what
it
is
because
you
are
doing
willful
fault
injection,
but
you
start
by
injecting
in
some
minimum
form
using
game
days
or
some
experimental
sessions
in
your
qa
in
your
pre-fraud.
A
That's
that's
the
first
step.
Second
step
is
you
start
actually
developing
some
meaningful
chaos,
experiments
related
to
your
services
and
put
them
into
qa
or
pre-production,
and
once
you
achieve
that,
you
will
start
realizing
the
value
that
you
are
achieving
or
the
returns
are
going
to
be
much
higher.
A
You
also
execute
some
chaos
states
sometime
in
the
near
timeline
of
the
update
software
update
itself
in
a
random
way,
and
you
can
do
this
in
pre-production
or
in
prediction
also.
So
the
final
stage
of
chaos
maturity
model
is
you
have
a
system
where
you
are
doing
chaos
testing
whenever
there
is
a
change
introduced
and
validating?
A
If
there
is
a
degradation
in
reliability
or
at
least
maintaining
the
same
level
of
reliability,
if
not
improved
reliability,
so
git
ops
will
become
very,
very
important.
As
you
start
using
chaos,
engineering
at
scale
or
in
production,
it
has
to
be
guaranteed
that
that's
the
last
level
of
maturity.
Somebody
should
know
that
whenever
I
introduce
a
change,
there
is
going
to
be
a
fault
testing
in
pre-product
for
sure,
and
it's
going
to
be
random.
It
could
be
within
a
week
within
a
day
or
few
hours.
A
Whenever
the
app
change
is
introduced,
it
gets
deployed
through
githubs
and
you
use
the
same
githubs
concept
to
also
pull
in
the
chaos
test.
On
top
of
your
application
update
so
you're
guaranteeing
there
is
going
to
be
chaos
testing
after
your
change
is
pushed
into
production.
So
that's
a
definite
way
of
telling
your
team
that
there
is
going
to
be
chaos
testing,
and
there
is
somebody
that
is
going
to
look
at
the
reliability
benchmarks
after
the
changes
are
pushed
into
production.
A
How
do
you
pull
a
chaos
experiment
on
top
of
an
application
change
that
is
rolled
out
right?
So
there
are
two
parts
to
it.
One
is
about
how
your
chaos
experiments
are
being
managed
by
your
devops
teams
and
they're
generally
put
in
in
a
git
repository,
and
your
team
is
going
to
make
some
updates
to
the
chaos
experiments
and
whenever
there
is
a
change
to
the
chaos,
experiments
or
execution
of
that,
you
can
actually
see
them
in
the
ui
tools.
So
that
is
one
side
of
the
story.
A
A
To
configure
a
github
repository
in
the
configuration
database,
we
will
try
to
execute
some
chaos,
workflow
and
the
details
of
which
will
be
returned
to
the
backend
github
in
the
second
demo,
which
is
to
show
that
the
chaos
workflows
can
be
executed
based
on
a
event
or
to
a
deployment
or
to
the
system
in
the
back
end
right.
So
this
event
can
be
introduced
through
githubs.
A
That's
how
generally
it
is
done,
configuration
updates
are
rolled
out
through
githubs
onto
kubernetes
systems
and
in
our
demo
we
are
going
to
introduce
a
manual
change
to
a
deployment
which
otherwise
would
be
done
by
githubs,
and
when
we
do
the
change
to
the
deployment,
you
will
see
a
litmus
chaos.
Workflow
will
start
executing.
A
This
is
litmus
chaos
center.
Our
objective
is
to
create
a
git
up
based
backend,
for
the
configuration
store,
and
this
is
how
it's
deployed
it's
all
running
and
good,
and
the
back
end
store
is
right
now
by
default.
Mongodb-
and
you
see
some
even
tracker
as
well.
That's
used
for
backhand
guitars
and
we
have
one
agent
that
is
connected,
so
this
litmus
center
looks
all
healthy,
let's
go
and
do
some
deploy
keys
to
github.
So
this
is
where
you
do
in
settings.
A
A
A
A
A
A
A
A
So
event,
tracker
listens
to
certain
event,
tracker
policies
and
based
on
any
change
it
triggers
a
policy
so
looks
like
even
tracker
is
looking
good,
it's
listening,
so
we
just
need
to
create
certain
policies
and
let's
look
at
how
the
policies
look
like.
So
this
is
a
kind
event
tracker
policy,
it's
based
on
the
conditions
and
k
value
operator.
A
A
A
A
A
A
A
It's
gonna
look
good.
There
you
go
so
a
new
workflow
is
now
run
because
of
a
change
in
the
deployment
of
the
application.
The
same
engine
export
delete,
workflow
is
being
run
through
a
triggering
of
an
event.
So
that's
our
back-end
githubs
demo,
I'd
like
to
summarize
this
session.
By
saying
you
can
adopt
git
ops
for
chaos,
engineering
to
automatically
run
workflows
and
also
to
consider
git
as
a
back
end
for
your
configuration
store
of
your
chaos
database
and
it's
always
good
to
have
a
guarantee
of
random
fault
injection.
A
So
with
that,
I
would
like
to
thank
you
and
you
would
be
able
to
get
more
resources
about
litmus
and
githubs
at
our
docs.
You
can
join
us
on
the
litmus
channel
on
kubernetes
slack.
Thank
you
and
you
can
reach
me
at
my
twitter
handle
uma
undiscovered
mcgraw.
Thank
you.
Have
a
great
top
scone
as
well
as
kubecon
folks,
bye.