►
From YouTube: Case Study : Improving Resilience of Applications in Telco Environme... Uma Mukkara & Samar Sidharth
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Case Study : Improving Resilience of Applications in Telco Environments - Uma Mukkara, ChaosNative & Samar Sidharth, Orange
Though Kubernetes acts as a common abstraction and as a platform to build other platforms or applications, challenges are unique to each environment or area of operations. Telcos operate at a larger scale compared to the other environments. In these environments, the challenges of moving the applications to cloud native include a plan on the reliability of the whole service right from the beginning to the state of operations. In this session, we discuss a case study from Orange, the telecommunications operator, where they have been using Litmus, Kubernetes and other CNCF technologies to build and scale their application development process all the way till shipping them. We discuss their resilience goals of the cloud native application, scale factor, original challenges, impediments, the approach and the actual design along with the learnings in this case study. The session will wrap up with the key takeaways around resilience building technicals for large scale applications.
A
A
A
The
reason
why
these
challenges
occur
is
there
is
proliferation
of
micro
services.
There
are
too
many
of
them
and
they
are
shipping
quite
fast.
They
come
to
your
environment
faster
than
you
would
expect
so
this,
even
though
they
are
all
individually,
reliable,
well
tested
when
you
all
put
together
this
micro
services
to
form
your
application
service,
the
dependency
matrix
increases
a
lot,
and
any
fault
anywhere
really
means
that
you
may
have
a
problem
of
availability
in
your
service.
A
So
you
need
to
be
able
to
be
resilient
to
all
these
faulty
scenarios.
That's
really
the
reliability
challenge
and
the
solution
to
this
bigger
problem
is
adopt.
Chaos,
engineering
and
chaos.
Engineering
has
been
on
the
rise
for
the
last
couple
of
years.
We
are
seeing
a
lot
of
people
using
chaos,
engineering
or
project.
Litmus
is
an
evidence
to
that,
and
we
are
expecting
that
chaos.
Engineering
will
be
a
mainstream
solution
or
a
tool
set
in
the
very
near
future
and
chaos.
A
Engineering
is
being
adopted
for
the
overall
devops,
not
just
for
ops,
which
is
typically
the
case
in
the
last
decade
or
so.
But
now
we
see
chaos
engineering
being
used
both
in
pipelines,
qa
environments,
reliability,
testing,
test
beds,
all
that
stuff
performance
engineering,
so
chaos
engineering
is
emerging
as
a
greater
tool
set
for
developers
and
devops
in
general.
A
You
can
use
them
and
put
them
to
work
and
reduce
the
possibility
of
down
times
at
the
production
site
right
and
how
you
do.
This
is
put
the
chaos
experiments
together
in
various
forms
chain.
Them
build
steady
state
hypothesis,
closer
to
your
reality
and
go
deeper,
improvise
your
steady
state
hypothesis,
checks
and
and
then
start
fixing
your
either
configuration
issues
or
software
books
or
infrastructure.
A
Tuning
to
your
service
to
be
highly
available
and
litmus
is
born
out
of
clearly
such
a
need
and
it's
an
open
source
project
which
is
part
of
cncf
for
more
than
a
year
now,
and
we
have
developed
a
litmus
about
three
years
ago
and
donated
it
to
cncf
sandbox
project
last
year.
We're
doing
the
incubation
process
right
now,
hopefully,
will
be
an
incubation
project
in
the
near
future.
A
It
has
got
a
quite
a
good
adoption
with
a
thousand
plus
users
and
it's
a
pretty
stable
platform
with
2.0
as
the
latest
release,
and
we
are
seeing
quite
a
good
installation
rate
of
about
a
thousand
installations
per
day.
It's
a
complete
tool
set
for
anyone
to
do
chaos,
engineering
and
it
is
a
kubernetes
application
and
it
can
run
both
cognitives
targets,
attacks
against
cognitive
targets
and
non-cooperatives
as
well.
A
So,
together
you
develop
a
chaos
workflow,
which
is
nothing
but
a
chaos
scenario,
and
you
can
target
them
onto
your
kubernetes
resources,
kubernetes
platforms,
app
kubernetes
applications
and
also
a
bunch
of
non-kubernetes
use
cases
as
well.
It
could
be
any
cloud
platforms
or
vmware
or
bare
metal
physical
infrastructure
as
well.
So
where
do
you
use
litmus?
A
Litmus
first
of
all
has
got
readily
available
chaos,
experiments
to
a
large
extent
and
for
both
kubernetes,
as
well
as
for
non-cooperative
such
as
vmware
and
other
cloud
platforms.
So
you
just
need
to
create
chaos
scenarios
you
don't
need
to
start
from,
like
writing.
Chaos,
experiments
for
the
basic
scenarios,
so
what
you
would
be
doing
is
really
construct
this
chaos
scenarios
called
litmus
workflows
and
litmus
also
comes
with
a
powerful
feature
called
probes,
which
is
to
help
you
in
creating
a
steady
state
hypothesis
logic.
You
introduce
a
fault.
A
You
want
to
know
whether
my
system
is
still
working
as
expected
or
not
that's
a
difficult
way
to
express.
Is
my
system
working
properly
or
not?
So
litmus
probes
really
come
here
to
help
you
to
define
exactly
that
problem?
Declaratively,
you
can
tune
it.
You
can
get
it
as
close
to,
as
you
would
generally
describe
what
your
steady
state
hypothesis
is,
and
then
you
can
use
this
entire
end-to-end
chaos,
engineering
idea
on
for
multiple
use
cases.
A
So
how
do
you
get
started?
As
I
said,
we
have
got
a
bunch
of
experiments
already
available.
These
are
like
lego
blocks.
You
just
need
to
put
them
together
and
you
install
litmus
through
helm
and
you
get
chaos
center
or
you
start
running
your
basic
workflow,
invite
your
team
members
and
attach
it
to
your
prometheus
grafana
monitoring
system
and
everything
is
in
place.
So
it's
fairly
easy
to
start.
At
the
same
time,
it's
highly
scalable,
a
powerful
sdk
is
there
and
you
can
go
very
deep,
describing
your
complex
hall
scenarios.
A
So
the
solution
is
to
really
apply
litmus
into
that
kubernetes
system
and
keep
executing
various
different
experiments
and
continue
to
verify
your
openstack
continues
to
run
and
kubernetes
kubernetes
is
behaving
as
expected.
So
with
that,
let's
welcome
summer,
who
will
be
talking
about
violetness
and
how
and
he'll
be
taking
us
through
a
quick
demo.
B
B
B
Now,
let's
talk
about,
why
do
we
need
chaos
and
resiliency
testing,
as
we
all
know,
that
kubernetes
is
a
dynamic
and
a
complex
system
and
there
are
a
lot
of
activities
happening
under
the
hood,
which
means
kubernetes
component
can
interact
in
a
number
of
unpredictable
ways,
causing
emergent
behavior
as
deployment
grows
in
size.
So
does
the
number
of
possible
interaction
between
these
components
and
with
the
traditional
testing?
These
scenarios
are
hard
to
uncover
in
the
real
world
scenarios.
We
have
resources
that
are
customized
and
required
focus
testing
to
cover
these
scenarios.
B
Additionally,
litmus
has
a
pro
feature
that
enables
to
run
customized
validation
and
it
is
highly
configurable
wherein
we
can
configure
the
time
between
the
two
probe
executions
and
the
particular
instance
where
we
want
to
execute
the
probe
like
at
the
start
of
the
kiosk
injection
or
during
the
chaos
injection
or
towards
the
end
of
the
cause
injection.
This
gives
the
required
flexibility
to
run
the
validations.
B
B
B
It
also
offers
multiple
drivers
for
writing
test
cases
like
python
unit,
test,
bash
and
robot
framework,
so
towards
the
bottom
of
the
screen.
You
see
a
sample
output
from
the
cross
testing
processing
test
case,
which
includes
list
of
test
cases
along
with
projects
to
which
these
test
cases
belong,
and
the
tier
of
the
test
cases
like
healthcare.
B
B
B
Next,
we
move
on
to
the
chaos
injection
phase,
wherein
we
inject
the
kiosk
variant
to
the
identified
application
and
also
run
the
on
kiosk
probe.
So,
basically,
we
are
going
to
check
the
functionality
of
the
application
using
litmus
probe
at
the
time
when
the
application
is
under
stress
and,
finally,
the
post
validation
step.
So
apart
from
the
validating
the
final
state
of
the
application,
it
should
be
up
and
running
as
it
was
during
the
start
of
the
experiment.
B
You
can
also
have
some
additional
post
validation
probes
that
can
perform
the
custom
post
check
for
the
application,
along
with
the
other
actions
like
clean
up
of
the
resources
that
were
created
during
the
pre-validation
steps
coming
to
the
use
cases
resiliency
realizes
the
motto
of
this
cube.
Con
and
as
you
see
that
resiliency
is
also
at
the
center
of
our
use
cases,
we
can
utilize
the
open
source
tools
for
building
chaos
and
resilience
test
cases
around
kts
workload
and
infra.
B
B
B
B
B
So
first
is
the
unsealed
vault
probe,
which
is
running
at
the
eot.
That
is
end
of
the
test
and
the
second
probe
is
check
front-end
access.
So
this
is
basically
checksystem
checking
the
excess
of
the
vault
end
point,
and
this
is
running
in
the
mode
edge
so
which
means
that
it
would
run
at
the
starting
of
the
experiment
and
towards
the
end
of
the
experiment
and,
as
you
can
see,
there
are
different
parameters
which
are
such
exclamatory,
which
covers
the
gap
between
the
two
probes
or
the
timeout
parameter
for
the
probes.
B
B
B
The
final
verdict
is
passed
and
we
can
also
see
the
status
of
different
probes
and
pros
pass
through
pass
percentage
as
well.
So
this
is
hundred
percent
and
status
of
different.
Close
is
also
good.
So,
since
we
have
this
check
your
front-end
url
probe
as
edge,
so
it
was
first
checked
at
the
beginning
of
the
experiment
and
towards
the
end,
and
the
unsealed
part
was
only
towards
the
end.
So
it
is
the
post
chaos
probe.
B
All
the
all
the
word
parts
go
into
a
serial
state.
So
in
this
case
we
will
see
a
failure
scenario
where
all
the
parts
get
sealed
and
the
url
validation
will
be
filled
towards
the
end.
So
that
is
expected,
and
for
this
reason
we
have
configured
the
vault
probe
to
proceed
on
failure,
so
we'll
still
continue
with
the
experiment
and
unfill
the
wall
towards
the
end,
rather
than
failing
and
exiting
so.
This
is
also
an
option.
B
In
this
case,
the
only
difference
that
we
can
see
is
the
check
front-end
access.
Url
is
in
continuous
mode,
which
means
that
it
would
be
tested
from
starting
till
end.
So
basically,
it
would
check
at
the
start
of
the
experiment
if
it
is
accessible
and
during
the
chaos
injection
phase
and
towards
the
end
so
towards
the
end,
it
would
fail
since
we'll
be
deleting
all
the
parts
sequentially.
B
So
I
have
intentionally
added
a
30
second
delay
between
consecutive
gas
injections
so
that
the
port
can
recover
so
yeah.
So
now
the
connection
has
lost
as
soon
as
we
deleted
the
vault
zero
part.
So
now
the
pro
has
started
to
fail,
and
finally
it
will
mark
this
probe
as
fail
and
proceed
with
the
final
proof
execution
that
is
unsealing
of
the
vault.
B
B
B
B
B
Here
we
have
three
probes
that
are
running
out,
of
which
two
are
running
on
cures
and
the
last
one
is
running
towards
the
end
of
the
test
case.
So
the
first
on
chaos
probe
is
create
resources
which
is
basically
creating
open,
stack
resources
at
the
time
of
chaos
injection
and
similarly,
we
have
checkping,
which
is
checking
the
reachability
of
the
vm
created
during
the
create
resource.
B
B
B
B
So
we
have
kept
multiple
iteration
of
this
cross
injection
for
nova
scheduler
pod,
so
that
we
make
sure
that
it
is
overlapping
with
the
on
kiosk
probe.
So
we
want
to
make
sure
that
the
resources
that
are
being
created
is
overlapping
with
the
kiosk
injection
time,
so
that
we
can
actually
test
the
availability
of
the
services
toward
to
the
end
user
if
there
is
any
degradation
of
the
services
or
any
impact
in
creation
of
the
openstack
resources.
B
B
This
is
a
sample
output
of
the
cross.
Testing
based
group
containers,
so
resource
validator
is
basically
the
one
that
created
the
resources.
Checkping
was
a
connectivity
check.
Resource
deletion
was
the
cleanup
part,
and
this
is
a
another
probe
with
which
we
used
to
run
the
in
simple
rules
and
take
corrective
actions
during
the
chaos
injection
phase
or
post-cross-injection
phase.
B
A
So
the
summary
is,
you
will
be
able
to
do
deep
chaos
and
you
should
be
doing
deep
chaos
in
an
operating
environment
to
validate
the
functioning
of
your
application.
The
functioning
of
your
kubernetes
and
summer
and
his
team
at
orange
is
able
to
do
that
successfully
and
I
hope
you'll
be
able
to
do
something
similar
if
your
needs
are
similar
use
litmus
for
your
reliability
needs.