►
From YouTube: Chaos Engineering Applied To the FinTech Domain - Rajeshwar Vadheraju, FIS & Neelanjan Manna
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Chaos Engineering Applied To the FinTech Domain - Rajeshwar Vadheraju, FIS & Neelanjan Manna, Harness
Speakers: Rajeshwar Vadheraju, Neelanjan Manna
FIS, a Fintech company with more than 20,000 clients around the globe, offers Banking-as-a-Service Hub, which enables banks and corporations the ability to rapidly configure new financial services. The Delivery of “as-a-service” features across accounts, cards, and establishments is enabled by functional modules deployed on Kubernetes, which are used by thousands of customers each day. To bolster the resiliency of this critical infrastructure, FIS uses LitmusChaos to expose and help remediate the system flaws thereby ensuring highly available services for the customers. In this talk, Rajeshwar (FIS) & Neelanjan (Harness) will lay out the reliability challenges while delivering Banking-as-an-Service and demonstrate how chaos experimentation was leveraged as part of the organization’s “client-experience-year” initiatives to improve the banking APIs.
A
B
Hey
everyone
Thanks
for
attending
this
session,
a
software
engineer
at
hardness
and
a
core
contributor
to
the
cncf
litmus
chaos
project,
where
I'm
responsible
for
developing
chaos,
experiments
for
various
types
of
failure,
scenarios
and
the
requisite
tools
required
for
injecting
chaos
into
various
environments.
I'm
really
excited
to
be
here
with
all
of
you
over
to
you,
Raj.
A
All
right,
so
today
we
are
going
to
talk
about
chaos,
engineering
applied
to
the
fintech
domain.
Before
we
jumping
into
cassinating,
let's
spend
some
time
on
fintech
domain.
What
it
is
here
goes
a
problem
statement,
non-banking
services
companies
would
like
to
offer
banking
services
and
products
example
could
be
a
cab
company
such
as
Uber,
which
provides
digital
platform
tools.
Drivers
and
passengers
would
like
to
offer
auto
loan.
A
Why
would
they
do
that?
And
for
obvious
reasons
they
want
to
increase
their
revenue
streams
and
also
they
want
to
serve
their
loyal
customers,
make
them
happy
by
providing
some
competitive.
You
know
interest
rates,
so
if
so,
why
they
have
to
look
at
some
fintech
providers.
Why
can't
they
do
themselves
again?
For
some?
Obviously,
just
you
know.
Ubers
primary
business
is
not
banking,
a
b.
A
They
don't
have
banking
because
of
that
they
don't
have
banking
ecosystem
and
they
don't
have
banking
license
and
they
had
to
go
through.
Even
they
do,
they
would
go
through
a
lot
of
complaints
and
regulations
of
banking
sector,
which
is
too
much
for
them.
This
is
where
FIS
would
help
the
fintech.
Companies
are
pretty
consumers
where
it
exposes
banking
services
in
the
form
of
apis,
and
these
fintech
consumers
consume
those
apis
and
build
some
Innovative
products
provide
great
customer
experience
to
their
end.
Users
is
a
win-win
for
everyone.
A
A
So
the
complexity
of
this
architecture
is
not
intentional.
It
is
evolved
over
the
period
of
time,
just
like
any
other
organization.
Fis
architecture
is
also
evolved
for
the
last
three
decades
or
so.
If
I
see
Building
Products
in
the
financial
services,
it
build
it's
its
products
on
all
sorts
of
Technologies,
like
Mainframe
Windows
based
web
applications.
A
So
in
this
context
you
have
to
understand-
or
this
is
a
classic
distributed
systems-
architecture
out
there
and
distributed
systems-
are
a
picture,
brings
its
own
challenges.
The
challenges
around
resiliency,
where
things
could
go
wrong
in
anywhere
in
the
system.
What
do
I
mean
by
things
could
go
wrong.
Like
you
know,
there
could
be
a
network
partition
that
could
happen
between
the
systems.
There
could
be
a
service
disruption.
There
could
be
a
fault
that
could
be
happening.
A
This
is
where
chaos,
engineering
discipline,
could
help
identify
those
issues
and
resolve
them,
thus
providing
Superior
customer
experience.
So
what
is
chaos?
Engineering
I
mean
it's
a
disciplinary
practice
experimenting
on
a
system
in
order
to
find
how
resilient
it
is,
meaning
whether
the
application
can
withstand
with
the
turbulent
or
faulty
conditions.
A
A
And
what,
if,
if
the,
if
the,
if
there's
a
latency
on
the
downstream
system,
how
the
Upstream
system
is
impacted
right,
you
don't
understand
how
resilient
your
application
is
and
overall
stack
is
just
like
any
other
engineering
practice.
Chaos,
engineering
practice.
As
you
know,
three
pillars
depicted
here.
You
have
people,
you
have
process
and
you
have
ecosystem.
A
So
next
few
minutes,
I'm
gonna,
double
click
on
ecosystem
and
a
little
bit
of
one
process
and
explain
to
you
what
the
journey
that
FIS
went
through
and
the
third
process
that
we
went
through
and
how
we
put
together
this
ecosystem,
because
it's
not
just
a
chaos
tool.
That's
good
enough
to
implement
chaos,
engineering
practice,
but
you
need
to
Envision
and
think
about
ecosystem.
A
Find
the
toil
within
the
within
the
chaos
engineering
practice
and
automate
it
and
make
it
reputable
process,
and
second,
we
need
to
scale
it
so
why
we
need
to
scale
it.
We
need
to
scale
it
because
organizations
like
FIS
has
multiple
hundred
products
and
we
need
to
find
build
an
ecosystem
and
tools
and
the
processes
so
that
we
and
we
build
this
consistently
and
in
a
reputable
way
so
that
we
can
apply
this
across
multiple
multiple
products.
A
That's
how
we
scale
it
and
second,
we
want
to
automate
it
so
that
we
can
Implement
chaos,
engineering
for
every
release,
that's
going
into
production
because
chaos
itself
brings
in
some
additional
work.
So
we
want
to
identify
the
toil
around
that
automated
so
that
it
is
easy,
consistent
and
repeatable
process
for
for
the
application
or
application
teams.
So,
as
you
can
see,
there
are
four
or
five
pieces
in
this
ecosystem,
one
on
the
top.
You
see
a
CD
pipeline.
You
have
application
under
test
here.
A
A
Precursor
to
the
chaos
test
is
a
generating
load
on
an
application
under
test
and
put
it
through
some
load.
Why
we
want
to
do
that
because
most
of
the
issues
that
happen
in
production
happen
under
load.
That's
why
we
want
to
generate
some
load
and
the
application
that
we
are
injecting
chaos
into.
That's
where
the
load
generators
would
help.
Okay,
that's
great!
You
have
the
load
generator
generating
the
load
on
the
application,
and
you
have
chaos
tool,
injecting
chaos
through
its
agent
into
the
application.
A
But
we
want
to
monitor
the
health
of
the
application
when
we
are
injecting
the
the
chaos.
What
do
we
want
to
monitor?
You
have
several
metrics
up
and
down
the
stack
right.
You
have
a
matrix
at
the
application,
Level
your
Matrix
at
the
process
level
and
your
Matrix
are
the
host
level
such
as
you
know,
response
time,
throughput
error
rate
at
the
application,
Level
your
process,
CPU
process,
memory
utilization.
If
it's
a
Java
jvm
based
application
or
JBoss,
you
have
thread,
pool
connection
pools.
A
A
A
Now,
when
you
execute
a
chaos
test,
you
tell
Captain,
hey
Captain,
I
executed
this
chaos
test
for
this
period,
go
evaluate
for
this
five
minutes
or
this
10
minutes,
and
then
it
goes
pulls
that
slice
slos
pulls
the
metrics
from
the
Dyna
Trace
evaluate
and
give
you
gives
you
hey.
You
know
for
this
test,
you
you
said
you
wanted
a
10
10
error
rate
is
acceptable
because
you
introduce
chaos,
but
I
saw
like
a
30
percent
error
rate.
That
means
your
chaos
just
failed.
Then
you
further
dig
into
it
to
understand.
A
Why
that
why
why
there
are
so
many
errors,
whether
you're
introducing
a
latency
in
one
API
caused
errors
in
other
API,
there's
further
evaluation,
triaging
that
you
have
to
do
so,
but
what
Captain
can
help
you
with
is
can
help
you
with
avail
automating
that
evaluation
process,
that's
a
one
toil
that
you
have
that
is
eliminated
now.
This
is
all
good,
and
if
you
want
to
automate
this
further,
that's
where
it
says
CI
CD
would
help
you
what
cacd
can
help.
A
You
is
help
you
defining
a
workflow
around
chaos,
engineering
ecosystem
and
help
you
trigger
sellout
generator
through
the
through
Jenkins
cicd
Pipeline,
and
then
trigger
chaos,
experiments
and
finally
trigger
evaluations
that
will
talk
to
Captain,
which
will
give
you
a
Boolean
value
like
pass
or
fail,
using
which
you
can
make
a
decision,
whether
you
move
this
software
to
the
next
environment
or
production
or
wherever
you
want
to
or
not.
So
this
is
the
kind
of
ecosystem
that
we
have
built
to
a
automate.
A
B
Litmus
is
a
tool
set
to
do
Cloud
native
chaos,
engineering.
It
helps
both
developers
and
sres
to
automate
the
chaos
experiments
at
different
stages
within
the
devops
pipeline,
like
development
during
CI
CD
and
in
production,
which
leads
to
increased
resiliency
of
the
system.
It
adopts
a
kubernetes
native
approach
to
define
the
chaos
intent
in
a
declarative
manner
via
custom
resources.
It
uses
the
operator
pattern
and
relies
on
custom
resource
definitions
to
define
the
experiments
on
top
of
that.
B
Fitness
provides
chaos
into
dashboard
where
one
can
orchestrate
and
visualize
the
results
and
metrics
of
the
conducted
chaos
experiments
as
well
as
litmus
API
for
programmatically,
injecting
and
managing
the
chaos
from
a
top
level.
Litmus
chaos
uses
CRS
to
define
the
chaos
intent
and
manages
the
chaos
orchestration
via
operators.
B
There
are
different
CRS
such
as
chaos,
experiment,
chaos,
engine
and
Chaos
result,
while
a
key
also
operator
that
reconciles
the
chaos
engine
CR
in
the
course
of
a
chaos.
Experiment
execution,
given
that
a
service
account
with
sufficient
R
back
permissions
for
the
experiment
has
been
already
defined.
B
One
needs
to
define
the
chaos
experiment
CR
manifest
for
specifying
low
level
experiment
attributes
such
as
experiment,
tunables,
container
images,
Etc
and
the
chaos
Engine
That
Binds,
the
experiment
with
an
application
instance
as
well
as
defines
how
to
perform
the
chaos
experiment
in
terms
of
mounting
volumes
for
the
experiment,
parts
or
overriding
experiment,
tunables
retaining
or
deleting
experiment,
Parts,
post,
the
chaos
and
so
on.
Upon
the
creation
of
both
these
resources.
B
Chaos
operator
reconciles
the
chaos
engine
CR
to
create
the
chaos
Runner,
which
consumes
secure's
experiment,
CR
data
for
experiment
creation
and
then
creates
the
kubernetes
jobs
which
creates
the
requisite
experiment,
parts
for
running
the
experiment,
logic.
In
the
end,
the
job
also
updates
the
kiosk
result.
Cr,
which
summarizes
the
result
of
experiment,
runs
as
well
as
updates
the
kiosk
engine
to
bring
the
chaos
execution
to
an
end.
B
The
unique
value
proposition
offered
by
litmus
includes,
first
of
all,
being
Cloud
native
chaos.
Experiments
that
allow
you
to
validate
your
entire
application
stack,
beat
Cloud
native
kubernetes
resources
or
Cloud
infrastructure,
or
even
Legacy
infrastructure,
with
the
broad
range
of
experiments
that
it
that
is
offered
by
litmus
second,
being
least
privileged
principle.
Chaos
injection,
which
allows
for
chaos
Engineering
in
security,
sensitive
environments
using
granular,
R
back
over
individual
experiments
and
use
of
just-in-time
execution
of
privileged
containers
that
limit
the
abuse
and
misuse
of
Target
environments.
B
Sixth,
it
provides
declarative
custom
tasks
which
AIDS
at
partly
running
the
tasks
that
simulate
real
life
conditions
for
the
chaos
execution
so,
for
an
example,
running
a
load
generator
to
simulate
a
network
traffic
is
what
we
can
do
here.
Finally,
the
last
one:
quantification
of
system
resiliency
by
use
of
weighted
experiments
and
probe
success
criteria
to
quantify
the
result
of
your
system
as
a
metric
score.
B
A
A
Litmus
offers
wide
variety
of
experiments
here.
I
took
three
experiments
as
a
sample
and
I
want
to
introduce
how
we
measured
when
we
execute
this
experiment.
For
example,
when
we
execute
pod
HTTP
latency,
we
measured
third
pool
connection,
pool
utilization,
error
rate
and
throughput
Y,
so
again
back
to
that
microservice,
A
and
microservice
B
example.
If
you
introduce
latency
between
A
and
B,
we
want
to
understand
how
is
impacted.
The
resources
on
a
is
impacted,
pod
memory
hack,
if
you
introduce
a
memory,
saturation
type
of
event
and
a
micro
services
are
kubernetes
architecture.
A
The
part
is
killed
with
umkill
event.
We
want
to
understand
whether
that
the
APM
systems
are
alerted,
because
of
that
event,
is
a
service
response
time
of
apis
of
the
exposure
on
that
part
and
as
well
as
does
the
part
memory
hog
on
a
particular
part
or
a
boom
kill
event
of
a
particular
part
caused
any
wider
impact
right.
That's
how
it
helps
you
with
the
broader
stability,
resiliency
issue
and
understanding
the
stability
resiliency
of
your
application,
HTTP
status
code.
A
Foreign,
so
earlier
I
talked
about
Captain,
doing
a
chaos
Evolution.
So
how
about
the
example
of
a
chaos
Evolution
here
the
way
we
envision
chaos
Evolution
as
a
kind
of
three-phase
manner
your
pre-phase
Precast
phase
during
chaos,
phase
and
post
chaos,
phase
PKS
phase,
you
can
think
of
a
like
a
steady
state.
A
During
the
steady
state
you
may
Define
the
metrics
that
you
want
to
measure
and
you
you
define
the
thresholds.
Example
you
want
to
measure
throughput
error
rate
response
time,
CPU
memory,
connection,
pool
utilization,
third,
pull
utilization,
say,
for
example,
for
response
time.
You
want
three
seconds
through
put
100
transaction
per
second,
less
than
two
percent,
and
now
we
executed
the
test
study.
State
five
minutes
as
depicted
here
right
you
give
the
captain
saying:
hey
Captain
here
is
my
SLL
SLI
definition
and
here's
my
five
minute
window
I
executed
the
test
go
evaluate
it.
A
The
captains
comes
back,
saying
that
it's
all
looking
good
green,
steady
state
is
good.
Now
you
introduced
chaos
example
and
network
latency.
Now,
typically,
when
you
introduce
latency,
the
throughput
will
go
down.
The
error
rate
will
go
up,
but
your
hypothesis
is.
There
are
error
rate
could
be
impacted,
only
one
API,
where
you
introduce
the
latency
and
it
could,
you
know,
cause
maybe
five
percent
error
rate.
A
You
feed
that
your
slis,
that
you
want
evaluate
and
their
objectives
are
thresholds
again
in
the
CML
file
and
feed
it
to
Captain
and
kept
until
kept
and
hey
Captain
I
executed
this
chaos.
This
is
my
chaos.
Phase
is
between
10th
minute
and
15th
minute,
and
here
is
my
SLI.
Slr
definition
for
this
go
evaluate.
A
A
The
connection
pool
utilization,
you
know,
went
up
during
the
chaos,
but
it's
saturated
you're
expecting
it
to
go
out
to
maybe
50,
but
it
may
went
up
to
100
right
a
CPU
you're
expecting
to
go
up
like
five
percent,
but
it
went
up
to
50
percent.
So
those
are
the
things
that
Captain
can
automatically
evaluate
and
give
you
kind
of
some
sort
of
a
heat
map
and
pass
fail,
result
and
post
chaos.
What
you
need
to
do
is,
which
is
you
define
your
metrics
very
similar
to
your
study?
A
State,
the
post
phase
is
more
of
a
recovery
phase
right
where
you
remove
your
experiment
at
the
15th
minute
and
15
to
20
minutes.
You
want
to
understand
the
application.
Metrics
are
back
to
normal
right.
So
then,
your
slis
slos
are
very
similar
to
what
you
have
in
study
State,
and
then
you
give
the
captain
saying:
hey
Captain,
go
X,
go
evaluate
my
post
phase.
These
are
my
slos
slis
and
evaluated
and
kept
in
evalues
and
comes
back.
It's
all
looking
within
the
bonds
pass.
Otherwise
it
Market
fail.
A
So
that's
the
kind
of
a
power,
the
automation
that
Captain
brings
you
in
evaluating
your
chaos
experiment.
So
finally,
I
I
conclude
this
talk
here.
Finally,
I
will
leave
this
session
with
a
thought
that
usually,
the
chaos
in
leading
is
looked
at
either
more
of
a
SRE
problem,
but
it
can
help
you
validate
your
architecture
and
also
helps
you
with
the
you
know.
Measuring
business
metrics
thus
help
the
business
teams
ultimately
providing
greater
customer
experience
than
product
quality
and
also
validation.