►
From YouTube: The Mushroom Cloud Effect or What Happens When Containers Fail? by Alois Mayr, Dynatrace
Description
The Mushroom Cloud Effect or What Happens When Containers Fail? - Alois Mayr, Dynatrace
"Micro service architectures result in up to 20 times larger environments than their monolithic counterparts. In such big and interconnected environments container metrics will tell you about infrastructure health but not service health. Even if you have implemented service health checks to quickly react on service failures, in a resilient system you will see intermediary mushroom cloud effects of a large number of services being affected temporarily. How do you find out what really caused the problem and how to distinguish effect vs. cause?
In this session we will do post-mortem analysis by walking through different cases of failures we've observed in a real-world large e-commerce production environment and show you how to figure out what actually caused the failures."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
so
I
think
we
can
can
get
started
right
now.
Letting
lost
people
hang
in
here
great.
So
thank
you
for
joining
hope.
You
had
a
great
cube
run
so
far
and
cloud
native
come
so
far
and
hopefully
also
had
like
a
great
party
yesterday
yeah,
if
you
had
a
great
party
yesterday,
probably
you
feel
like
you
failed
today
in
the
morning
with
a
little
bit
of
hangover,
but
that's
all
good
things.
So
welcome
to
my
talk.
What
happens
when
containers
fail
or
like
the
mushroom
cloud
affect
what
happens
when
containers
fails?
A
So
my
name
is
Alice
Maya,
as
you
might
already
have
recognized.
I
have
a
very
German
accent,
but
not
from
Germany
I'm
from
Austria,
like
you
know,
small
little
country
in
the
center
of
Europe,
and
whenever
you
talk
in
the
US,
austria
is
some
kind
of
yeah.
It's
a
small
little
country.
I
mean
we
have
the
size.
I
think
the
austrian
population
is
about
same
population
as
Manhattan,
so
but
it's
at
least
block
sure,
but
whenever
I
talk
to
two
people
in
u.s.
A
they
always
may
connect
Austria
with
the
sound
of
music
and
that's
weird
fast,
because
in
Austria
no
one
knows
about
that:
I!
Don't
know
why
no
one
in
Austria
knows
about
the
sound
of
music.
So
that's
why
I
have
never
seen
the
sound
of
music
but
seems
to
be
a
pretty
funny
thing
so
likely
a
very
nice
and
landscape
they're
like
huge
mountains
and
all
that
cool
stuff.
But
yeah.
A
So
some
words
about
time
trace
whenever
I
give
talk
in
yeah
about
something
from
diners
race.
Most
of
the
people
don't
really
connect.
Dinitrate
was
like
more
on
cloud
platforms.
Diana
is
basically
is
around
since
2005
it
was
founded
in
austria
and
is
now
headquarter
in
boston
yeah.
We
have
around
1,600
1,700
employees
worldwide
and
about
say
400
500
engineers,
most
of
them
are
in
austria
and
in
poland.
So
most
of
the
engineering
happens
in
europe
and
we
have
a
volume
customers
across
all
industries.
A
So
like
financial
industry,
like
insurance
companies,
banks
manufacturing
industry,
all
at
all
the
things
and
over
the
time
of
the
last
decade,
we
have
seen
lots
of
different
problems
in
their
stacks.
So
when
you
go
back
in
2005
and
you
provide
monitoring
for
like
a
bank,
you
do
not
deal
with
like
modern
stuff.
A
These
days,
like
we
ma
so
you
you
deal
with
like
mainframes
yeah,
you
need
to
deal
with
huge
java,
app
java
stores
or
Java
or
top
met
applications
and
one
of
us,
but
nowadays
we
need
to
deal
with
like
more
modern
things
like
highly
dynamic
environments,
dr.
environments,
Cuban,
it
is
environment
where
everything
you
know
comes
and
goes
pretty
frequently.
You
know
it
contain
a
hidden
leaf
for
seconds
only,
but
you
also
need
to
be
able
to
deal
with
like
this
highly
dynamic
infrastructure
and
yeah.
A
A
Despite
that,
I
want
to
do
a
show
of
hands,
so
I
hope
every
one
of
you
has
somehow
filled
with
containers
at
this
time
who,
if
you
deal
with
containers
yeah
almost
all
that's
good
and
next
question
is
whose
life
has
become
easier
since,
like
there
are
containers
out
there.
So
since
there
is
communities
out
there
or
something
like
that,
test
become
easier
compared
to
like
the
monolithic
yeah,
that's
true,
that's
good,
and
who
of
you
feels
more
like
a
deaf
and
who
of
you
feels
more
like
an
OP
sky.
A
That
would
be
interesting
and
using
it
usually
at
this
question
the
deaf
guys,
all
the
deaf
guys
say:
okay,
it
has
become
easier
because
it's
really
easy
to
package
everything
into
a
single
image
and
let
the
OPSEC
I
care
about
the
rest
and
for
optical.
It
should
be
supposed
to
be
easier
because
you
can
to
run
the
container
with
the
image
right,
but
for
most
of
the
ops
it
became
not
really
easier
because
they
have
to
deal
with
an
additional
layer
of
complexity.
There
is
an
additional
layer
of
you
know.
A
You
have
need
to
take
care
of
different
versions
of
your
container
wrong
time.
You
need
to
take
care
about
the
orchestration
layer
need
to
deal
with
soft
different
networks,
so
you
have
a
lot
of
new
complexity
coming
in.
That's
why
it's
not
really
easier
for
Ops
guys
to
run
containers
yeah,
what's
needed
to
make
it
easier.
Yeah,
I
think
you
can
skip
this
question
since
we
are
at
cube,
come
here.
A
So,
as
I've
already
mentioned
in
the
introduction,
we
have
a
lot
of
experience
of
our
customers
where
we
have
seen
what
can
go
wrong
in
all
different
kinds
of
environments
like
in
huge
monolithic
in
flipper
applications
like,
for
example,
ecommerce
doors.
What
can
go
wrong
when
you
need
to
deal
with
SI
p?
What
can
go
wrong
if
you
want
to
run
a
huge
data
processing
cluster,
so
lots
of
different
stories
in
here
and
with
over
the
last
two
years?
We
also
learned
a
lot
of
things
of
running
containers
in
production
and
usually
I.
A
Do
always
like
this.
We
can
talk
about
a
lot
about
containers
and
this
time
I
want
to
talk
about
the
mushroom
cloud
effect
and
yeah.
I.
Think
most
of
the
most
of
you
would
immediately
know
what
I
mean
with
the
mushroom
cloud
effect,
but
probably
sometimes
our
people
are
in
there
that
not
really
get
the
first
place.
What
the
bathroom
cloud
effect
is
all
about
so
tldr.
A
A
It's
a
screenshot
of
a
real
time,
dependency
graph
of
all
components
within
an
environment
within
a
container
environment
that
has
failed
during
an
issue
and
all
the
components
how
their
components
have
been
connected
to
each
other
with
all
the
dependencies
in
there.
So
it
looks
like
a
mushroom.
It
costs
a
lot
of
components
in
there.
So
let's
put
some
layers
on
over
there
over
this
mushroom.
So
at
the
bottom
here
we
see.
Ok,
here
are
lots
of,
let's
say
talk
a
host
like
a
lots
of
Linux
machines,
running
a
doctor
engine
and
yeah.
A
So
this
is
probably
the
root
cause
of
many
problems
here.
Yet
there
is
like
there
are
all
the
hosts
running.
Actually,
application
containers
at
the
second
level.
There
are
all
the
processes
in
containers.
I
mean
you
know,
containers
of
just
like
isolated
processes.
That's
why
we
treat
processes
and
containerized
processes
the
same
way
so
to
say
that
there
are
lots
of
containers
there
and
all
the
processes
and
at
the
top
of
of
third
of
this
layered
picture,
there
are
all
applications
and
services.
A
A
Okay,
before
we
we
go
into
a
post-mortem
analytics
of
this
mushroom.
You
want
just
to
spend
some
some
words
about
how
people
build
the
system.
Nowadays,
usually
you
know
all
the
stories,
all
the
successful
stories
of
Netflix
and
all
the
famous
examples
out
there.
You
have
lots
of
services
or
microservices.
All
the
services
are
somehow
connected
to
each
other.
They
need
to
communicate
with
each
other.
You
also
have
dependencies
between
services.
A
So,
if
you're
in
service
a
talking
to
a
service,
be
you
have
a
dependency
between
those
two
services,
but
it's
not
only
on
the
service
level,
but
also
like
on
a
versioned
service.
So
if
you
have
like
a
service
aimed
version,
1
and
version
to
the
version
2
might
have
other
dependencies,
then
the
version
one
of
the
same
service.
So
we
have
really
like
a
very
complex
graph
of
all
dependencies
within
your
entire
environment
yeah
and
you
always
most
likely
have
like
a
requirement
of
zero
downtime.
A
When
you
build
such
a
system,
you
usually
build
it
that
it's
available
all
the
time
and
you
don't
want
to
have
downtime
when
you
upgrade
anything-
and
this
brings
me
like
to
our
yesterday's
closing
keynote
from
from
renton
Phillips,
who
just
demoed
us
how
easy
it
is
to
just
upgrade
a
community's
cluster
to
a
new
version
with
a
one
button
right,
yeah
and
there
are
some
some
other
platform
related
aspects
as
well.
So
it's
not
only
having
like
lots
of
services.
You
need
to
offer
to
your
customers.
A
For
example,
if
you
want
to
deploy
a
service,
you
will
most
likely
run
it
like
multiple
times,
like
in
five
containers
like
building
a
replica
set
for
five
instances,
so
that
you
run
five
containers
for
the
same
service
and
all
of
the
containers
are
a
funeral
and
the
underlying
infrastructure.
This
case
communities,
for
example,
I,
will
take
care
of
whether
your
services
are
still
up
and
running
or
not,
and
if
the
orchestration
layer
detects
it
there's
something
wrong,
it
just
kills.
A
The
containers
that's
a
new
one,
always
trying
to
have
a
decent
amount
of
containers
running
for
a
service
which
was
defined
by
the
operator
or
probably
you
to
also
auto
scaling
it
yeah.
So
in
this
case,
you
see,
on
the
left
hand,
side
we
have
like
one
service,
it's
called
the
booking
service
and
the
booking
service
basically
is
made
available
through
six
different
compares
and
yeah
one
container
out
of
the
six
disappeared
just
offline.
For
any
reason,
and
right
now,
it's
like
provided
by
five
different
containers.
A
As
already
mentioned,
we
don't
really
differentiate
between
processes
and
containers,
because
yeah
it
containers
I
mean
processes
in
containers
are
just
as
related.
That's
it,
and
those
five
processes
or
convenience
actually
run
on
five
different
host
and
in
this
case,
in
two
different
data,
centers,
meaning
in
two
different.
It
will
ability
stones
and
on
AWS
and
what
we
also
observe
is
yeah.
Okay,
here
is
like
the
one
container
that
is
offline
right
now,
and
what
we
also
observe
is
the
deployments
are
no
longer
static.
A
So
this
is
like
the
new
challenge
of
all
the
operators
in
order
to
figure
out
what
is
the
desired.
The
optimum
amount
of
containers
running
for
specific
service
for
a
specific
microservice,
because
you
need
don't
want
to
waste
your
resources.
You
want
to
save
resources
to
make
it
easier
to
schedule
other
tasks
or
whatever
and
save
money
on
on
renting
a
SS
resources,
for
example.
So
what
we
also
see
is
that
in
this
case,
in
the
morning
we
have
low
load
for
a
specific
service.
A
Let's
two
containers
are
enough
to
run
or
to
serve
a
specific
service
during
during
noon.
There
is
like
a
high-traffic,
a
peak
local
scenario
where
we
scale
up
to
I,
don't
know
seven
containers
and
the
afternoon
the
load
goes
down.
So
that's
why?
For
customers,
it's
enough
to
also
again
kill
some
of
those
containers
and
safe
resources
right,
and
this
perfectly
fits
basically
to
the
story
out
there
from
super
super
giant,
yes,
yep,
so
the
guys
out
there
that
also
try
to
like
save
resources
on
extending
and
shrinking
clusters.
A
Okay,
how
does
a
micro
service
environment
with
lots
of
services
look
like
in
practice?
It
looks
somehow
like
this,
so
in
this
case
we
see
we
can
just
taking
the
booking
service
here,
there's
a
single
service
which
is
called
booking
service,
basically
yeah
in
in
version
5,
and
this
is
just
one
service
out
of
1,000
services
in
this
cluster,
and
what
we
see
is
here
on
the
left-hand
side
is
that
the
deployment
complexity
of
those
of
this
service
we
see.
A
Okay,
this
single
service
is
being
deployed
through
five
different
containers
and
five
different
machines
on
to
data
centers,
but
on
the
right
hand,
side
we
see
all
the
communication
parts
between
of
this
single
service
so,
like
this
single
service
is
talking
to
all
those
blue
colored
bubbles
on
the
right
hand,
side
and
yeah.
It's
kind
of
a
mess
there,
because
lots
of
service
is
in
there
and
it
should
visualize
you
how
complex
such
an
environment
could
be.
So
it's
not
only
just
yeah,
we
see
the
containers
and
that's
it.
A
What
really
helps
you
is
figuring
out
how
these
services
really
connect
to
each
other
and
how
they
relate
to
each
other.
This
brings
me
to
the
next
slide,
which
is
when
you
have
a
micro
environment.
It's
really
all
about
service
dependencies,
so
you
really
need
to
figure
out
which
services
are
talking
to
each
other,
so
I
don't
know
who,
if
you
have
attended
the
keynote
yesterday
evening,
there
was
a
short
track
about
a
short
session
about
opened
racing
which
basically
try
to
solve
the
same
problem
right.
A
So
requests
coming
in
through
this
service,
our
like
spread
across
these
three
different
services-
and
you
will
see
like
how
they're
the
the
request,
those
through
your
entire
back
end
ending
matching.
If
this
is
again
the
booking
service,
the
starting
service,
each
of
those
services,
most
likely
is
deployed
again
multiple
times
with.
Let's
say,
five
containers
are
different
machines,
so
they
were
very
the
complexities.
This
is
pretty
pretty
high,
so.
A
What
actually
happens
when
containers
fail?
So
what
happens
when
this
single
container
fails
here?
Yeah,
so
usually
the
answer
is,
it
depends.
You
know
the
platform
is
built
for
resiliency.
So
that's
why
the
platform
takes
care
of
just
starting
a
new
container
of
the
same
image
and
everyone.
Everything
is
good
right,
so
usually,
if
a
single
container
fails
most
likely,
it's
not
a
problem,
but
what
happens
if
another
container
fails
like
if
two
or
three
containers
of
the
same
service
failed
or
if
a
failing
container
has
an
impact
of
a
container
for
completely
different
service?
A
We
complete
a
different
doctor
image.
What
happens?
It
really
depends
and
that's
why
it's
important
to
know
how
these
containers
relate
to
each
other
and
how
the
services
relate
to
each
other
right.
So
in
this
case
a
single
training
container
do
not
have
any
problem
or
do
not
have
any
impact
on
other
services.
A
Okay,
so
what
we
have
learned
is
basically
yeah.
Just
kidding
failures,
obviously
lead
to
yeah
issues
and
might
lead
to
a
mushroom
tart
effect
where
you
have
like
a
snowball
effect.
So
if
you
contain
a
fail
and
it's
an
important
big
container,
it
most
likely
destroys
other
containers
and-
and
it's
like
a
snowball
effect,
then
and
now
I
want
to
walk
through
one
of
Bach
want
to
walk.
Through
one
example,
we
have
observed
where
single
campaign
are
really
a
cost,
yeah
a
little-little
out-pitch,
and
this
again
is
some
kind
of
a
mushroom
cloud.
A
You
know
it's
basically
the
same
visualization
as
before,
but
with
less
components
included.
So
in
this
time
we
see
that
there
was
an
outage
or
there
was
a
problem
and
and
all
those
components
affected.
You
can
see
here
at
the
bottom
that
they're
basically
were
to
Linux
machines
and
at
the
top
there
was
a
couple
of
services
that
Tomcat
services
in
Apache
web
service
engine
X.
There
are
some
databases
in
there,
so
lots
of
different
components
and
how
they
relate
to
each
other,
and
now
I
just
want
to
to
walk
through
step-by-step
what
happened.
It.
A
At
this
issue,
so
what
was
the
problem?
So
we
have
seen
that
this
problem
at
a
customer
where
the
customer
had
a
real
yeah
real
huge
pain.
So
they
used
to
had
like
a
huge
monolithic
system
and
they
had
an
outage
were
in
them
on
the
loot
exit
system
and
they
were
not
able
to
to
figure
out
the
root
cause
of
the
outage.
They
were
not
able
to
find
a
solution
for
the
outage
fast
enough,
so
they
lost
a
lot
of
money
during
the
outage.
A
That's
why
they
decided
they
want
to
get
rid
of
the
monolithic
application
and
migrate
to
like
a
micro
service
application
and
what
they
did
was
that
pretty
nicely
sliced
down
the
monolithic
application
into
micro
service
environment
and
they
used
dr.
for
just
running
the
micro
services
because
yeah
it
was
I,
think
back
in
2014,
also
15
a
pretty
nice
way
and
how
to
do
that
and
the
only
problem
was
that
containers
were
pretty
new
to
them
and
they
were
really.
You
know,
mm
angry
that
probably
the
micro
service
environment
will
also
fail
right.
A
A
That's
why
they
decided
that
they
will
just
you
know,
mount
a
LOX
petition
on
every
host
within
every
container
running
on
the
host,
so
that
every
container
on
a
host
shares
the
same
lock,
petition
of
the
underlying
host
okay.
So
this
was
like
a
design
decision
of
the
operators
just
to
make
sure
that
if
a
container
fails,
if
something
goes
wrong,
that
they
will
have
access
to
the
log
files
stored
on
the
on
the
petition.
A
So
that
was
the
reason
why
they
wanted
to
have
like
a
shared
locks
petition
and
they
have
no
rotation
or
stuff
like
that
in
place
and
no
proper
log
management.
But
what
happened
at
the
certain
point
in
time?
The
locks
petition
ran
out
of
space.
You
know
if
you
have
like
a
LOX
petition.
All
containers
writing
the
application
logs
into
the
same
locks.
Petition
I
mean
it's
just
a
matter
of
seconds
minutes
hours,
I,
don't
know,
and
at
the
certain
point
in
time
the
Lord's
petition
ran
out
of
space
yeah.
A
So
how
did
the
problem
basically
evolved
over
time,
so
the
orchestration
layer
they
used
had
some
health
checks
for
every
container
and
the
container
we
are
not
able
to
write
logs
anymore.
That's
why
the
health
check
failed.
So
what
was
the
job
of
the
orchestration
layer?
Then
it
just
killed
a
container
because
it's
not
health
again
healthy,
again,
just
kill
the
container
and
start
a
new
container,
because
that's
the
job
of
devastation
layer.
So
everything
is
fine
right,
okay,
but
there
yeah
we're
still
no
free
free
space
on
the
locks
petition.
A
A
Yeah
I
mean
yeah.
The
orchestration
layer
again
detected
that
the
health
check
failed,
just
killed
the
container
again
and
started
a
new
one,
so
it
was
like
kind
of
an
infinite
loop
of
killing
and
starting
containers
do
to
like
failing
health
checks.
All
the
time
because,
like
the
Loch
petition
was
vs,
was
full
I
mean
it
wasn't,
really
infinite.
A
So
actually
it
just
took
a
couple
of
minutes.
I
guess
then
happened
this,
so
the
next
petition
went
out
of
space
so
valley,
dr
invalid
dr..
It
was
like
the
default
configuration
at
that
time
where
dr.
you
know,
stores
like
the
container
file
system.
So
with
every
container
you
start
you
get.
You
need
a
specific
space
in
order
to
have
like
a
space
available
for
the
container
file
system
and
if
you
kill
the
container,
the
old
file
system
of
the
container
will
not
be
removed.
A
So
that's
why,
after
yeah,
some
minutes
or
so
the
valley
polka
directory,
ran
out
of
space
as
well.
So
there
was
additionally,
there
was
one
more
problem
in
there,
so
we
have
a
full
locks
petition
at
full
value,
vodka
and
petition.
The
real
problem
was
that
Valley
doctor
was
on
the
route
petition,
so
basically
the
whole
linux
machine
was
dead
which
lead
to
yeah
that
you
know
cluster
nodes,
so
this
single
cluster,
node
or
after
wild
mark
to
the
cluster
nodes,
we're
no
longer
available
to
to
run
containers
and
that's
a
bad
situation.
A
If
you
have
a
cluster
just
for
running
application
containers-
and
you
can't
run
any
containers
anymore
because
of
that
problem,
so
so
how
this
problem
has
affected
our
services.
You
know
it's
just
like
containers
that
do
not
run,
but
you
you
always
run
containers
by
purpose.
In
this
case
there
were
multiple
from
services
effective.
A
At
the
top
of
the
graph
I
mean
the
resolution
would
be
better,
then
you'd
see
like
Tom,
kept
engine,
excellent
and
all
the
things,
and
we
also
observed
an
increased
failure
rate,
which
basically
means
that
other
services
tried
to
connect
to
consume
these
services
and
just
go
to
timeout
or
whatever
right.
So
we
detected
increase
trailer.
It's
here.
A
So
the
problem,
how
how
the
problem
could
have
been
avoided?
Yeah,
it's
it's
pretty
obvious.
Using
don't
do
that
stupid
things
of
sharing
locks,
petition,
use,
lock,
drivers
remove
containers-
if
you
don't
need
him
anymore.
So
just
we
let
just
remove
containers
if
they
exceed
10
or
20
or
whatever,
and
move
your
value
proposition
or
move
like
a
petition
where
you
store
like
container
file
systems,
you
know
to
another
petition,
then
then
the
root
petition-
it's
like
I,
think
a
good
idea
in
this
case.
So
but
this
is
not
really
specific
to
to
contain
environment.
A
I
mean
this
can
happen
in
all
traditional
systems
as
well
right,
but
it
wasn't
really
was
really
a
problem
in
this
environment
because
the
people
there,
the
customer,
were
really
surprised
that
this
could
happen
in
new
technologies
as
well.
So
what
did
they
do
so
when
they
have
learned
that
this
specific
case
wasn't
a
good
idea?
They
wanted
to
be
prepared
for
black
friday
and
they
don't
wanted
to
to
run
into
the
same
problem
again
during
like
their
best
time
in
the
year
where
they
sell
the
most
product.
A
A
So,
after
the
three
days
during
their
load
test,
more
than
13,000
containers
were
included
and
about
3,400
services
have
been
tested
during
the
three
days,
and
this
is
like
again
the
mushroom
cloud
which
I
have
shown
you
initially
at
the
beginning,
where
we
basically
analyzed
820
billion
dependencies
between
all
those
containers,
services,
processes
and
and
and
hosts,
and
only
like
two
hosts
to
dr.
hosts,
really
had
some
tcp
issues.
Basically,
suspicious
TCP
connectivity
issues
basically
means
they
were
offline
yeah.
A
A
A
Otherwise,
you
won't
be
able
to
figure
out
whether
something
goes
wrong
or
not
right,
and
what
we
have
learned
is
it's
really
important
to
have
an
automation
and
automated
approach
in
order
to
figure
out
the
root
cause
of
a
problem
right
by
analyzing
all
those
dependencies
when
we
figure
out
okay,
when
a
process
dies
because
the
container
was
killed,
what
is
the
effect
of
the
kilt
container
to
a
service?
What
is
the
effect
to
another
service?
A
A
A
A
So
this
is
like
the
dashboard
of
of
our
demo
environment,
the
demo
environment.
Isn't
really
it's
like
traditional
environment
running
multiple
things
there,
and-
and
this
is
like
one
screen
you
have
seen
before-
let's
pick
some
of
those
guys
where
we
see
okay,
this
case
we
run
I,
think
you
need
to
click
10,
make
this
guy
or
not,
where
we
see
ya
how
the
services
are
related
to
each
other
and
how
they
deployed
how
they
are
deployed
on
on
machines.
As
you
can
see,
there
is
a
windows
machine
we
so
we
have.
A
A
Let's
take
this
guy,
so
what
we
have
is
our
agents
in
checked
into
docker
containers
and
are
able
to
see
what's
going
on
within
your
applications.
In
this
case,
we
see.
Okay,
we
have
one
application,
which
is
a
web.
Application
is
vmware
eztravel
com
which
basically
sends
requests
to
our
yeah
front-end
in
the
back
end,
and
this
front
end
has
calls
to
other
services
like
a
nothing
occasion,
service
booking
service,
and
we
just
can
let
the
show
here.
A
Okay,
this
service
is
like
he
deployed
only
one
process
and
talks
to
two
other
services
there,
but
we
can
also
jump
into
a
service
flow
which
basically
shows
you
how
this
single
service?
How
dare
the
requests
through
this
service
traverse
through
the
system
and
end
up
somewhere
in,
for
example,
in
a
MongoDB?
A
So
the
MongoDB
we
see
okay,
what
are
the
queries
to
lick
MongoDB
and
how
long
they
took
and
its
aggregations
here
yeah.
So
we
have
505
career.
Is
there
and
the
average
time
is
40
milliseconds,
but
we
can
also
check
okay,
who
actually
is
calling
this
MongoDB
and
figure
out
like
backwards.
So
what
are
the
service
is
clearing
my
database
right.
A
A
So
in
this
case
we
see.
Okay,
we
have,
on
the
left
hand,
side
two
problems.
Basically,
two
through
applications
are
impacted,
so
these
are
two
websites
which
we
have
and
the
root
cause
basically
was
so
that
the
impact
was
a
user
action,
degradation,
user
action,
duration
degradation,
which
basically
means
the
response.
Time
was
too
high
right
for
the
easy
travel
b2b
data
web
site
and
the
slowdown
most
yeah
crazy,
but
the
root
cause
basically
was
that
the
check
destination
service
in
the
background
caused
the
problem
all
right.
Let's
check
this
out
yeah.
A
What
we
see
here
is
okay
during
that
period
of
time,
where
we
figured
out
that
there's
a
problem
in
your
environment.
We
see
okay
yeah
at
this
time
here
the
response
time
went
up
to
12
seconds
and,
let's
just
analyze
the
response
time
here
and
we
would
see
okay,
the
code
execution
took
like
four
seconds
longer
than
it
used
to
take
in
the
comparison
time
frame
before,
and
we
even
see
like
okay
at
the
code
level,
what
what
was
the
impact
on
the
code
level.
A
So
what
was
the
problematic
situation,
the
border
and
the
code
level
here?
Let's
go
back
and
the
mushroom
cloud
I
have
shown
you
before
here
in
this
specific
problem.
This
also
is
like
a
kind
of
a
mushroom
cloud
where
we
again
see
like
all
the
hosts
at
the
bottom
of
the
graph
and
at
the
top
of
the
graft
applications
and
services,
and
we
can
even
replay
that,
where
we
see
okay,
how
a
specific
problem
in
this
case,
the
problem
is
the
check
destination
service
in
the
back
end
has
evolved
over
time
right,
yeah.
B
B
A
A
So
let's
go
back
so
here
is
here
all
operating
all
our
VMs
or
ec2
instances
or
whatever
you
just
need
to
place
an
agent
on
the
machine
and
whenever
we
detect
that
you
start
a
new
process
beat
like
a
JVM,
a
tomcat
and
no
chess
process,
a
docker
container
running
a
tomcat.
We
just
inject
the
agent
into
the
process,
for
instance
in
Java.
This
basically
means
we
adapt
the
JVM
startup
parameters
and
eject
our
agent
into
the
JVM.
A
A
Okay,
what's
going
on
within
the
JVM,
we
put
all
the
data
to
our
cluster
and
in
our
back-end
we
can
all
correlate
and
analyze
all
the
information
coming
from
different
parts
in
your
environment,
coming
from
the
host
itself
coming
from
the
agents
running
in
the
processes
in
the
JVM,
but
also
when
we
detect
that
your
your
service,
for
example,
delivers
HTML
right.
We
are
also
able
to
inject
a
specific
JavaScript
snippet
which
allows
us
to
track
what
the
user
is
doing
in
the
browser.
A
So
if
the
user
hits
a
button
in
the
browser,
this
was
the
example
I've
shown
you
before
with
like
the
the
impact
too
easy
travel
be
2
b.com,
where
we
have
detected
that
for
the
end
user,
it
was
a
slowdown
of
like
two
minutes.
We
are
able
to
detect
that
because
we
we
inject
agents
into
into
HTML,
so
we
can
measure
what's
going
on
in
the
user
browser
what's
going
on
in
the
in
the
back
end.
A
B
A
Okay,
so
the
question
was
how
we
analyzed
the
older
information
during
the
three
days,
locusts
right,
so
the
three
days
load
test
wasn't
done
by
us.
It
was
a
lotus
by
the
customer.
The
customer
just
wanted
to
test
the
infrastructure
and
had
our
monitoring
solution
enabled
right.
We
didn't
really
do
anything
there.
A
We
just
were
there
just
might
on
everything,
but
what
we
have
detected
was
basically
that
some
of
the
services
had
higher
failure
rates
head
out
pages
because
of
two
hosts
had
too
much
traffic
that
they
did
really
that
they
weren't
really
able
to
deal
with
the
traffic
right.
So
we
didn't
do
anything
the
customer
decided
to
run
their
load
tests.
We
just
captured
all
the
information
and
visualize
them
and
our
in
our
back-end.
We
just
run
our
our
analytics.
B
A
So
they're,
so
everything
you
can
see
here
is
just
like
a
visualization
of
all
the
the
monitoring
information
we
get
from
from
from
the
agents
deployed
on
the
cluster
right,
like
everything
on
the
host
process,
is,
let's
jump.
Let's
take
this
guy
see
if
there's
something
interesting
in
there,
yeah
I
mean
that's
a
boring
one.