►
From YouTube: Real-Time Application Maps for Proactive and Actionable Visibility - Aloke Guha, OpsCruise
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Real-Time Application Maps for Proactive and Actionable Visibility - Aloke Guha, OpsCruise
Today’s observability provides volumes of time-series data, statistical trends including anomaly detection and correlational analyses. We argue that operations teams need an integrated and cohesive understanding of the application that maps interdependencies across microservices and dependencies on the orchestration and infrastructure services. We show that beyond metrics, logs, and traces, capturing configuration information are necessary for creating a complete application maps for gaining deeper insights into the application behavior. In addition, establishing a standard approach capture the attributes of the complete application environment will enable automated detection and causal analysis of application problems. We will present some early findings on building real-time actionable application maps for cloud applications.
A
Good
morning,
let's
get
this
team
loaded,
so
I'm
a
look,
a
little
kou
helm,
the
CTO
and
co-founder
of
abstruse
and
today
I'm
going
to
talk
about.
How
can
we
provide
Enterprise
Ops
teams?
What
do
I
call
proactive
and
actionable
visibility?
That's
a
lot,
but
as
I
go
through
it,
hopefully
that'll
make
sense,
but
it's
related
to
some
of
the
talks
that
our
previous
speakers
have
already
talked
about.
You
heard
about
the
gap
between
observer
really
and
control.
You
heard
about
unification.
A
You
heard
about
the
the
whole
notion
of
trying
to
use
open
source
and
not
getting
locked
all
of
those
play
into
this.
So
that's
what
we'll
do?
Let's
start
with
where
Enterprise
ops
are
today
right.
What
are
the
challenges
faced?
I
would
say:
they're
three
broad
areas
right,
one,
the
the
total
volume
of
data
explosion
that
has
happened
now,
right
that
hasn't
changed
is
only
gonna
get
worse.
It's
volume,
velocity
and
transients.
That's
one
big
sector,
one
fact
set.
The
second
is
what
I
call
lack
of
in-depth
visibility
right.
A
One
of
the
key
things
that
has
happened-
and
it's
only
gonna
get
worse
in
some
sense-
is
the
steering
that's
happening
and
gives
us
obsession
right,
makes
it
harder
and
harder.
The
other
one
is
the
long
chain
dependencies.
We
already
know
about
those
right,
but
the
other
bigger
problem
is
also
where
Ops
teams
are
coming
from
and
their
challenges
right.
We
have
a
lot
of
different
tools,
open
source
and
proprietary
right.
You
have
different
teams
working
on
them
and
you
don't
have
the
full
knowledge
across
them.
A
So
we
still
have
war
rooms
right,
there's,
a
fundamental
shortage,
not
everyone
is
at
Google
and
Netflix
right.
So
where
is
observability
today?
If
you
look
at
it
and
you
know,
we
are
presenting
that
around
the
room,
we
have
multiple,
diverse
different
data
sources.
Today,
right
we
have
time-series
metrics.
We
can
do
all
kinds
of
things
with
that.
We
have
traces
and
dependencies.
We
just
heard
about
that
logs
and
events
and
a
lots
of
all
kinds
and
clearly
the
the
challenge
that
the
teams
face
is
how
do
I
put
them
together
in
fiction.
A
You
already
seen
that
not
only
that
the
teams
have
very
sophisticated
tools
already
today
to
take
advantage
of
these
pieces.
The
question
is:
how
do
we
bring
them
all
together
and
where
is
it
where
as
this?
But
what
is
the
prevailing
wisdom
and
the
current
state
and
the
enterprise
to
be
able
to
handle
the
data?
So
if
you
look
through
it,
you
know,
one
of
the
ways
people
are
handling
to
sprawl
is
obviously
leveraging
open
source.
That's
already
happening
right.
A
This
conference
we
are
in
with
the
whole
cube
con.
The
CNC
of
star
is
a
testament
to
that.
The
second
area,
where
multiple
dependencies
you've
already
heard
the
wisdom,
is
get
all
the
data
out
there
collect
all
the
metrics
use
all
the
data
sources
right
and
if
there
are
transients
record
everything
right
so
now
we
have
essentially
a
very
large
number
of
objects,
a
lot
more
metrics,
more
logs
and
more
alerts.
The
scale
just
went
bigger
and
guess
what
war
rooms
are
still
here
right,
and
why
is
that?
A
At
the
end
of
the
day,
if
you
look
at
some
of
the
work
that
was
done
recently
at
Cornell,
the
complexity
of
the
system
itself
is
hard
right.
The
dependencies,
the
heart
individual
components,
have
multiple
aspects:
clustering,
different
types
of
nodes,
right
master
versus
data
nodes,
everything
scales,
and
even
if
you
don't
have
the
Deathstar
of
Netflix
or
Amazon,
the
total
number
of
interactions
that
happen
make
it
very
very
hard
to
understand.
What's
going
on
before
something
hits
the
fan
right.
So
the
question
is:
what
can
we
do
to
help
them
right?
A
So
a
case
in
point:
I
was
talking
to
a
financial
sector
customer
a
couple
of
weeks
ago.
Very
savvy
guys
have
been
there
on
running
their
services
on
Amazon
Cloud
right
has
instrumented
everything
from
monitoring
tools,
he's
turn
on
all
Prometheus
he's
collecting
data
from
kubernetes.
He
has
information
from
his
cloud.
Vendor
he's
instrumented
every
service.
He
rode
at
micro
meter.
A
He
can't
predict,
what's
gonna
happen
so,
and
he
wants
to
reduce
that
time
that
we've
all
heard
right,
the
Holy,
Grail
or
reduced
the
MTTR
MTD,
so
where's
the
gap
coming
from
I
want
you
to
go
to
back
to
basics
when
the
first
time
the
word
observable
really
was
coined,
which
is
over
six
decades
ago,
calleman
1960,
observable
T
is
when,
without
knowing
the
initial
state,
you
can
determine
the
state
of
the
system
right.
Those
of
you
studied
control
theory.
A
If
you
know
the
system,
how
do
you
determine
it
by
looking
at
the
inputs
and
outputs
standard
that
logic
hasn't
changed
for
60
years?
It
won't
change
second
necessary,
but
not
sufficient
condition
is
that,
in
order
to
control,
you
should
be
able
to
steer
the
system
to
a
desired
state
if
you're
getting
to
a
certain
service
level.
A
That
means
you
need
to
know
what
the
control
inputs
are
right.
That's
very
simple
theory
is
understood,
but
if
you
look
at
what's
going
on
in
recent
on
complex
distributed
systems,
the
problem
is
at
the
time
of
Kalman.
We
didn't
have
large
spatial
temporal
systems.
If
you
go
back
to
the
Netflix
interaction
diagram,
the
number
of
internal
interaction
and
States
is
huge.
A
So
just
building
a
control
system
by
looking
at
the
all
the
metrics
itself
is
insufficient.
It's
not
going
to
work
you're
not
going
to
get
any
insight
into
what's
going
on,
and
even
if
you
were
to
look
at
every
component
tree,
what
is
the
control
input?
Who
tells
you
that?
How
do
you
find
out
associated?
What
is
the
number
of
variables
state
kafka
right,
kavkaza,
160,
metrics,
you
know
build
a
box
wait
160,
plus
for
one
of
the
components
you
might
have
really
click
the
weather,
so
we
have
two
fundamental
gaps.
A
Even
if
you
look
at
the
principles
of
control
observability.
So
if
I
were
to
summarize
that
we
agree
with
releasing
tools
from
right,
you
can
now
normalize
and
bring
all
the
data
from
open
standard
tools
open
standard
instrumentation.
You
can
measure
dependencies
by
looking
at
data
from
different
metrics.
You
can
record
all
the
changes
you
have
three
things
left.
You
need
to
extract
and
obtain
the
structure
of
the
application
with
the
above
number
two.
You
need
to
understand
the
control
variables
of
the
controller
in
entities
of
every
component
within
that
system
and
associated
with
that.
A
You
need
to
reduce,
what's
called
the
dimensionality
problem.
The
dimensioning
problem
is,
it's
only
gonna
get
hard.
It's
an
np-hard
problem
right.
We
need
to
break
this
problem
down
into
better
ways
of
writing
it.
So
how
would
we
go
about
to
address
this
gap
using
the
above
definition?
One
is
we
need
to
will
extract
the
structure
from
the
application
level,
the
orchestration
layer,
the
cloud
layer
right
from
those
capabilities,
and
the
information
is
already
there
right.
You
have
configuration
from
communities,
you
get
it
from
your
cloud
VM,
which
is
open
access.
A
Api's,
you
get
the
information
from
the
componentry
for
the
containers
or
the
SAS,
and
the
containers
case.
It's
standard
information
that
comes
with
it.
If
you're
looking
for
behavior,
you
pull
in
the
flow
information.
The
data
that
going
between
interactions
flow
in
and
out
between
the
containers
or
you
can
use
traces,
for
example,
the
usage
which
refers
to
the
services
and
the
resources
being
used
by
every
service
or
container
and
the
traces
itself.
A
We
don't
need
the
full
traits
we
just
needed
between
the
interactions,
so
you're,
not
waiting
post-facto
after
the
whole
trace
has
gone
and
then
control
knobs
for
each
of
them.
You
need
to
know
from
curated
opinionated
knowledge
right
dependencies.
You
can
use
the
same
information.
Look
at
the
east/west
dependency
in
the
top
there.
You
can
extract
the
information
from
the
companies
layer
from
the
Kuban
estate
metrics
and
then
the
cloud
layer.
You
want
to
put
this
all
together.
It's
highly
diverse
data,
but
using
the
open
collectors
right
and
the
API
is
to
the
cloud
vendors.
A
You
can
collect
configuration
all
the
metrics
at
the
different
levels,
the
floors
or
traces
and
the
logs
and
events.
You
need
to
normalize
this
information
right
to
build
that
structure
itself
and
then
as
well
to
each
component
build
the
model
that
comprise
the
behavior,
so
you're,
essentially
dividing
up
a
distribute
system
into
a
graph
of
componentry
that
talk
to
each
other
at
different
layers.
A
Right
once
you
have
this
information
you're,
basically
getting
the
state
every
period
as
the
data
is
being
pulled
in
from
your
instrumented
application,
the
state
of
the
system,
as
is,
and
if
you're
in
a
predictor
move.
We
are
seeing
whether
there's
a
deviation
in
that
state
right
once
you
have
that
you
can
process
that
information
and
you
can
do
more
or
more
downstream
processing
right.
A
How
would
you
implement
this?
Well?
This
is
an
example
of
how
we've
instrumented
it,
so
those
orange
boxes
represent
open
collectors
that
sit
behind
an
existing
monitoring
plane.
So
they
said
behind
kubernetes.
They
said
behind
primitives
to
collect
the
states
and
events
they
collect
the
metrics
kubernetes
metrics
they
collect
API.
They
configure
the
configuration
information
and
all
the
data
from
the
cloud
vendors
or
your
VMs
configuration
files
that
have
access
to
flows
and
traces.
A
The
idea
is
to
give
you
give
the
enterprise
team,
which
have
a
different
group
which
are
a
different
team
kind
of
a
consistent
single
view
of
what's
happening
right,
you're,
not
talking
about
the
the
analysis,
that's
going
to
happen
after
the
incident
we're
talking
about
primarily
as
it's
happening
as
much
as
we
can
to
make
them
as
proactive.
They
can
be
that's
the
fundamental
goal,
we're
not
doing
it
as
a
transaction
by
transaction
level.
That's
a
different,
different
environment
altogether,
different
use
case.
A
You
know
after
this
talk
or
to
this
evening,
but
let
me
give
you
an
example,
so
this
is
collecting
the
data
from
those
three
collectors
right
from
kubernetes
from
primitives
from
the
cloud
level,
as
well
as
the
trace
information,
and
what
this
represents
is
just
to
give
an
example
on
the
left
are
the
blocks
and
structure
are
the
hierarchy
of
the
different
components
in
this
customers?
Application
in
this
case
he's
running
elastic
search.
A
You
can
identify
the
master
node,
the
pods,
for
the
data
nodes,
what
they
are
talking
to
or
their
connectivity
in
which
direction
right.
You
can
see
on
the
far
side.
Those
are
collectors
that
are
open
and
sitting
on
the
monitoring
plane.
Those
are
the
gateways
and
then
collecting.
This
information
is
providing
you
an
every
level
which
part
is
talking
to
which
way.
What
are
the
directions?
What
is
the
morphology
of
the
service
container?
A
Similarly,
for
that
environment,
for
that
part,
which
kuben
is
notice,
it's
sitting
on,
you
have
the
second
level
map.
And
then,
if
you
go
one
level
down,
you
have
essentially
the
three
layered
view
of
going
from
the
Container
services,
all
its
neighboring
ones,
upstream,
downstream
to
the
corresponding
kubernetes
environment
and
all
the
pods
that
sure
that
kubernetes
cluster
and
similarly
the
the
pods
that
are
sitting
on
the
cloud
nodes.
A
A
So,
if
you're
talking
about
tracing,
you
can
now
basically
build
the
same
map
coming
from
the
outside
in,
for
example,
here
it's
showing
one
coming
from
the
ingress
side
going
to.
In
this
case
it
happens
to
be
one
of
the
nodes
of
Kafka
and
trace
that
into
the
corresponding
one,
and
you
can
follow
this
in
real-time
because
you
have
the
data.
This
is
another
view
from
a
traffic
perspective.
It's
called
a
Sankey
diagram.
A
A
We
need
two
pieces
of
information
that
we
have
to
capture.
One
is
the
curated
knowledge
that
we
have
from
the
aspects
of
two
sorries.
Let
me
step
back
in
order
to
be
able
to
pull
this
data.
We
were
using
rules
to
map
the
structures
right
and
those
rules
are
I
with
open-ended
rules.
We
know
how
pods
use
common
area
Deak
the
kubernetes
cluster.
How
does
the
cluster
sit
on
top
of
the
cloud
nodes
right?
A
Similarly,
when
you're
talking
about
an
application
like,
let's
say
elasticsearch,
we
know
which
are
the
performance
related
metrics,
which
have
the
demand
related,
metrics
and
so
on.
So
for
all
of
those
information
that
goes
in
there,
those
rules
as
well
as
when
there
are
problems
that
surface
all
those
rules
are,
can
be
embedded
in
the
system
itself.
A
So
where
there
is
lack
of
information
because
remember,
the
Ops
teams
are
not
involved
in
the
Deaf
side.
That's
where
you
were
to
use
machine
learning
techniques
to
fill
in
those
gaps,
but
the
downstream
processing
can
then
say
when
your
deviations
there
take
the
data,
use
the
structure
analyze
it
categorize
it
and
essentially
come
up
with
the
actions,
because
you
know
where
the
control
inputs
are.
So
the
idea
is
gonna.
Give
you
that
full
chain.
A
Once
you
have
this
level
of
Zarate,
to
cut
down
the
gap
between
observe
and
control,
you
see,
if
you
know
it's
fully
observable
and
you
know
with
the
control
entities,
are
you
can
make
changes
so
just
kind
of
summarizing
at
a
high
level?
What
we
are
suggesting
is
in
order
for
the
ops
team
to
get
more
in
a
proactive
mode,
essentially
integrate
all
the
metrics.
The
events,
traces
and
configuration
in
one
area
use
them
with
the
created
knowledge
to
build
up
the
silver
all
structure
in
reach
and
build
up
the
structure.
A
We
should
require
a
combination
of
some
machine
learning
to
be
able
to
process
that
data
for,
for
example,
causal
analysis,
recommendations
for
fixes
and
so
on
so
forth,
I'm
not
going
to
speak
about
those
areas,
but
the
idea
is
to
use
this
as
a
starting
point
for
being
enable
all
of
this.
For
the
enterprise,
ops
teams,
we
are
running
with
a
number
of
customers
and
working
on
this
would
love
to
kind
of
have
discussions.
A
A
A
A
So
there
so
I'll
go
back
to
this
diagram
here,
so
the
third
aspect
of
it
under
the
metrics
or
traces.
There
are
a
couple
of
different
ways:
we
can
pull
that
data,
it's
basically
a
network
level
information
right,
so
we
are
looking
at
think
of
it
as
layer,
four
layer,
seven
information
that
we
provide
to
be
able
to
get
information
of
communications
and
the
data
that's
moving
from
one
to
the
other.
We
can
leverage
SQ
as
well
so
between
those
as
well.
A
So
there
are
multiple
ways
of
getting
there,
but
that
in
from
again
it's
an
open
framework,
so
we
can
get
that
to
essentially
add
to
the
containers
that
we've
already
discovered
are
using
service.
Yesterday,
okay,
any
other
trace
environments
in
your
trace,
instrumentation
yaga
should
talk
to
you
see
how
we
didn't
take
advantage
of
that
as
well.
Yes,.
A
Extremely
important
because
the
labels
that
come
at
the
different
levels,
right
from
the
application,
continues
to
kubernetes,
etc,
can
change
and
then
again,
when
the
containers,
as
you
know,
Diane
come
back
again
so
able
to
consistently
map
it
and
have
that
is
extremely
important.
So
one
of
the
real
wares
we
use
communities
as
the
source
of
truth
is
to
try
to
put
the
labels
in
there.
In
fact,
we
encourage
the
users
to
actually
put
the
labels
there,
so
we
can
identify
this
component.
A
B
A
Think
I
need
to
give
you
maybe
discuss
separately
at
this
point
when
we
are
talking
about.
Let's,
let's
take
an
example
of
elasticsearch
as
an
example
right
in
the
elasticsearch
level,
we
identify,
for
example,
the
control
mechanisms
are
how
we
can
change
the
configuration
sizes,
the
resources
that
you
give
to
it
as
an
example
right
those
we
already
know
to
measure
the
performance
of
elastic
searches,
the
search
index
performance
itself.
So
those
are
the
indicators,
and
the
controller
in
this
case
is
specific
to
that
component
itself.
A
So
we're
talking
about
identifying
what
are
the
specific
entities
that
control
it
if
you're
talking
about
effective
behavior
in
the
performance,
if
we're
talking
about
a
generic
container,
it
would
be
the
resources
that
it
needs
to
do
what
it
asks
you
right
again.
Typically,
the
infrastructure
resource
could
be
services
that
it
depends
on
as
an
example,
ketchup
that
makes
sense
and
and
it
by
the
weight
varies
right
from
component
to
component.
If
you
go
to
SAS,
it
will
be
different
right.
C
A
A
C
D
So
fantastic
correctly
like
it,
it's
like
that
we
are
pushing
like
all
that
praises
matrix
to
that
centralized
agent,
all
these
agents
and
does
it
support
like
sampling,
because
winner,
deeper
and
large
systems,
so
we
have
to
follow
that
are
to
sampling,
because
the
metrics
and
locks
are
all
in
silos.
So.
A
Two
things
one:
we
are
not,
we
not
essentially
I,
think
the
word
you
used
was:
are
you
going
and
putting
agents
in
there?
Is
that
what
you
say
you
know
we're
not
okay,
so
take,
for
example,
the
Prometheus
gateway,
which
is
the
second
one.
It
actually
sits.
There
only
talks
to
the
Prometheus
server
and
pulls
data
from
the
Prometheus
server.
Okay,
it's.
A
Kubernetes
it
talks
to
the
kubernetes
server.
We
don't
instrument
anything.
So
if
we're
talking
about
traces,
we
can
go
talk
to
whatever
tracing
mechanism
you
have
or,
for
example,
using
the
information
from
any
kind
of
network
metric
circle,
and
so
the
idea
is
to
be
not
instrument.
Anything
more
than
you
already
have
deploy
the
open,
instrumentation
nixon's.
D
A
And
leverage
that,
as
opposed
to
going
in
instrumenting,
so
we
don't
want
to
touch
or
instrument
the
application
code,
don't
want
to
do
additional
information
other
than
what
you
can
already
do
with
open
instrumentation.
The
only
caveat
there
is.
We
have
to
do
talk
to
the
cloud
vendor
or
the
infrastructure
vendor
deponent,
that's
what
we
have
to
do.
Mix-Ins
yeah.
D
A
That's
what
we
do
internally
on
the
mapping
with
the
roots.
Okay,
there's
one
other
thing
you
mentioned,
which
is
what
is
the?
What
is
the
frequency
you
get
the
data
right,
so
this
would
depend,
for
example,
on
how
you've
set
to
collect
data
from
Prometheus.
So
if
you're
collecting
de
prometheus
or
whatever
is
sampling
interval,
we
would
basically
match
that
sampling
interval
within
the
limit.
So
we
are
not
going
to
force
the
sampling
and
the
volume
at
the
end
of
the
data.
At
the
end
of
the
day,
all
that
data
is
yours.
A
C
E
C
A
So
the
example
originally
is
projecting
into
our
book,
but
there
are
multiple
parts
to
your
question
number
one
is
yes,
this
all
each
of
the
data
formats
are
different
right
summer
time.
Series
events
are
coming
in
at
a
certain
event,
within
the
sampling
field,
etc.
You're
right
we
have
to
put
and
use
time
series
database
in
order
to
display
the
information
that
we
have
on
the
dashboard.
We
have
to
use
the
graph
database
etc
right.
So,
but
that's
what
we
do
to
a
provide
the
information
to
you.
That's
not
the
real
thing.
A
That's
the
visual
aspect
of
the
real
information
is
what
we
have
that
we
can
process
and
analyze,
but
the
next
part
of
your
question,
which
is
it
can
implicit
in
that,
is
how
do
you
essentially
make
all
these
very
different,
metrics
congruent
and
that's
where
the
rules
come
in
everything
from
the
structure
and
the
dependencies
versus
what
the
meaning
of
a
metric
at
the
layer
coming
from?
Let's
say
a
server,
less
or
container
that's
running,
let's
say
Kafka
versus
what's
happening
at
the
Cuban
ease
level.