►
Description
Unified Monitoring of Containers and Microservices [I] - Nishant Sahay, Wipro Limited
Microservices are become critical for enterprise strategy towards simplifying their IT landscape. For a successful journey of microservice adoption, Container management, DevOps and Monitoring play an important role. Managing microservices in large-scale deployments are fraught with many unique challenges for enterprise IT.
Following are some of the key metrics of microservice monitoring which will enable the enterprises to manage their container platforms better:
1. Collecting logs, metrics from containers
2. Monitoring application running inside the container
3. Distributed tracing and the time taken by each service call.
4. Storage, analysis of collected metrics, logs
5. Performing RCA and anomaly detection on the collected logs and metrics
This session would explain how to harness the power of Zipkin with the intelligence of Spark ecosystem and the flexibility of ELK+ Beats to create a unified monitoring solution. Key features of this solution are – utilization of distributed tracing, infrastructure metrics to manage containers. All this is done through visualization, correlation and predictive monitoring
About Nishant Sahay
Nishant is a senior architect at Wipro Technologies with extensive experience in data architecture, design and visualization. Nishant is a Sun Certified Developer and IBM Certified IT Specialist with work experience spanning roles that include Software development, design and architecture. Nishant works at the Open Source COE lab at Wipro, where he is responsible for research and solution development in the area of big data and middleware. Nishant's current focus is on service monitoring and log analysis using machine learning and data science.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Oh
very
good
morning
and
thanks
a
lot
for
joining
me
in
the
session
on
unified
monitoring
of
containers
and
microservices
I'm
Michonne
Sahai
were
affiliated
with
the
Pro
limited
and
working
in
the
open
source.
Co
a
lab
and
Wipro
I
have
been
working
on
in
fine
monitoring
and
computer
vision
since
last
couple
of
years
and
basically
involved
with
the
data
analysis.
Team
I
would
also
like
to
take
the
opportunity
to
thank
my
colleague,
pavani
Anand.
She
is
an
architect
and
Wipro
limited.
A
She
was
a
co
speaker
for
this
particular
talk,
but
due
to
some
personal
emergencies,
she
had
to
take
leave.
She
had
been
instrumental
in
conceptualizing
this
entire
unified
monitoring
framework
and
has
deep
expertise
and
log
analytics.
So
coming
back
to
today's
talk,
the
goal
of
the
presentation
is
I
would
like
to
explain
how
we
have
utilized
the
power
of
Zipkin,
along
with
the
intelligence
of
apache
spark
ecosystem
and
the
flexibility
of
elastic
locks.
Gabbana
plus
bead
stack
to
create
the
signified
monitoring
framework,
so
I
have
divided
this
particular
talk
in
three
different
parts.
A
In
the
first
section,
I
will
be
setting
up
the
context
as
to
why
unified
monitoring
is
required
with
respect
to
micro
services.
Containers
in
the
second
section,
we'll
go
through
the
solution
overview
and
in
the
final
section.
I
have
solution
demonstration
in
which
will
be
catering
to
few
use
cases
with
respect
to
the
unified
monitoring,
including
anomaly
detection.
A
So
here
we
are,
this
is
a
typical
micro
services
use
case,
which
you
can
actually
map
it
to
any
of
the
micro
services
use
case,
which
might
you
be
implementing.
So
for
me
this.
This
can
map
to
a
online
e-commerce
application
and
historically,
which
was
a
monolithic
app
and
then
it
was
not
able
to
scale
up
had
issues
with
respect
to
speed
of
go
to
market,
and
then
we
adopted
the
micro
services
architecture.
A
That's
the
micro-services
accurate
picture,
since
it
is
build
on
modulite
dilation.
So
it
helps
us
in
reducing
the
complexity.
So
each
of
this
high-level
module
can
be
considered
as
a
service.
Now
and
each
of
the
service
functionality
can
map
to
a
business
capability.
So
so,
with
micro
services
we
achieved
service
isolation
or
to
services
are
not
tightly
coupled
with
each
other.
They
have
their
own
release
cycle.
They
they
don't
have
a
very
tight
integration
between
each
other.
They
can
not
provide
high
scalability.
A
No,
we
are
not
actually
scaling
up
the
entire
monolithic
app
if
needed
need
to
scale
up
rather
than
we
can
pick
and
choose
the
services
which
are
of
most
and
utmost
importance
and
not
having
a
high
resource
requirement
that
can
be
scaled
up.
It
is
giving
us
the
flexibility
in
in
the
sense
that
no
services
can
be
written
in
their
own
programming
languages,
so
you
can
actually
write
account
service
or
that
in
a
Python
and
whereas
your
customer
service
can
be
written
in
a
Java
programming
language
and
finally,
it
gives
us
a
better
resource
utilization.
A
Since
it,
you
can
actually
scale
up
and
scale
down
services
as
per
the
tech
requirement
that
you
have
in
hand
but
again,
like
everything,
micro
services
comes
with
a
cost
and
the
cost
is
more
related
to
Network
overheads.
Now
we
have
n
number
of
services
which
are
interacting
with
each
other
over
HTTP.
So
we
have
a
high
number
of
network
overhead.
Then
again
we
have
services
which
can
be
written
in
their
own
programming
languages,
so
they
have
their
own
deployment
approach.
They
can
have
their
own
release
cycles.
So
again
it
becomes
an
ops
nightmare.
A
So
finally,
this
awesome
trio
comes
in
where
we
have
the
micro
services
which
can
be
deployed
in
containers
and
run
in
kubernetes
cluster.
So,
finally,
what
we
have
achieved
is
service
isolation,
as
we
have
already
seen
with
respect
to
micro
services,
the
stag
or
mystic
hosting
again,
with
the
combination
of
containers
and
micro
services
and
the
container
orchestration
and
the
cluster
cluster
and
the
workload
management
which
is
taken
whereby
kubernetes.
A
Now
we
have
a
lot
of
moving
pieces,
we
have
a
huge
number
of
micro
services,
a
same
number
of
containers,
pods
nodes
data
centers,
and
there
are
applications
which
are
not
yet
moved
into
kubernetes
cluster.
They
are
still
running
on
bare
metal
in
their
own
private
data
centers.
So
the
amount
of
data
that
is
getting
generated
with
respect
to
the
logs
and
metrics
is
overwhelming,
and
these
data
that
is
getting
generated,
the
the
sheer
volume,
the
velocity
velocity
and
the
variety
is
simply
mind-boggling.
A
The
the
data
driven
the
data
different
approach
actually
affects
both
the
top
line
and
bottom
line
of
the
enterprise's
and,
as
per
the
study
in
2015-16
by
Enterprise
Management
Associates,
almost
65%
of
enterprises
use
more
than
10
monitoring
apps
to
keep
tab
of
their
applications
and
their
infrastructure,
and
the
same
studies
also
states
that
almost
71
percent
of
the
enterprise's
actually
spend
around
5%
hours
to
fix
an
average
performance
issue.
Now
that
is
quite
obvious.
A
In
your
hand,
how
can
you
go
about
and
try
to
figure
out
where
that
particular
problem
is,
and
this
is
that
it
becomes
very
imperative
for
enterprises
to
have
a
very
strategic
and
a
robust,
unified
monitoring
framework
in
place.
So
this
is
where
our
need
for
monitoring
solutions
came
in
like
as
I
have
mentioned,
we
there
is
a
hybrid
deployments.
There
is
a
high
data
volume.
A
It
has
a
customizable
report
and
dashboard,
so
we
want
to
have
at
least
a
single
window
through
which
a
business
or
an
ops
can
actually
go
into
the
monitoring
solution
and
figure
out
where
the
problem
is.
We
need
to
have
a
support
for
programmable
interfaces
again
with
respect
to
flexibility
of
enterprise
integration.
A
So
the
list
of
assets
that
we
are
monitoring
as
of
now
we
have
lists
in
the
infrastructure,
containers,
platforms
and
applications
from
infrastructure
perspective.
We
are
monitoring
most
of
the
bare-metal
virtual
machines,
IOT
devices
network
services,
like
HTTP
d,
then
from
container
platform
perspective.
It
is
the
kubernetes
docker
containers,
the
messaging
platforms
web
servers
and
from
application
perspective,
is
the
application
logs
application,
monitoring
frame,
monitoring
logs
and
even
the
events
and
traces.
A
A
So
again
we
are
taking
into
consideration
the
same
micro
services
use
case
typical
use
case
that
we
had
shown
in
the
previous
one
of
the
slide,
and
we
were
targeting
an
online
e-commerce
application
and
specifically
for
from
the
demo
perspective,
there
is
one
this
particular
API,
the
pradhan
for
API,
which
will
show
in
the
latest
stage
which
that
API
is
actually
orchestrated
through
multiple
services.
That
is
a
broad
detail,
pradhan
mantri
and
the
broad
review.
A
So,
from
the
conceptual
point
of
view,
like
all
the
monitor,
monitored
assets,
the
data
in
the
form
of
logs
and
metrics
are
pushed
through
the
exporters
into
the
data
acquisition
layer,
the
exporters
being
the
Zipkin
emitters.
The
elastic
beats
the
locks
strippers
and
in
through
the
data
acquisition
layer
consisting
of
the
Zipkin
collector
and
the
Kafka
lock
stash.
The
data
is
pushed
into
elastic
DB,
where,
finally,
it
is
being
utilized
by
Apache
spark
for
predictive
analysis
for
Zipkin
and
Cabana
for
visualization
and
by
Sentinel
for
alerting
and
reporting.
A
So
the
main,
the
first
main,
important
layer
of
our
solution,
is
the
ingestion
layer
which
consists
of
mainly
the
Zipkin
emitters
and
the
elastic
beats.
So
the
micro
services
are
both
fast
and
flexible,
but
in
case
of
a
performance
issue
which
comes,
and
it
is
very
tough
to
debug
and
do
any
kind
of
a
profiling.
This
is
the
reason
being
like
the
micro
services
or
a
single
service.
A
I
will
never
give
you
the
complete
picture
of
your
entire
application
and
in
a
normal
production
deployment,
of
where
these
services
are
having
a
very
complex
kind
of
an
integration.
So
it
is
very
tough
to
find
out
where
a
particular
service
is
deployed
in
which
particular
container
in
which
particular
pod
or
a
host.
So
this
is
where
distributed
tracing
helps
us.
So
whenever
a
request
passes
on
from
each
of
the
services,
metadata
is
actually
intake
injected
into
that
particular
request.
A
And,
finally,
those
information
are
aggregated
and
you
get
a
services
transaction
graph
in
almost
a
near
real-time.
So
our
emitters,
like
in
our
application,
the
demo
application,
which
consists
of
node.js
and
spring
boot
apps
we
are,
we
are
using
the
spring
sleuth
and
the
Zipkin
J's
components
for
emitters.
You
have
the
Ruby
emitters
and
there's
a
lot.
Many
emitters
provided
in
the
github
open
communities
which
can
be
utilized
as
per
your
application
needs,
which
are
now
pushed
into
the
Apache
Kafka
and
from
there
the
Zipkin
collector
picks
it
up
and
pushes
into
elasticsearch.
A
So
Zipkin
provides
the
capability
both
of
Kalat
collating
the
trace
information,
as
well
as
for
visualizing
them.
The
second
mode,
most
important
component
in
our
ingestion
layer
is
the
elastic
beats.
The
elastic
beats
are
open
source
edge
based
shippers,
which
are
written
in
golang,
so
they're
by
default.
Well,
quite
a
few
beats
are
available
made
available
by
elastic
like
the
metric
beat
file,
beat
packet
beat
heartbeat
or
beat.
These
are
the
few
default
beats
which
are
available
by
elastic,
and
you
can
also
look
into
the
community.
A
Driven
beats
where,
like
few
of
the
most
famous
which
we
have
utilized,
are
the
the
coffe
cup,
the
spar
with
the
spring
beats
and
the
Kafka
beat
so
now.
All
the
as
beats
we
can
understand
from
the
names
of
this
beat
metric
bit
is
more
for
capturing
the
infrastructure.
Metric
information
file
bit
captures
all
the
log
related
information
packet
bit
is
more
about
its
nips.
The
network
traffic
and
gathers
the
network,
metrics
information
and
those
are
finally
pushed
into
the
elastic
dB
and
so
elastic.
A
Db
is
again
an
open
source
text
search
analytics
engine
and
it
it
actually
can
store
a
huge
volume
of
data,
and
you
can
do
fast
searches
and
also
you
can
do
fast,
analytical
operations
on
the
data
that
is
getting
stored
in
it,
and
so
it
internally
uses
bkd
tree
algorithm
for
multi
node
multi
data
dimensional.
So
it
actually
helps
in
optimizing
the
disk
space
as
well
as
also
optimizes
the
indexing
and
the
search
that
can
be
made.
A
So
now
that
we
have
all
the
data
of
ship
from
our
api's
and
from
our
micro
services
and
our
kubernetes
cluster
containers
into
the
elastic
DB.
So
the
next
part
for
our
unified
monitoring
solution
is
to
actually
do
some
kind
of
predictive
analysis,
not
that
API
the
protein
for
API.
We
want
to
make
sure
that
in
case
there
is
an
issue
with
respect
to
the
the
latency
there's
a
high
latency
or
there's
a
huge
number
of
requests
that
is
giving
a
for
not
for
kind
of
a
status.
A
So
we
want
to
make
sure
that
it
is
more
of
a
proactive
nature
in
which
the
ops
team
are
notified,
not
when
the
actual
SLA
has
gone
beyond
the
required
limit.
So
this
is
where
we
are
doing
the
predictive
analysis
approach
in
which
we
are
using
apache
spark
ecosystem,
which
has
the
streaming
ml,
live
and
spark
SQL
component.
A
We
are
utilizing
the
pi
spark
in
the
spark
or
api
for
model
generation,
and
so
the
data
collection
is
already
done
by
the
elastic
search
and
the
pre-processing
is
done
by
the
spring
art,
the
spark
RDD
and
this
park
SQL
and
finally,
the
model
ml
model
generation.
We
are
utilizing
the
k-means
for
anomaly,
detection
so
and
then
the
data
that
is
being
so
we
have
a
time
window
once
the
model
is
generated
for
within
that
particular
time
window.
A
A
scheduler
runs
and
fetches
the
data
from
elastic
DB
and
send
it
across
to
the
model
to
infer
if
there
is
any
kind
of
an
outlier
that
need
that
is
present
in
the
data
in
that
particular
time
window,
and
then
it
pushes
it
on
again
back
to
the
DB
for
the
next
component.
That
is
the
Sentinel,
the
alerting
and
reporting
component.
So
Sentinel
is
again.
A
It
is
a
plug
in
node.js
plugin
for
Cabana,
which
again
falls
into
the
category
of
extreme
extensible
programmable
interface
that
we
are
actually
adhering
to
in
our
framework
and
it
actually
works
in
three
main
steps.
So
we
have
a
watcher
configured
in
our
Sentinel
which,
in
the
scheduled
interval
of
time,
sends
out
a
request
to
the
elastic
DB
with
a
script
and
events.
This
script
is
executed
and
then
we
can
actually
it
will.
A
It
will
either
give
a
positive
or
a
negative
response,
and
if
in
case
the
response
is
positive
or
if
there's
an
alert
that
need
to
be
triggered,
so
the
action
is
taken
and
alert
is
triggered.
You
can
also
configure
reports
in
Sentinel
and
schedule
the
interval
when
the
report
need
to
be
generated
and
send
out
to
the
ops
or
the
business
teams.
A
Not
with
you
know,
you
have
the
author
lab
with
you
know
it
may
be
in
your
email
and
so
using
that
particular
outlier
detail.
The
Sentinel
pushes
the
the
the
information
through
the
email
which
can
now
be
accessed
through
the
URL
to
the
Zipkin
visualization,
so
it
this
is
what
the
main
screen
in
the
Zipkin
UI,
which
actually
gives
us
the
service
dependency
as
well
as
you
can
capture
the
latency
with
respect
to
each
of
the
services
that
I
being
invoked
in
that
particular
request
and
Zipkin
has
a
direct
integration
with
Cabana
UI.
A
So
you
can
actually
then,
with
a
click
of
button,
you
can
go
to
Cabana
UI
asper,
the
URL
that
you
have
configured
it.
You
can
go
to
a
business
view
as
per
the
URL,
that
you
have
configured
or
to
an
ops
view.
So
so
now,
so
this.
This
is
how
we
actually
have
configured
all
the
components
in
our
stack
for
the
unified
monitoring,
so
I
know
jump
onto
the
solution.
Demonstration.
A
So
now,
as
we
have
seen,
that
the
main
API
that
we
wanted
to
monitor
was
the
the
the
pradhan
for
api
of
the
micro
services
use
case
that
we
had
picked
up
for
the
online
e-commerce
application,
so
we
will
cater
to
three
different
features
in
our
solution
demonstration.
This
is
more
of
a
require.
This
is
a
recorded
demonstration
because
there's
a
lot
of
moving
part
and
there
are
some
emails
that
need
to
be
accessed,
and
thus
we
have
recorded
this
demo.
A
So
the
first
one
would
be
more
about
the
the
unified
monitoring
of
containers
and
micro
services
in
a
kubernetes
cluster.
The
second
one
would
be
about
the
distributed
tracing
and
third
is:
we
will
identify.
We
will
introduce
an
anomaly
in
one
of
the
services
and
try
to
figure
out
through
our
unified
monitoring
framework
in
a
more
proactive
manner.
A
So
this
is
the
microservices
at
rest,
so
we
have.
This
is
a
Cabana
UI
at
as
of
now
the
cluster
is
the
kubernetes
cluster
is
done
so
we
have
all
the
metrics
shown
as
zero
like
the
nodes
that
the
deployment
containers
and
the
libel
pods,
then
we
will
start
up
our
the
kubernetes
cluster
and
we
have
provided
the
size
s3.
A
And
we
can
see
all
the
pods
and
all
the
different
deployments
that
are
present
in
that
particular
trust
cluster.
With
the
few
of
the
pods
in
the
cube
system
and
most
of
the
application-specific
pods
are
in
the
default
namespace
and
we
can
see,
there
are
three
nodes
up
and
running,
and
now,
if
we
go
to
our
the
kubernetes
overview
dashboard,
we
can
see
that
there
are
three
nodes
running
with
the
20
pods
and
14
deployments,
and
there
are
25
containers.
A
And
so
again,
this
kevanna
dashboard
is
highly
customizable.
So
we
have
picked
and
choose
those
particular
metrics
which
we
wanted
to
showcase
for
this
particular
solution
demonstration.
But
you
can
actually
get
other
charts
and
graphs
in
here
as
per
your
needs.
So
this
is
for
the
that
it
gives
us
the
top
nodes
by
memory
usage.
So
we
have
three
node
lustre
in
which
it
is.
A
We
are
having
the
three
nodes
with
the
memory
usage
Pro
displayed
with
them,
and
it
gives
us
the
information
regarding
the
net
by
node
and
network
out
for
each
of
the
three
nodes.
And
then
we
go
to
the
parts
like
the
second
tab
is
for
the
pods
overview,
so
it
will
give
all
the
parts
that
are
running
in
your
cluster
and,
on
the
right
hand,
side.
It
lists
outs,
all
the
pods
that
are
running
the
prod
review
service,
the
detailed
service
and.
A
Then
we,
since
we
are
looking
more
from
the
perspective
of
memory
and
the
CPI
utilization
in
our
use
case,
so
we
have
listed
on
the
top
memory
intensive
parts
and
the
top
CPU
intensive
parts,
and
this
is
the
cloud
tagging
of
all
the
contain
all
the
containers
that
are
running
in
our
cluster.
And
finally,
we
can
also
see
what
are
the
top
and
CPU
intensive
containers
and
the
top
memory
usage
for
by
the
containers.
So
again,
as
I
said,
like
all,
the
graphs
are
customizable.
A
So
now
we
can
go
to
a
port
for
a
specific
node,
so
we
will.
We
can
actually
drill
down
from
a
node
level,
go
to
a
pot
level
and
in
that
particular
part
we
can
actually
figure
out
what
are
what
which
container
is
running,
so
we
can
in
that
particular
node.
We
see
that
the
the
list
of
memory
intensive
ports,
the
Zeppelin
deployment,
the
prod
detail
service.
Similarly,
we
we
can
see
this
top
CPU
intensive
ports
and
in
that
particular
part,
the
the
one
which
we
have
selected.
A
The
prod
detail
service
is
the
container
that
is
running
and
it
gives
us
the
the
CPU
information,
as
well
as
the
memory
usage
information
for
that
particular
container
and
port.
So
now
we'll
go
into
the
distributed
tracing
part
wherein
so.
This
is
the
URL
which
which
is
accessible
for
the
pradhan
for
API,
and
now
once
we
have
make
up
quite
a
few
number
of
requests
on
that
particular
API.
We
can
go
to
the
Zipkin
and
a
zip
in
zip.
A
So
these
are
all
the
information
captured
by
these
pin
emitters
and
pushed
it
into
the
the
elastic
DV
through
the
Zipkin
collector
and
know
if
we
utilize
this
particular
trace
ID
and
go
into
our
the
container
tab
in
our
Cabana
dashboard.
We
can
actually
see
that
the
entire
information
is
also
captured
in
Cabana
with
respect
to
all
these
services,
and
we
can
see
on
the
right-hand
side
what
is
the
kind
of
the
services
whether
it
is
a
server
or
client,
because
internally,
those
services
are
calling
a
MongoDB
which
is
the
storage
for
them.
A
So
so
this
is
the
pradhan
for
service,
which
I
was
talking
about,
and
this
is
the
normal
trend
which
we
have
captured
for
the
last
twenty
to
fifty
days.
This
is
the
trend
which
we
have
simulated
through
jmeter
as
to
initially
in
the
in
the
any
initial
part
of
the
day,
the
the
we
have
a
quite
less
number
of
requests
that
are
coming
in
and
slowly
those
requests
are
going
up
and
it
is
more
about
the
HTTP
code
200
and
it
Peaks
up
at
around
4
to
5
p.m.
A
This
is
more
about
the
intuition
or
the
business
and
the
technical
team
who
can
come
up
as
to
decide
how
many
clusters
need
to
be
configured
for
your
particular
model,
and
now
we
have
the
clusters
configured.
We
have
four
different
clusters
with
the
centroid
available
in
them
and
then
we
will
configure
the
alert
in
a
sentinel.
So,
as
I
have
said,
the
Sentinel
has
Watchers
alarms
and
reports.
So
we
are
configuring
up
pradhan
for
service
anomaly,
which
will
run
in
every
five
minute
and
there's
a
script
which
it
needs
to
invoke
in
every
five
minute.
A
And
if
there
is
a
positive
response,
then
it
will
send
out
a
report
and
an
alert
as
configured
in
the
system.
And
finally,
we
will
introduce
some
anomaly.
So
this
is
my
prod
review.
Service
and
I
am
introducing
an
anomaly
with
respect
to
the
memory
conception.
I
have
written
a
small
piece
of
code
which
actually
runs
which
actually
eats
up
one
zero.
A
Alright,
as
of
now,
we
have
the
version
9.0
running
and
then
I
I
I'll
change
the
image
for
that
particular
deployment
from
9.0
to
10.0,
the
one
which
we
have
pushed
into
the
talker
and
make
some
it
requests
through
our
browser
and
even
through
jmeter
scripts,
so
that
we
are
creating
a
right
kind
of
anomaly
and
at
that
particular
time
through
the
Sentinel
and
alert,
is
sent
across
as
an
email
to
our
to
our
inbox.
And
here
we
have
to
do
two
emails.
One
is
the
report
and
other
is
the
alert.
A
A
This
is
the
actual
alert
for
that
particular
time
window
in
which
there
was
an
anomaly
detected
and
if
you
click
on
this
particular
alert,
it
takes
you
directly
to
the
second
UI,
and
here
we
can
actually
see
that
the
pradhan
for
service,
the
latency,
has
increased
from
0.78
millisecond,
which
was
earlier
to
around
13.
Second,
as
of
law
and.
A
And
we
can
we
have
listed
down.
We
have
figured
out
that
this
is
the
particular
part,
and
this
is
the
particular
container
which
is
which
is
creating
that,
but
that
outlier
now,
if
we
go
to
the
pods
overview
and
try
to
see
what
is
the
status,
so
we
can
see
that
the
prod
review
service
is
actually
consuming
around
2.2
GB
of
RAM
earlier
it
was
around
800
MB
and
then,
if
we
drill
down
further,
we
can
within
the
container.
A
Also,
we
can
see
that
the
CPU
is
picking
up
and
also
the
memory
is
pretty
high
compared
to
the
normal
scenario,
when
the
prodigy
service
was
working
in
a
correct
manner,
and
then
we
can
actually
roll
out
that
particular
image
which
we
had
deployed
from
10.0
to
9.0.
And
if
we
go
back
to
the
current
status
after
the
the
the
deployment
has
been
backed
out
from
10.0
to
9.0,
we
can
see
that
law.
The
proud
of
your
service
memory.
A
Consumption
is
within
the
limit
of
846
MB
and
the
other
CPU,
and
the
memory
metrics
are
also
well
within
the
range
so
yeah.
So
this
is
what
we
wanted
to
showcase
as
to
how
the
the
unified
monitoring
framework
we
are
utilizing.
All
the
the
the
component
with
respect
to
the
elastic
amana
Zipkin,
distributed
logging,
apache,
spark
and
helping
us
in
our
proactive
issue,
detection
and
also
for
monitoring
of
our
clusters.
A
Again,
like
with
respect
to
the,
we
wanted
to
monitor
the
entire
ecosystem
of
an
enterprise,
and
we
wanted
to
make
sure
that
even
the
visa
we
didn't
want
it
only
to
limit
ourselves
within
the
you
can
say
the
kubernetes
or
that
particular
ecosystem.
So
in
order
to
monitor
the
entire
enterprise,
which
can
have
private
data,
centers
applications
which
are
running
on
bare
metal-
and
we
also
had
a
very,
very
kind
of
very,
very
good-
you
can
say
the
performance
with
related
to
elastic
DB.
So
we
took
up
this
particular
approach.
B
Hello
over
here
over
here,
sorry
yeah.
Can
you
give
us
some
more
specific
examples
of
like
how
you
have
used
spark
machine
learning,
predictive
modeling
to
solve
the
problem?
I
hear
you
mentioned,
like
you
have
used
like
a
k-means
exact,
connects
in
a
business
terms
like,
for
example,
I
have
an
application
which
are
micro-services.
Can
you
speak
about?
A
So
so
so
what
we
targeted
this
was
you
can
say
that
this
is
not
a
multivariate
model
that
we
had
created.
This
was
a
single
variant
just
for
the
demonstration
purpose,
but
again
you
can
have
mobility.
You
can
have
a
combination
of
latency.
You
can
have
a
condemnation
of
HTTP
status
code
and
also
the
time
window
in
which
that
particular
data
data-
you
are
actually
capturing,
and
then
you
can
have
a
normal
trend
and
then
use
a
came
in
cluster.
A
Again
again
this
when
we
have
used
the
caming
clustering
in
Apache,
it
is
more
of
a
Spock.
It
is
more
about
the
intuition
with
which
we
we
capture
that
K
value.
So
so
for
us,
it
was
more
about
capturing
the
latency,
the
HTTP
status
code
and
the
time
window
in
which
those
particular
parameters
are
captured,
and
then
we
passed
it
on
to
create
of
specific
clusters,
and
this
is
more
about
the
application
status
at
a
particular
time
with
respect
to
latency
and
the
HTTP
code.
So
that
is
what
we
were
targeting.
B
A
Them
for
most
of
the
metrics
and
in
logs
there
are
default
dashboard
available
provided
by
Cabana,
and
if
you
want
to
be
more
intuitive-
and
you
want
to
get
more
insights
into
the
data,
then
you
can
easily
create
those
dashboards
with
all
the
metrics
and
the
logs
information
that
are
captured.
So
it
is
very
easy,
like
I
am
NOT
having
a
background
of
Cabana,
but
again
when
I
started
working
on
unified
monitoring,
it
was
pretty
intuitive
more
of
a
dragon
dog
kind
of
stuff.
So.
C
My
name
is
Andy
Santos
I
work
for
eBay
right
I'm.
Actually
one
of
my
team
actually
focuses
on
exactly
the
same
thing,
which
is
the
observability
right
so
coming
back
to
Cabana.
We
actually
keep
Ana
has
this
limitation
because
they
have
to
license
called
ex
back
so
they're
trying
to
get
from
the
open
source
into
the
into
more
commercial
right.
So
there
are
certain
plugins.
They
are
not
keeping
as
an
open
source,
but
we
internally
we
actually
contribute
it
back.
We
have
an
open
source
tab,
but
we
actually
created
a
lot
of
plugins.
C
For
example,
if
you
wanted
to
create
like
a
tail
command
prompt
with
a
little
banner,
you
don't
have
a
way
to
do
that,
but
we
created
a
plug-in
to
do
that.
It's
actually
pretty
simple
to
do.
You
can
use
JavaScript
to
do
a
lot
of
this
one
for
someone
other
one
is
that
we
find
it
interesting.
Is
that
you'd
want
it
to
do
a
heat
map
right
within
the
bigger
clusters
right?
How
do
you
see
like
the
density
of
the
deployment
stuff
like
that?
C
You
can
do
a
heat
map
and
we
have
the
code
that
does
that
will
be
more
happy
to
share
it.
But
the
other
thing
that
I
have
a
question
is
there:
what
is
the
skill
that
we
deal
with
and
I'm
actually
interested
to
see
how
we
can
potentially
collaborate
and
open
source
that,
because
we
actually
dealing
with
a
much
larger
scale,
there,
we've
done
this
architecture
peep
or
a
so.
Our
scale
is
about
10
million
matrix
per
second
okay
from
the
locks
eyes,
with
dealing
with
about
10
petabytes
a
day
and.
B
C
C
So
on
the
high
level
up
emoting
happy
to
go
in
details,
so
we
actually
split
this
into
two
different
stories.
One
is
a
hot
air
and
the
other
one
is
called
here
dry.
The
hot
air
is
anything
within.
Let's
say
two
weeks:
you
can
define
that
depending
on
how
much
one
are
you
willing
to
do
that
right,
yeah
and
then
the
coastal
rates
are
something
that
for
metrics
we
store
it
in
the
open,
tsdp.
All.