►
Description
How to Monitor Application Performance in a Container-Based World - Fabio Giannetti, Ken Owens, & Dave Roberts, Cisco
Application performance monitoring has always been a difficult problem for developers to implement, but it’s only getting harder. As application architectures shift to container-based microservices, the number of components typically rises and it becomes more difficult to understand how any given microservice is related to the total application performance profile. In this talk, we’ll discuss how Cisco developed a monitoring solution for Mantl, Cisco’s open source container stack based on Kubernetes. The solution leverages response-time data from Mantl and data from a number of open source projects, including Google cAdvisor, OpenStack Monasca, and Apache Kafka. We’ll discuss how we architected the solution, the features it provides, and why network traffic monitoring is critical when dealing with microservices.
About Fabio Giannetti
Fabio Giannetti brings 16 years of experience to his role as Principal Cloud Engineer at Cisco where he spearheads work on the company’s containerized, microservices platform. Prior to Cisco, Giannetti held multiple senior engineering and research positions at Hewlett-Packard, first focused on printing technologies and later on cloud computing. Giannetti has contributed to the OpenStack cloud computing project and written for multiple ACM, IEEE, and other publications. He is an inventor or co-inventor on 10 patents. Giannetti holds a master’s degree in Computer Science Engineering from Università degli Studi di Genova, in Italy.
About Ken Owens
Ken Owens is Chief Technology Officer, Cloud Native Platforms at Cisco. Ken is responsible for creating and communicating technical/scientific vision and strategy for Cisco’s cloud native technologies. He brings a compelling view of technology trends in enterprise IT (e.g. infrastructure, computing, SaaS, virtualization, and cloud) and evangelizes the technology roadmap for the business. Before joining Cisco in 2014, Ken spent over 7 years at Savvis as the Chief Scientist, CTO, and VP Security and Virtualization Technologies. Prior assignments include 5 years as a network security architect at A.G. Edwards & Sons, Inc., and Edward Jones brokerage firms in St. Louis, Mo, and 10 years in the design and architecture of communications systems and components for Erlang Technologies, Tellabs, and Wiltel. Ken holds bachelor’s and master’s degrees in electrical engineering from Missouri University of Science and Technology.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Today
we're
going
to
talk
a
little
bit
about
sort
of
what's
the
problem
statement.
What
are
the
motivations
in
terms
of?
Why
are
we
talking
about
this
with
at
the
cloud
native
con?
What
kind
of
provide
you
a
foot-post
solution
that
we've
been
working
on
within
their
cloud
native
community,
as
well
as
in
the
OpenStack
community,
and
then
we'll
give
you
a
few
key
takeaways
and
next
steps?
A
And
so
what's
the
problem
statement,
I
kind
of
like
to
look
at
problem
statements
in
terms
of
sort
of
a
user
story
right
so
as
an
application
owner
I
want
to
kind
of
create
my
application
to
do.
I
can
proven
and
be
the
best
that
it
can
become
and
most
secure
can
be
the
best
performance
it
can
have
and
want
to
expose
and
express.
A
My
like
into
my
sensitivities
or
my
my
application
intent
in
a
set
of
like
so
like
business
level,
objectives
or
business
statements,
and
then
we
have
to
kind
of
monitor
the
what
my
expectations
are:
the
application
from
a
kind
of
a
business
objective
against
now
it's
actually
performing
in
the
environment
and
make
real-time
decisions
and
actions
against
that,
and
so
some
of
the
motivations
are.
You
know
pretty
simple
right.
A
Let's
look
at
you
know
how
do
I
sort
of
provide
real-time
performance
and
scalability
of
an
application
against
what's
happening
against
that
application
in
real
time
and
then
the
difficult
part
is
sort
of
how
do
we
collect
the
data?
How
do
we
measure
the
actual
performance
of
this
of
the
application
when
you're
deploying
as
a
set
of
containers,
because,
as
you
know,
containers
are
individual
small
pieces
of
source
code
running
in
different
locations
and
they're
loosely
coupled
and
so
be
able
to
kind
of
determine
how
your
application
is
performing?
A
When
you
have
a
bunch
of
micro
services
running
all
over
the
place,
it's
not
a
super
easy
thing
to
do.
So.
How
do
you
sort
of
collect
the
data
aggregate
that
data
and
get
a
picture
of
how
the
actual
applications
before
I
mean
not
how
container
one
and
container
through
yoga
forming
but
the
actual
appliqué
itself?
A
And
so
we
prototype
this
in
something?
Then
we
call
it
Cisco
shipped.
It
said:
Cisco
shipped
I,
oh
it's
freely
available
to
use
and
playing
around
with
this
sort
of
like
a
if
you
will
kind
of
open
sourcing,
a
testbed
for
figuring
out
how
to
make
applications
before
and
better,
and
so
it's
open
for
you
guys
to
use
and
give
us
feedback
on.
A
This
is
so
what
the
interface
looks
like.
This
is
the
one
of
the
configuration
tabs
in
deploy
and
so
the
whole
eye.
The
whole
idea
of
shipped
was
so
looking
at
when
you
look
at
the
sea
and
the
CNC
F
definition
of
cloud
native.
This
is
sort
of
the
micro
services
architecture
piece.
How
do
you
compose
your
application,
how
you
develop
and
build
your
services
and
connect
to
other
services,
and
so
that
you
know
the
kind
of
the
building
peace
and
the
deployed
piece
are
independent
of
each
others.
A
You
can
develop
your
application
on
your
laptop
or
in
a
set
of
services
in
the
cloud,
and
then
we
go
to
deploy.
You
can
create
different
deployment
environments
and
as
part
of
that
deployment
configuration
in
addition
to,
like
the
you
know,
the
I
as
provider
and
some
of
the
information
around
you
of
your
deployment
environment.
You
can
go
into
this
configuration
tab
for
policies,
and
the
idea
here
is
that,
as
you
can
see
on
the
screen,
you
have
sort
of
average
load.
You
have
memory
usage,
you
have
sort
of
the
you
know.
A
The
numbers
amount
of
network
bandwidth
is
being
consumed.
You
sort
of
have
more
what
I
call
kind
of
business
business
logic
around
what
the
application
is
supposed
to
be
doing,
and
it's
more
on
the
sensitivity
of
the
application.
It's
not
a
specific
number.
You
can
define
specific
numbers,
but
the
whole
at
the
point
of
this
is
sort
of
defining
more
on
the
sensitivity.
A
I'll
get
more
of
that
in
a
second
and
so,
on
the
right
hand,
side.
We
have
policies
that
you
define.
So
this
is
like
a
simple
scale-up
policy
that
when
you,
when
your
memory
hits
a
certain
percentage
in
this
case,
seventy
five
percent-
you
would
scale
up
the
number
of
containers
needed
to
support
it.
You
have
this
limit
this
graph.
A
You
know
in
this
case
I
want
to
maximize.
You
know
the
sort
of
like
Network
thresholds
or
the
security
thresholds
and
I
want
to
kind
of
minimize.
You
know
my
limits
on
the
letting
you
know
the
load
average,
and
so
you
can
kind
of
move
these
dots
around
and
create
your
own
sensitivity
per
application
and
at
the
bottom
of
the
of
the
page,
is
where
you
define
actions,
and
so
we
kind
of
showed
you
a
scale
down
action
that
we
defined
here
and
so
the
whole
point
of
this
is
as
you're
developing
your
application.
A
Have
you
know
a
number
of
you
know
amount
of
usage
right
now,
that's
causing
my
application
to
need
to
scale
up
now.
That
period
is
over
and
I
can
scale
it
back
down
automatically
without
any
user
intervention,
and
so
that's
sort
of
the
in
the
deployed
tab.
Configuring,
the
application
policies
when
you
get
to
the
run
tab
is
a
little
bit
different
now.
This
is
where
you
can
kind
of
look
at
for
this
environment
I've
deployed
into
which,
in
this
case
is
a
testing
environment.
I
won't
start
looking
at
certain
parameters
right
and
so.
A
First
off.
We
want
to
like
kind
of
point
out
how
many
containers
are
running
and
then,
as
you
scale
up,
you
see
the
number
of
containers
increase,
as
is
scaling
from
like
two
to
three
containers.
You
see,
you
know,
it'd
be
like
a
red
circle.
Saying
hey
you
know,
half
of
them
are
up.
The
other
half
are
coming
up
once
they're
all
up.
It
turns
out
green.
Like
you
see
here,
you
also
have
the
different
environments
you
can
deploy
into,
and
so
a
big
part
of
this
application.
A
You
know
model
that
we
wanted
to
push
out
into
the
industry
was
that
you
shouldn't
have
to
like
change
your
policies
just
based
on
what
you're
deploying
to
if
you're,
deploying
to
your
test
environment
or
your
QA
or
your
production.
You
want
your
policies
to
function
the
same
way,
regardless
of
what
environment
you're
in
and
so
you
can
apply
the
same
policies
across
multiple
environments
in
the
exact
same
way
and
then
obviously
there's
a
lot
of
micro
service
specific
pieces.
A
You
want
to
be
able
to
monitor,
like
in
this
case,
we're
showing
CPU
and
memory
and
disk,
and
then
the
data
were
trying
to
monitor
in
this
case
was
the
network
type
of
utilization
and
network
load?
And
so
that's
what
was
causing
you
know
in
the
in
this
case
we're
looking
at
memory
usage
in
network
at
the
same
time
and
so
from
an
architecture
standpoint
to
kind
of
give
you
an
idea
of
what
we're
going
to
show.
A
You
have
we
created
an
app
intent
engine
to
kind
of
manage
and
create
policies
and
do
analytics
around
the
policies
and
the
main
things
we're
trying
to
do
without
app
intent.
Engine
is
this
kind
of
scheduled
actions
to
look
at
the
security
aspects
around
the
actions
to
do
alarm
and
notification,
so
it
doesn't
just
scale
up
your
set
of
services
without
telling
you
this
event
caused
me
to
scale
up
and
then
trying
to
compound
these
different
methods,
metrics
and
actions
across
all
of
your
different
container
services.
A
You
deployed
now,
while
leveraging
like
kubernetes
a
marathon
for
sort
of
that
action
piece
that
go
back
into
the
system.
We
also
wanted
to
kind
of
tie
this
into
the
social
piece
of
it,
so
we
leveraged
Cisco
spike,
which
is
a
collaboration
tool
kind
of
like
slack
but
ties
into
WebEx
as
well.
So
you
can
kind
of
create
like
meeting
rooms
and
do
video
conferencing
and
there's
an
issue,
so
we
sort
of
also
send
alerts
into
what
we
call
a
spike
room.
B
Thanks,
okay,
so
how
we
always
solve
this?
We
we
did
three
things,
because
in
order
to
have
a
comprehensive
and
understanding
of
your
application,
you
need
to
get
sources
from
different
ways
and
different
places.
So,
in
order
for
us
to
compare
apples
with
apples,
what
we
did
we
used
a
unifying
measurement
for
months
and
we
didn't
want
to
reinvent
the
wheel.
So
we
use
something
that
I
used
to
work
with
in
OpenStack
and
there
is
a
format
very
similar
to
matrix
2.0.
B
So
there
is
a
standard
and
the
project
in
OpenStack
is
called
Manassas
who
use
the
manasa
format.
It's
a
very
simple
format
to
describe
measurements,
so
we
use
a
single
measurement
for
mark
across
the
bar.
The
other
thing
we
do.
We
use
a
measure
and
emboss
why
we
want
to
do
that,
because
we
want
to
have
the
ability
to
send
measurements
and
have
other
tools
to
collect
those
measurements
and
operate
on
those
re-inject
them.
So
then
we
can
keep.
B
Data,
update
and
label
enable
alarming
in
a
label
our
function
for
the
lazarum
bass
will
actually
use
Kafka,
since
it's
a
very
strong
and
robust
bus
itself
and
then
compound
measurement.
That's
the
other
interesting
thing,
so
we
understand
and
all
the
metrics
will
be
enough
to
understand
arms
or
to
create
actions,
and
so
some
of
those
needs
to
be
modified
by
other
components,
to
create
what
we
call
compounds.
B
So
this
is
an
example
of
the
Monash
convention
manager.
So
you
see
that
there
is
a
name
of
the
measure
which
would
be
your
metric
right
and
then
there
is
a
timestamp
and
value
and
in
the
middle
that
is
what's
called
dimension.
Those
dimensions
are
the
specific
in
no
specific
labels
that
you
want
to
push
into
the
metric.
So
these
come
from
the
darker
labels.
They
come
and
we
inject
we
found
out
injecting
at
least
four
of
them
project
ID,
which
is
equivalent
of
a
talent
ID.
B
If
you
wish
service
ID
refers
to
the
micro
service
that
belong
to
the
project,
which
will
be
the
application
and
in
the
environment
talking
about
like
you
know,
this
is
this
development
or
this
staging
and
then
the
host
where
they
are
running,
we
can
correlate
a
lot
of
data
and
we
can
do
a
lot
of
things
with
this
simple
label
so
couch
you
can
attach
more
and
an
integrate
more.
What
we
discovered
is
with
four
you
can
do
already
a
lot
and
so
go
back
a
little
bit
into
the
details.
B
In
flux,
DB,
and
if
you
know
a
little
bit
about
time,
series
data
bases
tags
basically
restore
once
per
time
series.
So
they
don't
occupy
space
and
they
will
become
very
very
fast
to
to
query
and
then
because
in
flux,
DB
already
has
a
plug
in
form
graph
are
basically,
we
can
leverage
there
and
go
and
do
the
views
I'm.
Sorry,
a
little
bit
out
of
focus
does
when
I
scale
them
up
and
they
I
can
query
in
flux
to
be
through
graph
on
and
I
can
create.
B
B
The
other
thing
is
alarming,
so
I
can
create
thresholds
on
any
of
these
metrics
right,
but
only
there
once
these
metrics
are
identified.
Now
you
see
that
my
application
in
turn
engine
with
alarming
works
out
of
the
bus
itself,
so
I
don't
need
to
wait
for
the
data
to
be
stored
in
the
database
before
I
cannot
create
an
alarm.
So
in
this
way,
I
have
really
inline
alarming,
which
will
happen
at
the
time
the
measure
has
been
written
and
know
by
the
time,
its
database.
Why?
B
Because
I
want
to
decouple
the
time
it
will
takes
to
ride
the
data
to
the
database,
so
one
one
I
should
do
is
I
want
to
make
sure
that
I
ride
a
huge
amount
of
data
in
a
single
go,
so
I
got
performance
from
the
database
and
I.
Don't
wait
that
time
if
something
is
happening,
mining
for
structure
in
the
mean
time
so
there
I
can
write
things
like
this.
B
B
Cpu
memory
usage
is
50%
in
any
given
day
of
the
week.
50%
is
good
or
it
could
be
a
problem,
but
when
I
reach,
maybe
the
weekend
or
when
I
have
I,
know,
I
expect
more
workload,
then
that
50%
is
actually
not
a
big
deal
anymore.
So
what
I
can
do
is
that
I
can
have
these
policies
that
gonna
add
some
data
and
based
on
this
extra
information,
create
a
new
metric
and
then
the
new
metric
will
be
the
one
that
will
be
evaluated
for
a
year
real
alarm
or
no.
B
The
next
step
we
did
is
you
know
in
order
to
understand
how
your
application
behave,
especially
around
microservice
is
not
enough
to
just
have
the
traditional
measurements
from
you
know:
CPU
network
I/o
and
blah
blah
blah.
You
also
need
to
understand,
what's
going
on
at
the
load
balancer,
because,
depending
on
the
response
time
that
you
get
from
your
microservices
there,
it
could
be
effect.
Domino
effects
on
yours
on
your
application
right
right
now,
microservice
is
responding
slowly
than
used
to
and
they
will
create
some
timeout
in
other
micro
services
and
it
can
create
some
problem.
B
So
what
we
did?
We
stand
it
traffic,
which
is
the
load
balance
and
using
our
solution
and
again
we
added
from
traffic
the
ability
to
generate
the
same
metrics,
but
this
time
on
on
the
Kafka
bus
itself
and
on
traffic
we
can
get
a
project
ID
from
the
messaging.
We
can
get
a
service
ID
or
we
can
get
the
container
ID
because,
of
course,
load
balancing.
We
don't
know
from
which
container
the
traffic
is
routed
or
not,
but
that
didn't
stop
us
to
make
the
major
connection
between
the
service.
B
So
we
can
correlate
the
data
and
so
adding
this
to
the
pot
again,
you
know
the
load
balancer
can
send
the
data
there
and
doesn't
change
anything
to
our
pipeline
because,
of
course,
is
in
the
same
formats.
It's
consuming
this
before
is
just
the
new
metric.
It's
going
to
be
shown
and
it's
new
metrics
that
can
be
displayed
on
the
graph
on.
So
now
we
have
the
ability
to
get
response
codes.
So,
for
instance,
there
are
a
lot
of
500
in
a
short
amount
of
time.
B
Something
is
going
real
wrong
or
you
know
for
hundreds
or
403
for
unauthorized.
So
someone
is
trying
to
do
something
spooky
or
maybe
simply
assert
it's
a
spider.
That's
why
what
Microsoft
cannot
assess
another
one
anymore?
So
we
are
now
are
able
to
identify
all
this
situation,
simply
creating
alarms
around
those
situations,
and
then
we
have
the
response
time.
So,
if
you
see
spikes,
we
can
capture
those
and
say
hey.
You
know
this
should
be
brought
to
attention
and
then
we
can
correlate
a
spike
with.
B
Maybe
that
particular
micro
service
and
the
micro
service
may
have
some
memory
issues
increase
in
memory
or
in
CPU
usage,
and
we
can
do
all
this
sort
of
correlations
and
then,
lastly,
we
were
working
on
doing
security
scan.
We
started
with
host
security
scan,
so
we
had
another
project
in
Cisco
Colorado
and
they
were
kind
to
work
with
us
on
basically
producing
the
same
matrix
in
the
same
format.
B
What
they
will
do,
they
will
periodically
go
and
scan
the
hosts
of
the
kubernetes
or
of
the
Mises
cluster,
and
if
there
were
some
problems,
they
had
like
a
bunch
of
scans
that
will
report
that
we
returned
a
value
from
zero
to
one
saying:
zero
is
okay
one.
There
is
a
problem
and,
depending
on
the
value,
it
will
the
severity
right.
One
was
the
worse
than,
and
0.5
means
some
for
the
test,
what
if
they
ended
all
these
kind
of
stuff.
So
now
we
could
use
that
to
do
things
like.
B
Oh,
that
host
didn't
pass
the
security.
There
is
something
dodgy
about
it.
Maybe
we
go
and
just
stop
to
use
it,
and
then
kubernetes
will
move
the
container
out
or
in
case
of
situation
where
it
wasn't
as
severe
you
get
something
like
0.2
0.3.
What
you
could
do
you
could
say
well,
at
this
point,
I
have
some
application.
There
will
be
more
sensitive
to
security
issues,
so
just
go
and
talk
to
the
to
the
to
the
scheduler
and
create
an
anti
affinity
rule
for
that
particular
host.
So
I
don't
want
to
take
any
risk.
B
B
We
never
got
to
the
point
of
doing
security
scan
on
the
container
life,
but
that
was
our
next
objective
to
go
and
scan
the
containers
themselves.
So
if
you
look
at
a
complete
solution,
it
looks
like
this
right
now.
You
have
the
adviser
for
the
container
themselves,
the
load
balancer
you
can
have
both
host
and
container
vulnerability
scans.
You
can
have
collecti
to
take
the
host
matrix
and
then
you
know,
if
you
ran
on
an
open
stock
infrastructure,
you
can
leverage
there
or
VMware
or
any
of
the
other
public
clouds.
B
B
So
I
think
what
we
discovered
is
there,
as
a
mention
at
the
beginning,
we're
very,
very
we've
standardized
in
the
format
and
adding
few
labels
to
our
micro
services
and
applications.
We
were
able
to
do
a
lot
of
correlation
of
the
data
and
we
done
root
cause
analysis.
I,
don't
think
we
are
there
yet
I.
Don't
think.
Is
that
easy?
But
definitely
we
did
we
struct
a
lot
of
mileage,
just
just
taking
care
of
the
architecture
in
their
way
and
applying
very
simple
solutions,
and
so
that
the
corrective
actions
we
can
take.
B
B
What
should
be
the
corrective
actions
for
the
application
to
be
run
correctly,
and
we
want
the
IT
department
to
set
limits
within
what
can
be
done
for
it
says
you
cannot
scale
infinitely
or
you
know
you
can
now
take
a
scale
up
to
an
infinite
amount
of
memory
of
things
like
that,
but
beside
there
should
be
really
application
owner
to
spend
the
cards
in
the
right
place,
use
them
in
the
right
place,
and
then
you
know.
Yes,
the
ability
to
see
a
monitor
how
the
application
is
working,
how
evolves
and
what's
happened
so
next
step.
B
So
as
as
Ken
mentioned
this
technology
is
a
POC
is
meant
to
illustrate
the
capabilities.
What
can
we
do?
I
said
what
we're
really
looking
for
is
to
you
know,
make
this
concert
accepting
the
industry,
and
you
know,
taking
as
a
way
forward
for
application
owner
and
for
defining
high
level
application
description.
B
So
if
you
look
at
a
keynote
today,
they're
aware
all
those
layers
that
the
Community
Foundation
is
defining
the
very
top,
there
is
a
cost
allocation
and
that's
the
part
that
we
are
looking
here
and
so
now
all
of
the
features
that
I
described
we
are
implemented.
But
it
is
a
good
chunk
and
you
can
go
and
test
them
and
you
know,
can
mention
are
available
at
Cisco
ship
to
I/o.
So
you
can
go
there
and
create
very
simple
applications,
and
then
you
can
define
those
policies.
B
You
can
have
at
them
a
run
time
and
you
can
play
with
the
policy
the
scaling
and
all
these
kind
of
stuff
and
I.
Think
what
we
all
really
like
to
do
is
to
see
what
is
the
space
within
the
parameters
community
to
basically
take
an
approach
similar
to
this
and
make
sure
that
you
know
we
have
a
unified
matrix
with
a
path.
So
we
create
an
ecosystem
where
other
players
can
consume
matrix
and
produce
matrix
on
the
bus.
And
you
know
we
enable
all
sort
of
complex
computation
compound
computation.