►
Description
You’re Monitoring Kubernetes Wrong - Loris Degioanni, Sysdig
"I hate to say it, but there’s a lot of bad advice out there on how to monitor your containers in production. In this talk I hope to add some much-needed clarity on how to best monitor containers to manage the health of your applications.
My advice is built off our experience creating container monitoring solutions, and the information we’ve gotten from working with hundreds of customers deploying Docker. I’ll cover key questions like:
*Why is it so hard to get visibility into Docker containers?
*How far can you get with the Docker stats API?
*What metrics really matter for your containerized applications?
*How does Kubernetes improve monitoring? How does it get in the way?
*What open source tools can help with these challenges?
In addition to this theory, I’ll get into some real examples that will ground our discussion. You’ll walk away with a new appreciation of what it takes to monitor your environment right, as well as a few ideas that you can put into practice right away."
About Loris Degioanni
Loris Degioanni is the CEO and founder of Sysdig. He is the creator of the popular open source troubleshooting tool sysdig, and the open source container security tool Falco. Prior to founding Sysdig, Loris co-created Wireshark, the open source network analyzer, which today has 20+ million users. Loris holds a PhD in computer engineering from Politecnico di Torino and lives in Davis, California.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
First
of
all,
I'm
going
to
show
you
how
to
build
your
kubernetes
monitoring
system.
So
you
start
with
data,
can
be
anything
application,
infrastructure
system
process.
Just
all
of
your
data,
you
pipe
it
into
a
big
honking
data
engine,
and
then
you
just
get
unicorn
Aran
and
Rainbow
Dash
ports
that
simple
many
people
that
get
involved
with
kubernetes
or
any
modern
infrastructure
very
often
make
this
assumption.
It
turns
out.
Sometimes
the
unicorns
don't
really
show
up
as
expected.
So
that's
what
we're
going
to
talk
about
today.
A
A
Let's
start
with
a
bit
of
context
and
a
bit
of
where
were
coming
from
so
the
keynote
this
morning,
sort
of
created
a
good
timeline
of
how
things
are
moving
and
we're
definitely
in
the
middle
of
something
quite
revolutionary,
so
I'm
putting
three
things.
You
know
you
know:
PCs
servers,
virtual
machines
and
containers
and
what's
interesting
to
me,
is
that
this
time
things
many
things
are
changing
quite
dramatically.
In
particular,
the
unit
in
the
server
era
was
the
Machine.
A
The
unit
in
the
water
machine
era
was
still
essentially
the
Machine
a
virtualized
version
of
the
machine,
but
you
were
still
working
with
a
complete
host
that
a
complete
operating
system.
Now
the
unit
is
the
service.
The
the
component
of
your
application
in
what
Yorkist
rating
are
just
little
pieces
of
an
application,
not
machines
anymore
orchestration
was
none
for
the
server's
orchestration
for
virtual
machines
was
essentially
its
piece
that
you
were
bolted
on
and
in
fact,
that's
one
of
the
reasons
why
we
don't
have
a
standard
for
did
while
orchestration
now
with
containers
is
native.
A
Essentially,
that's
why
we're
here
because
the
two
things
are
evolving
together
and,
of
course,
you
know
we're
going
from
monolithic
infrastructure
infrastructures
to
service
base
infrastructures.
This
means,
of
course,
the
things
are
becoming
more
complicated,
more
complicated
for
many
reasons,
they're
becoming
much
better,
of
course,
but
they
are
also
becoming
more
complex.
In
particular,
these
beautiful
slide
is
my
artistic
representation
of
a
three-tier
application
with
a
cache
and
the
web
server
and
the
database,
and
this
is
how
we
were
used
to
think
about
our
our
applications.
And
now
we
have
more
something
like
these.
A
Where
we're
services,
the
teruki
straight,
is
so
each
of
the
gray
boxes
here
is
essentially
now
the
machine,
physical,
virtual
and
the
application
components
move
around
on
these
machines
to
implement
services
essentially,
and
not
only
they're,
orchestrated
in
arbitrary
places,
but
they're
often
moved
to
a
different
place.
So
you
end
up
with
something
that
can
be
overwhelming,
especially
if
you
want
to
understand
what's
happening.
If
you
want
to
monitor
it
and
if
you
don't
adapt
to
this
new
way.
A
The
thing
that
is
especially
exciting
for
me,
however,
is
how
revolutionary
something
to
like
kubernetes
can
be,
not
only
from
the
technical
point
of
view
yeah,
we
will
be
used
to
think
about.
You
know,
load,
balancers
and
web
servers
and
databases
and
replication
controllers
and
pet
sets,
and
all
this
kind
of
stuff
and
of
course,
the
technical
side
is
very
exciting
and
in
its
progress.
A
Couple
more
distributed
and
more
people
based,
so
how
do
all
of
these
things
impact
monitoring
and
monitoring
our
infrastructures
I'm,
going
to
focus
on
four
things
that
I
believe
are
becoming
harder
nowadays,
number
one
is
getting
the
data,
definitely
harder
now
with
all
these
moving
parts
and
also
with
the
pigness
of
containers,
number
two
making
sense
of
the
data
number
three
troubleshooting.
So
what
do
we
do
when
something
is
going
wrong
to
understand
the
root
cause
and
number
four
people
so,
starting
with
getting
the
data?
What
make
makes
container
great?
A
They
are
simple,
they're
small
they're,
isolated.
They
contain
only
the
tendencies
they
they
really
need.
So
what
I
was
saying
before
you
know?
We
started
by
considering
container
sort
of
lightweight
operating
systems
that
could
be
started.
You
know
faster
than
a
than
a
full
bottle
machine.
Well
in
practice,
they're,
not
in
practice.
We
want
to
keep
them
as
small
as
possible.
We
want
to
keep
them
as
simple
as
possible
because
we
want
to
be
able
to
download
them
quickly
start
and
quickly,
move
them
move
them
quickly.
A
So,
and
also
you
know,
keep
it
as
simple
as
a
bunch
of
benefits
that
we
are
all
aware
of
only
problem
keeping
their
simple
also
tends
to
make
things
a
bit
more
opaque.
So
typically,
there
are
two
approaches
that
I'm
going
to
describe
for
getting
inside
and
extracting
metrics
or
doing
troubleshooting
inside
containers.
The
first
one
is
pulse,
so
this
is
a
primitive
that
kubernetes.
That
is
designed
exactly
for
this.
So
essentially,
you
have,
for
example,
year
two
two
containers
running
on
top
of
the
of
the
same
machine.
What
you
do
is
you've
wrapped
them.
A
A
Another
way,
which
is
the
one
we
use
in
systick,
is
based
on
instrumenting,
underneath
the
containers
so
containers
at
this
point.
I'll
just
left
the
well,
they
were
there
and
what
happens
is
there's
a
component
in
the
operating
system,
a
kernel
module
the
seeds
underneath
these
containers
captures
the
data
and
by
hooking
a
couple
of
places
in
the
kernel
level,
in
particular,
for
example.
This
is
the
system
:
enter
and
exit
interface
in
the
kernel.
A
This
means
that
the
red
circles
are
able
to
capture
stuff,
like
storage,
inter-process,
communication
network
activity,
resource
use,
a
and
but
essentially
everything
that
that
each
of
the
containers
does
when
it
talks
to
to
the
operating
system.
At
that
point,
there's
one
container,
which
is
this
is
the
container
which
is
in
charge
of
collecting
this
information
and
doing
stuff
with
it.
I
will
show
you
a
couple
of
demos
in
the
later
part
of
my
presentation.
A
A
A
This
is
the
container,
takes
care
of
instrument
in
the
operating
system
kernel
and
then
takes
care
of
starting
extracting
this
information
and
collecting
it,
and
this
brings
me
to
some
guidelines
and
talking
about
stuff
that
very
often
people
do
wrong
and
by
the
way,
probably
if
there's
something
that
you
want
to
bring
home
from
representation.
This
slide
is
the
most
important
one.
So
you
should
not
be
involved
in
instrumenting
for
monitoring,
so
I
was
showing
you.
A
You
know
the
approach
that
we
have
a
system
that
works
by
saturating
it
and
making
sure
that
one
of
these
containers
runs
in
your
infrastructure
and
that's
it
so
in
a
general
way.
There
are
ways
in
kubernetes
can
help
us
making
deployment
of
monitoring
really
transparent
and
making
configuration,
maybe
not
completely
unnecessary,
but
at
least
the
bare
meaning
in
particular.
Ideally,
you
should
not
be
involved
in
producing
anything
that
is
not
a
custom
metric.
What
I
mean
is
everything
else.
You
know
infrastructure.
A
Basic
metrics,
like
CPU
memory
and
disk
application,
matrix
data
base
matrix
response
times.
All
these
kind
of
stuff
can
be
collected
automatically
and
should
be
collected
automatically.
It's
not
only
nice
to
have,
because
you
know,
of
course,
the
less
work
you
do,
the
better
its,
but
nowadays
it's
a
muster
because
in
heavily
instrumented
instrumented
infrastructure
and
an
overly.
A
Orchestrated
infrastructure
there's
just
no
time
to
go
and
worry
about
the
second
measure,
so
the
only
metrics
that
you
should
pipe
out
are
the
ones
that
you
produce
manually.
We
should
be-
hopefully
you
know
a
limited
set
because
a
bunch
of
stuff
is
already
collected
for
you.
The
other
thing
is
every
single
metric
should
be
taped.
A
This
is
very
important
because
there's
no
way
to
make
a
sense
of
this
stuff
unless
you
have
proper
taking
the
reflects
the
logical
organization
of
your
infrastructure,
let
me
show
you
what
I
mean
for
a
second,
so
here
I
have
Imus.
Is
the
cloud
and
I'm
just
selecting
one
of
the
namespaces
and
now
I'm
charting
a
metric,
which
is
the
a
request
time
accession,
the
response
time
for
this
service.
A
By
I
don't
know
replica
set
and
at
this
point
I'm
able
in
Disney
space
to
see
that
the
difference,
the
different
response
times
for
the
different
replicas
sets
that
I'm
running
or
maybe
I
want
to
do
it
by
something
like
pod
dot
name
and
here
I'm,
seeing
essentially
out
the
response
times
of
the
different
pods.
It
is
infrastructure
compared
to
each
other.
Maybe
I
want
to
do
it
by
image.
A
So
this
ability
to
segment
stuff
first
of
all
needs
to
reflect
how
kubernetes
organized
things.
So
what
what's
happening
here?
The
metric
is
collected
from
any
container.
These
containers
move
around
in
your
infrastructure,
but
all
these
containers
are
a
tags.
You
know
some
tags
are
explicit
because
you
give
them
to
your
containers
and
some
tags
are
implicit
because
they
they
can
be
essentially
collected
from
from
kubernetes.
All
of
these
is
put
together,
and
then
this
metric
is,
you
know,
aggregated
in
different
ways,
depending
on
the
aggregation
that
you
do.
Tagging
is
critical.
A
It's
what
really
makes
a
difference
in
a
modern,
modern
monitoring
tool.
Tagging
must
be
integrating
with
kubernetes,
so
any
tool
that
doesn't
talk
natively
to
kubernetes
to
extract
the
tags
is
not
a
good
tool
and
by
the
way,
there's
there's
quite
a
bit
of
work
from
both
vendors
in
community.
To
make
this
work
well
again,
I
showed
you
an
example
with
60
cloud,
but
there
you
know
just
make
sure
that
your
tool
does
that.
The
other
thing
that
is
very
important,
that
is,
you,
should
not
be
involved
with
tagging.
A
The
amount
of
information
that
you
need
to
send
to
your
data
engine
by
the
way.
Every
time
you
take
a
metric,
you
essentially
make
a
duplicate
of
the
dimetric
right.
So
if
metrics
are
tagged
by
default-
and
you
have
you
know
like
tens
of
tags
for
each
of
the
matrix,
essentially
you
need
tens
of
the
storage
and
tens
of
the
responsiveness
in
ten
times
the
responsiveness
in
your
monitoring
tool.
A
So
it's
not
trivial,
but
on
the
other
hand,
the
problem
is
the
more
complex
the
difference
infrastructure's
become
and
the
more
having
all
of
the
data
is
very
important.
So
you
want
to
be
able
to
do
this
kind
of
stuff
interactively
like
I,
showed
you
and
not
decide
a
priori.
What
kind
of
segmentation,
or
what
kind
of
dashboards,
or
what
kind
of
data
you're
going
to
export?
All
of
the
data
should
come
with
all
of
the
tags,
and
then
you
can
use
it.
You
know,
after
the
fact,
which
brings
me
to
troubleshooting.
A
So
another
thing
that
is
becoming
more
challenging
is
okay.
You
know,
I
seen
the
trend
of
my
key
performance
indicators,
I'm
receiving
alerts.
I
know,
there's
something
wrong
now.
What
now
it's
not
trivial
because
again,
containers
are
opaque
and
they
make
troubleshooting
even
harder
than
the
metric
collection.
In
particular.
A
Containers
are
designed
to
be
small,
so
all
of
the
tools
that
we
were
used
to
rely
on
you
know
you
SSH
into
the
Machine.
You
do
a
nice
trace.
You
do
a
PS,
you
do
a
top.
You
do
whatever,
if
config
all
of
that
very,
very
likely
you're
not
going
to
have
it
and
it's
going
to
be
complicated,
essentially
to
go
and
understand
how
things
work.
So,
let's
go
in
a
shell
for
a
moment
and
I'm
going
to
run
our
open
source
tool.
A
So
that
takes
makes
use
of
the
same
technology,
a
kernel
module
underneath
the
containers
that
I
showed
you
in
the
previous
slide.
What's
interesting
here
is,
as
you
can
see,
I'm
executing
this
command
with
a
parameter
that
points
it
to
my
local
kubernetes
master,
which
means
that
not
only
I'm
seeing
container
information
but
I
abused
like,
for
example,
kubernetes
deployments
that
show
me
the
deployments
that
are
running
on
the
bonded
machine.
So
what
happens?
A
A
So
again,
let's
do
two:
let's
go
to
kubernetes
services
and
we
can
see,
for
example,
that
there's
a
job
application,
let's
drill
down
into
it
by
clicking
enter,
and
we
can
see
that
there
are
three
pods
on
these
inside
this
specific
service.
We
can
click
on
one
and
now
we're
seeing
the
containers
there's
only
one
container
in
this
pod.
Let's
keep
going
now
we're
seeing
the
processes
inside
the
container
and
we're
seeing
you
know,
nginx
a
couple
of
scripts.
So
let's
go
inside
the
engine,
GNX
worker
and
now
I'm
inside
the
process.
A
In
this
case,
just
log
file,
by
the
way
I
can't
read
down
even
further
by
clicking
on
the
vibe
I'm,
seeing
the
specific
data
that
has
been
written
on
that
file
or
I
can
look
at
the
connections
and
I
can
see
the
different
connections
that
this
container
is
established.
And,
again,
you
know,
I
can
select
a
specific
connection
and
I'm
seeing
the
data,
including
the
HTTP,
requests
and
responses
that
are.
A
A
The
accept
I
can
see
the
receive
from
and
in
the
right
on
the
sockets
with
the
data
I
can
see
that
the
socket
was
shut
down
in
the
middle,
but
then
some
data
was
still
received
by
the
socket
and
like
in
The
Naked,
City
Clause,
so
we
went
from
essentially
the
old
data
and
the
machine
drilling
down
and
narrowing
down
to
a
specific
connection
through
essentially
a
user
interface
that
is
oriented
toward
this
use
case.
So
again,
this
is
a
complete
open
source
tool.
It's
totally
fun
to
use.
A
There's
the
link
on
my
slides,
I
think
it's!
You
know
something
that
can
really
be
beneficial
for
troubleshooting
containers,
which
is
something
that
we
very
often
oversee.
Last
thing
that
I
want
to
talk
about
these
people.
So,
as
I
told
you
before,
I'm
very
excited
about
trying
to
rethink
monitoring
in
a
way
that
is
a
bit
more
people
oriented.
So
micro
services
are
about
people.
A
Micro
services
start
by
us,
wanted
to
make
our
team's
more
decoupled
in
our
development
teams
scale
better,
so
can
monitoring
reflected
today,
SSD,
we
are
announcing
a
new
feature
called
cystic
teams,
which
is
designed
exactly
for
this.
So
the
way
it
works
is
you
get
all
of
the
data
in
a
way
in
the
way
I
showed
you
before,
so
you
know
application
container
of
matrix.
A
It
all
goes
in
a
central
database
and
it
this
is
correlated
with
the
kubernetes
service
metadata,
and
then
the
data
is
filtered
and
pushed
to
different
teams,
based
essentially
on
not
only
the
physical
topology
but
the
kubernetes
kubernetes
metadata.
So
this
is
designed
to
reflect
people
organization.
It's
heavily
integrated
with
kubernetes
I'm,
going
to
show
you
a
little
Dominican.
It
supports
isolation
and
customization.
A
So
if
you
are
part
of
a
team
that
is
monitoring
a
specific
service,
you
will
only
see
the
data
for
for
that
service
and
the
data
will
be
filtered
even
if
those
containers
or
pods
run
on
machines
that
are,
you
know,
share
with
other
people.
It's
completely
self
service
and
it's
centralized.
What
does
it
mean?
Let's
go
back
here,
can
you
read
this
all
right?
So
this
is
a
deployment
I'm
running
a
local
kubernetes
on
my
butter
machine.
So
what
I'm
doing
here
is
just
I'm
going
to
start
a
service.
A
So
you
know
this
is
a
deployment
standard
stuff
you
can
see.
I
will
have
three
containers
its
WordPress
I'm,
using
the
WordPress
image,
but
there's
something
different
here,
there's
an
annotation
section
and,
as
you
can
see,
there's
a
some
context
here.
So,
first
of
all,
who
are
the
team
members
for
for
this
deployment,
so
I
use
essentially
three
email
addresses.
Then
I
want
a
couple
of
dashboards.
A
This
is
you
know,
WordPress
services
top
talking
to
my
sick.
Well,
so
I
would
like
the
service
of
a
view
dashboard
and
there
was
I
would
like
the
Mexico
dashboard.
Then
I
would
like
to
get
my
alerts
in
two
different
ways.
One
is
a
couple
of
people
should
receive
an
email
when,
when
an
alert
is
filed
and
I
would
also
like
to
have
another
posted
on
a
select
channel
and
then
I'm
configuring
a
couple
of
others.
A
So
the
service
starts
since
the
cloud
talks
to
the
kubernetes
watch.
Api
detects
that
the
service
is
starting
reads.
The
annotation
understands
that
this
service
is
annotated
as
a
team
creates.
A
team
for
this
service
looks
who
are
the
users
of
the
team
and
sends
an
email
to
the
users
that
have
not
been
registered
to
see
the
cloud
so
I'm
going
to
activate
my
user.
A
Now
I
have
my
own
customized
monitoring
environment
that
has
been
dynamically
created
on
the
fly
for
me
when
the
service
was
created,
I
can
see
the
three
containers
that
are
part
of
the
service
and
only
those
three
containers
everything
else
is
fitted
for
me,
no
matter
what
those
containers
go
this
this
will
follow
them
and
everything
else
all
of
the
metrics
that
are
not
related
to
these
containers.
All
of
the
tags
everything
else
is
filtered
out
so
that
my
my
environment
is
is
isolated.
I
can,
of
course,
you
know.
A
A
These
are
already
populated
with
the
data,
and
all
of
this
data
again
is
completely
collected
out
of
the
box,
in
particular,
for
example,
the
sequel
queries
are
collected
by
looking
at
the
network
traffic
by
sitting,
underneath
the
containers
decoding
the
queries
and
ensuring
the
slope
queries.
But
the
point
here
you
know
these
dashboards
are
essentially
now
part
of
this
team
and
they've
been
created
automatically
I
didn't
need
to
log
in
the
monitoring
tool
to
create
these
dashboards
I,
just
declared
which
what
I
wanted
when
I
created
the
service.
A
If
I
go
to
all
earth,
I
am
my
two
others
that
have
been
created
and
they
are
already
active
and
in
looking
at
the
data,
but
if
I
go
to
settings,
I
can
also
see
that
my
notification
channels
have
been
created
and
I
have.
You
know
the
twinners
that
are
specified
in
the
service.
These
are
the
guys
that
we
received
alerts
and
they
also
have
the
slack
Channel,
both
of
which
have
been
essentially
configured
so
I.
A
Don't
know
we
use
like
extensively
SSD
and
being
able
to
essentially
dynamically
create
like
channel
that
will
be
tied
to
a
specific
service
is
something
that
I
sort
of
really
love
as
a
concept
and
just
shows
me
the
power,
the
power
of
this.
Of
course,
you
know
a
new
user
has
been
created
and
this
user
is
part
of
the
team
deployment
prod
WordPress,
which
was
created
dynamically,
but
I
can
always
switch
to
the
default
theme,
and
essentially
these
users.
A
The
privilege
is
to
also
see
the
full
infrastructure,
and
so
now
you
know
by
just
switching
the
team
here
in
the
user
interface
I
can
go
and
look
at
you
know
the
full
infrastructure
and
I
can
see.
You
know
the
my
service
data,
but
now
I
can
see
every
other
services
data
and
they
can
see
it
in
a
way
that
is
both
logical,
so
namespaces
and
pause,
and
it's
kind
of
stuff
or
physical,
like
machines
and
containers,
and
so
on.
A
Use
cases
for
these
you
know
development
versus
production,
so
having
the
ability
to
segment
things,
maybe
in
the
namespace
level,
18
per
micro
service,
which
is
what
I
showed
you
before
platform
as
a
service,
which
is
another
very
strong
use
case
for
kubernetes,
tends
to
match
pretty
well
this
kind
of
this
case,
because
it's
possible
essentially
to
have
the
past
themes
that
start
their
applications
and
magically.
They
also
get
the
link
essentially
to
their
monitoring
environment
for
the
application,
even
if
their
little
application
list
only
is
just
an
experiment
and,
of
course,
their
strict
environments.
A
We
talk
quite
often
to
people
that,
for
example,
work
in
I,
don't
know
banking
and
you
want
to
have
a
platform
running
kubernetes,
but
you
want
strong
insulation
even
from
the
monitoring
point
of
view,
because
some
containers
market
might
contain
sensitive
information.
So
in
this
way
it
is
possible
to
clearly
isolate
the
containers
because
all
of
the
data
is
filtered
out
and
that's
it.
So
these
are
some
links
again
the
open
source
tool
you
find
it
on
on
github
or
on
our
website
since
the
cloud.
This
is
the
website
and
we
announced
it
today.
A
Yes,
so
with
sista
cloud,
there's
a
pretty
sophisticated
notification
set
of
functionality,
so
you
can
I
showed
basic
stuff,
but
you
can
you
know
pager
duty,
Amazon
SNS,
you
can
route
them.
You
can
split
them.
You
can
do
you
can
do
a
bunch
of
stuff.
Essentially
you
can
have
different
channels
that
they
do
different
things.
So,
from
this
point
of
view,
it's
like
any
good
modern
monitoring
tool.
C
A
A
That
would
be
essentially
a
standard
file
format
and
compatible
with
other
tools,
and
so
we
decided
to
pick
essentially
a
pickup
format,
so
the
format
per
se
is
binary
and
is
the
same
format
that
can
be
opened
by
TCP
dump
or
Wireshark
or
any
other
networking
tool.
But
of
course,
six
League
is
easy.
It's
a
pretty
sophisticated
tool.
So
there
are
many
ways
to
take
this
data
and
convert
it
to
you.
Man
readable.
A
A
First
of
all,
the
old
collection
system
that
I
showed
you
in
the
slides,
that's
all
open
source,
so
the
whole
instrumentation
is
open
source
in
the
whole
data
collection,
the
user
interface,
the
common
line
user
interface,
that
I
showed
you
is
open
source
and
then
essentially,
is
a
company.
We
build
a
tool
on
top
of
on
top
of
these,
to
collect,
to
collect
this
data
and
offering
you
essentially
the
dashboards
and
all
this
kind
of
stuff.
That's
essentially
how
we
structured
this.
B
A
A
C
C
A
It's
the
same,
you
get
exactly
the
same
information.
Well,
if
it's,
if,
if
it's
in
a
container
and
it's
in
a
pod
and
it's
running
on
top
of
kubernetes,
the
benefit
is
that
you
get
all
of
the
tagging
out
of
the
box.
So
that's
the
additional
benefit.
You
know
the
context
that
you
don't
take
if
you
are
not
running
a
container.