►
Description
Prometheus: The Unsung Heroes [I] - Alejandro Ramirez, Swisscom Directories AG
Prometheus is a fantastic tool, Prometheus/Kubernetes integration is amazing, PromQL is expressive…. We have all heard the superlatives and the articles, but this talk explores those tools of the Prometheus ecosystem that are not under the spotlight, but should not be discarded as mere footnotes to the saga.
We will discuss how mtail and the snmp_exporter allowed us to instrument our load balancer appliance and give us more insights into our performance in such a critical component of our system.
About Alejandro Ramírez
Alejandro Ramirez Local.ch (Swisscom Directories AG) Senior System Engineer Zürich, Switzerland Alejandro works at local.ch, one of the country’s highest traffic websites, ensuring that the platform runs smoothly while at the same empowering other engineers in the team to accelerate development getting software as quickly and reliably as our users expect it, embracing DevOps and breaking old silos.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Each
one
gets
about
four
million
unique
visitors
a
month,
so
we
have
some
quite
some
traffic
and
I'm
as
a
tech
leader
from
from
DevOps,
so
yeah.
So,
let's
begin
so
on.
Some
here
is
I.
I
really
wanted
to
try,
like
this
mark
marketing
angle
tone
to
my
talk
and
I
think
it
seems
that
yeah
it's
working
so
basically
the
in
every
story.
There
is
always
some
some
things
that
don't
get
the
recognition
that
they
that
they
should.
In
our
case,
there
is
a
lot
of
hard
work
that
it's
not
really
connected
to
to
kubernetes.
A
Oh,
it's
really
nice!
When
you
have
that
wind
field
approach
and
you
can
like
have
your
white
box
monitoring,
building
into
your
apps
and
get
the
the
data
that
you
want
from
them.
That's
like
the
the
ideally
scenario,
but
there
are
some
other
tools
that
you
can
use
and
I'm
going
to
show
you
two
of
them
during
this
talk
that
should
enable
you
to
get
those
things
that
does
not
with
build
team
monitoring
and
get
them
with
your
alerting
and
monitoring
with
primitives
going
on
yeah.
A
So,
for
example,
in
history,
I
have
here
two
examples:
I'm
going
to
go
really
fast,
because
time
is
a
little
bit
short
but,
for
example,
most
people
have
not
heard
about
the
Maurice
Helaman
and
the
guy
just
yeah,
just
he
just
developed
40
vaccines.
So
you
can
imagine
the
impact
he
has
on
history,
and
probably
you
don't
know
who
this
mr.
bottle
Auggie's
and
he
just
he's
a
great
about
saving
1
billion
people
on
earth.
So
yeah
as
I
say
it's
a
little
bit
of
hyperbole
to
the
talk,
but
I
was
thinking.
A
Yeah
I,
don't
really
know
what
you
guys
are
monitoring,
maybe
you're,
not
monitoring
a
nuclear
plant
or
a
missile
silo,
or
something
like
that.
So
it
could
be
that
you
theorized
with
monitoring
as
well.
So,
let's
get
with
the
with
the
stories
of
what
was
our
problem,
nor
was
the
motivation
we
had
it
so
in
in
local
we
have
a
closed
source,
the
load
balancer,
it's
a
right.
Now
it's
called
brocade.
A
It
has
changed
like
names
like
three
or
four
times
in
the
last
four
years,
and
the
problem
is
that
it's
a
closed
source,
the
open
balancer.
So
there
is
no
way
that
we
can
really
go
into
the
code
and
actually
instrument
the
monitoring.
It
does
provide
some
build-team
graphic,
so
it
has
a
web
UI
where
you
can
go
and
query
some
stuff,
but
there's
really
basic
crafting
and
also
it's
not
available
to
all
of
the
developers
and
it's
not
really
on
the
same
place
with
the
with
with
the
other
data
sources
that
we
already
have.
A
So
actually
our
load
balance
is
not
just
one.
Little
parameter
it's
actually
a
cluster
of
load
balancers
and
there
are
four
separate
boxes
so
before
I
started
working
in
mistakes
in
the
company,
someone
said
okay,
but
we
need
to
use
to
have
this
date
and
have
some
kind
of
monitoring
and
they
realized
that
one
of
the
features
that
local
answered
offered
was
actually
that
you
could
log
to
a
file,
but
also
you
could
have
a
syslog
facility.
A
So
you
could
like
tell
her
the
feeds
log
server
and
tell
you
what
you
go
and
you
could
have
a
like
a
specific
format
and
how
to
send
it.
So
somehow
came
up
with
a
bytom's
vitam
script
articles
syslog
to
carbon
and
basically
what
it
is
that
received
the
syslog,
a
stream
of
logs
chopped
it
down,
navigate
it
and
then
push
it
to
to
graphite
and
that's
how
we
were
working
for
the
last
I,
don't
know
four
years,
but
I
think
you
can
already
see
what
really
the
problem
is.
A
That
first
of
all,
is
this
syslog
to
carbon.
The
virtual
machine
died,
then
no
moratorium
for
us
so
that
that's
it
that
that's
the
end
and
also
doing
changes
to
this
script.
That
which
happened
fairly
regularly,
so
some
kind
of
a
log
change
or
we
wanted
to
to
get
some
specific
graph
into
graphite.
Doing
changes
was
really
scary
because
a
if
you
just
like
playing
Jenga
I,
did
just
think
one
bit
of
the
bottom
and
everything
falls
apart,
and
no
monitoring
and
people
are
going
crazy.
So
this
was
not
really
working
for
us,
so
yeah.
A
So
recently,
the
everyone
here
I
guess
we
begun
moving
to
two
coronet
s
and
we
started
exploring
Prometheus
and
we
got
really
excited
about
the
things
it
can
do
and
how
it
works.
But
we
notice
we
have
all
of
these
devices
which
we
want
to
get
the
data
from.
So
we're
going
to
talk
about
em
till
a
little
bit
right
now.
A
So
anything
is
if
you
go
to
github
and
you
go
to
it's
part
of
the
of
the
Google
organization,
but
it
seems
that
it's
just
developed
by
one
guy,
so
you
see
the
community
and
it's
just
one
on
one
computer
and
basically
what
it
is
is
it's
a
tool
for
extracting
met
the
metrics
from
application.
Look
so
you
can
you
have
a
log
and
actually
tells
the
log,
and
you
have
the
set
of
regular
expressions
and
based
on
those
regular
expressions.
You
can
expose
your
metrics
for
other
metric
systems.
A
A
So
in
its
most
basic
form,
a
an
M
tail
program
is
a
would
be
this
one
day.
I
think
this
will
be
like
the
smallest
thing.
You
can
do
this
with
just
a
line
count,
so
you
see
they
have
what
they
call
export
variables
with
a
counter
type
there
and
a
regular
expression.
That's
just
looking
for
the
end
of
the
line
and
just
increasing
the
counter.
So
that's
like
very
simple
basic
stuff,
so
yeah
I
have
a
bunch
of,
and
this
one
is
pretty
simple
and
I
hope
you
can
greet
this
of
their
figures
better.
A
All
right,
but
I
can't
ride
like
this,
so
I'm,
gonna,
cheat
and
I.
Have
it
this
thing?
Okay,
scripted!
So
just
line
count.
So
I'm
not
going
to
go
through
the
steps
of
actually
building
them
tail
because
it's
just
a
normal
goal.
Installation
just
go
build
and
you
already
have
the
things
there.
So
I
have
installed
my
laptop
and
then
there's
also
container,
which
you
can
run
so
yeah
here
at
examples
and
in
one
directory
I
have.
A
I
have
a
the
locks
which
I
want
to
to
take
a
look
at
and
the
other
directory
called
props.
We
have
the
MPL
programs,
you
can
have
multiple
programs
and
when
MTEL
runs
it
will
just
like,
unite
them
and
evaluate
all
the
rules
and
see
which
them
which
apply
to
the
set
of
rules
that
you
have
there.
So
this
is
just
a
simple
log.
I
found
some
wearing
the
internet
not
really
relevant
to
us,
but
it
will
serve
a
purpose
just
regular
log
file.
A
Then
we
have
our
basic
program
that
we
discussed
earlier
and
now
yeah
you
see
it's.
This
is
like
the
the
web
interface
it's
office
offers
and
we
can
go
to
Prometheus
and
we
can
see
that
right
now
it
says:
ok
lang.
The
program
is
called
line.
Count
em
tail
running
on
my
laptop
and
it
has
0
heaps
while
because
it's
expelling
the
the
log
file-
and
nothing
has
happened,
it's
just
a
standard
logs,
also,
let's
pipe
something
to
that
alright
and
yeah.
A
Well,
so
now
you
have
a
the
counter
increase
in
it
and
it's
ten,
it's
really
basic,
but
it's
already
very
useful
just
in
it
fix
it
in
this
most
basic
form.
So
that's
yeah!
So
let's
go
on
with
the
with
the
presentation
now
you're
going
to
go
with
a
little
bit
more
complex
thing,
so
I
like
coffee,
I,
guess
everyone
likes,
coffee
and
and
and
a
coffee
machine
would
be
one
of
those
things
that
I
don't
expect
it
to
have
already
a
prometheus
end
point
for
scraping.
That
will
be
fantastic,
but
it
does
not.
A
But
let's
think
that,
for
some
reason
it
has
some
serial
connection,
we
can
actually
get
logs
out
of
the
out
of
the
out
of
the
coffee
machine.
So
you
see
a
very
basic
structure
with
the
timestamp
and
you
see
the
the
steps.
It's
it's
taking
two
to
brew
the
coffee
or
a
cappuccino
or
whatever,
and
we
want
to
strut
some
information
out
of
it.
So
this
is
yeah.
This
is
one
device.
We
cannot
read
instrument
as
I
say
so.
A
With
this
example,
we're
going
to
see
a
lot
of
the
basic
concepts
of
M
till
it
should
be
really
good
for
you,
starting
what
we
want
to
do
so
here.
On
top
use,
we
have
the
tool
exported
bars
that
M
Cal
has
it's
a
Gaussian,
an
account
and
where
you
see
there
by
type
and
stage
those
would
be
like
the
dimensions
of
the
metric
that
we
would
have.
A
Then
there
is
the
other
back
here
that
is
called
hidden,
which
will
not
appear
as
a
as
a
metric
when
we
scrape
Prometheus,
but
it's
actually
very
useful
for
things
like
timers
and
we're
going
to
see
example
afterwards
and
then
the
last
line.
There
is
just
how
you
define
a
regular
expression
just
to
make
your
things
a
little
bit
neater
and
what
we
have
there
is
one
of
these
python
named
capture
group.
A
So
yes
bonus
time
before
we
go
on
at
myself,
I'm
not
really
familiar
with
go.
So
there
was
really
strange
for
me
so
time.
Parse
works
very
differently
from
what
you
have
may
have
used
in
the
in
the
past,
for
example,
the
Java
or
any
other
C
language
is
a
with
where
you
have
this
percentage
B
for
for
a
year
percentage,
H,
capital,
H,
4/4
time
something
like
this.
But
what
they
do
is
that
that
goal
actually
does
it
with
a
mnemonic.
A
So
you
see
you
have
this
date
that
there
that
you
see
on
the
top,
and
that
translates
to
one
two,
three,
four,
five,
six,
seven
and
basically
what
what
they're
doing
is
that
they're
formatting
the
dates
by
example.
So
you
you
have
your
time.
Format
cannot
point,
but
you
have
your
your
and
know
your
your
time
some
sample
and
then
on
the
on
the
right.
You
see
the
the
format
and
how
you
would
tell
it
what
format
to
expect.
A
A
Ntelos
also
borrows
a
little
bit
the
syntax
from
Python
decorators,
so
you
can
create
a
decorator
which
can,
for
example,
in
this
case,
extract
the
time
out
of
the
law,
because
sometimes
we
want
to
do
timers.
We
are
not
we
caring
when
the
load
arrive
to
Amtel,
but
actually
the
time
starting
from
the
log
itself.
So
in
this
case
the
decorator
is
actually
extracting
the
the
time
stamp
from
the
law.
Senators
there,
the
current
time
stamp
and
then
we'll
just
continue
to
evaluate
the
next
of
the
of
the
code.
A
A
A
A
And
there
we
go
so
with
this
very
simple
program.
We
were
able
to
get
some
really
interesting
metrics,
so
here
we
can
see,
for
example,
that
the
form
of
the
cappuccino
and
the
lat
is
quite
differently,
because
obviously
it
has
a
little
bit
more
milk
and
we
check
that
that
the
brew
time
is
somehow
similar.
But
maybe
it's
not
exactly
the
same,
and
also
we
have
this
the
warnings
so
really
with
a
real,
simple,
regular
expression.
In
some
logic,
there
we're
able
to
extract
quite
useful
information
from
from
the
logs.
A
So
again,
let's
go
so
I
think
go
on
talking
about
this
thing
on
our
load
balancer.
We
have
some
code
similar
to
this.
It's
not
exactly
the
thing,
but
basically
we
are.
We
are
parsing
the
logs
for
five
hundred
and
two
hundred
and
three
hundred
and
things
like
that,
and
actually
the
what
we
end
up
is
the
graphs
that
took
like
this
and
when
things
catch,
fire
they're
really
useful
to
see
what
how
things
are
going
wrong.
A
So
it's
a
pretty
useful
tool,
so
I'm
going
to
wrap
up
run
out
of
empty
in
with
good
on
time.
So
is
it
perfect?
Absolutely
not
it's
really
yeah.
They
did,
for
example,
the
regular
expression
set
is
not
is
not
the
full
complete.
For
example,
positive
and
negative
Luca
rounds
are
not
really
support,
and
that
would
have
been
really
useful
for
things
like
things
where
you
have
a
double
pipes
as
a
delimiter,
for
example,
or
something
like
this
would
make
the
regular
expression
a
little
bit
more,
simpler
and,
most
importantly,
right
now,
I.
A
Don't
know
why
the
Edit
there
is
that
currently
a
blog
in
the
in
the
in
the
in
the
latest
release.
There
is
no
releases
on
the
latest
commits
which
actually,
if
you
keep
it
running,
and
it
finds
a
parse
error,
it
goes
crazy,
which
is
stopped.
Logging
and
the
matrix
would
remain,
would
not
increase.
There
was
an
issue
there
and
they
fix
it,
and
then
they
roll
back
for
no
reason.
So
you
have
to
go
to
a
specific
commit
if
you
want
to
to
get
this
thing
right
so
yeah.
A
So
it's
a
it's
not
really
perfect,
but
the
alternatives
we
could
use
flow
into
your
log
stash
too
our
stuffing
elasticsearch.
But
then
you
would
need
some
kind
of
logic
to
strategy
extracting
from
from
elasticsearch
and
then
get
it
in
the
super
meat
is
because
we
have.
We
want
to
have
the
same
monitoring
logic
going
on,
so
it's
a
little
bit
complicated
and
we
don't
care
about
that
too
much.
There
is
another
exporter.
A
That's
called
the
grub
exporter,
which
does
a
similar
thing
to
this
one,
but
it
uses
a
different,
regular
expression
engine
which
is
quite
the
I,
think
it's
the
same,
one
that
that
logstash
uses
and
yeah
on
the
on
the
on
the
Gipper
github
page,
the
guy
says
basically
yeah.
This
one
uses
more
memory
and
yeah,
so
we
went
with
with
empty
land.
So
far
we
are
happy
with.
We
would
like
more
people
in
the
community
to
actually
help
them
bring
this
to
a
more
stable
thing,
because
it's
quite
useful
but
yeah.
A
That's
why
we're
here
to
spread
the
word?
Basically
so
now
we're
going
to
talk
about
the
SNMP
exporter,
which
was
another
part
of
the
puzzle
for
having
the
the
load
balancer
actually
documented,
and
the
resident
the
exporter
is
yeah
I.
Think
it's
a
very
self
defining.
What
it
does,
actually
it
will
scrape
Network,
Devices
or
servers
or
whatever
that
provide
the
the
SNMP
monitoring
option
and
we'll
expose
them
in
a
format
that
Prometheus
understands.
A
There
are
two
components
to
it:
one
is
the
exporter
itself,
which
does
the
scrape
in
an
offering
of
exposing
the
metrics
to
permit
it,
and
then
there
is
a
really
useful
generator
to
which
actually
can
create
the
information.
Without
you
going
crazy
with
the
with
the
with
SNMP
tree,
which
is
yeah,
always
hell
I'm,
going
to
go
really
fast
through
this
one,
because
I
am
actually
expecting
that
people
know
what
SNMP
is.
A
But
it's
a
really
well-known
protocol
has
been
around
for
a
long
time,
some
as
third
iteration
of
version
and
the
first
tool
security
was
just
like
yeah.
We
have
this
community
string
and
that's
our
secret
and
we
pass
it
on
clear
sex
knows
and
version
3,
it's
a
little
bit
better.
What
you
get
yes
difficult
and
and
all
the
stuff
this
is
we
what
we
you.
A
A
There
is
this
whole
concept
about
the
object
identifiers
and
how
the
tree
the
tree
works.
So
basically,
a
ID
is
basically
as
a
set
of
numbers
defining
an
entering
in
the
tree.
You
can
with
that
afterwards,
it's
a
very
interesting
and
long
discussion.
There
see
how
it's
worked.
There
then.
The
other
part
is
the
the
management
information
base
which
is
actually
provided
by
your
vendor.
Usually,
if
you
have
a
Cisco,
router
or
I,
don't
know
juniper
stuff
like
that
they
will
provide
an
MIB
for
your
device,
which
actually
says
look
this
this
device.
A
That's
here
we
offer
these
metrics
for
you.
You
can
find
them
with
this
with
these
oh
Ivy
numbers
and
yeah.
So
it's
it's
the
definition
of
what
things
are
offered
and
it's
like
the
best
real
guide
for
to
know
the
capabilities
of
your
device
as
an
MP
wise,
so
the
generator
tool,
as
I
said
before.
It's
really
useful
tool
because
you
don't
have
to
call
the
tree
to
get
the
to
the
DOE
IDs
that
you
are
interested
in
right
now.
It's
currently,
you
have
to
build
it
separately,
the
further
than
the
SNMP
exporter
tool
itself.
A
Basically,
because
you
need
to
have
this
net
SNMP
tools
installed
and
also
you
need
to
provide
your
own
vendor
MIB.
So
so
Brian
decided
against
that
that
that
should
not
be.
It
does
not
really
make
sense
to
have
a
thing
inside
the
yeah.
That's
an
exporter
itself.
You
have
a
very
simple
generator
file
that
looks
like
this,
so
you
basically
tell
it
the
name
of
the
module,
in
this
case
the
x
CX
T
M.
It
tells,
with
version
of
this
MP
you're,
going
to
be
using
for
that.
A
A
So
it
was
a
a
lot
of
data
being
exported
there,
but
not
really
used,
because
we
could
not
reach
it
easily
and
then
the
other
part
is
the
export
that
which
will
just
within
with
a
file
generated
by
the
generator,
will
cut
the
tag
their
IDs
and
expose
the
targets
yeah.
So
this
is
what
it
would
look
like,
so
I
think
I'm
going
to
go
a
little
bit
farther
and
so
on
to
the
demo.
A
A
Okay,
so
this
is
our
generator
file
and
what
I'm
going
to
do
it,
because
I
didn't
want
to
connect
to
any
strange
device
or
anything.
What
I'm
going
to
do
is
that
I'm
going
to
run
a
docker
container
that
actually
exposes
an
empty
matrix.
There's
an
empty
message
that
the
we're
gonna
see
exposed
are
basically
that
of
the
brutal
machine
where
docker
runs
on
a
Mac.
So
it's
a
yeah
a
little
bit
strange,
but
yeah
in
this
case
we're
going
we're
going
to
work
this
or
ID
that
you
see
here.
A
That
is
part
of
the
of
the
management
information
tree
and
especially
it's
specifically
the
mi
v
dot
to
entry
in
that
that
tree
that
it's
very
safe
to
use
for
demos,
because
if
you
have
a
SNMP
device,
this
thing
with
the
interface
must
be
declared
so
via
so
here
on
the
bottom.
You
see
that
I'm
I
already
running
the
container
that
exposes
SNMP,
metrics
and
I
just
did
a
basic
snmpwalk,
so
you
guys
can
see.
A
So
now
what
I'm
doing
right
now
is
that,
as
I
mentioned,
the
generator
tool
is
not
actually
built
into
the
Senate
exporter.
So
you
cannot
use,
for
example,
the
existing
container
for
the
SNMP
exporter
for
that,
but
you
can
really
easily
create
your
own
docker
image
and
just
add
this:
the
SNMP
tools
that
you
need
to
generate
it.
That's
what
I
did
in
this
case,
so
I
I,
just
took
the
generator
file.
I
showed
you
earlier
and
I
I
told
it
via.
Please
export
the
the
configuration
that
you
generated
to
the
SNMP
file.
A
So
we
can
see-
and
here
you
see
it
created
a
lot
of
things-
and
this
is
this
is
something
that,
if
you
use
SNMP
exporting
the
path
before
it
was
a
Python
to
this
generator
to
was
really
meeting.
I
was
really
a
pain
in
the
ass
to
get
the
oids
correctly
and
get
the
the
types
correctly
and
all
that
stuff.
So
really
the
generator
is
the
a
fantastic
tool
that
is
going
to
save
you
a
lot
of
time
and
it
really
lowers
the
barrier
to
to
use
in
SNMP
exporter
on
so
now.
A
What
I'm
doing
is
that
I'm
actually
running
the
SNMP
exported
and
using
the
generated
file.
So
now
it's
already
running
and
it's
waiting
for
Prometheus
to
to
come
over
to
something
to
come
over
and
actually
ask
for
it
to
scrape
a
target.
So
let's
see
we're
going
to
open
it
and
we're
going
to
use
the
target
hope
this
works
yeah.
It
works
so
here
on
the
on
the
bottom.
A
What
what
you
see
is
that
you
actually
walk
the
target
with
the
with
the
with
the
OID
that
I
want
it
with
the
subtrees
and
actually
generated
the
information
here
and
export
it
in
Prometheus
format.
So
that's
really
fantastic.
Just
just
already
with
this,
you
can
do
a
lot
of
stuff
and
then
you
can
create
your
own
alert
and
you
can
have
all
kinds
of
important
metrics.
For
example,
we
have
a
fiber
of
the
connecting
two
data
centers
and
we
had
some
monitoring
stored
in
some
NMS
server.
A
But
now
we
can
have
it
on
a
on
our
graph,
an
eye.
That's
what
really
in
our
faces.
We
can
see
what's
the
bandwidth
there
and
you
can
really
do
very
important
stuff
with
it
with
SNMP.
So
let's
go
one
step
further
and
we're
going
to
actually
run
from.
It
is
I'm
running
Prometheus
in
a
container
in
my
laptop
right
now
and
that
one
should
yeah
escape
this
target.
A
So
for
me
to
use
and
look
at
the
targets
and
well
happily,
it's
a
the
endpoint
if
it's
working-
and
you
see
here
that
it's
using
the
local
docker,
a
module
name
that
I
gave
it,
and
this
is
the
target
that
we
are
using.
So
that's
nice.
So
now,
let's
see
how
the
configuration!
How
do
we
tie
up
these
two
to
the
to
exporter?
Basically,
the
important
parts
here
would
be
would
be
the
modeling.
A
So,
for
example,
for
for
things
like
a
switch,
we
don't
care
further
about
the
same
matrix
as
for
a
server
or
for
a
router.
So
you
can
create
different
modules
and
you
could
say
that
what
models
to
apply
for
a
specific
set
of
targets
and
then
the
replacement
in
this
case,
because
I'm
working
in
special
conditions
and
inside
the
the
docker
p.m.
we
up
to
this.
This
is
usually
just
localhost
on
ninety
one.
A
A
Okay,
so
wrap
up.
What
would
be
an
alternative
for
this?
You
could
have
something
like
a
full
fledge
NMS,
which
is
like
the
official
thing
to
have
when
you're
using
the
SNMP
target.
But
then
again
you
have
the
same
problem
that
you
have
another
monitoring
system
and
another
set
of
routing,
alerting
rules
and
all
that
stuff,
and
we
would
like
to
really
get
one
working
but
working
well
and
not
really
having
everything
spread
and
getting
woken
up
at
the
middle
of
the
night.