►
Description
Prometheus instrumentation: the Practical Way - Aditi Ahuja, Couchbase
Instrumenting applications to expose meaningful metrics is the key to harnessing the power of Prometheus. The native Prometheus client libraries offer a convenient way to define various metrics about essential behaviours of your application in a form of basic metric types: counters, gauges and histograms. Applying this to more complex cases might be challenging.
In this talk you will learn about the instrumenting real application in an example of Thanos (metric data store extending the long term storage capabilities of Prometheus) compaction microservice. The audience will learn practical instrumentation approaches on production-grade software from basic to more complex cases. The complex case is monitoring various stages and estimating the potential compaction durations, which can vary widely based on the data. Aditi will explain Go client implementation using the official Prometheus library, but the same can be generalized to other languages.
At the end of this talk, you will know how to instrument applications and how to unit test that setup! This talk is for everyone looking to start out with instrumenting code and tap that into Prometheus.
A
Good
morning
I'm
aditi-
and
today
I
will
be
talking
about
with
getting
started
about
getting
started
with
prometheus
instrumentation
from
the
ground
up
if
you're
a
beginner
looking
to
instrument
your
application
to
tap
into
prometheus
this
one's.
For
you
so
a
little
bit
about
me,
I
am,
I
was
in
lfx,
mentee
last
fall
at
the
thanos
project
and
I'm
currently
an
intern
at
couch
pace.
A
A
A
A
A
Briefly,
instrumenting
code
could
be
walked
through
a
four
stage
process
defined
by
the
questions.
What
should
I
instrument
also
covering
the
y?
How
do
I
get
the
data
for
it?
How
do
I
organize
it?
That
is
what
types
of
metrics
represent
this
data
the
best
and
finally,
the
implementation,
the
nuts
and
bolts
to
set
the
context
for
one
of
the
examples.
Thanos
is
a
metric
data
store,
extending
the
long
term
storage
capabilities
of
prometheus
so
taking
the
example
of
some
common
thanos
processes,
compaction
down
sampling
and
retention.
A
These
are
each
multi-stage
processes
which
can
take
varying
amounts
of
time
based
on
the
data
to
be
processed.
These
lagged
observability,
and
hence
these
were
the
perfect
candidates
to
instrument
to
gather
observable
metrics.
That
was
the
focus
of
my
project
during
my
mentorship,
so
looking
at
some
terms,
metrics,
this
is
a
fairly
common
term,
but
just
reiterating
so
each
time
series
has
a
unique
identifier,
ideally
providing
some
idea
about
its
function
called
a
metric
name.
A
A
A
Counters
can
be
set
to
answer
the
question:
how
often
since
they're,
mostly
used
in
getting
the
rate
of
increase
and
gauges
can
also
be
reset
to
zero
on
demand,
unlike
counters,
where
that
happens
only
on
restart
histograms.
These
sample
observations
based
on
frequency
and
slot
these
into
specified
buckets.
A
A
A
A
In
the
thanos
processes,
the
data
was
segregated
into
groups
based
on
the
minimum
time,
and
each
of
these
were
computed
per
group.
So,
coming
to
my
thought
process
in
on
how
I
picked
the
matrix
metric
types
to
represent
these,
since
these
values
could
increase
or
decrease
and
would
need
to
be
reset.
Like
I
discussed
earlier,
I
went
with
gauges
and
since
each
group
required
a
separate
identifiable
value,
I
used
gauge
vectors
labeled
with
the
group
ids.
A
I
have
an
intermediate
map
group
blocks,
which
you
can
see
in
the
snippet,
where
I
store
the
results
for
each
group,
and
this
is
how
its
update
a
gauge
is
updated
in
client
golang
and
after
all,
the
blocks
have
been
processed.
I
reset
it
to
zero.
The
first
line
of
the
snippet
and
each
group
key
becomes
a
label
for
each
group.
A
A
A
The
default
prom
config
yaml
supports
this,
and
it's
it.
The
scrape,
config
scrapes
port
9090,
every
15
seconds,
as
I'm
sure
you'll
know
the
code
on
the
top
right
serves
these
metrics
on
on
the
custom
port
on
the
custom
endpoint,
and
I
have
added
a
job
to
the
config
just
below
it
and
having
all
of
these
default.
Metrics
in
the
default
registry
is
actually
an
anti-pattern
and
should
be
and
will
be
removed
in
the
next
version
of
the
client.
A
A
So
now,
let's
look
at
some
more
examples
of
client
golang,
so
I
first
create
the
end
point
with
this.
With
a
handler
on
the
same
port,
and
here
I
have
used
a
custom
handler
with
a
specific
registry.
Unlike
the
previous
example,
with
the
default
metrics,
I
use
the
default
handler,
which
came
with
the
default
registry,
and
hence
the
default
metrics
were
already
registered.
A
Let's
discuss
the
more
detailed
example
in
this
example,
I
will
be
discussing
a
small
application
involving
apache
kafka,
an
event
streaming
platform
which
can
also
be
used
as
a
message
queue.
I
will
be
using
their
go
client
to
create
and
delete
topics
produce
and
consume
messages
and
instruments
to
expose
some
basic
metrics.
A
When
creating
a
topic,
I
have
a
stroke
topics
created
with
metrics,
for
both
the
possible
cases
successfully
and
unsuccessfully
created.
So
the
successful
metric
represents
topics.
The
net
number
of
topics
created
at
the
end
of
the
run
categorized
by
labels,
and
hence
it
is
a
gauge
so
when
creating
when
creating
this,
I
use
promoto
to
register
these
without
an
explicit
registration
step
in
the
prometheus
package.
Creating
and
registering
metrics
is
separate
with
creation
preceding
registration.
A
A
So,
since
this
talk
is
about
a
quick,
practical
introduction
to
prometheus
instrumentations,
a
few
things
to
keep
in
mind,
while
naming
metrics,
the
namespace
should
specify
the
domain
or
the
application
of
the
metric.
Ideally
here
I
have
used
kafka,
the
suffix
should
have
the
unit
or
the
word
total
for
accumulating
counts
like
counter.
I
have
used
total
here
and
avoid
adding
the
label
name
into
the
metric
name,
since
that's
a
redundancy,
an
example
of
that
would
be
naming
the
message.
A
A
So
I
categorize
label
topics
by
this
really
simple
method
based
on
their
suffixes
either
is
order,
even
so,
I
add
to
the
gauge
with
the
label
for
that
topic
and
specifying
that
label
makes
sure
that
only
that
time
series
gets
incremented,
so
the
recommended
practices
for
code
instrumentation
make
a
good
point
about
adding
a
metric
for
anything.
Interesting,
that's
worth
a
log
here,
since
I
think
tracking
the
number
of
errors,
while
creating
topics
is
worthwhile.
A
A
A
So
the
histogram
buckets
here
indicate
message:
consumption
latencies,
which
are
less
than
the
specified
buckets
in
the
buckets
before
it.
So
there
are
multiple
ways
that
this
can
be
specified.
The
first
is
the
method
I
have
shown
explicit
demarcations
and
the
second
when
they
are
evenly
spaced
and
the
count
is
known,
is
using
linear
buckets
and
finally,
if
they
are
not
linear,
one
can
also
use
exponential
buckets.
A
The
message
latency
here
is
the
duration
between
writing
a
message
and
returning
from
the
right
function,
so
the
histograms
have
observations
added
using
the
observe
method.
This
is
the
timer
in
the
client,
golang
library
and
the
duration
is
calculated
using
timer.timers,
which
can
be
used
with
histograms
or
gauges
or
summaries.
A
A
Consuming
messages,
so
the
message
consumed
struct
is
pretty
similar
to
the
previous
one
and
each
time
a
message
is
successfully
consumed.
The
gauge
message
count
for
that
topic
is
incremented
using
the
decrement
decremented
using
the
decrement
function.
If
not,
this
error
2
has
a
corresponding
metric.
A
A
A
A
To
test
the
send
message:
function
too:
I
do
something
similar.
I
have
specific
structs
for
the
subtests.
I
run
the
functions
to
be
tested
again
and
get
the
metric
values
for
the
labels.
So
you
can
see.
I
have
my
custom
registry,
I
reset
it
all
and
I
get
metrics
with
label
values
and
I
check
for
assertions.