►
Description
Counting with Prometheus [I] - Brian Brazil, Robust Perception
Counters are one of the two core metric types in Prometheus, allowing for tracking of request rates, error ratios and other key measurements. Learn why are they designed the way they are, how client libraries implement them and how rate() works.
About Brian Brazil
Brian Brazil is a core developer of Prometheus, and the founder of Robust Perception. He has developed and maintains components and features across the Prometheus ecosystem including the Python and Java clients, and many exporters. He wrote many of the best practices and guidelines for those looking to use Prometheus, and publishes regularly on the Reliable Insights blog.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
my
name
is
Brian
Russell
I'm
here,
to
give
you
a
talk
on
counting
with
Prometheus.
So
to
give
you
some
background
on
Who
I
am
I,
am
one
of
the
four
core
developers
of
radius
studying
in
Dublin
live
in
Dublin
worked
at
Google
for
a
while
worked
on
many
open-source
projects,
and
these
days
I'm
the
founder
of
robust
perception
who
does
consulting
and
support
for
Prometheus.
So,
if
you're
looking
for
that,
talk
to
me
afterwards,
so
Prometheus.
A
If
you
aren't
generally
aware,
give
you
some
background,
it's
the
metric
space
time
series
data
base
after
flight
box,
poetry,
so
solving
the
same
sort
of
space
thats
a
graphite
would
be
not
the
same
sort
of
space
that
elk
would
be.
It
supports
labels
which
are
multi-dimensional
tags
or
key
value
pairs,
there's
no
ordering
on
them
and
it's
got
unified
graphical
ownership
and
the
architecture
is
service.
A
Discovery
figures
that
stuff
that,
if
where
everything
is
all
your
pods
or
whatnot
from
kubernetes
or
console,
and
then
those
scrape
some
things
like
see,
adviser,
Cassandra,
J,
metrics,
Porter,
your
own
applications,
whatever
stores
the
data
locally,
has
ruled
alerts,
and
so
the
pretty
complex
query
language
was
Pompey.
L,
sin,
theta,
SS,
alert
browser
and
Ravana
or
other
dashboarding
solutions
can
get
the
data
there's
something
up
and
I.
Think
the
six
seven
kilohertz
range,
I'm
kind
of
hearing
your
echo,
so
I'm
just
appears.
Okay,
so
goes.
The
actual
topic
of
discussing
counting
counting
is
easy.
A
We're
all
taught
us
in
primary
school
or
ads
were
in
Germany
kindergarten.
It's
just
one,
two,
three
nice
and
simple.
The
problem
is
that
we're
looking
at
monitoring
events
or
monitoring
interested
monitoring
for
a
metric
system,
and
we
want
to
tell
how
many
of
those
have
happened
inside
last
five
minutes.
But
the
thing
is
that
we're
not
getting
seeing
every
single
event
because
we're
a
metric
space
system
of
the
logs
based
system,
what
we're
doing
is
sampling
every
now
and
then
hey.
What's
going
on
20
seconds
later,
hey,
what's
going
on
and
in
problem.
A
It's
sometimes
just
data
loss
because
of
network
interruption,
and
there
the
other
end
is
slower
in
the
middle
of
doing
GC
or
something-
and
you
know
we
have
this-
then
partial
view
of
sample
data,
and
how
do
we
process
that
answer?
The
question
of
how
many
requests
did
I
get
inside
the
last
five
minutes?
A
So
I
should
say
that
there
are
basically
two
types
of
core
metrics
that
you're
going
to
come
across
in
any
monitoring
system
with
also
weirder
stuff
like
hydrologic
log.
Ignore
those-
and
you
have
the
gauge
gauge-
is
a
snapshot
of
state,
so
canonically
your
temperature
or
your
memory
usage
is
a
snapshot.
Who
knows
how
it's
actually
moving
was
just
telling
you
right
now,
I
am
using
50
megabytes
of
RAM,
they
go
up,
they
go
down,
counters
are
the
other
base
type
and
well.
A
We
need
to
have
a
small
explanation
over
the
next
30
minutes
or
so
I
thought
a
counter
is,
but
the
thing
is
we're
trying
to
track
is
events.
Events
are
something
that
happens.
All
right,
it
might
be.
Http
requests
might
be
a
function
call,
it
might
be
an
exception.
It
could
be
anything
an
event.
Logging
system,
like
your
elk
stack
or
anything
in
our
gray
log,
would
have
every
single
event,
but
not
too
much
information
about
them,
because
you'll
run
into
bandwidth
issues.
A
metric
space
system
ran
up
in
previous
graph.
A
A
So
there's
a
few
different
approaches
that
you
will
see
if
you
look
across
all
the
different
modeling
systems,
like
some
IOT
devices
and
switches
to
applications
and
client,
libraries
and
the
first
and
simplest
one
you'll
see
in
lot
of
devices
particularly
actual
physical
hardware,
devices
is
the
resetting
account.
It
starts
at
zero.
Every
time,
there's
an
event
at
the
interested
increment
and
then
whatever.
However,
the
data
is
transferred
for
every
push
or
pull
it
sends
the
current
count
and
resets
to
zero,
and
then
you
no
more
events
come
in.
A
It
has
the
data
and
then
the
data
is
transferred,
it's
reset
to
zero,
and
it
keeps
on
working
like
that,
nice
and
simple,
and
here's
how
this
works
in
normal
operation.
So
this
is
showing
10
per
second
and
it
comes
along
under
the
big
spike
up
to
50
per
second
for
a
10
second
period.
So
my
simple
example-
and
if
you
look
at
what's
going
on
it's
fine,
it
captures
this.
A
It's
a
little
laggy,
that's
fine,
but
it
perfectly
captures
what's
going
on
and
that's
all
fine
and
the
problem
is,
though,
that
it's
not
quite
perfect.
So
one
thing
if
a
transfer
fails
you've
lost
at
that,
and
the
thing
is,
though,
that
you
might
have
to
realize
that
this
is
not
a
random
uncorrelated
event.
A
You
know
speaker
pulling
in
the
data
promised
to
release
each
see
half
of
the
information
we've
got
three
places
or
someone
just
texted
me
in
their
workstation,
using
a
full
model
of
that.
That's
not
gonna
work
out
too
well.
So
this
is
not
generally
a
good
approach,
except
in
very
specific
circumstances,
and
so,
if
we
presume
that
we
lost
a
single
point
right
here
at
the
failure,
we
don't
see
it
at
all
in
our
graph
and
this
the
blue
lines,
the
true
line-
and
we
just
see
it-
everything
was
perfect.
It
was
normal.
A
There
was
nothing
out
of
the
ordinary
you'd,
completely
miss
the
package,
ooh
that
big
5x5
familiar
data,
so
it
has
that
disadvantage
the
second
approach.
Now
this
one
is
very
popular,
so
drop
wizards
meter
for
everyone,
familiar
with
the
very
popular
java
library,
top
wizards
and
it's
porting
into
basically
every
other
language.
If
anyone's
using
go
metrics
in
go,
that's
trough,
wizard.
Basically,
it's
a
port.
If
there's
a
few
orders
in
other
languages
as
well,
it
works
the
same
way
as
units
load
averages.
A
So
it's
a
bit
way
of
saying.
Well,
we
take
the
current
value
and
we
take
20%
of
this
and
80%
of
the
old
value
and
we
add
them
together
and
the
next
tick.
We
do
the
same
thing
again,
so
it
takes
the
current
value,
adds
it
in
and
smooths
things
from
history,
and
you
do
this
regularly,
every
5-10
seconds
or
whatnot
and
there's
a
weighting
factor
deciding
how
much
you
care
about
old.
How
much
you
care
about
you!
A
It's
adopted,
for
example,
would
evaluate
that
every
5
seconds,
that's
how
it
works
and
in
normal
operation
here's
how
it
looks
out,
so
that
works
mostly
yeah,
and
that
comes
along
it's
easy
and
because
it's
normally
averaging
Oprah
one
minute.
It
just
plays
this
overtime,
and
so
it's
actually
reading
the
accurate.
What
respect
for
a
one
makers
average-
and
it
just
explanation-
see
the
stays
here.
A
There
are
problems
with
the
expiration
of
the
average.
The
first
is
that
the
codes,
your
dis
king,
exponentially
events,
are
not
considered
equivalently
if
you're
transferring
data
every
10
seconds,
and
it's
doing
that
take
every
five
seconds.
Data
into
five
seconds
receiving
your
scrape
master,
is
more
than
the
five
seconds
or
force
because
of
the
decay,
so
that
makes
the
map
kind
of
tricky
to
figure
out.
If
you're,
not
sampling
is
the
same
as
the
tick
interval.
A
An
SPL
is
five
seconds
is
quite
frequent
to
the
data,
and
it
also
means
because
it's
got
built
in
one
minute,
five
minutes
and
fifty
minute
weightings.
If
you
want
some
other
race
or
what
to
get
things
faster
or
slower,
you
can
do
that
and
it's
kind
of
resilient
missing
a
scrape
right.
You
can
see
that
ish
will,
you
know,
will
probably
get
here
you're
kind
of
the
information,
but
you're
not
going
to
see
for
the
full
spike,
but
it's
not
terrible,
but
it
has
issues
and
there's
also
a
general
issue.
A
We
need
to
talk
about
your
Radha
come
across
continuously
in
signal.
Processing
might
also
know
system
like
he
knows,
even
with
diagram
is
aliasing
so,
depending
on
exactly
and
precisely
where
you
squint,
when
you
straight
when
the
data
is
coming
from,
you
might
see
artifact
and
something
called
like
in
the
question
challenge
here,
which
states
you
can
only
see,
features
that
are
see
half
the
frequency
of
what
you're
scraping
at
and
I've.
A
Even
then,
the
part
of
the
problem
at
to
extract
that
frequency
so
moving
a
second
in
either
direction
when
you're
evaluating
something
to
make
a
massive
difference
in
what
you
see
so,
who
here
is
used
for
fatter
and
had
a
pile
of?
It
is
more
to
different
status
sentence
and,
had
you
know
graph
and
hit
refresh
and
the
shape
of
the
completely
changed,
and
then
they
were
freshly
changed
again.
That's
all
aliasing
effects,
because
basically
we're
scraping
people
do
not
see
the
true
nature
of
the
graph.
A
This
happens
all
the
time
and
it's
just
the
fact
of
life
and
now
probably
shouldn't
scrape
more.
You
switch
two
logs
at
that
point
because
trying
to
do
that
sort
of
analysis
of
metrics,
you
know
how
fast
you
need
to
scrape
with
every
millisecond
nanosecond
microseconds
you
switch
logs
and
you
think
you're,
where
this
is
the
thing
that
happens,
and
this
also
brings
me
to
something
else
and
a
confusion
that
sometimes
come
off
print
for
reuse.
It's
like
why
don't
you
work
like
graphite
summarize
function.
A
It
gives
me
the
right
answer
turns
out
it
does
it
gives
you
an
answer
which
would
appear
to
be
correct
via
and
then
I
even
Alice.
So,
while
those
is
that
say
summarized
in
its
simplest
form,
say
right:
how
many
requests
did
I
have
inside
the
last
hour
and
we'll
go
and
look
at
data
points
in
that
last
hour
and
add
them
up
because
they've
already
universe,
the
gauges
and
and
there
it
is?
A
The
problem
is
that
up
to
that
point,
the
1302
I
rely
on
the
hour
that
1302
point,
let's
say
der
for
15
minutes
until
two
minutes
from
the
hour
and
30
minutes
from
outside
the
hour.
So
it's
all
accurate
because
they're,
including
all
this
data
and
you're,
missing
all
this
data,
so
you're
getting
an
aliasing
issue
there,
so
summarize,
is
not
accurate,
because
the
data
is
extremely
unlikely
to
be
aligned
and
even
imagine
normally
you're
going
to
have
things
staggers
because
you
want
to
spread
load
around.
So
it's
always
going
to
be
an
accurate.
A
This
is
not
just
me
ripping
on
the
summarized
function.
It
is
not
possible
for
the
metric
space
system
to
get
this
white
Prometheus
also
produces
the
wrong
answer.
It's
merely
a
different
wrong
answer.
I
believe
it's
a
better
wrong
answer,
but
just
be
aware,
these
does
all
trade-offs
we
made
here
like
this
is
not
an
unreasonable
way
to
do
it
in
graphite,
but
because
of
this
issue,
you're
going
to
you
know,
the
answer
is
just
not
what
you
want.
A
An
accurate
answer,
use
logs
and
but
thing
is
both
previous
solutions
are
kind
of
designed
with
the
belief
that
the
monitoring
system
on
the
other
end
is
relatively
dumb
right.
We've
got
something
if
you
just
go
to
graphite,
which
doesn't
have
a
phone
getting
federal
defendants
of
that
powerful
query
language
or
something
like
open,
Fiesta
Vishal.
You
very
recently
started
to
get
a
query
language
only
like
an
the
release
like
two
months
ago.
That
starts
to
get
one
if
you've
got
something,
that's
basically
just
a
dumb
data
store.
A
A
It's
you
know
if
I
give
you
the
system
where
it's
only
a
dumb
database
having
some
inaccuracies
is
better
than
not
having
nothing
right.
That's
just
engineering.
We
have
to
make
trade-offs,
but
we
can
do
something
a
little
better.
So
the
approach.
The
third
approach
is
the
approach
taken
by
communities
counters
and,
like
the
resetting
counters,
we
have
a
character
to
start
to
zero
and
isn't
corrected
in
the
exact
same
way
and
it's
regularly
transferred,
but
it
is
never
reset.
A
It's
only
every
step
in
the
process
restarts
and
then
the
rate
function
takes
its
in
and
title
is
how
quickly
it
increases
or
decreases
over
time
well
in
Grace's,
only
increases
tenders,
topiaries
and
so
a
normal
operation
will
see.
This
is
the
correct
answer.
Mathematically
because
this
example,
the
examples
purposely
don't
of
aliasing
because
that's
you
know
another
problem,
but
this
is
the
100%
correct
answer.
What
one
minute
average
looks
like.
So
that's
exactly
what
we
launched
out
of
the
correct
mathematical
answer,
because
everything's
aligns
perfectly
and
those
resilient
to
failed
transfers.
A
You
lose
resolution
or
data.
So
that's
good!
You
can
still
see
that
hey,
there's
more
requests
than
usual,
just
can't
tell
exactly
when
they
happened.
Multiple
places
can
pull
that
a
from
before
pushed
over
whatever
no
problems,
and
you
can
choose
whatever
time
you
want
savage
over
that
you
want.
Data
is
considered
uniformly.
A
It
doesn't
matter
if
it
was
like
the
first
five
seconds
to
the
last
five
seconds
in
ten
minute
it's
over
to
the
uniformly
and
it's
extremely
implemented
easy
to
implement
on
the
client
side,
because
you
just
the
character
that
goes
up
you,
you
know,
ease
new
texts
and
so
on,
but
that's
robbery,
impleaded
intimate
now.
If
there
is
a
fails
transfer,
let's
say
for
whatever
reason
that
this
transfer
around
here
fails.
You
just
see
the
data
you
know
10
seconds
later,
so
that's
you
know
pretty
good.
We
still
Hey.
A
A
Well,
so
now
perfect,
but
still
pretty
good,
and
so
then
looking
at
these
ways,
they
talk
about
the
rate
function
itself,
because
we
were
confused
about
some
of
the
behaviors
of
the
rate
function
because
it
turns
out
the
internals
are
complicated
and
took
us
a
while
I
think
Bjorn
did
a
lot
of
work
for
this,
as
did
I
and
because
they're
the
real
world
kicked
in
processes
restart.
It
turns
out
yeah
escapes,
get
missed
time
periods
of
omission
because
of
aliasing
they
rarely
align
perfectly
and
that
you
know
things
start
and
stop.
A
Maybe
your
project
goes
away
or
you
deploy
a
new
version.
All
these
things
happen
and
we
need
to
deal
with
them
in
a
way
that
is
largely
correct
or
these
good
enough,
because
we
can't
do
it
perfect,
because
this
is
all
metrics
well
perfect.
You
need
logs
and
to
ignore
general
relativity
entirely
difficult.
So
the
first
problem
we
have
is
counter
resets
right
counters
moves
at
to
zero
and
basically,
just
normally
only
ever
happens
when
a
process
restarts.
A
So
you
imagine,
it's
they're
still
seeing
the
same
port
same
target
restarts
it's
now
a
zero
because
she
wants
to
have
maintained
straight
across,
runs
right
so
because
we
know
the
counters
only
every
increase.
What
we
can
do
is
say
hey
that
counter
just
went
down.
Therefore,
it's
reset.
So
if
we
see
lost
10
to
5,
while
we
know
that
in
total
well
went
from
10
to
5,
so
the
counter
reset
there.
A
So
we
had
10
so
went
north
of
10
to
the
10
of
increase
where
10
to
5,
so
this
by
of
increase
so
15
total
and
that's
what
we
see
by
contrast,
if
you're
looking
at
graphite
or
inputs
to
be
there,
it's
the
closest
thing
they
have
to
the
rate
function
is
the
non-negative
derivative
function
and
what
I
would
do
is
would
actually
ignore
that
drop
and
just
report
your
increase
of
10.
So
we
get
a
slightly
better
answer
here
and
the
second
question
of
myth,
scrapes.
Well,
we
missed
a
scrape.
It's
fine.
A
We
said
that
data
points
around
it
and
everything
just
works,
but
if
what
happens
just
at
the
edge
of
the
time
period,
we're
considering
that's
more
complicated
and
because
it's
very
rare
that
the
race
will
align
perfectly
when
the
scrapes
are
especially
what
you
have
more
than
one
target
and
the
fact
that
things
are
in
per
media
to
be
staggered
on
one
millisecond
intervals,
based
on
the
hash
of
the
target
ables,
so
they're
going
to
be
spread
all
over
at
five
minutes
pretty
evenly
and
under
10,000
millisecond
resolution.
So
you
know
have
fun
with
that.
A
So
think
of
me,
if
information,
if
data
points
are
near
the
edge,
we
need
to
extrapolate
edge
to
get
the
right
answers,
and
this
is
where
one
of
the
big
confusions
around
great
patents,
so
about
increase,
which
is
rates
brudder,
so
race
returns
per
second
results.
Increase
is
just
syntactic
sugar
around
that.
A
So,
if
you
have
rate,
5m
increase
is
the
same
as
saying
rate
5m
x,
300,
that's
only
syntactic
sugar,
because
people
keep
on
asking
yes
produce
anyway,
and
so
let's
say
we
have
data
and
at
t
equals
1
dot
has
10
T
equals
6
is
12
and
T
equals
levels
13.
So
this
has
clearly
gone
up
by
3
over
this
time
period
and
we
want
to
look
over
what
pull
from
T
equals
0
at
T
equals
15
and
see
what
the
rate
is
per.
A
Second
now
you
could
look
at
this
and
say
hey
over
10
seconds.
It's
gone
up
by
3,
therefore
hey
it
said
3
over
10
seconds
and
that
the
increase
is
free.
However,
it
could
look
over
to
15
seconds
it's
really
over
4.5.
So,
even
though
we've
got
all
the
integers
here,
because
they've
extended
from
the
10
seconds
that
we
have
in
data
to
the
15
seconds,
we
need
to
add
an
extra
5
seconds
per
sabasha,
so
just
hit
internet
more
and
we
end
up
with
4.5.
A
It
isn't
the
correct
result
on
average,
because
on
average,
if
you
look
at
it
at
different
periods,
it
is
4.5.
It's
just
because
you're
told
hey
it's
free.
Therefore,
it's
literature,
no
it's
a
floating
point
number
and
that's
how
integers
end
up
as
floats
I,
have
a
diagram
for
this
one.
So
your
data
points
are
at
1
and
11
and
I
got
the
other
data
point
here.
It's
an
average.
So
that's
fine,
and
if
we
look
at
literally,
we
just
say:
hey,
that's
3,
but
the
time
period
is
further.
A
So
you
have
to
extrapolate
back
to
nine
point,
seven
and
forward
to
fourteen
point
two
effectively
and
we
end
up
then
with
the
4.5
across
rather
than
the
tree.
That
would
be
here.
So
that
makes
sense
it's
way
on
yeah.
This
is
why
that
answer
happens
and
as
well
at
time
series
of
creation
destroyed.
If
we
always
extrapolate
out
the
edge,
the
problem
is
of
a
time
series.
Let's
say
we're
doing
an
increase
over
an
hour
and
time.
A
Tears
only
exist
for
five
minutes
with
way
overestimate
through
all
of
the
stop
relation
at
the
edge,
which
will
be
a
problem.
So
we
need
to
detect
that
so
we
basically
get
all
the
points
see
how
far
they
are
apart
from
each
other
average.
That
does
that's
good
enough
and
what
we
do
is
if
the
first
point
is
within
one
interval,
plus
some
slack
will
extrapolate
and
on
the
other
end,
if
same
thing
at
the
end
is
110%
of
the
interval
will
extrapolation.
A
If
it's
not,
it
means
the
time
series
we
think
it
started
or
stopped
and
we'll
just
extrapolate
cific
assess
the
interval
principal
is
if
the
time
series
disappeared.
Well,
it
could
have
disappeared.
The
process
could
have
died
anytime
from
the
scrape
and
when
the
next
script
should
roughly
on
average,
it
disappears
in
the
middle.
So
an
average
will
get
the
same
results
and
they
do
the
same
to
start
something
appearing
and
one
other
saying
we
know
counters
can't
go
negative
because
they
start
at
zero
and
only
ever
increase.
So
we
don't
extrapolate
below
zero.
A
So
here
are
all
the
examples.
So
this
is
the
diagram.
This
took
quite
a
while
to
produce,
looks
very
silly
if
you
don't
know,
what's
going
on
of
them
so
we'll
take.
This
is
the
simplest
example
here
this
fella,
the
big
long
one,
and
what
we're
doing
is
that
this
is
just
happens
to
start
perfectly
aligned
so
simple,
but
at
the
end,
because
the
next
data
point
will
be
here
outside
the
window.
A
We
extrapolate
forward
and
we
calculated
based
on
that
first
point,
and
that
last
point
with
this
one
here
going
back
one
and
at
the
start.
Well,
we
don't
exactly.
It
goes
zero,
but
it
is
within.
You
know,
an
interval,
so
we
extrapolate
back
to
there
and
at
the
end.
Well,
it's
not
close
enough,
because
the
next
point
will
be
there
and
that's
within
the
window.
A
Could
you
know
it's
here,
so
we
just
replace
50%
and
then
this
one
here
we
extrapolate
at
the
end
once
again
to
50%,
because
we're
saying
on
average,
it
probably
disappeared
halfway
through
the
slides
will
be
online
later
I.
Also,
my
slides
online,
you
because
yeah,
you
need
this
text,
but
this
is
what's
going
on.
This
is
just
explaining
all
the
cases
of
how
we
try
to
get
this
as
right
as
possible,
so
we're
correct
mostly
on
earth.
A
And
there
is
another
issue:
is
that
and
sometimes
happens
with
just
time
series
don't
always
exist,
so
we
kind
of
presume
that
will
see
the
time
series
it
will
see.
The
0
will
see
everything
increasing
from
there.
The
problem
is
sometimes
time
series
spring
force
with
a
value
of
one
and
then
keep
that
one
forever
and
we
look
at
that
mathematically.
Well,
it's
okay!
What's
the
increase
in
well,
there
isn't
an
increase
its
flash.
A
Well,
someone
has
just
created
this
this
series
and
once
it's
a
yes
I,
would
the
fact
that
one,
because
that
means
an
error,
I
care
about.
Well,
we
have
no
idea
when
that
increased
from
zero
to
one
happened,
because
when
time,
series
data
Boyce
standpoint
for
all
we
know
this
application
has
been
running
for
years.
That
increased
happened
a
decade
ago
and
we
just
never
saw
because
release
is
only
turned
up.
So
we
can't
assume
anything.
We
don't
know
and
time
series
database
wouldn't
have
this
information.
A
All
and
thieves
is
a
single
time
series
with
no
context
because
that's
how
it
works,
and
so
the
thing
is
that
we
are
generally
intermediates
we're
not
trying
to
catch
100%
of
everything.
So
answer
as
well,
we'll
see
the
next
increase,
so
you
fightin
to
have
you
know
a
different
solution
if
you're
trying
to
get
100%
reliability
because
previous
makes
several
trade-offs
around
availability
versus
precision
and
accuracy.
A
And
so,
if
you
do
care
about
this
logs
logs
are
more
actors,
especially
if
you
get
the
type
that
are
actually
you
know
CP
or
make
sure
your
counters
are
initialized
when
your
process
starts
to
zero.
So
we'll
see
that
because
then
the
race
is
just
down
to
that
first
scrape
or
just
other
from
the
fact
that
it's
wooden,
if
you
actually
care
about
that,
because
really,
if
you're,
indicating
hey
I'm
in
an
error
stage,
that's
more
engaged
in
the
counter
tape
and
there's
not
a
problem,
and
this
is
the
general
problem.
A
You'll
see
across
most
monitoring
systems,
a
novel
scientific
ones
tend
to
approach
this
differently.
There's
lag,
you'll
notice
in
all
the
diagrams
we
had
of
what
was
going
on,
that
there
was
lag
of
the
Lisa
scrape
interval
of
seeing
the
spike,
because
all
of
them
are
looking
back
in
time
right.
So
if
I've
got
a
5
minute
rate
in
a
given
time,
it's
really
a
5
minutes
average
of
the
last
5
minutes.
It's
not
centered
on
now
and
similarly
we're
resetting
counters
will
have
the
exact
same
issue.
A
Exponential
moving
averages
is
much
more
complicated
to
explain,
but
the
exact
same
thing
is
happening.
It's
always
a
lagging
view
and
that's
way
it
is,
and
one
of
the
reasons
for
this
is
that
if
we
were
looking
at
future
data,
that's
sort
of
becomes
very
complicated
because
it
means
if
I
was
to
evaluate
a
query.
And
then
new
data
comes
in
the
value
of
the
query.
Changes
that
will
be
very
hard
to
deal
with.
You
know
in
your
head,
so
one
other
things
to
keep
in
mind.
A
Is
that
rates
roll
in
reader
comparable
for
the
same
period
and
if
you're
impairing
a
5
minute
race
and
a
10
minute
race
or
try
to
divide
them?
That
doesn't
make
sense
sadistically,
so
try
to
keep
someone
race
in
your
entire
intermedius.
You'll
tag
me
at
the
worst
I
ever
came
across
was
was
it
17
or
23,
or
something
like
that
rates
inside
a
single
file
yeah
because
when
I
admit
understand
how
things
work
single
rate
inside
interpreters?
A
Unless
you
know
what
you're
doing
maybe
might
be
ok
insert
special
circumstances
as
long
as
you
never
compared,
and
so
we
look
to
have
Prometheus
those
things.
There
are
a
few
more
complications
than
the
science,
because,
yes,
it's
a
cap
rate
goes
up.
That
sounds
easy
to
implement,
but
you
need
to
worry
about
the
real
world
which
includes
things
like
threading,
because
threading
is
a
ting
and
has
been
for
some
decades.
A
Concurrency
handling.
What
you
have
available
for
you
varies
by
language.
Mutex
is
pretty
much.
Every
language
has
new
taxes,
but
they're
also
the
slowest
so
you're,
normally
talking
about
100
nanoseconds
for
a
mutex
except
Python,
which
is
100
microseconds
or
in
fact
was
it
500
microseconds,
it's
ridiculously
so
I
prefer
reasons
I'm
aware
of,
and
then
there's
a
taluk,
so
tommix
tend
to
be
around
the
10
15
nanoseconds
age
and
then
the
best
one
of
all
is
the
actually
of
per
processor
values.
A
So
you
notice
your
cache
coherency
across
the
processors
and
so
Java
actually
approximates
this,
so
the
Java
side
actually
handles
conservancy
the
best
and
but
the
public's
are
perfectly
fine,
forever
juice
and
some
languages
only
have
new
taxes
like
the
go
client
is
using
Atomics.
The
Ruby
client
doesn't
actually
have
any
concurrency
handling
because
Ruby,
because
it's
presumed
it's
only
single
threaded.
So
why
would
you
need
it
yeah?
A
So
by
comparison,
if
you
look
at
how
the
droplet
of
nature
h-has
in
current
four
numbers,
so
that's
actually
using
using
the
atomic
sort
of
same
double
adder,
we
have
the
Java
ties
anyway.
The
increment
is
quite
efficient,
but
there's
four
of
them,
because
it
has
to
shoot
digital
and
it's
got
the
one
minute,
five
minutes
and
fifteen
minutes
and
also
has
to
do
the
decay
logic
as
well.
Every
five
seconds
so
works
out
the
drop
of
the
Demeter.
If
you
the
benchmarks
it's
about
six
times
slower
than
previous
counter,
just
your
instrumentation.
A
I
will
mention,
know
the
job
wizard
counter,
which
is
actually
a
gauge
because
it
can
go
down
and
is
as
fast
as
prometheus
counter
and
because
it's
using
the
exact
same
double
adder
that
they've
Java
client
uses
so
that
one's
fine
is
just
the
meter
which
is
doing
all
this
exponential
and
decaying
Windows.
That
actually
has
performance
implications
and
there
is
an
inner
implication
which
I
wrote
recently
in
the
blog
post:
stay
pretty
stable
for
the
child.
A
So
normally,
if
you
are
using
the
PI
compliant
or
one
of
the
owners,
you'll
notice,
it's
my
metric
set
by
the
labels
and
then
increment
it.
But
if
not,
you
know,
specified
labels
inside
the
increment
call
here,
and
this
is
because
we
have
this
thing
returns,
what's
called
a
child
across
the
languages
and
it's
basically
a
map
here,
a
concurrent
hash
map
using
Java
and
that
mapping
actually
expensive,
so
it
takes
about
a
hundred
nanoseconds
or
so
Ned
Java.
A
So
if
you
think
about
that
increment
I
told
your
comment
is
taking
between
10
and
20
nanoseconds
to
thing
you
have
intended,
it
is,
and
then
you
add
a
hundred
nanoseconds
just
to
find
the
thing
you
need
to
increment.
That's
pretty
serious,
like
that,
makes
things
six
times
slower,
that's
bad!
So,
instead,
if
you
can
keep
a
pointer
to
this
child,
that's
much
faster,
and
this
is
why
we
do
it
this
way.
A
So
someone
if
they
know
in
advance
what
label
values
they
have,
which
is
a
fairly
common
thing
and
you
can
generous
no
cash
stash
as
a
pointer
directly
to
the
child,
the
increment
and
that's
much
faster.
So
if
you
are
doing
anything
like
that,
that
helps
can't
all
will
do
it
and
that's
handy,
but
if
you're
taking
hey
I'll
cash
it
inside
a
concurrent
hash
map.
There's
no
point
in
doing
that,
because
this
already
is
a
concurrent
hash
map.
A
You
can't
get
more
faster
than
that
for
me,
some
Java
and
similarly
any
other
languages
and
Dinard
is
a
relation
to
anti
passing
new
systems
or
people
will
create
a
map
from
metric
names.
Symmetric
objects!
Don't
do
that!
You
know
the
name
of
your
metric
just
keep
a
pointer
because
otherwise,
well
your
app
once
again
we're
going
to
have
this
hit
and
there
are
some
disorders
as
best
practices,
then
mention
around
around
counters
use
seconds
for
timing
seconds.
A
There
is
standard
everyone
uses
seconds
right
and
then
put
that
name
inside
metric
and
there
are
other
systems
you
can
see
it
sometimes
with
other
things
that
they
only
support
integers
inside
the
database.
So
the
developer
has
to
choose
well
I'm,
going
to
use
seconds
microseconds
milliseconds
seconds
minutes
hours
days,
I've
seen
all
of
these
instrumented,
sometimes
inside
the
same
binary
like
there
was
a
time
when
Prometheus
itself
was
using
four
different
time
units
in
the
text
edition
there
isn't
Prometheus.
No,
we
might
have
better.
A
This
is
silly
just
use
seconds,
we're
using
phone
point
numbers
so
it'll
automatically
handle
the
exponent,
because
the
thing
is
that
what
units
you
just
have
in
your
graphs,
it's
display
level
problem
that
should
be
hanging
up
a
graph
and
it
shouldn't
be
the
concern
of
someone
who's.
Actually
writing
the
code.
It's
just
to
write
new
seconds.
Profiler
will
take
care
of
it.
That's
the
way
to
approach
this
in
the
same
way
that
you
don't
choose
your
time
zones
down
in
your
code.
A
Don't
choose
your
units,
that's
the
problem
for
the
display
there
solve
the
display
there.
Similarly,
increase
function,
it's
handy
for
display,
but
in
your
rules
in
New
York's
position,
still
one
use
range
to
record
just
so
once
again,
it's
per
seconds
nice,
consistent,
simple
and
as
I
mentioned
increase,
is
only
syntactic
sugar
for
race
did
that
same
and
there's
no
extra
logic
in
there.
It's
just
a
multiplication
to
make
your
life
of
them
easier.
A
There
is
one
other
rate
function
called
Irish:
it's
not
an
angry
function,
although
I
think
that's
mentioned
in
the
docs
and
in
fact,
I
need
to
clean
up
the
docs,
because
that
jokes
getting
a
little
stale.
Oh
it
does.
It
looks
at
the
last
two
points
in
the
range
right
and
just
says
how
faster
increasing
so
it's
extremely
responsive.
A
You
notice
what
race
it's
a
one
minute,
a
version
the
same,
the
yesterday's
many
windows
are
kind
of
a
1
minute
average,
but
that
means
any
Spiker
so
on
its
average
over
a
minute
and
which
can
be
handy
if
that
is
very
spiky,
because
it
turns
out
humans
are
really
not
good
at
dealing
with
spiky
data
see
what
average
admission,
but
if
so,
there
is
irate.
If
you
want
to
see.
A
A
In
the
perfect
case,
like
you'll
see
this
works
exactly
the
same
and
give
us
that
same
perfect
answer.
We
solve
the
setting
counsellors,
which
is
good,
I,
know
most
alike,
though,
as
I
mentioned
happens
as
well,
when
a
sales
transfer,
let
us
say
this
transfer
here,
fails
we
missed
this
and
we'll
still
see
a
spike.
It
will
just
be
not
quite
as
big
so
you're
still
seeing
this
fight
it's
resilient.
It's
not
the
perfect
answer,
but
we
have
reasonably
gracefully
handled
a
failed
scrapes
that
happen
because
well
there
was
an
overload
going
on.
A
So
this
brings
a
question
to
be
careful
of
missing
data.
The
query
range
facially
endpoint.
So
that's
what
Griffin
is
using
basically
says:
hey
here's
a
star,
here's
an
inch
if
they
already
displease
his
discretion
at
each
of
those
data
points.
It's
just
a
handful.
It's
a
slightly
optimized!
Where
of
saying
run
this
query
many
many
times
five
seconds
apart
or
ten
seconds
apart
for
20
seconds
apart.
A
But
if
you
have
a
10
minute
step
like
you're
saying
is
right
run
is
over
the
last
week
every
10
minutes
with
a
5
minute
race,
you're
missing
after
each
of
those
requests,
is
only
looking
at
5
minutes
of
data
and
it's
got
a
5
minutes
before
it's
missing.
So
you
need
to
be
careful
to
need
to
make
sure
your
rate
range
is
generally
at
least
the
size
of
your
step
and
in
I
race.
You
want
your
step
to
be
no
larger
than
your
sample
resolution.
A
So
they'll
always
have
those
2
points,
even
if
it
is
a
failed
transfer.
So
keep
that
in
mind,
otherwise,
you're
really
missing
data
and
not
getting
a
true
average
they're
missing
and
also
just
mention
the
compound
types
in
the
scholar
summary.
How
do
you
track
like
the
scene?
Prometheus
2,
counters
yay,
you
just
count
tilted
requests
your
carry
the
total
latency.
You
get
the
rates
of
those
divided,
and
now
you
have
average
latency
o
for
any
time
period.
A
It
also
has
the
same
count
of
some
that
somebody
has,
but
really
is
forecasting
cartels
in
Prometheus
because
it
all
does
is
had
counters,
which
are
buckets
and
which
indicates
the
histogram
you
take
the
reed
to
both
of
those
and
which
you
can
aggregate
up
then,
and
then
you
figure
out.
Ok.
So
where
is
the
90th
percentile
here
so
that
works
fine?
But
beware
of
cardinality
explosion
because
there's
general
rule
in
Prometheus,
you
probably
want
to
keep
your
cardinality
below
10
on
the
instrumentation
and
histogram
as
a
box
of
the
cardinality
of
10.
A
So
you
need
to
be
careful
general
with
cardinality
there's,
maybe
literally,
for
every
Prometheus
you're
allowed
a
handful
of
exceptions,
so
choose
wisely,
where
you're
going
to
be
so
sometimes
hey.
If
you've
got
12
times
the
logs,
you
digress.
Well,
there's
your
cardinality,
that's
life,
but
you
should
be
very
careful
in
instrumentation,
particularly
in
libraries,
not
to
blow
out
how
much
performance
from
you
can
deal
with,
because
monitoring
is
not
free
monitoring
costs,
money
and
time
and
resources
and
humans
do
have
to
six.
A
You
know:
go
to
retune
everything
so
at
summer,
eats
without
quartiles
are
basically
free
as
soon
as
you're
starting
adding
labels
or
quantiles
or
histograms.
Things
start
to
get
more
expensive
just
because
things
are
multiplying
edge
and
that's
main
talk,
the
resources
itself.
You
have
previous
websites.
These
are
dead
mailing
list.
The
source
code
is
in
functions
that
go,
and
it's
also
my
own
blog,
which
has
some
posts
around
this
stuff
and
communities
generally.
So
any
questions
we
have,
you
have
15
minutes
for
questions
or
fourteen
minutes.
A
Okay,
so
the
question
is:
are
recommended,
scrape
intervals
on
Prometheus.
It
basically
boils
down
to
how
much
data
and
how
accurate
about
you
need
it,
because
the
way
I
see
it.
Yes,
you
could
scrape
once
a
second
and
get
really.
You
know
granular
data,
but
you
could
also
scrape
once
every
60
seconds
and
have
60
times
in
time
series
for
basically
the
same
performance
cost
and
which
of
those
is
more
likely
to
help
you
in
an
emergency
like
having
all
your
libraries
interspaces
that
they're
having
this
one
metric.
A
That's
awesome
and
I
would
generally
aim
for
breath
like
just
having
things
liberally
instrumented
with
no
labels
all
to
your
codebase.
Just
so
we
can
figure
out
roughly.
Where
is
my
problem?
So
I
would
generally
I've
found.
Personally,
one
minute
is
actually
fine
for
the
vast
majority
of
things,
and
the
only
time
you
need
down
at
the
second
range
is
when
you're
trying
to
bug,
like
maybe
literally
a
handful
of
metrics
when
you're
trying
to
debug,
you
know
burst
subtract
that
are
in
the
seconds
range,
but
breath
is
generally
best.
A
B
Which
just
will
do
like
an
API
generic
API
red
base?
What
are
the
kinds
of
things
we
would
measure
if
you
don't
know
anything
about
the
API
as
the
light
rate
and
like
latency
or
whatever
I
wonder
if
there
is
a
place
in
the
internet
for
people
who
are
currently
collecting
those
like
I
know:
they're
the
power
it's
re.
Brooke
has
5,000.
B
B
A
In
a
shinto
instrumentation,
what
instruments
was
the
first
part
of
the
question
if
you
look
at
the
best
practices
and
primitive
stereo,
the
instrumentation
best
practices
answer
that
question,
which
is
by
Rotom,
and
the
second
question,
then,
is:
are
there
common
patterns
around
using
things
like
summaries
and
so
on?
So
the
thing
is:
there's
about
five
really
really
common
patterns
in
Prometheus.
It's
like
taking
your
race,
there's
aggregating
that
race
there's
using
historical
toleration,
aggregation
and
there's
summary
gain
equality
house.
There
are
very
simple
patterns,
and
not
very,
very
explicitly.
A
Don't
put
them
in
the
dog
eventually
drum
like
to
do
this,
but
those
are
the
normal
patterns
like
five
or
six
Lamoni
that
you
need
and
opt-ins
the
others.
Only
the
histogram
and
summary
require
any
told.
The
rest
is
just
decoration,
take
a
sum
but
yeah
everything
beyond
that
tends
to
be
so
weird
that
we
basically
need
to
ask
a
prompt
to
our
experts
watching
if
you're
doing,
because
you're
doing
complex
stuff,
yeah
every
person.
A
A
C
A
So
question
is
joy,
needs
put
the
scrape
interval
into
race.
The
answer
is
no
because
it
has
the
vasa
worked
at
house,
but
the
range
should
cover
at
least
two
points,
preferably
tree
to
cover
a
missing
scrape.
So
normally
you
know
two
and
a
half
times
at
least
I'd
like
if
you
had,
let's
say
a
10
second
scrape
interval.
You
probably
want
a
thirty
second
race
at
a
minimum
longer,
if
you
actually
want
to
average
selfish.