►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Doing Things Prometheus Can’t Do with Prometheus - Tim Simmons, DigitalOcean
The current Cloud Native Observability dogma is that metrics (and logs and traces) are “not good enough” and that this brave new world needs brave new Observability tools. This is false.
This session will focus on how to utilize Prometheus and friends to solve problems that are typically cited as limitations. This talk is for anyone interested in learning how Prometheus can solve the majority of your Observability problems, no vendor required.
An outline of this talk is:
- How to thoughtfully utilize existing Observability tools
- Deploying High Availability Prometheus
- Effectively interacting with high-cardinality data
- Long-term metrics storage
- Doing “machine learning” on metrics
- Handling thousands of alerts in a sane way (https://twitter.com/timsimlol/status/1145790451129167872)
- How to measure *everything* with Prometheus
- Fostering a healthy Observability culture with SLOs
https://sched.co/UaXp
A
Hello
everybody
welcome
in
this
is
my
talk
called
doing
things.
Prometheus
can't
do
with
Prometheus.
My
name
is
Tim
Simmons
I'm,
an
engineer
on
the
observability
team
at
digitalocean,
my
team
primarily
runs
centralized
logging,
metrics
and
tracing
infrastructure.
You
can
follow
me
on
Twitter,
but
I
wouldn't
recommend
it,
because
that
is
roughly
the
content
that
you
will
be
there
for.
A
Digitalocean
is
a
cloud
company
that
is
focused
on
simplicity.
We
have
everything
you
need
to
run.
The
workloads
that
you
want
to
run.
My
primary
job
at
do
is
caring
for
a
prometheus
fleet
that
cares
about
metrics.
That
kind
of
support
the
entire
cloud
first
line
in
my
CFP
kind
of
this
a
little
was
clickbait
to
try
and
get
my
talk,
accepted
and
entice
you
to
come
to
my
talk.
Nothing
is
like
this
cut-and-dried.
A
You
know,
there's
a
lot
of
nuance
and
saying
something
like
that
and
like
there
might
be
like
a
cool
tweet
to
do
or
whatever,
but
like
I.
You
know
this
is
not
that's.
Nothing.
I
could
be
easily
convinced
and
I'm
totally
wrong
here,
but
I
do
kind
of
believe
this.
If
you're
willing
to
invest
the
time
and
energy
in
kind
of
understanding,
the
tools
and
the
outcomes
that
you're
looking
for
the
old
pillars
that
nobody
is
so
fond
of
anymore,
I
think
they're
actually
really
really
good
and
good
enough
for
you.
A
The
thing
that
is
not
good
enough
is
us,
and
this
shouldn't
really
come
as
a
surprise
to
anybody
right.
The
problem
with
technology
is
always
the
people
using
it.
You
know
people
will
say
things
like
I,
don't
have
time
to
do.
Good,
observability
or
you
know,
I,
don't
have
time
to
learn
these
complicated
tools,
but
these
are
the
tools
that
are
giving
you
insight
into
how
the
thing
that
like
pays,
your
bills
are
performing.
So
if
there's
anything
that
is,
you
know
complicated.
That
is
actually
worth
learning
and
understanding.
A
Isn't
it
being
able
to
do
this
like
understanding
your
customers,
experience
I,
think
if
you
spend
a
lot
of
time
making
sure
your
customers
are
happy,
you
know
your
existing
customers,
I
think
the
new
ones
will
kind
of
figure
themselves
out,
I.
Think
the
hardest
problem
in
the
observability
space
is
actually
just
convincing
your
boss,
your
leader,
that
it
is
worth
doing
and
worth
doing
well
and
the
way
that
you
can
reliably
serve
customers
and
do
this
is
by
intelligently
instrumenting
your
software.
So
you
can
see
what's
going
on,
but
to
produce
those
best
outcomes.
A
The
goal
here
for
us
today
is
for
you
all
to
start
thinking
about
observability
like
I,
do
at
least
for
30
minutes
or
so
because
I'm
right
and
hopefully
that
is
something
that
is
generally
useful,
as
you
kind
of
leave
here
and
go
any
lunch
or
whatever,
but
I
think
it
will
also
kind
of
help.
You
understand
where
I'm
coming
from
when
I
start
explaining
how
we've
done
some
more
advanced
things
with
Prometheus,
and
that's
probably
the
real
reason
that
you're
here
anyway,
but
while
I
was
writing
the
talk
and
talking
about
those
things.
A
We
often
increment
or
set
metrics
anytime,
something
happens.
So
if
you
have
a
system
that
is
doing
things
every
time
something
happens.
You're
gonna
set
some
sort
of
metric
for
it.
Applications
that
are
utilizing
Prometheus
serve
a
literal
web
page
with
metrics
and
plain
text
that
you
can
go
to
and
look
at,
and
that
is
called
exporting
metrics
and
I'm.
A
Looking
at
metrics
and
logs
and
traces,
there
are
a
couple
of
themes
that
kind
of
instantly
emerge
to
me.
There
are
a
lot
of
promises
floating
around
about
people
giving
you
observability
for
free
or
you
know,
being
able
to
add
it
on
to
something
at
any
time,
and
you
know
get
the
kind
of
outcomes
that
you
want.
In
my
experience,
when
you
just
kind
of
rubs
observability
on
something
like
that,
you're
not
really
gonna
get
the
best.
The
best
outcome,
these
patterns
and
these
tools
need
to
be
kind
of
thoroughly
understood
and
then
utilized
thoughtfully.
A
This
all
takes
time
and
labor
and
because
of
that
time
that
it
takes
to
kind
of,
learn
and
understand
this
and
do
it
well.
People
are
starting
to
talk
about
going
beyond.
You
know
we're
gonna
correlate
these
signals.
We're
gonna,
build
simple
unifying
tools,
some
sort
of
overarching
thing
that
is
going
to
make
this
all
easier,
but
you
can't
really
do
that
because
these
things
are
all
so
different
and
also
kind
of
nuanced,
and
it's
hard
to
do
any
of
that
stuff.
Well,
without
that
core
understanding
and
I
don't
want
anybody
to
sleep
on.
A
You
know
those
old
kind
of
fundamental
pillar
concepts
and
wait
for
the
next
big
thing,
because
it's
going
to
be
built
on
this
same
stuff.
How
about
making
sense
here?
Because
what
I'm
trying
to
say
is
that
none
of
this
is
easy.
You
heard
it
in
the
keynote
yesterday,
you
know
it's,
it's
not
easy
and
we're
trying
to
make
it
easier,
but
even
with
that
top
quality
stuff
that
you're
gonna
get
for
free,
you
still
have
to
understand
how
to
transform
that
into
a
good
result.
A
I
want
us
here
today
in
a
headspace
where
we
are
treating
these
tools
like
the
critical
infrastructure
that
they
are
not
as
something
that
comes
later,
because
that
is
the
way.
The
way
that
we
effectively
maintain
software
and
keep
customers
happy
and
keep
dollars
flowing
getting
really
good
at
this
is
worth
the
investment
and
it's
really
fun
so
Prometheus.
A
The
reason
you
probably
added
this
talk
to
your
schedule
is
because
I
was
gonna,
be
talk
about
doing
fancy
stuff
with
Prometheus
and
I
will,
but
you
need
to
be
ready
for
a
little
bit
of
a
letdown,
because
the
more
I
thought
about
this,
the
more
I
realized
there
isn't
special
sauce.
These
are
kind
of
logical
thought,
processes
that
result
from
being
really
familiar
with
this
tool,
and
these
are
hard
problems
to
solve,
and
the
tools
can't
be
good
at
everything.
A
A
It
is
beautiful
in
its
simplicity
in
that
way,
it's
very
easy
to
kind
of
set
up
and
explore
and
start
to
understand
what
the
value
you're
gonna
get
out
of.
It
is
in
the
future.
As
long
as
resource
usage
is
kept
in
check,
the
network
is
up
and
the
things
that
are
exporting
those
metrics
are
up
you're,
basically
good
to
go.
Sometimes,
though,
the
network
does
break
or
queries
make
Prometheus
sad
or
exporters
go
down,
so
people
want
to
start
talking
about
doing
high
availability
Prometheus.
A
This
is
the
kind
of
thing
that
can
happen
so
I
have
an
instance
here
that
was
running
perfectly
well
for
a
while
it's
a
short
there,
but
you
know
this
thing
is
running
for
weeks.
At
a
pretty
constant
resource
utilization,
then,
all
of
a
sudden
it
doubles
triples
in
resource
usage,
because
somebody
queried
it
big
bad,
but
if
you've
got
ever-
and
this
is
fine,
but
if
you
run
out
of
CPU
or
you
run
out
of
memory
to
processes
query,
then
you
can
slow
an
instance
down
to
the
point
where
it
becomes
unresponsive.
A
This
is
kind
of
the
same.
The
only
thing
that
I
see
kind
of
routinely
you
know
operationally
and
running
Prometheus
for
people
is
that
sometimes
you
know
people
make
it
sad.
So
what
happens
when
you
do?
That
is
that
you
might
end
up
with
a
gap
in
metrics
when
you're
querying
later,
because
you
will
put
that
server
in
a
situation
where
it
can't
go
out
and
reach
for
metrics
anymore.
A
It
also
means
you
know
a
gap
in
a
graph
if
it's
a
normal
day
is
maybe
not
such
a
big
deal,
but
when
it
when
it
is
not
scraping
metrics
like
that,
it
is
also
not
evaluating
alerts.
So
if
you
do
have
something
terrible
happen
in
that
space,
not
only
will
you
not
be
able
to
see
it,
you
won't
get
alerted
for
it,
and
that
is
actually
pretty
bad.
The
way
that
folks
generally
work
around
this
is
by
deploying
replicas
of
Prometheus
that
scrape
the
same
things
and
evaluate
the
same
alerts.
A
A
This
I
now
realize
there
is
a
status
API
that
you
could
probably
also
query.
So
maybe
do
that
too?
Ideally,
you
could
manually
failover
those
instances
in
your
group
so
because
maybe
there
is
a
network
issue
between
Prometheus
and
the
things
that
it
is
scraping,
and
you
know
about
that,
because
you've
got
an
alert
from
somewhere,
but
Prometheus
is
actually
like:
hey
I'm
doing
Gucci
man
like
there's
less
there's
less
metrics
coming
in
and
I'm
actually
happier.
But
then
you
query
and
say:
hey
we're
all
my
metrics.
A
If
you've
got
a
network
segregated
Prometheus
somewhere
that
is
not
subject
to
that
issue,
you
could
go
and
poke
your
proxy
as
they
actually
go.
Look
over
here.
We
need
to
see
what's
going
on
and
then,
ideally
you
would
not
rotate
back
to
that
other
instance
that
has
those
holes
until
the
other
one
has
problems
so
yeah.
A
So
this
is
a
really
high
availability
right,
because
there
are,
there
are
situations
where
this
could
become
unusable,
but
you're
not
really
going
to
be
able
to
tolerate
a
massive
network.
Failure
in
a
system
that
uses
the
network
to
go
out
and
reach
out
for
metrics
so
deal
with
it,
I
guess,
but
you
can't
minimize
those
gaps
right
and
hopefully
you
can
be
able
to
find
them.
You'll
find
those
metrics
that
you
need
somewhere.
A
If
it's
really
important-
and
you
know
as
bad
as
I
made,
that
sound
I've
been
running
from
atheists,
pretty
large
infrastructure
for
years,
and
there's
only
been
a
couple
of
cases
that
I
can
remember
where
I
didn't.
Actually
have
something
that
somebody
needed
due
to
either
a
network,
or
you
know
a
bunch
of
instances
going
down.
There
are
options,
though,
if
you
are
kind
of
wanting
to
go
harder
in
that.
In
that
way,
there
are
things
like
say:
a
nose
query
or
promix
Y,
and
these
promise
to
fill
those
gaps
in
your
metrics.
A
You
know,
even
if
one
of
them
goes
down
and
you
can
have
just
like
one
thing-
that
you're
querying
the
way
they
do.
This
is
actually
by
fanning
out
queries.
So
if
you
have
four
instances
that
are
scraping
the
exact
same
thing,
two
of
them
go
down,
this
thing
will
actually
query
all
four
see
that
there's
a
gap
in
the
first
two
and
then
fill
it
in
you
know
by
deduplicating
data
and
giving
you
one
solid
graph.
A
There
are
some
risks
here,
though,
so
you
will
have
decreased
query
performance,
at
least
in
a
little
bit,
because
there
is
CPU
time
spent
once
that
proxy
goes
fans
out
and
then
kind
of
coalesce
as
those
things
together.
You
also
can
pay
double
the
cost.
For
latency
of
your
query
so
say
you
do
a
big
fat
query
that
returns
like
40
Meg's
of
results.
A
Maybe
your
network
is
slow.
Maybe
it's
not
whatever
that
that
result
will
go
from
Prometheus
to
this
proxy.
Do
some
work
correspondingly
larger
amount
of
work
with
the
larger
your
data
set
is
and
then
send
it
back
along
to
you
and
if
you're
running
queries
that
are
like
15
18
20
seconds,
you
will
kind
of
feel
this
a
little
bit
more.
A
The
other
risk
is
that
if
someone
does
a
really
big
query
or
loads
up
a
big
fat
killer
dashboard
over
like
eight
weeks
or
something
these
queries
will
fan
out
to
all
of
your
Prometheus
and
DDoS
all
of
them,
and
then
you
are
creating
that
gap
that
you
are
trying
to
avoid
in
the
first
place,
because
none
of
your
replicas
are
up
there's
also
some
increased
operational
overhead.
You
know
this
is
another
component
to
run
and
because
Prometheus
is
generally
so
easy
to
operate.
A
This
thing
can
actually
be
as
hard
to
operate
as
Prometheus
itself,
but
there
are
other
benefits
too.
So
you
can
get
something
like
a
global
query
view.
So
you're
not
changing.
You
know
data
sources
in
Griffin
or
trying
to
figure
out
which
Prometheus
has
the
metrics
that
you're
looking
for,
and
that
is
pretty
cool.
Let's
talk
about
cardinality
for
a
second.
This
is
a
quick
refresher,
because
this
is
not
something
that
is
intuitive
for
people
when
they
start
using
Prometheus.
But
it's
probably
the
most
important
thing
for
you
to
know.
A
Every
unique
permutation
of
label
values
in
Prometheus
creates
a
new
time
series,
so
you
can
see
in
that
example,
below
I
have
three
different
time
series,
because
there
are
three
different
kind
of
combinations
of
potential
metric
label
values.
The
official
recommendation
from
Prometheus
is
that
any
individual
query
should
match
hundreds
and
not
thousands
of
time
series.
So
you
see
I've
done
a
query
that
kind
of
matches
three
here,
but
you
can
maybe
think
about
how
if
I
was
doing
a
sum
across,
you
know
HTTP
requests
across
my
entire
application.
A
How
this
could
easily,
you
know,
find
hundreds
or
thousands
of
time
series
you
can
work
out
a
query
at
a
single
point
in
time
to
figure
out
kind
of
how
dangerous
it's
going
to
be.
So
you
can
see
like
how
many
times
series
are
gonna
be
returned.
How
long
did
your
query
take
and
you
should
do
that
before
you
go
and
graph
something
because
the
way
to
the
graph
works?
Is
it
evaluates
your
individual
query
at
many
many
many
points
in
time
and
then
connects
them
all
together.
A
So
if
your
grade
takes
10
seconds,
when
you
evaluate
it
for
the
last
5
minutes,
it's
not
gonna
work
to
you
great,
if
you're
doing
it
over
a
week.
So
basically
what
you
should
do
when
you're
creating
metrics
is
to
try
and
avoid
labels
that
have
unbounded
potential
values
and
then
watch
the
number
of
total
labels
that
you
put
on
a
metric
because
you've
got
a
big
metric
already
and
then
you
add
another
label
that
just
has
ten
values
you
just
10x
the
cardinality
of
that
metric.
A
So,
if
you're,
following
what
I,
basically
just
told
you
to
do,
is
not
create
metrics
when
you're
creating
metrics.
In
my
experience,
running
Prometheus
for
people
generally,
any
query
that
is
worth
doing
touches
thousands
of
time
series,
some
of
them,
even
hundreds
of
thousands,
so
we've
got
to
find
a
way
to
work
with
this.
Luckily,
if
you're
querying
under
about
mm
time,
series
you're
totally
fine
and
then,
if
you
even
go
up
to
like
10,000
or
so
I'm,
not
even
worried
about
you
going
beyond
that,
though,
we
might
have
to
do
some
work.
A
If
you're
going
to
do
those
higher
cardinality
queries,
there
are
some
things
you
can
do
operationally
to
Prometheus
to
help
limit
the
worst
of
the
pain.
So
you
need
to
have
significant
amount
of
memory
and
CPU
head
room
while
prometheus
is
running.
A
queried
instance
is
a
completely
different
animal.
Like
I
showed
you
earlier,
you
can
set
a
limit
on
how
many
samples
I'm
sorry
I,
should
give
you
a
guideline
on
head
room.
I
usually
do
50%,
it
still
dies.
A
Sometimes
so
you
know
it's
just
money
right,
so
you
know
just
keep
on
you're
getting
up
or
just
run
it
in
kubernetes
and
let
it
eat
your
entire
cluster.
Whatever
you
can
set
a
limit
on
how
many
samples,
a
single
query,
can
load
into
memory,
so
samples
means
individual
values
inside
of
the
individual
time
series.
So,
if
you're
evaluating
a
query
that
touches
a
thousand
time
series
over
two
weeks,
it's
gonna
have
to
evaluate
that
query,
potentially
thousands
of
times.
A
So
then
you
were,
you
know,
multiplying
that
number
of
time
series
by
a
big
number
I
usually
set
this
to
like
50
million.
If
somebody's
doing
something
that
is
more
than
50
million
like
I,
don't
want
them
to
do
it,
and
prometheus
generally
will
like
kind
of
calculate
the
number
of
samples
that
it's
gonna
load.
It
takes
like
five
seconds
or
so
and
then
go
hold
on
Jack,
pretty
sure
and
I
usually
say
no,
but
I
have
been
known
onto
occasion
on
occasion
to
bump
up
this
limit.
A
For
somebody
who
wants
to
do
more,
you
can
also
set
the
limit
on
maximum
concurrent
queries
that
are
being
done
in
Prometheus.
You
should
absolutely
do
this,
I
usually
set
it
to
one
per
CPU
core
on
the
instance.
The
reason
for
this
is,
if
you
are
doing
like
20
queries
at
a
single
point
in
time
on
a
box
that
has
like
eight
cores,
it
is
going
to
have
to
do
like
a
lot
of
context.
A
Switching
and
it's
not
gonna
go
well,
it's
better
to
wait,
the
you
know
quarter
of
a
second
or
whatever
for
another
query
to
finish
and
then
execute
your
new
one
rather
than
just
kind
of
doing
them
all
at
once.
What
usually
happens
when
you
do
something
like
this
without
this
limit
is
you're
loading,
a
big
dashboard
with
a
hundred
queries
on
it
and
they
will
just
all
go
back.
I'll
come
back
failing
and
then,
if
you
you
know
this
limit,
then
they
can
all
just
roughly
come
in.
A
A
Unfortunately,
they're
the
you
know,
it's
a
big
cloud,
so
they're
answering
that
question
out
of
Prometheus
would
require
a
query
across
you
know
a
giant
global
set
of
data,
so
I
shard
this
stuff
regionally.
This
means
that
people
are
a
little
bit
sad
at
first,
because
they're
like
well
now
I
have
to
do
my
query
for
every
region,
but
you
still
get
your
answer
and
that's
better
than
no
answer
at
all.
And
it's
honestly,
it's
not
that
hard
to
stitch
it
all
together
like
if
you
really
really
care.
A
Isn't
it
worth
doing
a
little
bit
of
work?
I,
don't
know,
that's
how
I
feel
about
it,
but
you
can
also
federates.
Oh,
if
you
you
know
still,
you
could
do
a
regional
sharding
and
then
create
kind
of
a
layered
cake
of
Prometheus,
where
you
are
aggregating
something
at
a
regional
level
and
then
only
putting
that
stuff
in
the
next
tier
up.
So
then
you
could
do
that
global
query
again,
just
a
little
bit
more.
You
know
kind
of
labor
that
you
have
to
do.
A
But
hey
answering
your
question:
when
you
start
going
really
hard
with
queries,
you
need
to
be
kind
of
careful,
be
careful
with
your
aggregation,
so
scope
down
to
only
the
things
that
you
actually
care
about.
You
know
increase
your
step
size
of
your
query
so
that
you
have
at
first
the
lowest
resolution
that
you
can
stand
and
then
you
know
the
same
thing
with
a
smaller
kind
of
time
window
and
then
scale
up.
So
if
you're
gonna
do
some
sort
of
big
query
over
a
week,
you
know
maybe
start
with
an
hour.
A
You
know
five
minute
resolution
and
then
kind
of
work.
Your
way
up,
instead
of
just
like
punching
it
right
in
the
mouth,
with
the
bigquery
there's.
Another
thing
that
you
can
do
is
you
should
be
able
to
understand
kind
of
your
query
pattern
and
the
questions
that
you're
going
to
be
trying
to
answer
with
these
metrics
and
then
create
the
metrics
and
query
to
match.
A
So
this
is
a
lot
of
stuff
to
care
about,
but
again
you're
doing
something,
that's
really
hard.
So
you
have
to
care
a
little
bit.
It's
like
you're
supporting
customer
I,
don't
know
like
what
is
more
important
so
doing
these
kind
of
things.
I
routinely
help
folks,
query
metrics
that
have
hundreds
of
thousands
of
time-series
and
if
you
listen
to
you,
know,
vendors
and
things.
This
is
something
that
you
absolutely
cannot
do
it
for
me,
Thea's
and
I'm
here
to
tell
you
they're
lying
it's
just
a
little
bit
harder
than
it.
A
It's
just
a
little
bit
hard,
but
it's
fun
I
like
to
give
you
an
example
of
crafting
metrics
that
match
desired
outcomes,
because
this
is
something
that
I,
don't
think
people
are
doing
enough.
So
you've
got
some
sort
of
web
service
that
has
potentially
thousands
of
endpoints
that
you
might
care
about.
Latency,
for
you
want
to
know
per
endpoint
latency.
You
want
to
know
latency
per
API
version,
and
then
you
want
to
know
global
latency.
Just
one
single
number.
A
You
should
create
a
separate
metric
for
each
one
of
those
questions,
so
you
will
have
one
metric
that
has
a
label
value
for
every
endpoint
in
your
app
and
then,
if
you
tried
to
do
a
global,
you
know
latency
calculation
on
that
everything
would
fall
apart.
That's
fine!
Because
now
you
have
a
separate
metric
for
global
latency.
That
is
tiny,
tiny,
tiny
and
it
comes
back
instantly
and
that's
really
good
and
then
another
one,
for
you
know
some
sort
of
group
that
you
care
about.
A
A
You
got
to
know
what
to
query
for
what
but
I
think
it's
a
fair
trade-off
to
being
able
to
you
know
actually
answer
your
questions
like
I
said:
spend
a
little
time
understand
your
tools,
because
they're
really
important
long-term
storage
is
something
that
people
routinely
ask
for
in
prometheus
Prometheus,
pretty
recently
released
disk
backed
retention.
So,
basically,
if
you
provision
a
big
old
disk
on
your
prometheus
instance,
you
can
tell
it
to
just
fill
up
the
entire
disk
with
metrics
and
you
know
start
deleting
them
when
it
runs
out
of
disk
space.
A
You
can
do
this
with
a
block
storage
volume
as
well.
If
you
keep
the
maximum
block
size,
that
is
the
size
that
Prometheus
size
and
time
where
Prometheus
will
compact
your
metrics
down
to.
If
you
keep
that
relatively
low
like
three
to
seven
days
when
you
run
out
of
disk
space
and
Prometheus
needs
to
delete
metrics,
it
will
delete
by
block.
A
So
if
you
you
know,
can
let
this
compact
down
like
four
weeks
or
something,
and
then
you
run
out
of
disk
space
you're,
not
deleting
four
weeks
of
disk
space
or
metrics
and
then
freeing
up
like
half
of
your
disk.
You
can
kill
little
little
tiny
bits
at
a
time.
This
has
a
little
bit
more
overhead
when
you
start
querying,
but
in
my
experience
it's
not
so
bad
I
turned
this
on
in
our
infrastructure
about
250
days
ago.
A
So
my
experience
doing
really
long-term
queries
is
somewhat
limited,
but
doing
kind
of
some
of
the
stuff
I
talked
about
before
I've
had
like
perfect
easy
success.
Doing
kind
of
long-term
queries
and
250
days
or
a
year
or
2
years
is
a
lot
more
than
people
need,
even
just
like
four
to
eight
weeks
is
often
enough
for
for
every
question.
There
are
new
solutions
in
this
space,
though
things
like
fan,
oohs,
m3
and
cortex.
A
They
offer
long-term
source
as
well
as
a
lot
of
other
potential
benefits,
so
that,
like
global
query
view
I
was
talking
about
earlier
multi-tenancy
down
sampling
compaction.
You
know
pushing
metrics
natively
scaling
horizontally.
These
are
solving
really
big
problems
compared
to
just
vanilla
Prometheus
and
they
come
with
a
correspondingly
large
amount
of
operational
overhead,
so
the
cost
to
run
any
of
these
things
is
much
much
more
than
Prometheus
and
it
needs
to
be
staffed
appropriately.
If
you're
gonna
do
something
like
this.
A
My
opinion
is,
if
you're
looking
at
something
like
this
just
for
long-term
storage,
you
should
think
about
whether
you
really
need
to
keep
that
data
long-term
in
the
first
place.
In
my
experience,
it's
a
thing:
people
think
they
want,
but
almost
never
use,
and
the
problem
is
that,
especially
if
you're
considering
a
big
solution
like
one
of
these
big
boys,
the
expense
and
operations
and
money
is
a
lot
higher
for
the
relative
utility
that
you're
actually
getting.
My
recommendation
for
long-term
storage
is
to
throw
that
big
fat,
disc
Prometheus
and
just
let
it
go
hog-wild.
A
If
that
doesn't
work,
you
know
you
could
think
about
setting
up
a
long-term
prometheus
instance
and
federating
only
that
kind
of
aggregated,
long-term
trending
data
that
you
really
want
to
keep.
And
if
you
do
that
and
keep
that
data
relatively
small,
you
know
even
just
like
four
four
gigs
of
RAM
in
a
you
know:
100
gig
disk
you're
gonna
keep
that
stuff
for
years,
but
I
say
it
again.
You
have
to
do
that
labor.
A
You
have
to
think
about
the
solution
that
you
want,
and
you
know
kind
of
care
about
your
outcomes
and
then
you've
got
a
campaign
for
the
time
to
do
that
and,
ironically,
it
might
even
be
easier
to
convince
some
leaders
that
you
should
stand
up
a
large.
You
know
new
cool
system
because
it
has
kind
of
that
cloud
native
clout,
but
if
you
don't
need
all
of
that,
you
really
shouldn't,
because
it's
gonna
be
very
expensive.
A
Once
people
start
getting
value
out
of
metrics,
a
common
idea
that
folks
have
is
wanting
to
do
machine
learning
or
anomaly,
detection
on
and
I'm
sure.
This
is
possible.
I've
heard
a
lot
of
amazing
stories
about
what
machine
learning
can
do.
I've
personally,
never
seen
on
even
an
outcome,
good
or
bad.
A
So
this
is
a
good
example
from
the
Prometheus
blog
awhile
ago.
So
you've
got
a
handful
of
instances
of
like
a
web
application
that
are
serving
you
know,
kind
of
similar
requests.
You
can
examine
the
group
of
latency
metrics
to
find
outlier
instances
that
might
be
running
kind
of
slowly.
You
can
do
this
by
checking
out
request
durations
average
that
are
sticking
out
by
two
standard
deviations,
and
then,
additionally,
you
could
mitigate
any
false
positives
if
your
data
is
really
closely
clustered
together
by
checking
that
the
value
must
also
be
20%
higher
than
average.
A
So
this
is
an
absolute
unit
of
a
query,
but
you
can
break
it
down
into
its
component
pieces
fairly
easily
to
kind
of
understand
it.
You
know,
spend
an
hour
understand
the
tool
you
get
it
another
one
I
like
to
do
is
trying
to
find
spikes
and
odd
events.
So
if
you
got
something
that
is
normally
very
very
consistent
and
you
can
examine
some
sort
of
recent
amount
of
time
like
5
minutes
or
10
minutes
compared
to
the
last
24
hours
or
so
you
can
see
if
you've
got
some
sort
of
change.
A
This
is
a
great
kind
of
first
alert.
If
you
were,
you
know,
looking
you're
kind
of
starting
on
your
journey
toward
you
know
doing
this
kind
of
thing,
and
you
just
want
to
know
when
something
happened.
This
is
a
good
thing
to
kind
of
know.
What's
going
on
there,
the
only
other
thing
that
is
better
than
catching
a
problem
right
away,
though,
is
catching
it
before
it
even
happens.
The
predict
linear
function
in
Prometheus
uses,
or
it
predicts
a
future
value
based
on
a
backward-looking
linear
regression.
A
So
the
canonical
example
is
alerting
you're
gonna
run
out
of
disk
space
before
you
do
imagine
how
useful
that
would
be
on
like
a
database,
for
example.
This
is
great
that
whenever
you
have
some
sort
of
capacity
or
saturation
metric,
you
know
it's
a
little
bit
similar
to
the
last
example,
because
usually
something
has
started
happening
that
has
changed
your
future
in
a
way
that
is,
that
is
undesirable,
but
it's
good
for
longer-term
stuff,
like
a
disk,
filling
up
release
or
like
a
TLS
certificate
expiring
as
well.
A
You
can
write
an
application
that
exposes
metrics
for
just
about
anything,
so
in
fact,
most
of
the
ones
that
you
would
want
to
write
probably
already
exist
out
there,
a
digitalocean
we've
written
exporters
to
quantify
capacity
and
our
fleet
in
various
ways,
look
at
our
events
pipeline
and
look
at
individual
customer
or
customer
resources,
and
these
are
all
things
that
are
incredibly
custom
right.
Only
only
they
would
be
useful
to
us
a
lot
of
times.
A
There's
one
real
gotcha
that
I've
seen
by
people
a
lot
when
writing
exporters
for
the
first
time
that
I
want
to
try
and
warn
you
about
this.
Isn't
the
go
library-
and
it
happens
to
you
when
you
are
kind
of
writing,
you're
forced
exporter
and
like
reading
a
documentation
and
doing
it
kind
of
the
naive
way?
Basically,
what
happens
is
you
will
create
a
new
time
series
with
some
combination
of
labels?
And
you
will
you
you
will
choose
a
label
value
that
that
represents
something
ephemeral,
so
say:
you're
exporting
metrics
about
a
queue
name.
A
That
is
only
there
for
five
minutes
or
so
what
will
happen
using
this
kind
of
naive
way?
That
is
very,
very
easy.
To
use
them
in
the
client
is
that
metric
will
then
be
present
in
the
metrics
endpoint
that
your
application
exposes
as
long
as
the
application
is
up
so
until
it
gets
restarted
so
then
you'll
be
clearing
your
stuff
in
Prometheus
later
or
you
know,
get
an
alert
for
something.
You
know
some
cue
name
or
whatever
you're
like
wow.
Oh,
that
shouldn't
be
like
that.
A
You
know
that
shouldn't
be
there
and
then
we
go
one
look
and
it
won't
be
there
like.
Why
are
my
metrics
wrong?
The
workaround
is
to
use
a
pattern
called
const
metrics
in
the
go
library.
This
pattern
essentially
gathers
the
metrics
at
scrape
time
so,
instead
of
kind
of
just
you
know,
bopping
a
little
function
or
something
inside
your
your
code.
A
You
can
kind
of
either
just
leave
it
to
entirely
a
thing
that
happens
in
scrape
time
if
it's
really
fast
or
you
can
keep
the
kind
of
information
that
you
want
to
be
exporting
metrics
about
in
some
sort
of
other
abstraction
in
memory
in
your
code
and
then
build
them
at
the
time
that
the
scrape
happens.
You
should
always
be
doing
this
when
the
source
of
truth
for
the
metrics
that
you're
exporting
is
not
your
code
itself.
A
So,
if
you're
doing
something
like
you
know,
doing
a
database
query
and
then
kind
of
generating
metrics
from
that
or
poking
around
in
some
sort
of
stats
socket
or
writing
an
exporter
for
some
sort
of
large
system.
That
you're
kind
of
you
know
just
exporting
some
kind
of
health
stuff
about.
This
is
what
you
should
do.
I,
don't
expect
you
to
remember
that
what
I
would
hope
is
that
you
kind
of
remember
this
moment
and
then
maybe,
when
you're
doing
this
someday,
you
can
like
tweet
at
me
or
go
read.
A
You
see
up
on
the
right
and
that
that
tweet
is
actually
my
alert
manager
routing
tree,
and
you
wouldn't
even
know
it's
a
tree
because
there
are
so
many
lines
on
it.
Those
are
all
the
places
that
the
alerts
can
possibly
go.
There's
a
lot
of
good
literature
out
there
about
what
you
should
alert
on
I'd.
Actually,
I,
don't
know
if
I
said
this,
when
you
are
creating
kind
of
that
alert
manager,
config
make
an
abstraction
and
generate
that
config.
So,
like
I,
don't
I
don't
really
have
to
care.
A
A
There's
some
good
literature
out
there
about
what
should
be
alerted
on
I'm,
not
gonna
like
dive
deep
here,
because
that
could
be
a
whole
talk,
but
alerts
should
prompt
action
and
then
have
all
the
context
attached
to
the
alert
necessary
to
perform
that
action.
So
write
a
playbook
attached
to
your
alert.
Also,
if
you're
doing
slack
alerts
use
the
tool
that
Ullaeus
wrote
to
create
beautiful
slack,
you
just
don't
get
like
viscerally
angry.
Whenever
you
see
them
the
pinnacle
of
mindful
observability
is
implementing
s
ellos.
A
So
you
should
read
the
chapter
in
the
SRA
book
on
it's
really
like
the
entire
SRU
book,
but
essentially
the
idea
is
that
you
should
be
defining
the
type
of
performance
experience
that
your
users
can
expect,
measure
it
and
then
take
action
to
maintain
that
level
of
service.
This
is
the
most
important
thing
to
measure
right.
So
talking
about
being
outcomes
oriented
this
is
the
outcome
for
your
customers.
This
is
the
thing
that
you
should
be
measuring,
but
quantifying.
A
It
is
actually
perfectly
really
hard,
but
it's
an
essential
exercise
and
the
first
few
times
you
do
it.
It
might
actually
require
some
work.
You
might
even
need
to
change
the
architecture
of
your
application
to
be
able
to
measure
that
thing
in
the
right
place,
but
once
you've
measured
it
you
can
then
set
an
objective.
So
this
is
often
like
some
number
of
nines
and
then
you
can
give
yourself
an
error
budget.
So
that
would
be
like
the
amount
of
time
that
you're
allowed
to
file
your
SLO.
You
can
use
that
then.
A
As
a
simple,
you
know
this
one
simple
thing
as
a
as
a
factor
in
any
decision-making
that
you
might
have
to
do.
So
if
you
have
a
service,
that's
consistently
violating
its
SLO
and
as
an
organization
you
decided,
the
DES
allows
are
important.
You
can
then
make
the
leap
that
you
should
not
be
doing
any
work.
That
is
not
innovative
fixing
that
problem.
This
is
a
great
way
to
push
back
on
decision
makers
that
might
be
trying
to
push
new
work
on
you
or
new
features,
or
something
because
it's
really
hard
to
arguing.
A
It's
not
achieving
your
SLO
Prometheus
can
help
you
set
this
kind
of
system
up
and
I
want
to
show
you
briefly
how
to
do
that.
So
this
is
the
99th
percentile
query
latency
in
micro
me,
Thea
stands
for
infrastructure
over
24
hours,
measured
hourly,
say:
I
wanted
to
define
availability
to
be
that
99%
of
my
requests,
complete
in
under
a
second
I,
can
express
that
in
a
Prometheus
query
as
a
binary,
1
or
0
did
this
happen,
or
did
this
not
happen
for
a
time
period
using
this
less
than
boolean
operator
in
Prometheus?
A
So
you
can
see
on
the
previous
graph.
There
was
one
kind
of
hour
where
my
latency,
my
99
percentile
latency,
was
over
a
second.
So
now
in
this
graph
it's
a
zero,
so
I
failed
for
that
hour.
Then
you
can
use
a
Prometheus
sub
query
which
are
awesome
to
find
that
exact
same
data
on
that
graph,
which
was
generated
by
you,
know
evaluating
one
query
at
24
points
in
time
and
drawing
all
the
lines
together.
You
can
get
all
that
data
in
one
query
so
that
you
can
then
use
it
in
another.
A
Later
you
see
all
those
ones
and
zeroes
there.
We
can
then
take
the
average
over
time
of
that
data
set
and
because
it's
expressed
in
those
ones
or
zeros,
we
can
calculate
a
percentage
of
time
that
we
met
at
RS,
although
so,
if
you're
doing
a
time
based
SLO,
this
is
a
very
easy
way
to
do
that.
So
you
can
see
that
we
actually
had
an
availability
of
95.8%
over
those
24
hours.
So
if
we
wanted
99%
availability
than
we'd,
be
in
violation
for
simplicity,
here,
I
evaluated
the
queries
every
hour
in
real
life.
A
Second
and
I
am
actually
counting
the
the
requests
that
were
less
than
or
equal
to
a
second
and
then
my
total
number
of
requests,
so
I
can
get
a
more
exact
number
of
requests
that
actually
took
less
than
a
second
because
I'm
using
that
histogram
bucket.
Another
problem
that
you
might
run
into
here
is
that
if
you
don't
have
any
requests
for
five
minutes,
you
know
you
would
be
dividing
by
zero
here,
which
is
not
allowed
in
Prometheus,
and
it
would
break
your
entire
query.
A
So
using
this
little
trick
with,
or
you
can
sanitize
that
time
where
you
had
no
requests
as
a
one
in
your
kind
of
binary
zero
to
one
by
using
any
metric
that
is
always
extant
and
then
we
can
use
the
same
pattern
as
we
did
in
the
previous
example.
So
we
can
calculate
the
percentage
of
time
over
the
last
week
where
latency
was
less
than
one
second
and
feed
that
into
an
SLO
all
right,
everybody
we
did
it
remember
doing.
A
Observability
well
requires
time
and
effort,
but
your
reliability
is
maybe
the
only
thing
that
is
important
enough
to
justify
that
kind
of
investment.
You
need
to
do
it
anyway,
if
you're
gonna
want
the
best,
if
you
want
the
best
outcomes
because
low
effort
inputs
are
not
enough
for
your
customers,
you
should
pick
some
tools
and
get
really
good
at
using
them.
I
recommend
from
atheists,
because
it's
just
excellent
software
and
that's
it.
If
you
have
questions
you
can
come
up
and
ask
me
afterwards
over
here
or
outside
or
something
but
I
want
to.