►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro + Deep Dive: Prometheus - Julius Volz, Prometheus & Richard Hartmann, SpaceNet
Prometheus is an open-source monitoring system and time series database. It features a multi-dimensional data model with a powerful query language and integrates many aspects of systems and service monitoring: from the instrumentation of services over the collection and storage of metrics data, all the way to dashboarding and alerting. Native support for various service discovery mechanisms also make it particularly suitable for dynamic cloud-based environments. In the introduction, Prometheus co-founder Julius Volz explains the architecture of Prometheus and shows its advantages over traditional monitoring systems. For the deep dive: Join Prometheus maintainers to learn more about the design, project goals and history of Prometheus. We will cover in depth concepts of Prometheus to enable you to get a better understanding of what it is all about. The talk will cover topics including federation, relabelling and PromQL joins. In the end, you will leave with an advanced understanding of Prometheus.
https://sched.co/MPiP
A
Good
morning,
everyone
who
just
went
actually
to
the
keynotes
who
made
it
there
Wow
okay,
half
about
cool
all
right.
So
this
is
a
90
minute
session.
It's
combined
intro
and
deep
dive
session
for
the
Prometheus
monitoring
system.
So
the
first
half
I'm
just
going
to
give
a
complete
introductory
beginners
talk
to
what
is
Prometheus.
How
does
it
work?
Why
is
it
awesome
and
the
second
half
of
the
talk
is
really
just
completely
free
form.
A
Prometheus
maintainer
z'
being
asked
any
questions
by
you,
which
will
be
led
by
Richard
after
me,
so
in
about
like
30
or
40
minutes.
Alright,
so
I
am
Julius
I'm,
the
one
of
the
cofounders
of
the
Prometheus
monitoring
systems
and
nowadays
I
just
freelance
around
it.
I
help
companies
use
it,
do
training
software
development
around
it
and
so
on.
So
let's
start
with
you
know
what
prometheus
actually
is.
A
Prometheus
was
especially
made
for
dynamic
environments
like
kubernetes,
even
though
kubernetes
that
didn't
even
exist.
Yet
when
we
created
Prometheus,
we
explicitly
do
not
do
some
other
things,
so
we
try
to
keep
the
scope
of
Prometheus
fairly
limited
and
simple.
We
don't
do
logging
or
tracing.
We
only
do
numeric
time
series
data
where
you
have
individual
values
that
have
some
kind
of
development
over
time
for
logging,
which
you
still
want
to
have
as
well.
You
might
use
a
system
like
Loki
or
elasticsearch
for
tracing,
maybe
you'll
use,
Jaeger
or
Zipkin.
A
We
give
you
a
way
to
define,
alerting
rules
that
can
be
maybe
very
complex,
but
they're
always
explicit,
and
you
have
to
define
what
means
what
a
bad
condition
actually
means.
So
the
system
doesn't
do
any
kind
of
automatic
anomaly
detection,
just
squinting
at
your
data
and
kind
of
guessing
that
something
might
be
wrong.
Prometheus
itself
also
only
has
a
local
on
disk
storage,
which
is
naturally
somewhat
limited
in
its
scalability
and
durability,
because
it's
a
single
node
system
and
there's
no
built-in
replication.
So
we'll
talk
about
long-term
storage
and
scalability
tiny
bit
more
later.
A
There's
ways
to
do
that,
but
it's
not
part
of
Prometheus
itself,
Prometheus
started
in
2012
at
sound
clouds.
That
was
when
both
I
and
Matt
kind
of
came
from
Google
and
tried
to
make
sound
clock
more
stable
and
back
already
had
a
cluster
scheduler
before
darker,
even
existent,
so
obviously
no
kubernetes.
Yet
no
Griffin
I
yet
know
none
of
these
fancy
tools
and
the
OP
the
open
source
monitoring
tools
back
then
weren't
really
suitable
for
making
sense
of
dynamic
cluster
scheduling
with
containers
floating
around
and
all
that.
So
this
led
to
the
birth
of
Prometheus.
A
It
was
open
source
from
day
0,
but
we
only
really
published
it
in
2015
and
then
in
2016.
It
became
the
second
member
project
of
the
cloud
native
computing
foundation,
which
is
hosting
this
conference
here
today
after
kubernetes,
and
you
can
find
us
at
Prometheus
I/o
if
you
go
to
slash
community
you'll,
also
find
ways
to
get
help
and
become
a
contributor
which
would
be
extra
awesome.
A
So
let's
have
a
look
at
the
basic
system,
architecture
of
what
you
know
how
Prometheus
works
and
what
it
allows
you
to
do.
First
of
all,
we
want
you
to
be
able
to
get
metrics
out
of
the
things
you
care
about,
and
the
things
you
care
about
can
be
different
things.
For
example,
they
might
be
applications
services
running
in
your
cluster,
where
you
control
the
code,
your
own
web,
api,
server,
etc,
etc.
A
In
those
examples,
the
best
thing
to
do
is
really
just
take
a
Prometheus,
client
library
included
into
your
process
and
use
it
for
two
things
to
track.
Metrics
counters
histograms
gauges
summaries,
basically
just
keeping
keeping
track
of
internal
metric
state
and
then
the
second
part
is
exposing
that
over
an
HTTP
endpoint,
because
Prometheus
is
a
pull
based
monitoring
system
and
expects
to
be
able
to
go
to
your
instance
and
ask
it
about
the
current
value
of
all
its
time
series
that
it
is
tracking.
A
Now
you
might
have
some
stuff
where
you
cannot
directly
add
stuff
at
Prometheus,
metrics
to
the
code
itself,
like
a
Linux
virtual
machine
or
my
sequel,
daemon
or
the
cgroups
container
information
on
a
machine
and
there's
many
many
of
these
existing
systems
where
you
will
not
be
able
to
add
directly
a
previous
endpoint.
So
for
this
we
have
the
concept
of
an
exporter
which
you
run
as
a
sidecar.
A
Next
to
the
thing
you
want
to
measure
and
then
Prometheus
just
talks
to
the
exporter
and
the
exporter
gets
the
metrics
in
the
backend
in
whatever
third
party
format.
That
is
and
translates
them
back
to
promises
server
to
the
promises
format.
So,
finally,
we
add
a
Prometheus
server.
The
Prometheus
server
is
the
heart
of
the
parameters
ecosystem.
Obviously,
you
configure
it
to
pull
on
a
periodic
basis.
It's
like
every
15
seconds.
A
Every
30
seconds
are,
for
example,
common
values
to
pull
metrics
from
all
the
targets
that
are
configured
in
it,
and
then
it
stores
those
over
time
as
time
series.
So
just
a
quick
interview,
what
do
we
actually
transfer
on
these
arrows
here?
What
does
that
format?
Look
like.
It
just
looks
like
this
without
the
colors,
of
course,
but
the
colors
make
it
more
readable.
It's
really
just
a
text-based
format
that
we
defined
in
Prometheus,
where
every
time
series
has
a
single
sample.
A
It's
just
the
current
value
of
each
time,
series
that
is
being
sent
at
that
moment
in
time,
and
so
this
is
really
really
dead,
simple
to
generate
from
even
a
bash
script
if
you
need
to.
But
if
you
are
in
a
more
higher
level
programming
language,
you
might
use
one
of
our
client
libraries
to
do
proper,
tracking
and
exposition,
so
the
hurdle
to
get
metrics
out
of
things
you
care
about
is
actually
pretty
low.
As
long
as
you
can
pull
we'll
talk
about
the
data
model
a
bit
more
later,
okay
cool.
A
So
now
we
have
the
parameter
server:
it
stores
it
in
a
very
highly
efficient
times
use
database,
and
then
you
might
be
wondering
well,
how
does
Prometheus
actually
know
where
all
these
targets
are?
We
don't
require
you
to
statically
configure
each
one
of
your
kubernetes
pods,
for
example.
Obviously,
Prometheus
integrates
with
different
service
discovery
mechanisms
to
allow
you
to
do
that
and
automatically
discover
what
is
out
there.
A
Finally
to
do
useful
stuff
with
the
data
we,
for
example,
grow
fauna.
You
know
the
most
popular
dashboard
builder
has
native
support
for
Prometheus.
As
a
graphing
back-end.
You
can
use
primitives
web
UI
to
dig
around
in
the
data
yourself
or
just
build
any
kind
of
automation
against
Prometheus's,
HTTP
API
yourself
as
well,
then
Prometheus
itself
in
the
server.
A
You
can
build
your
own
as
well,
and
so
this
is
just
the
basic
architecture,
and
then
you
have
like
many
of
these
different
exporters
and
integrations
and
stuff
that
is
built
around
it.
Basically,
typically,
you
have
many
Prometheus
servers
in
your
organization,
because
these
are
you
know.
The
individual
server
is
not
a
horizontally
scalable
thing,
so
you
know
you
might
chart
them
by
function
or
you
can
build
trees
that
form
yeah
like
a
hierarchical
topology
to
achieve
different
scaling
goals.
A
So
the
for
these
four
points,
our
4
major
reasons
why
I
think
primitives
was
successful,
especially
back
then
nowadays,
maybe
they're,
not
even
so
special
anymore,
but
I
think
in
2015,
when
we
released
it,
they
were
a
bit
more.
So
you
know
a
data
model,
that's
dimensional,
then
a
query
language
that
goes
along
with
it
to
do
powerful
things,
the
simplicity
and
efficiency
of
a
single
parameter,
server
and
then
integrating
with
service
discovery
to
make
parameters
work
in
the
net
in
dynamic
environments.
A
First,
the
data
model,
so
I'll
go
through
these
four
points
in
detail:
Prometheus
fundamentally
tracks
time
series,
so
these
are
values
over
time
where
the
identifier
stays
the
same.
For
example,
the
identifier
on
an
abstract
level
might
be
something
like
the
temperature
in
this
room
versus
the
temperature
in
the
other
room
right,
and
we
just
want
to
kind
of
track
the
same
thing,
but
over
time
how
it
develops,
and
for
tracking
that
over
time
we
add
timestamp
value
pairs,
which
we
call
samples
that
belong
to
the
time
series
timestamps
in
Prometheus
are
always
in
64.
A
Keep
it
simple.
There
are
basically
millisecond
UNIX
timestamps
the
sample
values.
Also
only
can
have
a
single
type,
which
is
always
float
64,
which
works
remarkably
well
for
systems
monitoring,
use
cases,
and
then
the
big
question
is
here:
what's
the
identifier
like?
How
do
we
identify
time
series,
because
this
will
inform
how
we
search
and
query
for
metrics,
we
decided
to
go
for
a
dimensional
key
value
based
model
where
we
start
with
a
metric
name
at
the
beginning.
This
is
just
the
name
of
the
aspect
of
a
system.
We
are
monitoring
in
this
case.
A
This
is
a
counter
metric.
How
many
HTTP
requests
has
a
server
handled
since
process
start
up?
It
just
goes
up
and
up
and
up
and
now
we
want
to
sub
differentiate
certain
dimensions
in
there
like
was
it?
What
was
the
path
of
the
requests,
the
status
code,
etc,
etc?
So
these
we
just
attach
as
what
we
call
labels
key
value
pairs
and
this
data
model
is
pretty
flexible.
A
It
avoids
having
a
higher
key,
as
we
had
in
earlier
monitoring
systems
like
graphite
or
stats
D,
where
the
metric
name
is
somewhat
akin
to
a
path
name
with
higher
key
components.
And
then
often
you
know,
if
you
add
a
new
dimension,
you're,
not
sure
where
in
the
hierarchy
would
wanna
stick
it
which
component
has
which
meaning.
So
this
is
a
bit
more
flexible
and
explicit
and
useful
in
queries.
A
So
now
to
make
use
of
this.
Prometheus
comes
with
its
own
new
query
language.
That's
called
prong
QL,
the
Prometheus
query
language.
It's
explicitly
not
a
sequel
style
language,
because
sequel,
star
languages
tend
to
be
a
bit
obtuse
when
trying
to
do
time.
Series
operations,
so
prompt
QL
is
especially
optimized
to
having
a
16th
way
of
expressing
the
typical
computations.
We
want
to
do
with
time
series
in
the
systems
monitoring
context.
A
A
And
now
you
want
to
know
well,
you
know
give
me
all
the
partitions
that
are
larger
than
100
gigabytes
in
my
infrastructure,
but
that
that
are
not
mounted
on
the
root
amount
point
you
might
just
start
with
a
metric
name.
Then
filter
down
with
a
negative
matcher,
basically
saying
give
me
only
the
x
views
that
belong
not
to
the
route.
A
Another
common
thing
you
might
want
to
do
is
calculate
request.
Error
rate
ratios
so
like
how
many
errors
are
you
getting
in
comparison
to
the
total
number
of
requests
for
this?
You
could
take
the
purse
and
rate
of
the
500
status
code,
requests
and
divided.
Here's
here's
a
division
operator.
So
this
is
all
one
expression
divided
by
the
total
number
of
requests,
so
the
request
rate
of
that-
and
this
would
give
you
ratio
a
single
number
bit
boring.
A
Sometimes
you
might
want
to
preserve
some
dimensions
like
the
path
give
me
this
ratio
for
every
path
in
my
web
application.
For
that
you
just
add
a
modifier
to
the
some
aggregator,
and
then
it
preserves
some
dimensions
and
now
is
where
magic
actually
starts
happening,
because
you're
no
longer
dividing
one
number
by
another
number
you're
dividing
a
whole
set
of
numbers
by
a
whole
other
set
of
numbers
on
the
left
on
the
right
hand,
side
and
prom
KL
automatically
joins
these
sets
of
time.
A
Series
based
on
identical
label
sets,
so
it
will
divide
the
error
rate
for
one
path
by
the
total
rate
for
that
same
path
and
propagate
that
as
an
equally
labeled
element
into
the
output.
So
this
whole
label
based
automatic
joining
is
really
powerful
and
there's
no
more
complex
ways
of
doing
this.
If
labels
don't
exactly
match
up
on
both
sides.
A
Just
another
example
won't
go
really
into
this,
but
we
allow
you
to,
for
example,
track
latency
distributions.
In
a
histogram
metric
and
then
have
a
preview
language
function
which
allows
you
to
say,
hey
I,
want
to
calculate
the
99th
quant,
the
99th
percentile
from
the
cystogram
over
all
of
my
instances,
so
I
want
to
aggregate
away
the
instance
time
engine
and
I
wanted
average
that
over
five
minutes.
This
is
what
this
expression
does
and
then
you
get
you
know
for
all
your
paths
and
all
your
methods,
you
get
the
99th
percentile
latency
as
an
estimate.
A
Now,
if
you
learned
that
language,
which
admittedly
can
warp
your
brain
a
bit
at
the
beginning
but
then
becomes
really
useful,
you
can
start
using
it
either
in
the
built-in
expression
browser
or
even
graph
those
expressions
over
time.
But
this
is
not
really
good
for
building
persistent
dashboards
with
all
the
dashboarding
features
that
you
used
to.
So
for
that
just
use
go
use,
go
Phenom,
it
has
native
Prometheus.
A
Now
really
cool
thing
is
alerting
and
how
it
integrates
with
this
language
in
a
unified
way.
So
alerting
is
no
longer
a
separate
system
that
just
runs
check
scripts
somewhere
on
hosts.
First,
like
the
the
whole
idea
of
primitives,
is
you
collect
anything?
You
can
imagine
as
a
metric
first
if
it's
just
like
a
boolean
value,
0,
1
or
2,
or
it
can
enum,
because
that
is
stored,
read
efficiently,
and
then
you
define
alerting
rules
based
on
the
collected
data.
A
So
in
this
example,
I
took
an
expression
that
we
had
earlier,
which
gives
us
the
error
rate
ratio
for
every
path.
We
multiply
it
by
a
hundred
to
get
to
a
percentage
and
we
filter
it
by
five
to
get
all
the
paths
that
have
an
error
rate
ratio
higher
than
5%,
and
so
this
would
really
give
you
an
alert
for
every
path
that
is
currently
misbehaving
and
yeah.
This
can
get
way
more
complex
as
well.
A
Third
Point
operational
simplicity,
you
know,
Prometheus
as
a
single
binary
service
and
single
node
service
is
pretty
easy
to
get
started
with.
It
only
needs
a
small
configuration
file.
It
puts
its
local
data
on
disk
and
off
you
go.
You
can
get
high
availability
for
the
purpose
of
alerting
by
just
running
two
identical,
primitive
servers
which
pull
the
same
data.
So
they
know
nothing
about
each
other
they're,
not
a
cluster.
A
They
just
do
the
same
thing:
they
ingest
the
same
data
generate
the
same
alerts,
alert
manager,
we'll
d-do
Platt
for
you
and
it's
written
in
go
a
single
server.
If
it's
really
beefy
one
can
also
do
quite
a
lot.
Obviously
it's
not
infinitely
horizontally
scalable,
but
it
can
do
in
the
lower
number
of
millions
of
series.
If
you
really
really
overdo
it,
you
might
be
able
to
get
like
a
million
plus
samples
on
ingestion
per
second
and
the
storage
on
disk
of
each
time.
A
Series
ample
is
also
really
efficient
about
like
1
to
2
bytes
per
sample,
so
the
local
storage
is
really
only
meant
as
an
ephemeral
window
of
data
that
you
keep
maybe
4
weeks,
maybe
4
months.
Some
people,
like
this
guy
who's,
currently
looking
in
his
phone
Richie,
he's
like
keeping
years
in
there
and
we
tell
them
well,
that's
not
really
what
it's
designed
for.
A
That
converts
or
another
thing
I
really
like,
is
the
nose
who
has
heard
of
Thanos.
Okay,
you
should
all
hear
of
Thanos.
This
was
only
like
one-sixth
of
people
or
so
Thanos
was
also
built
by
people
who
are
like
both
core
Prometheus
people
and
had
another
need.
Basically,
you
know
getting
a
unified
view.
That's
long
term
storage,
unified
view
with
durability
over
multiple
parameters.
Servers
in
your
organization.
This
works
a
bit
differently
than
the
remote
write
pattern
that
I
showed
before.
Basically,
you
add
Thanos
sidecar
to
all
of
your
existing
chrome,
easy
service.
A
This
could
be
s3
GCS
or
something
you
host
you're
on
your
own,
some
object,
storage,
and
then
you
can
just
point
Ravana
at
the
Stannis
query
component,
which
integrates
data
from
long-term
storage
with
the
short
term
data
from
your
life
through
meters
instances
and
then
has
some
magic
for
actually
also,
if
you
have
an
H
a
pair
of
primitives,
to
only
show
you
the
data
from
one
in
a
consistent
way
and
so
on.
So
it's
really
cool
last
point
of
the
selling
points
of
Prometheus
is
how
it
works
with
dynamic
environments.
A
So
the
old
world
was
static
for
a
long
time.
Now
it
hasn't
been
anymore.
First
came
dynamic
VMs
then
we
put
cluster
schedulers
like
who
it's
on
top
and
we
have
microservices
on
top
of
that,
so
things
are
moving
really
quickly
and
automatically,
and
so
on
so
like
how
old
does
a
monitoring
system
still
make
sense
of
what
is
supposed
to
be
where
our
answer
obviously
is
service
discovery
in
three
different
ways
here.
A
So
Prometheus
uses
service
discovery
for
three
different
purposes
that
are
related
but
different.
The
first
is
really
your
monitoring
system
should
know
what
is
supposed
to
be
where,
with
some
push
based
monitoring
systems,
they
kind
of
receive
data
from
whatever
sending
data,
but
they
don't
really
know
if
someone
should
report
in
but
isn't
so
that's
the
first
purpose,
then
the
second
one
is
the
service
discovery.
Information
usually
gives
you
technical
information
for
how
to
actually
reach
a
target
and
pull
metrics
from
it.
A
A
This:
is
an
engine
X
part
of
a
certain
other
label
quality
and
so
on,
and
then
parameters
can
map
that
into
your
time,
series
data
Prometheus
has
built
in
discovery,
support
for
a
bunch
of
cloud
providers
and
clusters,
cluster
managers-
and
it
also
has
generic
mechanisms
like
DNS,
consul
and
so
on.
You
can
also
build
your
own
plugin
based
mechanism
based
on
a
file
watcher
if
the
built-in
ones
are
not
sufficient.
A
So
as
a
conclusion,
these
for
selling
points
really
help
make
Prometheus
a
suitable
fit
for
the
cloud
native
world
and
making
sense
of
a
lot
of
the
data
that
happens
there.
Cool
so
I
briefly,
say
thanks,
but
then
I
immediately
skip
to
the
Prometheus
deep
dive
and
open
QA.
So
spare
your
questions
for
that
Ritchie
is
going
to
lead
that
and
I'll
hand
over
to
Ritchie.
B
He
didn't
know
he
was
volunteering,
but
he
does
know
okay.
So
once
again,
this
is
open
Q&A
for
technical
or
whatever
reasons
we
are
required
to
call
this
deep
dive,
but
it's
basically
an
open
Q&A.
There
are
no
stupid
questions,
so
please
any
level
of
question
you
feel
comfortable,
asking
just
ask
them:
it's
totally
fine.
We
will
then
assign
questions
to
whoever
is
standing
here.
B
B
Okay,
okay,
so
everyone
else.
This
is
how
we
have
to
hold
this
thing
course.
Unfortunately,
since
EF
can
only
afford
to
mix
per
room,
even
though
I
complain
every
single
year
anyway
yeah.
So
we
no
way
not
the
other
question.
So
any
any
level
of
questions
is
fine,
please,
just
whatever
you
want
to
ask
ask
it.
We
only
ask
one
thing:
don't
have
like
a
lightning
talk,
so
it
should
be
a
real
question.
It
shouldn't
be
a
statement
of
course
that
that
basically
takes
away
from
question
time
for
everyone
else.
B
C
You
get
data
from
a
node,
but
this
node
was
out
of
service
for
something
like
I,
don't
know,
20
minutes,
so
you
don't
get
any
matrix
from
the
node
for
20
minutes.
So
how
can
we
kind
of
deal
with
this?
All's
when
you
make
some
things
like
some
other
time
or
this
kind
of
operation
to
get
a
consistent
result,
I
hope
my
yeah
so.
B
C
B
Okay,
so
if
the
question
is
how
to
deal
with
outages
and
not
being
able
to
reach
nodes
which
are
up
but
not
seen
by
Prometheus,
you
have
to
put
a
Prometheus
in
a
place
where
it
can
be
seen
and
very
where
you
can
see
the
nodes.
So
if
you
have
this
kind
of
issue,
it
makes
sense
to
have
a
paresis
nearer
to
whatever
your
workload
is,
maybe
have
a
local
instance
or
whatever
and
then
use
remote
read,
write
or
federación
to
get
data
to
wherever
you
need
it,
that's
more
or
less
it.
B
A
Yeah
and
generally
Prometheus
takes
the
stance
to
gaps
in
data.
To
just
say
like
oh
well,
you
know
you
have
a
gap.
It's
fine,
because
primitives
is
not
the
system
that
tries
to
keep
a
hundred
percent
of
your
data
forever
consistently.
It
tries
to
tell
you
like
what
is
wrong
right
now
and
if
I
cannot
reach
your
note
at
that
moment,
like
that's
the
main
problem
and
we
don't
care
anymore,
like
what
are
the
actual
metrics.
Of
that
note,
you
need
to
figure
figure
that
out
after
getting
alert
for
that.
D
F
E
E
So
if
you
want
to
be
multi-tenancy
that
way,
the
kubernetes
prometheus
operator
makes
it
pretty
easy
to
do
that
as
far
as
I
understand
and
then
yeah,
you
know,
cortex
does
it
just
a
nose
have
multi-tenancy
they're
working
on
it
only
I
think
last
time,
I
spoke
to
bar
tech
about
it
panels,
there's
ways
of
doing
multi-tenancy
and
panels
as
well,
so
there's
a
whole
bunch
of
different
ways.
Yeah.
Does
that
answer
your
question
yeah?
A
G
A
H
G
E
Sure
so,
there's
a
remote
write
protocol,
which
is
using
proto
buffers
and
snappy
encoding
in
a
HTTP
body.
You
can
set
an
end
point
in
the
Prometheus
config
that
will
tell
Prometheus
to
send
the
data
it's
scraping
to
somewhere
right
and
then
I
mean
there
are
tens
of
people
that
have
implemented
this
protocol
on
the
server
side
to
accept
data,
so
there's
tens
may
be
exact,
exaggerating,
like
Julius
implemented
an
adaptive
influx
that
then
got
merged
into
influx.
As
far
as
I
understand,
there's
one
for
open,
TS,
DB
cortex
accepts
it.
E
E
One
other
thing
to
add
is
recently:
we
rewrote
the
remote
write,
API,
spi
I
guess
so
now
it
actually
tails
the
Prometheus
right
ahead
log.
So
it
deals
really
well
now
with
network
outages.
If
there's
a
network
outage,
Prometheus
will
buffer
up
on
disk
the
samples
and
when
the
network
comes
back,
it'll,
try
and
resend
them.
Okay,.
I
I
The
first
example
there
is
some
mathematic
function
like
percentile
that
we
already
seen,
but
what
about
having
some
more
complex
mathematic
functions,
so
this
is,
for
the
first
part.
The
second
one
is,
for
example,
I
have
machine
one
machine,
a
that
the
trigger
one
alert
but
I
have
also
HTTP
be
trigger
error
that
I
want
to
trigger.
Only
the
C
alert
is
the
sum
of
these
two
others
together,
so
aggregate
other
thing
inputs.
Thank
you.
Okay,.
J
E
J
E
G
K
So
I've
got
a
question
around
I
guess
you
see
Prometheus
being
used
in
things
like
linkers,
II
and
other
different
projects
and
you'll
get
to
a
stage
potentially,
where
you've
got
a
few
different
instances
of
Prometheus
on
your
clusters,
maybe
across
different
clusters.
Are
there
any
kind
of
recommended
practices
about
how
you
and
get
those
communicating
and
kind
of
architects?
From
that
perspective
that
you
can
share.
J
J
The
next
question
is:
how
do
I
do
graphing
and
well
there's
dashboard
templating
so
on
a
graph
Anna
and
so
a
single
graph
Anna,
because
you
only
need
one
and
just
have
a
hey,
Jay,
my
sequel,
somehow,
wherever
you
usually
do,
then
you
can
have
a
global
media
spooling
aggregated
data
by
Federation,
because
if
you
want
to
get
something
like
global
requests,
you
cannot
calculate
that
in
a
per
cluster.
Fruitiest
need
to
get
all
the
data
somewhere,
so
normally
you'd
get
all
the
aggregated
data
into
a
cluster.
J
For
me
to
see
if
it's
got
like
multiple
kubernetes
and
then
I
create
there
and
then
up
to
a
global
where
you
do
more
maths
and
do
it
there.
This
is
not
everyone's
taste.
Once
you
add
hoc
queries,
like
that's
fine,
when
you
know
all
your
queries
in
advance,
because
you
make
that
all
nice
and
efficient
with
Federation,
in
which
case
you
get
to
do
other
options
so.
E
Yeah
so
Federation,
the
other
option
is
to
use
remote
write
to
push
to
some
central
database,
of
which
you
know,
we've
just
listed
a
bunch
of
them.
I
represent
cortex,
so
cortex
is
a
horizontally
scalable
implementation
of
the
Prometheus
API.
So
that
way
you
can
use
the
same
prom
QL
expressions
you
would
on
your
Prometheus
nodes
in
the
global
way.
You
know
against
a
single
horizontally,
scalable
cortex
cluster,
but
you
can
do
the
same
with
influx
DB.
L
Yeah
I'm
Donna
smitten
err,
so
the
first
way
is
to,
if
you
don't
want
to
use,
pushing
matrix
or
federating
so
replicating
the
the
data
in
a
multiple
places,
you
can
essentially
just
use
a
tano's,
which
is
a
solution
that
allows
you
to
have
a
long
term
search,
but
also
it
was
global
querying.
So
you
put
your
you
start
some
component
on
the
global
level
that
does
prom
query
evaluation
and
grabs
data,
queries
them
directly
from
the
parameters,
instances
or
object
storage,
which
is
yet
another
option
you
can
choose.
Oh
thank
you.
B
H
J
The
question
is
data
cleanup,
so
I
assumed
this
2
part
was
just
deleting
data,
you
don't
know
any
need
and
implicitly
also
down
sampling.
So
the
first
thing
to
note
is
that
per
medias
itself
is
pretty
efficient
in
terms
of
data
storage.
We're
talking,
1.6,
1.7
bytes
per
sample
is
typical
in
production.
If
you
go
to
a
another
system
like
graphite
or
mrtg,
it
is,
might
you
might
be
talking
40
bytes,
100,
bytes,
200
White's,
depending
on
what
system
so
first
off,
because
Prometheus
is
so
efficient.
J
Your
needs
to
clean
up
data
is
substantially
less
now.
There
is
the
retention
time,
which
you
can
specify
a
time
limit
for
the
Prometheus
and
we'll
take
care
of
it.
There's
also
a
size
limit
as
of
two
three
versions
ago.
Another
option
you
have
then
is:
there
is
no
down
something
inside
Prometheus
itself.
That's
kind
of
a
more
cluster
storage
thing
which
Janus
has
and
it
turns
out
to
be
useful.
You
need
like
an
hour
because
of
complexity,
and
then
the
other
thing
is
there
is
a
delete
API
in
Prometheus,
so
you
can't
delete
data.
J
G
E
You
can
you
can
build
recording
rules
and
then
you
can
filter
something
cable.
You
can
build
recording
rules
in
Prometheus
which
does
the
aggregation,
and
then
you
can
use
relabeling
rules
on
your
remote
right
path
to
drop
everything
else
and
only
send
recording
rules.
So
yes,
but
I,
we
don't
do
that.
We
just
send
it
all.
Okay,.
M
J
Prometheus
is
relatively
Domon
houseworks
if
there's
no
awesome,
optimization
or
so
on.
You
have
to
add
the
recording
rules
yourself
and
if
you
ask
it
to
do
something,
it
will
do
it
these
days.
We
have
isolation
of
what
not
not
when
that
happens.
But
if
you
ask
it
to
go
across
ten
gigs
of
data,
it
is
going
to
go
across
ten
gigs
of
data
and
then
that
just
takes
time
because
it
takes
up
with
time
to
process
ten
gigs
of
data.
J
If
you
want
a
recording
rule,
you
might
be
able
to
cut
that
down
to
a
hundred
Meg's
or
something,
but
at
the
end
of
the
day,
if
you
want
to
process
a
lot
of
data
you're
processing
it
lost
if
it's
going
by
a
remote
reread,
you
know
we
have
to
send
all
that
data
over
the
wire
against.
If
it's
just
on
side,
one
Prometheus,
then
that's
only
going
over
to
system
bus.
So
it's
going
to
be
quite
a
bit
faster.
E
N
I
have
a
question
about
the
about
the
scrape
interval
so
Julius
in
the
presentation
mentioned.
A
typical
scrape
interval
might
be
fifteen
or
thirty
seconds,
so
I'm
wondering
is
there
guidance
there
for
a
certain
number
of
metrics,
a
certain
number
of
scrape
endpoints.
What
the
practical
lower
limit
is
for
this
great
interval
and
then
similarly
for
the
evaluation
interval
of
alerts,
depending
on
how
many
rules
of
complexity
of
those
rules
etc.
B
So
it
kind
of
depends
on
your
data
and
what
you
want
to
achieve.
Generally
speaking,
we
suggest
within
one
premises.
Instance,
you
keep
the
same
great
time
for
everything,
so
you
don't
start
mixing
within
what
one
instance
you
can,
but
did
there
a
certain
reason
why
why
you
probably
shouldn't
course
it
will
make
your
life
easier
in
some
regards,
but
this
is
basically
just
a
question
of
how.
What
does
your
work
look?
Look
like.
J
Percona
does
state
they
are
asked
for
sub
second
to
be
supported,
but
I
would
personally
consider
it
at
below
fifteen
or
five
seconds
you're
getting
into
profiling
and
even
getting
endpoints
to
respond
that
quickly.
You
know
it
can
be.
It
can
be
difficult.
Why
are
you
you
used
to
SNMP?
You
know
it's
slow,
but
but
do
think
about
which
to
is
appropriate
like
if
you
want
to
every
now
and
then
just
hammer
something
to
get
detailed
stats.
You
can,
if
you're
doing
that,
all
the
time
you
might
end
up
using
too
much
CPU.
B
If
you
start
getting
into
networks
black
box,
this
part
is
something
which
potentially
you
want
to
hit
every
single
second
for
for
ICMP
probes,
maybe
even
for
HTTP
stuff-
to
really
get
a
good
picture
of
how
stuff
changes
over
time.
But
this
is
more
on
the
networking
side,
not
so
much
on
the
service
side
itself.
O
J
P
J
The
toad
is
there,
it's
not
too
hard
to
add
them,
but
you
have
to
add
them
in
the
source
code
and
we'd
have
to
ask
the
question
of
okay.
How
useful
is,
is
that
very
similar
to
something,
or
we
already
have
and
so
on,
because
we
already
have
11,
aggregators
and
50
something
functions
and
we're
just
adding
a
function
like
we
have
a
standard
deviation,
which
is
population
standard
deviation.
J
Well,
we
don't
have
sample
standard
deviation
and
if
someone
to
request
that
I'd
ask
the
question
of,
why
do
you
care
it's
basically,
the
same
number
and
because
one
is
just
dividing
by
n
when
it's
divided
by
n,
minus
1?
Is
that
that
sort
of
question?
Well,
if
you
asked
like
I,
think
we've
got
a
pretty
good
coverage,
which
aggregator
are
you
considering.
P
J
Q
Hi
I
was
wondering
if
you
thought
about
separating,
writing
and
scraping
of
reading,
because
I
have
a
lot
of
users
in
my
cooperation,
but
I
prefer
dashboards
and
when
they
open
it
on
one-month
period,
they
just
kill
the
primitives
because
of
killed
and
I
was
thinking.
Can
you
separate
reading
to
writing
so
that
when
they
kill
the
permit,
is
it
continue
to
scrape
yeah.
A
So
first
little
answer
is
like
there
are
some
sanity
limits
in
Prometheus
to
say,
X
number
of
crews.
At
the
same
time,
and
as
of
which
version
I
forgot
a
couple
of
versions
back
now,
there's
sample
limits
to
how
much
one
query
can
touch.
But
of
course,
so
you
have
the
problem
yeah
during
an
outage,
everyone
wants
to
see.
What's
going
on
and
everyone's
firing
up
their
grip
on
you,
we
haven't
really
thought
about
completely
separating
that
yet
because
it's
so
it's
all
one
process
and
it's
one
server.
A
So
it's
a
bit
hard
to
do,
but
there's
different
things
you
can
do.
You
could,
of
course,
run
one
Prometheus
that
or
multiple
parameters
is
that
just
for
being
queried
and
not
during
your
alerting
workload,
but
then
also
what
gitlab,
for
example,
is
doing
or
well
they're,
using
a
proxy
between
griffin,
ax
and
prometheus,
that
is
called
trickster
from
Comcast.
A
H
J
H
L
Just
watch
you
can
always,
you
know,
fall
back
to
you
like
you
know:
integration,
sistex,
ten
extension
systems
like
cortex
or
tano's.
They
all
essentially
move
the
data
away
from
the
scraping
Park
so
to
keep
from
queues
mainly
for
scraping-
and
maybe
you
know
local
evaluations,
and
you
can
actually
make
read
buff
more.
You
know,
scalable
complex
as
well
and
and
and
resilient
and
isolated
with
a
little
scraping,
part
and
and
cortex
has
already
caching
enables
you
have
this
trickster
built
in
tunnels.
Hopefully
have
that
soon
as
well.
So
that's
another
option.
All.
M
R
So
I
have
two
questions,
so
the
first
one
is
that
I
see
the
Prometheus
server
reached
out
to
each
exporter
and
applications
to
get
metrics.
So
what
if
I
want
to
send
metrics
in
the
other
direction,
for
example,
there
may
be
infrequent
events
triggered
by
explicit
user
actions,
so
I
wonder
what
how
can
I
get
metrics
in
the
case.
The
second
question
is
because
I
am
a
Java
developer,
I'd
like
to
monitor
like
JVM
statistics
and
performance,
so
I
wonder
if
you
have
like
built-in
good
support
for
monitoring
JVM,
like
GCE
statistics,
etc.
J
So
I
can
answer
it.
The
Java
question
as
the
client
Java
maintainer
and,
like
all
client
libraries,
we
try
to
export
out-of-the-box
what
we
can
that
make
sense
which
includes
all
the
VMS
the
JVM
starts.
The
thread
starts
to
memory
pool
stats
in
the
case
of
Java,
for
technical
reasons,
you
do
have
to
call
default
export
stuff
initialize,
which
is
in
the
client,
Java
hotspot,
module
and
then
you'll
get
all
of
those.
And
the
other
question,
then,
is
what
happens
when
I
want
to
push
infrequent
events.
J
So
if
this
is
on
a
long-running
server
and
there's
an
event,
you
increment
a
counter
and
prometheus
will
get
it
after
that.
It's
next
scrape.
If
this
is
a
batch
job
which
is
running
every
hour
every
day
or
what
not
at
the
end
of
its
run,
it
would
push
the
metrics
to
push
gateway
and
then
Prometheus
will
scrape
that
regularly.
Does
that
answer,
questions.
A
You
want
to
track
front-end
metrics,
like
how
many
users
are
clicking
a
certain
button
on
my
website
yeah.
So
for
that
first,
you
would
need
to
send
that
event
to
something
in
your
data
center.
That
can
count
that
event
with
done
read
this
button
with
a
certain
so
and
so
has
been
clicked
so
many
times.
B
A
J
O
S
Like
you
can
do
two
things
you
can,
you
can
say
only
Prometheus
one
writes
to
the
remote
right
and
Prometheus
2
is
H
a
or
there
are
systems
out
there
like
time
scale
now,
cortex,
you
just
add:
a
new
external
label
called
underscore
underscore
replicas
to
your
Prometheus,
and
the
remote
system
will
deduplicate
at
in
this
time.
If
Prometheus
one
goes
down,
the
remote
system
will
look
at
it
and,
like
I,
haven't
received
any
data
for
15
seconds
I'm
going
to
accept
data
from
prometheus
2
4.
O
F
So
what
we
do
on
openshift,
for
that
is,
we
simply
have
Prometheus,
always
firing
an
alert,
and
then
we
pass
that
down
to
the
entire
pipeline
and
whatever
paging
service
you
use.
You
have
that
paging
service
page
you
when
that
alert
is
not
firing.
So
it's
a
Deadman
switch
like
untrained,
for
example
right.
If
something
is
not
touching,
the
switch
its
firing.
A
So
the
steadman
switch
thing
is
kind
of
last
resort
that
you
can
add
that
max
mentioned.
So
it's
always
firing
alert
and
then
an
external
service
notifies
you
if
it
does
not
receive
that
alert
every
one
minute
or
so
right.
So
something
in
your
lodging
infrastructure
is
completely
dead
now
and
then
it
alerts
you,
but
before
that
you
can
already
met
Amanat
or
prometheus
normally
using
parameters,
so
Prometheus
itself
also
exposes
Prometheus
metrics
alert
manager
as
well.
A
So
you
would
have
a
meta
monitoring,
Prometheus
setup,
which
just
scrapes
your
other
Prometheus
components,
and
then
you
can
have
really
more
complex,
alerting
rules
to
find
and
they're
like
hey
those
Prometheus
stopped
ingesting
or
hey.
Those
queries
are
really
slow
in
this
one,
etc,
etc.
And
then
you
can
have
like
proper,
alerting
rules
and
then
this
last
one
that's
the
whole
pipeline
kind
of
into
end
test,
that's
more
as
a
like
last
resort,.
S
G
S
Essentially,
you
can
do
both.
It
depends
on
your
use
case
for
remote
right.
It
is
just
to
ship
all
the
data
to
a
remote
source
so
that
you
can
visualize
or
do
operations
on
the
data
in
the
remote
source,
but
the
query
API,
if
you
just
want
to
ask
Prometheus,
hey,
give
me
data,
you
can
just
use
the
query
API
and
the
remote
there's.
Also
a
remote
read
API,
where
you
can
get
raw
data
from
Prometheus.
You
can
also
use
that.
G
B
So
one
question
from
our
side:
cars
people
are
drifting
off
slowly
and
this
is
directly
the
inverse
from
all
the
other
cube
concert.
The
deep-dive
had
a
lot
more
visitors
than
the
intercession
and
we
always
did
the
deep
dive
the
same
thing
or
the
same
way.
So
first
question
is:
who
of
you
is
aware
that
promises
has
a
booth
in
in
the
sponsor
section?
B
B
B
Q
It's
like
regarding
allergy
I
was
wondering
if
you
have
plans
to
include
a
rating
based
on
thresholds
based
on
levels.
I
mean,
for
example,
I
will
just
take
something
out
of
the
rhyme
I
weight
of
one
server.
It
can
be
fresh
or
can
be
different
from
a
database.
One,
a
storage,
server
and
I
don't
want
to
duplicate
the
Aleuts
and
I
just
want
to
set
different
fresh
service
for
the
same
area.
So.
J
If
this
is
your
own
infrastructure,
I
would
advise
duplicating
alerts.
If,
however,
these
are
coming
from
different
customers
and
there's
like
some
database
or
UI
or
even
source
control,
they're
coming
from
there
is
a
technique
for
doing
this,
and
it's
on
my
blog
somewhere
I
forget
the
name
which
I
think
it's
thresholds
based.
J
Yeah
using
time
series
the
threshold
sounds
about
right:
I've
got
over
200
blogs,
it's
difficult
to
remember
all
the
funny
names
I
came
out
push
yeah,
but
this
may
skew
trick
with
group
left
and
using
thresholds.
But
if
it's
your
own
infrastructure
honestly,
it's
simple
it's
just
if
it's
your
own
infrastructure
that
you
were
running
for
yourself,
it's
easiest
to
duplicate
the
alert.
T
Hi
as
a
software
developer,
that
provides
software
that
should
expose
metrics
to
Prometheus.
Do
you
have
anything
that
can
assist
me
to
ensure
that
those
metrics
are
provided
by
integration
tests
or
something
so
that
the
tag
in
in
quality
or
integration
tests
or
with
something
else
than
some
errors?
I
can
ensure
that
the
software
that
is
to
be
shipped
will
expose
those
metrics.
A
Unless
you
build
it
all
yourself
and
check
that
it
works
there,
I
would
actually
be
really
cool
if
there
were
more
was
more
tooling
around
this
like
if
a
binary,
somehow
better
could
register
its
metrics
and
possible
label
values
with
some
registry
and
then
later
on.
You
can
check
in
your
Heine
can
have
like
a
checker
tool
run
across
your
software.
The
rules
based
on
it
and
alerting
routing
and
so
on,
doesn't
exist.
Yet
what
another
thing
that
exists
is
unit
testing
for
alerts,
so
you
can
actually
also
with
it's
also
another
prom
tool
command.
A
You
can
write
basically
a
spec,
you
can
say
like
okay,
given
this
file
that
defines
alerting
rules
and
given
those
input,
data
I
expect
to
get
the
following
output
alerts.
So
you
can
write
unit
tests
for
your
alerting
rules,
but
you
have
to
like
specify
you
know
the
so
assumed
data.
You
cannot
actually
check
against
some
binary
that
you
have
that
that
binary
would
produce
that
data.
So
it's
still
limited.
L
U
Also,
if
you
I'm
not
sure
you
if
this
was
the
exact
same
thing
that
you
mentioned
but
in
in
the
NGO
client,
there's
also
a
test,
you
tool
that
actually
does
a
scrape
of
the
internal
registry,
basically
and
I.
Think
it's
like
test
you
together
and
compare-
and
you
can
just
say
this-
is
the
metric
output
that
I
would
have
expected
if
I
had
done
a
scrape
of
this
application
and
like
in
kubernetes.
This
is
used
like
all
over
the
place
for
a
lot
of
metrics
and
that's
why
we,
how
we
basically
test.
F
J
The
other
thing,
if
you
do
have
a
full
CIC
dating
and
at
least
in
your
release
processes,
you
might
be
checking
as
part
of
the
release
that
hey
has
process.
Cpu
usage
gone
up
relative
to
requests
and
check
that
sort
of
thing
for
your
key
metrics.
Something
else
to
keep
in
mind
is
how
much
should
you
unit
test,
because,
if
you
think
about
log
lines
like
application
logs,
how
often
do
you
unit
test?
J
The
answer
is
well
when
it's
rebuilding
and
other
important
stuff,
like
that
and
I'd
say
it's
similarly
for
metrics
like
if
I'm
writing
an
or
PC
library,
the
metrics
are
part
of
the
main
feature
set,
so
those
should
probably
be
tested.
But
if
it's
just
like
a
troll
away
like
compression
just
in
case
I
ever
need
it.
I
wouldn't
personally
unit
test.
That
and
part
of
the
reason
for
that
is
like
yeah
I've
seen
systems
like
big
systems,
and
it
turns
about
about
maybe
10
percent
of
them.
The
metrics
are
broken.
J
Somehow
such
as
you
forgot,
the
registration
or,
and
you
it
just
isn't
monitoring
the
wrong
thing.
If
you
add
unit
testing,
then
that
number
should
go
down
to
maybe
5%,
because
you're
monitoring
the
wrong
thing,
but
because
you
now
have
a
rule
that
all
metrics
must
be
unit
s.
Did
you
get
far
fewer
metrics,
so
I
think
it's
better
to
have.
J
Okay,
we'll
have
more
match,
but
you
know
5%
more
than
were
broken
Robert
and
okay
everything's
unit
tested,
but
we
have
fewer
metrics
because
then
every
little
bit
of
data
you
can
have
20
bugging
and
outage
we're
trying
to
figure
out.
What's
going
on
inside,
a
piece
of
code
is
useful,
sometimes
even
if
it
is
wrong.
So.
J
So
big
things,
a
lot
of
the
things
these
days
are
happening
in
the
ecosystem,
so
we've
got
things
like
Tanis
and
cortex
in
Prometheus
itself,
like
I
personally.
Think
it's
reasonably
near
to
feature
completion
as
in
what
Prometheus
core
should
have
like
stuff
coming
up,
is
the
TLS
stuff
and
which
is
currently
slowly
going
to
denote
exported
elsewhere
and
bulk
in
Porsche
and
backing
back.
Inserting
recording
rules
is
being
worked
on
and
there's
some
UI
stuff
for
paradius
we
have
yeah
open
metrics
is
always
supports
metadata.
J
U
Some
of
us
have
discussed
this,
not
not
necessarily
everybody
there.
We
may
be
looking
into
improving
some
of
our
API.
It's
like
the
remote
read
API
we've
thought
about
making
investigating
how
we
could
make
some
of
our
API
streaming,
but
I.
Think
like
this,
you
can
see
the
theme
I
think
we're
not
really
planning
any
major
feature
work
in
core
Prometheus
yeah.
That's.
A
Q
L
Yeah
there
is
the
possibility,
like
that.
There
are
definitely
lots
of
advantages
of
this,
so
you
actually
have.
As
a
user
like
you,
don't
have
got
many
options
and
there
is
joint
force,
for
you
know
better
and
right
support,
but
yeah,
as
you
can
imagine,
merging
the
two
projects
are
or
many
projects
are
not
easy,
like
you
can
see
on
open
telemetry,
just
doing
that.
It's
it's
quite
epic
adventure
and
long
one
as
well.
L
Politics
as
well
and
yeah
from
from
my
side
like
as
a
maintainer
I,
don't
want
to
care
about
the
politics
that
much
I
want
to
build
something
that
works
for
everyone
and
and
solve
all
the
use
cases.
So
yeah
definitely
will
look
forward
to
make
it
simple
and
more
easy
to
understand
and
choose
from
and
maybe
even
including
to
produce
ecosystem
as
some
recommended
tool.
Yeah.
V
S
V
U
Maybe
coming
back
to
your
question
about
the
two
projects,
merging
I
think
some
of
the
work
that
we're
investigating,
like
the
remote
read
API
being
streaming,
is
actually
that
this
could
be
like
an
inter
exchange
format
for
both
projects
potentially
like
as
I
said
like
this
is
totally
exploratory.
I,
don't
know
if
that's
gonna
happen,
but
it
seems
some
of
us
agree
that
this
could
be
a
good
direction
so
that,
because
both
projects
essentially
promised
long-term
storage
right.
U
If
we
now
build
a
new
thing
and
it's
not
compatible
with
the
old
thing,
then
we've
basically
broken
that
trust,
and
we
definitely
don't
want
to
do
that.
So
remote,
like
remote,
read
improving
that
and
maybe
making
that
the
API
is
that
both
of
these
projects
work
with
that
wouldn't
also
make
merging
much
easier.
I
know,
I
think
on
a
political
level.
I
think
we
like
we're
all
good
friends,
so
we
would
love
to
work
more
closely
together
and
cortex
has
really
cool
features.
That
tano's
would
like
tahno
says
really
cool
features.
U
S
B
There's
maybe
one
more
take
on
this
I
mean
you.
You
have
currently
to
pain
pressure
points
within
Prometheus
when
it
comes
to
stuff
which
you
can
solve
by
by
charting,
and
this
is
basically
the
index
and
the
storage,
and
that's
why
it
just
makes
sense
to
address
both
in
one
single
thing,
the
other
just
for
your
information.
We
have
this
on
the
community
site.
B
Of
course
you
like
for
for
stuff,
which
we
want
to
do
and
such
our
def
summit
notes
are
out
in
the
open
and
we
will
probably
push
publish
something
next
week
or
so
after
Q
Khan
will
also
sit
together
and
do
Tim
team-building
slash
dev
summit.
So
there
should
probably
be
some
results
from
this
as
well.
Q
The
first
time
I
installed
poetess.
It
was
maybe
maybe
yeah
one
year
ago,
and
it
was
the
default
value
for
retention
for
30
days
and
I
was
like
what
why
I
set
it
to
for
201
year?
No,
not
actually
asking
any
question
because
I
didn't
want.
I
did
not
know
at
the
time
that
the
storage
wasn't
wasn't
compliant
with
that
kind
of
retention.
Now
it
is,
it
seems
it
is.
Q
B
Keep
in
mind
that
promise
is
coming
from
a
place
where
the
most
important
thing
is
to
to
alert
a
human
about
things
going
wrong,
and
that
is
one
of
the
core
things,
the
other.
If,
if
I'm
being
evil,
I
might
say
that
Google
doesn't
care
about
anything
which
is
beyond
14
days.
So
that's
where
that
was
coming
from
no
you're,
not
getting
the
mic
and
they
have
other
systems
in
play
in
place
and
such
for
anyone
who
has
this
as
their
own
system.
It's
obvious
that
you
need
to
have
long-term
retention.
It's
it's
obvious.
Yeah.
B
Q
J
Yeah,
so
something
else
to
keep
in
mind
with
the
15
days
is
that
that's
a
default
from
tree
storage
engines
ago
and
that
storage
engine
was
nowhere
near
as
efficient,
but
it
was
topping
out
about
50,000
samples
per
second
and
yeah.
That's
where
I
comes
from,
and
we
can
only
change
that
value
on
a
major
release
and
the
proposal
to
change
it
on
v2,
but
it
was
talked
about.
J
It
was
never
actually
made
because
it
would
be
a
breaking
change
for
users,
because
if
we
suddenly
bump
that
to
infinity,
which
I
believe
was
a
proposal,
users
were
on
a
disk
space
under
monitoring
falls
over.
So
we
will
at
some
point,
presumably
change
that
default,
but
we
can
only
safely
do
it
in
a
major
version
change
and,
to
be
honest,
I'm
not
sure
there
ever
will
be
a
Vitry.
There
might
be
at
some
point,
but
you
know
that
that's
it's!
B
We
would
probably
have
a
three
point:
X
just
for
a
long-term
storage,
but
no
to
be
fair
and
and
I
mean
there's
this
more
like
of
a
discussion
here
in
front
and
not
so
much
Q&A.
But
this
14
days
thing
is
stuck
in
the
head
of
a
lot
of
early
adopters
and
to
me
it
caused
a
lot
of
damage
in
in
the
mind
share
of
the
community.
Of
course,
it
was
something
which
a
lot
of
people
just
fundamentally
disagreed
with
and
for
various
reasons
back
then
that
was
the
thing
which
was
officially
recommended.
B
A
I,
try
to
sell
Prometheus
as
a
metrics
based
monitoring
and
alerting
stack
and
I.
Don't
even
mention
time,
series
database
or
well
I
say
it
happens
to
include
a
time
series
database
for
that
purpose,
but
it's
not
a
time
series
database
front
and
center.
So
it's
pretty
specialized
one
for
systems,
monitoring.
W
This
is
mainly
just
for
the
thinness
and
the
the
cortex
owners.
I
know
Tom's,
not
here,
but
ya,
know
I.
Think
I
I
would
love
to
ask
a
question
of
you
know
if
there's
any
interest,
essentially
and
like
looking
at
other
open-source
long-term
storage
projects
to
work
to
work
with
when,
when
you
consider
I
guess,
like
a
more
longer-term
plan
for
a
Prometheus
native
long
term,
store
I
know
that
just
something
the
indexing
side.
At
least
you
know,
a
lot
of
work
has
gone
into.
Obviously,
I
have
a
bias
and
there's
a
reason.
W
I'm
asking
this
question:
it's
because
I'm,
the
the
maintainer
of
three,
which
is
an
open
source,
long
term
storage,
but
you
know
I,
think
we
are
heavy,
a
users
of
Prometheus
Pronk
you
all.
We
love
the
ecosystem,
we'd
love
to
be
more
involved.
So
maybe
this
is
not
the
right
format
to
be
asking
the
question
I
mean.
S
We
try
it
like.
We
are
impressed
by
the
insane
scale
at
m3.
Has
you
have
more
scale
than
any
of
us
here
which
is
like
insane,
which
super
impressed
and
we've
actually
looked
at
m3
like
I
know,
Fabian
looked
at
m3
before
starting
Thanos
and
there
was
no
documentation
back
then,
and
we've
tried
to
open
issue
says
like
yeah.
S
If
it's
all
a
big
huge
black
box
that
too
complex
back,
then
it
I
think
it
changed
a
lot
now
and
we
should
take
another
look,
but
yeah
I
mean,
if,
like
it
just
happens,
that
we
know
tano's
has
a
lot
of
great
things,
codex's
a
lot
of
great
things,
but
we
should
also
look
at
what
m3
can
add
to
the
table.
Given
the
scale
and
everything
that
you've
achieved.
Also.
S
Again,
like
the
whole
point
about
the
mergers
first,
we
actually
come
come
come
up
with
common
interfaces
that
Thanos
and
cortex
like
integrate
with,
and
in
the
end
like
you
can,
you
can
done
a
component
of
Thanos
and
you
can
own
a
component
of
cortex
because
they're
the
same
api,
s--
and
m3
could
come
in
implement
those
same
api
sand
you.
You
could
also
swap
it
within
three.
B
Okay,
so
we
are
Prairie
closing
four
minutes
early,
so
we
are
probably
closing
four
minutes
early.
Thank
you
very
much
for
staying
until
the
end
come
by
the
booth
grab,
free
stickers
or
a
notebook
or
something
we
have
them
under
the
table.
But
if
you
ask
for
them
yeah,
thank
you
very
much
and
see
you
again
soon.