►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Prometheus Project Intro – Julius Volz, Prometheus (Any Skill Level)
About Julius
Julius co-founded Prometheus and lead the project to success at SoundCloud and beyond. He now focuses on growing the Prometheus community and is the main organizer of PromCon, the first conference around Prometheus. In a previous life, Julius was a Site Reliability Engineer at Google.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Alright,
thanks
for
thanks
for
coming,
everyone,
I'm
Julius
waltz
one
of
the
cofounders
of
the
Prometheus
monitoring
system,
and
this
is
really
like
an
intro-level
talk
if
you're
already
using
Prometheus
and
you
already
kind
of
know
what
it
is.
Maybe
it's
not
so
useful,
so
I'm
just
giving
you
a
chance
to
leave
now
anyway.
So
in
this
talk,
I'm
going
to
just
tell
you
what
prometheus
is?
What
are
the
basic
components?
A
A
We
do
not
do
certain
other
kinds
of
things
just
want
to
be
clear
on
this.
Prometheus
only
does
metrics.
That
means
like
individual
values
that
change
over
time,
but
that
have
some
kind
of
time-series
identifier.
We
do
not
do
logging,
where
you
want
to
store,
like
all
the
details
of
an
individual
event
or
request
tracing
so
for
logging.
You
might
use
something
like
elasticsearch
for
tracing,
maybe
something
like
Zipkin
and
we
still
recommend
like
having
metrics
logging
and
tracing
in
your
infrastructure.
A
We
allow
you
to
define,
alerting
rules
that
could
be
possibly
very
complex
but
they're,
always
very
explicit,
so
Prometheus
doesn't
do
any
magic
machine,
learning,
style,
anomaly,
detection,
just
looking
at
your
data
or
saying
like
Oh,
something
looks
really
weird
here.
Let's
page
someone
also,
the
parameter
server
itself
is
kind
of
one
node
system
and
you
can
scale
it
manually,
but
it's
not
a
horizontally
scalable
system
where
you
just
press
a
button,
add
more
machines
and
it
kind
of
clouds
up
and
that's
also,
the
storage
by
default,
as
it
is.
A
Infirmities
itself
is
not
a
durable
storage.
If
you
lose
the
node
that
part
of
the
storage
is
gone,
it's
more
of
a
really
like
real
live
monitoring
system,
but
we
do
have
ways
of
integrating
long
term
storages
that
I'm
going
to
mention
later
so
we
started
from
Ethier's
in
2012
at
Sound
Cloud
out
of
a
real
need.
A
Matt
and
myself
just
came
from
Google
at
that
time
and
Sound
Cloud
was
a
company
who
already
had
in-house
built
like
a
cluster
manager.
Basically
before
docker
or
kubernetes
or
Maysles
really
existed,
and
you
know
they
had
hundreds
of
micro
services
running
on
there
and
we
needed
something
to
monitor
that
I'll
get
to
that
in
a
second.
So
we
started
it
at
Sound.
Cloud
brought
it
to
relative
maturity,
beginning
of
2015.
We
really
published
it
took
off.
How
can
use
it
all
that
now
it's
part
of
the
cloud
native
computing
foundation.
A
Many
companies
are
using
it
it's
kind
of
for
many
people.
It's
almost
like
a
default
if
you're
using
kubernetes,
it
fits
in
very
well
and
yeah
plenty
of
contributors,
big
project-
maybe
I-
should
say
here.
Also,
it's
a
bit
special
in
comparison
to
some
of
the
other
CN
CF
projects
and
that
it
didn't
really
have
one
engineering
company
that
intentionally
created
this
thing
to
put
in
the
open
source
world.
It
was
more
of
a
byproduct
like
we
created
it
because
we
needed
it,
and
then
we
thought-
oh,
let's
open
sourced
this,
but
SoundCloud
wasn't
really.
A
A
So
yeah
the
motivation
was
really
SoundCloud
already
had
this
in-house
build
class
2
scheduler,
they
had
hundreds
of
micro-services
thousands
of
instances,
and
we
were
just
trying
to
figure
out
why
SoundCloud
was
down
all
the
time
or
slow
or
whatever,
and
it
was
very
hard
like
if
there's
a
latency
spike
using
the
existing
monitoring
to
stack,
then
graphite,
stats,
D,
ganglia,
New,
Relic,
etc,
etc.
They
either,
you
know
in
terms
of
the
data
model
or
in
terms
of
the
query.
A
Language
or
the
amount
of
history
they
could
take
into
account,
didn't
really
give
us
the
the
detailed
insight
that
we
needed
to
to
figure
out
where
exactly
what
was
going
wrong.
So,
finally,
we
decided
to
build
a
new
solution
which
eventually
became
Prometheus
to
tackle
all
these
issues,
I
would
say:
yeah.
Let's
first
have
a
look
at
the
overall
architecture.
A
A
You
can
configure
these
in
scalable
trees
or
other
kind
of
you
know
scalable
architectures,
but
let's
just
start
with
a
single
one.
You
configure
that
Prometheus
server
to
fetch
metrics
over
HTTP
in
a
format
that
we
define
and
the
metrics
can
really
come
from
any
kind
of
thing
that
can
serve
metrics
over
HTTP.
So
you
know.
Typically,
we
recommend
you
to
if
you
have
your
own
applications
like
an
own
web,
app
API
server,
put
metrics
directly
into
your
own
code,
white-box
instrumentation.
That
tells
you
exactly
what's
going
on
inside
the
application.
A
Those
are
the
best
metrics
and
then
Prometheus
can
also
go
directly
to
your
code
and
know
it's
like.
Oh,
you
know
if
I
cannot
pull
from
your
thing
it's
down
and
all
these
nice
side
effects
for
some
things.
You
cannot
directly
do
that.
You're
not
gonna,
like
input
Prometheus
metrics
into
the
Linux
kernel
or
into
the
my
sequel,
daemon
right,
and
for
that
we
have
this
exporter
model
where
you
run
a
sidecar
job
right
next
to
the
thing
that
has
metrics
in
some
other
kind
of
format.
A
Already
like
the
Linux
kernel
over
the
proc
file
system,
for
example,
the
node
exporter,
does
that
for
host
metrics
or
for
Linux,
then
Prometheus
scrapes,
the
node
exporter
and
the
node
exporter
in
the
background
just
goes
to
the
proc
file
system
and
some
other
to
some
other
syscalls
to
figure
out
actual
host
metrics
translates
them
to
prometheus
metrics
and
shipped
them
back.
Synchronously
to
Prometheus
Prometheus
has
a
local
time
series
database,
where
it
stores
all
these
time
series
over
time.
It's
a
configurable
retention
could
be
like
weeks,
sometimes
months.
A
A
I
would
say
these
are
my
favorite
selling
points
of
Prometheus
over
other
monitoring
systems
back
in
the
day,
at
least,
and
still
nowadays,
I
think
the
first
to
kind
of
go
hand
in
hand.
Prometheus
has
a
dimensional
data
model
that
allows
you
to
record
in
relatively
high
detail
for
a
time
series
based
system
what
is
happening
where
and
then
to
make
sense
of
that
collected
high
detail
data.
A
We
give
you
a
powerful
query
language
to
compute,
whatever
you
kind
of
currently
want
to
know
about
that
data,
a
single,
primitive
server
is
pretty
simple
to
operate
and
still
relatively
you
know,
efficient
I'll
get
to
that
later
and
the
point
that
makes
Prometheus
specifically
work
really
well
in
dynamic
cloud
environments
and
cluster
schedulers
is
the
service
discovery
integration?
How
it
knows
what
it
should
monitor
all
the
time,
so
I'm
going
to
hop
through
all
these
four
points
in
row
and
go
into
some
details.
First,
the
data
model.
A
You
know
in
all
time
series
have
some
kind
of
identifier
and
then
they're
just
like
turns
them
value
times
them
value
times
them
value
and
in
prometheus
all
the
timestamps
are
in
64
in
millisecond
precision.
Let's
just
keep
it
simple,
that's
it!
That's
all
you
need
for
systems
monitoring,
that's
what
we
say.
The
value
type
is
always
a
float64
that
works
for
pretty
much
everything
you
can
increment.
That,
like
for
I,
don't
know
forgot
like
285
years,
1
million
increments
per
second
before
you
reach
the
first
non
consecutive
integer.
A
So
at
that
point
you
should
like
reset
counters
at
some
point,
or
so
so
it
doesn't
really
become
a
problem
but
yeah.
So
the
the
big
difference
is
kind
of
how
we
identify
time
series
compared
to
systems
that
came
before.
Let's
compare
it
to
the
graphite
and
stats
the
metrics
model,
where
you
had
a
long
metric
string
that
had
some
dots
in
the
middle
to
two
separate
components
that
form
the
implicit
hierarchy.
In
this
example,
you
have
like
an
HTTP
requests
happening
on
nginx
servers
on
different
hosts.
A
That's
the
IP
component
there
and
then
different
paths
and
different
status
codes,
but
the
hierarchy
here
is
implied
and
implicit.
You
know
you
need
to
kind
of
know
what
each
component
means,
but
also
you
know
why
would
the
status
code
be
lower
in
a
hierarchy
than
the
path
for
example,
or
why
why
wouldn't
the
host
come
first?
Also,
if
you
add
another
component,
would
you
add
it
in
the
end
in
the
beginning,
somewhere
in
the
middle,
it's
kind
of
hard
to
extend
as
well?
A
In
Prometheus
we
go
a
bit
of
a
different
route
where
we
put
the
metric
name
front
and
center.
It's
basically
the
central
aspect
of
a
system
that
you
are
monitoring
like
the
number
of
HTTP
requests.
We
have
counted
since
a
given
process
has
started,
and
you
know
the
process
exposes
that
and
then
it
can
have
can
put
key
value
pairs
on
that
metric
to
further
DIF
up
differentiate
dimensions
on
that
metric,
for
example,
you
know
the
process
itself
would
add
the
path
and
the
status
labels
onto
these
metrics,
but
then
Prometheus
when
it
scrapes.
A
The
target
also
adds
some
labels
of
that
target.
Saying
like
this
came
from
this
service
or
which
we
call
job
this
job,
and
this
particular
process
instance
so
you're
getting
some
labels
from
the
instrumentation
in
the
process
and
some
labels
coming
from
the
Prometheus
server,
because
it
knows
what
it
actually
just
scraped
and
yeah.
This
gives
you
more
explicit
mapping
of
keys
to
values
and
there's
no
hierarchy
anymore
either.
A
Now,
if
you
query
that
becomes
a
bit
more
obvious,
y1
is
nicer
than
the
other
in
the
kind
of
graphite
model
you
have
to
know
where
to
put
asterisks.
Let's
say
you
want
to
query
all
the
HTTP
requests
for
all
the
engine.
X's
that
end
up
with
a
500
status
code
would
look
like
in
the
top
four
for
graphite,
it's
kind
of
implicit.
You
need
to
know
what
it
all
means
for
for
Prometheus
you
just
you
start
with
a
metric
name
and
then
you
just
add
random
label
filters.
A
You
could
also
have
negative
or
reg
X
filters
here,
just
selecting
the
time
series
that
you
really
care
about,
and
you
don't
even
have
to
know
what
other
labels
exist
on
those
time
series.
So
really
you
want
these
kind
of
key
value
levels
as
first-class
citizens,
which
also
fits
nicely
with
kubernetes
and
docker,
which
also
have
the
same
label
concept.
A
So
now
we
have
collected
all
the
useful
data.
Now,
let's
do
cool
stuff
with
it.
Prometheus
defines
a
new,
completely
custom.
Query
language,
it's
not
sequel
based
which
was
a
bit
of
a
like.
Some
people
get
surprised
by
that,
but
we
think
that
what
we
have
works
better
than
a
sequel
based
language
for
the
kinds
of
computations.
We
typically
want
to
do
in
time
series
based
monitoring.
A
A
Saying
by
path
could
add
different
labels
in
there
as
well,
and
then
it
would
preserve
this
dimension,
and
this
is
where
some
of
the
actual
cool
automatic
join
magic
happens
in
from
QL.
If
you
have
a
binary
operator
between
two
whole
sets
of
time
series,
then
it
will
automatically
join
them
up
on
identical
label
sets.
So
you
know
the
500
error
rate
for
the
path
on
the
Left
will
divide
it
by
the
will
be
divided
for
the
total
rate
for
that
same
path
on
the
right
and
that
gets
propagated
into
the
output.
A
So
it's
kind
of
this
vector
based,
arithmetic
slang,
guack,
you
can
do
more
fancy
stuff.
For
example,
you
can
track
Layton
sees
with
histograms
in
your
instrumentation
and
then
use
a
query
function
doing
career
time
to
say,
like
give
me
the
99th
percentile,
for
example,
aggregated
over
all
the
instances
and
as
averaged
over
a
5
minute
time
span,
so
you
can
choose
the
percentile,
the
aggregation
level
and
the
overtime
during
Korea
time,
and
then
you
get
these
answers.
So
these
are
just
some
examples.
A
You
can
combine
many
more
crazy
constructs
into
very
long
crazy
from
QL
statements,
but
you
know
this
I
think
does
for
beginning
now.
Once
you
know
a
bit
of
this
language,
you
can
start
looking
at
data
with
it
either
in
this
built-in
expression
browser
in
parameters
itself.
That's
in
the
primitive
server,
for
example,
just
looking
at
the
current
value
of
everything
in
your
infrastructure,
there's
also
a
simple
built-in
graphing
interface,
but
then,
once
you
want
to
have
more
bells
and
whistles
nice
dashboards
that
you
can
save
and
share
with
your
colleagues.
A
Of
course,
you'll
want
to
use
CRO
fauna,
Accra
fauna
has
native
support
for
Prometheus
backends.
So
that's
like
what
we
recommend
for
people
to
use
for
serious
dashboarding
and
then
the
cool
thing
I
think
is
that
Prometheus
also
integrates
alerting
in
a
unified
way,
with
the
query
language
and
based
on
the
data
that
is
collected.
So
you
know
in
the
Nagios
style
world,
you
would
run
periodic
checks
on
machines
that
don't
really
have
much
history.
They
just
check
if
something
is
currently
running
or
the
CPU
is
currently
high
or
so
in
prometheus.
A
It's
it's
a
bit
different.
We
just
collect.
We
collect
all
the
data
as
time
series
first
and
then
you
can
formulate
any
prompt,
qlae
expression
on
top
of
that
data
that
can
take
into
account
different
amounts
of
histories
or
also
different
aggregation
levels.
You
could
say
like
is
the
whole
service
bad
or
is
a
particular
process,
bad.
You
can
decide
on
or
something
in
between
right
and
decide
like
what's
the
relevant
alerting
level
and
the
way
that
works
is
you
say
like
in
this
example,
we
say
we
want
to
create
an
alert,
many
500
errors.
A
You
configure
that
in
your
Prometheus
server,
it
will
evaluate
it
every
couple
of
seconds.
It
will
run
this
prompt,
ql
expression
that
is
in
yellow
here,
and
this
is
a
repetition
of
something
I
had
earlier
where
it
calculates
like
that
per
path,
error
rate
ratio.
I
multiply
it
by
a
hundred
to
get
from
a
ratio
to
percentage
and
then
I
filter
it
to
only
get
the
paths
that
have
more
than
5%
error
and
in
the
lodging
role
in
prometheus
works.
A
This
way
like,
if
there's
no
output
from
an
expression,
everything's
good,
but
any
any
time
series
that
gets
outputted
from
the
expression
becomes
an
alert
and
then
an
alert
manager.
We're
premature
sends
these
individual
alerts.
They
can
be
grouped
into
single
notifications.
Like
you
know,
if
you
have
an
alert
per
machine,
for
example,
that's
done
and
there's
100
machines
down,
then
a
lot
manager
could
like
group
that
into
a
single
notification
you
can
also
say
you
know.
Every
output
element
has
to
be
bad
for
at
least
5
minutes.
A
That's
the
statement,
the
for
statement
there
before.
Actually,
you
know
bugging
someone
around
about
it.
That's
important
for
some
kind
of
robustness
and
you
can
add
extra
labels
and
annotations
that
can
be
used
in
a
lot
manager,
routing
and
in
the
actual
human
readable
snippets
that
you
get
in
your
alerts.
A
Prometheus
is
you
know,
single
static,
go
binary
kind
of
server,
and
you
just
run
it
on
a
machine.
It
doesn't
do
any
clustering
because
we
think
that's
really
hard
to
do
right
and
the
clustering
might
also
be
the
first
thing
that
breaks
in
the
face
of
a
network
outage.
So
you
know
if
you
want
to
have
high
availability
for
alerting
everything
in
parameters
is
built
around
high
availability
for
alerting,
not
necessarily
for
infinite
data
retention
or
something,
but
there
we
just.
A
So
you
don't
actually
get
two
notifications
as
long
if
both
are
still
running
yeah
but
yeah
like
getting
started
with
preemies
is
pretty
simple:
there's
like
people
running
it
on
a
Raspberry
Pi
at
home.
Of
course
it
gets
more
complicated
once
you
have
really
big
set
ups
and
you
need
to
think
about
scaling
and
topology.
A
single
server
is
relatively
efficient
like
we
can
do
yeah.
It
depends
on
the
hardware.
A
But
if
you
have
a
big
machine,
you
can
do
a
couple
of
millions
of
active
series
at
a
given
time
and
over
a
million
samples
ingested
per
second,
and
then
we
store
those
samples
as
roughly
like
something
between
one
and
two
bytes
per
sample
on
disk.
It's
a
custom
time
series
database.
So
this
is
good
for
keeping
weeks
months.
A
Some
people
go
out
on
a
limb
and
keep
like
only
few
metrics
but
for
years,
but
then
we
tell
them
like
yeah,
there's
no
durability,
guarantees
for
people
who
do
want
to
keep
data
more
durably
and
longer.
There's
some
decoupled,
remote
storage
support
where
you
can
tell
Prometheus
to
ship
all
your
all
its
samples
to
remote
endpoint
and
then
that
can
choose
to
store
it
in
a
remote
system.
A
For
example,
we
have
like
in
Flex
DB
supports
that
now,
there's
yeah,
we
have
adapters
for
graphite,
there's
cortex
from
weave
words,
there's
several
systems
that,
like
basically
over
ten,
integrations
in
this
space
now,
where
that
allow
you
to
write
Prometheus
samples
to
a
scalable,
durable
backends
system,
and
some
of
them
also
allow
you
to
read
that
data
back
through
Prometheus,
again
Prometheus
as
I
promised
works
really
well
in
dynamic
environments.
You
know.
A
It
will
tell
you
what
kind
of
part
it
is
that
you're
monitoring
its
like
environment
production
type,
name,
engine
X
whatever,
and
then
it
you
can
choose.
If
you
want
you
to
add
that
as
labels
to
your
time
series.
So
then,
you
can
actually
select
and
aggregate
on
that
data,
so
we
support
I
think
like
roughly
ten
different
types
of
services
coverage.
Some
are
kind
of
lyric,
note
level
ones
to
discover.
A
You
know
instances
on
machine
providers,
a
AWS
as
a
Google,
others
are
more
cluster
manager,
ones,
kubernetes,
obviously,
and
then
their
generic
ones,
DNS
is
pretty
agnostic
and
console
zookeeper.
You
can
even
build
your
own
one
if
Prometheus
doesn't
support
it
yet
with
a
file
watcher
based
interface
so
like
in
conclusion,
you
know
promises,
helps
you
monitor
and
make
sense
of
infrastructure
that
is
dynamic
and
complex
with
these
four
nice
properties
and
yeah.
Basically,
that's
that's
it
for
today
and
I'm
happy
to
take
questions.
A
A
A
So
there's
like
the
alert
itself
and
then
there's
a
final
incident
management
portion
which
can
happen
outside
of
Prometheus.
The
alerting
in
Prometheus
itself
is
really
based
on
kind
of
a
sliding
window
system
where
it
really
just
checks,
is
something
bad
right
now,
given
the
window
of
time
that
you're
currently
looking
at
in
a
sliding
window
fashion
and
if,
at
some
point
the
badness
shifts
out
of
that
window,
the
alert
will
also
resolve
in
prometheus.
A
Yeah,
okay
yeah.
So
as
long
as
the
problem
persists,
Prometheus
will
actually
like
every
evaluation
interval
like
which
could
be
15
or
30
seconds
sent
to
alert
manager
like
this
problem
is
still
active
for
this
particular
label
combination
and
then
an
alert
manager.
You
configure
for
a
given
grouping
of
alerts
by
labels,
which
you
can
choose
their
send
like
repeat
this,
every
two
hours,
for
example,
or
once
per
day.
A
If
this
problem
still
persists-
and
you
can
also
say
they're
like,
for
example,
if
hundred
hosts
go
down
and
you
have
an
alert
based
on
all
of
them,
but
you
want
to
group
them
in
a
lot
manager.
Then
you
can
say
well,
you
know
maybe
first
I
get
alerts
about
70
of
them
being
down
and
then
I
don't
want
to
send
out
a
notification
yet
because
maybe
30
others
will
come
just
like
a
couple
of
seconds
later.
A
So
you
could
say
in
a
lot
manager
like
wait,
30
seconds
in
case,
there's
more
alert
elements
coming
into
the
same
grouping,
so
you
can
still
group
them
together
and
these
kind
of
features
there.
But
at
some
point,
if
from
easiest
things,
the
problem
is
gone
gone
then
it
will
like
tell
a
lot
manager.
Oh
it's
resolved,
and
it
will
also
disappear
in
alert
manager.
Yeah.
A
A
Yeah
so
question
is
yeah
better
to
run
prometheus
inside
the
kubernetes
cluster
or
outside
we
recommend
inside,
because
then
a
lot
of
stuff
gets
taken
care
for
you
automatically.
You
get
the
service
account
stuff
mounted
in
automatically
and
it
works
network
wise
to
reach
all
the
pods
automatically
right
so
yeah.
We
recommend
that
and
then
of
course,
it's
nice
to
still
maybe
have
some
prometheus
outside
of
the
cluster
or
some
other
meta
monitoring
that
checks,
whether
the
one
in
the
cluster
actually
works.
Yeah.
A
Yeah
yeah
how
many
Prometheus
services
per
instances
to
monitor
I
think
it
really
depends
on
how
many
the
the
the
bottleneck
is
usually
the
number
of
time
series.
So,
if
your
instances
export
like
five
times,
here's
each
you
can
have
either
multiple
thousands
of
instances,
but
if
each
instance
has
like
ten
thousand
times
here
is
that
it
exposes
to
Prometheus
and
you
can
have
far
fewer,
you
know.
The
budgets
that
you
have
to
think
about
most
is
a
single
primitive
server
can
do
if
it's
a
big
one
can
do
like.
A
Maybe
a
couple
of
million
of
time
series
at
a
given
time.
The
time
series
is
every
unique
combination
of
labels
that
is
currently
relevant,
so
metric
name
plus,
like
all
the
you
have
to
multiply,
to
get
like
all
the
different
values
that
each
label
can
have
and
multiply
that
for
all
the
different
possible
labels
to
get
the
total
number
of
serious
you're
going
to
ingest
and
then
times
the
number
of
instances
or
pods
you
have,
and
that
gives
you
an
idea
of
your
budgeting.
There.
A
Yeah
like
it
starts
to
slow
down
ingestion,
but
you
might
also
all
just
find
that
your
queries
don't
return
anymore.
If
you
have
like
one
very
common
error
at
the
beginning
for
Prometheus
users
is
like
once
they
find
out
how
useful
labels
are.
They
put
information
into
the
label,
values
that
is
too
unbounded
in
cardinality,
like
a
user,
IP
address
or
email
or
something,
and
that
could
be
millions
of
those
right
and
that
gets
multiplied
with
all
the
other
cardinalities
of
all
the
other
labels.
A
Yeah
Rob
med
features
for
Prometheus
in
the
year
in
a
year
from
now
or
so
so.
Prometheus
as
a
whole
is
probably
not
going
to
change
that
much
anymore
in
in
terms
of
its
scope
and
what
it
wants
to
do.
There's
some
things
we
have
like.
We
have
an
official
road
map
section
on
the
site
that
has
shrunk
and
shrunk
and
truck
every
time
we
have
crossed
out
some
of
those
items.
A
One
item
that
still
on
there
is
making
better
use
of
the
metadata
like
when
you
instrument
something
on
the
on
the
side
of
a
process
that
exposes
metrics.
It
actually
exposes
still
some
metadata
about
the
metrics
like
what
each
metric
means
like
help
string
and
also
whether
it's
a
counter
or
gauge
histogram
or
summary
metric
and
the
server
when
it
stores
that
information.
A
Just
the
first
thing
it
does,
is
throw
that
metadata
away,
because
we
don't
have
any
facility
to
store
and
make
use
of
that
yet,
and
so
one
roadmap
item
is
still
somehow
temporarily
aggregating,
at
least
that
map
that
metadata
in
memory
and
Prometheus,
so
that
we
could
give
you
some
hints.
Maybe
if
you're
formulating
a
query
and
you're
doing
and
I
can
rate
over
a
gauge,
we
can
tell
you
like
oh
you're,
trying
to
do
something
that
might
be
not
as
you
intended
here.
A
A
We
had
this
initial
interface,
where
Prometheus
just
sends
samples
in
a
in
a
real-time
almost
near
time,
real-time
kind
of
way
to
a
different
endpoint
that
I
talked
about,
but
then,
if
you
permit,
the
server
is
down
for
or
the
network
connection
to
a
hosted
service
that
does
the
long-term
storage
for
you
is
like
down
for
10
minutes
and
you
just
lose
10
minutes
of
data
and
there's
some
other
approaches
that
are
more
about
shipping
whole
chunks
from
a
Prometheus
to
some
remote
system.
One
of
them
is
called
Thanos
and
yeah.
I
really
recommend.
A
A
Ok,
if
you
have
multiple,
so
if
one
Prometheus
server
is
no
longer
enough,
then
as
I
said,
Prometheus
doesn't
do
automatic
horizontally.
Like
push
button
scaling,
you
have
to
kind
of
think
about
how
you
split
up
the
load
over
your
Prometheus
instances.
If
you
search
for
blog
post,
I,
think
Prometheus
scaling
and
Federation
that's
a
good
overview.
There's
different
strategies.
You
could
like
one
example.
A
So
you
have
the
global
view,
but
not
all
the
detail
in
the
top,
and
in
the
leaves
of
that
tree
you
would
have
the
very
detailed
instance
level
view,
but
no
longer
the
global
view,
that's
the
kind
of
hierarchical
scaling,
there's
some
ways
of
doing
limited
types
of
horizontal
charting
as
well,
where
you
can
like.
You,
have
one
service
in
one
data
center,
that's
just
10,000
instances
and
they
cannot
be
scraped
with
one
Prometheus.