►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: Prometheus - Matt Layher, Fastly & Ganesh Vernekar, Grafana Labs
Prometheus is an open-source monitoring system and time series database. It features a multi-dimensional data model with a powerful query language and integrates many aspects of systems and service monitoring: from the instrumentation of services over the collection and storage of metrics data, all the way to dashboarding and alerting. Native support for various service discovery mechanisms also make it particularly suitable for dynamic cloud-based environments. In this introduction, members of the Prometheus team explain the architecture of Prometheus and show its advantages over traditional monitoring systems.
https://sched.co/Uajs
A
I
work
mostly
on
the
node
exporter,
welcome
to
the
Prometheus
introduction
session.
So
it's
worth
noting
this
is
the
intro
we're
going
to
have
the
Prometheus
deep
dive
tomorrow
morning,
I
believe.
So,
if
you
have
questions
about
more
advanced
aspects
of
Prometheus,
please
join
us
there
and
also
we'd
like
to
give
a
special
shout
out
to
Julius
Bowles,
one
of
the
co-creators
of
Prometheus
for
his
help
with
the
slides.
So
first,
what
exactly
is
prometheus
prometheus
is
a
metrics
based
monitoring
and
alerting
stack.
It
is
an
entire
ecosystem
of
components.
A
So,
for
example,
you
have
things
like
our
instrumentation
libraries.
The
Prometheus
client
can
be
exposed
from
several
different
languages,
such
as,
for
example,
Go,
Python,
Java,
Ruby,
C,
rust,
all
kinds
of
different
things,
and
there
are
also
Prometheus
exporters
and
those
operate
in
slightly
different
ways.
So
the
Prometheus
server
itself
is
your
mech
suit.
Metrics
collection
and
storage
system
actually
gathers
time
series
and
stores
them
in
this
database,
and
then
once
you
have
that
data
you
can
make
use
of
it
by
querying
Prometheus,
creating
alerts.
A
So,
for
example,
if
you'd
like
to
know
when
you've
reached
a
certain
level
of
memory
pressure
or
if
your
disk
space
is
about
to
run
out,
you
can
do
that
and
also
we
support
dashboarding
via
the
Prometheus
API,
using
software
such
as
griffons,
but,
what's
cool
about
prometheus?
Is
it's
really
meant
for
all
levels
of
your
stack?
So
not
only
can
use
Prometheus
for
your
own
applications.
You
can
instrument
things
like
the
Linux
kernel
or
web
servers
like
Apache,
nginx
and
etc.
A
Finally,
Prometheus
is
made
for
dynamic
cloud
environments,
so
lots
of
folks
to
play
Prometheus
with
things
like
kubernetes.
But
that
being
said,
it's
also
very
useful
for
the
home
user
as
well.
So
in
fact,
I
run
Prometheus
at
home.
Just
for
monitoring
my
home
services
and
I
actually
have
like
alerting
stuff
to
my
phone.
So
if
something
goes
down,
I
get
notified
about
it,
so
you
can
run
it
in
production,
but
it's
also
a
lot
of
fun
to
use
at
home.
A
So
Prometheus
is
a
lot
of
things,
but
there
are
certain
things
that
are
out
of
scope
for
the
project.
So,
for
example,
Prometheus
does
not
focus
on
logging
or
tracing.
These
are
different,
observability
primitives
and
while
they
are
quite
useful
for
a
theist,
is
squarely
focused
on
metrics
collection.
In
addition,
Prometheus
does
not
provide
automated
anomaly
detection
on
its
own.
So,
for
example,
you
could
gather
data
with
Prometheus
and
that
feed
that
into
a
different
system
to
do
things
like
anomaly
detection,
but
Prometheus
does
not
feature
this
built-in.
A
Finally,
the
Prometheus
server
stores
all
of
its
data
to
a
local
disk.
It
does
not
necessarily
provide
scalable
or
durable
long
term
storage.
So
if
you
want
to
store
time
series,
for
example,
multiple
years
that
could
get
kind
of
problematic,
but
there
are
workarounds
for
this
as
well
so
a
little
bit
of
the
history
of
the
project,
the
Prometheus
project
was
originally
created
in
2012
at
Sound
Cloud,
designed
to
address
some
of
the
flaws
in
existing
monitoring
systems.
A
The
project
became
open-source
and
was
worked
on
for
a
couple
of
years
before
being
fully
publicized
in
2015
on
places
like
hacker
news
in
2016
we
joined
the
CN
CF
as
I
believe
the
second
project
after
kubernetes
and
then
Prometheus
version
1.0
was
released
a
year
later
we
really
supreme
atheist
version
2.0,
and
this
was
done
primarily
so
we
could
replace
these
storage
back-end.
So
now
we
have
a
new
storage,
back-end
called
TS,
DB
or
time
series
database.
A
A
So
first
we
have
our
targets
and
these,
for
example,
could
be
applications
that
you
and
your
team
are
developing,
so
something
like
a
web
application
which
could
be
written
in
something
like
say,
a
PHP
or
Ruby
or
Python
and
like
an
API
server
written
and
go
or
some
other
language.
So
these
are
targets
you
control
and,
as
such
you
can
use
the
Prometheus
client
libraries
to
instrument
your
own
applications.
So
we
feature
client
libraries
for
languages
such
as
Go
Java
Python,
but
there
are
also
community
bindings
written
for
languages
like
rust
and
C
as
well.
A
However,
not
all
the
software
in
your
stack
is
necessarily
something
that
you
will
control
so,
for
example,
you
probably
deployed
on
a
Linux
virtual
machine.
You
might
be
connecting
to
a
my
sequel
server.
You
might
want
to
look
at
the
C
group
utilization
or
something
on
your
machine,
but
these
saut.
This
software
does
not
necessarily
expose
native
Prometheus
metrics.
So
this
is
where
Prometheus
exporters
come
in,
and
an
exporters
job
is
basically
to
query
metrics
from
some
source,
such
as
my
sequel,
using
that
sources
native
interface.
A
So
here's
an
example
of
what
happens
when
you
expose
Prometheus
metrics.
If
you
were
looking
at
something
like
the
slash
metrics
endpoint
on
one
of
your
instrumented
services
or
one
of
the
exporters,
you
might
see
a
metric,
something
like
this.
So
there's
a
bit
of
metadata
here
in
cell
phone
for
humans
and
all
so
indicates
the
type
of
the
metric.
But
here
we
have
a
time
series
called
HTTP
requests
total.
There
are
a
couple
of
labels
here
in
between
the
brackets,
and
labels
are
how
we
partition
data
into
different
dimensions
within
Prometheus.
A
A
So
now
you
have
your
from
easiest
server
in
the
middle
prometheus,
well,
on
regular
basis,
reach
out
to
all
these
different
targets.
So
your
own
applications,
as
well
as
your
exporters,
but
Prometheus
needs
to
find
a
way
to
actually
get
a
hold
of
these
applications
and
figure
out
where
they
live.
In
order
to
do
so,
you
can
configure
a
static
target
list
if
you'd
like.
However,
Prometheus
also
has
built-in
support
for
service
discovery,
using
mechanisms
such
as
DNS.
A
So,
for
example,
you
can
query
an
SRV
record
and
get
a
list
of
targets
for
Prometheus
to
scrape
you
could
use
coop
the
kubernetes
integrations.
So,
for
example,
if
you
have
pods
that
are
coming
up
and
down
at
different
intervals,
Prometheus
will
notice
that
and
create
time
series
and
also
flush
out
stale
time
series
for
those
pods.
You
can
integrate
with
the
service
discovery
of
your
cloud
provider
or
something
like
AWS
and
finally,
we
also
have
a
custom
service
discovery
hook.
A
A
So
once
you're
actually
gathering
this
data
at
regular
intervals,
and
you
have
some
time
series
being
built
up
within
Prometheus,
you
want
to
actually
do
something
with
that
data,
so
Prometheus
itself
features
a
web
UI
and
the
web
UI
is
good
for
kind
of
experimenting
and
exploring
with
different
queries.
So
it
features
kind
of
a
basic
interface
that
you
can
use
good
natural
explain
that
a
little
bit
more
later,
but
it
also
shows
things
like
the
current
configuration
of
the
Prometheus
server.
A
The
current
targets
that
are
loaded
into
memory
and
whether
or
not
they
are
healthy
and
also
things
such
as
the
run
time
flag
configuration
for
Prometheus.
If
you
want
to
do
a
more
advanced
use
case,
we
recommend
the
use
of
software
such
as
grow
fauna
and
grow
fauna,
enables
you
to
create
persistent
dashboards.
So,
for
example,
you
can
do
10
or
12
different
Prometheus
queries
at
once
and
create
a
tactical
dashboard
for
you
and
your
team
to
help
understand
the
health
of
your
service.
A
A
So,
for
example,
if
you
want
to
set
up
an
alert
in
prometheus
that
says
say
if
I
have
over
80%
memory
utilization
on
this
machine
fire
and
alert,
you
can
do
that
and
then,
if
you
configure
Prometheus
to
to
reach
out
to
an
alert
manager
component,
it
will
actually
send
a
hook
whenever
that
is
firing
with
alert
manager.
You
can
dispatch
that
alert
to
different
locations,
so
say,
for
example,
are
using
something
like
pager
duty.
You
can
issue
a
page
immediately
or
I
actually
use
an
app
on
my
phone
called
pushover
from
my
home.
A
So
some
of
the
major
selling
points
of
Prometheus
include
the
multi-dimensional
data
model
because
of
the
fact
that
you
can
have
a
time
series
with
many
different
labels
and
then
many
different
label
values
as
well.
You
can
really
easily
slice
and
dice
your
data
into
a
lot
of
different
dimensions
and
gain
a
lot
of
insight
into
what
your
applications
are
doing.
Prometheus
also
features
a
very
powerful
query:
language.
We
call
prom
QL
and
it's
a
simple
inefficient
go
server,
so
Prometheus
is
aesthetically.
Links
go
binary.
A
If
you
really
want
to,
you
can
just
go,
build
or
even
cross,
compile
it
SCP
it
up
to
a
Linux
machine
and
just
run
it,
and
it
will
pretty
much
just
work.
Finally,
since
Prometheus
supports
things
like
service
discovery,
it's
a
great
fit
for
applications
that
run
on
top
of
things
like
kubernetes
or
are
integrated
with
some
other
existing
service
discovery
mechanism.
A
So,
let's
discuss
a
little
bit
about
the
Prometheus
data
model.
So
first,
what
exactly
is
a
time-series?
You
can
think
of
it
as
an
identifier.
So,
for
example,
our
HTTP
requests
total
time
series
and
that
we
have
a
vector
here
and
the
vector
contains
some
tuples.
Each
of
these
tuples
contains
a
timestamp
of
when
a
sample
was
gathered
and
also
a
value.
A
So
here's
an
example,
time
series
we
have
this
HTTP
request,
total
it's
coming
from
a
job
called
nginx
and
we
have
an
instance
label
attached
as
well.
So
if
you
have
multiple
different
nginx
servers,
you
can
actually
filter
on
whichever
one
you'd
like
to
query
or
see
all
of
them
in
aggregate
by
using
something
like
prompt
QL.
So
the
result
of
this
provides
a
pretty
flexible
architecture.
The
Prometheus
metrics
are
laid
out
in
a
flat's
namespace,
so
there
is
no
hierarchy
involved
here.
A
B
You
Man,
so
I'll
be
just
going
through
some
of
the
points
which
Matt
already
mentioned
here.
So
let's
talk
about
querying.
We
already
have
data
in
the
parameters
right
now,
using
the
exporters
and
all
the
other
applications.
So
let's
see
how
we
can
get
the
data
using
the
queries,
yeah,
so
prom
QL
is
a
new
query
language
which
was
built
with
the
prometheus
and
it's
great
for
time.
Series
computations
I
will
be
showing
some
examples
and
it's
not
a
SQL
style
querying
language.
So
this
is
a
very
basic
query.
B
B
But
if
you
don't
buy
a
want
it
bird
path,
we
can
just
remove
the
buy
path
in
the
query
and
we
can
aggregate
it
or
all
the
requests
that
we
are
getting
for
the
errors
and
going
little
more
into
this.
There
is
also
histograms,
and
this
query
gives
you
the
the
what's
the
latency
of
99
percentage,
of
the
queries
of
the
requests
that
we
are
getting.
So
these
are
just
a
high-level
examples.
I'm
not
going
inside
the
queries
what
it
actually
happens.
These
are
just
examples
that
you
can
do
with
the
query
language.
B
So
we
just
give
the
query
in
the
U-
and
here
you
have
the
list
of
time
series
that
it
matches
and
the
value,
and
if
you
want
to
see
the
graph
of
a
graph
for
that,
you
can
see,
there
is
another
tab
and
when
you
just
click,
you
get
the
graph
for
the
forest
that
you
run,
and
this
is
just
ad
hoc.
It's
it's
not
persistent.
You
have
to
run
the
query
every
time
that
you
want
to
see.
So,
if
you
want
to
have
persistent
dashboards,
you
can
use
graph
on
ax.
B
You
can
have
panels
with
time
ranges
mentioned
in
them
with
the
query
that
it
has
to
run
in
the
panels
and
every
time
you
open
the
graph
owner
dashboard,
the
panels
will
be
the
same.
Obviously,
the
data
will
be
different
because
the
time
changes
yep
and
now
talking
about
the
alerting,
so
the
main
alerting
stays
inside
the
Prometheus.
Where
you
define
the
alerting
rules.
B
So
that's
where
the
alert
manager
comes
into
play.
So
the
prometheus
has
these
rules
create
handles
the
alerts
and
pushes
it
to
alert
manager
and,
as
might
say,
the
alert
manager
takes
care
of
sending
it
to
different
hooks,
like
slack
or
page
utility,
etc,
and
about
the
operational
simplicity.
Prometheus
is
a
single
binary
and
it
just
runs
on
your
localhost.
B
It
uses
the
local
storage,
it
doesn't
need
any
object
store
and
if
you
want
to
run
it
in
higher
liability
mode,
you
just
turn
to
Prometheus
with
the
same
config
and
to
and
scrape
the
same
set
of
targets
in
both
the
Prometheus
so
that,
if
one
prometheus
goes
down,
you
have
another
Prometheus
which
is
running.
Obviously,
the
local
stores
are
different
for
both
of
them
and
yeah
efficiency.
It's
very
easy
to
have
a
moderate
set
up
of
Prometheus,
which
can
take
more
than
a
million
sample
per
second
and
have
millions
of
series.
B
I
have
personally
seen
a
Prometheus
instance
that
can
manage
more
than
10
million
series
on
a
single
host
yep
and
it's
good
to
cue,
keep
few
weeks
to
months
of
data
in
the
Prometheus.
But,
as
you
know,
the
disk
is
not
reliable,
so
you
don't
want
to
run
a
Prometheus
with
a
very
high
scale
for
like
to
keep
months
of
data.
For
that
we
have
something
called
remote
storage.
Prometheus
has
an
endpoint
called
remote
write,
using
which
you
can
write
the
data
from
the
Prometheus
to
some
long-term
storage
solutions.
B
This
long-term
storage
solutions
are
not
strongly
a
part
of
froma
test.
These
are
community
projects,
for
example,
cortex
and
Thanos.
There
were
introduction
sessions
for
cortex
and
Thanos
I
hope
you
have
visited
that
if
you
wanted
the
long
term
storage,
so
I'm
not
going
in
depth
about
them
yep.
So
you
have
Shin
for
long-term
storage
and
yeah.
B
So
Matt
mentioned
about
service
discovery,
for
example
Kuban
it
is,
you
have
parts
coming
and
going
and
you
might
have
thousands
of
parts
and
you
don't
want
to
list
the
IP
and
port
of
all
those
parts
in
your
config.
So
for
that
we
have
something
called
disco
service
discovery
to
handle
this
dynamic
environment
where
it
talks
to
the
DNS
server
or
any
service
discovery
that
you
have
in
your
system
to
know
what
should
be
present.
For
example,
it
will
give
a
list
of
hundred
parts
that
should
be
present,
but
we're
using
the
alerting
rules.
B
You
can
come
to
know
which
parts
are
not
there,
for
example,
yeah
and
using
service
discovery
I,
you
come
to
know
where
to
pull
the
matrix
from
and
the
service
discovery
also
gives
some
metadata
about
the
parts
and
other
hosts
that
you
have,
which
you
can
attach
to
the
matrix
that
you
are
pulling
during
the
scraping
period.
Yep
and
Prometheus
all
already
has
built-in
services
support
for
AWS
Google,
given
it
is
DNS,
console
and
lots
of
other
stuff,
and
we
guess
more
will
be
coming
soon.
Yeah
and
yeah
I
would
like
to
conclude
here.
B
I,
don't
want
to
go
too
deep
into
all
the
Prometheus
aspects,
so
you
feel
comfortable
in
this
talk.
If
you
want
to
get
more
in
depth,
I
recommend
you
to
attend
the
DPI
session
tomorrow
or
we
have
a
Prometheus
booth
in
the
sponsor
showcase
area.
Where,
if
you
have
any
questions,
you
can
just
come
talk
to
us.
So
we
looked
at
the
dimensional
data
model
like
the
label
values.
Each
label
value
is
like
a
dimension
in
the
time
series
database.
It's
a
powerful
query
language.
It's
just
like
my
fancy
mathematics.
B
It
can
do
anything
that
you
want
with
it.
It's
simple
and
it's
efficient
yeah,
it's
very
simple
to
write.
It's
a
single
binary
and
the
service
discovery
is
what
it.
What
makes
it
beautiful.
Yep!
Thank
you.
So
we
have
yeah.
We
have
another
Prometheus
maintainer
here,
so
v3
will
be
answering
questions
and
unluckily.
We
have
just
one
mic.
So
if
you
think
you
are
going
to
ask
questions,
I
recommend
you
to
come
a
little
bit
to
the
forward.
So,
as
we
came
here,
I'll
repeat
the
questions.
Any
questions,
yep.
B
Yeah,
so
his
question
was
one
of
the
problem.
With
times
is
databases
the
write
amplification
for
SSDs
like
if
you
are
doing
too
much
writes
the
SS,
is
going
to
wear
out
yeah.
So
that
was
a
big
problem
with
Prometheus
1.0,
where
we
used
to
just
randomly
write
the
data
to
the
disk
FM,
not
wrong
and
with
Prometheus
2
dot.
Oh,
we
don't
write
the
data
as
soon
as
it
comes.
We
store
it
in
the
memory
and
to
reduce
the
write
amplification.
B
We
write
it
in
the
size
of
page
blocks
so
that
we
don't
write
the
entire
block
of
SSD
unnecessarily
and
once
we
have
written
the
like
Prometheus
has
in
memory
part
and
persistent
part,
and
once
we
write
the
data
to
the
persistent
part,
we
don't
modify
it
so
that
we
don't
have
to
write
a
lot.
So
there
is
something
called
compactions
which
takes
care
of
other
stuff
like
joining
the
blocks,
but
yeah.
We
write
it
in
chunks
of
SSD
block
so
that
we
don't
wear
out
this
as
days
yeah.
B
Yep
so
his
question
was
he
her
to
keep
mind
of
how
they'll
be
read:
load
concentrate
load
which
is
going
to
increase
the
something
like
well
yeah?
How
do
you
maintain
a
constant
write
rate?
Yeah
yeah,
a
constant
write
rate.
I
can
say
it
depends
on
the
system
capability
like
we
saw
we,
we
can
ingest
millions
of
samples
per
second.
So
if
you
want
to
increase
that,
you
need
to
have
a
faster
CPU
and
a
lot
of
RAM
so
that
it
can
hold
in
the
memory,
but
it
will
hit
a
bottleneck
somewhere.
B
B
Like
in
parameters,
we
scrape
and
end
point
multiple
times,
so
how
do
we
avoid
duplicate
values?
So
if,
at
a
time
stamp
0
you
have
a
value
of
10
and
if
I
times,
I'm
10,
you
have
a
value
of
10
again,
so
I
would
not
say
it's
a
duplicate
value.
It's
just
the
value.
That's
present
at
that
particular
times
time,
which
is
so
you
just
store
it.
So
when
you
look
at
the
history,
historical
data,
what
you
have
you
can
just
say
at
this
time
stamp
you
have
this
value
and
the
time
stamp.
B
B
D
I'm
not
exactly
sure
what
you're
fishing
for,
but
there's
I
mean
there's
cube
state
matrix,
which
is
exporting
a
whole
lot
of
kubernetes
metrics
from
your
cluster
than
the
cubelets
exports
of
the
FBI
server
exports
Jeff.
So
you
can
certainly
check
like
how
much
resources
you
have
left
the
interesting
thing
and
that
might
go
into
what
you
are
fishing
for
is
there's
the
the
auto
scaling
in
promise
in
kubernetes.
Can
there
are
adapters
we
can
use
Prometheus
to
inform
the
auto
scaling.
So
usually
you
do
I
mean
the
vanilla.
D
Auto
scaling
is
pretty
vanilla,
but
you
often
want
to
do
something
like
all
the
scale
based
on
request
on
CPU
usage
on
RAM
usage
or
like
on
a
Vittori
things
which
you
promised
this
or
is
actually
giving
you
and
then
you
can
loop
back
the
promises
or
into
this
cameo
imposition
the
auto
scaling
decisions
all
right.
So
that's
not
exactly
scheduling,
but
not
your
is
that
going
into
the
direction
you
were
fishing
for.
D
E
Yeah,
so
my
question
is
about
the
primitives.
The
data
storage,
it
looks
like
Prometheus
is
storing
holiday-themed,
disk
or
SSD.
Then
what,
if
the
the
part
of
Prometheus
be
rescheduled
already
started,
then
how
Prometheus
achieves
the
data?
Persistency
is
prometheus,
using
a
persistence
volume
or
any
other
strategy.
B
It's
a
pure
deployment
that
you
use
a
persistent
storage
so
that,
once
you
restart
the
Prometheus,
you
have
the
access
to
the
same
this
that
you
ran
on
so
I.
Don't
we
have
a
persistent
part
of
the
Prometheus
and
there
is
an
in-memory
part
in
the
Prometheus.
So
whenever
we
get
samples
from
the
hosts,
we
write
it
to
something
called
writer
headlocks,
which
is
a
very
fairly
basic
concept
in
the
databases.
B
So
we
also
store
it
in
the
memory
and
we
write
it
to
the
disk
in
the
form
of
Records
which
we
can
replay
later
and
coming
to
your
point,
we
also
write
it
in
the
blocks
of
SSDs
and
yeah.
So
if
the
Prometheus
goes
down,
it
comes
back
again.
The
persistent
part
of
the
promise
is
still
there
on
the
disk,
but
the
in-memory
part.
We
can
replay
it
using
the
write,
a
headlock
that
we
write
whenever
we
get
the
samples.
B
B
So
that
handling
doesn't
come
under
promises
exactly
you
have
to
make
sure
that
when
parameter
is
rescheduled
you
attack,
you
are
attached
to
the
same
volume
you
can
do.
If
you
are
using
Cuban,
it
is
I
think
you
can
do
it
using
the
stateful
sets
where,
if
we
restart
the
Prometheus,
you
will
have
the
same
identifiers
and
you
will
have
the
same
persistent
volume
attached
to
it.
So
I
want
to
have
the
same
storage.
If
you
stateful
sets
yep.
F
If
you're
running
really
large
clusters
with
lots
of
data,
we
said
running
multiple
Prometheus's,
we
just
do
it
in
high
availability
and
they'll
just
grab
from
the
same
thing,
so
that
doesn't
really
help
you
with
the
you
know
with
scaling
Prometheus.
Is
that
not
a
problem
in
practice?
I?
Don't
have
really
large
clusters,
so
I'm
just
curious.
If
you
can
speak
to
that
at
all,
yeah.
B
B
So
yeah,
but
in
that
case
you
will
have
a
problem.
How
will
you
have
a
global
view
of
all
the
Prometheus
together?
So
in
that
case
you
can
do
multiple
things.
One
is
something
called
Prometheus
Federation
where
you
have
where
you
wear
one
Prometheus
talks
to
another
Prometheus
like
it
runs
a
query
to
another
Prometheus
and
stores
that
data
instead.
So
if
you
have
multiple
Prometheus,
you
can
have
a
single
parameters
which
talks
to
this
Prometheus
at
a
lower
level
and
aggregates
at
a
data.
B
Were
that
that's
like
one
time,
storage
of
the
metrics,
but
the
I
mentioned
cortex
and
Thanos.
So
Thanos
has
multiple
models
which
you
can
run
independently.
So
tano's
has
a
query
or
model
module
and
you
don't
need
to
run
the
entire
Thanos
infrastructure.
So
I
can
just
run
a
single
query
model
which
will
talk
to
this
premier,
all
the
Prometheus
instances,
and
you
can
have
a
global
view
by
querying
the
query
and
model
of
the
Platanos.
So
these
are
the
two
solutions
that
I
know
of
yeah.
G
D
B
D
Saying
because
this
is
a
very
frequently
asked
question
and
some
people
formulated
as
a
statement
or
as
a
question,
they
say
Prometheus
doesn't
scale
which,
depending
on
the
perspective,
you
were
that
viewer.
That
is
actually
true,
but
this
what
Ganesh
just
said
about
the
showing
you
usually
try
to
short
a
long
dimension,
so
that
you
don't
need
that
global
view
like
the
NIV
or
Prometheus,
was
created
at
SoundCloud.
We
just
had
one
promise
your
server
Pro
team.
D
Essentially
they
could
do
whatever
they
want
with
their
premiere
server
and
it
would
like
be
big
enough
for
the
team
and
it's
it
happens
almost,
never
that
you
actually
want
to
aggregate
the
heb
or
requests
from
the
one
team
with
you
one
of
the
others.
So
you
usually
don't
have
that
problem,
so
something
I
think
to
the
current
day.
They
have
never
used
one
of
those
like
global
views,
permanent
storage,
whatever
solutions,
and
they
are
fairly
happy
with
that
and
yeah.
D
So
arguably
it
kind
of
skills
there's
another
trickery
you
can
do
when
you
actually
have
one
like.
Let's
say
you
have
10,000
instances
of
the
same
service
and
you
actually
have
to
scrape
it
all,
and
then
you
could
do
like
a
shoring
or
like
every
Prometheus
server,
scrapes
every
tenth
instance
and
then
do
some
aggregation
trickery.
But
I
really
wouldn't
go
down
that
path,
and
it's
in
in
practice.
I
thought
it's
necessary
for
99.9%
of
you.
E
D
Faq,
of
course,
so
the
original
idea
is
kind
of
old
style.
We
have
our
trusted
private
network
and
we
just
everybody,
can
scribe.
Everybody
is
all
HTTP
and,
of
course,
that's
kind
of
a
not
acceptable
anymore,
so
Prometheus
as
a
server.
It's
ironic,
because
the
Prometheus
server
has
actually
declined,
part
of
the
HTTP
request
and
the
monitored
clients,
technicolor
HTTP
servers
exposing
the
metrics.
So
the
server
is
perfectly
happy
to
just
use,
TLS
and
bigger
tokens
and
whatever,
like
all
those
things
and
kubernetes,
that's
actually
happening,
usually
right.
So
that
works
quite
nicely.
D
The
the
problem
is
more
the
part
of
the
who
exposes
metrics
or
even
the
AP
of
the
parameter
server
itself,
because
then
you
have
to
implement
and
TLS
server
with
authentication
and
everything
and
the
lame
answer
of
the
prometheus
community
was
so
far.
Yeah
just
set
up
and
reverse
proxy.
It's
not
our
our
problem
like
we
want
to
solve
one
problem
and
solve
it.
Well,
UNIX
philosophy,
but
then
people
got
upset
that
they
always
have
to
set
up
like
a
sidecars.
For
that,
like
a
good
example,
is
you
have
the
node
exporter?
D
It's
not
called
the
Nordics
for
you,
like
you,
want
to
expose
metrics,
let's
say
my
Seco
right,
so
my
cycle
runs
in
a
port
and
then
you
run
a
sidecar
for
my
sequel,
which
is
the
mystical
exporter,
and
then
it
exposes
the
metrics
in
unsecured
HTTP.
So
you
want
now,
unlike
nginx,
sidecar,
to
reverse
proxy
for
HTTP,
so
you
have
the
cycler
for
the
sidecar
and
we
even
had
an
example
at
some
table
where
we
had
where
the
monitored
thing
was
already
a
sidecar.
D
So
we
at
the
site
before
the
sidecar
for
the
sidecar,
and
then
it
gets
ridiculous.
So
luckily
there
is
an
effort.
Ongoing
whoa
want
to
implement
like
a
reference
implementation,
how
we
in
the
promises
community
one
server
sockets
to
do
TLS
and
that's
done
for
the
Nordics
model
right
now.
It's
almost
ready
to
merge,
but
that's
true
for
like
month
right
so
anyway,
this
is
the
experiment,
and
once
we
have
one
good
way
to
do
that,
we
want
to
push
it
everywhere
through
the
ecosystem,
and
then
you
can
still
not
do
get
into
your
sidecar.
A
So
one
alternative
approach-
I've
seen
certainly
all
exporters
there
is
their
efforts
being
made
to
add
like
TLS
to
the
exploration
stuff.
One
alternative
approach
I
have
seen
is:
if
you
have
many
exporters
running
on
one
machine,
you
bind
all
of
them
to
localhost
and
you
build
some
sort
of
proxy
that
aggregates
all
of
them
and
then
that
proxy
has
TLS
or
similar
and
it
will
expose
the
metrics
and
everything
on
its
own
machine.