►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Prometheus Intro and Deep Dive - Julius Volz, PromLabs; Björn Rabenstein, Grafana Labs; Matthias Rampke, SoundCloud
As the 2nd oldest project in the CNCF, you have probably heard about Prometheus before. Nevertheless, the project maintainers will give you an introduction from the very beginning, followed by a deep dive into the exciting new features that have been released recently or are in the pipeline. You will learn about many opportunities to use Prometheus, and maybe we can even tempt you to contribute to the project yourself.
A
All
right
with
slight
delay,
welcome
to
the
prometheus
introduction.
Unfortunately,
I
am
not
julius
julius.
Couldn't
make
it
due
to
reasons
so
I'll
have
to
do
some
powerpoint
karaoke,
for
you,
my
name
is
matthias,
I'm
an
engineer
at
soundcloud.
I
let
me
make
my
notes
bigger,
so
I
can
read
them
all
right.
I
work
on
the
production
engineering
team,
which
is
the
successor
of
the
team
that
originally
built
prometheus.
A
Yes,
in
the
first
part
of
the
session,
I
will
give
a
very
brief
introduction
to
prometheus.
This
will
not
be
an
in-depth
guide.
We
have
like
30
minutes
to
get
through
everything,
so
we'll
just
be
a
teaser
a
show
of
hands
actually
who
here
is
not
yet
using
prometheus,
there's
a
few
over
here
and
there
all
right
for
the
rest
of
you.
A
This
is
going
to
be
a
very
brief
refresher
and
then
we'll
get
to
bjorn.
Who
is
going
to
walk
us
through
some
news
and
updates?
We
have
about
half
an
hour
together
with
the
delay.
We
may
not
get
to
q
a
if
you
have
any
questions.
The
prometheus
pavio
in
the
prometheus
booth
in
pavillon
2
is
always
staffed.
You
can
always
find
us
there,
and
you
can
always
ask
questions.
A
Prometheus
is
a
metrics-based
monitoring
system
at
the
time
we
created
it,
because
there
was
nothing
that
was
able
to
monitor
our
growing
zoo
of
microservices.
We
were
already
deploying
to
a
dynamic
container
based
platform
and
we
really
needed
some
more
details
than
what
nagios
was
giving
us,
which
was.
Is
it
up
right?
Prometheus
is
not
trying
to
cover
all
your
bases.
Prometheus
is
a
metric
monitoring
system.
Typically,
you
will
combine
it
with
other
systems
for
log
collection
and
tracing.
A
A
The
client,
the
ins,
the
instrumentation
in
the
client
itself,
is
very
lightweight
because
and
that's
a
big
difference
to
push-based
monitoring
systems.
Your
application
doesn't
put
anything
on
the
network
at
the
point
of
an
event
happening
at
the
point
of
an
http
request,
happening,
we're
not
trying
to
send
metrics
anywhere.
All
we're
doing
is
incrementing
some
numbers
in
memory,
so
it's
blazing
fast
on
the
hot
path
of
your
application,
then,
and
then
these
current
values
of
typically,
these
counters
are
exposed
as
an
http
endpoint
that
can
be
scraped.
Asynchronously.
A
A
Hopefully
so,
there's
a
lot
of
interesting
metrics
in
software,
that
you
don't
control
and
that
don't
have
native
prometheus
or
open
metric
support
more
and
more
things
do,
but
if
they
don't,
they
need
a
little
help
from
a
friend.
We
call
this
friendly
exporter
and
we'll
talk
a
little
bit
more
about
exporters
later
today.
So
in
the
poll
architecture,
the
prometheus
server
is
the
thing
that
reaches
out
to
all
the
things
that
you
want
to
monitor
and
in
this
example,
we
have
only
one,
but
you
can
have
as
many
as
you
want.
A
A
You
probably
want
to
look
at
your
metrics,
that's
kind
of
the
point.
So
typically
you
like
there
is
a
web
ui
in
prometheus.
I
would
really
only
recommend
it
for
exploration
and
debugging.
It
is
not
a
dashboard.
We
used
to
have
our
own
dashboard
builder,
but
then
grafana
came
around
and
we
said
all
right.
This
is
so
much
better.
Just
use
grafana
it's
the
tool
of
choice.
However,
everything
is
api
driven,
so
you
can
totally
build
your
own
automation,
your
own
visualization,
whatever
you
want
on
top
of
prometheus.
A
Finally,
of
course,
in
monitoring,
if
there
is
a
problem,
we
do
want
to
tell
someone
about
it,
and
this
is
where
alert
manager
comes
in
alert
manager
handles
grouping
alerts
together,
so
that
you're
not
overwhelmed
by
a
million
fine-grained
alerts
happening
at
once,
and
this
is
totally
a
thing
that
happens
right.
We
have
a
lot
of
alerts
where
we
just
say
all
right.
Tell
me
if
any
of
the
end
points
on
any
of
my
instances
has
an
error
rate
larger
than
10
or
1
percent.
A
You
have
a
thousand
instances
and
you
have
a
thousand
endpoints.
That's
a
million
right
that
can
happen.
Alert
manager
will,
if
you
configure
it
that
way,
group
it
back
together
and
say:
yo,
you
have
a
problem:
here's
a
million
things
that
are
wrong.
You
may
want
to
look
into
them,
but
it
doesn't
like
we
used
to
before
this.
We
used
to
have
pager
storms
where
you
would
like
in
an
incident.
You
would
have
one
person
just
sit
there
and
like
acknowledge
pagerduty,
because
there
was
so
much
coming
in.
A
A
So
often
these
timestamps
are
in
a
regular
interval,
but
they
don't
have
to
be
and
the
values
change
over
time.
So
what's
the
idea?
Oh
yeah
and
all
the
timestamps
are
just
integers.
All
the
values
are
float64s
so
far.
That
will
change
in
the
future.
But
fundamentally
this
is
what's
happening
right.
It's
a
series
of
at
this
time.
The
value
was
this
float.
A
A
A
So
in
this
example,
for
example,
you
might
be
interested
in
this
single
time
series
in
particular,
or
maybe
you
want
to
see
all
the
status
200s
of
across
all
the
paths
on
all
your
instances
right.
So
that's
why
the
path
is
an
is
a
label
here,
whereas
you
wouldn't
want
to
mix
together,
http
requests
and
cpu
time
used.
That
makes
no
sense,
so
those
two
should
have
different
names.
A
That's
the
the
rough
guideline
we
can
give
there
frequently
we
use
counters
like
this.
They
represent
the
total
number
of
events
since
some
unspecified
point
in
time.
This
can
be
the
time
that
your
instance
started.
It
doesn't
have
to
be.
It
is
absolutely
valid
to
start
your
counter
at
a
random
number
that
you
just
pulled
out
of
your
head
or
in
other
yeah.
So
there's
many
ways.
Typically,
they
start
at
zero
and
they
just
count
up
over
time
until
the
instance
restarts,
but
it
doesn't
have
to
be
this
way.
A
B
A
A
A
A
So
and
afterwards
we
are
not
necessarily
as
long
as
we
scrape
often
enough.
We
can
reconstruct
how
many
requests
happened
in
any
one
minute.
Interval
ten
minute
interval,
seven
and
a
half
minute
interval,
if
you
really
want
to
so
that's
why
we
use
these
ever
incrementing
counters,
rather
than
just
recording
the
rate
at
any
point
in
time,
we'll
see
how
that
plays
out
later.
A
A
So
this
is
just
a
sneak
peek.
The
most
important
hurdle
to
overcome
is
trying
to
think
in
sql
if
you've
done
sql
before
some
of
it
seems
familiar
and
it
seems
like
you
can
kind
of
try
and
translate
between
the
two.
Don't
that's
not
a
helpful
way
to
think
about
promql
think
about
prom
ql
as
managing
time
series
and
like
multiple
time
series
and
matching
them
up
and
working
with
them.
A
A
A
A
The
rate
function
here
is
what
looks
at
the
time
window.
This
is
what
looks
at
the
difference
between
when
the
counter
started
and
the
counter
ended.
Plus
it
does
some
magic
to
compensate
for
if
the
counter
reset.
In
the
meantime,
it
understands
that
it
understands
how
to
interpolate
values
if
we're
asking
for
a
time
window
that
doesn't
quite
align
with
when
we
scraped
so
and
just
like
the
scalar
calculation
we
just
looked
at
this
happens
individually
for
every
time
series.
A
So
we're
doing
this
one,
this
rate
separately
for
every
path
for
every
instance
for
every
status,
except
in
the
in
the
second
one,
for
every
status
here,
we're
only
doing
it
for
the
ones
that
have
status
equals
500.
So
we
don't
do
unnecessary
calculation,
but
we
do
calculate
the
rate
individually
for
each
of
them
because
they
will
have
reset
at
different
points.
In
time
right,
we
cannot,
we
can't
turn
this
around.
A
A
What
should
we
do
if
we
try
to
divide
some
metrics
that
have
labels?
So
here
we
are
summing
things
up,
but
we
are
not
summing
over
the
path
dimension,
we're
preserving
the
path
dimension,
and
then
we
are
dividing
the
two
and
then
the
division
again
happens,
one
by
one
for
the
ones
that
fell
out
of
for
the
multiple
series
that
fell
out
of
the
sum
by
we
divide
each
of
them
individually.
A
Alerting
again,
I
can't
go
into
any
of
the
details
here.
The
very
very
short
story.
Is
you
write
an
expression
like
this
one
that
only
returns
data
when
there
is
a
problem
and
if
there
is
a
problem
that
you
want
to
alert
someone
about
alerts
have
labels
and
annotations
labels,
help
alert
managers
sort
out
the
routing
annotations
help
humans
sort
out.
The
resolution.
A
A
In
practice.
You
can
actually
go
quite
a
long
way
with
prometheus
without
actually
having
multiple,
the
storage
and
collection
engines
are
quite
efficient.
We
have
many
prometheus
servers
with
millions
like
10
million
active
time
series
is
no
problem.
If
you
give
it
a
good
but
not
outrageous
amount
of
memory
and
a
million
samples
per
second
on
a
single
server.
No
problem
memory
is
really
your
limiting
factor.
You
need
to
keep
up
two
hours
worth
of
samples
in
memory.
A
A
We
have
not
found
an
upper
limit,
yet
as
much
as
your
storage
can
handle,
we
haven't
found
the
point
where
prometheus
really
breaks.
So
it's
really
just
how
big
of
a
disk
are
you
willing
to
give
it,
and
if
that's
not
enough,
there
are
related
projects
that
add
scaling
and
aggregation
across
many
prometheus
servers.
A
Circling
back
to
exporters,
not
everything
speaks
for
me
feels
yet
so
exporters
help
you
bridge
this
gap
from
other
metric
systems
that
you
may
already
be
using
from
other
system.
Specific
metrics
like
the
linux
kernel,
metrics
the
mysql,
whatever
database
you're
using
or
you
can
diy
it,
because
they
only
need
to
return
a
number
when
they
are
being
asked
over.
Http
exporters
are
much
simpler
to
write
than
you
think,
and
I
would
encourage
everyone
to
write
an
exporter
just
for
fun.
A
B
Dive
yeah,
I
hope
I'm
on
now
so
julian
was
supposed
to
be
here.
He
is
missing
for
the
same
reason
as
julius.
Now
you
can
guess
what
reason
that
is,
I'm
still
here
like
we
wanted
to
show
off
that.
For
me,
this
is
a
true
multi-stakeholder
project.
We
are
four
people
all
from
different
companies,
although
originally
myself
and
julius
and
mr
we
all
worked
for
soundcloud,
that's
kind
of
the
common
origin,
but
he's
still
at
soundcloud,
I
met
a
company
called
grafana
labs.
You
might
have
heard
about
them.
B
Julius
is
running
his
own
gig
and
julian
came
later
to
the
project
is,
I
think,
he's
the
most
active
person
in
the
roommates
project
right
now
and
he
is
running
a
new
thing.
Ollie,
it's
also
quite
interesting,
so
he
can't
make
it.
This
is
called
the
deep
dive
section.
It's
a
bit
weird
that,
in
contrast
to
the
olden
days,
we
now
have
a
deep
dive
and
intro
session
in
the
same
30
minutes,
which
is
very
short,
also
completely
different
audiences
right,
but
first
you're
all
somewhere
in
between,
and
you
actually
like
this.
B
So
I
decided
to
not
dive
deep
into
a
certain
topic,
but
just
give
you
like
a
flashlight
of
stuff
that
is
new.
Julian
is
also
giving
a
talk.
He
wanted
to
give
a
longer
version
of
this
at
prometheus
day,
but
he
couldn't
make
it
so
he
will
record
this
and
shared
by
his
blog
or
twitter
account.
So
totally
look
for
that
look
out
for
that
it
will
be
a
nicer
version
of
this.
B
This
is
essentially
just
pointers,
also
just
looking
at
more
or
less
the
last
12
months
and
only
looking
at
prometheus
prometheus,
which
is
the
prometheus
server,
but
that's
only
one
component
of
the
huge
ecosystem.
So
there's
even
more
than
that,
so
we
just
go
through
a
few
things.
In
version
2.25,
we
embraced
feature
flags.
B
This
is
actually
a
bit
longer
ago
than
a
year,
but
I
like
to
give
it
kind
of
an
honorary
mention
here,
because
feature
flags
were
probably
pretty
much
like
in
kubernetes.
We
should
have
had
this
idea
way
earlier.
It
allowed
us
to
be
like
a
bit
less
risk-averse
with
new
features.
Originally
prometheus,
half
of
the
world
is
using
it.
So
we
are
fairly
conservative
in
terms
of
stability
and
breaking
people.
So
now
we
we
have
feature
flags.
One
reason
to
have
them
is
to
be
really
explicit
about
experimental
features.
B
So
we
had
this
before
we
just
said
we
had
this
feature,
don't
use
it
in
production.
It
will
change
any
time,
but
people
just
used
it
in
production
because
they
didn't
even
know
it
was
an
experimental
feature.
So
now
you
have
to
switch
it
on
with
the
feature
flag
marks
it
really.
Clearly.
That
is
one
reason
to
use
them.
B
Also,
we
can
use
a
feature
flag
to
introduce
a
breaking
change
without
having
a
new
major
release
because
you
switch
it
on
deliberately
and
then
you
have
a
new
behavior
which
is
breaking,
but
by
default
you
don't
switch
it
on
in
this
case.
If
we
like
the
feature-
and
we
have
a
major
release
like
in
10
years
or
so
with
prometheus
3,
then
we
can
make
it
we
can
make
it
a
no-op
and
it's
just
on
forever.
B
Similarly,
if
an
experimental
feature
that
is
not
breaking
it's
fine,
we
just
make
it
the
default
and
then
the
feature
flag
becomes
a
knob
as
well,
but
it
will
still
work
right.
It
will
just
warn
you
and
then
the
final
reason
to
use
the
feature
flag
is
when
you
really
want
to
change
the
behavior
in
a
way
that
wouldn't
even
go
into
the
next
version.
It's
just
a
different
mode
of
prometheus.
If
that
feature
becomes
stable,
it
would
become
a
normal
flag
of
changing
behavior.
B
I
have
examples
for
each
of
this
in
all
the
new
features
here.
So
what's
new
impromptu
l,
these
are
they
by
now
they
look
kind
of
incremental
those
changes,
but
if
you
use
them,
it's
kind
of
great.
The
first
thing
is
here:
we
allow
negative
offsets
now
offset
allows
you
to
query
something
in
the
past.
Now
you
can
curry
stuff
in
the
future,
predicting
the
future
yay.
Of
course,
that
doesn't
work
even
for
prometheus,
but
sometimes
you
run
query
historical
careers,
like
you
want,
are
interested.
It's
like
the
temperature
perps
last
month.
B
How
did
it
compare
to
the
average
around
this
evaluation
time?
And
then
you
you,
you
kind
of
from
your
point
in
the
past.
You
look
into
the
future
and
use
the
negative
offset.
Similarly,
we
have
an
add
modifier,
where
you
can
kind
of
pin
parts
of
your
query
to
a
certain
point
in
time:
they're
even
like
helpers.
This
is
the
end
of
a
range
query
if
you're
on
a
range
query,
so
this
is
pretty
special,
but
it
allows
a
number
of
things.
B
This
query,
for
example,
finally
solves
the
surprisingly
hard
problem:
how
to
draw
the
top
five
lines
in
like
traffic
or
whatever
it's
a
bit
more
complicated.
There
is
a
link
here.
It's
also
clickable.
If
you
look
at
the
slides
in
the
pdf,
where
there's
a
blog
post
explaining
this,
this
is
busting
caches,
so
we
caches
could
assume
that
you
never
look
into
the
future,
and
so
we
thought
it
might
be
a
breaking
change,
but
in
the
end
we
decided
not
so
it
became
stable
and
it's
now
default.
B
You
can
just
use
it
in
the
newer
version.
Trigonometry
happened.
Finally,
it's
a
bit
of
a
problem
that
people
want
all
kinds
of
functions
and
we
don't
like
want
to
implement
a
million
functions
from
gl,
but
this
finally
made
it
so
you
have
sine
function,
cosine,
even
a
10
2,
which
has
two
parameters
and
we
use
the
from
qr
label.
Matching
it's
a
bit
weird,
but
if
you
really
need
this,
you
will
appreciate
it
happen.
B
2.31
this
converts
from
normal
degrees
into
radians,
which
is
the
default
and
people
use
prometheus
to
monitor
wind
turbines,
which
is
pretty
cool
present
over
time,
is
also
a
nice
little
thing
that
tells
you.
If
there
is
a
sample
at
all
in
this
time
range,
there
are
use
cases
for
that.
Most
of
you
haven't
run
into
it.
I
guess
so.
This
is
a
biggie
agent
mode.
It's
also
an
example
for
a
feature
flag
that
will
become
just
a
normal
flag
because
it
just
changes
how
prometheus
works.
There's
some
irony
matthias
mentioned
this.
B
It's
also
ironic
that
all
the
cloud
metrics
cloud
provider
in
ingest,
prometheus
metrics
now,
which
was
supposed
it,
was
created
to
kill
them
right
and
now
we
all
love
each
other
again
but
like
yeah,
so
you
can
mix
and
match,
but
if
you're
really
relying
on
your
cloud
provider,
it
feels
like
why
should
I
run
a
fully
fledged
prometheus
setup?
So
now
you
can
just
slim
it
down
with
the
agent
thing.
B
You
have
no
local
storage,
only
the
wall,
you
have
no
valuation
engine,
it
all
consumes
fewer
resources
and
it
the
only
thing
it
does.
It
sends
everything
via
remote
right
to
your
metrics
provider.
It
could
be
actually
quite
cool,
like
a
lot
of
our
customers
are
doing
this
and
I
wouldn't
do
it,
but
I'm
also
like
very
conservative,
sre
ish
anyway
remote
write,
receiver,
that's
kind
of
the
flip
side
of
that
coin.
At
some
point,
we
realized
we
need,
like
just
10
lines
of
code
to
let
prometheus
read
its
own
remote
write
protocol.
B
This
turns
prometheus
into
push-based
monitoring
system,
which
should
never
happen
right.
So
this
is.
This
is
why
it
was
a
feature
flag
initially,
because
we
were
really
like
kind
of
concerned.
People
will
abuse
this.
They
will
assume
prometheus
is
now
push-based
monitoring
system.
I
still
don't
recommend
this
for
like
production,
but
it's
just
nice.
You
want
to
just
ingest
something
that
comes
over
remote
right.
Just
in
a
toy
test,
setup
want
to
play
with
something,
or
you
have
just
a
tiny
amount
of
metrics.
Then
it
might
just
work.
B
So
it's
now
stable,
so
it's
on
by
default,
but
please
don't
try
to
ingest
humongous
amount
of
production
metrics
in
this
way.
Okay.
What
else
do
we
have?
This?
Is
that
was
great
fun.
We
had
a
year
almost
a
decade,
long
discussion
about,
should
we
expand
environmental
environment
variables
in
our
config.
Many
programs
do
that.
For
me,
this
has
this
purity
ambition
that
you
have
one
and
only
one
way
to
configure
things,
and
this
is
in
your
config
file.
B
If
you
really
want
to
just
write
a
write,
a
wrapper
script
that
will
expand
the
environment
variables
and
then
write
out
a
config
file,
but
so
many
people
wanted
it.
So
we
had
a
long,
long,
long
discussion
and
finally,
we
settled
on
it
should
be
allowed
in
the
external
label
section
and
only
there,
which
is
the
most
requested
use
case
for
that.
Personally,
I
would
have
done
it
more,
but
now
I
don't
want
to
open
up
this
kind
of
works
again,
so
it
is
still
a
feature
flag.
Why?
B
Because
somebody
might
have
had
a
dollar
curly,
brace
like
not
meant
as
an
environment
variable,
but
as
the
real
thing.
So
it
is,
strictly
speaking,
of
breaking
change
unlikely
that
somebody's
doing
this.
But
with
I
don't
know,
trillions
of
committees,
users,
it
might
happen.
So
it's
a
feature
flag
until
prometheus
3,
which
will
happen
in
2031.
I
don't
know
no
timeline.
B
Okay.
This
is
an
interesting
one
and
julian.
I
will
cut
this
really
short
because
we
are
short
of
time,
but
this
is
all
for
downstream
distribution
and
library
users,
so
normal
users
would
will
notice
nothing
of
this.
But
julian
was
really
he
did
a
lot
of
stuff
here.
That
has
a
little
twitter
storm
about
this,
and
he
will
certainly
talk
about
this
in
his
longer
version
of
this.
B
So
it's
essentially
finally,
can
you
prometheus
code
base
more
or
less
properly
as
a
go
module
you,
the
the
ui,
which
is
has
a
million
react
dependencies,
is
packaged
as
kind
of
some
pre-compiled
source
package,
so
it
can
hopefully
be
packaged
in
deviant
properly.
We
we
publish
proto
buffers
on
both.
What's
what's
his
name,
I
always
get
that
right,
don't
build
right
and
what
was
the
middle
one?
I
can't
see
this
on
my
notes
here:
plug
plugins.
B
That's
that's
the
coolest
thing
right,
so
we
have
build
time,
plug-in
system
prometheus,
binary
growth
and
growth,
because
we
support
a
huge
amount
of
service
discovery
mechanism
and
every
service
discovery
mechanism
comes
with
a
huge
client
library,
azure
gcp
whatever.
So
this
is
the
main
reason
why
the
prometheus
binary
is
big.
Most
people
don't
care,
but
some
do
so.
You
can
just
switch
this
all
off
at
build
time,
and
then
you
have
very
slim,
nice,
small
communities,
binaries
all
right.
What's
coming
very
quickly.
B
The
most
important
thing,
in
my
view,
is
the
new,
sparse
high
resolution,
histograms
very
cool
stuff.
I
also
have
worked
most
of
the
time.
It's
my
heart
and
soul
into
it,
so
I'm
clearly
biased
but
ganesh
who
also
worked
on.
This,
will
give
a
very,
very
cool
talk
later
today.
Please
all
go
there
also
alert
manager.
I
will
see
more
love
now.
The
current
maintainer
has
more
time,
and
my
colleague
josh
will
also
work
more
on.
It
gives
the
talk
here
pretty
cool
example
for
something
outside
of
prometheus.
B
If
you
wanna
look,
I
know
I
haven't
done
this
like
that.
We
have
loads
of
design
dogs
in
the
pipeline.
A
lot
will
happen.
You
can
see
this
in
our
documentation.
We
also
want
a
more
formalized
proposal
process
for
new
stuff.
Another
new
thing
it's
important
for
julian.
He
will
maintain
this.
We
we
will
do
2.37
will
be
a
long-term
support
version
where
you
get
security
updates
for
a
longer
time,
also
very
cool,
and
I
think
that
should
be
it
right
yeah.
Thank
you
very
much.