►
From YouTube: Prometheus – What it is, What is New, and What is Coming - Richard Hartmann & Julien Pivotto
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Prometheus – What it is, What is New, and What is Coming - Richard Hartmann, Grafana Labs & Julien Pivotto, Inuits
Prometheus is a metrics-based monitoring and alerting system and also the project with the second longest tenure within the CNCF. As such you have probably heard about it by now. We will give you a short introduction to Prometheus, what it is and why it was such a big deal when it was initially released. In all those years since then, the project has only gained speed, which provides us with the opportunity to tell you about all the exciting new features that have just been released or are in the pipeline, including opportunities to contribute to the project and its wider ecosystem.
B
So
let's
talk
about
what
prometheus
is,
that
is
the
quick
101.
It's
inspired
by
google
sporkman.
It's
a
time
series
database
internally,
working
with
64
64-bit
values.
For
for
all
numerical
data.
There
is
over
a
thousand
community-created,
instrumentations
and
exporters
way
more
non-public
ones.
It
is
for
metrics
not
for
logs.
Dashboarding
is
done
by
grafana.
Even
before
I
joined
cofounder
labs,
it's
a
highly
dynamic
and
built-in
service
discovery,
which
means
that
you
have
a
lot
of
different
ways
to
put
data
or
information
about
what
systems
to
monitor
into
prometheus
quite
easily.
B
It's
built
in
with
kubernetes.
You
can
do
zone
transfer
transfers
through
dns.
You
can
have
a
firebase
service
discovery
where
you
can
put
in
your
own
stuff
from
your
own
systems
and
there's
dozens
of
other
things.
How
you
can
get
the
information
about
what
to
monitor
into
prometheus
without
any
additional
work
on
your
it's
literally
taking
it
from
your
your
operators
from
your
orchestration
from
everything
which
which
you
might
have
it
doesn't
have
a
hierarchical
data
model.
It
has
an
in-dimensional
label
set
what
that
means
is
normally.
B
If
you
have
hierarchical
models,
you
might
have
continent
country
and
customer,
and
then
you
need
to
select
your
your
data
by
customer
across
all
all
continents
and
all
of
the
sudden,
your
hierarchy.
Data
model
is
already
wrong
for
that
type
of
query.
So
by
just
attaching
labels
and
allowing
you
to
slice
and
dice
your
n-dimensional
matrix.
However,
you
want
you
can
simply
select
by
whatever
you
want
for
this.
You
are
using
prompyo,
which
is
a
functional
language
which
is
basically
doing
vector
math.
B
What
that
means
is
you
set
your
language
up
once
or
your
calculations,
and
then
you
just
toss
whatever
amount
of
data
into
into
this
into
this
calculation
and
get
the
results
set
up,
no
matter
how
much
data
comes
or
goes
away,
you
always
get
out
what
you
want.
That's
similar
to
how
weather
predictions
and
such
are
made.
This
language
is
used
for
everything
within
prometheus
for
processing,
graphing,
alerting,
exporting
everything
where
you
work
with
the
data
you
always
go
through
control
super
nice.
B
B
It's
a
pull
based
system
which
gives
you
super
nice
properties
around
asserting
that
everything
which
should
be
there
or
is
is
part
of
your
server.
Discovery
is
also
monitored,
and
you
always
know
what
the
latest
status
is
and
if
it
goes
away
or
if
it
goes
down,
you
see
this
immediately,
which
gives
you
very
nice,
alerting
properties,
very
important
concept.
B
Usually
in
prometheus
land,
every
service
should
have
its
own
metrics
endpoint,
which
is
unfamiliar
for
people
who
are
used
to
agents.
There
are
various
advantages
to
to
having
distant
metric
endpoint,
because
you
are
not
tied
with
your
versions,
and
you
don't
have
this
huge
uplift
when
you
have
an
upgrade
or
anything.
That
being
said,
we'll
be
talking
about
agents
in
a
bit,
and
we
have
super
hard
api
commitments.
B
B
What
you
can
do
with
individual
events
is
that
you
merge
them
into
counters
or
histograms,
which
is
something
which
is
also
done
extremely
often
in
the
networking
scene,
which
had
a
lot
of
those.
What
is
now
cloud
native
scaling
issues
like
two
decades
ago,
where
you
have
this
super
cheap
way
of
of
compressing
data
along
the
axis
which
you
care
about
and
then
emitting
this,
this
distilled
information
about
whatever
you're
looking
at
it's
super
highly
efficiency
for
both
storage
and
transmitting
yeah.
B
B
I
even
know
people
who
literally
print
f
in
their
c
code
and
then
just
dump
this
file
on
on
the
web
endpoint,
and
that
is
their
prometheus
integration
and
it
works
like
we
have
all
the
tooling
and
such
to
make
this
nicer.
But
if
you
so
choose
you
can
literally
print
f
or
even
just
echo
in
your
shell
script.
It
is
really
easy.
B
One
of
the
indirect
effects
of
this
is
that,
while
communities
and
prometheus
were
created
completely
distinctly
of
each
other,
they
have
inherently
been
designed
for
each
other
and
they're,
also
written
with
each
other
in
mind.
So
we
have
people
who
are
on
sick
instrumentation
at
kubernetes
and
they're,
also
on
premises
team,
which
already
tells
you
how
much
coordination
goes
on
between
the
two
projects
and
the
one
thing
which
is
actually
recommended
by
kubernetes
to
to
monitor.
B
Kubernetes
itself
is
prometheus
and
they
put
insane
amounts
of
work
into
cube
state
metrics
to
make
this
super
high
efficient
to
get
all
this
data
out
of
kubernetes
on
the
just
instance
of
monolithic,
prometheus
level,
there's
long
term
storages
and
such.
But
if
you
run
a
single
prometheus,
you
can
get
more
than
2.5
million
samples
per
second
an
instance
we've
seen
more
than
60
000
samples
per
second
in
corn.
B
B
It
started
with
scaling,
ingestors
and
curious
and
then
took
the
code
off
of
thanos
to
scale
a
storage
horizontally
if
you're
guessing
that
thanos
is
playing
to
do
the
same.
With
the
cortex
scaling
code
for
ingestion
clearing,
you
would
be
right
as
a
pipe
dream
of
myself.
I
would
like
to
see
core
thanos
at
some
point.
A
What's
new,
so
yes,
so
now
we
will
see
what
you
have
missed
if
you
have
not
upgraded
promoters
in
the
last
year
and
we
see
a
lot
of
users
actually
not
updating
their
fund
automatic
systems.
But
let's
have
a
look.
The
first
point
is
service
discoveries,
so
in
the
last
year,
so
it
means
like.
Since
I
think
july
last
year
we
have
added
five
new
service
discoveries
to
primitives,
which
brings
the
number
of
service
queries
by
french's
like
to
13
or
something
like
that.
A
We
have
added
digital
oceans,
scalway
ethner,
eureka,
docker
and-
and
this
is
only
the
beginning,
since
we
know
that
we
are
getting
at
least
three
more
new
service
calories
in
the
very
really
next
release
of
primitives,
and
there
is
even
more
to
come
from
the
community.
This
enables
you
to
use
practice
in
a
lot
of
new
use
cases
as
well.
A
Then
you
have
tls
and
basic
authentication.
So,
while
promoting
has
always
been
able
to
scrape
using
tls
and
basic
authentication
the
targets
now,
it
is
also
able
to
export
its
metrics
and
to
export
its
interfacing,
tls
and
basic
authentication,
which
means
that
you
don't
no
longer
need
a
previous
proxy.
If
you
want
to
secure
your
promise
proxy.
A
A
Chrome
ql
also
has
seen
a
lot
of
changes
in
the
last
year.
First
of
all,
we
have
including
new
functions,
some
of
them,
which
were
really
like
really
wanted
by
the
community
like
last
overtime,
which
enables
you
to
take
the
latest
sample
over
a
certain
range.
We
also
have
some
new
features
to
write
better
queries
and
to
get
more
insightful
data
like
the
at
modifier
and
promql
it's
disabled
by
default,
because
it
can
break
some
assumptions
that
some
caching
proxy
might
have
about
the
prom
ql.
A
The
next
feature
is
like
the
negative
offset.
So
when
you
are
exploring
your
data
informatives,
you
can
decide
to
apply
an
offset
to
your
data
to
say,
hey,
I
want
to
see
the
data,
but
like
one
hour
back
now,
you
can
also
do
that,
like
one
hour
up
front
which
will
help
you
debug
and
deep
dive
into
your
matrix
a
lot
more
easily,
and
then
we
have
also
composite
duration,
which
have
landed,
which
means
that
you
no
longer
need
to
think
about.
Like
oh,
oh,
do
I
write
1
hour,
30
minutes.
A
A
A
It
is
a
different
way
to
share
metrics
on
the
federation
which
exists
in
primitives
because,
like
the
remote
right
also
enables
you
to
pass
on
like
the
steel,
markers
and
a
lot
of
other
information
and
basically
like
what
it
enables
you
is
like
if
you
want
to
run
a
promise
on
the
edge,
if
you
ever
use
cases
that
requires
you
like
a
more
like
push
model,
while
still
keeping
all
the
promises
of
formatives,
you
can
now
do
that.
So
community
still
remains.
A
The
next-
and
this
is
one
of
the
most
weighted
features
that
we
have
seen
and
one
of
the
most
exciting
thing.
It
is
the
exemplars.
So
basically,
what
is
an
exemplar?
An
example
is
a
way
to
attach
an
external
data
to
a
matrix
set,
which
means
that
next
to
your
metric,
you
can
now
have
like,
for
example,
a
trace
id.
This
is
the
more
common
use
case,
so
it
enables
you
like
when
you
have,
when
you
have
an
actual
alert
or
a
dashboard,
you
can
directly
see
an
example
of
a
slow
metric.
A
And
last,
let's
finish
with
the
alert
manager,
so
we
have
two
new
features
coming
to
the
art
manager,
one
of
them,
which
is
time-based
muting,
which
means
that
you
can
decide
now
directly
in
the
act
manager
to
decide
that
some
teams
should
not
receive
somewhere
else
during
the
weekend
or
out
of
business
hours,
and
you
can
control
that
for
each
route.
So
this
is
like
a
nice
addition.
That's
required
before
that,
a
lot
of
stuff
happening
in
prom,
ql
or
in
the
promises
upfront.
A
Now
you
can
do
that
directly
in
the
alert
manager,
and
the
second
feature
is
the
negative
matches.
So
now
you
can
match
you
can
create
the
silence
that
will
match
the
errors
that
do
not
match
certain
labels.
So
you
can
decide,
for
example,
to
put
everything
but
production
in
silence.
You
can
decide
to
put
everything,
but
something
in
silence,
which
is
like
also
going
to
be
helpful
for
the
people
who
actually
operate
and
are
on
call
with
the
earth
merger.
B
That
has
several
different
dimensions.
The
maybe
most
obvious
one
is
that
historically,
as
I
said
initially,
we
have
treated
even
experimental
features
and
interfaces
as
functionally
stable
and
and
immutable,
which
is
great
for
for
for
stability,
but,
as
julian
just
said
it,
it
would
not
enable
certain
use
cases
like,
for
example,
if
you
have
caching
proxies-
and
you
know
that
you
have
caches
in
before
in
front
of
you
and
you
don't
want
to
ever
break
them.
B
We
want
to
open
up
more
more
flexibility,
more
innovation
or,
in
the
case
of
remote,
read,
write
where
we
literally
just
run
the
experiment
as
the
first
version.
So
we
have
this
stable
basis
and
then
we
can
innovate.
On
top
of
this,
a
lot
of
our
old
assumptions
are
being
revisited
and
we
are
deliberately
enabling
more
use
cases.
B
Read
write
are
totally
valid
use
cases,
and
so
we
are
trying
to
enable
more
and
more
of
those
use
cases
to
actually
enable
people
to
to
operate
with
the
stuff
within
the
prometheus
main
org
and
not
spreading
this
out
to
the
wider
community
and
as
such,
creating
differences
in
in
different
approaches
by
upstreaming
all
of
this
or
re-upstreaming.
In
all
of
this,
we
hope
to
just
make
it
easier
to
reuse
the
code
which,
which
we
have
and
to
just
yeah,
make
it
easier
to
use.
B
B
B
We
already
have
two
specifications
out.
This
is
one
open
matrix,
which
is
the
exposition
format,
as
you
saw
earlier,
and
the
prometheus
remote
write
specification,
which
is
how
you
can
bulk
bug,
push
data
from,
for
example,
an
agent
or
a
prometheus
server
to
long-term
storages
or
in
certain
pipelines
where
you
can
then
mangle
your
data
and
and
put
it
onwards.
The
nice
thing
about,
starting
with
those
two.
B
It's
basically
once
the
interface
to
get
any
data
out
of
systems
into
a
prometheus,
compatible
system
and
then,
at
the
second
step,
once
it
is
already
in
a
prometheus,
compatible
system
emitting
all
that
data
to
whatever
the
consumer,
long-term
storage
or
whatever.
Is
that
also
automatically
covers
both
pull
and
push?
B
We
have
a
variety
of
test
suites
already
we
have
a
prom
compliance
test
suite.
We
have
a
remote
write
compliance
test,
suite
an
open,
metrics
compliance
test
street.
We
are
thinking
about
having
psdb
and
data
correctness
test
suites
as
well.
We
might
even
have
more.
That
remains
to
be
seen.
It's
also
a
little
bit
of
how
and
who
wants
to
have
what?
Basically,
just
wherever
we
see
usage
of
aspects
of
prometheus,
we
want
to
just
make
sure
that
all
of
this
is
is
done
in
a
compatible
format.
B
This
thing
is
compliant
with
or
not
compliant
with
yeah,
and
also
we'll
be
working
with
scenes.
You
have
to
actually
have
a
mark
of
approval
or
some
such
the
calls,
with
the
lawyers
and
everything
and
with
like
the
logo
and
such
that's
still
outstanding
and
coming
to
to
all
of
those
things
which
are
coming
and
there's
lots
more
in
in
those
documents
which
are
also
all
links.
So
you
can
just
click
them
through
in
the
slides.
B
Everything
which
we
do
is,
as
of
this
year
recorded
and
open
to
join.
We
already
always
publish
like
our
meaning
notes
and
everything,
but
it
was
a
remnants
of
this
or
being
in
person
that
we
didn't
truly
realize
that
we
could
just
make
all
of
this
online
and
public
to
join
and
everything.
We
have
always
invited
people
to
the
in-person
things,
but
that
doesn't
scale
whereas
doing
an
online
supervised
case.
We
must
record
it
and
and
publish
it,
but
also
it
is
open
to
join.