►
From YouTube: Prometheus - Intro, Deep Dive, And Open Q+A - Goutham Veeramachaneni & Ganesh Vernekar, Grafana Labs
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Prometheus - Intro, Deep Dive, And Open Q+A - Goutham Veeramachaneni & Ganesh Vernekar, Grafana Labs
Speakers: Goutham Veeramachaneni, Ganesh Vernekar
Prometheus is the de facto standard in cloud-native metrics monitoring and beyond, in large part because Kubernetes is literally designing its custom metrics engine for Prometheus. With ever more people adopting cloud-native technologies, we will cover a mix of intro/101 content, a deeper dive into current developments, and open Q&A at the end.
A
All
right
cool,
so
what
is
Prometheus
Prometheus
is
a
metrics-based
monitoring
and
alerting
stack
when
I
say
stack.
It's
end
to
end,
as
in
you
get
the
instrumentation
libraries
to
instrument
your
applications
and
also
exporters
to
get
data
from
other
systems
that
are
not
instrumented
with
Prometheus.
A
It
also
has
the
scraping,
which
means
the
collection
and
the
storage
layer,
and
it
also
has
a
querying
alerting,
dashboarding
layer,
which
is
the
UI
and
the
way
you
interface
with
all
the
data
that
you
collect,
and
this
is
for
all
levels
of
the
stack
as
in
you
can
monitor
systems
applications
to
your
hobby
projects
to
weather.
To
anything,
it's
very
generic.
What
differentiates
it
from
the
older
monitoring
systems
is
that
it's
made
for
highly
Dynamic
Cloud
environments
as
in
when,
for
example,
communities
where
pods
come
and
go.
A
Prometheus
is
really
really
good
for
environments
like
that,
where
you
don't
have
static
hosts
but
have
pods
that
come
and
go
and
change
over
time.
Cool
Prometheus
does
not
do
any
logging
or
tracing
it's
very
focused
on
metrics.
It
does
that
one
thing,
and
does
it
really
really
well,
it
doesn't
do
any
automatic
anomaly.
Detection,
you
have
to
write
the
alerts
yourself.
We
have
a
really
powerful
query
language
as
you'll
see,
but
you
have
to
write
the
alert
queries
yourself.
You
do
not
going
to
suggest
any
anomaly
detection
or
anything.
A
It
doesn't
have
scalable
or
durable
storage.
So
what
does
that
mean?
So?
Essentially
with
Prometheus?
You
can
comfortably
store
up
to
a
few
terabytes
of
data
as
and
you
can
only
scale
to
a
single
node
and
you
can
store
you.
The
typical
attention
is
two
weeks,
but
you
can
also
store
a
few
months
of
data,
but
typically
this
is
not
for
storing
years
and
years
of
data
or
like
scaling
Beyond
a
single
node
yeah.
So
it
started
off
in
2012
at
SoundCloud.
They
were.
A
It
was
a
bunch
of
ex-googlers
who
moved
to
SoundCloud
and
they
really
missed
the
monitoring
system
that
they
were
used
to.
So
they
started
working
on
Prometheus.
They
started
adopting
it
internally
at
SoundCloud
at
scale,
and
they
finally
published
it
end
of
2015
I.
Think
a
joint
cncf
and
v1.0
was
released
in
2016.
V2.0
was
released
in
2017..
You
might
expect
we
three.org
to
come,
but
we
are
at
B
2.40
right
now
or
something
so
essentially
we're
still
on
v2.0.
It
works
really
really
well
and
yeah.
A
It
was
a
complete
rewrite
of
the
storage
engine
and
it's
kind
of
future
proof
so
coming
to
the
architecture.
So
you
have
your
applications
that
you
want
to
monitor
that
you
want
to
collect
Matrix
from
the
way
you
do
this
is
you
instrument
your
applications
with
Prometheus
or
Prometheus,
compatible
client,
libraries
and
these
client
libraries
expose
the
HTTP
server.
So
essentially,
you
need
to
expose
your
metrics
on
a
HTTP
server.
You
do
not
push
your
Matrix
to
a
central
system.
You
just
say:
oh,
if
you
hit
this
API
endpoint,
which
is
slash
Matrix.
A
You
cannot
run
a
bunch
of
queries
and
you
can
get
the
analytical
data
or
you
can
get
the
metrics
out
of
the
number
of
connections
the
size
of
the
DB
and
things
like
that
or
for
Linux.
You
can
basically
scrape
proc
the
proc
file
system
to
get
the
data
out,
so
we
have
exporters
which
you
run
alongside
these
systems.
What
these
exporters
do
is
they
just
basically
go
talk
to
the
service
in
its
own
native
language,
in
its
own
native
format,
and
take
that
Matrix
data
and
convert
it
to
Prometheus?
A
A
Now
Prometheus
needs
to
discover.
What's
what
services
exist
and
that's
where
service
Discovery
comes
into
picture,
and
this
is
something
really
really
powerful
because
suppose
you're
running
a
collection
of
services,
maybe
say
I
wanna,
you
say:
I
want
45
of
my
services
to
run.
You
have
Auto
scaling
up
that
goes
to
47
or
43.
You
have
you,
have
a
dynamic
number
of
services
running
and
now
you're
monitoring
them
with
a
push-based
system.
So
you
don't
know
when
only
41
of
them
are
running.
A
It's
really
hard
to
detect
of
one
or
two
services
are
malfunctioning,
but
with
Prometheus
it
talks
to
the
kubernetes
API
or
it
talks
to
the
service
Discovery
provider.
It
knows,
oh,
there
is
45
Services
running
and
here's
the
eyepiece
of
all
of
these
I'm
gonna
go
and
try
to
get
metrics
from
them.
If
someone
one
of
them
is
down,
we
will
know
immediately
because
Prometheus
will
fail
and
it
will
have
a
metric
that
says
this
service
is
down
and
we
can.
We
can
alert
on
it.
A
So
that's
one
of
the
advantages
of
the
push
the
pull-based
system
over
push.
So
that's
the
service
Discovery
part
of
Prometheus,
and
then
we
have
the
UI
side
of
things.
So
we
have
a
web
UI,
which
is
really
powerful.
I
still
regularly
use
it,
even
though
I
work
for
dafana
and
it
works
really
well
for
debugging
purposes
and
for
dashboarding
and
for
you
to
look
at
regular
dashboards.
A
We
recommend
using
grafana,
there's
also
a
lot
of
people
who
wrote
a
lot
of
automation
with
our
API,
where
you
just
call
Prometheus
with
fromql
and
it
gets
it
gives
you
all
the
data.
So
there's
this,
you
can
say
the
data
usage
side
of
things
and
also
we
have
alerting
so
Prometheus
has
so
you
can
write
alerts
in
Prometheus.
Prometheus
will
evaluate
these
alerts
against
the
data
in
the
Prometheus
and
then
it
sends
it
to
an
alert
manager
which
basically
detoes
and
routes
alerts.
A
A
One
thing
I
do
want
to
mention
is
the
robust
form
of
alerting
that
happens
with
Prometheus,
because
the
very
little
amount
of
dependencies
it
has
network
is
very
prone
to
break,
and
if
you
put
your
alerting
on
a
system
on
a
distributed
system
or
on
a
system
that
relies
on
networking,
it's
less
robust
than
putting
it
in
Prometheus
and
it's
as
close
to
data
as
possible.
That's
really
cool
one
really
cool
thing
about
Prometheus
yeah.
The
selling
points
of
Prometheus
are
its
dimensional
data
models.
A
So
when
the
data
model
came
out,
it
kind
of
literally
changed
the
world.
We
will
talk
about
that.
We
have
a
really
nice
and
Powerful
query
language
built
for
Prometheus
and
the
Prometheus
style
data.
We
have
a
simple
and
efficient
server,
even
though
it's
a
single
node
server
that
doesn't
scale.
Typically,
it
really
scales
vertically
on
a
single
node
and
also
our
very
powerful
service,
Discovery
mechanisms
that
we
are
constantly
adding
new
service,
Discovery
mechanisms
to
and
improving
so
talking
about
the
data
model.
What
is
the
time
series?
A
In
my
mind
time
series
is
an
identifier
for
a
list
of
timestamp
values,
for
example
the
stock
price
of
a
particular
stock.
That's
a
Time
series,
because
it's
a
timestamp
there's
a
value,
the
weather
or
temperature
in
a
particular
City,
there's
a
lot
of
different
cities
for
each
City.
There's
a
weather,
dot
I
mean
weather
for
Detroit
weather
for
Chicago
and
stuff,
like
that,
that's
a
Time
series.
But
how
do
we
identify
this?
Each
time?
Series?
A
That's
where
Prometheus
shines-
and
it
is
it-
has
this
label
based
time
series,
for
example,
HTTP
request
total
with
the
labels
that
describe
that
a
particular
thing.
So
it's
flexible,
it
doesn't
have
hierarchy.
There's
no
Dots
here,
it's
not
HTTP
request,
total
dot,
nginx,
dot,
200
or
anything.
So
there's
no
Dots
here
you
can
add
and
remove
labels
without
your
queries
and
dashboards
breaking
which
is
really
powerful
and
like
in
the
previous
model.
A
If
you're
used
to
graphite
or
stats
T,
you
can
you
have
like
HTTP
request,
total
dot,
nginx
dot,
200
or
something,
and
you
have
to
guess
what
the
200
is.
You
have
to
guess
what
engine
exists,
but
here
it's
a
very
explicit
watch.
Each
of
those
labels
mean
and
that's
really
powerful
for
the
querying.
We
have
our
own
language
called
promcule.
It's
a
functional
query
language,
it's
built
for
Prometheus
and
Prometheus
style
data.
It's
not
SQL,
which
I
think
is
a
huge
benefit.
A
It's
it's
very
easy
to
reason
with
and
once
you
get
used
to
it,
you
cannot
literally,
you
can.
You
cannot
live
without
it,
and
one
really
cool
thing
I
want
to
mention
is
today.
If
you
are
a
Google
Cloud
customer,
you
can
go
to
stackdriver
and
before
you
used
to
have
this
clunky
UI,
where
you
have
to
click
a
lot
of
things
to
understand,
but
you
can
now
use
promptql
to
query
that
data.
It's
really
nice
and
really
powerful
and
more
and
more
vendors
are
adding
support
for
it.
A
Yeah,
for
example,
let's
see
how
selecting
all
the
partitions
that
are
more
than
100
GB
capacity
that
are
not
mounted
on
root
looks
like
so.
First
we
have
the
node
file
system,
bytes
total,
which
is
symmetric.
So
if
you
select
this,
this
is
going
to
give
you
ideally
the
amount
of
disk
that
the
file
system
is
using
and
when
I
say
not
root.
You
can
say
Mount
Point,
not
equal,
slash
if
you
want
it
on
a
particular
route.
A
You
just
say:
Mount
Point
equals
that
particular
path
and
you're
going
to
get
the
data
for
that
particular
path.
Now
this
is
in
bytes
and
we
have
to
convert
it
to
GB,
so
1.
E9
1,
divided
by
one
E9
and
that's
greater
than
100,
so
it's
greater
than
100
GB,
and
it's
going
to
give
you
a
list
of
all
the
time
series
or
metrics
that
match
that
condition.
And
here
you
can
see.
Okay
on
this
particular
instance,
this
particular
Mount
point
is
taking
more
than
100
GB
Sigma,
oh
118
GB,
and
things
like
that.
A
It's
kind
of
very
easy
to
reason
with
at
least
the
easier
queries,
so
you
can
write
really
ugly
promcule,
but
it's
better
than
other
languages,
I
think
yeah
and
here's.
What
is
the
ratio
of
requests
across
all
my
service
instances?
So
you
can
say
ratio
of
Errors,
so
you
can
just
say:
hey
give
me
the
list
of
I
mean
give
me
the
rate
of
errors
and
give
divided
by
the
total.
A
It
will
give
me
the
ratio
of
error,
so
here
it's
like
2.9
percent
of
my
requests
are
errors
and
then
you
can
also
add
custom
groupings
to
it,
for
example,
give
it
to
be
by
path.
You
just
say
sum
by
path
and
you
can
Group
by
path
and
divide
by
path,
and
you
can
see
that
okay,
here's,
my
worst
performing
endpoint,
which
is
basically
about
nine
percent
of
my
requests-
are
errors
on
this
topics.
Endpoint.
A
So
you
can
write
a
query
that
says:
if
my
error
threshold
is
more
than
this
set
me
an
alert,
you
can
get
that
query
your
alertable
page,
you!
You
can
get
that
query
and
then
you
can
dive
into
it.
You
can
add
Dimensions
slice
by
Dimensions
to
understand
exactly
what
is
causing
that
and
alerting
is
very
similar.
So,
basically,
you
write
an
expression
that
matches
a
list
of
Time
series
and
whenever
there's
a
there's,
a
Time
series
there
that
alert
condition
is
matched.
A
You
can
send
yourself
an
alert,
for
example
this,
if
you
say
more
than
if
I'm,
seeing
more
than
five
percent
of
Errors,
send
me
an
alert,
but
one
really
cool
thing
is
the
four
period
here,
four
or
five
minutes.
So
you
can
have
a
some
weird
freak
networking
incident
where
you
know
you
just
have
like
a
blip
of
errors
and
then
it's
back
to
normal.
It's
fine.
It's
just
a
small,
tiny
blip
of
Errors.
You
don't
want
to
be
woken
up
in
the
middle
of
the
night.
A
For
that
you
will
want
to
be
ideally
woken
up
if
this
is
continuing
regularly,
so
you
can
configure
a
four
period
and
only
if
this
condition
is
true
for
a
particular
amount
of
time
you
get
paced
and
that's
that's
also
really
powerful.
It
reduces
the
number
of
pages
that
you
get
and
then
you
can
add
labels,
and
then
you
can
route
by
labels
and
you
can
do
a
lot
of
really
cool
things
for
alerting
yeah.
A
A
It
can
scale
to
1
million
samples,
a
second
with
tens
of
millions
of
active
series,
and
this
is
more
than
enough
for
most
organizations.
You
don't
need
a
distributed.
Storage.
You
don't
need
a
complicated
monitoring
system
to
run.
You
can
just
run
Prometheus,
which
is
single,
node,
extremely
robust
and
focus
on
your
applications.
Not
your
monitoring,
that's
really
powerful,
and
it's
really
simple,
really
simple,
to
reason
with
and
really
simple
to
operate.
A
We
also
use
the
gorilla
compression
on
our
data,
which
means
a
16
byte
like
sample,
which
is
a
time
stamp
and
value
compresses
down
to
just
one
to
two
bytes,
like
1.6
bytes,
a
sample
on
average
or
1.2
bytes
of
sample
on
average,
which
means,
even
with
tens
of
millions
of
active
series
on
a
single
node.
You
can
keep
like
two
to
three
weeks
of
data
or
even
months
of
data
without
having
to
resort
to
a
distributed
or
network
storage,
and
that's
really
powerful.
A
Some
people
keep
years
of
data
on
it,
but
they
have
you
have
to
take
regular,
backups
and
stuff,
but
it
is
possible.
There's
nothing
inherently
broken
in
Prometheus
that
you
cannot
store
more
than
this.
It's
just
the
limitation
of
disk
space.
That
is
basically
stopping
you
yeah.
So
one
thing
I
want
to
highlight
is
the
exporter
ecosystem.
Again,
Prometheus
is
a
pull-based
system.
A
It
needs
to
go,
go
and
scrape
metrics
from
data
from
things,
and
these
things
might
not
be
taught
in
the
Prometheus
protocol
and
we
have
a
huge
ecosystem
of
exporters
when
I
say
huge
I
mean
massive
like
every
time.
Somebody
is
like
how
do
I
monitor
this
with
Prometheus?
The
first
thing
that
Google
is
the
thing
Prometheus
exporter
and
I
will
always
almost
always
find
one,
including
speedtest.net
exporter,
or
like
a
router
exporter,
for
like
Fritz
box
back
in
Germany.
A
B
Yeah,
so
in
the
Deep
time
we
will
see
what
is
the
few
of
the
highlights.
What
are
new
in
Prometheus
in
the
last
one
year,
we'll
mostly
look
at
the
Prometheus
server
itself,
though
there
are
lots
of
other
projects
under
promise
which
have
done
great
stuff
and
we'll
see
what's
coming
next
recapping
some
features
of
from
KL,
which
have
been
presented
before
two
of
the
things
which
let
you
control
the
time
of
a
Time
series.
B
B
C
B
These
were
the
new
things
in
the
promql
now
before
I
talk
about
this
particular
feature.
Prometheus
has
a
feature
called
remote
write
where
you
can
configure
Prometheus
to
send
all
of
it,
its
data
that
it's
getting
to
send
it
to
a
remote
storage,
for
example,
Thanos
or
cortex
for
long
term
storage
and
in
version
2.25
Prometheus
received
the
capability
to
receive
the
data
from
the
remote
right.
So
you
could
have
a
Prometheus
remote
right
into
another
Prometheus.
B
So
this
was
added
to
make
the
remote
storage
so
easy,
and
up
to
recently
Prometheus
released
a
new
version
every
six
weeks
and
once
a
new
version
was
released,
we
rarely
ever
put
the
bug
fixes
into
older
release,
and
we
always
ask
you
to
upgrade
to
the
latest
release.
If
you
wanted
a
bug
fix.
This
was
not
really
helpful
lately
and
starting
tour
37,
which
was
released
in
July.
We
we
have
started
releasing
the
long
term
support
releases.
First,
one
is
tour
37,
which
was
released
in
July.
B
And
one
another
fallback
before
this
particular
feature
was
that
if
for
a
Time
series,
you
got
a
sample,
let's
say
for
a
timestamp
thousand
and
for
the
same
time
series
due
to
some
timing
issues.
A
new
sample
came
for
a
timestamp,
let's
say:
900.
Prometheus
would
reject
it
because
it
could
not
support
timestamps
that
came
out
of
order.
Every
new
sample
had
to
be
newer
than
the
sample
that
came
before,
but
starting
to
rot
starting
version
2.39.
B
We
added
an
experimental
support
in
Prometheus,
where
you
can
say
if
the
auto
40
sample
or
the
old
sample
that
comes
within,
let's
say
two
hours
in
this
configured
example.
We
can
ingest
it
in
Prometheus
right
now,
so
Prometheus
no
longer
will
reject
out
of
order
samples
and
you
can
configure
it
to
have
let's
say
100
year
old
sample
and
it
will
still
work
fine,
and
this
is
the
last
thing
that
I
am
going
to
talk
about
in
the
Deep
dive,
which
I
am
very
excited
about.
B
B
C
Thank
you
hi.
Thank
you
for
your
explanation,
all
of
them
and
I'm
happy
to
use
that
as
a
pro
medius
and
recently
I
found
it.
There's
the
Kuba,
Pro
middle
stats
and
ham
chart
I'm
deployed
it
to
some
secret
name
is
a
little
bit
strange.
So
I
opened
some
issue,
but
nobody
checking.
So
could
you
check
the
issues
that
the
issue
number?
Is
the
I'm?
Really
sorry
for
that?
But
the
issue
is
nobody
see
this
though
issue
number
is
the
2400
2
530
33?
So
could
you
locate
it
as
if
you
have
a
time?
C
D
E
F
He
didn't
really
have
a
lot
of
oversight
when
doing
it,
so
he
added
a
label
for
the
path
of
the
request
coming
in
the
problem.
Was
some
of
the
paths
had
unique
IDs
in
them?
This
went
to
prod.
After
about
a
week,
we
ended
up
with
a
label
that
had
a
cardinality
of
like
500
000.,
as
you
can
imagine,
Prometheus
did
not
handle
that.
Well.
Well,
actually,
I'm,
not
sure
if
Prometheus
didn't
handle
it
well,
our
cluster
didn't
handle
it
well.
Are
there
any
I,
guess
I'm
not
aware
of
it?
F
B
A
We
now
have
scrape
level
limits
where
you
can
limit
the
number
of
Series
in
a
script,
and
you
can
be
like
okay,
if
somebody's
sending
me
more
than
five
thousand
cities
or
ten
thousand
series
depends
on
the
script
conflict,
you
can
just
the
script
will
just
fail
and
then
you,
the
developers,
will
come
to
you
and
say:
I
can't
see
my
metrics
and
they
can
go
to
the
targets
page
and
see
oh
I'm,
pushing
a
lot
of
data,
probably
garbage.
So
we
have
all
those
limits
in
place:
they're
not
part
of
the
global.
A
E
G
B
So
we
have
tested
this
in
production
for
a
bunch
of
different
loads,
the
depending
on
the
rate
at
which
you
are
getting
out
of
order
samples.
There
will
be
a
bit
of
increase
in
the
CPU
and
the
memory
I,
don't
think
it
affects
a
lot.
But
again,
it
all
depends
on
the
percentage
of
samples
percentage
of
Time
series
that
are
getting
out
of
order,
samples
and
the
rate
at
which
you
are
getting
out
of
order
samples.
B
But
still
the
memory
will
be
capped
up
to
some
point,
because
once
a
time
series
gets
up
to
30
out
of
40
samples,
we
just
flush
it
to
the
disk
and
cap
the
memory
consumption
of
a
particular
time
series
yeah.
So
you
have
to
just
try
it
out
and
see
how
it
performs
in
your
environment,
so
it
it
can.
There
can
be
no
difference.
There
can
be
a
little
higher
difference,
so
it
all
depends
got
it.
Thank
you.
H
B
A
A
By
the
way,
we
also
have
our
Prometheus
quarterly
Community
call,
and
you
we
discuss
all
the
latest
and
greatest
things
there,
and
you
can
also
always
always
bring
your
questions
and
directly
talk
to
the
maintainers
and
community
members
in
the
community.
Call
this
Google
for
it.
Prometheus
Community,
call
and
you'll
find
it.
B
And
we
also
have
monthly
Dev
Summits,
where
we
discuss
all
the
development,
that's
happening
in
Prometheus
and
we
openly
take
decision
on
what
we
want
to
do
next
and
if
you
have
anything
in
your
mind
that
you
want
to
propose,
the
doc
is
public.
So
you
can
also
search
for
Prometheus,
Dev
Summits
and
take
part
in
them
every
month.
D
Hey
so
it's
been
a
while
since
I
last
checked,
I
think
it
was
like
last
month,
but
I
I
ran
into
an
issue,
and
it
was
I'll
give
some
backstory
here.
If
anyone
has
ran
istio
and
tried
to
scrape
his
geometrics,
you
know
that
the
cardinality
is
a
little
ridiculous,
especially
when
you're
doing
a
pod
monitor
with
sidecars
right.
So
in
one
in
one
particular
cluster
that
we
had,
we
run
around
to
maybe
300
clusters.
Our
setup
is,
we
have
Prometheus
and
everything
everything
remote
writes
to
mimir,
right
and
kudos
to
bemir.
D
I
haven't
been
able
to
break
it
until
this
thing
happened,
but
so
what
essentially
happened
was
it
was
it
was.
It
was
my
fault
because
I
I
misconfigured
the
scrape
config
or
the
remote
right
config,
so
I
didn't
allow
it
enough
capacity
to
write
to
write
out.
So
basically,
it
was
scraping
too
much
and
it
couldn't
keep
up
with
the
load
that
was
getting
because
my
Max
shards
wasn't
high
enough
and
it
the
I
did
some
testing
my
Max
requested
shards.
D
Is
there
a
possibility
that
we
can
input
logic
into
the
remote
right,
config
that
if
it's
out
of
date
for
a
specific
amount
of
time
to
just
don't
even
try
sending
it
just
drop
it
right
away
like
because
I
feel
like
because
the
only
solution
to
this
particular
issue
and
it's
lucky,
because
I
don't
really
care
too
much
about
the
data
on
the
pods,
so
I'm
just
using
empty
dirs.
So
what
I
did
was
I
just
ran?
D
It
ran
a
script
through
and
just
deleted
all
of
the
pods
and
had
the
staple
set
restart
up,
and
you
know
the
fresh
right
ahead
log.
But
if
I
was
running
PVS
on
there,
then
we
would
still
run
in
the
same
issue.
It'd
be
a
huge
problem.
So
again,
I
remember
there
being
an
issue
about
this.
I
haven't
checked
in
about
like
a
month
and
a
half,
but
is
there?
Is
that
being
looked
at
or
is
that
a
possibility
that
we
can
Implement
into
this.
A
Yeah
I
mean
so
first
thing
is:
you
can
have
an
alert
that
says
if
I'm
out
of
sync
for
more
than
60
seconds
page
me,
that's
first
thing
I
would
say
and
I
would
run
alerting
on
Prometheus,
not
on
memer,
because
remember
it
doesn't
get
the
data.
You
can't
alert
it's
more
robust
to
always
do
alerting
on
Prometheus.
That's
one
layer
of
things
and
yeah
I
mean
like
I,
think
you
can
do
a
few
things.
D
D
A
D
Yeah
and
another
question
too,
when
it
comes
to
the
when
it
comes
to
scrape
configs
themselves,
there's
the
re-labeling
configs
and
the
metrics
relabeling
configs,
but
there's
also
the
re,
the
remote
right,
relabeling
config,
when
it
when
it
comes
to
service,
monitors
and
pod
monitors
right,
it
would
be
nice
if,
like
we,
can
kind
of
have
like
The
Best
of
Both
Worlds
of
it,
storing
some
data
locally,
but
also
like
the
stuff
that,
like
is
really
like
heavy,
like
memory
usage
wise.
We
send
it
but
don't
store
it
locally.
D
If
that
makes
sense
kind
of
like
we
do
them,
you
do
a
metrics
relabeling
to
keep
certain
metrics,
but
then
you
would
have
to
do
like
at
a
global
level.
Stop
like
the
remote
right
for
some
metrics,
but
if
we
could
put
that
into
another
section
of
like
the
Pod
monitor,
the
service
monitor
is
like
remote
right,
relabeling
or
something
like
that.
I
think
that
would
be
really
is.
Are
there
plans
for
that,
or
is
it
just
going
to
be
set
on
the
global
level.
D
D
A
E
B
A
J
Hey
I've
heard
about
some
updates
with
prom
ql
language
server,
as
well
as
another
project
called
prom
lens
and
I
know.
Some
difficulty
that
folks
have
is
with
learning
prom
ql.
You
know,
particularly
like
with
our
developers
and
I,
was
just
wondering
if
there's
any
plans
to
like
integrate
that
with
Prometheus
UI
or
graphonics.
I
A
By
the
way,
prom
lens
was
open,
source
I
think
two
days
ago,
with
the
collaboration
from
the
problem,
Labs
I
mean
I,
don't
know
what
julius's
company's
problem
Labs
and
chronosphere,
and
we
have
full
plans
to
integrate
it
into
Prometheus.
It's
part
of
the
prom
like
it's
in
the
Prometheus
org
right
now,
and
you
should
see
it
in
the
coming
months.
A
K
A
Prometheus
is
highly
available
in
that
you
run
two
Prometheus
servers
that
don't
talk
to
each
other,
that
don't
know
anything
about
each
other
and
you
can
do
deduplication
of
alerting
at
the
alert
manager.
Level
which
is
highly
available,
so
I
would
say,
Prometheus
does
solve
the
high
availability
problem
in
its
own
way.
It's
a
it's,
a
very
workable
solution
that
works
for
most
people.
Prometheus
is
inexpensive
enough
that
you
can
run
two
Prometheus
servers
and
be
happy
about
it.
A
A
I
A
So
Ganesh
also
talked
about
the
remote
right
receiver.
You
can
push
remote
right
into
it
and
in
The
Collector
I
give
a
talk
on
Monday.
It
should
be
up
already
I
think
so.
Basically,
you
can
use
the
collector
to
write
otlp
into
the
collector
and
then
right
remote
right
into
Prometheus,
so
you
can
convert
otlp
data
into
Prometheus
and
Maps
really
really
well.
Having
said
that,
I
do
I'm,
also
part
of
the
open,
Telemetry
community
and
I
always
keep
proposing
adding
otlp
ingestion.