►
From YouTube: Leveraging OpenTelemetry for Your Prometheus Pipeline - Goutham Veeramachaneni, Grafana Labs
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Leveraging OpenTelemetry for Your Prometheus Pipeline - Goutham Veeramachaneni, Grafana Labs
OpenTelemetry metrics are now GA and it promises to be fully compatible with Prometheus! People are looking to use OTel to migrate from their existing monitoring system to Prometheus or use one of the many data transformation capabilities of the Collector. But trying to use it with Prometheus throws a lot of questions: 1. What are the advantages of using OTel in my Prometheus pipeline? 2. Which of the 3 Prometheus receivers and 3 Prometheus exporters do I use, and when to use which? 3. Can I replace Prometheus or the agent with the OTel Collector? 4. Can I use Prometheus instrumented applications with OTel and vice-versa? When does it make sense to rewrite your instrumentation? 5. What is the migration path I could use? Come attend this talk by Goutham Veeramachaneni for answers to the above questions and understand the different components inside OTel Metrics and how they interact with the Prometheus eco-system. The talk will dive into the common scenarios and pipelines, the gotchas and when and how to migrate from Prometheus to OTel for metrics and vice-versa.
A
So
yeah
we're
gonna
talk
about
a
bunch
about
the
collector
about
the
receivers
and
exporters,
and
specifically
I
will
talk
about
some
of
the
use
cases.
You
can
use
open,
Telemetry
and
Prometheus
together
cool
first,
let's
talk
about
Prometheus.
So
Prometheus
is
a
monitoring
and
alerting
system.
It's
metrics
based-
and
this
is
typically
the
architecture
for
Prometheus.
A
So
you
have
your
applications
and
the
exposed
metrics
on
a
HTTP
endpoint
that
Prometheus
scripts.
So
it's
a
pull-based
system
and
then
you
can
point
your
grafana
to
your
Prometheus
and
Prometheus
also
runs
alerting
and
sends
it
off
to
an
alert
manager
and
recently-
and
it's
between
more
and
more
common.
You
don't
want
your
data
just
in
Prometheus.
You
kind
of
are
also
writing
it
to
a
long-term
store
somewhere.
It
could
be
your
own
on-prem.
It
could
be
a
Cloud
solution
with
remote
right.
A
So
that's
about
Prometheus.
Let's
talk
about
open
telemetry,
so
there's
a
lot
of
text
on
this
slide,
but
the
thing
that
resonates
with
a
lot
of
people,
including
me,
is
the
like
one
of
the
main
compelling
reasons
why
open
Telemetry
exists.
Let's
say
you're
an
Enterprise
and
you
decided
to
invest
in
one
minute.
You
did
an
evaluation,
you
said
I
want
observability.
Let
me
pick
this
vendor
and
then
you
tell
your
developers:
hey
here's
the
SDK
for
this
vendor.
A
If
you
want
observability
instrument
your
applications
with
this
and
then
two
years
later,
this
vendor
gets
Acquired
and
the
prices
go
up
or
you
need
to
move.
You
found
a
really
nice
open
source
solution
and
you
want
to
move
to
this
open
source
solution.
Now,
if
you
go
to
your
developers
and
tell
them
hey
now,
you
have
to
instrument
re-instrument
your
thousands
of
lines
of
code
with
the
new
sdks.
That's
just
not
going
to
happen.
A
It's
going
to
take
a
year
at
least
to
migrate
from
vendor
one
to
vendor
two
and
this
migration
is
the
reason
why
in
many
Enterprises
there's
like
six
different
observability
vendors-
and
this
is
lock-in
and
that's
kind
of
where
open
Telemetry
steps
in
so
open
Telemetry
is
a
set
of
standards,
sdks
and
middleware.
So
the
idea
is
you
instrument
your
applications
with
open
Telemetry.
Once
you
send
open
Telemetry
data
to
the
open,
Telemetry
collector
and
from
there
on,
you
can
basically
send
to
whatever
vendor
you
want,
which
means,
if
you're
an
Enterprise.
A
If
you're
a
platform,
team
and
you're
you're
evaluating
a
new
vendor,
you
don't
need
to
change
anything
in
the
application,
so
you
don't
even
need
to
tell
the
developers
anything.
You
can
just
say:
okay
I'm,
going
to
change
the
collector
configuration
which
I
control
I'm,
going
to
send
it
to
vendor
1
and
vendor2
and
I'm
going
to
compare
performance.
Does
it
really
work?
Are
we
getting
the
same
value
out
of
it?
A
So
that's
the
kind
of
key
value
of
open,
Telemetry,
I,
think
and
today
I'm,
going
to
focus
on
the
middleware
side
of
things
which
basically
contains
the
open,
Telemetry
collector
and,
as
you
can
see
on
the
slide,
it's
a
conduit
that
allows
collecting
sources
to
destinations.
What
does
that
mean
so
zooming
into
that?
A
So
that's
the
open,
Telemetry
collector,
it's
a
big
blue
box,
but
inside
it,
so
it
has
again
in
the
lightning
top.
We
learned
that
it
has
the
receivers,
processors
and
exporters.
The
receivers
are
used
to
collect
and
receive
data.
In
this
case,
open
Telemetry
supports
a
wide
range
of
receivers,
not
just
for
metrics,
but
also
for
traces
and
logs
and
I'm.
Just
focusing
on
the
metric
side
of
things
here
and
here
it
can
ingest
otlp
format,
which
is
the
open,
Telemetry
format.
You
can
just
stats
the
cloud
watch.
A
It
can
also
scrape
metrics
from
Prometheus,
and
then
you
have
the
processors
where
you
can
modify
this
data.
Add
extra
labels
or
like
do
some
aggregations
like
run,
recording
rules,
kind
of
like
reduce
the
number
of
labels,
reduce
the
number
of
cardinality
and
once
you're
happy
with
modifying
your
data,
you
can
basically
send
it
off
to
a
bunch
of
different
sources.
Like
you
know,
you
can
send
it
off
to
all
the
vendors,
like
all
the
major
vendors
have
Integrations,
with
the
open,
Telemetry
collector.
A
So
let's
look
at
the
Prometheus
receiver,
so
this
is
different
ways.
You
can
ingest
data
from
Prometheus,
so
the
Prometheus
receiver
is
basically
just
a
wrapper
over
the
Prometheus
config.
So
if
you
have
a
Prometheus
that
you're
using
to
collect
data,
you
can
just
copy
that
yaml
out
and
paste
it
into
this
into
this
Prometheus
receiver
at
all
the
parts
remote
right,
everything
you
can
just
like
copy
and
paste
it
into
this.
We
just
support
the
script
config
and
the
global
labels.
A
So
if
you
want
to
migrate
from
an
existing
Prometheus
installation
and
use
the
collector
use
the
Prometheus
receiver,
also,
if
you
are
running
a
serious
Prometheus
workload-
and
you
want
to
monitor
it,
I
would
also
suggest
using
the
Prometheus
receiver.
It
works
really
really
well.
It
is
a
wrapper
over
the
permit,
the
same
Prometheus
creping
code.
So
it's
exactly
the
same
data
that
you
can
kind
of
scrape
with
Prometheus
and
it
works
quite
well.
There
were
a
bunch
of
compatibility
issues,
but
slowly
every
one
of
them
is
getting
fixed.
A
You
know
our
friends
at
eBay
have
also
kind
of
pointed
some
out,
some
of
those
out
and
they're
all
most
more
or
less
fixed
right
now,
and
then
we
have
the
simple
Prometheus
receiver.
Why
does
this
exist?
So
if
you
look
at
this,
this
kubernetes
service
Discovery
conflicts
and
they
look
pretty
scary.
Now,
if
you
have
a
single
application
that
you
want
to
Monitor
and
you
want
to
say,
hey,
here's
a
Prometheus,
here's
an
endpoint
that
exposes
Prometheus
I
want
metrics
from
it.
You
don't
want
to
write
a
lot
of
complicated
code.
A
You
want
to.
You
want
to
have
something
very
simple
and
that's
where
the
simple
Prometheus
receiver
comes
in,
so
you
can
basically
say:
hey
use
the
simple
Prometheus
receiver,
here's
the
end
point
and
here's
the
collection,
interval
and
there's
a
few
other
features
you
can
add,
but
that's
basically
it
now
the
on
the
flip
side.
You
cannot
do
service
Discovery
with
this
or
it
doesn't
make
sense
to
do
Services
query
with
this.
So
for
serious
Prometheus
use
cases,
I
would
still
use
the
Prometheus
receiver.
A
But
when
you
have
a
single
one-off
system
that
you
want
to
monitor,
I
would
use
the
simple
Prometheus
receiver
in
the
end,
no
matter
what
you
use
the
simple
parameters:
receiver
converts
it
it's
a
wrapper
over
the
normal
Prometheus
receiver,
so
you
get
the
same
data.
So
that's
fine
and
then
you
have
the
Prometheus
receiver.
A
So
the
reason
this
exists
is
because
for
Prometheus
for
you
to
collect
Prometheus
data,
you
need
to
scrape
a
dash
Matrix,
a
HTTP
endpoint
to
get
the
Matrix
and
not
all
services
expose
this
data.
For
example,
MySQL
doesn't
have
a
slash
Matrix
endpoint,
so
you
can.
You
have
to
run
a
different
service
that
basically
talks
to
the
MySQL
or
MySQL
API
converts
that
metrics
into
Prometheus
format
and
gives
it
for
you
to
a
script.
A
Now
this
means,
if
you
want
to
monitor
MySQL,
you
have
to
run
the
collector
the
MySQL
exporter
both-
and
this
is
a
simple
wrapper-
that
kind
of
lets
you
run
just
one
service
and
say:
hey
run.
A
sub
process
is
the
process
for
the
data.
There
are
some
security
implications
for
which
is
why
it
is
deprecated.
I
would
not
recommend
this
I
would
run
this
separate
process
normally.
A
Cool,
so
that's
the
Prometheus,
exec
receiver
and
so
I
I
mean
the
processes
that
convert
from
different
Matrix
format
to
the
Prometheus
Matrix
format.
They're
called
exporters-
and
here
you
can
see,
there's
the
Apache
exporter,
postgres
exporter,
MySQL
exporter,
funnily
enough
I'm
going
to
talk
about
exporters,
but
nothing
to
do
with
Prometheus
exporters.
I'm
going
to
talk
about
the
exporters
in
the
Prometheus
collector,
even
more
confusingly.
The
first
exporter
I'm
going
to
talk
about
is
the
Prometheus
exporter
so
yeah.
A
So
now
you
have
your
collector
you're
ingesting
a
lot
of
data,
and
you
want
to
kind
of
use
this
and
send
it
to
a
Prometheus
or
Prometheus
compatible
system.
So
there's
some
multiple
ways
to
do
this.
One
of
them
is
the
Prometheus
exporter
and
the
other
one
is
the
Prometheus
remote
right
exporter,
there's
also
the
Google
managed
Prometheus
remote
right
exporter,
which
you
can
basically
use
to
send
data
to
Google
managed
Prometheus.
There
used
to
be
when
I
proposed.
A
This
talk
the
AWS
Prometheus
remote
right
exporter,
which
needed
a
special
way
of
doing
authentication
to
send
to
AWS,
but
we
Now
supported
in
Prometheus
core,
so
that
has
been
deprecated
and
even
removed.
Funnily
enough,
I
spent
30
minutes
trying
to
find
the
commit
that
removed
it.
I
could
not.
My
git
skills
were
not
good
enough
yeah.
So
let's
talk
about
the
Prometheus
exporter,
so
the
Prometheus
exporter
basically
takes
all
your
metrics
and
exposes
them
on
a
slash,
Matrix
endpoint.
A
Now
you
can
point
your
Prometheus
at
the
open,
Telemetry,
collector
and
say
hello:
Prometheus
go
and
scrape
the
open,
Telemetry
collector,
it's
very
it's
very
simple
like
it
just
takes
all
the
metrics
converts
them
into
the
Prometheus
formats
and
puts
them
on
metrics.
It
has
a
couple
of
interesting
things.
One
is
send
time
stamps
and
Metric
expiration.
So
in
Prometheus
we
kind
of
expect
the
metric
like,
even
though
the
metric
has
not
been
updated
for
a
long
time.
We
expect
it
to
be
scraped
it.
A
Even
if
you
didn't
update
it,
we
still
want
to
scrape
it.
We
just
wanted
to
show
the
same
graph,
but
typically
when
you
do
that
forever,
you
just
metrics
the
list
of
metrics
that
you
have
grows,
so
you
have
to
control
it
with
metric
expiration
you
with
in
this
config.
It
says
you
know
after
three
hours,
if
you
don't
see
any
updates
to
this
metric,
remove
this.
The
other
thing
is
send
timestamps
true.
So
when
Prometheus
scrapes
your
application,
it
says
okay,
the
value
at
this
timestamp
that
I'm
scraping
time
dot.
A
Now
is
the
value
that
you
see
here.
Five
minutes
later,
we
promise
describes
it.
It's
like
okay
and
at
the
timestamp
five
minutes
later.
It's
also
the
value
is
the
same,
but
when
you're
collecting
data,
the
data
has
a
timestamp
when
it
comes
in
and
when
you're
exposing
it.
You
kind
of
you
have
to
decide,
should
I
use
the
original
timestamp
or
should
I
just
let
Prometheus
assign
time
dot
now
so
some
for
some
use
cases
it
makes
sense
to
send
the
timestamp
for
some
use
cases.
It's
fine
do
not
send
the
timestamps.
A
A
So
if
you
have
a
scrape
interval
of
one
minute,
you
basically-
and
you
have
like
a
total
of
hundred
thousand
series-
you
can
say
Okay
per
second
I'm,
going
to
ingest
on
average
100
000
divided
by
60.,
but
if
it's
all,
the
hundred
thousand
are
Exposed
on
one
endpoint,
there's
no
load
balancing,
there's
no
spread
of
load
there.
So
that's
one
issue.
A
The
other
issue
is
so
in
the
Prometheus
format.
Whenever
you
expose,
you
expose
the
metric
first,
you
expose
the
metric
type
and
the
metric
help
text
and
then
all
the
data
about
that
metric.
Now
you
can
have
the
same
metric
with
different
help
texts
across
different
teams
and
the
moment
you
have
something
like
that.
This
exporter
just
breaks
it
just
doesn't
work.
A
A
A
A
So
if
you
have
your
applications
and
if
you
have
Prometheus-
and
you
still
want
to
use
the
collector
in
the
middle
I
would
still
recommend
remote
writing
to
Prometheus
rather
than
using
the
slash
Matrix
endpoint
so
yeah,
that's
the
different
receivers
and
exporters
for
Prometheus
there's
some
gotchas.
When
can
sorry.
A
There's
some
gotchas
when
converting
from
otlp
to
Prometheus.
One
of
them
is
that
dots
are
not
supported
in
Prometheus.
We
just
don't
support
dots
but
otlp
supports
dots,
which
means,
when
you
have
dots,
they're,
converted
to
underscores,
which
means,
if
you,
which
means,
if
you
convert
from
otlp
to
Prometheus,
to
TLP
your
data
is
different.
A
That's
that's
kind
of
okay,
but
the
other
bigger
issue
is,
if
you
have
otlp
and
you're
ingesting
logs,
Matrix
and
traces,
your
Matrix
labels
will
be
different
from
your
logs
and
traces
labels
just
because
the
dot
is
not
supported.
So
jumping
from
one
signal
to
another,
with
Prometheus
is
going
to
be
a
little
more
difficult.
A
So
my
recommendation
don't
use
thoughts,
dots,
no
I'm
kidding,
so
we're
trying
to
I
mean
at
least
it's
in
the
it's
part
of
the
next
Dev
Summit,
to
allow
dots
in
Prometheus.
Let's
see
if
the
Prometheus
Community
accepts
it,
I
will
be
trying
to
push
for
it.
That's
one
gotcha,
the
other.
The
other
gotcha
is
that
resource
labels
are
sent
to
a
single
metric.
A
A
Ideally,
people
expect
all
of
these
resource
labels
to
end
up
in
all
the
metrics,
but
that's
not
the
case.
So
the
service,
namespace
and
service
name
label
end
up
in
the
job
resource
I
mean
in
the
job
label
for
Prometheus,
metrics
and
service
dot
instance.
Dot
ID
ends
up
in
the
instance
label,
but
every
other
resource
label
ends
up
in
a
special
metric
called
Target
info.
A
So
now,
if
you
want
to
use
resource
labels
in
your
queries,
you
need
to
do
a
really
ugly,
join
in
prom
quill,
to
kind
of
add
this
resource
label
to
the
metrics
that
are
that
have
the
same
job
and
instance
labels.
It's
not
pretty,
but
it
can
be
done
and
typically
that's.
This
is
something
that
also
confuses
people,
and
the
other
thing
is
exponential:
histograms
don't
work
yet
so
pramit
I
mean
open,
Telemetry
has
passed,
histograms
and
Prometheus
has
or
will
have
exponential
histograms.
A
A
A
So
if
you
are
looking
to
leverage
that
in
open
Telemetry,
just
a
gotcha
that
it
might
not
work
as
well
in
Prometheus
cool
going
to
some
of
the
use
cases
or
how
you
can
basically
add
The
Collector
to
the
Prometheus
pipeline,
so
one
thing
is:
if
you
have
a
Prometheus
pipeline
that
looks
like
this
and
you're
like
I
love,
open,
Telemetry,
I
want
to
adopt
the
SD
case.
I
don't
want
to
have
a
vendor
lock
in
you
can
do
that.
A
You
can
add
open,
Telemetry
sdks
to
your
application
and
every
single
open,
Telemetry
SDK
already
supports
a
slash,
Matrix
endpoint.
You
don't
need
to
change
any
of
your
pipeline.
You
can
happily
modify
your
applications
to
use
open,
Telemetry
SDK.
That
works
the
other
common
way.
I
mean
this
is
also
getting
more
and
more
common
where
you
have
Prometheus
to
just
access
the
collection
agent.
It's
doing
all
the
remote
writing
to
a
long
term
store
and
everyone
queries
and
relies
on
the
long-term
store
for
alerting,
and
you
can
use
the
open,
Telemetry
collector
here
now.
A
Why
would
you
do
this?
The
thing
is
when
you
centralize
on
a
long
term
store
you
kind
of
also
want
to
allow
developers
to
maybe
use
open
Telemetry.
So,
instead
of
just
using
Prometheus,
you
can
also
add
more
sources
and
use
open,
Telemetry
collector
to
get
Matrix
from
all
these
sources
and
write
to
this
single
Prometheus
long
term
store.
You
can't
do
that
with
Prometheus.
A
Having
said
that,
I
typically
give
a
gotcha
with
this
approach
here.
If
you
look
at
the
long
term,
store,
does
the
alerting,
which
means
your
one
collecting
data
and
then
sending
it
over
Network
to
a
distributed
system
and
then
expecting
that
distributed
system
to
do
alerting.
There
could
be
network
issues,
which
means
the
data
could
be
delayed,
which
means
your
alerts
might
not
work,
and
typically
it's
less
robust
than
asking
Prometheus
to
do
the
alerting.
Well,
it's
it's
also
very
robust,
and
a
lot
of
people
are
okay
with
that
level
of
robustness.
A
You
can
experiment,
you
can
see,
and
this
kind
of
liberates
you
to
do
some
experimentation
and
basically
lets
you
avoid
lock-in
and
I
think
that's
really
really
powerful
yeah.
This
is
something
I
want
to
mention
if
you,
in
the
end,
if
you
do
want
to
insert
the
open
Telemetry
collector
in
between
your
applications
and
Prometheus
use,
remote
right
do
not
use
metrics
for
all
the
reasons
that
I
mentioned,
and
this
is
very
important
yeah.
That's
all
the
content.
I
have
any
questions.
A
C
C
C
A
So
the
question
is
I
mean:
would
you
in
the
future
deprecate
remote
right
and
ask
people
to
use
otlp
if
the
endpoint
supports
otlps,
but
I've
already
suggested
adding
otlp
into
Prometheus
itself
and
it
was
not
accepted?
I
will
try
again,
maybe
in
the
future
if
it's
accepted
and
tested
at
scale
yeah.
Why
not?
That
would
be
awesome.
B
A
A
B
Yeah,
so
one
one
thing
you
mentioned
was
that
resources
were
added
that
label
as
sort
of
a
separate
metric
and
there's
this
part
to
enabled
resources
as
regular
labels
in
all
metrics
is
so
any
thoughts
around
that
sort
of
exploding.
The
number
of
labels
you
have,
even
though
there
would
be
sort
of
the
same,
continues
the
right,
so
they
will
probably
be
rather
compacted,
but
any
potential
issues
with
Prometheus.
If.
A
You
do
that
so
essentially,
the
the
resource
labels
could
be
a
lot
of
different
labels
and
adding
to
adding
them
to
every
single
metric
doesn't
make
sense.
However,
you
can
use
one
if
you
have
a
resource
label
that
you
really
want
to
be
part
of
the
metric.
You
can
use
the
processor
to
add
that
resource
label
to
that
metric
or
to
all
the
metrics,
but
typically
because
of
the
wide,
varied
nature
of
resource
labels.