►
From YouTube: Deep Linking Metrics and Traces with OpenTelemetry, OpenMetrics and M3 - Rob Skillington
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Linking Metrics and Traces with OpenTelemetry, OpenMetrics and M3 - Rob Skillington, Chronosphere
Metrics and traces are two pillars of Observability and are often used in a complementary fashion. Metrics can give you a high level view of application’s responses and performance and tracing can give you a detailed view of requests through applications. Often when using metrics in graphs or alerts you want be able to jump to an example of a request represented by a given metric datapoint which is difficult to do today. In this talk we show an example of this using an OpenTelemetry exporter to publish trace IDs as exemplars using the OpenMetrics exposition format. We then walk through configuring Jaeger as a tracing backend and M3 as a metrics backend to store the trace ID alongside a datapoint. We show how it is then possible to go from a metrics graph that visualizes the latency of your application to a trace that fell into a latency bucket using the deep link of the trace ID.
https://sched.co/UaXX
B
So
a
little
bit
about
myself,
I'm
the
CTO
at
a
startup
called
chronosphere.
Previously
it
was
the
m3
and
n
30
B
technical
lead
at
uber,
and
these
are
open
source
metrics
platforms
that
are
compatible
with
Prometheus
is
a
remote
storage,
back-end
and
I'm
gonna
open
metric,
a
contributor
which
is
a
CNC
F
project
and
I've,
been
on
that
committee
for
two
years
now.
So
I
wanted
to
talk
to
you
about
the
state
of
monitoring
a
little
bit
just
to
cover
the
context
of
logs
metrics
traces.
B
They've
been
made
more
useful,
of
course,
with
the
advent
of
fast
log
search,
whether
that's
using
indexing
or
or
not,
they're,
usually
not
structured.
Much.
Some
have
had
more
success
here
at
structuring
their
logs
than
others,
and
that
essentially
comes
down
to
at
your
organization.
You
know
what
kind
of
logging
frameworks
you're
using
what
kind
of
metadata
you're
attached
what
kind
of
restrictions
you
have
on
your
log
messages,
but
at
the
end
of
the
day,
still
a
lot
of
us
tend
to
have
a
significant
amount
of
uncertain,
unstructured
components
to
most
log
messages.
B
So
log
output
from
a
process
can
all,
of
course
be
straight
into
one
place
and
collected
somewhere
and
then,
of
course,
you
can
do
your
slicing
and
dicing
on
it.
But
fundamentally,
there's
usually
some
unstructured
nough
stew
parts
of
your
logs.
So
now
I
want
to
talk
to
you
a
little
bit
about
metrics.
So
you
know,
metrics
have
also
been
around
forever,
as
well.
They're,
usually
a
little
bit
more
structured
than
logs.
B
They
basically
because
of
this
big,
become
faster
to
query
and
aggregate
for
specific
queries
that
you
that
you
know
ahead
of
time
and
with
lots
of
log
data.
You
know
if
you're
socially
frequently
frequently
searching
that
and
aggregating
log
mess,
roar
log
messages
and
visualizing
them.
They
tend
to
be
a
little
bit
slower
at
that,
because
you
know
the
very
data
structures
that
that
they
are
are
slower
to
aggregate
by
the
collection
time
and
also
slower
to
query
and
aggregate
at
query
time.
B
B
The
the
last
two
three
or
four
years
have
has
seen
an
acceleration
or
people
interested
in
this
topic,
and
you
know
I
wanted
to
share
some
numbers
here
that
were
presented
at
prompt
con
just
two
weeks
ago,
actually
about
the
install
base
of
Prometheus,
so
the
number
of
Prometheus
instances
being
used
in
Griffin
ER
has
grown
four
to
5x
year
of
a
year.
For
the
past
four
years
last
year,
in
2018
54,000
people
had
a
Prometheus
datasource
installed
in
their
governor
this
year.
B
Two
hundred
forty
two
thousand,
which
is,
is
incredibly
huge,
huge
amount
of
growth.
So
you
know
talking
a
little
bit
about
tracing
what
that
is.
This
is
just
one
actually
pretty
poor
and
not
very
interesting
trace,
but
it
this
is
an
example
of
what
tracing
his
it's
essentially,
the
way
of
looking
at
event,
data
broken
down
by
the
operations
that
your
code
paths
are
taking.
So,
whether
that's
in
a
you
know
a
model.
B
Or
a
micro
service
architecture,
it
essentially
has
parallel
operation
data
stitched
together
and
relate
into
each
other
visually
with
relevant
timing
information.
So
you
can
see
on
the
wall
clock.
You
know
how
long
things
are
taking
executing
certain
requests
and
response
flows
and
then,
of
course,
yeah.
This
is
even
a
worse
measure,
perhaps
of
popularity
than
the
Google
trend
search,
but
there's
nothing
like
Griffin
or
stats
for
for
any
kind
of
tracing
projects
out
there.
B
One
perspective
on
on
these
three
signals
you
know
you
could
think
of
logs
is
essentially
when
they
are
all
string
to
one
place
as
a
ton
of
different
unique
events,
essentially
and
when
they're
stream
to
one
place,
it's
they're
all
bundled
together
and,
yes,
you
can
aggregate
and
slice
and
dice
in
them
using
a
log
back-end,
but
this
is
how
at
least
they're
structured
internally
and
strained
to
get
to
your
back-end,
so
a
perspective
on
traces.
These
are
submitted
and
basically
assembled
as
linked
events.
B
So
events
that
might
have
happened
from
different
applications
in
your
in
a
microservice
architecture
or
different
components
in
monolith,
architecture,
all
stitched
together
and
and
how
they're
related
together
is
intrinsically
captured
as
part
of
a
trace
and
then,
of
course,
metrics.
You
know
is
basically
a
report
or
a
categorized
view
of
the
events
coming
out
of
applications,
and
so
it's
it's
really
like
with
taking
a
spreadsheet
and
then
creating
a
graph
from
the
raw
events
that
you
capture
in
a
spreadsheet.
B
That's
that's
really
what
metrics
has
been
you
know
typically
used
for,
and
that's
that's
why?
Obviously
it's
it's
made
in
a
format
that
can
be
easily
aggregated
at
collection
and
query
time,
and
it's
much
better
for
volumetric
analysis,
because
you
take
essentially,
you
know
samples
at
at
periods
in
times,
rather
than
just
collecting
every
single
thing.
The
drawback,
of
course,
though,
is
that
you
don't
have
the
event
data
anymore.
B
So
when
you
need
to
root
cause
an
issue,
you
need
to
drill
down
further,
you
need
to
see
maybe
request
response
details
between
two
micro-services
in
your
architecture
doesn't
have
that,
but
again
there
many
reasons
why
it's
it's
very
they're,
very
useful,
so
I
wanted
to
talk
to
you
a
little
bit
today
now
about
what
does
combining
logs,
metrics
and
traces
today.
Look
like
so
increasingly
more
observability
platforms
are
providing
at
least
two
signals,
perhaps
all
three
four
four
observability
for
monitoring
and
for
instrumenting
your
systems.
So
here
we
see
Prometheus
as
well
as
Lukie.
B
You
know
this
is
basically
all
two
projects
that
you
can
use
in
the
graph
on
a
UI
to
basically
view
metrics
and
traces,
sorry,
metrics
and
logs
rather,
and
then,
if
we
talk
a
little
bit
about
what
the
integration
looks.
Like
with
metrics
and
traces
today,
there
usually
is
that
there's
a
few
products
out
there
that
do
this,
but
they're
all
basically
using
some
form
of
matching
of
metric
metadata,
basically
like
labels.
B
And
you
know
if
you
do,
if
you
do
a
if
you
have
a
lot
of
very
good
structure
internally
to
how
you
collect
metrics
and
traces,
and
you
do
a
stellar
job
at
basically
keeping
these
systems
configured
very.
Similarly,
in
the
the
tags
and
labels
that
they
use,
then
you
can,
then
you
can
basically
narrow
down
the
search
space
that
you're
looking
for
and
usually
you'll,
find
a
representative
trace
some
of
the
time,
depending
on
your
sampling
strategy,
of
course,
and
we'll
talk
more
about
that
in
a
minute.
So
you
know
this.
B
This
is
a
this
works
for
again,
if
you're,
basically
very
meticulous
and
how
you
can
figure
everything
if
you
sample
it
a
hundred
percent
and
basically
have
made
sure
that
this
is
going
to
work.
Well,
unfortunately,
you
know
that's
not
the
real
world
and
a
lot
of
the
time
most
of
the
useful
graphs
that
you're
looking
at
in
a
metric
dashboard
as
well,
has
some
kind
of
aggregate
function
applied
which
actually
will
drop
the
metadata.
So
the
labels
for
Prometheus
metrics
and
the
context
then
is
basically
lost.
B
B
So
it's
it's
a
lot
harder
to
go
from
a
metric
graph
straight
to
a
trace
and
again,
of
course,
it's
only
magical
when
you
store
every
trace,
otherwise
you
might
just
be
jumping
to
the
the
tracing
UI
and
there's
just
nothing
there,
because
the
trace
for
that
data
point
wasn't
sampled
and
you
know
one
thing.
That's
interesting,
of
course
here
is
that
actually
saman
from
slack
on
Monday
gave
a
talk
and
the
observability
practitioners
summit.
B
B
If
you're,
looking
at
a
500
internal
server
error
that
happened
on
a
specific
HTTP
route
that
has
very
low
frequency
traffic,
then
you
you,
you
want
to
be
able
to
look
at
that
exact
issue
and
you
want
to
have
high
confidence,
is
going
to
be
traced
there
to
be
able
to
go
directly
from
a
graph
to
a
trace.
So
that
brings
us
to
deep
linking
metrics
and
traces.
So,
essentially,
you
know
what
deep
linking
metrics
and
traces
are.
B
So
we
first
we're
going
to
show
a
Jager
example:
application,
sorry,
the
Jager
example
application
and
open
source.
This
has
been
modified
to
basically
use
the
the
new
open,
metrics
x
position,
format,
which
is
a
Prometheus
extended
metric
disposition
format
with
support
for
exemplars,
and
then
we're
going
to
show
you
how
some
of
the
changes
we're
trying
to
upstream
to
grow
finer
to
to
leverage
the
deep
links
that
we've
made
between
metrics
and
traces.
The
first.
B
This
is
our
Jager
demo
application,
it's
basically
generating
a
whole
bunch
of
synthetic
transactions
which
you're
simulating
requests
and
and
basically
generating
metrics
from
that
as
well
as
traces.
And
then,
if
we
jump
over
base
to
our
our
metrics
here
in
Griffin
er,
which
are
coming
back
from
from
m3
actually
so
we're
using
Prometheus
to
scrape
these
metrics
and
then
Prometheus
is
remember.
Writing
these
metrics
to
m3
with
the
exemplars
attached
all
the
way
from
the
emit
open,
metrics
exposition
format.
To
the
back
end.
B
We
can
see
here
that
trace
IDs
show
up
next
to
data
points
in
our
graphs
now,
and
so,
if
we
just
like
silo
out
the
500
internal
server
errors
that
that
are
occurring
in
some
of
these
simulated
requests,
we
can
see
that
we
can
just
go
directly
from
one
of
these
errors
in
the
graph
to
a
representative
trace.
And
then
you
know
if
we
want
to
try
to
drill
down
into
this
issue,
say:
we've
been
paged
we've
just
looked
at
our
graph,
we've
clicked
on
a
on
a
trace.
B
B
So
that's
kind
of
basically
the
crux
of
what
we're
doing
here,
we're
just
enabling
the
ability
to
go
from
a
single
jump
from
a
metric
dashboard
and
a
single
data
point
exactly
directly
to
a
directly
to
a
trace,
and
so
you
know
I
wanted
to
explain
a
little
bit
more
about
this
and
then
of
course,
kind
of
work
through
step
through
piece
by
piece.
What
how
we
receiving
this.
So
this
is
essentially
the
picture
of
all
the
demo
and
then
like
what
we're,
what
we're
doing
so.
We
have
an
application.
B
We're
using
open
telemetry,
which
is
a
instrumentation
SDK,
essentially
open
telemetry,
makes
it
a
little
bit
easier
to
collect
metrics
traces,
logs
and
export
them
to
a
variety
of
different
supported
backends.
So
you
know
whether
that's
a
vendor,
whether
that's
Prometheus
and
whether
that's
Jaeger
for
traces,
it
doesn't
really
matter.
Do
you
write
the
code
once
and
it
exports
to
these
different
formats
and
and
backends,
and
so
I
wanted
to
dive
in
a
little
bit
more
into
open
telemetry.
So
you
can
see
basically
here.
B
If
you're
familiar
with
the
Prometheus
exposition
format,
it
might
look
very
familiar
and
basically
exemplars
here
are
captured
as
a
trailing
comment
next
to
a
metric
value.
So
essentially
here
we
have
some
counters
for
HTTP
requests,
total
broken
down
by
different
status
codes,
and
we
also
have
some
histogram
histogram
values
that
were
also
tracking
and
each
one
of
these
has
a
corresponding
trace
ID.
B
Essentially,
we
are
collecting
the
the
metrics
and
traces
together,
because
we
do
that
we
are
able
to
associate
the
trace
ID
to
the
metrics
that
gets
basically
transmitted
to
Prometheus,
using
the
open,
metrics
exposition
format,
and
then
Prometheus
is
essentially
taking
that
exemplars
during
added
memory
and
then
sending
it
can
then,
of
course,
send
it
on
to
a
a
back
end
and
then,
in
this
example,
we're
gonna
use.
You
know
we
are
using
a
3
DB
here
and
3.
Db
is
essentially
an
open
source
project,
as
I
said,
which
came
out
of
Heber.
B
It
was
a.
It
was
a
basically
the
core
back
end
for
the
metrics
platform
at
ruber
I'm
and
is
you
know,
Prometheus
from
my
storage
compatible
back-end
and
it
it
kind
of
howls
more
than
10
billion
metrics,
said
uber
and
ran
across
thousands
of
machines
comprising
2
to
3%
of
the
computer
footprint
at
re-roof.
So
we
use
it.
We
used
it
at
over
for
a
global
view
of
metrics
data
across
regions.
B
So
diving
it
a
little
bit
more
to
how
Prometheus
and
3
work
together
here.
Essentially
Prometheus
scrapes,
the
exemplar
and
as
I
mentioned,
it
will
keep
it
locally
in
memory.
So
you
can
kind
of
have
that
same
experience
that
you
saw
in
the
dashboard
just
using
Prometheus
loom
and
then
in
this
demo
we're
writing
it
to
m3,
which
actually
stores
it
in
durable
storage.
B
Next,
before
you
write
next
year
metric
for
whatever
retention
period,
you've
configured
in
through
your
and
3db
to
store
that
exists
or
metrics
for
so
here
you
can
see
that
a
single
storage
really
a
single
stored
value
bit
packs
in
the
m3,
DB,
column,
format
and
basically
time
stamp,
has
the
sorry.
The
first
part
of
this
component
is
the
timestamp
Delta
Delta
bits.
The
second
part
is
the
float64
value,
deltas
or
bits,
and
then
third
part
is
exactly
the
exemplar
bits
now
trace.
B
Id
has
relatively
high
entropy
because
it's
usually
based
on
a
UUID
and
uu
IDs
I'm,
providing
you
with
high
entropy,
so
there's
no
collisions.
However,
there
are
certain
things
you
can
do
to
try
and
compress
this
a
little
bit
better
than
then
you
might
might
think,
but
that's
a
topic
for
another
day.
So.
C
B
So
you
know
when
you're
talking
about
applications
or
services,
maybe
not
in
a
single
container
but
across
many
containers
that
process
thousands
or,
if
not,
hundreds,
of
thousands
of
requests
in
a
given
time
window.
You
know,
especially
if
you're,
basically
collecting
data
points
at
a
60
second
resolution,
but
you
can
also
obviously
step
that
down
to
lower
as
well.
This
is
relatively
efficient
because
you
have
on
the
metrics
back
inside
because
you
have
a
single
data
point
stored
with
a
single
trace
ID
correlated
to
that
data
point.
B
So
it
turns
out
that
using
metrics
aggregation
is
actually
a
really.
It
has
a
lot
of
upsides,
basically
for
keeping
traces
that
are
interesting
to
the
human.
Firstly,
it
aggregates
them
across
time.
So
basically,
it's
going
to
make
sure
that
you
consistently
get
a
certain
amount
of
traces
at
a
useful
time.
Interval
that's
meaningful
to
you.
B
Metric
tags
are
also
great
capturing,
unique
combinations
of
events.
So
you
know
you
already
break
up
your
metrics
by
era.
Success
era,
status
code,
the
latency
bucket
that
a
data
point
is
falling
into
sorry.
A
request
is
falling
into
for
a
representative
data
point.
This
also
ensures
basically
unique
combinations
of
types
of
traces,
because
you
know
a
lot
of
the
sampling
today,
head
base,
sampling
and
adaptive
sampling
and
tracing
is
all
based
on
as
soon
as
you
start
a
trace.
The
incoming
data
of
the
request.
B
So,
basically,
you
can
try
to
categorize
and
take
one
of
each
type,
but
that
type
can
only
be
calculated
using
information
at
the
beginning
of
a
trace.
So
you
know
what
I
should
peer
out
or
what
services
are,
but
it
can't
use
things
like
how
long
that
request
took
or
how.
If
that
request
like
served,
an
error
and
the
reason
it
can't
is
because
it
has
to
decide
whether
it's
going
to
sample
it
at
the
beginning
of
the
trace,
so
putting
it
all
together
again.
B
Basically,
using
metric
aggregation
can
allow
you
to
select
the
interesting
traces
across
tags
and
dimensions
that
are
meaningful
to
you
in
your
labels
for
your
metrics.
So,
basically,
in
this
model
we
wait
until
metrics
are
aggregated
and
then
we
basically
call
out
to
a
cache
that
is
holding
all
old
traces
that
were
sampling
and
for
this
model
to
work,
you
really
need
a
hundred
percent
sampling
at
the
application
layer
and
then
holding
those
100
percent
of
traces
in
in
a
cache.
B
You
can
spread
that
across
like
a
typical,
well-known
cash
like
Redis
or
memcache
D,
or
something
and
essentially
once
metrics,
have
been
weighted
and
collected.
You
basically
choose
exactly
the
trace
IDs
that
you
chose
to
correlate
to
your
data
points
out
of
that
cache
and
make
sure
that
they're
stored
in
your
tracing
back-end,
and
it's
worth
mentioning
here,
of
course,
that
you
know
you
can
of
course
have
that,
in
addition
to
your
default,
one
to
two
percent
sampling
strategy
using
head
based
sampling
or
adaptive
sampling,
and
so
you
know,
I
wanted
to
kind
of
give.
B
Obviously
a
update
of
the
status
of
you
know
this
project.
This
is
the
current
end-to-end
demo
that
I
ran
today,
however,
that
demo
realize
on
a
lot
of
upstream
changes,
some
of
which
should
be
merged,
some
of
which
haven't
some
of
which
also
require
discussion
with
the
community
and
developers
of
Prometheus
and
of
open
telemetry.
So,
basically,
the
demo
is
also
currently
using
Jaeger
to
export
the
traces
and
the
open,
metrics
exposition
format
itself.
B
Due
to
basically,
our
RF
is
trying
to
use
the
the
current
head
development
check
out
of
open,
telemetry
and
and
not
quite
being
able
to
make
that
work
with
with
our
current
demo
and
I
also
wanted
to
mention
that
the
Capri
Thea's
client
library
requires
significant,
and
there
needs
to
be
an
in-depth
discussion
with
the
maintainer
x',
which
they're,
obviously
we're
kind
of
in
talks
about
that
on
how
to
make
this
a
clean
kind
of
change.
So
a
shout
out
like,
as
I
mentioned,
to
to
everyone,
who's
been
like.
B
Basically,
oh,
the
Prometheus
contributors,
a
few
people
that
co-founder
helping
discuss,
review
and
and
make
some
progress
with
us
on
on
this.
And
if
you
want
to
download
this
slides
at
the
end
of
the
talk
which
I'll
I
didn't
wasn't
able
to
do
right
before
the
talk
so
I
apologize
about
that.
But
basically
here
here
are
those
links
you
can
download,
slides
and
then
kind
of
like
Splunk
around
the
different
projects
and
with
that
I
just
want
to
say
thank
you
for
being
here.
B
D
B
That's
a
good
question:
I've,
definitely
directed
organizations
where
your
instrumentation
coming
from
a
mobile
like
phone
application,
is
transmitted
to
the
back
end
I'm,
not
sure,
if
that's
exactly
kind
of
what
you're
mentioning
but
basically
yeah,
there's.
Obviously
the
ability
to
link
client-side
traces
with
server-side
trace.
It
does
require
like
a
pretty
robust
tracing
pipeline.
Where
you
know
your
mobile
client
knows
how
to
send
up
relevant
metadata
that
you
captured
internally,
so
that
it
can
be
linked
to
the
tracing
data
and
the
server
back
inside
and
a
lot
of
a
lot
of
companies.
B
Every
everyone
I've
seen
they're
all
doing
that
a
little
bit
differently.
So,
unless
they're,
using
like
some
vendor
solution,
that's
out
of
the
box,
but
even
then
I
don't
know
of
any
vendor
solution,
that's
doing
tracing
that
works
both
in
the
mobile
application
and
the
backend
and
joins
the
traces
together.
I,
just
I
just
haven't
seen
that
it
out
there
generally
available.
B
Great
question:
so
that's
what's
really
nice
about
this
implementation.
Is
that
you're
not
actually
adding
a
label
to
the
metric
itself,
which
generates
you
know,
would
generate
a
new
time
series
in
the
backend.
This
is
actually
taking
the
existing
time
series
that
already
has
an
existing
set
of
labels
and
right
next
to
the
float64
value
that
you
capture
and
the
time
stamp
that
you
capture
for
that
data.
Point
you're
packing
a
few
extra
well,
a
lot
of
extra
bits.
B
Actually,
if
you're
recording
a
UUID
to
record
the
data
points,
so
it
doesn't
change
the
cardinality
of
your
metrics
whatsoever,
but
it
does
make
the
bytes
per
sample
go
up
by
a
significant
amount,
depending
on
what
you're
using
for
trace
IDs.
Now
that
would
be
scary
if
every
single
metric
had
a
trace
ID
correlated
to
it.
But
if
you
really
there's
definitely
ways
to
basically
configure
this
so
that
only
certain
metrics
may
be
your
RPC
metrics,
maybe
you're
like
front-end.
You
know:
HT
PG,
RPC
metrics
actually
get
this
annotations
on
the
trace.
Ids!
B
Sorry
off
trace
IDs
to
them,
which
means,
like
you
know.
Basically,
only
a
few
percent
of
your
overall
metrics
ever
get
an
exemple
are
associated
with
it
and
that
helps
control
the
overall
extra
space
required
for
music
exemplars
that
we
found,
but
thankfully
it
doesn't
touch
the
Nellie
at
all
as
well.
No
matter,
no
matter
how
you
can
figure,
it.
B
Or
stored
a
combination
of
metrics,
yes,
so
that's
a
great
question
as
well.
You
know,
if
you
basically
typically
use
Prometheus
out
of
the
box.
You
are
kind
of
guaranteed
that
if,
as
you
have
like
a
container
label,
so
you
know
basically
it's
gonna
subdivide
all
your
metrics
by
the
process
that
actually
gathered
them
because
of
the
the
label
on
the
container.
B
Basically
within
m3,
is
a
little
bit
different
in
that
it
has
an
aggregation
to
here
that
sits
in
front
of
that,
if
you're
using
just
default
m3.
That
actually
is
an
embedded
process
that
happens
in
a
coordinator,
and
it's
called
the
entry
coordinator
that
sits
as
a
side
car
next
year,
Prometheus
instance.
But
if
you're
not
running
a
side
car
model,
then
you
have
a
central
replicated
in
free
aggregator
server.
B
The
metrics
will
flow
through
that
and
you
basically
configure
rules
which
look
very
you
know
which
basically
are
recording
rules
in
Prometheus
land
to
to
like
kind
of
mesh
metrics
together
to
reduce
the
cardinality
for
across
the
queries.
So
it's
kind
of
that
process
of
that's
what
determines
of
like,
which
labels
end
up
in
your
metrics.
Eventually,
thank.
E
D
Here
not
the
question
about
the
database
is
it?
Is
this
certificate
good
good
for
storing
apprentice
events
and
then
match
them
to
okay,
not
for
trade,
not
was
trace
lot,
but
with
metrics,
maybe
with
race
locks
as
well,
because
events,
they
usually
start
for
one
hour
and
it's
they
need
to
be
storage
as
well.
Just.
B
D
B
B
Yeah
great
question:
so
you
we
merged
one
PR
now
that
the
rest
are
all
is
still
open
end
or
in
discussion.
The
yeah
that
we
merged
was
adding
the
support
for
pausing
the
exemplars
in
the
open,
metrics
exposition
format.
However
Prometheus
right
now,
even
if
using
a
client
library
that
supports
that
exposition
format,
of
which
right
now
is
only
the
Python
client
library,
Prometheus
will
will
do
nothing
with
that.
It
basically
throws
it
away.
F
Amazing
idea
to
connect
traces
and
metrics,
as
you
were
talking
I,
was
thinking
whether
you
all
have
thought
about
the
other
way
around.
So
instead
of
having
a
foreign
key
within
metrics
and
connecting
that
to
traces,
let's
say
having
a
notion
of
a
foreign
key
within
the
trace
as
baggage
and
then
using
that
to
connect
to
metric
so
I'm,
imagining
all
the
labels
associated
with
the
metrics
as
tags
into
the
traces
and
then
tail
sampling
based
on
unique
combination
of
those
and
keeping
every
unique
trees.
F
B
A
very
interesting
idea:
yeah
I,
definitely
will
help
you
honest,
no
I
haven't
thought
about
that
I.
Think
what's
interesting
is
when
you
introduce
aggregations
to
this.
A
lot
of
this
stuff
becomes
possible
because
you're,
basically
holding
a
whole
lot
of
state
in
memory
as
data
is
flowing
through
the
system
problem.
With
that,
that's
a
district.
That's
a
relatively
stateful
system
that,
as
you
scale
up
you
know,
is
obviously
another
get
another
distributed
system
that
you're
running
alongside
your
database.
So
it's
it's.
Definitely
not
a
small
feat
to
do.