►
From YouTube: Story of Correlation: Integrating Thanos Metrics with Observabi... Bartłomiej Płotka & Kemal Akkoyun
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Story of Correlation: Integrating Thanos Metrics with Observability Signals - Bartłomiej Płotka, Red Hat & Kemal Akkoyun, Polar Signals
The CNCF Incubated Thanos project with the large open-source community continues to push boundaries regarding observability and monitoring using Prometheus-based metrics. Together with the Prometheus community, it improves the metric story for Kubernetes clusters and beyond. Things like improved performance, better scalability, debuggability, security, metrics backfilling and query QoS is only the tip of the iceberg. As we know, observability nowadays comes in many flavours. Bunching them together is not a trivial side, given many shapes and collection points. Aside from metrics, we have logs, traces or even continuous profiling. In this talk, Kemal and Bartek, Thanos maintainers, after a quick overview of Thanos, will explain how Thanos can be integrated with those non-metric observability signals. The audience will learn an example, end-to-end ways to correlate multiple observability backends with Thanos for enhanced observability and monitoring experience.
A
Hello,
everyone
we
are
here
on
the
maintainers
track
and
learn
about
thanos
and
yeah
some
kind
of
introduction
about
tanos
and
we're
super
excited
to
be
here
in
person.
Finally-
and
you
know
it's
our
first
really
kubecon
after
after
covet,
so
it's
amazing
to
be
here-
and
hopefully
there
will
be
some
interactive
time
where
you
can
ask
some
questions,
because
this
is
yeah.
We
can
have
this
versus
whatever
we
have
in
virtually
that
wasn't
so
so
possible.
A
So
today,
we'll
learn
about
thanos,
but
we
also
learned
about
other
projects
in
the
ecosystem,
open
source
projects
that
really
are
fulfilling
the
whole
observability
story
because,
as
we
know,
metrics
are
kind
of
the
cheapest
and
the
most
reliable.
You
know
first
observability
signal
where
you
have
your
instrumentation.
You
have
your
monitoring,
maybe
you've
probably
used
maybe
with
tunnels,
and
then
you
end
up
having
a
kind
of
reliable
and
and
quick
to
use.
You
know
alerting
and
debugging
platform,
but
it's
not
everything
right.
A
There
are
amazing
tools
that
you
can
combine
with
and
we'll
discuss.
You
know
what
are
the
challenges
there
and
what
are
the
ways
to
mitigate
those
problems,
but
before
that
short
introduction,
my
name
is
bartek
plotka
and
I'm
principal
software
engineer
at
red
hat.
I
maintain
many
many
projects
in
open
source
and
the
biggest
one
are
prometheus
and
thanos
really
mainly
about
goldbank
and
my
passion
towards
golank
and
open
source
kind
of
made.
Me
also
create
a
book,
and
this
book
is
coming
in
december.
A
B
Everyone,
the
name,
is
camera,
I'm
also
a
maintainer
of
couple
of
open
source
project,
not
as
many
as
biotech,
I
guess,
and
recently
I
joined
polar
signals.
We
are
an
observability
company
and
we
are
building
continuous
profiling
products
and
I
I
we
have
a
open
source
offering
called
parka
we're
gonna
actually
demo
on
it.
B
B
Prometus
plus
plus
prometheus
has
certain
problems
like
it's
just
a
single
binary,
and
you
can
only
just
vertically
scale
it
and
when
you
have,
when
you
need
highly
available
system,
you
need
some
additional
facilities.
That's
where
thanos
comes
in
with
thanos.
We
are
providing
highly
available
horizontal
scalable
data
storage
for
prometheus.
Basically,
so
thanos
is
nearly
five
years
old
now
and
we've
been
an
incubating
project
nearly
three
years.
If
I'm
not
mistaken,
we
recently
crossed
like
400
contributors
and
we
have
a
lot
of
slack
users.
So
we
are,
we
have.
B
We
are
a
really
vibrant
community,
like
we
always
active,
and
we
answer
your
questions.
So
if
you
want
to
reach
out
like
just
join
our
slack
channel-
and
we
just
recently
crossed
the
10k
stars
on
github,
which
is
the
most
important
metric
of
all,
as
you
all
know,
we
have
many
many
adapters.
Many
enterprise
adapters,
like
the
people,
are
using
this
in
production
and
like
with
really
really
highly
scalable
environments.
B
Okay,
now,
let's
see
how
we
actually
get
from
prometheus
to
thanos
in
prometus,
we
have
different
components
and
all
these
components
kind
of
kind
of
like
living.
In
the
same
binary,
we
do
this
because,
like
with
a
single
binary
with
a
single
process,
we
can
actually
provide
more
reliable
solutions
and
it's
easy
to
operate
like.
Of
course,
there
are
some
drawbacks
like
you
need
to
install
prometus
closer
to
your
workload.
So
for
that
like
for
each
component
by
with
thanos,
we
created
another
dedicated
component
like
for
the
query
pieces.
B
We
we
have
thanos
querier
for
scraping
pieces.
We
have
a
side
carve
where
you
can
actually
deploy
near
prometheus
and
for
the
rule
engine
we
have
another
thing
called
thanos
ruler,
so
thanos
has
different
deployment
models.
One
of
them
is
as
this
in
described
here.
It's
a
side,
core
deployment
model.
You
can
just
put
the
put
the
sidecar
module
near
your
prometeuses
and
you
can
have
a
global
view
of
all
the
prometuses
that
you
connect
with
the
sidecar.
B
Another
model
is
to
have
an
object,
storage
for
like
long-term
retention,
then,
with
the
help
of
the
site
car,
we
can
ship
all
the
blocks
to
the
object
store
which
is
like
cheap
to
store,
and
then
you
can
just
query
using
another
component
called
thanos
store
api
store,
which
we
come
to
the
api
part.
One
last
thing
to
mention.
We
also
have
a
another
component
called
thanos
receiver,
which
gives
you
the
ability
to
actually
push
your
metrics
directly
to
another
storage.
B
It
depends
on,
depending
on
your,
like
net
network
topologies.
You
may
find
this
useful
rather
than
having
a
pool
based
model,
so
how
we
actually
do
this
thanks
to
the
store
api,
which
is
a
flexible
api
for
us.
Each
component
kind
of
like
exposes
this
api
and
from
the
sonos
query
perspective.
This
is
seamless
right.
So
when
we
check
the
store
api
like
thomas
ruler
is
a
store.
B
Api
receiver
store
itself
even
sidecar
itself
when
we
say
the
store
api
like
it's,
not
just
a
metrics,
but
we
have
all
other
meta
information
that
are
available
with
that.
We
can
get
all
the
examples.
For
example,
we
can
get
the
alerts,
we
can
get
the
recording
rules
like
everything.
So
with
the
flexibility
of
the
store
api,
we
can
just
like
interchange
all
these
components.
B
So
what
we've
been
this
recently
working
on
is
couple
of
improvements,
mostly
around,
like
performance
improvements
or
kind
of
helping
to
community
to
use
different
solutions
for
their,
like
caching
backends,
for
instance.
If
you
prefer
to
use
redis
now
you
can
actually
add
redis
as
your
store
cache
backend
one
other
cashback
can,
we
add,
is
group
cache
which
is
easier
to
operate.
It's
not
an
external
dependency.
You
can
just
embed
this
into
your
thanos
components
and
you
don't
need
anything
anything
else.
B
B
B
The
pr
is
already
there
like.
We
keep
working
on
that.
But
if
you
want
to
know
about
more
know
more
about
queer
pushdown,
we
have
another
talk
coming.
I
guess
it's
in
two
hours
right
from
philip,
he
will
be
going
through
all
the
recent
changes
about
the
query,
pushdown
api
and
the
big
news.
Okay,
we
finally
met
in
person,
and
we
had
our
dev
summit
and
we
decided
that
our
apis
are
pretty
stable.
B
We
haven't
changed
them
for
the
couple
of
even
years
right,
so
it's
time
to
actually
bump
the
tunnels
to
version
one.
For
that
we
have
couple
of
agenda
items
that
we
need
to
take
care
of,
but
we
targeted
the
next
cube
cone
for
the
thanos
1.0.
Basically,
this
doesn't
change
anything
for
you.
I
guess,
like
the
apis
are
pretty
stable.
We
just
need
to
clean
couple
of
flags,
maybe
so
that
we
won't
like
be
supporting
some
experiments
anywhere.
Anything
else
to
add
in
here
yeah
it's
great
cool
and
it's
demo
time.
A
A
So
it's
written
in
kind
of
golang
unit
test,
and
I
will
talk
about
that
maybe
later,
but
I
would
start
and
meantime
it
starts
pretty
quickly,
but
then
metrics
have
to
propagate.
So
it
takes
some
time.
I
will
discuss
the
actual
setup,
so
the
setup
is
following.
I
wanted
to
kind
of
mimic
like
you
know
the
proper
production
setup
that
you
usually
would
have
with
tunnels,
and
this
is
one
of
many
options
and
as
camel
mentioned,
there
are
kind
of.
Oh
you
see
it's
starting.
A
There
are
kind
of
three
major
deployment
models
of
channels,
so
this
is
the
most
complex
one
that
you
probably
you
should
do
at
the
at
the
end
of
your
journey
with
thanos
like
that,
the
ultimate
sas
kind
of
metrics,
but
at
the
end,
how
it
works.
We
have
we
I'm
I
kind
of.
Let's
imagine
we
have
like
customer
clusters,
so
a
separate
cluster
under
you
know
different
network
where
we
run
our
applications,
so
I
have
demo
applications.
It's
like
simple
ping
server.
It's
like
a
simple.
A
You
know,
slash
ping
http
and
I
have
finger
so
kind
of
clients
that
are
spamming.
Those
things
within
the
pink
pink
web
server
and
I
want
to
observe
my
pink
and
make
sure
it's
healthy
and
I
can
debug
that
and
I
have
full
observability.
So
I
kind
of
installed
lots
of
goodies
into
that.
So
first,
I
instrumented
that
with
metrix
and
I
used
grafana
agent,
but
I
could
use
promoters
as
well
and
rafana
agent
or
promote
use
in
this
case
as
well.
Actually,
graphing
is
embedding
prometheus.
A
You
know
we
scrape
metrics
from
this
pink
application
and
then
remote
right,
so
kind
of
export
to
a
separate
cluster
and
we
call
it
observatorium,
because
why
not
it's
kind
of
like
you
know,
imagine
you
have
like
observability
centralized
cluster
and
it's
very
simple
setup
at
the
end.
Maybe
you
know
it's
a
comp,
the
most
complex
deployment,
but
it's
still
kind
of
simple.
You
only
have
three
micro
services.
Query.
A
A
Imagine
and
then
we
do
similar
with
tracing
and
we
use
kind
of
push
api
from
otlp,
so
open
telemetry
protocol
and
that
pushes
to
grafana
agent
and
then
agent
is
able
to
push
that
using
some
jager
format
to
jagger,
which
is
right.
Now.
The
symbols
jager
in
memory,
storage
and
again
jager
allows
us
to
store
traces
and
query
them
and,
and-
and
you
know,
all
of
those
free
systems
has
some
ui
to
use.
A
But
we
we
want
to
make
it
even
more
nicer.
So
we
want
to
add
continuous
profiling
right.
So
what
it
does
is
that
we
install
some
agent
on
the
customer
side
and
this
is
parker
developed
by
polar
signals.
It's
open
source
as
well.
All
of
this
is
open
source
and
it
scrapes
a
special
p,
prof
endpoint,
so
instrumentation
for
profile
golang
has
it
java
has
it,
but
there
are
also
the
other
ways
how
you
can
grab
those
continuous
profiles.
So
with
this
we
have
really
really
nice
overview
of
what's
happening.
A
We
have
all
the
magic
observability
signals,
metrics
logs
tracing
and
continuous
profiling,
but,
as
you
can
imagine,
this
gets
a
little
bit
complex
to
use.
Not
only
you
know
operate,
and
hopefully
you
know
I
mean
industry
is
moving
to
making
it
easy
to
operate
and
scale
and
use.
But
what
about
usage?
What
about
user?
Imagine
I
have
to
go
to
thanos
ui
rafana,
ui,
jugger,
ui
and
then
parka
ui,
it's
just
kind
of
like
scary,
and
you
have
to
learn
three
different
kind
of
search
for
different
search
engines
and
and
apis,
it's
kind
of
crazy.
A
So
I
was
thinking
it's
actually
not
that
hard
to
implement
a
microservice
that
will
understand
those
apis
for
you
and
will
explode
it
in
a
very
simple
correlation
mechanism.
So
we
wrote,
you
know
a
very
complex
collateral
ai
that
will
essentially
correlate
and
hopefully
will
help
us
in
our
journey
and
I'm
joking
of
course.
A
I
I
wrote
that
two
days
ago
it's
very
simple
service
and
it's
very
yellow,
but
the
point
of
this
demo
is
to
make
sure
is
to
show
you
how
easy
is
to
implement,
with
100
line
of
codes,
like
amazingly
useful
correlation
mechanism,
that
ideally,
we
can
bring
up
to
all
those
uis
someday.
Maybe
if
we
embed
it
in
deep
browser,
so
maybe
it's
easy
to
have.
Maybe
you
know
at
some
point
we
can
work
with
those
projects
like
thanos,
loki
and
jagger
and
any
other
to
really
embed
this.
A
This
kind
of
service
in
in
in
our
browser,
so
we
can
kind
of
show
you
way
around
okay,
but
enough
talking,
let's
see
if
this
will
work,
so
the
situation
is
following
right.
We
have
some.
Let
me
quickly
rearrange
things
so
we
start
with
thanos
ui
and
thanos
allergic
ui.
So
imagine
we
have
our
finger
ping
service
alert.
We
are
happy
nice,
we
can,
you
know,
go
to
conference
and
just
enjoy
our
time.
Unfortunately,
we
are
on
call-
and
you
know,
imagine
you
get
paged
and
you
get
paid
on
your
phone.
A
You
check
and
there
is
a
link
to
maybe
thanos
ui
that
contains
an
alert
and
we
can
see
there
are
too
many
errors.
Well,
this
is
what
I
can.
I
can
read
from
it
right
and
there's
some
complex
query
to
make
that
to
to
express
that,
but
what
I
do
next,
hopefully
there
is
some
run
book.
There
is
some
links
that
maybe
to
dashboard
or
something,
but
what,
if
you
don't
have
those?
A
What,
if
you
are
in
the
middle
of
the
night
suddenly-
and
you
really
want
to
fix
it
very
very
quickly-
so
maybe
you
go
to
thanos
ui,
and
what
do
you
do?
What
do
I
query?
Maybe
I
can
just
search
something:
okay,
there
are
some
requests
http
sure,
but
there
might
be
more
services
that
I'm
observing.
How
do
I
get
to
the
information?
I
really
want
right.
I
have
to
understand,
prompt
you
well,
I
need
to
really
understand
what
is
happening.
So
maybe
there
are
things
we
can
improve
here.
A
Now,
let's
go
to
our
logging
signal
and
I
open
you
know
just
main
graffana.
You
know
kind
of
ui
and
if
you
are
familiar
with
this,
you
can
you
know
you
can
do
a
lot
here,
but
you
have
to
think
and
like
in
the
moment
on
the
you
know,
uncle
kind
of
page
and
it's
incident
situation.
It's
really
hard
to
kind
of
find.
You
know,
find
the
correct
data
source
and
really
try
to
you
know.
Maybe
I
don't
know
find
something
right.
A
It's
kind
of
hard
and
the
same
with
tracing
you
go
to
tracing
and
say:
oh
I
want
to
what
should
I?
What
should
I
do?
What
pack
I
should
use
like?
How
do
I
find
my
not
only
service
that
is
instrumented
with
those
traces,
but
also
you
know
the
exact
situation,
exact
error
that
is
happening
right
and
again,
because
we
have
four
things.
There
is
also
parka
and
you
know
how
to
use
those
this.
You
know
kind
of
query
filtering
mechanism
of
parka,
so
at
the
end
it's
kind
of
crazy.
A
So
this
is
where
we
thought
it's
kind
of
simple,
to
write,
automation
that
will
understand
those
things
and
you
know
again
take
it
as
an
example.
It's
it's
just.
You
know,
I'm
not
front-end
developer,
so
I
I
didn't
build
a
nice
ui
for
that,
but
think
about
it.
What
I
have
is
like
a
simple
ui
or
like
simple
input,
so
I
copy
the
alert
name,
big
service
too
many
errors.
I
just
put
into
that,
so
being
serviced
too
many
errors.
A
I
want
to
use
exemplars
and
I
will
tell
you
what's
that
later
on
and
I
click
correlate
and
voila.
It
actually
gave
me
a
lot
of
links,
useful
links.
I
can
use
so
let's
go
through
this
page
and
you
know
imagine
it
is
like
embedded
in
your
browser.
So
it's
easy
to
use,
but
correlation
this
ai
found
that
there
is
an
alert.
It's
firing
for
a
certain
service
with
certain
instances
with
certain
job
and,
what's
more
amazing,
is
that
it
found
out
that
this
alert
is
actually
using
http
request
total
counter.
A
That
is
also
instrumented
with
exemplars
and
exemplars
is
essentially
a
way
to
provide
an
example
request
id
that
contributed
to
those
errors.
So
we
know
that
there
is
a
certain
request
that
goes
among
many
that
cause
this
alert
to
fire
and
we
put
that
into
exam
exemplar
right
and
might
be
thousands,
but
we
we
won't.
A
We
only
want
one
example,
so
you
can
see
that
this,
my
my
service
queried
actually
tunnels
and
makes
like
two
different
api
calls
and
gathered
that
there
is
a
request
id
that
is
correlated
to
those
errors
that
cause
alerts
to
fire.
So
it
constructed
useful.
You
know
links
for
us,
so,
for
example,
we
can
go
and
check.
A
You
know
exactly
the
the
query
behind
the
alert
here,
so
we
can
see
confirm
yeah,
error
rate
is
you
know
above
65
percent
right
and-
and
you
know
even
like
what
are
the
actual
error
rate
requests
per
second,
it's
slower
than
then
it's
less
than
one
per
second,
but
you
know
in
this
actual
case:
it's
not
very
useful.
So
maybe
let's
try
a
different
correlation.
Maybe
you
know,
as
I
think
it
was
amazing,
talk
on
prometheus
day
from
bjorn
around
anomaly
detection,
and
he
mentioned
that.
A
You
know
it's
it's
it's
useful
to
have
those
theories
that
you
want
to
check,
but
only
a
few
of
those.
So
we
look.
We
click
from
theories
that
what
can
go
wrong?
What
could
go
wrong
and
you
know
where
is
the
best
place,
to
look
for
the
reason
why
things
are
failing?
So
let's
go
to
the
logging,
maybe
logging,
you
know
it's
useful,
you
can
see.
I
don't
need.
I
didn't
need
to
craft
this
query
I
have,
and
you
know
you
don't
need
to
learn
to
look
ul.
A
A
Is
there
error
yeah?
I
don't
think
it's
there.
Well,
we
are
unlucky
here
at
this
time.
Logging
was
not
very
helpful.
Let's
try
tracing
so
with
tracing.
Let's
go
that
correlation
go
where
you
know
kind
of
directed
us
to
tracing.
You
can
see
that
immediately
they
gave
us.
You
know
this
correlator
gave
us
the
trace
view
right,
which,
which
exactly
is
a
spans
for
this
request
that
failed
and
contributed
these
alerts.
It's
very,
very
useful
information.
A
You
can
see
it's
an
error
and
you
know
the
whole
request
took
some
time,
but
there
is
this
evaluate
ping
function
which
took
one
hand
over
100
one
second,
and
we
can
see
it
turned
error
because
decided
to
not
return
success.
Okay,
that
was
not
helpful,
but
at
least
we
know
a
function,
so
we
already
knew
something
right.
A
So
we
are
unlucky
this
time,
but
who
knows
maybe
on
next
dollar
that
that
information
would
be
helpful.
So
what
we
can
do,
let's
go
to
the
last
link,
which
is
parker
and
let's
check.
If
continuous
profiling
will
tell
us
anything-
and
you
can
see
you
know
our
caller
rate
calculator,
crafted
a
a
query
that
checks
the
exact
job
and
profiling.
A
You
know
in
the
exact
moment
when
this
you
know
kind
of
example,
exemplar
was
happening
and
already
told
us
a
lot.
So
we're
actually
spending
a
lot
of
cpu
time.
Let's
see
so,
apparently
we
spent
five
minutes
in
total
across
all
the
pings
on
this
nasty
back
function.
Right
and
if
you
go
to
my
code
and
search
for
this,
actually
let
me
copy
what
happened
here
so
yeah.
This
is
my
pin
code.
A
Let's
search
oh
yeah,
I
have
this
nasty
pink.
That
just
does
you
know
nothing
and
burns
cpu
exactly
this
so
yeah
I
found
error,
just
delete
it
and
now
it
should
work
right.
So
this
is
kind
of
the
story
we
wanted
to
show.
Should
we
show
experimental
feature
if
this
is
actually
working?
So
let's
try.
So
we
have
another
link
here
and
this
is
like
so
what
you
have
seen
is
totally
doable
right
now
with,
like
master
master
or
main
of
all.
A
Those
projects
like
latest
releases,
but
parker
is
going
crazy
with
features,
and
one
thing
which
is
amazing
is
that
we
can
use
exemplar
so
trace
id.
So
usually
we
have
profiling.
That
is
across
all
the
things
that
were
happening
in
the
binary
but
trace
id
or
like
we
can
label.
You
can
filter
out
things
by
certain
labels
and
we
could
filter
out
essentially
only
cpu
time
spent
for
this
request
and
it's
experimental
and
it
was
not
working
before
whoa.
It
works
all
right.
A
So
yes,
oh,
my
god,
yeah
and
you
can
see
that
correlator
crafted
a
query
which
directly
asked
for
request
id,
and
you
can
see
that
just
this
request
took
exactly
one
second
of
cpu
time.
This
is
incredible
because
if
you
have
a
large
binary
that
has
do
lots
of
things,
and
maybe
it
does
a
lot
of
other
cpu
time
work.
You
wouldn't
see
the
exact
problem
performance
problem
in
this
request,
so
this
is
yes,
amazing.
Congratulations
to
polar
signal,
for
you
know
really
yeah
hacking
profiles,
it's
amazing
all
right!
A
So
that's
for
the
demo
side.
I
want
to
show
you
one
more
important
thing
so
how
we
even
constructed
this
exemplar
and
those
kind
of
few
all
those
observability
signals
together,
the
key
of
of
how
we
were
able
to
correlate
those
stuff
is
in
this
function.
It's
like
a
middleware,
so
it's
not
like
very
visible
in
the
code,
because
it's
hidden
in
this
helper
that
wraps
your
http
handler
for
ping
and
in
the
go,
but
it
will
look
very
similar
in
other
languages.
A
So,
as
you
can
see,
we
create
a
registry
from
it
use
registry
from
our
official
client,
golang
library.
We
create
a
metric
for
http
request,
total.
Does
this
work
yeah,
then
we
grab
exemplar
if
there
is
any
so
well
grab
trace
id
really,
if
from
tracing
library,
that
I
wrote
that
wraps
open,
telemetry
library
by
the
way-
and
I
wrote
myself
because
I
wrote
another
one,
because
it's
kind
of
like
open
telemetry
library
was
too
complex
to
kind
of
use
for
normal
humans.
A
So
this
trace
id
is
connected
to
our
http
request,
total
metric,
which
is
kind
of
like
logically
and
symmetrically,
making
sense
right
and
of
course,
we
have
to
wrap
with
some
tracing
library
to
have
this
trace
id
and
all
of
it
right.
So
I
will
not
explain
everything,
but
you
can
grab
my
slides
and
you
can
reproduce
everything
it's
like
in
our
bw
correlator
repository.
A
You
can
run
a
unit
test
and
it
will
spin
all
those
things
and
to
achieve
this,
I
use
this
one
library
you
can
see
here
efficient,
go
end-to-end
and
if
you
want
to
learn
more
join
our
talk
today
on,
you
know:
4
p.m.
50
50
and
there
are
jessica
mitchell
here
who
will
be
discussing.
You
know
just
this
end-to-end
framework,
because
it's
running
containers
on
your
local
laptop,
so
you
can
totally
test
on
production.
Many
many
things
and
we
use
that
to
test
from
thanos
on
every
every
pr.
A
So
we
spin
up
the
whole
tunnels
deployment,
complex
systems
and
ultra
fast.
So
I
recommend
checking
that
and
we
also
want
to
mention
that
we
do
mentorship.
So
if
you
have
any
new
joiners
into
you
know
open
source,
we
are.
We
are
there
to
help
you
and
you
can
also
watch
a
video
from
lucas
and
to
learn
more
and
with
that.
I
would
like
to
thank
you
and
yeah.
B
I
guess
before
the
questions,
one
disclaimer
like
you've
seen
that
we
added
examples
to
the
alerts,
which
is
a
feature
that
we
are
we're
going
to
add
in
client
call
link
pretty
soon.
Next
version
same
goes
for
parker
as
well.
The
trace
id
queries
will
be
included
in
the
next
release,
so
this
is
very
bleeding
edge.
Demo
and.
A
C
Hello,
okay,
yeah,
so
you
had
to
well
you
you
built
this
on
on
what
appears
to
be
not
very
deep
tweaks
to
the
entire
ecosystem,
right,
changing
a
few
things
on
each
client,
and
maybe
how
far
are
we
from
this
being
the
default
every
of
every
one
of
these
products
out
of
the
box
having
been
configured
so
that
you
could
just
plug
in
this
correlator?
And
is
this
an
initiative,
something
that
is
that
is
happening
in
in?
Is
it
a
conversation?
That's
happening.
A
There
are
definitely
conversations,
but
it's
really
like
organically
growing.
I
think
we
are
there
because
we
already
you
know,
build
stable
metric
packages
that
are
metric
libraries
for
different
languages
with
exemplars.
Now
we
have
to
find
out.
You
know
how
to
work
with
other,
like
you
know,
tracing
libraries
and
logging
libraries
to
to
create
those-
let's
say
auto
instrumentations,
for
maybe
your
framework
of
preference,
and
you
know
I
think
you
can
discuss
about.
B
B
Yeah,
so
the
most
important
thing-
I
guess
here
is
the
labels
right.
All
these
tools
are
based
on
the
label
models.
That's
why
we,
how
we
actually
correlate
those
there
are
a
couple
of
initiatives
to
just
like
ease
up
the
correlation
like
open
telemetry
which,
but
you
need
to
do
that
in
the
instrumentation
size
right.
So
it's
kind
of
like
tightly
coupled.
B
A
Any
instrumentation,
like
one
package,
you
install
and
you
have
all
this-
you
know
right
side
thing.
You
know
by
by
just
one
click
of
the
button:
that's
the
kind
of
idea.
So
we
have
this
part
done
and
and
then
you
know,
we
have
to
correlate
all
those
kind
of
efforts
within
different
libraries
as
well
to
have
instrumentation
part.
But
I
I
think
what
you
I.
B
Wanted
you
to
say
I
will.
I
will
come
to
that
for
this
still
with
the
people
of
labels,
you
need
a
little
bit
instrumentation
right.
You
need
to
grab
that
tracing
idea
and
put
that
as
in
label
what
we
are
also
working
on
with
the
parka.
There
is
an
agent
which
is
an
ebpf
based
agent
and
collect
those
metrics
by
just
using
ebpf,
which
is
zero
instrumentation.
B
A
Even
without
labels,
I
think
it
can
work,
but
the
idea
is
that
there's
a
bpf,
you
know
kind
of
observability
direction,
going
on
as
well
to
kind
of
have
all
logging
metrics
and
tracing
and
profiling
just
by
ebpf.
So
hopefully
we'll
see
more
improvements
on
the
side,
but
really
there
is
nothing
better
than
instrumenting
your
own
application
owning
the
code.
I
trust
you.
I
mean
trust
me
because
we
we
we
have
amazing
observability,
because
we
yeah
we,
we
kind
of
kind
of
added
those
statements,
and
this
is
like
called
open
box.
A
D
B
B
It's
kind
of
the
holy
grail
everyone.
Everybody
wants
to
go
there
right,
but
those
discussions
are
very
early
stage.
Obviously
we
probably
will
be
part
of
that
discussion.
We
are
building
this
open
source
tool,
so
there
is
tag
observability,
there's
open,
telemetry,
subgroups,
so
yeah
just
be
part
of
those
discussions
yourself
as
well.
If
you
want
to
see
any
feature
that
you
want
to
add
be
edit
but
yeah,
we
are
just
like
scratching
the
surface
surface
of
this
profiling
area.
Awesome.
Thank
you.
A
It's
worth
to
mention
that,
still
even
with
open
telemetry
having
all
the
signals
in
the
world,
you
have
to
run
multiple
agents
because
a
more
scalable
solution
like
if
not
normal
production,
you
won't
wouldn't
survive
one
binary.
Having
all
the
signals
you
have
to
have
different,
maybe
open
telemetry,
for
example,
just
different
instances
of
open
telemetry,
but
still
you
have
to
scale
those
across
metrics
and
traces
and
profiles,
and
probably
logging
at
some
point,
but
those
are
like
different
characteristic
of
performance
of
collecting
those.
A
So
you
have
to
manage
that
in
practice
you
have
to
manage
that
in
separation,
so
maybe
it's
one
binary
is
helping.
I
am
open.
Telemetry
is
doing
an
amazing
job
on
on
kind
of
getting
apis
to
to
work
together.
So
that's
amazing,
but
the
agent
side,
you
have
to
still
probably
running
the
separate
agents.
So
that's
that's
kind
of
the
the
unfortunate
reality
yeah.
Thank
you.