►
From YouTube: Workshop: Analyzing and Visualizing OpenTelemetry Traces with SQL - John Pruitt, Timescale
Description
Workshop: Analyzing and Visualizing OpenTelemetry Traces with SQL - John Pruitt, Timescale
Speakers: John Pruitt
If you are planning to attend the workshop, follow these instructions:
Bring your laptop and charger. This will be hands-on!
Make sure you have Docker installed. You will need it to run de demo.
The day before, go to tsdb.co/opentelemetry-demo and follow the instructions in the repo to deploy the demo environment we'll be using. This is a lightweight microservices demo that can be installed quickly. You will also have the chance to deploy it at the beginning of the workshop.
We also recommend that you have psql installed (the PostgreSQL client), but this is not absolutely necessary.
All these instructions are also included in the workshop.md file of the tsdb.co/opentelemetry-demo.
Workshop Description:
Of the three observability data types supported by OpenTelemetry (metrics, logs, and traces), the latter presents the most potential. Tracing exposes the inner-workings of requests processing in modern, cloud-native architectures.
Tools typically visualize a single trace, showing how an individual request traversed your entire system. This approach helps with distributed debugging and analysis. However, looking at a single trace is of limited value.
But what if you stored tracing data in a SQL database? You could ask global questions about your system and uncover answers aggregated from thousands of traces at once. You could discover in real-time upstream and downstream dependencies between services or components. You could find slow execution paths, where the error rate spiked since the last deployment, or where the request rate suddenly dropped - and not just that it occurred but pinpoint the cause. Thus, tracing analysis is a proactive tool to help you spot issues before your customers do. In this workshop, you will learn how to do precisely that. Concretely, we will ingest OpenTelemetry traces into a PostgreSQL database, and we’ll learn how to build powerful custom dashboards in Grafana using SQL that will immediately give you a deep level of insight into your distributed systems, helping you make the most out of your tracing data.
First, we will explore a lightweight open-source system designed to demonstrate the power of tracing and SQL that anyone can easily download and run. Then, we will use this demo environment to explore how to search and visualize traces using Jageer and Grafana, and as the last step, we will learn how to create a new Grafana dashboard to visualize throughput, response time and a recursive query for analyzing downstream dependencies through tracing data.
A
Welcome
to
the
workshop,
this
is
intended
to
be
hands-on.
So
if
you're
wanting
to
follow
along,
go
ahead
and
go
to
this,
this
link,
either
via
the
qr
code
or
the
the
link
up
there
and
download
the
repo
and
then
run
docker
compose
up
and
the
route
of
the
repo
you'll
be
deploying
a
system
that
we
will
use
to
build
some
grafana
dashboards
and
explore
traces.
A
A
And
this
is
a
preview
of
what
we're
going
to
build
today,
the
system
you're
going
to
deploy,
which
I
should
say
the
the
postcards
on
the
table-
have
the
link
as
well
the
qr
code.
So
if
you
missed
it
before
it
can
catch
up
we're
going
to
deploy
a
system,
that's
going
to
generate
traces,
no
metrics,
no
logs
and
we're
going
to
build
some
grafana
dashboards
using
that
data.
A
A
So
the
repo
actually
has
two
different
versions.
The
question
was:
are
these
manual
traces
or
automatic
today
we're
going
to
use
the
manual
version,
so
the
code
is
instrumented
manually,
there's
also
a
version
in
the
repo
that
uses
only
auto
instrumentation
and
please
feel
free
to
ask
questions
as
we
go.
A
So
what
we're
going
to
do?
First,
we're
going
to
talk
about
the
demo
system.
Just
what
make
sure
you
understand
what
you
just
deployed,
then
we're
going
to
talk
a
bit
about
tracing
background,
just
to
make
sure
we
understand
enough
to
write
sql
and
then
we're
going
to
build
a
dashboard
using
the
time
series
aspect
of
tracing,
then
we're
going
to
build
a
second
dashboard,
focusing
on
the
tree
structures
and
then
wrap
up
the
demo
system.
Again,
here's
the
link,
pull
it
down.
It
all
runs
a
docker.
A
When
you
get
to
the
repo,
you
know,
you
hit
the
code
button,
you
can
either
use
git
clone
download
it
as
a
zip.
I
also
want
to
note
that
there
is
a
workshop.md
file
in
the
root
with
notes.
So
basically
everything
I'm
going
to
talk
about
is
is
recorded
in
that
that
markdown
file
as
well.
So
if
you
fall
behind
or
if
later
on,
you
want
to
revisit
and
dig
in
a
little
deeper,
you
can
consult
that.
A
A
So
what
are
you
deploying?
We've
implemented
a
an
absurd
password
generator.
The
point
of
this
is
not
to
build
a
good
password
generator
it's
to
emit
interesting
traces.
So
what
we
have
are
four
services
that
each
return
a
random
character.
So
the
lower
service
serves
up
lowercase,
characters,
upper
uppercase,
digit,
digits
and
special
special
case.
A
Also
in
docker,
we've
got
an
observability
stack,
the
traces
flow
into
the
open,
telemetry
collector.
First,
that's
not
strictly
necessary,
but
it's
in
the
box
to
play
with
right.
Now
it's
configured
only
to
do
batching
no
sampling,
though
the
collector
then
forwards
those
traces
to
the
prom
scale
connector,
which
puts
them
in
the
prom
scale
database,
which
is
a
time
scale.
Postgres
database.
A
A
A
A
A
So
you're,
probably
familiar
with
looking
at
a
trace
in
jager
right
you
go
over
to
jaeger.
You
can
see
the
tree
structure
over
here
by
the
nested
spans
and
you
can
see
the
time
spiri
time
series
aspect
over
here
with
the
gantt
chart
now
the
length
of
the
lines
here
is
the
duration,
and
you
also
get
the
start
and
end
times
so.
This
is
this
is
cool
and
this
is
a
powerful
tool,
but
you
typically
you're
only
looking
at
one
trace
at
a
time.
A
A
So
also,
let's
look
at
the
perspective
from
the
code
side.
This
is
a
function
in
one
of
the
services
we've
deployed
is
python.
You
can
see
on
line
37,
we
create
a
span
and
give
it
a
name.
That
span
has
a
context
and
the
beginning
of
the
context
and
the
end
of
the
context
will
determine
the
start
time
and
the
end
time
you
can
attach
attributes
to
a
span.
A
A
A
The
other
thing
I
want
to
mention
here
is
again
like
this
is
not
a
great
password
generator.
It's
created
to
make
interesting
traces,
and
so
these
calls
to
work
are
intentionally
slowing
down
the
code
just
to
make
interesting
data
and
we've
got
you
know:
random
exceptions
being
purposefully
emitted
as
well.
A
So
the
tracing
spec-
let's
say
we
wanted
to
ingest
these
these
traces.
This
is
a
screenshot
of
the
protobuf
definition
for
a
trace.
There's
a
lot
there.
It
would
take
us
all
day
to
do
this,
probably
more
so
we're
not
going
to
do
that.
That's
why
we're
piggybacking
on
prom
scale,
which
has
done
the
hard
work
for
us.
A
So
promscale
is
going
to
land
this
data
in
a
postgres
database
in
a
number
of
tables,
but
today
we're
going
to
focus
on
only
one
view
and
that's
the
span
view.
So
each
row
in
the
span
view
corresponds
with
a
single
span
right
and
many
traces
will
be
represented.
In
this
view,
the
the
view
has
many
columns
as
well,
and
we're
only
going
to
look
at
eight
of
them
and
I
think
you'll
be
surprised
at
just
how
much
value
we
can
get
out
of
eight
columns.
A
The
service
name
column,
unsurprisingly,
gives
you
the
name
of
the
service
that
emitted
the
span.
The
span
name
corresponds
to
the
name.
You
give
the
span
when
you
create
it,
which
is
typically
like
a
function
name
or
some
sort
of
operation,
the
start
time
in
time
and
duration.
That's
the
time
series
aspect
of
a
span
and
the
using
the
trace
id
and
the
parent
span
id
you
can
find
a
parent
and
therefore
you
get
the
tree
structure
of
a
span
of
a
trace.
A
A
A
So
grafana
gives
you
a
start
time
and
an
end
time
using
that
ui
filter
and
what
we're
going
to
do
is
compare
the
start
times
of
the
span
to
that
time,
window
that
we've
selected,
and
so
as
you
can
imagine,
some
spans
will
be
fully
contained
in
that
window.
Some
will
overlap
and
some
will
will
be
excluded
from
the
window
and
note
that
that's
true
within
a
trace.
So
if
this
were
a
trace,
some
spans
would
from
the
trace
would
be
included
and
some
would
be
excluded.
A
Okay,
so
in
grafana
there's
a
macro
called
a
dollar
underscore
underscore
time
filter.
You
can
use
that
to
tie
your
queries
to
that
ui
widget
and
the
way
we're
going
to
do
that
is
in
the
where
clause
we'll
use
time,
filter
and
we're
going
to
filter
on
the
start
time
that
expands
to
something
that
looks
like
this,
where
the
start
time
is
between
the
beginning
and
the
end
of
the
filter
that
we've
selected.
A
A
A
in
the
demo.
Folder
are
a
number
of
dashboards.
The
ones
we're
going
to
use
for
the
workshop
are
workshop.
1
workshop,
1
finished
workshop
2
and
workshop
2
finished.
So
what
I've
done
is
workshop.
One
and
workshop
one
finished
are
copies
of
one
another
in
workshop.
One
all
of
the
queries
are
commented
out,
so
we're
gonna
talk
about
each
one
and
we
will
uncomment
them
and
enable
them
as
we
go
workshop.
One
is
finished
workshop.
One
finished
everything's
enabled.
So
if
you
want
to
cheat
just
go
to
the
finish,
one.
B
A
A
You
in
there
now,
okay
cool
yeah,
so
we
jump
into
workshop
one,
and
you
should
see
something
that
looks
like
this
and
nothing's
working
right.
So
what
we're
going
to
do
is
we're
going
to
enable
this
this
panel
right
over
here
that
says
number
of
traces
in
the
time
window
and
when
we
do
it's
going
to
look
something
like
this.
A
So
behind
this
panel
is
one
query.
The
query
is
going
to
look
at
the
span
view
that
we've
been
talking
about
it's
going
to
use
the
macro
from
grafana
to
filter
on
the
time
window
and
we're
going
to
filter
where
the
parent
span
id
is
null
and,
as
I
mentioned
before,
the
root
span
of
every
trace
has
a
null
parent
spin
id.
So
basically,
what
we're
saying
is
give
me
all
of
the
root
spans
in
this
time
window
that
we've
selected,
and
why
is
that
important?
A
We
know
that
every
trace
has
a
single
root
span,
so
by
counting
the
number
of
root
spans
in
the
time
window.
We
know
the
number
of
traces
in
the
time
window
right
so
go
back
to
grafana
hit
edit
on
the
panel,
and
you
should
see
some
comments
in
the
sql.
Take
those
comments
out
hit
apply
and
you
should
see
a
number
come
up
so
right
now
I've
got
my
filter
set
to
five
minutes
in
the
last
five
minutes.
There
are
a
hundred
and
sorry
712
traces
again.
This
is
a
terrible
password
generator.
A
So
next
we're
going
to
do
throughput
we're
going
to
build
a
panel
that
looks
like
this.
So
if
we
know
every
trace
has
a
single
root
span
and
we
don't
have
sampling
turned
on.
We
know
that
every
trace
corresponds
with
a
request.
We
can
compute
the
throughput
in
the
system
by
counting
spans
over
time
right
or
counting
traces.
A
A
I
didn't
want
you
all
to
have
to
type
so
we're
going
to
make
this
easy
and
just
uncomment
up
with
the
comments
there
and
hit
apply.
So
what
do
we
see
here?
We
see
that
at
least
on
my
system.
I
had
a
peak
of
46
traces
in
a
10
second
window
in
the
last
five
minutes
and
I've
seen
as
few
as
five
so
already.
We
know
that
there
is
variability
in
the
throughput
of
this
system
over
time
and,
if
I
scroll
back
out
to
say
15
minutes,
we
see
a
pattern
even
more
clearly
right.
A
A
A
All
right
so
now
we're
going
to
look
at
a
histogram
of
latencies,
so
we've
seen
the
throughput
we've
seen
the
top
ten
slowest
ones,
but
the
top
ten
slowest
one
doesn't
really
give
us
a
picture
of.
You
know
what
are
the?
What
are
the
fastest
ones?
Look
like
what
do
the
average
ones
look
like,
so
this
histogram?
What
it's
going
to
tell
us
is
it's
going
to
break
our
latencies
down
into
a
number
of
buckets
and
then
count
in
those
buckets
for
the
entire
time
window.
A
A
A
A
A
A
A
A
So
now
we're
going
to
build
a
histogram
of
latencies
over
time
and
what
this
essentially
gives
you
is
a
think
of
it
as
like.
You
have
to
kind
of
think
of
3d
right,
because
it's
not
only
going
to
give
you
this,
the
slowest
and
the
fastest
in
each
10.
Second
bucket,
it's
going
to
give
you
the
span,
the
the
the
spread,
and
it
will
show
you
where
most
of
the
most
of
the
traces
are
landing
within
that
spread,
so
the
brighter
box,
the
the
more
traces
in
that
box.
That
makes
sense.
A
A
Hit
apply
and
now
so
you
can
see
right
above
the
p95
latencies.
That's
only
telling
us,
you
know
essentially
the
slow.
What
the
slow,
slow
requests
look
like
now.
With
this
picture
we
see
for
each
vertical
slice,
it
broken
down
into
buckets,
and
the
color
corresponds
with
how
many
traces
fall
into
that
bucket
right.
So
we
can
see
there's
variability
in
in
the
latencies
and
we
can
see
the
spread
right.
A
Okay,
so
you
may
say
this
is
great.
I
know
my
system
is
slow
and
I
know
it's
variably
slow,
but
where
do
I
go
from
there?
You
know
what?
If
I
want
to
nail
this
down
to
a
service
or
an
operation,
how
do
I
figure
out
where
to
go
optimize?
First?
So
that's
what
we're
going
to
build
now
we're
going
to
build
this
doughnut
chart
this
pie
chart.
A
What
we're
going
to
do
is
for
every
operation,
which
is
like
a
function
we're
going
to
compute
the
enti,
the
amount
of
execution
time
that
was
spent
in
that
operation
for
the
15
minute
window
right
and
you
could
say:
okay,
the
function,
that's
spending
that
we're
spending
the
most
execution
time
in
is
the
bottleneck
and
that's
where
we
should
go,
spend
our
time.
Optimizing.
A
So
we
have
this
query:
this
one's
going
to
be
a
little
bit
more
advanced,
so
go
back
to
the
thought.
Each
span
has
zero
or
more
children
a
parent
span's
time
frame
encompasses
all
of
his
children.
So
if
we
want
to
know
how
much
time
is
spent
in
a
span
exclusive
of
his
children,
we
have
to
subtract
the
child's
duration
from
the
parents
right
and
that's
going
to
get.
Let
us
pinpoint
whether
the
time
was
spent
in
the
parent
or
in
the
child.
A
A
A
A
A
A
I'll
move
it
to
five
minutes
instead
of
15.
okay.
So
what
do
we
see
here?
We
see
that
in
this
five
minute
window
we
spent
10.4
minutes
of
execution
time
in
the
digit
service
in
the
random
digit
operation
right
it
vastly
outstrips
all
of
the
rest
of
the
system,
so
realize
we're
not
just
looking
at
one
process,
not
just
looking
at
one
service
we're
looking
at
all
five
services,
we're
looking
at
you
know
a
thousand
traces
for
five
minute
window.
A
So
you
know,
maybe
there
are
all
different
kinds
of
requests
happening
in
that
five
minute
window
and
we've
pinpointed
a
function.
That
is
our
culprit.
So,
with
this
panel
alone,
you
you've
identified
the
bottleneck,
and
you
know
exactly
where
to
go.
To
spend
your
time
optimizing
right.
That's
pretty
cool.
A
A
A
All
right,
so
we
see
that
this
table
verifies
what
we
see
in
the
pie
chart
the
digit
service
has
a
random
digit
operation
on
average,
it's
taking
over
a
second
to
complete,
at
least
on
my
laptop
and
the
p95
direction
is
4.2
seconds.
So
not
only
is
it
slow
on
average,
it's
extremely
slow
in
the
slowest
case
we
also
see
you
know
the
next
slowest
operation
is
in
the
same
service.
It's
the
render
digit
and
you
know
the
p95
of
it-
is
363
milliseconds
on
my
laptop.
A
A
A
A
So
what
is
this
orange
bar?
It
is
the
random
digits
function
from
the
digit
service.
Unsurprisingly,
like
that's
what
all
of
the
patterns
are
pointing
to,
but
we've
just
confirmed
it.
What
else
can
we
see?
We
can
also
see
that
occasionally,
this
blue
bar
is
tall
and
this
red
bar,
you
know,
and
those
look
periodic
as
well.
We
see
the
red
bar
is
longer
here.
Then
it
goes
away.
It
comes
back
here.
A
A
A
So
we
have
a
grafana
dashboard.
Now
we've
used
nothing
but
traces,
no
metrics,
no
logs.
I
don't
know
about
you,
but
we
started
out
with
a
very
simple
query
and
built
upon
it.
So
even
the
the
most
advanced
queries
in
here
are
understandable
and
yeah.
You
know
I,
I
think
you
get
a
lot
of
value
out
of
this
any
question
before
we
move
on
yeah.
A
I'm
not
sampling
at
all,
so
right,
yes,
so,
theoretically,
if
you
were
sampling,
throughput
is
actually
obviously
not
going
to
be
throughput
right.
You
could
actually,
if
you
knew
like
you,
were
doing
10
samples
or
something
maybe
you
could
multiply
it
scale
it
up
and
get
a
approximation,
but
it
wouldn't
be
exact
right
slowest
traces.
B
A
A
A
So
if
we
were
to
query
a
trace
a
given
trace,
we
use
a
explicit
trace
id
here
and
started
getting
the
span
ids
and
the
parent
span
ids
out.
I
mean
this
is
what
it
looked
like.
It's
just
a
list
of
numbers-
and
I
don't
know
about
you,
but
like
my
brain,
can't
make
sense
of
that.
That
doesn't
really
tell
me
anything,
but
we
can
use
sql
to
make
sense
of
this
to
do
that.
A
A
A
Nobody
wants
to
do
this
right
and,
if
you're
familiar
with
postgres,
you
may
know
the
the
generate
series
function.
We're
going
to
consider
that
cheating
for
now.
That's
that's
the
nice
way
to
do
it
right
so
ooh.
You
know
if
we
wanted
to
count
to
a
million
nobody's
gonna.
Do
it
that
way,
that's
just
gross.
A
This
with
this
is
a
common
table
expression
and
you
use
the
recursive
keyword
here
to
say
this
is
a
recursive
common
table,
expression
aliased
as
x,
and
the
pattern
is
you'll
have
a
query
union
with
another
query,
and
the
second
query
is
going
to
reference
x,
so
it's
referencing
itself
and
how
this
works
is.
The
first
query
is
the
initialization
step:
it's
where
you
start
in
the
recursion.
A
A
A
A
So
we're
going
to
look
at
spans
from
the
same
trace
where
this
iteration
span
id
is
the
previous
iteration's
parent
span
id
and
then
we're
going
to
go
one
more
up
and
we'll
say
where
this
iteration
span
id
is
the
previous
iterations
parents
band
id,
and
in
that
way
we
start
here
we
go
up,
go
up
looping
recursively
until
we
stop,
which
is
would
be
where
the
parent
span
id
is
null.
There's
nothing,
nothing
else
to
match
on.
A
A
A
Why
are
they
sending
more
or
let's
say
I'm
responsible
for
service
x,
I'm
getting
the
same
workload,
but
now
they're
running
more
slowly.
Why
you
may
want
to
look
downstream
and
say
something
in
in
my
stack
is
running
more
slowly
now
than
it
was
before.
Show
me
everything.
That's
downstream
of
me
all
right,
so
we're
going
to
build
a
table
that
looks
like
this
go
back
into
grafana
and
go
back
into
the
demo
folder
and
swap
over
to
workshop
2.
B
A
A
Going
to
look
at
the
span
table
we're
going
to
filter
by
the
time
window.
We're
going
to
filter
by
this
dollar
service
ties
us
to
the
picker
for
the
service
name,
and
the
dollar
span
ties
us
to
the
picker
for
the
span
name,
we're
we're
gonna
get
the
trace
id
span,
id
parent
spam
id
service,
name,
spam
name,
the
duration
and
we're
gonna
count,
so
we're
gonna
start
the
counting
at
zero.
A
So
that's
where
our
recursion
starts,
and
what
we're
going
to
do
right
here
is
we're
going
to
take
the
span
for
each
iterative
step.
Look
at
the
previous
step
where
we're
in
the
same
trace
and
this
steps
span.
Id
is
the
parent
of
the
previous.
So
we're
walking
up
the
tree
right
and
what
we're
going
to
do
is
we're
going
to
increment
the
distance
by
one.
A
So
we
know
distance
one,
we're
directly
called
the
one
we
saw
we
selected
on
distance.
Two
was
the
parent
of
that
distance.
Three
is
paired
with
that
and
we're
going
to
pull
all
those
out.
We're
gonna
throw
out
distance
zero,
which
is
the
one
we
picked
and
we're
going
to
get
the
p99
percentiles
p95
percentiles.
A
A
A
A
A
A
So
we're
going
to
take
the
same
recursive
query
that
we
had
before
for
upstream,
but
we're
going
to
do
a
distinct
on
the
concatenation
of
the
service
name
and
the
span
id
to
give
it
an
id.
So
this
grafana
widget
needs
an
id
for
each
node
and
we're
going
to
use
the
service
name
and
the
span
name
as
titles
and
subtitles.
A
A
A
So
on
the
first
iteration
we
actually
don't
have
enough
information
to
draw
an
edge,
so
we're
gonna
start
with
a
null
child
service
name
and
a
null
child's
band
name
on
the
second
and
following
iterations.
We
know
the
prior
iteration
and
this
iteration,
so
we're
able
to
draw
a
line
between
the
two,
so
you'll
notice
that
we've
got
p
for
parent
we've
got
the
parent
service
name
and
the
parent
span
name
and
x
is
the
previous
iteration.
So
we've
got
the
child's
service
name
and
the
child
span
name.
A
So
we've
looked
at
the
last
five
minutes,
which
we
know
from
the
previous
dashboards,
probably
between
a
thousand
and
twenty
five
hundred
traces
and
we've
looked
at
all
of
the
distinct
traces
in
that
time
window
that
are
upstream
that
and
we've
looked
at
all
those
traces.
We've
found
the
ones
the
spans
that
start
in
the
the
digit
slash
and
we've.
A
A
A
So
now
we
have
a
dynamic,
real-time
map
of
not
just
our
service
dependencies,
but
our
operations
within
those
services.
So
it's
like
an
x-ray
of
of
all
the
services
at
once
right,
and
this
is
actually
an
easter
egg.
You
might
want
to.
You
might
say:
well
why
on
earth
is
the
lower
service
calling
the
digit
service
there's
zero
reason
why
it's
a
purposeful
bug
in
there
for
for
us
to
find
and
dig
through
and
try
to
go
fix.
A
Questions
cool,
okay,
so
let's
do
the
same
thing
for
downstream
we're
gonna
do
the
nodes!
First,
it's
the
same
query
as
before
we're
just
you
know.
We
swapped
that
one
relationship
to
go
downstream
instead
of
upstream,
so
you
can
go
over
to
the
the
dashboard
enable
or
edit
this
one
go
to.
The
first
query
enable
it.
A
A
A
A
That's
pretty
cool,
so
if
we
combine
the
previous
dashboard
in
this
one,
the
previous
dashboard
showed
us
that
there
was
this
performance
issue
and
it
showed
us
the
exact
operation
where
that
performance
issue
was
right.
Now
we
can
go
over
to
this
dashboard
and
we
can
filter
on
the
operation
that
we
identified
in
the
last
and
we
know
exactly
what's
upstream
of
it
exactly
what's
downstream
of
it.
So
we
know
if
we
optimize
this
thing,
what's
it
going
to
impact,
hopefully
for
the
positive.
A
A
A
A
Yeah
right
so
we're
doing
the
distinct.
Let's,
let's
go
back
to
the
query,
so
you'll
see
right
here,
we're
using
distinct
here,
so
we're
finding
all
of
the
distinct
paths.
So
it's
not
necessarily
the
case
that
any
one
request
would
follow
all
of
this.
It's
actually
not
the
case.
We
know
that
some
of
them
go
down
this
path
and
some
of
them
go
down
this
path,
but
we're
we're
building
a
picture
of
all
of
the
distinct
paths.
A
Yes,
yeah,
so
I
know
this
widget,
you
know
I'm
not
super
familiar
with
it.
I
know
there's
a
way
to
color
these
nodes,
so
I
think
you
could
compute.
You
know
the
throughput
or
the
latency
like
we
did
before
and
then
use
that
to
color
these
yeah
I'd
love
to
see
a
way
to
color
or
like
impact
the
size
of
the
edges
as
well.
But
I
don't
think
this
can
do
that
this
particular
ui
widget
yeah
right.
I
was
sending
some
feature:
requests
to
grafana.
A
A
We
don't
really
have
any
enterprise
features
and
yeah.
So
that's
what
this
is
about.
You
download
it
you've
got
this
data
in
the
database
and
unless
you're
you're,
you
know
strong
with
sql
and
have
some
imagination.
You
don't
really
know
what
to
do
with
it
right.
I
actually
built
this
for
myself
when
I
I'm
on
the
an
engineer
on
the
team
building
prom
scale,
and
you
know
I
needed
some
demo
data
to
play
around
with
to
make
sure
that
my
schema
worked
and
make
sure
these
queries
actually
worked
and
so
yeah.
C
A
It
also
supports
prometheus
metrics,
so
you
can
have
metrics
and
traces
forwarded
to
prom
scale
and
put
in
the
same
database.
So
what
I
haven't
done
in
this-
and
I
hope
to
expand
on,
is
let's
put
both
in
the
database
metrics
and
traces
and
now
with
sql.
You
can
correlate
the
two
and
build
something
even
better.
You
know
even
deeper
in
science.
A
A
C
A
Databases
they're
in
the
same
database
they're
in
different
tables
and
different
schemas,
although
the
schemas
may
change
so
kind
of
bring
them
together,
but
yeah
you'll
have
certain
tables
that
the
metrics
land
in
a
certain
tables
that
the
traces
land
in
and
then
based
on
the
metadata
that
you
attach
to
either
metrics
or
traces.
You
can
correlate
the
two.
A
A
That's
a
really
good
question,
so
the
question
is
around:
you
know
we
give
we
we
give
a
way
to
store
the
data
and
with
sql
we
give
a
way
to
query
the
data,
but
you
know
what's
what's
in
the
box
what's
out
of
the
box
right?
So
that's
that's
the
beauty
of
sql.
I
mean
it's
a
it's
a
turing
complete
language
for
data
analytics.
So
the
the
cool
thing
is
we're
giving
you
the
data
and
a
forum
with
a
language
to
go
with
it.
Where
you
can
answer.
A
C
C
A
A
Cool,
so
now
you
have
a
running
trace,
generating
playground
on
your
system.
Take
it
home
play
around
with
it.
If
you
have
any
questions,
timescale
has
a
community
slack,
you
can
jump
in
there,
there's
a
prom
scale
channel
and
find
me
or
anyone
else
in
there
with
a
little
tiger
logo.
Next
to
their
name.