►
From YouTube: 6. #everyonecancontribute cafe: Grafana Tempo
Description
Blog with URLs: https://everyonecancontribute.com/post/2020-10-28-cafe-6-grafana-tempo/
A
So
yeah
hello,
we're
gonna,
look
into
grafana
temple
today.
This
is
something
we
discussed
like
30
minutes
ago
because
it
was
released
yesterday
or
announced
yesterday
and
I
don't
know
how
to
start
exactly,
but
maybe
nicholas,
do
you
want
to
start
in
and
share
the
blog
post
you've
been
reading
about
it.
B
And
then
you're
going
into
the
preparation
of
temple
yeah,
it
could
start
so.
My
name
is
nikka
smith,
I'm
working
as
senior
developers,
engineer
phosphority,
currently,
I'm
mostly
doing
their
blockchain
stuff
related
operations
and
helping
customers
a
lot
of
with
compliance
related
services.
Yes,
that
would
be
for
my
side,
so
I
won't
hand
over
to
who
I
think
I
will
take
alejandro.
D
A
Yes,
of
course,
I'm
working
for
comedia,
we
have
our
own
enterprise
cms
system
and
I'm
working
on
the
team,
that's
responsible
for
the
cloud
offering
of
the
cms
service
and
yeah.
The
main
of
my
focus
is
on
monitoring
and
the
automation.
So,
basically
the
whole
monitoring
stack
around
krapana,
prometheus
terraform
and
python
yep.
That's
it
okay.
A
From
austria,
but
I'm
living
in
germany,
I'm
the
crazy
one
who
knows
everyone
in
this
round.
I
do
love
observability
and
monitoring
yeah,
and
we
kind
of
found
out
that
there
is
something
going
on
or
something
new
this
week,
because
grafana's
observabilitycon
happens
and
we
learned
about
grafana
temple
and
was
like
yeah.
Maybe
we
wanna
just
try
it
out
and
before
the
session
nicholas
shared
a
blog
post
with
me-
and
probably
it
just
like
makes
sense
to
talk
about
a
little
bit
about
what
graffana
tempo
is.
B
C
B
Yep,
so
really
talking
a
little
bit
about
grafana
temple,
graffana
temple
is
a
new
services
was
spin
up
from
people
from
grafana
labs.
Mostly
it
will
help
you,
like
you,
see
in
the
tea,
so
the
tea
is
mostly
related
to
tracing.
So
a
trace
is
a
time
span
where
an
action
happens
mostly
so
they
can.
For
example,
it
tries
to
be
something
like
at
an
h,
p
call
so
in
distribute
systems.
B
B
So
if
you
are
coming
back
from
a
normal
programming
language
or
from
a
program
range,
at
least,
you
know
probably
also
traces,
because
you
know
the
familiar
concept
of
stack
traces.
So,
where
you
can
see
in
your
compiled
language,
you
can
see
the
stack
trace
where
you
can
see
which
function
was
called,
which
time
zone
and
with
which
values
mostly
in
distributed
systems.
We
have
a
bigger
problem
to
solve
because
it
doesn't
run
only
in
one
machine.
It
can
be
run
on
multiple
machines
and
for
that
are
mostly
traces
in
this
common
pattern.
B
B
B
A
B
Need
to
press
record
all
right.
Retort
is
always
enabled
when
you're
on
this
always
so,
and-
and
here
we
can
currently
also
see
trace
from
the
browser.
Mostly
so
we
can
see
in
the
timestamp
when
it
was
happened,
and
later
we
can
see
the
full
graph
when
it
was
finished
all
the
requests,
mostly
this
shouldn't
be
doing.
On
the
front-end
side.
B
We
are
more
interested
now
with
tempo
and
the
back-end
side
when
one
service
towards
another
service,
mostly
so
these
are
the
trace,
and
then
we
can
digging
more
into
the
topic
of
how
our
application
is
currently
working,
because
we
have
currently
in
the
observability
on
space.
We
have
three
truths
that
we
mostly
using
to
analyze
all
of
this,
so
we
have
lots
for
dating
when
the
event
has
happened
so
which
data
was
sent
mostly
there.
B
Then
we
have
metrics
for
counting
stuff
and
in
the
last
point
we
have
mostly
traces
to
visualize
which
hack
action
was
done
and
also
in
this
step
of
this,
which
value
has
it
mostly,
as
you
probably
know,
some
familiar
tracing
to
it.
So
there
are
some
tools
like
jagger,
for
example,
or
the
other
two
that
was
the
first
papers
that
comes
from
also
google,
I
think
open
siptin,
or
was
it
uber
or
one
of
them?
B
B
One
trade-off
of
this
that
you
need
to
take
is
mostly
because
you
don't
engage
in
the
full
trace,
so
you
can't
search
about
it
mostly.
You
need
at
least
now
the
trace
id
to
see
where
the
trace
happened.
It
is
the
main
difference
between
dragger
and
also
our
tempo,
and
also
the
short
introduction
a
little
bit
about
tracing
mostly.
B
A
I
think
it's
really
great.
I
just
found
an
old
presentation
of
mine,
which
I
need
to
share
with
you
as
well,
in
the
chat
which
I
did
like
explaining.
What
open
tracing
and
open
metrics
are.
This
was
kind
of
the
the
first
ideas
of
like
how
to
how
to
explain
what
it
is
and
also
like
saying:
hey.
We
want
to
maybe
have
a
combined
solution.
A
How
does
it
feel?
Because
you
also
mentioned
yeah
like
jaeger,
tracing
and
open
tracing,
was
just
a
little
bit
before
that,
and
I
think
it's
called
open
sensors,
which
was
the
the
implementation,
and
then
there
was
kind
of
the
announcement
last
year.
I
think
about
open
telemetry,
which
can
replace
jaeger,
but
it
won't
so
it
will
coexist,
and
people
started
to
like
with
integrations
and
saying
hey.
A
A
Be
it,
for
example,
when
you
run
ci
cd
deployment
and
you
want
insights
into
the
application
stack.
The
call
stack
the
timing
points.
How
long
does
an
sql
query?
Take
one
node
in
in
the
kubernetes
cluster,
for
example,
is
fast,
the
other
one
is
not,
and
these
are
all
the
insights
you
can
get
out
of
that.
The
perfect
example
is
tracing
my
sequel
connections,
but
there
are
lots
of
others,
and
I
need
to
be
switching
back
to
the
blog
post,
which
I
closed
somewhere.
In
my
many
tabs.
A
B
B
Then
you
have
a
full
time
so,
for
example,
your
operation,
at
least
for
example,
creating
a
user
could
take
two
seconds
and
to
see
the
full
spend.
Probably
you
see
only
the
estuary
takes
at
least
from
all
your
from
all
your
operations
take
at
least
one
and
a
half
seconds.
Mostly,
then
you
see,
okay,
you
need
to
look
only
on
the
sql
query
and
improve
this.
Instead
of
looking
in
probably
over
engineering,
the
performance
of
your
applications
so
to
see
the
function,
get
faster,
chord
and
so
on.
B
This
would
be
one
of
the
fellow
mains,
mostly.
What
is
more
also
interesting,
what
is
really
in
the
blog
post
from
dufam?
He
saw,
let
me
think
where
it
was
so
mostly
traces
is
only
one
part
and
when
you're
going
down
a
little
bit
in
the
blog
post,
mostly
drawing
when
you're
entering
tempo
yeah
spur
out
for
id.
So
because
mostly
yeah
driving
a
little
bit
further
down.
A
C
B
B
B
A
In
the
morning,
you
want
to
see
everything
which
kind
of
influences
the
current
state,
and
I
think
it's
one
of
the
things
it's
it's
critical
to
see.
Okay,
there's
a
trend
and
something
is
going
on
and
by
the
way,
these
are
the
log
lines
or
these.
These
are
the
the
additional
insights
you
want
to
map
into
that
and
tracing
is
just
another
layer
or
another
data
format
which
needs
to
be
mapped
into
this.
B
Liked
this
blog
post,
mostly
when
we
talking
about
observability,
because
it
explains
it
so
easy-
mostly
to
see
where
currently
what
are
metrics,
what
are
trace
traces
and
where
correlate
to
each
of
them.
So
because,
mostly
we
have
the
three
parts
in
here
also
the
same
like
doing
for
tempo
and
mostly
when
you're
going
now
down
in
the
blog
post.
It's
so
we
have
the
three
sections
so
there's
only
the
events,
so
the
retrest
stop
going
a
little
bit
further
down
yeah.
B
So
this
is
the
most
interesting
graphics
that
I
know
about
this,
and
here's
typically
also
to
the
breast
they've.
Also
edit,
is
the
range
where
you
have
a
high
volume
of
data
that
you
need
to
store
and
a
low
low
volume.
So
that
means
probably
everything
from
us
knows:
okay,
we
want
to
implement
logging.
So
mostly
you
get
some
expertise
from
consultants
from
excellent
transactions.
They
say
you
need
to
set
up.
B
Elasticsearch
elasticsearch
is
really
high
data
intensive
workload
and
also
it
needs
to
be
big
resource
requirements
instead
of
from
computing,
so
memory
and
cpu
and
you
need
to
set
up
distribute.
So
it's
a
huge
impact,
mostly
the
value
you
get
mostly
from
loading
is
for
most
people
or
not
for
most
people,
but
for
some
people
using
it
not
enough
to
get
the
value
out
of
vlogging.
B
So
you
have
mostly
a
bigger
loading
system
than
your
whole
application
state
so
and
for
developers
or
for
mostly
people
that
want
only
to
see
how
it
currently
to
relate
to
each
other
is
mostly
probably
a
good
way
that
you
only
have
in
metrics
and
only
the
traces
that
you
can
see.
The
aggregation
and
laureen
can
be,
and
additionally,
I'm
stepped
to
that
or
you
can
see,
for
example,
typically,
the
most
people
have
mostly
metrics
and
loading
stuff,
but
right
now
not
tracing.
B
So
there's
a
dependent
step
of
this,
and
what
currently
tempo
does?
Is
they
reduced
a
little
bit
the
volume
of
the
traces
in
terms
of
the
storage
that
is
required
to
save
all
these
traces?
Because
they're,
saying
hey,
you
can't
do
a
full
search
on
all
our
traces.
We
were
storing
only
the
ids,
mostly
in
the
budgets,
so
you
need
to
know
what
you
search
michael,
already
treated
a
cruel
post
about
that.
Where
was
it
also
explained
from
bjorn
plunker?
I
think
he
was
told
he's
also
one
of
the
creators
of
tenors
well.
B
A
B
A
B
B
Now
it
was
in
the
pulse,
so
it
doesn't
matter
in
the
end.
So
mostly
so.
This
is
mostly
the
cruel
part
about
temples
that
we
are
now
able
to
use
in
when
we
are
using
the
most
famous
telomer
related
framework,
so
open
terminally,
open
tracing,
I
don't
know
the
other
icon.
I
know
I
know
only
jager,
but
this
is
17
right
now.
B
B
But
yeah
and
then
you
can
indexing
or
saving
them
into
tempo,
and
then
you
can
also
having
the
correlation
on
rafana
but
yeah.
That
is
the
main
idea
behind
that,
mostly
instead
of
saving
them
and
fully
searchable,
you
need
only
to
know
the
index
id
and
you
will,
when
you
have
a
full
tooling
stake
with
prometheus
later
with
phase
suppliers.
B
You
have
the
full
angle
of
all
the
support
that
you
probably
need
to
having
a
good
observability
on
a
lower
cost
of
infrastructure,
but
also
scalable.
So
if
the
big
important
point
about
it
is
they
say:
okay,
we
want
to
make
things
easier,
but
we
can,
because
of
that
we
can
easily
scale
them.
So
we
simplify
stuff.
We
have
not
the
rich
feature
to
set
mostly
so
like
when
you're
doing
tracing
with
zipline.
B
So
because
then
you
can
do
full
searches
on
the
traces
and
yeah,
and
this
is
the
main
idea
behind
that.
So
they
say:
okay,
we
have
a
little
small
set.
It's
like
the
same.
When
we
started
with
floaty
in
loki,
we
we
get
only
jason
or
indexing
jason
fights
mostly
well.
Now,
it's
not
innate
sin.
You
can
pass
json
fights,
mostly
it's
only
available
since
loki
to
zero.
B
A
You
can
have
a
look.
It's
it's
a
lot
to
unpack,
so
we
had
loki
in
version
1
1.0,
which
was,
I
think
last
year,
where
you
could,
which
use
the
same
format,
basically
the
same
format
for
metrics
and
logs.
So
you
could
just
fire
up
what
was
it
metric,
yeah
promisius
query
basically
was
looked
kind
of
the
same
like
loki
for
logs,
which
made
the
interface
in
grafana
way
more
easy,
and
you
didn't
need
to
learn
a
new
language
back
then,
okay,
examples
with
traces.
A
This
is
coming
up,
so
this
is
currently
in
development.
I
think
something
is
coming
next
week
in
that
regard,
but
I
could
be
mistaken,
yeah
and
temple
for
traces
and
blocks.
The
thing
is
how
to
try
this
out
now.
I
think
I
have
the
was
it
this
one?
No,
I
have
the
getting
started,
documentation
open
somewhere.
A
B
A
D
A
So
I
believe
that
in
the
in
the
trace,
which
has
like
the
logs
or
the
tags
inside,
there
is
just
like
in
an
indexed
number
or
some
detail
inside,
but
I'm
still,
which
one,
which
is
the
right
one.
This
is
the
right
one.
I'm
still
trying
to
understand
how
this
how
this
works
together.
I've
also
seen
that
tempo
has
a
an
open,
telemetry
exporter,
which
means
you
could
can
connect
it
there
as
well.
A
A
A
B
So
literally,
I
have
something
like
your
doctor
currently
running.
Probably
we
had
enough
enough
room
or
cpu.
We
will
see
if
it
breaks
or
not
mostly
okay.
So
first
up,
we
need
to
spin
up
the
tempo
battery,
and
so
for
that
we
create
a
simple
docker
network.
So
that
means,
instead
of
so
that
our
network
is
only
a
bridge.
So
when
you're
typing
docker
network
ls
seeing
the
default,
so
we
creating
our
own
network
for
the
tempo.
B
B
Now
this
loads
for
configuration,
so
this
is
the
configuration
file
mostly
so
it's
like
looking
a
little
bit
like
low-key,
so
with
the
injustice
to
collect
all
the
data
or
or
getting
all
the
data
collecting
them,
and
the
compact
does
the
job
of
doing
making
the
traces
smaller
after
one
time,
so
it's
b,
compactor
so
literally
and
for
storage.
Mostly,
we
will
save
it
on
a
file
right
now,
not
in
a
bucket.
So
it's
okay,
so.
B
B
B
C
B
Sometimes
now
we
will
see
where
they're
using
it
or
mostly
outside
to
anita's.
I
will
probably
I
need
to
open
a
second
page,
no
right,
not
yet
okay.
So
then
we
need
to
spin
our
rt.
So
now
what
we
did
is
to
we
started
the
back
end
so
little.
We
know
we
need
an
application
that
can
three
tempo
for
the
traces.
So
we
will
spin
up
the
temple
query
container
for
that,
so
downloading
it
and
starting
it.
So
probably
we
can
see
it
runs
on
a
different
part.
B
Okay,
so
mostly
it's
not
so
interesting,
it
uses
the
same
port
like
loki,
also
using
for
the
interesting
or
communication
part
but
they're
using
for
communication
they're
using
grpc
mostly
to
do
this.
So
that's
why
it's
called
gpc
storage
button,
and
now
we
should
able
to
see
something.
So,
let's
I
will
make
up
a
new
window.
B
B
D
B
A
B
B
A
B
B
B
B
Okay,
so
now
we
need
to
spin
up
everything:
okay,
all
right,
we'll
also
download
a
deep
dive
in
the
time.
So
what's
the
conflict
load
lights,
so
the
tempo
source
that
we
already
spin
up
is
the
back
end.
Mostly
I'm
gonna
make
it
a
little
bit
higher,
so
the
menu
will
be
probably
later
for
storing
all
this
stuff
temperature
is
for
jagger
ui,
mostly
the
synthetic
load
generator
is
an
instrumented
application
that
generates
load
and
we
have
prometheus
and
grafana
for
visibility,
mostly,
I
think,
and
how's
the
password
for
rafana.
B
C
B
D
A
A
B
A
B
A
D
D
B
A
B
And
here
we
can
see
like
I
explained
so,
where
the
service
trolled
it
so
how
many
times
in
it,
probably
okay,
so
I
will
change
my
windows
a
little
bit.
Give
me
one
minute,
except
we'll
see
that
so
you
can
see
a
little
bit
more.
Currently,
we
see
from
which
instance,
we
have
some
meta
informations
in
terms
of
text
and
process.
B
B
Yeah
yeah,
it
would
be
also
right
example
for
rookie,
okay,
and
so
this
is
literally
to
saving
all
the
stuff,
but
interesting
point
would
be
now
where
it
got
saved
literally
so
now,
like
I
said
before
archery,
so
we
have
also
an
object
store
and
this
time
it's
trolled
menu
menu
is
mostly
known
as
one
of
the
famous
object
stores
that
we
used
before.
So
it
was
one
of
the
first
implementations
that
implemented
the
s3
protocol.
B
And
it
has
a
lot
of
performance
regarding
it's
also
for
bit
routing.
If
you
want
to
access
it
and
you
can
do
a
lot
of
stuff,
this
is
literally
an
object
store
on
s3
base,
an
object,
store
yeah.
So
if
you're
using
ceph
or
something
else,
probably,
then
you
don't
need
it
directly
because
it
will
be
built
from
your
application
provider
so
from
your
own
search
provider.
B
But
when
you
have
idea
behind
menu,
is
that
you
having
a
storage
on
blocks
mostly
on
the
sides
that
you
can
easily
run
and
stay
mostly?
It's
also
written
bro
yeah,
and
now
we
need
to
log
in
let's
draw
tempo
and
super
secret.
I
will
copy
that
arty
interesting.
So
now
we
currently
can
see
here
on
the
data
that
things
that.
B
B
B
From
something
like
this
yeah
true,
so
literally
probably
you
have
also
the
options
to
have
multi
tenancy
so
that
we're
saving
for
each
traces
can
be
70
different
directories,
and
it
shows
us
some
information
about
what
is
the
starting
idea.
I
think
this
is
the
so
it's
like
in
sharding
concept,
mostly,
I
would
assume.
B
Id
no,
it's
not
atrocity
it's
something
else.
So
here's
here
I
know
all
our
stuff
is
safe,
so
all
our
traffic
generated
was
generated.
Probably
when
we
know
wait,
it's
it's
the
same
problem
because
I
talked
already
okay.
You
can't
do
it
by
searching
it
because
they
are
not
directly
indexed
in
terms
of
you
can
have
in
your
full
look.
A
A
B
B
A
B
C
B
A
I
think
I
think
you're
right
when
you
didn't,
when
you
did
not
start
yet
with
jaeger
or
maybe
like
open
sensors,
open
tracing
open
trinometry,
it's
a
good
way
to
get
started
easily.
So
you
have
something
where
you
see
something,
and
it's
like
beautiful
in
your
current
monitoring,
because
you
probably
use
grafana
already
with
prometheus
or
influx
db
or
whatever.
A
One
thing
I
I
did
within
with
adding
tracing
to
an
application
was
I
started
with
timing
points,
so
I
was
kind
of
measuring
how
long
a
config
compiler
took
and,
like
saying
hey,
I
want
to
probably
have
a
heat
map.
No,
it's
another
heat
map.
It's
a
call
stack
heat
map
would
a
flame
graph.
This
is
what
I
was
looking
for
to
see
the
call,
how
many
how
often
functions
are
being
called.
A
This
is
another
story
of
the
game,
but
kind
of
how
long
specific
things
take,
and
this
is
rather
easy
to
instrument,
or
you
are
kind
of
adding
if
you're
using
object,
oriented
programming,
you
are
adding
something
to
the
constructor
and
the
destructor
to
measure
the
lifetime
of
objects.
For
instance,
it's
like
it's
a
different
way
than
to
use
tracing,
but
it
could
work
as
an
example
to
get
things
going
rather
easy.
B
B
C
D
C
D
That's
the
tracing
stack,
the
fuse
deck
which
you're
seeing
in
jagger
or
something
like
that,
and
you
can
load
something
into
it
and
it's
a
really
easy
chase
format.
So
you
can
generate
stuff
and
put
it
into
this,
so
you
can
record
it,
but
you
can
press
load
and
then
and
get
this
information
from
different
stuff.
It's
it's
something
which
we
use
the
last
couple
of
months
and
it's
really
cool
because
you
have
an
easy
format
to
visualize,
something
which
is
only
happening
and
local
node,
for
example,.
A
B
A
C
A
B
A
Not
that
much
yes,
you
still
have
one
hour
to
go.
Okay,
that
is
the
meal.
No,
I
think,
probably
we
are
too
early
to
like,
say
hey.
Where
is
some
code
which
we
can
use,
but
I've
just
peeked
into
the
issue
tracker,
and
there
are
some
recommendations
about
adding
microservices
example
and
improve,
spend
red,
limiting
and
help
tests
and
and
even
more
things
which
are
being
dealt
with.
So
I
kind
of
expect
when
everyone
is
like
resting
from
observabilitycon
at
its
grandfather.
A
A
A
We
can
also
look
into
git
port
with
gitlab's
gdk.
This
was
also
one
of
the
ideas
we
could.
We
could
look
into
honeycomb.
This
is
something
I
also
wanted
to
do
for
a
while
in
the
english
coffee
chat
like
trying
this
out.
Maybe
combining
the
bee
lines
from
honeycomb
and
open
telemetry
with
tempo,
I
don't
know,
would
also
be
a
positive
possibility.
A
I
think
that
tracing
and
generally
speaking,
monitoring
and
observability
is
a
hot
topic,
and
this
is
getting
speed,
so
I
would
say,
let's
rest
a
little
and
next
week
we
will
jump
into
whatever
is
possible
and
if,
if
marcel,
can
like
prepare
something,
maybe
have
it
ready
in
a
markdown
format,
so
you
can
easily
create
a
blog
post
out
of
it.
This
would
be
nice
if,
if
this
works
for
you
just
in
case.
A
I
may
have
some
more
time
so
I'll,
let
you
know
if
I
make
it
until
next
week.
Yeah
just
just
take
your
time.
It's
no
no
pressure
intended.
If
it's
not
ready,
we
totally
find
something
else
to
try
out,
because
we
have
so
many
ideas
around
and
we
have.
We
have
nicholas
who
loves
to
create
to
do
live
things
so.