►
From YouTube: Tempo Community Call 2021-05-13
A
A
A
Agent
to
allow
you
to
you,
know,
log
create
logs
and
create
traces
in
a
manner
that
allows
you
to
kind
of
link
those
through
grafana
traces
to
logs
and
logs
of
trace
kind
of
jumps
around.
We
currently
use
loki
internally
as
our
kind
of
index
into
tempo,
because
tempo
is
a
trace.
Id
only
store,
but
other
back
ends.
A
A
C
A
A
Let's
talk
about
07-0,
so
we
released
that
between
the
previous
meeting
and
this.
So
let's
go
over
some
headlines
here.
Some
of
the
some
of
the
more
important
features
that
we
added
there.
Let's
see,
I
have
the
giant
list
up
here.
There's
a
ton
of
bug
fixes.
I
won't
go
into
all
the
details,
but
if
you
review
that
there's
a
number
of
performance
improvements,
one,
notably
that
I
really
like
that,
we've
started
to
run
internally.
A
A
I
think
we
were
finding
our
internal
bottlenecks
are
more
related
to
disk
or
I
o
throughput
than
they
were
necessarily
the
kind
of
the
amount
of
cpu
remember
spending
the
adjusters.
So
we
went
with
that
trade-off,
but
for
different
people
this
might
be
different.
If
you
had
very
fast
ssd's
directly
connected
tempo
and
some
kind
of
like
a
bare
metal
environment,
then
it
might
actually
be
quite
a
bit
better
to
stick
with
non-compressed
wall
page
based
access
index
access
is
sounds
boring,
but
it's
quite
good.
A
I
think,
and
one
of
a
great
improvement
here.
So
previously
we
were
pulling
like
an
entire
index.
Is
one
big
blob
which
really
reduced
the
size
of
the
indexes
we
could
keep,
because
once
they
get
to
omeg
two
megs
three
megs.
It
was
a
huge
amount
of
data
to
pull
for
each
block,
but
we've
changed
it.
So
we've
organized
our
indexes
in
terms
of
pages
and
it'll
pull
256k
at
a
time.
A
Those
might
be
the
highlights.
I
don't
know
if
we
did
go
to
go
116.,
that's
probably
worth
mentioning.
I
don't
know
if
there's
anything
else
in
here.
That
is
particularly
interesting,
but
I
think
those
are
the
ones
that
I
was
most
interested
in.
Is
there
anything
else?
The
team
remembers
that
that
is
worth
discussing.
A
A
And
it
was
kind
of
an
annoying
thing,
mainly
because
it
was
just
an
extra
piece
that
you
had
to
run
and
it's
very
confusing
to
people
why
it
existed.
Excuse
me,
but
grafana,
7.5.x,
now
and
and
tempo
0.7.0
directly
support
each
other
and
there's
no
need
for
this
kind
of
intermediary.
It
makes
our
examples
much
more
simple.
A
I
think
some
of
our
examples
still
have
it,
but
we're
looking
to
remove
those
entirely
and
grifona
can
now
query
tempo
directly,
which
is
just
makes
everything
easier.
Frankly,
I
think
we're
still
going
to
keep
the
tempo
query
piece
around.
It
is
nice,
I
think,
to
have
a
different
way
to
query
tempo
than
directly
out
of
grafana.
If
you
don't
want
to
use
grafana,
but
we're
going
to
stop
using
it
and
kind
of
just
like
move
it
out
of
the
examples
and
probably
push
people
to
stick
with
grafana
directly
to
tempo.
A
Oh
yeah,
that
is,
we
used
that
to
do
a
migration,
so
we
had
to
move
one
of
our
tempo
clusters
between
kubernetes
clusters,
one
of
our
tempo
instances
between
kubernetes
clusters,
and
we
had
no
way
to
tell
an
ingester
to
just
completely
flush
all
blocks
to
the
back
end.
So
we
needed
that
we
needed
a
way
to
tell
and
gesture
completely
push
all
blocks
to
the
back
end
before
we
shut
it
down.
So
we
didn't
lose
any
data
and
that's
why
we
added
that
piece.
E
A
So
I
don't
think
it
would
be
too
difficult
to
add
a
config
option
to
the
adjusters
to
just
do
that
on
shutdown
like
that,
be
the
default
behavior
on
shutdown.
We
wouldn't
run
it
that
way,
but
for
people
who
are
in
an
environment
where
that
made
sense,
it
would
be
fine.
It
would
make
shutdown
take
a
long
time,
but
you
would
be
welcome
to
run
that.
E
A
D
There
is,
there
is
one
thing
where
the
blocks
that
would
be
flushed
won't
be
immediately
queryable
I
mean,
I
think,
that's
more
of
a
like
cost
benefit
like
it's
worth,
getting
the
stateless
ingestors
right
without
persistent
volumes,
but
you
can
have
maybe
gaps
in
query
according
that's,
you
know
something
that
we
would
hope
to
address.
You
know
if
we
change
the
way
the
blocks
are
pulled
and
things
like
that,
but
that
I
mean.
B
D
D
Just
for
the
queries
to
repo
the
blocks
like
if
they
pull
every
five
minutes,
there
would
be
that
you
know
the
adjuster
will
flush
the
block
and
then
go
away.
Normally
the
adjusters
hang
onto
the
blocks
and
you
know,
set
respond
to
queries
for
that
data
because
they're,
the
only
pod
that
would
know
about
it.
So
you
know
for
a
few
minutes
until
they,
you
know
repo
the
blocks
that
those
traces
that
were
just
flushed
wouldn't
show
up
yeah,
but
I
mean,
I
think,
that's
probably
worth
it.
D
D
D
A
Okay,
so
we've
always
supported
ingestion
of
zipkin,
but
we
don't
use
zipkin,
so
they
were
said
a
surprise.
I
suppose,
when
we
actually
did
have
a
customer
who
you
started
using
it,
but
zipkin
purposefully
duplicates
span
ids
between
the
client
and
the
server.
So
when
the
client
reaches
out
it
creates
a
span
and
it
passes
that
span
id
and
in
jager
and
open
telemetry,
a
new
span
is
created
on
the
other
side,
with
a
new
span
id
button
zipkin
it
duplicates
that
purposefully.
It
uses
the
same
span
id
across
as
on
both
sides.
A
A
D
Sure
yeah,
I
think
this
is
always
kind
of
fun
to
go
through
as
we
kind
of
work
on
making
tempo
more
performance,
scalable,
and
also
I
mean
we
get.
You
know
discussion
for
things
like
you
know
how
many
different
pods
or
different
components
should
I
run
and
it's
hard
to
say,
because
workloads
vary
so
much
across
clusters,
so
this
is
just
kind
of
going
through
an
internal
cluster
that
we
have,
which
is
our
monitoring
cluster
yeah.
D
So
we
have
plans
to
go.
I
mean
I
guess
our
goal
is
probably
three
or
five
million
spans
per
second
and
then
higher.
What
three
or
five
million
would
be
kind
of
like
the
next
step.
Probably
we
might
even
be
able
to
get
that
with
the
current
design.
It's
just
it's
not
maybe
worth
us
like
sampling
that
much
just
for
our
own
internal
testing
but
yeah,
so
just
to
get
a
sense
of
like
the
amount
of
pods.
D
D
They
cut
blocks
at
700
megabytes
and
that's
snappy,
so
that's
after
compression.
So
I
think,
before
compression
that
was
somewhere
around
three
gigabytes,
I
think
just
to
get
a
sense
of
how
often
data
is
cut.
I
think
each
ingestor
is
cutting
a
block
every
couple
of
minutes.
They
have
50
gigabyte,
persistent
disks
now.
The
reason
for
that
is
they
don't
actually
need
that
much
storage,
but
it's
in
order
to
get
some
enough
bandwidth
right,
so
the
bandwidth
for
the
disk
scales
up
with
the
size.
D
This
isn't
on,
like
g
or
in
google
cloud.
So
I
don't
think
we're
running
into
any
bandwidth
limits
right
now,
but
I
think
we're
close
right.
So
that's
probably
like
something
else,
we'll
look
at
so
compression
for
the
wall
was
really
important
there,
because
that
cuts
down
on
just
the
how
much
data
is
being
written
to
disk
at
once
for
queries.
So
we
we
have
25
queriers
and
two
front
ends,
so
the
queries
are
kind
of
like
the
pods
that
are
doing
the
work
in
the
front
ends
or
serving
the
api.
D
This
was,
I
just
got
here
our
query
p90
for
looking
at
the
traces
somewhere
between
two
and
five
seconds.
So
most
of
the
time
it's
two
seconds,
but
we
have
occasional
long
tails
over
time
or
that
are
around
five
seconds.
P99
is
is
worse,
I
think,
maybe
it's
more
like
10
seconds,
I'm
just
going
to
put
that
in
here.
D
That
was
something
like.
So
maybe
something
like
that
we
had
36
compactors.
I
didn't
get
any
of
the
like
configuration
options
and
I'm
not
sure
that
we
really
need
that
many,
but
we
did
find
that
it's
good
to
have
that
many.
I
think
compacting
is
still
something
that
we're
working
on
tuning
and
determining
the
proper
number
of
yeah.
A
The
compactors
are
relatively
small
they're
like
a
few
gigs
of
memory,
each
and
one
cpu,
so
they're
not
doing
a
lot
of
work.
I
think
most
of
our
pieces
are
smaller
compared
to
say
cortex
or
loki,
but
I
don't
really
know
also.
I
had
no
idea
with
36,
I
would
guess
like
10
less
than
that.
Maybe
I
should
have
known
that.
D
I
think
you
know
when
we
increased
the
sampling
rate
to
1
million
spans
per
second.
The
block
list
was
growing
quite
a
lot.
I
think
it
was
higher
than
this
for
a
while,
and
so
I
would
say,
compactors
are
still
in
in
progress
thing
being
tuned
right.
There
was
the
compaction
time
window
that
we
reduced,
I
think
to
15
minutes
things
like
that.
Yeah,
the
maximum
block
size
we
increased,
and
then
we
we're
running
quite
quite
a
lot.
I
would
say
at
any
given
moment.
D
Some
of
them
are
idle,
though
yeah
yeah.
So
for
totals
that's
around
210
cpu.
I
didn't
give
them
one
moment
around
353
gigabytes
of
memory,
and
that's
there's
more
than
just
this.
That's
in
that
entire
kubernetes
name
space,
there's
a
couple
things
else
that
we
run
like
just
sidecars
and
things,
but
that's
the
total.
D
A
Thanks
definitely
at
a
million
space,
a
second
we're
actually
at
1.5
million
for
a
while,
sometimes
it's
hard
for
us
to
track.
What's
going
on
in
our
different
applications
to
know
why
we're
growing
or
shrinking.
So
for
a
couple
days
there
we
were
sitting
at
1.5
million
and
it
was
fine
also,
I
always
cringe
at
the
210
cpu,
which
just
seems
like
so
much.
The
good
news,
though,
is
we're.
A
Gonna
reduce
that
quite
a
bit,
there's
a
prn
now
in
maine,
although
it
broke
everything
that
significantly
reduces
the
the
cpu
usage
of
the
ingesters
about
70
reduction,
which
is
really
exciting
because
of
the
200
cpus,
I
think
like
150
of
them
are
ingesters
like
it's,
almost
all
investors
so
or
the
majority
is
just
so.
This
will
be
a
huge
reduction,
and
I
wanted
to
get
this
in
before
we
started
pushing
to
that
three
to
five
million
marty
was
talking
about.
D
Oh
yeah,
I
forgot
to
cover
this
part
here
just
just
quickly.
Okay,
so
we
have
two
retention
and
that's
about
56
terabytes
of
compressed
data,
25
billion
traces
yeah.
Yes,
you
know
I
was
thinking
one
metric
that
I'm
not
sure
tempo
has
that,
but
that
would
be
interesting
was
how
many
live
traces
are
on
each
ingestor
at
a
given
moment.
I
think
that's
another
thing
where
comparing
across
workloads
would
be
useful,
yeah
just
mentioning
it.
A
D
A
F
F
A
I
also
checked
the
link
to
the
reduction
pr
into
the
dock.
There's
some
really
juicy
graphs
in
there
on
our
little
test
bed,
I
had
point
one
cpu
for
the
ingestors
and
then
we're
down
to
almost
0.25,
so
just
a
huge
reduction
in
cpu
usage.
Basically,
what
we
did
is
not
marshall
proto,
so
we
marshal
at
the
at
the
distributor
level,
and
then
we
pass
just
bytes
like
blobs
of
bytes
to
the
ingester,
which
stores
it
directly
into
the
back
end,
and
we
only
re-marshal
it
if
we
have
to
like.
A
If
you
query
it
basically,
so
we
think
that's
going
to
be
a
huge
improvement
now
it
basically
broke
a
lot
of
stuff
that
we're
working
on
right
now,
and
I
really
wanted
to
have
this
in
our
operations
cluster
today
for
this
meeting,
but
it
didn't
work
out.
So
I
think
the
team's
deciding
between
one
of
two
pass
going
forward
and
we
have
prs
up
for
both
of
those
but
we'll
figure
that
out
cool
marty
you're
up
next.
Can
you
tell
us
about
exemplars.
D
Sure
yeah
I
mean,
as
far
as
I
understand
like,
I
think
they
work
well
in
755,
and
so
that's
the
version.
I've
worked
with
I'm
not
aware
of
what
the
changes
are
in
eight
though
yeah
cool,
oh
yeah,
let's
do
on
a
previous
community
call
we
kind
of
walked
through
exemplars
and
kind
of
a
demonstration
of
them.
So
I
think
we
could
look
that
up.
If
anyone
is
interested,
I
guess
I
would
say
they're
kind
of
still
experimental
in
in
progress.
D
So
what
that
inc?
What
is
new
is
in
previous
227
is,
it
will
now
store
the
so
it
always
would
scrape
the
examples,
but
now
they
will
also
be
written
to
the
write
ahead
log
and
what
that
does
previously.
The
examples
were
purely
in
memory.
So
if
the
pod
was
was
lost
or
restarted,
the
data
would
be
gone,
but
in
the
right
head
log
I
was
able
to
recover
some
of
the
data
and
I'm
not
sure
I
can't
remember
exactly
how
long
those
right
ahead
logs
are
retained,
so
that
they're,
not
permanent.
D
D
Additionally,
also
we'll
now
send
the
exemplars
to
a
remote
right
target,
and
so
the
goal
for
that
is
to.
There
are
other
tools
that
can
do
this,
but
currently
we're
working
on
receiving
that
data
in
cortex,
and
so
that
would
be
meaning
that
we
are
working
on
the
ingest
and
query
path
for
exemplars
and
cortex,
so
that
would
be
a
downstream
target
of
prometheus
premier.
D
This
would
be
running
and
scraping
the
exemplars
in
your
cluster
and
then
sending
it
to
a
cortex
back
end
yeah,
let's
see
so
I
was
mentioning
seven,
five,
five
and
so
for
querying
exemplars.
If
you
have
prometheus,
you
can
query
example:
ours:
I
would
go
with
seven
five:
five.
There
was
a
small
like
there
was
support.
Previously,
I
think
753
is
where
it
became
like
the
stability
was
there,
but
I
think
in
755
there
was
a
small
ui
fix,
so
I
think
I
would
I
would
go
with
that.
D
One
too
and
another
thing
I
think,
mario,
I
think
we
were
talking
about
this
recently.
So
another
thing
in
progress
is
exemplar,
supporting
grafana
agent
right
and
so
grifana
agent
makes
use
of
prometheus,
and
so
now
that
this
has
been
merged
and
released,
we
can
bring
that
functionality
into
grafana
agent,
and
that
will
also
be
able
to
scrape
and
remove
right
examples,
and
I'm
not
sure
if
that's
actually
yeah
do
you
want
to
talk
about
that
for
a
second.
G
D
A
Cool,
let's
cool
thanks.
So
member
list
is
next.
We've
had
some
discussion
on
memberless.
I
was
within
seconds
of
ditching
it.
In
fact,
I
told
the
team.
I
was
going
to
ditch
it,
but
we
stuck
with
it
because
the
community
really
wanted
it.
They
like
it,
and
so
we
have
since
made
some
tweaks
to
our
settings
and
found
a
lot
of
stability.
So
I'm
glad
people
pushed
me
to
keep
it.
A
I
was
about
to
give
up,
but
what
you
see
there
in
the
dock
is
our
current
settings
and
since
we've
had
these
settings,
our
issues
with
like
the
common
problem
was
like
unhealthy
compactors
compactors
just
sitting
in
there
forever
they're
impossible
to
forget.
We
couldn't
figure
out
why,
since
we've
made
these
changes,
we've
not
seen
that
at
all.
A
I
don't
think
I
can't
even
remember
the
last
time
we
had
to
sit
there
and
smack
forget
on
a
compactor
page,
praying
that
it
would
go
away
and
getting
really
frustrated,
and
you
know
going
to
extreme
measures
to
get
rid
of
it.
So
I
would
recommend
these
settings
if
you're
using
member
lists,
I
think
we
found
them
very
stable
and
please
give
us
feedback
if
you
don't
so,
we
can
continue
to
work
on
these.
A
G
G
G
G
G
So
yeah
these
logs
are
locked
with
a
label
which,
in
this
case
is
template.
You
can
override
that
and
then
you
can
select
any
of
the
three.
For
instance
we
can
set
for.
Yes,
let's
see
what
spans
are
being
processed
in
the
agent
and
there
you
have
there,
you
have
it.
You
see.
The
logs
are
just
a
pair
of
key
value
that
keep
key
value
pairs
which
describe
tax
properties
from
the
from
these
pants
and
then
what
is
cool
is
that?
G
Well,
sorry,
if
you're,
actually,
my
my
keyboard,
I'm
one
of
these
people
with
mechanical
keyboards,
that
you
can
change
like
the
interesting
searches
for
these
traces
that
you
couldn't
do
otherwise,
so
I'm
just
formatting
the
log
line
and
I'm
searching
for
traces
which
have
a
duration
longer
than
one
second
and
then
we're
here
the
field.
Then
maybe
I
want
to
search
that
also
have
also.
G
G
E
G
Sure
cool,
yeah,
so
very
powerful
tool,
all
right,
and
the
next
feature
is
that
we
added
remote
white
to
spawn
matrix.
So,
as
we
showed
last
week,
we
introduced
metrics
generated
from
the
spans
again
in
the
agent
or
in
this
tracing
pipeline,
but
how
they
were
being
exported.
The
metrics
was
just
by
the
primitive
supporter
from
the
auto
collector,
which
basically
just
exposed
a
matrix
handler,
which
you
would
have
to
scrape
so
another
option
that
we
have
introduced
is
instead
of.
G
Exposing
an
exporter,
maybe
we
can
use
the
remote
rights
capabilities
of
the
agent.
So
now
you
can
use
a
prometheus
instance
within
the
agent.
It's
like
the
very
same
case
as
for
unlucky
instance
from
the
previous
feature
and
directly
pushed
to
yeah
any
prometheus
like
backhand
like
prometheus,
cortex
or
anything
yeah
and
then
finally,
well,
we
have
just
mentioned
it
like.
A
A
A
A
Cool
well,
if
that's
it,
we
can
go
ahead
and
call
it
we're
a
little
early
but
no
big
deal.
Everybody
enjoy
your
whatever
week.
It's
left
and
we'll
see
in
a
month.
I
suppose
and
yeah
feel
free
to
jump
in
on
the
slack
channel
or
add
issues
to
the
repo
there's
a
community
forum
as
well,
to
ask
questions,
and
we
do
our
best
to
answer
all
these
quickly
and
to
engage
people
through
all
of
them
so
feel
free
to
kind
of
connect
to
us.