►
From YouTube: Tempo Community Call 2021-08-12
Description
Discussion of Grafana Tempo news.
- Version 1.1
- Update on Search
- New Maintainer
A
A
Conrad
has
been
with
us,
for,
I
guess
a
bit
over
three
months
and
has
just
done
great
work
in
tempo.
He's
been
an
agent
he's
been
doing
operational
stuff
he's
been
adding
code
he's
been
adding
features,
he's
been
instrumental
in
search
and
participated
with
that
in
the
hackathon
and
he's
even
contributed
to
some
of
our
upstream
dependencies
in
open,
telemetry
he's
kind
of
just
been
everywhere
everywhere.
He
sees
any
kind
of
need,
he
jumps
in
and
he
gets
some
work
done.
It's
been
fantastic
having
him.
So.
B
No,
I
mean
I
had
a
like.
You
know
a
really
good
time
working
on
temple,
it's
really
fun
and
yeah.
I
think
we
have
a
very
good
foundation
right
now,
so
I'm
looking
forward
to
everything
we'll
we
will
be
building
on
it.
So
we
search
and
we're
doing
you
know
all
these
reliability
improvements
so
yeah,
it's
really
cool
to
be
maintainer.
D
D
You
know,
because
if
you're
running
systems-
and
you
want
to
know
what's
going
on
so
visualization
and
observability
all
really
matters
in
that
and
so
yeah-
I'm
super
excited
to
be
here
after
you
know
using
grafana
for
so
long
and
I
love
all
the
tool
sets
so
and
yeah
I'm
just
getting
started
with
distributed
tracing.
So
tempo
is
gonna,
be
a
lot
of
fun.
I'm
looking
forward
to
it.
A
E
A
A
That's
right:
okay,
okay,
let's
swap
these
real
fast.
It's
all
part
of
the
show,
gentlemen,
all
right
there
we
go.
So
this
is
the
info
world.
One
we'll
talk
about
that
in
a
second
and
there's
nothing
really
new
in
here.
I
don't
think
for
anyone.
Who's
used
tempo.
It's
really
just
kind
of
this
high
level
overview
of
what
it
can
do
and
some
of
the
other
integrations
with
like
exemplars
and
such
I
really
wanted
to
show
it
just
to
kind
of
show
some
of
the
growth
tempo's
seen
some
of
the
attention
we're
getting.
A
A
So,
oh
man,
somebody
put
a
much
better
description
that
I
did
in
there.
Thank
you
very
much,
but
yeah
just
kind
of
wanted
to
bring
attention
to
some
of
these
things
that
are
going
on
in
in
tempo
cool
b,
one
one.
So
we
just
cut
v
one
one,
the
I'm
sorry,
we
cut
the
first
release
candidate
rc,
zero,
don't
install
it.
A
Coinrod
will
tell
you
why,
in
a
second
but
to
go
over
some
of
the
features,
some
of
the
new
things
that
are
coming
up,
that
we're
excited
about
that
we
think
everyone
will
be
will
be
able
to
use.
First
is
this
hedge
request
thing?
I
should
probably
be
linking
like
prs
here:
if
anyone
would
we're
capable
of
finding
those
pr
and
bring
them
up
I'd
appreciate
it.
Hedge
requests
basically
will
repeat
a
request:
a
get
request
to
the
back
end
after
so
much
time.
A
So
I
think
internally
we
have
it
set
to
500
milliseconds
or
something.
So
if
your
request
to
your
back
end,
gcs
or
s3
or
whatever
exceeds
500
milliseconds,
it
can
repeat
that
request.
It'll!
Do
that
once
so!
It'll
only
do
it
only
do
two
requests
at
most
and
really
this
just
to
get
around
kind
of
the
long
tail
of
the
back
end.
So
any
back
end,
even
something
as
stable
as
s3
or
gcs
has
a
long
tail,
and
you
can
easily
set
this
kind
of
timeout.
A
This
hedge
request
timeout
to
be
at
a
point
where
you're
only
going
to
be
repeating
a
very
small
percentage
of
your
costs,
like
say
one
percent
and
see
amazing
kind
of
an
amazing
impact
on
the
p99
of
tempo.
So
it
costs
almost
nothing
because
you're,
just
only
repeating
maybe
one
percent
of
all
gets
to
your
back
end
and
the
and
the
kind
of
consequence
of
that
is,
you
just
kill
the
p99
of
tempo.
I
think
we
are
seeing
like
five
or
six
seconds
p9,
which
is
not
great
at
all.
E
E
A
E
A
E
A
A
The
second
one
the
tenant
index
here
is
to
reduce
calls
to
the
back
end.
So
right
now,
every
every
compactor
and
querier
has
to
know
the
state
of
the
back
end
all
the
time
to
for
queries,
to
execute
queries
and
for
compactors
to
kind
of
reduce
your
blocks,
and
they
are
pulling
the
back
end
by
just
listing
every
block
over
and
over
and
over
again.
A
So
I
try
to
get
some
good
documentation
in
here,
because
it
is
a
kind
of
a
operational
change
that,
if
you
do
operate,
tempo
would
be
good
to
know
about.
So
you
kind
of
understand
the
way
your
system
is
working
to
help
understand
how
to
configure
and
also
maybe
what
could
go
wrong.
Some
of
the
failure
modes-
and
I
did
my
best
to
increa-
add
some
very
you
know
solid
documentation
around
that,
but
please
kind
of
review
that
and
if
you
have
any
concerns
or
some
of
the
documentations
make
sense.
A
B
F
Well,
yeah,
I
guess
about
the
tenant
index.
Did
we
call
out
it's
kind
of
beneficial
for
some
workloads
or
installations
where
just
the
pulling
itself
is
become
like
a
non-trivial
cost
like
it
can
become
significant,
and
so
this
really
helps
drastically
reduce
just
like
the
polling
operations.
Overall,
yes,
I
mean
yep,
I'm
not
sure
where
that's
come
up,
but
it
has
before
yeah.
A
It's
come
up
right.
There's
just
like
you
know,
there's
definitely
a
flat
cost
right
to
polling
on
our
gits,
but
also
people
who
have
run.
I
haven't
heard
this
from
tempo,
but
I've
heard
it
from
cortex
and
loki.
People
are
running
it
on-prem
and
maybe
they're
running
with
something
like
mineo
or
any
of
the
billion.
Like
on-prem
s3
things
you
can
run
sv
compatible
things
they're
having
maybe
issues
with
tempo
or
one
of
these
other
applications
overwhelming
their
back
end.
A
E
Cool
so
yeah.
The
next
link
change
is
about
caching
flexibility,
and
this
is
also
about
like
sort
of
larger
workloads.
I
would
say
it's
not
super
relevant
to
smaller
workloads
where
you
can
sort
of
fit
all
of
the
working
set
of
tempo
in
cash.
So
the
working
set
of
tempo
is
the
total
size
of
all
of
the
bloom
filters
for
all
blocks
in
the
back
end
right.
E
So
for
smaller
installations,
that's
relatively
smaller,
but
as
the
number
of
blockless
grows
like
I
think
we
had
22
000
blocks
at
some
point
in
our
back
end
and
then
the
size
of
the
bloom
filters
just
sort
of
shoots
up
to
120,
150
gigs
and
at
that
sort
of
level
it's
it's
not
really
advantageous
to
to
cash.
All
of
those
bloom
filters
you
may
want
to
like
pay
a
latency
penalty
and
and
just
sort
of
have
a
fraction
of
those
bloom
filters
in
memory.
So
that's
kind
of
what
we
implemented
with
this.
E
It
turns
out
that
so
for
the
lower
levels
in
the
compaction
level,
the
blocks
are
actually
they
have
a
lot
of
churn,
so
the
compactors
will
quickly
combine
level
zero
blocks
together
and
move
them
higher
up
the
level,
so
they
disappear
kind
of
quickly
and
also
super
old
blocks.
You
know
that
they're
going
to
get
deleted
soon,
so
it's
kind
of
advantageous
to
just
have
recent
blocks
that
are
higher
up
in
the
compaction
level
and
this
pr
sort
of
adds
the
ability
to
do
that.
E
G
E
It
works
so
so
we
have
this
section
on
caching
in
our
docs
and
if
you
scroll
all
the
way
down
over
here,
we
have
this
cache
size
control
and
this
sort
of
talks
about
what
we're
trying
to
do
here
is
to
is
to
just
store
a
fraction
of
the
bloom
filters
in
cash
and
for
the
rest
of
the
blocks.
Tempo
will
go
and
fetch
it
from
the
back
end
anyway,
there'll
be
a
slight
latency
cost
and
because
they're
not
in
cash.
E
Obviously
right
and
thanks
joe
nice
talk,
so
we've
added
two
sort
of
configuration
options,
the
minimum
compaction
level
and
the
max
block
age,
so
that
sort
of
limits
the
blocks
that
will
be
cached
and
if
you
sort
of
run
this
okay,
let
me
see
if
I
can
bring
this
image
up.
Let
me
share
this
step
instead,
so
this
is
sort
of
a
summary
command
that
we
added
to
our
cli
and
it
it
lays
out
how
the
blocks
the
bloom
filters
over
the
days
as
well
as
the
compaction
level.
E
So
you
can
see
that
there
are.
There
are
like
20
or
bloom
filters,
which
are
one
day
old
land
at
compaction,
level,
zero
and
so
on.
So
this
sort
of
breaks
down
how
the
bloom
filters
lay
out
over
the
days
and
compaction
levels,
and
you
can
see
that
the
the
ones
that
are
lower
in
the
compaction
level-
we
don't
want
to
catch
them
and
the
ones
that
are
really
old.
We
also
don't
want
to
catch
them
because
they're
going
to
get
thrown
out
anyway.
E
We
have
these
two
parameters,
you
can
just
say
cache
a
block
only
if
it's
above
the
level
2
or
if
it's
not
older
than
48
hours-
and
I
know
this-
this
is
like
a
little
complex.
So
if
you
have
any
issues
when
you're
using
it
like
feel
free
to
just
like
open,
pr's
or
issues,
and
then
we
can
improve
on
it.
But
this
is,
this
is
something
that
we
thought
would
work
really
well
for
the
first
pass
and
it's
already
started
showing
some
results.
E
F
Yeah,
we
would
really
like
to
hear
from
from
anyone
who
you
know
what
their
is
with.
Caching,
what
amounts
they're
using
if
these
settings
are
helping
just
be,
and
maybe
it
really
comes
into
play
for
larger
installations.
Like
you
mentioned,
our
environment
had
over
100
gigabytes
of
wood
filters
and
it's
impractical
to
kind
of
catch
all
that
so
yeah
yeah
we'd
just
like
to
hear
any
feedback.
A
Cool
sure,
so,
there's
a
lot
of
channels
to
reach
us,
like
slack.
Of
course,
this
meeting
all
kinds
of
different
ways
to
get
a
hold
of
us
make
an
issue
in
the
repo.
If
you
do
have
feedback
on
some
of
these
things,
we're
adding
or
you
know,
certainly
anything
to
help
you
operate
it.
Let
us
know
we
certain
we
kind
of
have
a
scale
which
is
good
because
it
lets
us
kind
of
find
issues
before
other
people.
A
They
can
also
be
bad
because
we
just
don't
see
some
maybe
of
the
issues
of
running
this
at
low
scale.
You
know
and
we're
maybe
fixing
issues
that
we
have.
So
we
highly
we
highly
suggest
or
encourage
everyone
to
help
us
know
what
issues
you're
running
to
running
tempo,
so
we
can
improve
things
and
and
help
other
or
help
you
all
get
things
going.
A
A
B
Yeah
so,
while
cutting
the
101
release,
we
have
been
noticing
a
couple
of
issues
with
memorists
in
our
largest
clusters,
so
yeah
in
this
release
we
have
updated
our
cortex
dependency,
which
brings
a
lot
of
upgrades
to
you
know
the
ring
to
memory
listen
to
other
stuff,
we
use,
but
this
also
changed
a
bit.
Our
member
list
is
configured
and
yeah
we're
we're
thinking
something
might
be
causing
trouble
there.
So
we're
still
investigating
that.
B
But
the
things
we
are
seeing
right
now
is
that
compactors
have
trouble
getting
ready
and
that
remember
this
is
causing
a
lot
of
cpu.
So
if
you
yeah
so
yeah,
basically
what
we're
noticing
we're
still
investigating.
If
you
have
to
change
the
config
somehow
so,
for
now
our
advice
with
the
101
release
candidate
is
you
know,
can
you
can
try
it
but
might
cause
some
issues?
B
So
you
know
keep
a
good
eye
on
that,
we'll
be
running
it
internally
and
we'll
we're
trying
to
track
down
on
the
issues
we're
seeing
again
as
similar
to
the
previous
part.
If
you
have
any
issues
running
memoryless,
just
let
us
know
create
an
issue
reach
out
on
a
community
slack
we're
constantly
trying
to
improve
it.
So
yeah.
A
A
A
It
sounds
funny
like
you
go,
you
know
to
the
ingester
ring
or
the
compactoring
and
click
forget
and
that's
been.
Memberless
has
generally
been
very
good,
but
this
one
like
haunting
issue.
We
can't
get
rid
of
I'm
kind
of
deep,
diving
it
now.
So
maybe
we'll
have
a
resolution
or
maybe
we'll
tell
you
I'll,
stop
using
member
list
and
we'll
remove
it
from
default.
So
I'm
not
sure
it's
going
to
be
one
of
those
two
directions.
A
F
A
If
you
install
one
one,
there's
nothing,
you
need
to
do,
but
we're
going
to
remove
support
for
these
in
one
two
and
there's
some
details
in
that
change.
Log
about
what
you
need
to
do,
which
is
basically,
you
need
to
upgrade
to
a
version
that
will
use
the
newest
block
version,
which
is
v2
and
kind
of.
Let
all
your
blocks,
all
your
older
blocks
fall
off,
but
the
changelog
has
some
really
good
details
added
by
marty
on
how
to
kind
of
migrate
forward.
A
It's
not
really
that
you
really
have
to
do
nothing
except
install
a
version
of
the
of
tempo
that
understands
both
the
v1
and
the
v2
blocks,
wait
for
all
v1
to
go
away
and
then
install
1.2.
So
just
heads
up
that
that
kind
of
change
is
occurring
and
now
service
graphs
are
up,
and
unfortunately
I
don't
have
a
ton
to
say
here,
because
mario
is
the
primary
kind
of
owner
of
that
on
our
team
and
he
is
currently
on
pto.
A
There
was
this
pr
that
we
dug
up
connor.
I
don't
know
if
you
have
anything
to
add
you're
welcome
to
say
I
have
nothing
to
add
joe
since
I'm
totally
putting
on
the
spot
here,
and
we
discussed
this
not
at
all
beforehand.
But
if
you
know
anything
about
service
graphs,
conor's
on
the
grafana
team
proper.
Do
you
have
if
you
anything
to
share
we'd
appreciate
it.
A
Once
mario
gets
back
he's
doing
the
work
in
the
agent
to
kind
of
generate
the
metrics
necessary
to
draw
this
graph.
Conor
and
andre
are
both
working
on
the
actual
code
to
like
display
this
in
grafana.
So
when
he
gets
back
we'll,
probably
maybe
look
to
have
a
demo
or
look
to
kind
of
show
a
little
bit
more,
maybe
next
month
at
the
community
call
in
september
cool.
A
F
Cool
yeah,
so
in
the
last
community
call
we
kind
of
walked
through
the
timeline
for
what
we
see
for
tempo
search,
and
I
think
we
maybe
we
did
a
live
demo
and
we
kind
of
showed
off
some
of
that.
So
we're
not
going
to
repeat
that
here.
None
of
that
has
really
changed.
We
just
have
some
numbers
from
some
internal
benchmarking
and
performance,
testing
and
stability,
and
things
like
that,
so
I'm
gonna
go
ahead
and
share
that.
F
So
that's
the
right
path
and
that's
kind
of
like
what
we're
focusing
on
here
and
the
investors
are
keeping
data
for
around
nine
minutes.
So
it's
that's
kind
of
like
the
environment
that
we
were
working
with
so
for
searching
so
after
letting
it
run
and
kind
of,
like
all
the
new
data
that
that's
present
on
the
injustice
has
been
indexed
and
things
like
that
and
he's
also
searched
meaning.
It's
searching.
Everything
on
the
ingestors
was
able
to
be
completed
in
about
1.5
seconds.
F
That
was
fairly
consistent,
and
that
means
it
was
searching
around
34
million
traces
or
about
66
gigabytes
of
search
data,
and
so
each
ingestor
there
were
40
of
them.
Each
one
was
doing
able
to
process
kind
of
over
one
gigabyte
per
second
or
so
for
ingester,
and
so
overall,
the
search
performance
we
felt
pretty
good
with
it
was
45
gigabytes
per
second
of
search
data
per
second
leading
to
the
1.5
seconds
for
the
exhaustive
search
now
in
exhaustive
searches.
F
So
searching
this
an
exhaustive
search,
we're
doing
by
searching
for
tags
that
don't
exist,
so
it
was
forced
to
search
everything,
but
other
searches
were
much
faster
if
they
were
able
to
be
completed
and
find
enough
results
sooner
than
that
cool.
So
that's
kind
of
the
performance
and
again
that's
just
over
around
nine
minutes
of
data.
So
this
is
promising,
but
you
know
what
we
want
to
search
a
lot
more
than
the
past
nine
minutes
of
data
resource
usage.
F
So
there
was
a
lot
more
overhead
to
turning
this
on
which
we
expected,
but
we
have
some
numbers
for
that
too.
So,
storage
wise,
it
was
around
1.5
gigabytes
of
extra
disk
storage
per
adjuster
for
the
additional
search
data,
and
so
that's
kind
of
like
the
66
gigabytes,
distributed
across
the
40
ingestors
for
on
the
right
path,
which
is
the
distributor
and
the
adjuster.
F
There
was
around
25
increase
in
cpu
30
in
memory,
and
so
the
cpu
is
required
for
just
processing
the
additional
data
because,
as
each
trace
comes
in
kind
of
like
that,
search
metadata
is
extracted
and
built
and
built
up
and
then
saved.
And
then
the
memory
actually
comes
from
storing
the
live
choices
with
their
search
data
in
memory.
A
lot
of
that
and
then
of
course,
just
other
other
work
going
on
cool
any
other
things
to
cover
about
the
performance.
D
F
A
F
Yeah,
so
the
the
ingesters
do
have
a
lot
of
work
to
do,
because,
although
the
distributor
extracts
it
initially
and
sends
the
bytes
over
to
the
adjuster
that
part's
kind
of
efficient
for
the
adjuster,
but
there's
still
a
lot
of
work
to
do
so
if
the
the
trace
is
coming
in
in
multiple
parts,
those
have
to
be
unpacked
and
repacked
together
and
then
after
it's
written
to
the
wall,
it
then
has
to
be
flushed
into
another
block.
So
there's
a
little
bit
more
work
to
do
there.
Okay,.
F
F
So
this
is
the
same
timeline
that
we
showed
off
previously,
so
this
hasn't
changed.
I'm
just
highlighting
here
kind
of
the
section
that
we're
talking
about.
So
this
is
phase
one
of
search,
so
tempo
will
have
an
api.
It
will
search
the
adjusters
and
we
will
kind
of
have
this
experimental
upcoming,
grafana
ui,
so
the
tempo
pieces
we're
targeting
for
one
two.
So
it's
not
the
release
we're
talking
about
today
in
the
in
this.
That's
one
one!
It's
one
two
is
the
one
after
this
and
then
the
grafana
version
is
to
be
determined.
A
It's
possible
you'd
would
just
prefer
continuing
to
use
your
continuing
to
use
your
logs
or
your
exemplars
for
discovery.
So
we
want
to
kind
of
maintain
that,
after
this,
so
for
a
while,
it'll
probably
be
like
a
flag
to
turn
it
on
a
feature
flag
and
then
eventually,
it'll
be
we're.
Flipped
and
it'll
be
a
flag
to
turn
it
off.
A
If
you
prefer
kind
of
you
know
doing
it
the
old-fashioned
way
and
having
that
level
of
performance
that
you
have
today,
because
we
do
know
some
people,
you
know
enjoy
it
for
that
kind
of,
like
extreme
high
volume
back-end
and
this
feature
may
not
make
sense
for
them
cool
all
right.
I
think
that's
roughly
the
end
of
our
stuff
questions,
any
other
agenda
items
or
concerns
or
questions
or
if
anybody
wants
to
you
know
tell
a
hilarious
joke.
A
C
A
All
right
all
right!
Well,
everybody
take
care,
let's
go
ahead
and
call
it.
If
there's
nothing
else
to
discuss,
and
we
will
see
you
in
about
a
month-
please
reach
out
on
our
slack
channel.
It's
probably
the
best
place
to
get
real,
quick
help.
We
also
have
community
forum.
Oh
there's
a
question
here,
one
sec:
we.
E
A
Have
the
community
forum,
which
is
a
great
resource,
the
grafana
forums,
to
ask
questions,
and
then
the
ish
or
the
the
github
repo?
We
do.
We
work
very
hard
every
member
of
this
team
to
be
very
responsive
on
all
of
these
different.
You
know
methods
of
contacting
us,
so
please
reach
out
on
them.
If
you're
having
issues
an
overview
of
the
grafana
tempo
deployment
would
be
pretty
helpful.
Cpu
memory,
storage,
any
config
changes
of
note
tanner.
G
Hey
I'll
turn
on
the
view
of
this
one
sure
yeah,
so
we're
at
shopify
we're
just
rolling
out
tempo
we're
up
to
about
I've,
gotten
it
up
to
about
60
million
spans
per
minute.
Now,
so
it's
getting
pretty
big
or
that's
and
that's
at
a
low
sample
rate.
So
we
have.
We
have
a
lot
of
span,
so
our
tempo
deployment
is
going
to
be
pretty
happy.
G
G
A
A
A
I
don't
know
completely
up
to
date.
This
cluster
is
taking
2
million
spans
per
second.
At
the
moment,
this
dashboard
is
awful.
It
needs
a
lot
of
love
and
we
use
it
every
day
or
I
do
at
least
injectors.
I
don't
we
don't
see
spikiness
on
the
ingestors.
Really
I
mean
unless
it's
below
the
prometheus
scrape
interval,
which
is
of
course
possible
right.
It
might
be
that
we're
just
not
seeing
something
that
could
be
no.
G
F
G
F
So
a
couple
versions
back
we
improved
the
adjuster
replay
to
not
actually
replay
like
it
used
to
actually
duplicate
all
the
data
into
a
new
wall
on
startup.
That
was
the
actual
replay.
But
there
was
oh
yeah,
but
maybe
it
was
in
1.0.
We
changed
it
to
kind
of
just
res
quickly,
rescan
the
wall
files
and
replay
a
lot
less.
I'm
honored.
C
E
G
So
it
seems
to
be
some,
investors
would
stop
running
or
they
would
restart,
and
so
I
think
I
triggered
this
currently
scaled
down
and
lost
two
adjusters
right
off
and
then
it's
kind
of
everything
has
struggled
to
come
back
a
little
bit
because
of
that
it
seems
like
so
then
some
get
unready
and
then
it
starts
to
kill
some
and
then
those
start
come
back.
They
start
doing
the
replay
walls,
so
they're
unready
for
like
five
minutes,
but
they're
kind
of
this,
like
vlog,
is
going
on.
A
A
A
A
F
A
G
F
F
F
F
G
A
I'll
help,
people
with
that
the
compaction
window
in
particular,
is
a
good
one
to
reduce.
So
I
think,
by
default,
it's
like
an
hour
and
when
you're
creating
a
huge
number
of
blocks,
reducing
that
window
like
five
minutes
or
something
will
allow
a
lot
more
compactors
to
participate
very
early
on
in
the
process
at
the
level
zero
which
will
reduce
your
block
list
quite
a
bit
okay.
E
G
F
Yeah
I
mean
I
guess
we
don't
have
numbers
that
we've
arrived
at
through,
like
any
sort
of
like
formula,
but
I
mean
maybe
we
could
you
know,
do
a
better
job,
with
some
guidance
for
different
levels
of
scale.
Like
this
environment,
you
know,
is
in
the
million
spans
per
second,
we
could
do
guidance
for
something
like
that
and
then
maybe
other
loads
like
100
000
or
something.
A
Yeah
we
have
20
40
40
40.
yeah,
it's.
It
is
a
bottleneck
of
the
system
that
compactors,
because
it
pulls
has
to
pull
the
entire
block,
which
is
a
lot
of
data
right,
rebuild
the
blocks
and
send
them
back
out.
I
don't
know
I
think
at
some
point.
We
don't
have
plans
in
the
immediate
future
deal
with
that,
but
I
think
to
get
into
like
the
tens
to
20
millions
of
space
per
second
per
cluster.
We
might
start
looking
at.
A
G
G
A
A
E
A
A
A
G
A
A
There's
eight
unhealthy
ones
here.
That's
also
part
of
the
reason
you
have
a
long
block
list,
although
just
scaling
up
would
be
helpful
so
with
those
compactors
sitting
there
that
aren't
actually
there
they
own
pieces
of
the
ring
and
the
other
compactors
are
basically
ignoring
entire
sections
of
blocks.
They
could
be
compacting.
A
F
A
A
E
E
A
A
Cool
awesome,
honestly,
it's
awesome
to
hear
you
having
success
at
1
million
spans
per
second
70.
000
blocks
is
enormous
and
we
need
to
work
to
get
that
down,
but
we
can
help
you
do
that.
I
would
kind
of
expect
it
to
just
fail
because
of
the
queue
size
and
the
queriers,
but
give
it
a
shot.
Let's
know
what
happens
when
you
do
that.
G
A
Yeah
man
reach
out
on
reach
on
slack
or
whatever
is
convenient
for
you,
as
you
kind
of
play
with
that
over.
Maybe
the
next
couple
days
or
weeks
we'd
be
glad
to
get
you
in
a
more
stable
spot
and
if
you
can't
get
your
injustice
stable,
that's
concerning,
and
we
need
to
dig
into
that.
So
if
you
can't
please
reach
out
and
we
can
share
logs
or
whatever
you
know
panic,
you
know
slight
stack
traces
or
whatever
you
can
get.
Okay,
cool
cool.
A
A
All
right
well
on
that
note,
unless
there
is
any
other
questions,
any
other
things
in
the
doc,
please
anything
for
the
agenda.
There
question
mark.
Oh
man,
we
got
good
notes
here,
good
job
ananya,
if
there's
any
other,
if
there's
not
any
other
questions
or
that
we
can
kind
of
just
go
ahead
and
finish
up
all
right,
everybody
take
care.
I
will
see
you
all
next
month,
if
not
before
tanner
I'll,
probably
be
hearing
from
him
in
the
next
couple
hours.