►
From YouTube: SlackTrace: A New Tracing Tool - Suman Karumuri, Slack
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
SlackTrace: A New Tracing Tool - Suman Karumuri, Slack
Trace data contains very rich information about a request execution. However, current tracing tools only expose that information as a trace view or a service graph, which severely limits the questions we can ask of trace data and diminishes the utility of tracing. However, from past experience, we found that these limitations arise because unlike logs or metrics, we can’t query raw trace data.
To query raw trace data easily, we designed a new span format called SpanEvent and built our tracing infrastructure called SlackTrace around it. In addition, to presenting the trace data as a trace view and a service graph, the SpanEvent format allows us to query raw span data using SQL queries which allows us to derive rich insights from trace data that is not possible with existing tracing systems. In this talk, I will present SpanEvent format and an overview of our SlackTrace infrastructure.
A
Hi
everyone,
my
name,
is
Suman
Cara
Marie,
my
pronouns
are
he/him
and
his
today
I'm
going
to
talk
about
slack,
trace
a
new
tracing
tool.
We
have
built
at
slack.
So
just
something
about
the
name.
Is
that
there's
a
talk
deadline?
The
project
doesn't
have
an
internal
name,
so
we
just
picked
that
because
that
was
the
top
of
my
hair
mind
at
that
time.
So
it
has
it's,
it's
not
how
we
trace
flag,
but
it's
the
name.
Is
there
just
for
the
sake
of
the
talk?
A
So
just
a
little
bit
about
me,
so
I'm,
a
senior
staff
engineer,
it
slack
and
I
work
on
the
monitoring
team.
There
I
lead
the
team,
I
love,
building
and
running
petabyte
scale,
observability
systems
and
at
twitter
thing,
precise
to
work
at
Twitter.
Pinterest
Facebook
like
that
strip.
There
and
I
ran
ELQ
at
a
lot
of
these
companies
and
we
also
replaced
al
khair
Twitter
with
a
system
called
log
lines,
which
is
a
log
search
platform
which
manages
about
two
petabytes
of
data.
A
Today,
I
was
a
tech
lead
for
Zipkin
project
at
Twitter
and
I
was
also
on
the
open
tracing
Advisory
Committee
customer
advisory
committee.
Something
and
I
also
built
end-to-end
tracing
at
Pinterest.
It's
the
system
is
called
pin
trace
on
the
matrix
front.
I
I
was
part
of
like
Twitter's
matrix
stack,
we're
an
open
DSD,
be
at
Pinterest
I.
We
run
at
slack.
We
run
like
one
of
the
largest
Prometheus
clusters
in
the
world.
A
I
also
wrote
an
in-memory
matrix
storage
engine
called
UV
that
for
open
DHCP
in
general,
I'm,
actually
very
interested
in
distributed
systems,
cloud
infrastructure,
infrastructure
and
general
systems.
So
I'd
like
to
start
with
a
really
really
quick
overview
of
tracing,
so
going
back,
it
is
slow,
is
the
hardest
problem
to
debug
in
a
distributed
system.
So
tracing
was
built
to
identify
where
the
bottlenecks
are
in
the
system
right,
everybody
builds
their
own
version
of
tracing,
so
I
mean
that
gave
rise
to
Rodrigo's,
tenth
rule
of
distributed
system
on
it.
A
Any
sufficiently
complicated
distributed
system
contains
an
ad
hoc
informally
specified
siloed
implementation
of
casa
tracing
right.
So
then
we
have
these
tracing
systems
today,
where
it's
a
bit
more
formalized
view
of
capturing
this
information
in
a
trace
and
then
analyzing
it
using
cold,
visualizations
and
analytics
tools.
So
what
we
did
to
trace
this
data
is
we
took
a
request,
centric
view
where
we
follow
a
request
along
its
execution
paths
of
all
its
micro-services
and
then
along
its
request,
execution
path.
We
capture
the
spans,
which
is
the
trace
data.
A
We
collect
this
data,
store
it
and
some
data
store
somewhere
and
then
generally,
these
systems
have
a
find
traces
view
where
you
search
for
the
representative
traces
and
then
you
look
at
one
of
those
traces
in
a
trace
view,
and
sometimes
you
also
look
at
this
data
in
a
service
graph
right.
This
is
generally
a
very
high
level
view
of
tracing
systems
right.
Some
tools
and
vendors
do
provide
analytics
on
top
of
this
trace
data
right.
A
So,
having
built
these
systems
right,
like
at
Twitter
Pinterest
and
like
slack
one
of
the
things
that
there's
an
emerging
theme
is
like
these
systems
are
very
complex,
like
all
those
like
Deathstar
diagrams,
you
see
like
in
slides
are
real
and
honestly.
Those
diagrams
actually
are
not
very
useful
in
practice
at
all.
It's
like
just
a
cool
visualization
of
how
complex
everything
is.
So
what
we
found
from
these
traces
like
using
these
traces
at
scale,
is
there
are
like
a
lot
of
problems.
You
need
a
lot
of
intuition
and
expertise
to
understand
defined,
representative
traces.
A
So
once
you
are
tracing
these
systems,
unless
you
know
more
or
less
what
you're
looking
for
and
search
for
the
right
thing,
it's
really
hard
like
it
would.
The
data
you
collect
won't
be
useful
unless
you
do
have
to
have
some
sense
of
what
you
are
knowing,
and
even
if
you
find
the
representative
trace
and
like
you,
you
are
looking
at
it
in
a
trace
view.
The
tools
stop
there.
They
don't
tell
you
what
slow,
like
it's
left
as
an
exercise
to
the
reader
slower,
why
it
is
slow
right.
A
A
Tracing
is
not
a
general
generally
is
not
the
tool
they
reach
out
to
ask
and
answer
questions
about
their
systems
performance,
so
service
graphs
are
cool,
but
outside
of
like
demos,
in
like
real
production
scenarios,
they
are
not
very
useful,
at
least
like
you
know,
in
my
experience
and
traces
cannot
be
used
for
triage.
So
if
you
have
a
pair
ongoing
production
issue
generally,
traces
are
not
used
for
triage.
A
They
are
generally
used
to
identify
bottlenecks
and
like
fake,
identify
bottlenecks
and
fix
point
issues,
performance
issues,
but
when
you
have
an
ongoing
incident,
traces
are
generally
not
the
tool
you
reach
out,
for
they
can
help.
If
you
know
that
hey
like
trace,
would
answer
this
question,
but
generally
they
may
not
be
right
and
there
are
other
problems
with
traces
like
you
sample
at
scale.
You
have
to
have
you
have
instrumentation
overhead
and,
as
a
result,
what
happens
is
when
you
have
your
you're
capturing
less
data?
A
As
a
result,
the
number
of
questions
you
can
ask
of
your
traces
generally
diminishes
at
scale.
So
what
happens
is
once
generally
tracing
systems
are
built?
They
are
deployed.
You
get
some
quick
wins
which
are
very
interesting
and
then
tracing
usually
falls
on
the
wayside.
It
becomes
a
nice-to-have
tool
as
opposed
to
a
must-have
tool
right
like
if
it
goes
away
like
if
your
prom
goes
away,
all
hell
breaks
loose
if
racing
goes
away,
yeah
right
so
true,
but
the
thing
is:
traces
have
very
rich
and
complex
data
in
them.
A
The
problem
we
have
found
by
building
these
systems
is
these.
The
data
in
the
traces
is
exposed,
as
literally
two
views
like
a
trace
view
and
a
service
graph
view,
it's
basically
like
taking
all
your
fans,
Prometheus,
metrics
or
logs,
and
everything
and
just
providing
a
single
dashboard
to
your
data
right.
That's
a
single
view
into
your
data
and
it's
pretty
constraining,
because
you
can't
ask
any
more
questions
of
the
data.
It's
like
you,
have
these
views
and
then
that's
pretty
much
it
right.
It's
a
waste
of
the
data
we
collect.
A
So
that's
why
we
want
to
make
trace
Reiter
more
useful
and
to
make
it
more
useful.
We
want
to
make
it
more
accessible
right,
that's
our
goal,
so
the
image
there
basically
shows
like
initially
there's
a
lot
of
hype
and
then
they're
deflated
exceptions
and
generally
that's
where
it's
stuck
right.
So
what
we
want
to
do.
We
want
to
access
this
traces
as
raw
data
right.
So
why
do
we
care
right?
So,
as
I
said,
the
we
only
get
fixed
views
into
the
data,
but
what
if
we
can
access
raw
data?
A
If
you
access
raw
data,
you
can
slice
and
dice
these
spans
across
various
dimensions,
to
get
unique
insights
right,
for
example,
we
can
mix
and
match
the
data
from
different
spans
to
ask
and
answer
different
sets
of
questions
that
we
couldn't
ask
before
right.
We
can
build
custom,
dashboards
and
analytics
all
the
things
we
do
on
logs
and
metrics
today
on
traces
as
well
right
to
get
the
value
out
of
them
right.
So
we
were
thinking
about
this
and
it
it
was.
I
came
across
a
strange
like
revelation
right.
For
me,
it
was
hey.
A
We
access,
metrics
and
logs
as
raw
data
right.
Why
not
traces
traces?
You
don't
have
you
don't
even
know
what
the
data
format
is.
You
just
go
to
Zipkin
UI
or
jäger
UI
or
your
tool,
and
then
you
just
put
in
some
information
and
then
it
spits
out
of
view.
That's
pretty
much
it
you
don't
access
raw
data
in
traces.
Why
is
that
so?
A
Having
thought
through
this,
and
we
have
identified
a
few
issues
and
like
we
got
a
few
insights
right.
So
if
you
look
at
like
tracing
today,
you
have
this
open
tracing
api's,
where
the
trace.
That's
how
you
instrument
your
applications
right,
open
tracing
is
a
complex
but
clean
API
right
to
instrument
you're
the
your
crazy,
your
application
for
tracing
right
and,
on
the
other
end,
you
have
cool
clean,
intuitive,
UI
s--,
which
is
you
have
you
or
traceview,
and
your
service
craft
right
to
look
at
this
data,
but
what's
in
between,
is
pretty
much
undefined.
A
Its
implementation
defined
it's
very
complex
right.
The
span
formats
are
very
complex
and
they
are
designed
to
fit
the
needs
of
the
tools,
the
questions
they
answer:
the
diverged
storage
systems,
they
use
the
diverse
query
languages
they
use
all
the
internal
formats
are
designed
for
to
fit
into
this
like
to
fit
into
this
diverse
model
of
off
spans
right.
A
So
what
is
complex
about
this
file
format
right?
So
we
looked
at
all
the
span
formats
or
at
least
like
the
publicly
available
ones,
and
here
is
what
we
found
right
most.
The
reason
span
formats
are
complex
is
they
are
very
network
centric
like
tracing
distributed,
tracing
started
from
a
very
network
centric
perspective,
so
the
span
formats
are
very
network
centric
and,
as
time
went
on
people
added
UI
things
that
are
needed
for
the
UI's
for
all
the
tools
into
the
sponsors
right.
A
So
the
the
span
formats
organically
grow
and
there
are
like
interesting
side
effects
of
this
right.
So
if
you
look
at
a
span
format
for
Zipkin
yoga
or
any
of
these
projects,
one
of
the
things
you
would
see
is,
for
example,
if
you
look
at
a
span
which
represents
how
much
time
something
happened,
and
then
you
have
two
ways
in
to
represent
the
sub
time
right.
A
One
way
is
you
can
use
that
you
can
use
your
child's
pants
to
represent
sub
parts
of
that
span,
and
you
can
also
technically
use
annotations
and
people
do
do
that
right.
So
there
are
two
ways,
and
then
there
are
some
UI
driven
aspects
to
these
pants.
For
example,
Zipkin
has
annotations
and
binary
annotations.
What's
the
difference?
A
Annotations
are
sub
events
in
a
span
that
you
would
look
at
that
are
shown
in
the
UI,
whereas
binary
annotations
are
the
same
things
but
they're
not
shown
in
the
UI,
because
they're
supposed
to
have
binary
data
right
and
there
are
like
logs
and
tags
in
these
pants.
What
are
logs
like
it?
This
is
like
something
from
a
storage
concern
right.
If
you
have
application
logs,
you
don't
want,
like
generally,
these
systems
are
stored
in
like
Cassandra
or
something-
and
you
don't
want
to
index
this
data
in
a
specific
way.
A
So
you
have
log
you
put
that
data
in
logs
and
then
you
put
different
data
in
tags,
which
is
like
a
more
columnar
friendly,
analytics
friendly
thing
right
and
then
there
are
Network
centric
concerns
right.
So
Zipkin
has
single
host
like
multi
host
spans,
whereas
span
events
can
belong
to
multiple
hosts.
Zipkin
Jaeger
has
single
host
span
where
every
span
can
only
be
produced
contains
the
data
produced
from
a
single
host
right,
their
wives,
and
why
receive
annotations,
and
then
there
are
complexities
or
long
span,
relationships
and
they're.
A
Just
basically,
too
many
span
formats
out
there
right,
we
each
with
its
own,
like
take
on
what
a
span
should
contain,
and
most
of
these
are
driven
by
what
the
tools
need
right
so
and
then
there
are
other
influences
like
we
talked
about,
like
the
data
model.
Of
these
spans
is
influenced
by
the
storage
systems.
We
use
the
query
model,
the
query
model
exposed
by
those
systems
and
the
latency
of
those
queries
all
of
these
impact.
A
The
span
format
today
right
so-
and
this
also
has
impact
on
how
much
the
cost
of
the
tracing
in
front,
which
query
languages
we
use.
So
what
we
do
is
we
built
to
address
these
complexities?
We
built
a
very
simple
span:
format,
which
is
pretty
minimalistic
called
a
span
event
right.
What
is
the
span
event?
A
span
event
is
a
very
simple
format
for
representing
a
span
right.
So
a
span
event
here
are
the
goals
of
this
new.
The
span
event
we
designed
span
event.
We
wanted
to
support
querying
raw
data
using
sequel.
A
We
want
these
fans
to
be
easy
to
produce
right.
We
want
this
like
a
span
event
format.
We
don't
want.
We
don't
care
if
these
fans
are
part
of
a
trace
or
not
right
and
a
span
event
should
be,
but
we
want
to
capture
the
full
fidelity
of
the
spans.
If
you
take
a
Zipkin
span,
jäger
span
or
an
open,
expand
or
open
sensor
span,
we
want
to
capture
all
that
data
in
our
span.
So
we
want
to
capture
that
full
fidelity.
We
just
don't
want
to
lose
that
data
right.
A
So
a
new
event
has
the
following
fields
like
we
have
an
ID,
which
is
the
unique
ID
for
an
event.
We
also
have
a
timestamp,
which
is
the
time
at
which
they
even
started
the
duration,
which
is
the
time
it
took
to
run
this
event.
We
have
something
called
a
parent
ID,
but
which
establishes
a
relationship
between
two
spans.
We
call
it
parent
ID,
but
it
establishes
a
relationship.
It
does
not
have
to
be
a
parent-child
relationship.
We
also
have
a
trace
ID,
which
is
an
ID
that
we
use
for
deeply
nested
spans.
A
Every
event
has
a
name.
Every
event
has
a
type
which
could
be
a
service
name.
We
also
have
tags
which
are
arbitrary,
set
of
key
value
pairs
for
all
these
events
right
and
we
additionally
have
a
new
field
called
a
span
type
which
is
similar
to
a
kind,
and
then
it
shows
the
type
of
Hispanic,
but
it's
an
annotation
whether
it's
a
client
span
server
span.
Anything
like
that.
So
the
reason
we
did
this
is
it
actually
looks
very
similar
to
the
existing
span
formats
out
there,
but
it
is
very
minimalistic
right.
A
A
It
is
also
a
span
event,
but
we
just
declare
that,
as
a
additional
child
span
that
the
UI
wants
to
do
something
about,
as
opposed
to
like
some
nested
structure
in
a
current
span
right,
we
have
point
spans
which
are
spans
that
happen
for
an
instant,
so
these
are
spans
with
duration
of
one.
We
also
have
one
off
spans,
which
are
spans
that
are
not
related
to
a
request
at
all.
A
It's
just
something
that
happened
in
the
system
right,
so
we
picked
Seco
as
the
query
layer
for
these
data,
because
sequel
is
widely
used
and
popular.
It
is
simple
to
get
started
with
this,
and
the
other
advantage
is
sequel,
ease,
verbose,
but
yet
very
expressive,
and
we
can
do
all
sorts
of
complex
joints
on
this
data
right
and
the
more
important
reason
we
chose
sequel
is
it's
supported
by
a
wide
variety
of
storage
packets
that
are
available
out
there
right.
A
So
we
took
all
of
these
ideas,
put
them
in
a
project
that
is
internally
called
feet,
tracing
and
then
the
goal
of
this
project
was:
we
want
to
represent
our
traces
as
causal
spanned
events
that
can
be
queried
over
sequel
right.
So
so
this
is
the
part
where,
like
I'll
talk
about
like
our
pipeline
and
infrastructure,
but
this
is
this
bits
are
not
very
exciting,
but
it's
like
pretty
standard.
So
we
have
our
apps,
which
generate
spans
right.
A
We
generate
span
events,
some
internal
apps,
and
then
we
also-
and
we
all
the
span
humans
that
are
generated
by
apps-
are
written
to
neuron,
which
is
slack's
internal
reliability
event
pipeline
and
it's
in
an
open
source.
Technology
of
that
would
be
scribe
ish,
and
then
we
also
have
a
history
TP
endpoint
for
ingesting
these
span.
Events
called
Wallace,
the
Wallace
service
takes
this
pan
events
right
simply
mehron,
and
then
we
also
have
trace
adapters.
So
the
thing
is
because
we
invented
in
your
span
format.
We
don't
want
to
invent
new
instrumentation
into
the
applications.
A
So
we
wrote
adapters
for
Jager,
spans
and
Zipkin
spans,
so
applications
just
can
just
point
to
point
their
endpoints
to
these
plans,
and
then
we
can
get
these
things
and
then
neuron
takes
these
plans,
writes
them
to
Kafka,
and
then
we
have
several
neuron
consumers
that
send
the
data
to
Honeycomb
our
internal
data
warehouse,
which
is
a
presto
cluster,
and
we
have
an
internal
Fork
of
Zipkin
to
handle
this
data
and
we
send
it
to
that
too.
So
these
are
just
like
some
use.
This
is
a
view
when
you.
A
A
We
can
build
heat
maps
and
dashboards
on
this
data,
both
within
honeycomb
and
also
within
our
internal
data
warehouse,
and
we
can
also
access
this
data
in
our
data
warehouse
using
complex
sequel
queries
here
is
an
example
of
a
UI
with
a
sample
query
that
shows
show
me
all
the
traces
today
that
have
an
API
method
and
that
came
from
a
specific
host.
So
it's
like
a
cross
span
join
query
right.
So
that's
like
the
heart.
That's
basically
to
prove
that
hey,
we
have
the
system,
it
works
right.
A
A
The
most
interesting
thing
for
me
is
we
were
asking
questions
of
this
data
without
adding
additional
instrumentation.
Generally,
whenever
you
have
some
question
about
how
something
works,
you
have
to
go
in
and
add
some
additional
instrumentation
at
a
counter
metric
whatever
to
get
the
insight,
but
with
trace
data.
A
We
do
have
to
do
that
most
of
the
time,
almost
every
single
time
there
is
a
production
issue
or
any
saying
we
could
go
into
the
system
and
just
formulate
a
new
query
on
this
data
and
get
an
answer
without
adding
additional
data
that
was
very
powerful
and
some
teams
today
do
use
traces
at
slack
as
a
result
for
trying
production
issues.
So
we
had
like
a
major
incident
where,
like
github
was
done
and
well
internal
instance
of
github
was
down
and
we
weren't
sure
why?
A
Because
we
made
a
massive
change
to
some
of
the
components
that
the
pipeline
depends
on
and
we
actually
trace
used
traces
heavily
to
triage
that
issue
and
identify
like
bad
hosts
and
like
take
them
down
very
quickly,
and
the
most
interesting
bit
for
us
is
how
we
use
the
trace
data.
I'll.
Give
you
three
examples.
Well,
I
only
have
two
on
the
slides,
but
I'll
give
you
three.
So
the
first
one
is,
as
I
said
before
we
have
we
use
memcache.
We
use
caching
heavily
in
an
architecture.
A
One
of
the
things
is,
caching
does
not
have
the
same
effect
for
all
the
customers
right.
So
we
have
memcache,
for
example,
for
large
customers.
There
can
be
hundreds
of
memcache
calls
in
a
single
request
right
for
small
customers,
they'll
be
tense,
so
the
perception
of
latency
from
memcache
for
a
given
customer
is
dependent
on
a
specific
endpoint
in
their
use
case.
A
So
what
we
were
able
to
do
is
we
were
able
to
ask
very
pointed
questions
that
you
couldn't
ask
off
of
your
data
before,
which
is
hey,
does
memcache
contribute
to
p99
latencies
for
this
large
customer
right?
So,
for
example,
if
we
want
to
make
our
performance
better
for
our
largest
customers,
how
much
should
I
invest
in
memcache
right
is
memcache
the
right
place
to
invest
a
trace
data
told
us
that,
yes,
it
is
because
what
we
found
is
from
the
trace.
Data
is
hey.
Our
largest
customers
always
hit
p99
latency.
A
So
we
cannot
just
look
at
99.
We
need
to
probably
look
at
three
nines
and
optimize
that
and
that's
one
inside
we
got
out
of
it
right
and
then
the
other
thing
we
found
interesting
answers
to
is
we
have
these
API
calls
for
IDR.
So
slack
provides
this
feature
called
international
data
residency.
We
data
is
actually
spread
across
for
legal
reasons.
A
Some
customers
want
the
data
in
a
specific
geographic
region,
so
when
we
launched
this
feature
generally,
when
you
launch
features
like
this,
where
you
keep
the
data
in
a
different
DG
or
graphic
region,
and
it's
in
a
different
AWS
data
center,
any
slowness
is
generally
attributed
to
this.
Like
cross
region
cause
right,
we
really
don't
have
data
from
matrix
and
logs
to
really
prove
one
way
or
the
other
whether
it
is
the
case
or
not,
but
with
traces
we
could
prove
whether
how
slow
the
idea
calls
were
making
the
actual
endpoint
calls.
A
For
example,
if
the
idea
calls
adds
100
milliseconds
of
latency
right,
a
cross
region
call
if
the
overall
call
takes
4
seconds.
That
hundred
milliseconds
is
not
optimizing,
that
100,
milliseconds
or
blaming
it
is
not
going
to
help
right,
so
it
has
helped
us
I,
didn't
compare
in
region.
Queries
with
cross
region
queries
to
know
whether
the
additional
latency
we
are
seeing
for
idea.
Customers
is
because
we
are
making
that
network
hop
across
the
region
or
not.
A
Similarly,
for
our
ekm
customers,
which
is
like
basically
encrypted
slack
messages,
we
can
also
say
whether
there's
latency
for
API
requests
is
coming
from
our
slow
customers.
The
third
bit,
where
we
use
traces
very
heavily,
is
I
mean
one
of
the
biggest
problems
with
tracing.
Is
you
have
to
like
trace
your
requests
across
the
stack
right,
but
a
slack
desktop
client
uses
WebSockets.
If
you
do
a
network
console,
you
would
see
it
so
we
use
WebSockets.
So
we
did.
A
We
maintain
a
persistent
connection
with
some
edge
network
and
one
of
the
interesting
things
is
we
look.
Traces
are
not
very
well
suited
for
tracking
a
request
through
those
through
WebSockets,
and
so
what
we
did
was
we
model
the
WebSocket
state
as
a
trace
in
itself,
and
we
have
a
client
trace
and
a
server
trace
right
and
what
we
did
with
this
is
now
because
it's
sequel
we
could
ask
hey
this
request.
A
Is
this
customer
is
experiencing
slowness,
easy
because
he
is
in
degraded
mode
right
because
is
a
state
of
the
websocket,
and
if
the
websocket
network
is
down,
we
go
to
the
back
end,
but
because
we
have
a
client
state
and
we
have
this
degraded
mode,
we
can
mix
and
match
independent
requests
right,
like
the
state
of
WebSocket,
is
not
even
a
trace.
It's
not
part
of
any
request,
but
yet
we
could
take.
A
This
data
makes
it
with
our
trace
data
in
a
sequel,
query
and
ask:
is
this
low
because,
because
the
customer
is
running
integrated
mode
or
not
right,
so
we
were
able
to
get
some
unique
insights
which
were
not
possible
if
we
couldn't
access
this
trace
data
as
raw
data.
Before
and
in
future,
we
want
to
build
a
canonical
instrumentation
based
on
trace
data,
as
I
said
like
we
were
able
to
ask
a
lot
of
questions
about
the
trace
data
that
we
couldn't
ask
before.
A
So
what
we
want
to
do
going
forward
at
slack
is
we
want
to
make
the
the
trace
data
as
the
default
canonical
instrumentation?
So
when
we
launched
a
new
feature
without
adding
any
additional
instrumentation,
we
want
the
trace
data
to
answer
basic
questions
about
the
health
and
performance
of
our
application,
and
we
are
also
going
towards
end-to-end
tracing
where
we
want
to
start
where
we
are
starting
to
trace
the
fly.
We
start
the
trace
on
your
client.
A
We're
in
the
data
warehouse
if
a
span
has
that
we
automatically
generate
the
automatically
generate
custom
analytics
for
users
without
users
having
to
do
anything
other
than
just
say
that
this
pan
has
this
tag
right
in
future,
we
want
to
open-source
the
system
and
using
this
system
we
want
to
bridge
the
gap
between
events
logs
and
traces.
We
also
want
to
at
some
point
investigate
like
a
more
concise
query
language
for
querying
this
data,
because
right
now,
sequel
is
honestly
verbose.
A
A
Format
that
we
designed
to
allow
querying
traces
as
raw
data
sequel
is
a
powerful
tool
to
query
these
pans,
and
we
have
put
all
of
these
ideas
together
in
feed,
tracing
or
slack
trace,
and
it
works
well
previously,
if
I
built
a
tracing
system,
I
had
to
knock
on
every
team's
door
and
say:
hey
like
please
use
tracing
it'll
be
very
useful.
Now
people
knock
on
my
door,
so
I.
A
B
A
A
C
A
Is
but
data
for
data
warehouse
is
presto,
so
it's
mostly
the
data
stored
on
s3
and
we
only
query
it
on
demand,
so
the
storage
costs
are
low.
We
get
infinite
retention
because
of
our
data
warehouse,
whereas
the
data
in
honeycomb
is
only
for
like
a
couple
of
weeks
at
this
point-
and
it's
only
it's
used
for
like
so
the
way
we
think
about
honeycomb
versus
data
warehouse
is
data
way.
Honeycomb
for
recent
data,
short
interactive
queries,
data
warehouse
for
complex
analytics
over
long
time
ranges
of
data,
okay,.
A
A
B
D
A
So
yeah
great
question,
so
what
we
actually
do
is
okay.
The
way
we
do
sampling
is
different,
so
we
have
a
one
percent
based
sampling
rate,
which
is
the
sampling
rate.
That
always
happens
right.
The
additional
one
percent
is
actually
our
C
II
person
or
some
team
turns
on
tracing
for
a
specific
team
or
for
a
specific
customer
or
specific
request.
A
So,
as
a
result
from
the
one
person
sampling,
we
get
like
a
general
idea
of
how
things
are
going
or
what
kind
of
things
our
systems
are
do
and
from
the
additional
1%
we
get
the
specific
data.
So
the
beauty
of
ours
is
like
this.
Tired
sampling
is
generally
tracing,
is
like
sampled,
and
then
you
don't
get
a
specific
request.
So,
with
our
sampling
scenario,
we
handle
both
specifics
as
well
as
generic
requests.
A
E
Okay,
yeah
really
interesting
talk,
I
just
have
a
question,
so
you
talked
about
1%
sampling,
but
now
you
have
these
interesting
sequel.
Queries
that
you
can
use.
Have
you
thought
or
are
you
thinking
about
perhaps
making
some
of
these
queries
that
become
interesting,
like
standing
queries
on
on
more
than
one
percent
such
that
the
traces
that
are
filtered
by
the
queries,
yeah.