►
From YouTube: Why, How to, and Issues: Tail-Based Sampling in the OpenTelemetry Collector - Reese Lee, New Relic
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Why, How to, and Issues: Tail-Based Sampling in the OpenTelemetry Collector - Reese Lee, New Relic
When you are running OpenTelemetry in production and your services are producing a firehouse of spans, the traditional and default head-based sampling approach won’t cut it. This is because traces are sampled at initiation, which can be useful for some environments, but for larger systems, it can mean you miss out on key trace data. This is where configuring the Collector to sample your traces after they have fully completed–tail-based sampling–becomes a great option. In this talk, you’ll learn about head- and tail-based sampling, and why the latter approach is useful for obtaining the highest level of granularity in troubleshooting. You’ll learn how to configure your OpenTelemetry Collector to do this, and see the implementation in a suite of microservices, with traces exported to Jaeger. You’ll also learn of the current issues with implementing tail-based sampling in the OpenTelemetry Collector in production so you can take the challenges into account for your own deployments.
A
A
Hola,
bienvenido
a
todos!
Thank
you
all
so
much
for
coming
out
today.
I
know
it's
been
a
long
week,
probably
for
all
of
you
and
it's
a
long
day
and
I'm
so
flattered
that
you
chose
to
come
to
my
session
for
your
last
session
of
the
day,
so
just
to
make
sure
we're
all
in
the
same
room.
My
talk
is
why
how
to
and
issues
tail-based
sampling
in
the
open,
telemetry
collector?
A
Oh,
that's
weird,
okay!
Well,
I'm
gonna
have
to
stand
here
all
right
to
start.
I'm
gonna.
Take
you
through
our
agenda
for
the
session.
I'm
going
to
start
with
a
brief
refresher
on
open,
telemetry,
collector
and
distribute
tracing
then
we'll
get
into
a
sampling
overview
where
I'll
cover
not
just
what
is
sampling,
but
also
why
sampling
and
then
we'll
see
sampling
in
action
with
a
live
demo
by
yours
truly
and
we
will
wrap
up
with
concerns
and
limitations
so
that
you
are
aware
of
the
challenges.
A
The
first
question
I'm
going
to
answer
for
you
today
is:
who
am
I
my
name
is
rhys
lee.
I
am
a
developer
relations
engineer
on
the
open,
telemetry
community
team
at
new
relic.
I
am
based
in
vancouver
washington.
I
am
passionate
about
helping
observability
end
users
get
and
understand
useful
data
from
their
systems.
So
I'm
very
pleased
to
share
this
presentation
with
you
today:
okay,
a
first
a
quick
refresher
on
these
core
concepts.
As
I'm
sure
most
of
you
are
already
familiar
with
these
at
this
point.
A
What
is
the
collector?
It
is
essentially
an
extremely
configurable
system
for
processing
telemetry
data.
It's
made
of
the
there's
three
main
components
that
access
that
telemetry
data
receivers,
processors
and
exporters,
and
some
of
the
things
that
a
collector
can
be
configured
to
do
include
sampling,
collecting,
host
metrics,
scrubbing
data
and
normalizing
data.
A
What
is
distributed
tracing
distributed
tracing
is
the
method
of
observing
requests,
as
they
move
from
one
service
to
another
in
a
distributed
system,
and
it's
important
for
helping
us
understand.
Our
systems,
such
as
our
service
connections
like
how
our
services
are
interconnected
and
can
be
useful
for
diagnosing
problems
such
as
where
latency
is
occurring.
A
A
A
A
A
A
Okay,
before
we
get
into
the
sampling
overview,
can
I
get
a
quick
show
of
hands?
Is
anyone
currently
using
trace
sampling
in
your
organization
or
know
anyone
who
or
know
what
your
organization
is
excellent?
Okay,
so
it
looks
like
some
of
you
are
probably
really
familiar
with
sampling,
but
in
any
case
I
will
cover
what
is
sampling,
why
we
might
want
to
sample
and
then
we'll
talk
about
head
and
tilde
sampling.
A
A
A
So
different
organizations
will
have
different
reasons
for
not
just
what
they
want
to
sample,
but
also
why
they
want
to
sample
some
teams.
They
want
to
see
only
interesting
traces
or
they
might
want
to
filter
out
noise,
such
as
health
checks,
and
I'm
going
to
pause
you
for
a
second
to
explain
what
I
mean
by
interesting.
Traces.
You'll
also
hear
me
use
the
term
traces
of
interest,
I'm
just
referring
to
the
specific
trace
data
that
you
or
your
teams
might
be
interested
in,
for
example,
as
an
app
developer.
A
A
A
A
So
it
never
has
to
wait
until
all
spins
and
requests
have
finished
and
it's
unbiased
because
it
never
looks
at
the
trace
to
make
a
sampling
decision
today.
Open
telemetry,
sdks
ship,
with
a
number
of
built-in
head-based
samplers
we've
got
parent-based
always-on
trace
id
ratio
based
and
there's
also
always
off.
A
I'm
going
to
talk
about
the
built-in
samplers
a
little
bit
now
so
the
open,
telemetry
default
sampler
is
actually
a
composite
of
two
of
the
samplers.
We
have
the
parent-based
sampler,
which
takes
a
required
parameter
for
what
you
want
to
use
for
your
root
spans.
So
that's
why
it
says
root
is
always
on
because
we
are
using
the
always
on
sampler.
As
the
name
suggests,
we
will
always
sample
the
span
of
the
root
span
and
the
way
the
sampler
works
is
by
asking
a
few
questions
of
every
span.
A
First,
are
you
root
span
of
yes?
Well
guess
what
I'm
always
going
to
sample
you
if
you're
not
a
root
span?
That
means
hey.
You
have
a
parent
span
and
I
was
going
to
ask
okay.
Was
your
parents
sampled,
if
yes,
hop
on
the
train,
you're
getting
sampled
and
if
not,
you're,
not
getting
sampled
and
in
other
words,
the
sampling
decision
gets
propagated
down
to
all
the
child
spans?
A
So
in
this
way,
this
is
how
the
default
sampler
works,
to
collect
every
span
of
every
request
in
your
system,
assuming
all
your
services
are
instrumented
with
open
telemetry
and
using
the
default
okay.
This
diagram,
which
I
also
created
painstakingly
lovingly
created
for
you
all,
shows
an
example
of
what
we
might
see
in
our
tracing
back
end
that
we're
using
the
blue
dots
represent
the
root
spans
and
the
blue
rectangle
represents
where
the
sampling
decision
is
made.
A
Okay,
the
name
of
this
sampler
is
a
mouthful
for
me,
so
this
sampler
uses
the
trace
id
to
make
a
sampling
decision
with
respect
to
the
sampling
rate
that
you
configure
in
open
telemetry.
When
we
combine
this
sampler
with
the
standard
random
trace
id
generator,
we
get
a
mechanism
by
which
to
do
probabilistic
or
random
sampling.
A
A
I
do
want
to
note
here
that
if
you
have
multiple
connected
services,
instrumented
with
a
variety
of
language
sdks
because
they're
in
different
languages,
the
open
telemetry
specification
recommends
that
you
use
this
only
for
root
spans,
and
so
how
earlier
we
saw
the
route
takes
in
the
os
on
sampler,
we'll
just
simply
switch
it
out
for
the
trace
id
ratio
based
sampler.
So
we'll
see
that
example
in
the
demo
coming
up,
so
here's
the
diagram
showing
an
example
of
what
we
might
see
in
our
chasing
back
end.
A
Okay,
so
that's
all
I
wanted
to
cover
for
head-based
sampling.
Let's
talk
about
tail-based
sampling,
the
biggest
difference
between
head-based
sampling
and
tilde
sampling
is
that
hotel-based
sampling,
as
the
name
suggests,
the
sampling
decision
is
made
only
after
all
the
spans
in
a
request
have
finished.
A
It
is
useful
for
efficiently
seeing
those
traces
of
interest
and
it's
optimal
because
we
get
to
keep
all
our
spams
in
context
and
don't
lose
and
don't
get
broken
traces
when
we
need
the
most
today
to
do
tail
sampling
using
open
telemetry.
You
have
to
stand
up
a
collector,
which
I
mentioned
at
the
beginning
and
implement
the
component
called
the
tail
sampling
processor.
A
Multiple
policies
exist
today,
and
you
also
have
the
flexibility
to
add
more
if
you
like.
This
is
just
a
short
list
of
the
policies
that
you
can
use
today.
I'm
going
to
take
you
through
the
screenshot
here,
so
this
shows
an
example
configuration
for
the
tail
sampling
processor
in
our
collector
config
email
file.
So
here
we've
got
our
tail
sampling
processor,
that's
the
name
of
it.
A
The
first
three
lines
here
are
all
optional:
configurable
settings
you
can
use
or
not
a
decision
weight
is
the
time
to
wait
from
the
beginning
of
the
span
or
beginning
of
the
trace
to
make
a
sampling
decision.
Numb
traces
is
number
of
traces
kept
in
memory
and
expected
new
traces
per
sec
is
expected
new
traces
per
sec.
A
And
finally,
we
have
the
policy
section
here.
There
is
no
default,
so
you
will
have
to
define
at
least
one
policy
to
use
the
tail
sampling
processor.
In
this
example,
I
have
two
policies
set
up:
I've
got
a
status
code
policy
and
a
probabilistic
policy,
and
this
will
essentially
get
me
all
my
error.
Traces
as
well
as
a
random
sampling
of
remaining
traces.
A
A
Good
great
awesome?
Thank
you.
So
if
you
have
any
lingering
questions,
please
hold
on
to
them
and
I'll
try
to
get
to
them
during
the
q
a
or
hopefully
some
of
them
will
be
cleared
up
with
this
live
demo.
I'm
about
to
show
you
okay,
so
first
I'll
give
you
the
context
of
the
demo
and
then
I'll
walk
you
through
this
scenarios
that
we're
gonna
see
and
then
we'll
do
a
quick
demo
reflection.
A
A
It
produces
exactly
one
error
and
for
the
purposes
of
this
demo,
I've
decided
that
my
choices
of
interest,
whoops,
are
choices
with
errors
and
we're
going
to
be
exporting
all
our
choices
to
jager,
so
we'll
be
looking
at
our
traces
of
jager,
and
my
goal
here
is
to
find
out
which
sampling
strategy
is
optimal
for
getting
me.
What
I
want.
A
A
A
A
A
A
So
here
we
have
20
traces,
you're,
so
correct
and
as
I'm
scrolling
through
here,
so
each
of
these
lines
is
a
trace.
If
I
click
on
one,
this
might
look
familiar.
It
was
in
the
screenshot
that
I
showed
you
earlier.
These
are
all
spans
and
you
can
see
like
the
different
services
and
if
I
scroll
down
a
little
bit
more
hey,
look
there's
my
error
trace
that
I
wanted
along
with
a
lot
of
not
so
interesting
traces.
A
Now,
when
I
come
in
here,
if
you
guessed
that
we
will
now
see
60
traces,
you
are
correct.
I'm
so
glad
that
worked
just
now.
Okay,
so
60
traces,
and
if
I
scroll
down,
I
will
see
hey
there's
one
of
my
ear
traces
and
I
should
see
one
more
there
you
are
and
then
our
third
one,
which
was
from
the
first
time
we
ran
the
load,
generator
perfect.
A
So
now
I'm
passing
in
the
sampler,
as
you
can
see,
I'm
only
using
on
bootspans
per
the
open,
telemetry
specification
recommendation.
That
was
a
lot
of
multi-syllable
words
at
once,
all
right
and
now
I'm
going
to
do
it.
For
my
second
service,
which
is
a.net
service,
and
finally,
I'm
going
to
do
the
same
thing
for
my
third
service,
which
is
a
python
service,
and
also
you
can
see
here
that
the
sampling
rate
that
I've
configured
is
25
percent.
A
A
A
A
Oh
and
that's
so
interesting,
so
I
actually
got
using
the
trace
id
ratio
base
sampler
this
time
it
looks
like
I
got
two
of
my
error:
traces
which
is
interesting.
I've
run
this
demo
and
had
zero
of
the
error.
Traces
show
up.
I've
had
like
one
air
trade
show
up,
so
it
really
is
randomized.
As
you
can
see.
A
A
A
You
could
use
the
default
as
well.
I
want
to
be
extra
sure,
so
that's
why
I'm
using
the
always
on
sampler,
so
I'm
going
to
go
ahead
and
comment
out
the
old
one
and
go
ahead,
and
now
I
want
to
show
you
what
that
looks
like
in
the
hotel,
collector,
config,
email
file,
so
you'll
remember
the
screenshot
that
I
showed
you
of
the
example
configuration
so
here,
I'm
using
the
decision
weight,
optional
setting
and
I've
already
got
a
policy
here.
A
A
A
A
A
A
A
A
Let's
see,
how
are
we
on
time
just
to
do
a
quick
demo
reflection
so
using
our
open
telemetry
default?
We
always
saw
our
traces
of
interest,
but
we
also
saw
everything
else.
So
then
we
wanted
to
do
to
get
fewest
traces.
So
we
used
the
trace
id
ratio
based
sampler
for
our
root
spans,
and
now
we
got
a
random
sampling.
We
didn't
always
see
our
traces
of
interest,
so
then
we
wanted
to
try
the
tail-based
sampling
processor.
A
A
A
We
have
to
hold
those
spans
in
memory
somewhere
before
the
last
band
has
finished,
and
this,
of
course,
can
eat
up
application
resources
if
the
if
the
spans
are
stored
locally
or
additional
network
bandwidth,
if
your
choices
aren't
stored
locally,
additionally
determining
the
interesting
traces
so
earlier
I
said
you
don't
need
a
ton
of
data
to
find
the
right
insights.
You
just
need
the
right
sampling.
A
A
Well,
if
you're
suddenly
experiencing
severe
network
congestion
and
your
tracing
solution
is
now
exporting
up
a
ton
of
traces
of
latency
you're,
going
to
see
a
spike
during
that
period.
In
data,
egress
storage
costs
and
potentially
data
ingest,
depending
on
how
your
back-end
vendor,
depending
on
their
pricing
model,.
A
However,
the
more
traces
you
have
been
kept
in
memory,
the
more
memory
you're
going
to
need
and
the
more
computing
and
processing
power
you're
going
to
need
to
look
at
each
span
to
see
if
it
fits
to
see
if
any
of
them
fit
your
bill
for
an
interesting
trace
after
certain
load.
One
collector
is
not
going
to
cut
it.
So
now
you
have
to
look
at
collect
a
deployment
pattern
and
think
about
load,
balancing,
there's
like
no
like
smooth
transition
to
getting
water
hold
on.
A
There
does
exist
today,
an
exporter
called
the
load
balancing
exporter
that
you
can
use
if
you're
running
multiple
instances
of
the
collector
with
the
tail
sampling
processor.
So
there
is
hope
and
finally,
the
future
of
the
tail
summing
processor.
It
sounds
a
little
ominous,
but
it's
simply
referring
to
there's
an
open
issue.
Right
now,
in
the
collector
contrib
repository
around
replacing
the
tail
sampling
processor,
the
conversation
is
more
around.
A
Will
breaking
up
some
of
the
policies
and
building
them
out
into
their
own
processors,
be
more
performance
than
or
be
more
or
as
performant
as
using
the
tail
sampling
processor?
If
you're
interested
in
this
discussion,
I
encourage
you
to
reach
out
to
jurassic
crowling.
He
is
for
the
grana
and
he
is
kind
of
the
mastermind
behind
this
idea.
A
And
I
think
we've
arrived
so
in
summary,
tail
sampling
is
great
for
efficiently
getting
you
what
you
want,
but
there's
going
to
be
a
lot
of
challenges
with
figuring
out
what
exactly
that
means
for
your
team,
as
well
as
implementing
it.
A
couple
of
quick
feature-ish
things
I
wanted
to
to
touch
on.
So
there's
work
being
done
right
now
to
add
probabilistic
log
sampling
to
the
open,
telemetry
specification,
which
is
pretty
cool.
A
I
encourage
you
to
head
to
the
collector
contrib
repository
to
find
out
more
and
also
wanted
to
talk
about
contributing
if
you're
interested
in
getting
involved
with
open
telemetry.
That
is
a
very
exciting
time
whether
you
want
to
do
co-contributions
or
help
improve
docs.
I
know
the
community
team
is
very
excited
to
have
all
your
great
minds
involved
so
definitely
head
to.
Oh,
actually,
I
have
the
slides
a
slide
at
the
end.
A
That
will
have
some
links
for
you
to
check
out
and
big
shout
out
to
everyone
that
is
on
this
list
and
all
of
you
who
came
today
and
for
anyone
watching
the
recording.
I
appreciate
you
all
so
much
for
being
here
and
all
right.
If
anyone
has
questions
please
head
to
the
mic,
it
looks
like
I
am
right
on
time,
but
I
see
a
gentleman
back
there.
A
So
let
me
leave
this
slide
here
for
you,
this
bar
code
here,
if
you
have
feedback
that
you
would
like
to
share
with
the
open,
telemetry
community
team,
about
your
experience,
whether
you've
used
it
or
like,
what's
stopping
you
from
using
it,
we
would
love
to
know.
And
yes,
what
is
your
question
that
I
can
hopefully
answer.
B
A
That
is
a
great
question,
and
it
might
so,
I
think
that's
one
of
the
the
challenges
right:
patel
based
sampling
in
general.
There
is
that
configuration
setting
decision
weight,
so
you
can't
have
it
set
to
a
longer
period
of
time.
I
think
the
default
is
30
seconds,
which
is
like
a
pretty
long
period
of
time.
A
A
I
do
encourage
you
to
you
know,
have
this
discussion
in
the
open,
telemetry
slack
channel
or
feel
free
to
come
to
one
of
the
special
interest
group
meetings,
but
I'm
also
actually
curious
about
that.
So,
if
you
want
to
hang
around
I'm
going
to
grab
your
contact
info
and
then,
if
I
find
out
the
answer,
if
you
found
out
the
answer
first,
like
let's
share
that.
C
D
So
there's
a
lot
of
talks
today
and
well
yesterday
and
going
to
be
tomorrow
about
ebpa,
for
it
bppf
lease
and
bp
of
that,
and
it
seems
that
ebpf
can
do
all
this
magic
like.
How
much
did
you
call
take
time
and
how
much
each
function
took
and
all
of
that
and
it
seems
to
be
replacing
like
distributed
tracing
in
that
way.
So
do
you
see
ebpf
as
a
distributed
tracing
killer,
because
you
said
that
distributed
racing,
its
implementation
of
a
telemetry?
D
E
E
There's
pros
and
cons
right,
and
so
one
thing
that
ebpf
doesn't
do
is
that
you
can't
really
throw
in
for
me,
or
at
least
when
you're,
trying
to
observe
you
can't
really
throw
tags
into
requests
and
so
trying
to
trace
something
across
multiple
hops
like
distributed
tracing
like
what
reese
is
showing
is
actually
a
little
bit
more
difficult
to
do
with
ebpf
right,
so
you're
good
at
capturing
spans
with
ebpf,
but
getting
distributed.
Traces,
that's
something
that
actually
open.
Telemetry
is
very
good
at
right,
and
so
there's
pros
and
cons
with
these
things.