►
From YouTube: 2022-10-20 meeting
Description
cncf-opentelemetry meeting-2's Personal Meeting Room
A
B
B
A
A
B
B
B
Josh's
write-up
of
our
last
discussion-
and
it's
really
good
I
hadn't
I
hadn't
read
it
before
now,
but.
D
I
read
it
yesterday
and
I
thought:
I
wasn't
there
two
weeks
ago,
so
it
was
I
thought
the
notes
were
well
done.
C
But
something
I
had
briefly
brought
up
towards
the
end
of
the
last
meeting
and
I
think
Josh
might
have
I.
Did
it
for
this
week's
or
this
meetings
agenda.
B
B
I
had
made
a
comment
that
you
could
do
this
somewhat
inaccurately.
B
B
C
C
So
I
had
one
more
topic
to
chat
about
which
was
more
like
a
SDK
side.
Tail
sampling
I
can
explain
what
I
mean
by
that,
but
but
before
that,
we
can
probably
start
with
the
linked
traces
and
see.
If
there
is
anything
we
can
do
to
improve
the
situation.
So
the
challenge
or
the
problem,
as
we
all
know,
is
since
the
link
traces
are
at
least
what
I'm
saying
in
Microsoft
link
traces
are
becoming
a
common
pattern,
particularly
for
say,
long
running
operations
or
like
anything
that
involves
like
a
producer
consumer
pattern.
C
Things
like
that
the
standard
use
cases
for
link
traces,
and
so
how
do
we
really,
if
I
take
it?
If
we
break
it
up
into
say
two
categories:
right,
head
sampling
and
then
tail
based
sampling
with
head
based
sampling,
since
the
probabilistic
sampling
is
based
on
a
single
Trace,
it
feels
like
there
is
no
easy
solution
to
say:
hey
like,
but
my
link
Trace
is
probably
not
going
to
get
sampled
in,
so
I
may
not
get
a
fully
consistent
group
of
all
my
linked
traces
together.
C
C
C
I
haven't
done
a
lot
of
thinking
about
it,
but
I
just
wanted
to
First
brainstorm
with
the
group.
If
there
is
a
if
there
has
been
already
some
discussions
around
this
in
the
past
since
I
joined
the
stick
only
recently
I
don't
have
all
the
context
so
that
that's
basically
my
the
summary
of
the
problem,
at
least.
A
I,
don't
recall
discussing
linked
traces
before
so.
This
is.
This
is
a
new
topic
now
I
I
haven't
used
link
traces
personally,
so
could
you
explain
me
to
me
how,
when,
when
we
want
to
link
to
another
Trace
from
S,
Pen
I
understand
that
this
link
is
from
a
span?
C
So
so,
when
the
link
is
created,
usually
the
destination,
so
you're
basically
saying
here
is
my
span
ID
and
the
trace
ID,
to
which
I'm
linking
to
so
usually
that
already
exists.
Okay,
at
the
time
you're
doing
the
linking
so
I.
D
A
A
B
So
in
a
deterministic
sampling
world
you
could
tell
at
the
point
you
created
the
link
if
the
trace
was
going
to
be
sampled,
and
so
you
could
sort
of
avoid
bothering
to
create
the
link.
If
you
knew
that
thing
wasn't
going
to
exist
in
your
in
your
storage
system,
but
that's
kind
of
a
negative
decision.
It's
it's
you.
You
could
decide
that.
B
B
I
I
don't
know
it's
just
like
an
observation
that
that
you
could
maybe
avoid
trying
to
create
a
a
link.
You
know
a
sampled
span,
then
we
have
a
link
to
a
non-sampled
span
and
it,
but
it
doesn't
I,
don't
know
if
that
helps
enough.
C
A
Going
back
to
the
example
that
you
provided
with
cons,
producer
and
consumer,
with
some
messages
of
my
sit
in
some
queue
for
hours,
I
understand
that
when
the
message
is
picked
up
by
a
consumer,
we
want
to
create
a
new
thread,
a
new
Trace.
At
this
point
right
and
well,
since
we
know
and
something
about
the
producing
Trace
we
could,
we
could
just
copy
the
sampled
flag.
B
It
also
assumes
you
have
access
to
the
entire
Trace,
whereas
what
may
have
happened
was
that
the
producing
Trace
ID
was
copied
in
to
some
package
of
data
that
flowed
through
your
your
processing
queue.
Yeah,
you
see
it
just
it's
one
more
piece
of
information
you
would
have
to
copy
of.
C
Yeah
yeah
it
it
could
work,
I
think
the
sampled
flag
would
have
to
be
I
agree.
That
could
be
another
data
point
and
then
it
becomes
like
a
new
type
of
sampler
which
looks
at
the
it's,
not
exactly
parent-based
sampling,
but
it's
kind
of
like
a
link
based
sampling
right.
It
looks
look
at
whether
my
thing
exactly.
C
Can
work
for
some
situations,
I
think
the
complexity
comes
in
as
because
of
the
end-to-end
mappings
right,
like
since
different
spans
in
a
trace
can
link
to
different
spans
in
different
traces
right
like
there
can
be
n
different
links
from
the
same
Trace,
maybe
not
a
80
scenario,
but
I
think
the
making
it
like
a
general
solution.
I
think
the
problem
comes
with
what.
If
my
the
link,
one
was
sampled
in
but
link,
two
was
not
sampled
and
I
think
there
are
probably
some
more
since
it's
basically
just
a
graph
yeah.
A
Well,
somewhat
related
problem:
well,
not
really
the
problem
space,
but
in
the
solution
space
is
a
desire
to
to
observe
complete
user
sessions
from
a
browser.
So
we
want
not
just
a
single
trace,
but
we
want
to
to
to
have
all
traces
that
belong
to
the
same
session
and
here
the
solution
that
I
I
think
might
work
is
that
when
we
see
the
first
trace
of
a
session,
so
we
need,
of
course
to
have
the
session
ID.
A
We
generate
the
R
value
as
as
usual,
but
later
when
there
is
a
new
Trace
being
created
for
the
same
session,
we
just
use
the
same
r
value
that
we
used
before,
so
we
are
not
generating
new
R
values
again
this
this
increases
the
chances
of
keeping
all
the
traces
belonging
to
the
same
session
as
a
whole,
because
they
all
share
the
same
r
value
right
so
in
general,
I
think
we
should.
We
should
think
about
providing
a
mechanism
in
SDK
to.
C
A
Well,
I
I,
I,
I
I.
Don't
like
this
idea
of
going
away
from
the
r
values
I
understand.
They
are
not
perfect,
but,
as
we
discussed
using
using
Trace
ID
and
a
hash
function
it
it
is
opening
another
box
of
problems
and
we
we
are
not
sure
how
we'll
how
it
will
go.
But
again,
the
the
problem
that
we
are
trying
to
solve
is
not
really
tied
to
the
r
values,
because
if
we
decide
to
go
with
random
bits
in
Trace
ID
again,
we
we
have
to
well.
This
is
no.
A
B
C
See
yeah
so
I
saw
someone
asked
in
the
slack
Channel
or
somewhere.
I
saw
something
about
this
client
or
the
browser
based
right.
It's.
B
I
mean
we're,
seeing
it
we're
seeing
a
ton
of
it
right
now,
people,
you
know,
there's
there's
been
a
lot
of
people
going,
wait:
real
user
monitoring,
ROM
whatever
is
awfully
similar
to
Telemetry,
and
how
do
we
you
know?
Are
there
ways
we
can
sort
of
unify
that
stuff?
And
you
know
you
have
the
problem.
You
have
all
sorts
of
interesting
problems
when
you
do
this
as
a
vendor.
Honeycomb
is
dealing
with.
B
You
know,
authentication
issues,
and
you
know
now,
like
a
collector,
becomes
a
challenge
like
you
get
these
front-end
teams
that
are
trying
to
do
Telemetry
and
they
don't
have
the
ability
to
stand
up
their
own
collectors
and
things
like
that.
It's
it's
really
interesting,
trying
to
figure
out
how
to
solve
some
of
these
problems,
but
but
we
are
seeing
a
tremendous
amount
of
interest
in
browser-based,
Telemetry
and
traces
and
all
that
kind
of
stuff.
So
I,
don't
you
know.
B
Smart
whatever's,
in
the
browser
smart
enough
to
not
send
everything
but
to
sample
it
in
some
useful
way,
and
so
now
you're
back
to
head
sampling,
which
you
know
has
all
the
problems
we've
been
talking
about.
So.
D
So
I'm
curious
from
like
the
customer
perspective
where
people
would
sort
of
argue
like
are,
are
the
following
two
problems:
equally
urgent:
the
the
negative
case
that
I
do
not
sample
a
sort
of
descendant
span
or
like
pseudo
descendant.
D
If
it
would
link
to
something
that
wasn't
sampled,
there
was
that's
like
potentially
the
case
addressable
with
like
memory
with
like
a
bloom
filter
or
Perfect
Memory
or
whatever,
but
then
the
other
question
was
like:
do
I
can
I
make
sure
that
I
sample
a
span
or
a
trace
which
may
give
rise
to
spans
that
link
to
it
like
arbitrarily
far
in
the
future,
so
I
guess
I
would
ask
like
those
two
perspectives
like
for
what
people
use,
links
for
and
I
won't
like
attempt
to
answer
this,
because
I've
never
used
links
but
I
understand
some
of
the
like
written
motivations
for
them.
D
A
With
sampling,
we
we
look,
we
look
at
the
algorithms
here
and
with
with
some
kind
of
academic
angle.
This
is
we.
What
we
discuss
here
is
very
pure
technical
issue,
but
in
practice
you
will
always
have
other
components
that
create
a
in
making
decisions.
A
So,
for
example,
when
we
see
let's
take,
let's
consider
this
linked
linked
traces
example.
If
we
want
in
our
consumer
to
create
a
new
trace-
and
we
see
that
we
were
linked
with-
we
want
to
link
with
a
trace
that
was
already
assembled.
This
is
only
a
hint
that
we
want
to
sample.
A
We
still
might
need
to
consider
other
things
like
General
throughput.
If
we
are
under
high
volume
of
throughput
of
of
traffic,
then
then
we
might
not
sample
after
all.
So
it's
just
a
hint
and
the
mechanism
that
we
are
talking
about
is
just
making
these
hints
possible,
but
it's
not
mandatory
for
the
consumer
to
follow
exactly
the
decisions
that
were
made
Upstream
and
the
same
is
with
regular
sampling
with
regular
traces.
A
I
talked
about
this
last
week
in
general,
you
may
have
some
notes
that
have
enormous
amount
of
traffic
and
they
have
to
make
their
own
autonomous
decisions
about
sampling.
Because
of
that.
B
Yeah
we
had
one
large
customer
today
asking
about
making
independent
sampling
decisions.
They
have
different
regions
that
their
systems
are
running
in
and
most
of
the
time,
their
transactions
don't
cross
regions,
but
sometimes
they
do,
and
so
they
want
to
be
able
to
not
have
massive
amounts
of
traffic
going
between
regions
but
but
just
sort
of
have
both
sets
of
regions
making
the
same
decisions,
or
at
least
mostly
making
the
same
decisions
and
and
yeah.
This
is
a
interesting,
interesting
question.
I
mean,
like
I,
think
you
know
when
I
think
about
it.
B
I
think
like
what
I
really
want
and
can't
have
is
Downstream
when
I'm
processing
that
thing
that
I
got
five
minutes
ago,
and
it
sat
in
my
Kafka
queue
for
all
that
time
and
now
I
finally
get
to
it
and
then
it
errors.
I
would
like
to
know
that
I
can
go
back
and
see
the
thing
that
caused
it
to
happen,
and
obviously
you
know
unless
I
just
hang
on
to
it
until
everything's
done
I
can't
you
know
all.
D
B
A
B
A
B
D
Yeah
yeah,
it
seems
yeah
I,
guess
like
how
I'm
leaning
toward
summarizing
this
conversation.
It's
like
we're
realizing
that,
for
some
perhaps
most
use
cases
for
spam
links
those
share
out
like
a
lot
of
sort
of
auxiliary
requirements
when
it
comes
to
sampling,
as
do
like
parent
and
child
traces
and
like,
and
so
it
sounds
like
you
know,
we're
wanting
the
same
sort
of
capabilities
of
like
you
know
deciding
all
at
once.
B
C
Yeah
I
think
the
session
concept
is
a
bit
easier
to
wrap
my
head
around
right.
Like
basically
you
you
start
a
session
and
then
you
do
a
bunch
of
things
within
that
session,
and
then
you
end
the
session.
So
there
is
like
a
natural
hierarchy
and
it
feels
like
a
retractable
problem
to
solve
like
achieving
consistent
sampling
within
everything
within
that
session.
It's
it
feels
more
self-contained,
whereas
the
linking
thing
is
like
not
that
self-contained
right,
because
you
start
a
trace
and
then
something
happens
to
it
and
then
could
be
minutes
later.
B
C
So
that
is
feels
more
more
challenging
of
the
two
problems
session
based
sampling
feels
like
a.
C
B
And
it's
kind
of
like
he,
you
can
think
of
the
R
value
as
Peter's
been
talking
about
it
in
well.
It's
a
session.
Id
I
mean
it
just
it
kind
of
kicks
the
problem
up
one
level,
but
it's
a
similar.
It's
a
similar
sort
of
thing.
There
is
some
number
that
is
shared
among
you
know
as
a
shared
context.
That
is
then
used
for
right
right.
A
Yeah,
well,
the
the
the
the
K
the
case
with
sessions
is
easier,
not
only
because
it's
easier
to
understand,
but
also
because
all
traces
are
created
within
a
session,
and
we
know
that
with
linking
traces,
the
the
number
of
scenarios
is
different
and
it
might
be
possible
to
link
traces
which
already
started
both
independently
and
somehow
they
got
linked
because
of
some
SDK
calls,
and
we
we
cannot
do
anything
about
that,
because
it's
too
late.
C
Yeah,
going
back
to
Spencer's
question
I
mean
the
scenarios
for
me.
I
personally
have
heard
a
lot
of
requirements
around
the
linked
traces
session
based
I
haven't
heard
yet,
but
I'm
I
can
totally
see
why
that
could
be
totally
important
as
well.
I
I,
don't
know,
I
mean,
depending
on
the
resources.
C
I,
don't
know
whether
we
should,
if
we
have
bandwidth
to
I,
mean
if
we
all
agree
there
are
problems,
then
we
can
potentially
have
different
work
streams
trying
to
solve
both
I
guess:
I,
don't
know
if
it's
a
prioritization
resourcing
question
between
the
two
or
whether
we
can
kind
of
put
efforts.
They
end
both
independently
because
they
seem
to
be
different
problems
and
different
solutions
for
both.
A
B
I
think
I
mean
you
could
toss
this
out
because
it's
just
kind
of
percolating
in
right
now,
if
we
had
the
concept
of,
let's
call
it
our
value
or
session
ID
or
whatever
it
wants
to
be,
but
basically
saying
if
you
want
your
links
to
be
sampled
according
to
the
same
rules
as
your
initial
praises
or
all
the
traces
in
a
you
know
a
session,
then
then
you
need
to
pass
on
this
r
value
and
propagate
that
through
the
system.
B
So
you
can
create
a
link
that
doesn't
contain
an
R
value,
and
in
that
case,
then
the
sampling
decisions
will
be
made
independently.
But
if
you
pass
on
the
R
value
as
part
of
the
link,
then
at
that
point
you
know
at
that
point
we
could
state
that
we
want
the
sampling
decision
to
be
made
in
the
same
way
yep.
So
this
could
become
an
optional
feature
of
linking
essentially
where
you
know.
B
If,
if
this
is
something
that's
meaningful
to
you,
if
you're
tracking
sessions
or
you'd
like
to
track
a,
you
know,
query
across
your
systems
that
propagate
it
over
time
with
multiple
traces,
then
you
need
to
pass
the
R
value
along
to
seems
like
a
plausible
pace
could
be
made
without
having
to
say
now.
Everybody
has
to
think
about
this
differently.
D
Yeah
and
as
I
was
trying
to
sort
of
just
steal.
What
we
were
talking
about.
I
was
looking
into
like
the
open,
Telemetry
API
spec
for
for
adding
a
link
to
a
span,
and
it's
like.
Oh
you
give
it.
The
span
context
and
I
was
like.
Let
me
double
check
that
what
is
in
the
spanning
context
and
the
span
context
contains,
among
other
data,
the
trace
Flags,
including
sampled
and
choice
State,
including
R
values.
D
So,
just
like
sending
additional
hints,
it's
actually
the
case
that,
like
those
are
part
of
the
data
that
is
already
being
relayed
in
order
to
be
able
to
create
a
link
back
to
a
span,
so
we're
not
so
like
in
this
direction
of
solution.
We're
actually
not
talking
about
adding
something
to
you
know
the
side
channel
that
you
propagate
metadata
through.
B
D
D
B
C
Thanks
Spencer
I
will
take
a
look
at
that,
so
I
think
one
main
open
issue
seems
to
be
the
r
value
like
proposal,
keeping
keeping
it
bringing
it
back
on
the
table.
So
Peter
is
this
something
you
would
follow
up
with
Josh
or.
C
No
I
think
I
think
in
the
last
meeting.
Josh
was
going
more
towards
the
trace
ID
Randomness
as
the
placement
for
the
R
value,
custom,
r
value
generation,
so
I
think
we
need
to.
A
Well,
we
definitely
support
this
new
random
flag.
We
want
to
have
it
because
it
helps
us
in
in
a
number
of
corner
cases
that
we
had,
in
particular
with
stale
based
sampling
based
on
R
values,
which
we
couldn't
do
without
when
when
we
want
to
to
sample
with
probability
different
than
a
power
of
two
now
with
a
random
flag,
we
can
do
it.
A
So
it's
a
it's
a
big
win
and
there
is
still
this
problem
that
some
people
apparently
really
want
to
have
something
with,
let's
say,
75
percent,
which
in
my
opinion,
doesn't
make
much
sense.
But
of
course
we
cannot.
We
cannot
argue
with
that.
If
they
want
to
do
this
well,
they
can.
A
They
can
use
the
new
mechanism,
but
I
I,
don't
I,
don't
think
what
Josh
wrote
really
eliminates
our
values
at
all,
because
they
are
still
needed
well.
At
least
P
values
are
needed
and
they
both
can
coexist
in
in
in
in
in
in
some
form,
so
yeah
I
hope
we
will
go
get
back
to
discussing
this
in
more
detail
and
while
we
are
waiting
for
this
random
flag
to
become
standard
which
Josh
expected
to
take
about
a
year
just
a
little
bit
pessimistic
in
my
opinion,
oh
we
have.
C
The
yeah
I
agree
they
can
potentially
coexist
and
I.
Don't
know
whether
in
the
session
based
sampling
yeah,
we
would
need
to
propagate
something
which
is
common
across
entrances.
Okay,
I
see
yeah,
they.
C
A
So
so,
going
back
to
the
session
example
with
our
values
is
fairly
simple,
because
we
just
need
to
ensure
that
all
these
new
traces
have
the
same
r
value
with
with
a
random
part
of
Trace
ID,
it's
much
more
difficult,
but
I'm
sure
we
can.
We
can.
We
can
work
something
out.
C
We
only
observation,
I
have
on
the
other
proposal.
The
non
r
value
based
proposal
is
in
the
in
the
latest
thing
that
Josh
has
written
up.
It
doesn't
necessarily
depend
on
the
w3c
spec
becoming
standard
and
implementing
right.
The
assumption
is,
it
assumes
that
the
last
seven
bytes
are
going
to
be
random,
even
if
that,
like,
irrespective
of
whether
that
flag,
new
w3c
flag
is
set
or
not.
A
A
C
A
So
if,
if
I
remember
reading
this,
that
meant
that
we
should
not
wait
for
wr3
C
committee
to
to
accept
this
new
random
flag.
We
can
just
use
it
right
now
and
and
use
and
treat
them
some
portion
of
Trace
ID
as
random.
If
this
random
flag
is
set,
we
still
will
test
the
flag.
Even
this
is
a
non-standard
flag.
Today
that
was.
B
A
This
is
this
lives
in
tray
status
next
to
the
sample
flag.
So
sampled,
black
is
just
a
single
bit
that
was
used
in
a
byte
that
was
passed
as
flags,
and
now
we
are
talking
about
using
the
second
bit
of
this
bike.
I
see.
A
B
C
It's
in
the
trace
parent
header,
the
last.
C
So
Peter
my
impression,
is
slightly
different
based
on
the
discussion
in
the
last
meeting.
It
was
that
the
new
tracity
based
ratio,
Trace
ID
ratio
based
sampler,
will
treat
the
last
seven
bytes
of
the
trace
ID
as
random
and
originally
The
Proposal
was
what
you
said,
which
is
look
at
the
new
flag.
If
the
flag
is
set,
then
use
it
like
you
get
a
guarantee
that
hey
the
last
seven
bytes
are
random,
but
then
I
think
the
question
was
about
hey:
what
happens
if
that
flag
is
not
set?
C
What
should
be
the
fallback
algorithm
and
that's
where
I
thought
the
decision
was?
We
will
still
do
best
effort
in
assuming
that
it
is
going
to
be
the
last
seven
bytes
are
random
because
most
of
the
current
implementations
of
Trace
ID
generation,
including
AWS
x-ray
they're,
all
the
last
seven
bytes-
are
already
randomly
generated,
so
it
doesn't
hurt
to
rather
than
some
other
custom
fallback
logic,
it
doesn't
hurt
to
just
use
the
same
thing,
because
there
is
higher
probability
of
generating
consistent
sampling
with
that.
No.
D
I
had
a
question
going
back
to
just
and
Link
question
and
just
use
cases
for
it
is,
is
the
the
downstream
span
that
typically
went
back
to
the
earlier
one
is
I'll
call
it
span.
B
is
B
like
almost
always
a
root
span,
or
is
it
ever
like
part
of
some
Trace
that
was
started
and
then,
and
then
you
know
is,
is
be
always
a
root
span?
That's
it.
A
No
I
I
was
just
saying
that
the
API
allows
you
to
link
anything
to
anything
right.
C
Yeah
I
think
to
add
to
that
yeah
there's
no
guarantee,
but
in
the
consuming
example,
if
a
new
Trace
is
being
started
based
on
that
yeah,
it's
likely
going
to
be
a
root
span,
but
there
are
other
use
cases
where
it
doesn't
have
to
be
a
root
span
at
all.
So.
D
C
There
is
yeah
I
think
the
challenge
is
with
this.
Propagation
is
again
I
just
need
to
think
through.
Where
you
have.
Let's
say:
Trace
one
span,
one,
it
is
doing
something.
Then
it
links
to
span
2
in
Trace
two
and
then
Trace
once
does
some
of
the
some
other
things
in
the
country,
and
then
there
is
like
a
link
to
a
different,
like
some
other
Span
in
Trace,
one
links
to
some
other
plan
in
Trace
three.
So
basically,
it's
just
a
random
graph
of
things.
C
So
it's
very
I,
don't
know
if,
like
the
decisions
were
taken
independently
by
some
two
traces
and
then
somewhere
down
the
call
tree
some
linking
happens
then
with
multiple
links
in
place.
I,
don't
know
how
this
kind
of
a
consistent
like
what,
if
I'm,
linking
to
two
different
traces
and
one
of
them,
decided
to
sample
one
of
them
decided
not
to
sample,
and
there
will
be
all
those
kind
of
cases.
So
it's
not
that
straightforward
as
the
session
based
one
but
I
think
it's
an
interesting
problem.
D
And
I
did
just
verify
that
it's
the
case
that
that
the
sampler
interface
among
the
data
that
it
receives
to
make
a
decision
are
the
links
that
will
be
added
to
the
newly
created
stand.
D
So
there
is
like
an
API
I
think
to
mutate
existing
span
to
like
add
more
links
to
it,
but
links
that
are
known
at
span.
Creation
time
are
eligible
to
be
utilized
in
some
way
to
make
a
Samsung
decision
and
like
that's
it,
that's
a
specification
that
exists
today.
So
that's
cool.
A
Yeah,
just
just
a
observation
so
with
with
linked
traces,
we
can.
We
can
almost
certainly
say
that
there
will
be
never
a
solution
that
could
satisfy
all
use
cases,
because
you
can
think
about
a
system
where
every
Trace
is
linked
to
everything
right.
So
we
could
have
a
mesh
of
everything.
Every
possible
Trace
is
linked
to
something
that
already
exists,
and
obviously
we
cannot
support
sampling,
synchronous,
sampling
of
all
these
traces
together,
because
that
violates
the
purpose
of
sampling.
D
Yeah
I'm,
almost
thinking
of
this
as
a
like,
a
like
a
head
sampling
solution
to
this
problem,
but
then
like,
as
you
noted,
Peter,
maybe
like
the
tail
sampling
collection
layer,
is
where
more
systemic
requirements
around
you
know.
Maximum
throughputs
and
things
are,
are
satisfied
and
maybe
a
bad
layer.
It
may
reserve
the
right
to
like
further
shed
Data
before
forwarding
it
on
to
your
storage
system.
B
D
B
C
Yeah
just
to
add
one
more
clarification
question
to
that.
So
is
that
a
combination
of
head
based
sampling
and
database
sampling,
where
you
you
did
some
head-based
sampling
and
then
you
were
doing
additional
pruning
in
in
some
tail
based
sampling
system
right.
C
Would
say
you
used
only
like
a
pure
tail
based
sampling?
Then
you
have
the
luxury
of
getting
all
the
even
the
linked
traces
and
then
at
tail
sampling
time.
I.
C
Try
to
achieve
consistent
sampling
by
by
having
some
logic
around
hey,
like
I,
saw
the
stress
it's
linking
to
the
other
stress
and
when
that
Trace
comes
I'm,
going
to
make
a
decision
just
because
I
decided
to
sample
in
this
interesting
trace
and
I
know
that
this
links
to
something
else
I
would
try
to
sample
in
the
other
linked
trays,
also
at
tail
sampling
time.
So
I
think
that
kind
of
goes
back
to
this
issue
that
can
share
I.
C
Think
those
were
kind
of
the
challenges
on
how
to
do
it
at
tail
sampling
time,
but
that,
but
once
hit
something
is
involved.
Then
I
think
the
probability
of
getting
consistent
sampling
across
linked
traces
goes
down
significantly,
because
now
you
you're
not
going
to
be
able
to
get
the
full
group
of
all
the
right.
B
Either
there's
a
timeout,
in
which
case
we
say:
okay,
we
think
we
probably
got
the
trace.
We
make
a
decision
and
we
send
it
on
and
then
we
hit
we
track
that
decision
for
a
little
while,
so
that,
if
new
spans
come
in
as
part
of
that
Trace
we
can,
we
can
also
send
them
along,
but
that
the
period
for
which
we
keep.
B
That
decision
is
not
that
big,
because
we're
kind
of
assuming
you're
all
part
of
the
same
trace
and
that
once
you
have
the
full
once
you
have
the
root
span,
you
know
there
may
be
some
asynchronous
things
that
trickle
along,
but
you
don't
need
they're
not
going
to
be
very
much
delayed,
but
in
this
world
we're
talking
about.
Oh
I
may
put
this
thing
into
a
queuing
system
and
it
may
be
a
long
time
before
it
comes
along.
B
So
we
can't
hang
on
to
that
because
you're,
because
then,
if
we're
just
waiting
to
make
the
decision
for
half
an
hour,
then
that
means
you
can't
look
at
anything
in
real
time,
and
so
in
those
circumstances,
then
you
have
to
keep
the
trace
decision
around
longer,
and
that
was
what
I
was
talking
about
in
that
issue
of
like
well,
maybe
there's
a
shared
redis
cache
or
something
like
that
where
you
can
refer
to
it
and
say,
and
at
a
trace
level
did
I
keep
this
thing
and
and
have
some
sort
of
high
problem.
B
At
least
you
know
that's
where
I
was
talking
about
the
bloom
filter
thing
like
and
maybe
I
don't
have
to
make
this
decision
perfectly.
Maybe
you
know
fast
and
close
to
right
is
good
enough
and
so
throw
it
in
a
bloom
filter,
and
then
you
can
keep
that
around
for
a
fairly
long
time
reasonably
efficiently
and
and
so
at
least,
you
can
make
a
mostly
consistent
tail
sampling
decision.
In
that
case,.
C
Yeah,
if
we
have
appetite
for
this
starting
it,
we
can
just
started
I
just
want
to
get
your
thoughts
on
it.
So
one
of
the
things
like
with
the
tail
sampling
I
think
there
are
multiple
places.
Tail
sampling
could
be
done,
for
example,
like
a
hotel
collected
style
like
a
external
to
the
process.
I
think
there
is
some
interest
from
some
of
the
people.
C
I'm
speaking
to
in
terms
of
hey,
can
I
do
this
site
within
my
process
right
or
at
SDK
time
like
using
like
say,
like
a
custom
filtering
processor
within
the
at
the
at
the
time
of
the
end
of
the
span
when
the
span
is
there?
C
The
challenge,
though,
is
this
means
that
at
the
sampler
interface
like
the
beginning
of
sampling,
it
has
to
lie
and
say:
hey
I'm,
going
to
use
an
always
on
sampler
right,
I
want
everything
and
then
at
the
span
creation
once
the
span
is
actually
done.
Then
this
filtering
kicks
in.
C
So
instead
of
sending
it
to
a
system
like
Hotel
collector,
it's
actually
doing
the
decision
locally.
So
it's
not
going
to
be
a
consistent
trace
and
all
that,
but
but
at
least
those
failed
spans
are
going
to
be
emitted
like
they
like
customers
who
care
about
here.
I
need
to
get
every
single
failed
span
or
a
high
latency
span.
They
they
will
get
the
failed
span
and
like
that
is
the.
That
is
the
good
part.
C
But
then
the
trade-offs
are
the
sampled
flag
is
going
to
be
always
set
to
true,
because
the
always
on
sampler
was
kind
of
like
the
initial
decision
right,
because
you're
kind
of
doing
a
late
decision
at
the
end
of
end
of
the
span.
So
if
some
other
Downstream
service
just
blindly
does
parent-based
sampling,
that
won't
be
right,
because
now
it's
not
exactly
the
parent,
exactly
didn't
decide
to
sample
it
in
so
I
was
just
curious.
If
there
has
been
any
discussions
around
this
topic,
I
did
I
did
prototype
it
out.
C
I
mean
it
seems
to
work,
but
I
think
it
comes
with
a
whole
bunch
of
trade-offs
in
terms
of
it's,
it's
basically
like
a
hacky
approach
to
tail
sampling,
but
but
for
people
who
are
very
sensitive
about,
hey
I,
don't
want
to
go,
manage
my
own
Hotel
collectors
or
like
send
data
out.
It
feels
like
a
something
worth,
considering
some
some
kind
of
a
solution.
A
I'm,
not
sure
if
you
are
familiar
with
was
Baltimore,
did
he
he
wrote
rate
limited,
consistent
probability,
processor
I
think
this
is
the
name
which
is
based
on
our
values,
but
does
more
or
less
what
you
described
so
it
it.
It
makes
decision
for
for
sampling
after
after
the
spans
are
complete.
So
this
is
a
database
sampling
technically,
but
it's
working
in
the
agent
in
in
the
SDK
sorry.
So
if
the
criteria
for
for
selection
would
be
changed
like
you
described,
I
think
this
could
be
adapted
to
to
work
as
as
you
want.
C
Okay,
thanks
Peter
I'll
check
it
out,
but
you
said
rate
limited
consistent
probability,
sampler,
okay,
I
see
so
so
it
makes
some.
It
has
some
selection
criteria
when
the
span
is
complete.
It
will
yes,
so.
A
Right
right,
so
the
the
goal
was
to
limit
the
rate
but
again,
but
have
it
done
in
a
fairly
consistent
way?
Yeah.
C
I
see
cool
I
can
reach
out
to
chat
more
about
it.
Yeah,
in
my
case,
it
is
not
rate
limited
directly,
but
it's
more
like
a
at
the
end
of
the
span.
Completion
I
checked
for
hey.
Is
this
a
failed
span?
If
so
always
continue
the
pipeline
like
send
it
to
the
exporters,
if
not,
do
some
probability
based
sampling,
like
one
percent
or
something.
B
But
essentially
the
thing
the
problem:
you're
bumping
up
against
is
a
thing
that
I
have
I,
think
was
I,
don't
know,
I'll
the
premise
of
span.
Immutability
is
I.
Think
a
a
pro
I
think
there
are
many
cases
under
which
it's
really
useful.
To
have
a
span
in
Flight
be
something
that
can
be
modified
baggage
comes
into
this
there
are.
There
are
a
lot
of
places
where
it's
like
hey
that
thing.
That's
up
there
that
haven't
actually
sent
yet
I
want
to
change
it
and
and
I
think
it
was
I.
B
Don't
know
naive,
maybe
to
assume
that
you
could
just
say
like
here's,
this
pure
pristine
thing
that
we
don't
mess
with
ever
once.
We've
created
it,
and-
and
so
that's
that's
where
this
comes
in-
that
that
I
feel
like
it
was
a
design
decision
made
early
on
in
this
whole
process
that
has
complicated
a
lot
of
relatively
common
use
cases.