►
From YouTube: 2022-12-15 meeting
Description
Instrumentation: Messaging
A
C
B
So
we
have
a
joint
meeting
today.
My
the
messaging
work
group
and
the
sampling
stick.
People
are
here
so
I
put
the
link.
D
For
combined
group
foreign.
D
See
most
of
our
sampling
crew
is
here
in
the
meeting.
Thank
you
all
for
the
schedule
change
and
for
coming.
I,
don't
usually
talk
in
this
meeting
or
I'm.
Actually,
this
is
my
second
time
joining
this
meeting.
So
I
would
like
to
welcome
to
the
sampling
crew
and
then
I
guess
hand
over
to
Johannes,
and
we
can
frame
this
discussion
about
sampling
and
links.
D
A
E
A
B
B
D
Me
yes,
apologize
I
was
waiting
to
hear
something
and
I
was
muted,
both
ways
so
in
words,
I,
have
pitched
an
idea
to
both
of
these
groups
and
there's
also
some
writing
in
the
link
to
the
issue.
That's
on
our
screen,
I
wanted
to
bring
these
groups
together
to
talk
about,
span,
links
and
I.
Think
the
question
really
starts
with
what
what
they
mean
and
I.
Don't
think
we
have
a
very
firm
picture
or
written
down
description
of
that
in
the
hotel
project
and
I'm
going
to
make
some
claims.
D
May
I
present
I'll
share
a
few
slides
just
to
give
a
quick,
quick
yeah.
Here
we
go.
D
Let's
see,
okay,
this
is
gonna,
be
quick
right,
wrong,
wrong
button:
okay,
I!
You
can
see
this
now
one
screen
of
text,
it's
sort
of
a
wall
of
text,
but
I
just
decided
to
write
down
everything.
I
was
thinking
of
in
one
page,
so
I
drew
a
picture
here
of
a
span,
we're
imagining
that
this
is
a
new
span
being
created
and
I
I
in
the
next
slide.
I
have
a
couple
of
screenshots
of
your
document
with
some
of
the
messaging
scenarios.
D
Diagrammed
and
I
want
to
talk
about
it
again,
but
this
is
sort
of
the
abstract
question
we're
talking
about
which
is
I'm
going
to
create
a
new
span.
It
has
links,
I
I
have
a
mental
model,
what
they
mean,
but
I
think
I
haven't
really
written
it
down
to
Hotel.
So
what
I
wanted
to
say
is
the
thing
that's
important
here
is
that
there
are
two
directions
to
a
link:
there's
The
Logical
Direction
and
the
sort
of
physical
Direction,
and
my
claim
is
the
sort
of
made
up
claim.
Is
that
spans
point
to
their
predecessors?
D
D
However,
the
direction
of
causality,
which
is
sort
of
the
the
kind
of
thing
we
use
to
describe
a
trace
moves
in
the
opposite
direction.
When
I
create
a
new
span
linking
to
some
other
spans,
they
caused
me,
they
existed
first
I'm.
A
new
span
which
is
linking
to
those,
therefore
those
other
spans,
caused
me
in
a
classical
tracing
scenario.
Where
your
traces,
a
tree
structure,
you
have
a
parent-child
relationship
where
the
parent
was
a
predecessor.
D
The
child
was
a
follower,
the
child
points
to
the
parent
and
when
we
assemble
that
trace
the
parent,
the
child's
included
in
the
parents,
Trace
I'm,
trying
to
claim
that
when
we
have
a
span
linkage
situation,
a
new
span
is
created
with
n
Span
links
that
that
new
span
logically
belongs
to
every
one
of
those
traces,
and
that
is
what's
not
contained
in
the
data
model
today.
So
if
we
believe
my
claim,
then
I
believe
we
have
a
problem,
Samplers
can't
do
what
I
just
described.
D
If
there
are
n
spans,
I
may
be
a
part
of
n
traces
and
I
might
need
to
export
that
spam
for
any
one
of
those
traces
and
I'm,
calling
that
a
problem
with
the
hotel,
sampler
API,
there's
no
way
for
a
sample
to
sit
today
to
say:
I
want
to
record
and
Export
the
span,
but
I
am
not
sampled,
meaning
my
children
will
not
be
sampled
automatically.
I
just
need
to
record
myself
because
I'm
linking
to
something
that
will
be
incomplete
without
me
and
the
reason
I
call
it.
D
That
incompleteness
is
that
much
like
with
the
sampling
scenario
just
for
parent
and
child
relationships.
We
have
this
problem
where,
if
you
lower
the
sampling
probability
from
your
parent
to
your
child,
you
might
sample
out
your
child
and
never
know
it.
It's
dangerous
to
lower
probability
as
you
as
you
descend
down
a
trace,
because
you
can
lose
information
and
have
incompleteness
without
knowing
it.
The
thing
we
can
that
what's
happening
with
Spam
link
today
is
you
can
lose
information
without
knowing
it.
D
Unless
you
record
the
follower,
you
will
never
know
that
the
proceder
had
an
association
and
we
don't
have
a
way
to
record
the
follower
unless
it's
sampled
today,
that's
the
problem.
I
was
the
reason
why
I
wanted
to
bring
these
groups
together.
I
think
we
can
solve
that
problem
by
by
producing
more
decisions
the
sample
could
make
and
that
that
the
more
decisions
are
I
want
to
export
this,
but
not
be
sampled,
and
that
leads
us
to
some
invariants.
What
do
you
get
at
the
end
of
the
day?
D
At
the
end
of
the
day,
these
these
linkages
are
asymmetrical.
The
things
that
preceded
me
have
already
made
their
sampling
decision
and
there's
nothing.
I
can
do
to
change
their
sampling
decision.
So,
if
I'm
linking
to
n
spans
those
ends
spans
are
probably
sampled
according
to
the
probabilities
that
they
used
themselves,
but
I
can
always
make
sure
that
I
am
exported
to
complete
another
trace.
The
follower
can
always
export
itself.
Therefore,
I
can
always
complete
the
things
I'm
linking
to
if
we
could
complete
the
sampling
logic
to.
D
Let
me
do
that,
of
course,
if
you're
a
vendor
who's,
counting
spans,
which
many
of
us
do
it
becomes
more
complicated.
However,
that
is
a
sampling,
technical
question
and
we've
solved
it.
So
I
don't
want
to
talk
about
it
now,
but
if
we
have
a
way
to
export
things
that
are
unsampled,
we
know
how
to
count
them
as
well,
but
that's
just
sort
of
a
corner
case.
D
I
have
now
made
my
entire
pitch.
I
want
to
just
sort
of
walk
through
a
little
bit
of
an
example
here.
So
I
took
a
couple
of
images
out
of
the
messaging
document:
220
Otep,
220
and
I,
just
scribbled
on
top
of
them
I'm.
Looking
at
this
particular
case
that
has
essentially
five
spans
and
I'm,
seeing
at
least
three
traces
that
are
possible
here.
There's
the
Trace
that
starts
the
producer,
create
its
child
is
a
publish,
and
then
it
calls
into
an
intermediary.
D
The
intermediary
May
create
a
trace
that
may
create
its
own
trace
and
link
to
the
producer.
Then
that
means
that
the
producer
Trace
will
include
one
span
from
another
another
Trace
ID,
which
will
help
complete
its
own
picture.
In
other
words,
the
orange
Trace
here
includes
three
spans:
two
are
in
a
trace
and
one
is
a
link
follower.
D
The
green
Trace,
similarly
I'm,
not
sure
where
the
root
of
this
Trace
would
be,
but
I
I
think
there's
probably
two
traces
here.
The
intermediary
might
have
a
batch
of
data,
that's
going
to
be
received,
and
that
could
be
one
trace.
The
receiver
could
link
to
the
to
the
intermediary,
in
which
case
so
one.
D
I
realize
that
there
are
many
different
variations
here.
I
was
trying
to
show
what's
possible,
so
so
I
think
you're,
right,
I
think
the
the
the
the
so
let's
ignore
tracing
the
intermediary
and
focus
on
this.
The
screen
span,
maybe
I
start
my
Trace
at
the
receive
operation.
I
call
the
process,
operation,
I,
call
the
settle
operation
and
now
I'm
linking
to
my
intermediary,
which,
if
it
did
Trace
itself,
was
already
already
started.
D
It
could
be
my
own
parent
or
or
it
could
not
be,
but
the,
but
ultimately
what
I'm
trying
to
say
is
in
order
to
complete
this
Trace,
which
includes
process,
settle
and
receive
I
might
want
to
include
a
span
from
the
intermediary.
It's
just
not
part
of
the
same
Trace,
and
so,
if
we're
sampling,
we
can
get
any
one
of
these
traces
to
be
complete.
We
just
probably
won't
get
them
all
complete
at
the
same
time
and
then
it's
a
numbers
game.
D
So
that's
all
I
wanted
to
say
is
I
think
that
we
could
extend
the
sampler
logic
to
support
exporting
these
spans
so
that
they
complete
traces
that
they're
linking
to-
and
that's
it
same
idea
in
the
next
diagram-
is
that
there
are
two.
There
are
two
traces
here.
One
covers
the
producer
side.
One
covers
the
consumer
side.
The
consumer
side
ought
to
get
a
copy
of
the
thing.
It's
linking
to
sorry.
The
producer
side
ought
to
get
a
copy
of
the
one
span
from
the
from
the
cons.
D
The
producer
Trace
gets
one
span
from
the
consumer
because
the
consumer
links
to
the
producer,
and
that
means
that
this
blue
Trace
will
include
anything
Upstream
plus
the
producer
span
and
one
span
from
the
consumer
and
then
there's
a
consumer
Trace
that
will
or
will
not
include
a
producer
whether
it
was
sampled
or
not,
because
it
was
decided
first
and
then
it
includes
its
own
team
spans,
and
that
is
my
pitch.
I
am
completely
done
and
I'm
going
to
unshare
and
let
us
discuss.
C
I
have
a
couple
of
questions
to
the
foundation.
Thank
you
for
the
pictures
it
was
great.
I
have
I
know
that
we
cannot
guarantee
that
with
no
links
before
spend
starts.
This
is
the
first
one
and
the
second
one
I
I'm,
not
sure.
If
links
impose
causality
in
any
way-
or
at
least
they
don't
they
don't
say
anything
about
the
direction
or
what
are
their
relationships?
C
Maybe
it's
I,
don't
know
certain
grandfather
from
I,
don't
know
who,
like
you
know
so
I
would
like
us
to
see
if
we
need
to
solve
this
problem
at
all
with
sampling.
C
So
there's
been
messaging
scenarios,
it's
common
that
some
messaging
systems
operate
with
pulling
so
they
go
and
they
ask
for
messages
right.
So
when
the
span
starts
right,
you
don't
know
what
you're
going
to
receive
and
if
you're
going
to
receive
anything,
so
you
know
only
that
this
by
the
end
of
your
operation,
you
can
fake
the
span,
but
you
will
lose
ambient
context.
You
will
lose
low
correlation.
You
will
also
by
shifting
exemplars
all
right.
C
C
You
cannot
change
links
attributes
at
this
moment
right
because
links
are
upgraded
and
they
probably
are
immutable,
and
then
you
will
need
some
to
find
some
other
way
to
attach
this
data
too.
But
it's
it's
I
think
it's.
The
second
point
is
a
separate.
The
first
one,
though,
is
important.
You
don't
know
what
you
will
receive
until
you
receive
it.
D
Thank
you
and
I
I
recognize
a
connection
with
this
other
topic,
which
I
have
absolutely
wrapped
up
in
some
of
the
issues
that
I've
that
we've
discussed
this
in
it's
the
question
of
whether
spam
links
can
be
created
after
spans
start
and
my
understanding
is
that
not
many
people
understand
why
we
don't
allow
that.
But
if,
if
you
end
up
discussing
why
we
why
we
don't
allow
spam
links?
After
start,
it
comes
back
to
a
sampling
question
again
and
I.
D
Don't
think
so,
and-
and
my
analysis
of
this
situation
says
that
it
is
unnecessary
to
limit
spam
link
creation
to
spanstart
and
I
I
was
trying
to
narrow
the
focus
of
my
little
pitch
but
the,
but
if
I
and
perhaps
I
could
well,
perhaps
I
should
represent
it.
D
But
but
let
me
let
me
describe
that
I
think
the
that
the
approach
to
sampling
that
I
have
described
is
compatible
with
a
future
version
of
otel,
where
we
do
allow
spam
links
after
start,
and
the
reason
is
that
the
property
that
we're
still
able
to
exploit
is
that
the
span,
which
is
the
follower
came
later
and
it
is
not
exported
yet.
But
the
moment
you
are
making
this
a
new
link
supposedly
had
a
set
link
or
add
link
operation.
That
span
is
still
alive.
D
You
may
have
already
made
that
spam
sampling
decision,
but
that
span
is
still
alive
and
you've
and
because
of
a
change
in
sampling
that
I
just
sort
of
sketched
around
right
now
the
span
is
recorded,
but
unsampled
now
I
can
decide
to
export
it
because
the
span
was
still
alive.
C
Yeah
makes
sense,
thank
you
for
the
clarification.
The
other
part
is
still
on.
So
imagine.
I
am
in
a
very
high
scale
scenario:
I
receive
hundreds
messages
per
each
batch
right
and
the
probability
that
any
of
them
are
sampled
in
is
maybe
one
so
do
we
understand
if
the
sampling
like
do
we
say:
okay,
people,
you
decide
which
sampler
to
use
and
therefore
how
can
we
allow
people
to
limit
their
rate
or
are
we
saying
go
ahead
and
Implement
your
own
thing.
D
I
I
get
I,
think
I
get
the
question.
Let
me
see
if
I
can
wave
my
hands
around
and
answer
so
we
are
now
talking
about
I,
guess
a
fan
in
scenario
where
I've
got
a
bunch
of
work
coming
into
my
span
and
I
cannot
and
and
I,
because
we're
sampling.
Let's
say
that
every
one
of
my
incoming
span
link
context
is
sampled
at
one
and
a
hundred
and
I've
got
a
thousand
links.
So
I
expect
10
out
of
my
thousand
to
be
present
on
average.
D
Now
the
decision
to
export
my
span,
the
one
span
that
will
complete
those
other
thousand
traces
is
going
to
add
one
more
span
to
the
output.
I've
already
got
10
spans
that
I'm
writing
I'm
going
to
add
one
span,
I'm
not
going
to
add
all
the
things
Downstream
from
me
because
remember
this
span
is
untraced.
All
of
its
descendants
are
not
traced.
D
So
it's
really
just
one
span
which
makes
me
think
of
it
as
an
O
of
one
cost,
like
I've
already
like
for
all
the
traces
that
I'm
sampling
I'm,
adding
one
spam
so
that
I
think
our
rate
limiting
still
works.
D
Thing
that's
hardest
about
this,
which
I
recognize
is
that
is
that
vendors
and
and
products,
and
that
use
Trace
data
now
would
have
to
begin
to
understand
that
when
there
are
links
you
may
receive
a
span
that
was
part
of
another
trace
and
just
because
it's
links
it's
being
recorded,
we
have
to
explain
that
to
the
vendors
they
have
to
tell
them
that
when
they're
building
their
traces
like
the
point
of
this
link
is
that
it
needs
to
be
assembled
into
another
Trace.
D
That's
what
it
means
and
I
think
that
right
now
most
a
lot
of
the
vendors
have
just
sort
of
shook
their
heads
at
scan,
links
and
said:
I,
don't
know
what
to
do
with
those
and
I
think
much
of
it
has
to
do
with
this
incompleteness,
which
is
when
you
don't
include
that
span.
You've
completely
lost
it
and
then
it
you
know
you
get
this
broken
user
experience,
so
we
would
have
to
make
sure
the
vendors
are
on
board
with,
like
here's,
some
spans
you,
you
can't
count
them
probabilistically
speaking.
D
They
were
not
sampled,
but
they
are
part
of
other
traces,
and
that
is
a
tricky
change
to
make.
B
I
have
a
question
to
trust.
When
you
told
me
to
talk
about
okay,
this
part
is
displaying
distant
part
of
the
traces
it
still
is.
Gonna
get
the
new
Trace,
ID
I
assume
so
in
this.
Basically,
it
is
suspended,
then,
has
the
links
to
the
other
spend
to
the
other
spans
from
different
phases.
So
this
band
then
will
basically
get
a
new
Trace.
Id
will
be
linked
to
the
other
spans,
but
will
not
basically
then
propagate
this
sample
through
decision.
Do
it's
like
children
or
descendants?
D
That's
the
idea
yeah
is
that
it's
it's
going
to
record
a
Spam
without
tracing
its
whole
Trace
and
and
then
as
you're
assembly,
as
you
receive
spans
at
the
vendor
side,
you're
going
to
say,
oh
I
got
Stan,
it
has
it,
and-
and
currently
we
do
not
have
a
way
to
say
this
span
was
not
sampled
like
we
just
don't
have
that
way
to
to
say
that
we're
not
including
the
transparent
in
the
in
the
span
data
structure,
which
is
like
a
tiny
Omission,
but
we
don't
have
that
information.
D
So
there's
a
couple
changes.
We
need
to
get
there,
but
so
then
the
idea
is
that
you
don't
like
change
the
trace
idea
of
the
span
you
just
like
it
has
links
which
lists
other
Trace
IDs
there.
They
belong
to
all
those
other
Trace
IDs
and
some
of
them
might
be
sampled.
E
So
one
one
issue
that
I
have
with
the
scenario
goes
back
to
one
of
the
examples
you
had
previously,
where
you
are
sampling
at
10
so
and
you've
got
three
traces,
so
you've
got
a
very
small
percentage
chance
of
having
all
three
of
the
traces
sampled
at
the
same
time.
E
So
what
that
means
is
that,
if
I
as
a
user,
want
to
find
one
of
those
situations,
it
becomes
very
difficult
to
actually,
you
know
see
a
actual
manifestation
of
that
and
I
I
just
think
that
results
in
a
fairly
poor
user
experience.
D
D
D
But
I
want
to
point
out
how
much
better
that
scenario
is
than
the
other
scenario,
which
is
I'm
a
producer.
I'm
I'm
writing
data
into
a
messaging
system
that
is
linking
back
to
me.
In
other
words,
that
messaging
system
is
creating
a
new
span
that
links
to
me,
but
that
messaging
system
span
was
not
traced.
So
and
under
this
scenario,
we
get
absolutely
no
information.
The.
E
Train
so
I
agree
with
you
there,
the
that's
all
the
worst
scenario.
I
prefer
actually
is
to
to
switch
it
and
have
the
the
linkage
associated
with
the
the
less
strongly
correlated
part
of
the
information
so
like,
for
example,
in
in
the
the
receive
scenario
that
the
Mia
Ludmila
mentioned.
Where
you've
got
to
receive,
and
you
don't
know
whether
you're
you're
getting
so
you
start
a
trace
right
and
then
later
on.
You
get
some
messages
at
that
point.
E
I
feel
like
the
link
should
go
to
the
the
receive
span
and
have
the
parent
be
the
the
remote
span.
So
so
it's
kind
of
switching
the
behavior
a
little
bit
such
that
the
default
cases
to
have
a
more
broad
perspective
of
what
the
the
trace
is
and
so
then
like
by
default.
E
The
sampling
gives
you
more
information
and
the
the
link
goes
to
the
things
that
kind
of
bring
the
the
units
together,
such
that,
oh
okay,
so
I'm
linking
to
the
the
span
that
represents
the
batch
instead
of
the
the
traces
the
batch
and
the
the
link
represents.
The
producer
does
that
make.
D
Sense,
I
think
it
does
I
I'm
absolutely
sort
of
trying
to
take
the
shortest
path
that
I
could
see,
which
is
to
say
that
right
now
we
have
a
way
to
encode
a
span
and
the
span
may
contain
a
link.
So
we
have
this
directed
pointer
and
we
can
only
record
it
from
one
side.
I
think
what
you're
saying
is
that
it
would
be
nice
if
we
could
record
it
from
either
side.
D
The
the
complication,
of
course,
is
that
the
thing
that
you're
referring
to
might
have
finished
already
and
not
be
present
and
I
I
can't
imagine
I
don't
want
to,
but
I
can
imagine
like
a
new
data
type,
which
is
like
fan,
link
record,
meaning
I'm
I'm,
recording
a
part
of
another
spam
out
of
band
because,
like
that
span's
long
gone
or
something
like
that
or
I.
E
No
so
like
the
the
producer
creates
its
own
span
and
reports
it,
but
then
the
consumer,
in
order
to
continue
that
Trace
would
say
that
that
producer
was
the
parent
and
follow
the
parents,
sampling
decision.
G
You
can't
you're,
basically
saying
even
long-lived
processes,
the
traces
effectively
open
and
still
an
active
Trace
for
hours
days,
and
and
that
you
don't
know
everything
until
you
know
it
all
right.
G
E
D
G
From
an
ingestion
point
of
view,
the
service
provider,
could
you
know
when
it
receives
these?
These
specialized
span
links?
Could
then
you
know,
look
up
the
existence
of
those
traces
and
annotate
the
data
it
receives.
To
note
you
know
you
could
actually
have
an
attribute.
That
says
here
is
the
one
of
those
special
span
links
that
actually
has
all
of
the
sources
that
it
that
are.
It
refers
down.
That's
right,
but
that's
an
ingestion
side
problem.
D
F
Oh,
but
what
you
suggested
will
work
only
if
each
span
contains
exactly
one
link
right,
because
then
it
can
just
reverse
the
roles
of
the
parent
and
the
link,
multiple
links.
Then
it's
not
possible
to
do
it.
D
I
think
you're
talking
about
Tyler's
suggestion
right
now,
yeah.
E
With
my
suggestion,
yes,
I'm
I'm
talking
about
like
a
fairly
specific
scenario
where
you
know
you've
got
one
remote
parent
context
and
a
an
ambient
context
right
now.
The
the
spec
generally
prefers
the
ambient
context
and
links
to
the
remote
I'm,
suggesting
that
in
certain
in
in
many
circumstances,
I
think
reversing
those
two
would
be
preferable
in
creating
a
more
cohesive,
Trace.
C
Yeah
I'd
like
to
entertain
the
idea
that
just
you
mentioned
the
three
span
links
as
a
separate
concept.
Imagine
I
received
this
thousand
messages
when
I
record
this
step,
do
I
record
a
thousand
links
okay
or
do
I
record
each
individual
link
that
has
that
was
sampled.
In
do
I
record
all
the
attributes
and
events
on
this
receive
span.
C
Would
it
make
sense
for
us
to
say:
okay,
spam
link
is
just
an
event,
and
this
is
the
idea
I
entertained
for
other
reasons
and
I,
and
maybe
we
should
just
say
Okay
this
event
as
a
parent
which
is
a
link
and
then
it
follows
the
parent
sampling
decision,
the
Remote
Link
sampling
decision,
and
then
we
record
this
event
and
then
we
know
everything
we
need
to.
D
A
D
Just
like
events
that
that
is
definitely
one
that's
occurred
to
me
and
as
someone
who
used
to
work
on
Modern
libraries
like
if
I
wanna,
if
I'm
like
in
the
middle
of
my
new
text
code
and
I,
see
somebody
else,
has
a
lock
I'm
going
to
record
a
span
event
saying
I
bumped
into
another
context
right
now
and
I
think
it
should
imply
a
link
like
like
me,
logging
another
spam
context
in
a
span.
That's
a
link
right
there.
D
D
Going
back
to
this
sort
of
hypothetical
about
suppose
we
add
a
way
to
add
spam
links
after
span
starts
I,
don't
think
it
would
break
anything
that
we've
that
wasn't
already
broken
and
I
think
it
has
the
potential
to
improve
some
of
the
debates
that
you've
had
in
this
group,
because
it
seems
like
there
is
a
desire
to
add
spam
links
after
start
and
I,
don't
see
how
it
breaks
sampling
in
ways
that
wasn't
already
broken
in
ways
that
we
can't
fix.
D
In
other
words,
is
what
I'm
trying
to
say,
yeah
and.
D
Tyler's
looking
for
a
simple
Trace
in
the
simple
case
and
that-
and
that
is
I-
think
I
I
heard
you
debating
that
last
week,
like
when
I
just
want
like
I,
have
one
parent
I
have
one
link
can
I
just
make
it
a
child,
a
link
right
then
that
makes
sense
like
wanting
the
simple
case
to
be
simple
but
I,
guess
I.
D
My
impression
from
the
outside
before
I
started
coming
to
this
meeting
was
that
a
lot
of
the
trouble
in
the
messaging
group
for
semantic
conventions
is
that
they're
batching,
often
in
both
directions
coming
in
and
out
of
this
server,
and
so
it
it's
never
obvious
how
to
trace
it.
D
I
think
and
I
think
that,
combined
with
the
fact
that
traditionally
spam
links
were
not
a
part
of
like
all
the
systems,
Like
Jaeger
and
Google,
staffer
and
stuff
so
like
we
are
all
kind
of
like
making
it
up
as
we
go
and
spam
link
don't
seem
to
work
so
we're
not
doing
them,
and
then
families
don't
seem
to
work,
so
we
can't
sample
with
them,
and
so
now
we
can't
even
add
fan
lights
after
spam
starts
like
it's.
Just
sort
of
like
an
accident
I
think.
C
D
D
This
idea,
which
I've
maybe
tossed
around
once
or
twice
before,
that
you
can
record
a
new
type
of
data
to
say,
here's,
a
link
that
happens
after
the
fact
and
you
wouldn't
even
need
to
like,
have
either
span
and
you
could
use
that
to
as
another
solution,
essentially
to
the
claim
I
the
claim
that
there's
a
problem,
which
is
that
you
have
no
way
to
record
the
follower
of
a
Spam
link
so
either
it's
by
recording
the
span
when
it
was
unsampled
or
it's
by
recording
a
new
piece
of
data
say
there
was
a
span
linkage
to
something
sampled
that
wasn't
sampled
itself.
C
C
C
You
get
the
the
cell
with
that
that
we
are
going
to
link
to
right
the
Upstream
thing
and
you
get
the
link
on
the
downstream.
But
let's
record
that,
okay,
there
is
some
small
set
I
think
there
is
some
sampling
changes,
but
not
in
the
sampling
decision.
So
basically
you
need
to
record
an
event
that
belongs
to
a
Spam
that
happened
elsewhere
to
the
remote
span.
G
Ahead,
I'm,
sorry,
I,
I'm,
I'm,
not
understanding
something
here,
I
think
what
you're
saying
is
you
have
an
event
that
exists
independent
of
either
span
that
has
essentially
references
to
both
of
the
other
spans
and
that
event
exists.
And
then
you
said
it
belongs
to
the
span
and
that's
the
part
I'm,
not
understanding.
Is
it
part
of
the?
Are
you
suggesting
it
as
part
of
the
trace
or
how
is
what
is
the
belonging
part
here.
C
Yeah,
so
we
now
have
span
events
right
and
they
are
essentially
will
be.
They
will
be
Logs
with
semantics
right,
so
you
have
two
different
signals.
When
you
create
a
span
event,
you
actually
create
an
independent
block
record
right.
C
I
think
this
is
the
the
future
of
the
logs
versus
events,
and
then
imagine
you
create
this
log
record,
which
is
a
child
of
it
points
to
its
parent
right,
but
remotely
it
happens
on
the
different
machine
right
and
then
it
says:
okay,
I
am
a
child
of
that
thing,
but
I'm
also
linked
to
that
other
thing
and
I'm
going
to
describe
the
properties
of
my
linkage.
C
The
attributes
right.
Does
it
help.
G
Yes,
thank
you,
I
think
my
it's.
My
lack
of
understanding
of
of
the
child
relationship
within
logs
I,
don't
know
how
that
works.
So.
C
B
Yes,
just
the
edge,
not
the
vertex
and
that
edge
basically,
then
B
can
exist
even
with
like
maybe
both
or
one
of
the
spans
are
not
centered
in,
but
you
still
don't
have.
Basically,
this
Edge
like
in
the
as
a
as
a
separate
event
or
log
record
I,
see
yeah.
C
D
Hearing
us
discuss
the
idea
of
these
leakage,
events
being
independent,
which
I
think
is
even
goes
even
further
than
the
claim
I
made
at
the
beginning
and
I.
Don't
disagree.
I
think
I
was
relying
on
this
nice
property.
We
have
that
we're
going
to
record
one
item
and
it
can
include
a
link
in
that
points.
Both
directions,
like
the
links,
are
bi-directional.
Basically
you're
going
to
record
it
somewhere.
D
I
had
been
trying
to
say
roughly
what
you
just
said
can
be
achieved
by
allowing
us
to
record
spans
that
weren't
sampled,
because
that
encodes
a
link
that
might
be
useful
for
something
that
was
sampled
so
as
for
whether
it's
a
unexported
unsample
or
an
exported
unsampled
span
or
a
completely
new
type
of
data
I,
don't
think
it
really
changes
the
the
data
representation,
like
the
the
graph
that
we're
drawing
as
long
as
all
the
data
arrives
at
the
back
end,
you
get
the
same
picture.
I
think.
C
It
doesn't
yes,
I
agree,
it
does
not
change
the
graph,
but
it
changes
it
answers
a
lot
of
it.
It
makes
it
more
flexible.
It
allows
to
answer
more
questions
like
do
you
record
links
that
are
not
sampled
in
what
information
you
put
on
the
spam
that
is
recorded
by
these
children,
it's
are
not
recorded,
so
I
I
think
it's
more
extendable
in
the
future.
It
can
allow
us
to
do
more
with
with
your
reporting,
probably
less
data
than
before
in
in
a
simpler
format.
F
D
The
the
logic
that
I
outlined
in
one
of
the
issues
was
a
little
bit
more
sophisticated,
something
like
and
I
and
I
have
pitched
this
for
a
world
where
we
can
add
spam
links
after
the
fact.
So
the
the
idea
was
to
include
one
or
two
new
sampling
decisions.
D
The
first
one
would
be
deferred,
which
is
to
say,
I,
am
not
sampled
I
have
to
make
that
call
up
front
but
deferred
in
a
sense
that
I
might
or
might
not
export
myself,
and
that
means
you
you're
that
we
have
this
notion
today
of
a
recording
span
which
is
not
exported.
It's
just
in
memory
for
like
inspection
or
Z
pages
or
whatever
we
have
so.
A
new
sampling
decision
could
be
I
want
to
defer
and
maybe
export,
and
that's
like
saying,
I'm
waiting
to
find
out
if
I'm
interested.
D
One
of
the
reasons
I
might
become
interesting
is
that
a
Spam
link
is
created
after
the
fact
that
links
to
something
sampled.
So
then,
you
could
imagine
transitioning
from
that
deferred
decision
state
to
the
I
am
now
deciding
that
I'm
going
to
write
myself
out,
because
I
contain
some
information,
interesting
information
which
will
include
the
span
and
all
the
stuff
that
was
in
the
span.
D
F
D
We
well,
if,
if
nothing
links
to
it,
so
you
create
a
span
with
no
links
to
start
with
and
no
new
links.
Well,
at
the
end
of
that,
Savannah
must
say:
I,
don't
I,
don't
want
to
record
this
or
export
this,
but
you
could
create
a
span
in
that
to
first
date
and
then
create
a
link
to
another
sampled
span,
another
sampled
Trace,
which
would
tell
the
sampler
oh
this
span
has
a
link.
D
That's
part
of
another
thing,
I'm
going
to
record
it
or
export
it,
so
that
deferred
decision
would
be
because
I
found
a
link
change
to
the
actual
export.
D
No,
no
I'm,
actually
thinking
about
in
the
SDK
itself,
like
you
have
a
live
span.
It's
deferred
decision
making
until
the
span
adds
the
span
end
time
like
that's
when
I
want
to
make
the
decision
so,
but
the
only
reasons
that
I
propose
in
my
thought,
experiment
was
because
you
came
in
with
links
which
case
you
can
make
the
decision
right
away
or
you're
going
to
allow
spin
links
after
start,
which
case
you're,
going
to
defer
that
decision
foreign.
D
F
This
suggestion
is
only
for
links
that
join
are
still
dispense,
started
right.
D
The
reason
to
use
deferred
yeah
like,
but
but
if,
if
links,
can't
be
added
after
span
starts,
then
I
can
make
every
decision
up
front,
which
is
to
say
that
I'm
going
to
make
my
own
independent
sampling
decision,
whether
I
have
parent
or
not,
and
then,
if
any
span
links
were
sampled
I
will,
if
I
decide
not
to
sample
in
any
spam
links,
were
sampled.
I
will
now
record
myself
on
export
myself,
unsampled.
F
D
And
that
gets
back
to
the
the
diagrams
I
have,
which
is
to
say
that
you
can
get
a
complete
trace
on
one
side
or
on
the
other
side,
but
the
chances
of
getting
both
of
those
at
the
same
time
are
probability.
The
probability
multiplied
yeah
and
but
but
but
but
in
order
to
complete
the
trace
on
essentially
on
the
left
side.
I
need
to
get
that
one
stand
from
the
the
external
Trace
to
complete
it.
Otherwise,
I
have
no
information
about
what
that
Trace
was
doing.
G
F
D
D
B
Yeah
I
mean
from
from
my
point
of
view
here
when
I
basically
say:
okay,
I'm
the
customer
and
from
that
solution
a
charge
that
you
propose
Dependable,
that's
what
benefit
they
get
I
would
say:
I
don't
get
complete
traces,
that's
just
like
the
same
as
before.
Basically,
I
I
will
not
get
the
complete
path
of
the
message.
I
will
also
not
be
able
like
on
the
backhand
side
like
to
count
those
those
bands
at
least
and
kind
of
have
them
in
some
kind
of
your
extrapolation
included.
B
There
will
not
get
that
benefit,
but
what
benefit
I
would
get
is
basically
that
I
can
correlate
several
like
producers
together
like
if
there's
ever,
producers
and
all
the
messages
end
up
in
are
received
in
one
single
batch
I
can
make
this
connection
and
see
that
and
basically
what
I
can
also
get
and
I
think
that
is
interesting
for
messages.
I
can
see
this
one
Edge,
basically
from
the
from
the
producer
to
the
consumer,
because
I
have
this
one,
then
consumer
span
and
I
see:
okay,
for
example.
B
This
message
was
consumed
by
I,
don't
know
10
different
consumers.
I
will
see
that,
because
I
will
see
all
the
10
consumer
spans
and
I
can,
for
example,
also
make.
Then
I
can
see
the
latency
between
like
the
producer
and
the
consumer
like
in
those
single
ads,
but
the
interest
then
trust
the
other
Trace
will
be
kind
of
broken
off
when
they
will
not
see
much
more
from
that
point
on
and
I
think
that
is
a
that
is
definitely
I.
B
Think
an
improvement
for
messaging
scenarios,
because
just
seeing
okay,
for
which
kind
of
consumers
are
related
to
this
producer,
and
what
is
the
latency
I
think
that
is
one
thing
that
the
that,
from
the
messaging
point
of
view,
definitely
is
interesting.
B
How
how
well
that
fits
in
the
overall
like
open,
Telemetry
picture
I
would
need
to
think
more,
but
it's
I
think
it's
definitely
from
my
point
of
view
and
interesting
solution
and
then
other
like
smart
question.
I
had
about
like
this
default
sampling
and
I
wondered
that,
if
I
deferred
and
the
same
thing
for
the
span
on
the
on
the
receiving
side,
basically
what
that
they
need
is
the
idea
to
do
this
for
the
complete
trace
for
every
Span.
D
Yeah,
that's
kind
of
a
consequence
of
this
is
that
I
would
have
to
configure
a
heavier
weight.
Sampler,
that's
going
to
defer
some
of
the
time
and
I
would
and
I
if
I'm
never
going
to
defer,
that's
just
cost
costing
me
something
or
if
I'm
never
going
to
export
unsampled,
and
that
and
that's
that
would
mean
like
right
now.
D
Today
we
have
a
handful
of
Base
built-in
Samplers
that
are
described,
and
we
would
be
adding
to
that
and
saying
that
now
you
have
a
a
sampler
mode
which
says
you
know,
follow
all
the
existing
logic
plus
export.
My
spam
anytime
I
link
to
something
sampled,
which
would
be
another
option,
as
you
say,.
E
Does
the
span
link
include
whether
or
not
that
parent,
the
the
span
that
it's
linking
to
was
sampled?
Does
the
sampling
decision
get
included
in
that
link.
D
That
is
a
defect
that
I
mentioned
and
I
think
I
need
to
think.
We
need
to
think
about
that
like
right
now.
The
span
does
not
record
record
the
trace
parent
header,
which
includes
the
sampling
Flags.
They
include
Trace
State,
but
not,
and
we
include
Trace
ID
and
span
ID.
There's
one
field:
that's
not
included
there,
which
is
the
the
rest
of
the
trace
parent,
which
is
any
flags
that
were
set,
which
includes
the
sampled
flag
right
now
so
and
there's
there's
obvious
solution
to
that.
But
you're
right,
we
don't.
D
A
D
See
I'm
trying
to
think
about
in
the
actual
SDK
whether
you
get
it
like
you've
got
to
spam
context,
object,
I'm,
thinking
about
the
go,
SDK
I
believe
you
can
extract
the
sample
flag
from
the
context
object
it's
just
that
we
don't
record
it
on
The
Wire.
F
D
Yeah,
that's
that
well,
I
guess
invisibility
is
for
the
trace,
is
what
we're
trying
to
say
and,
and
my
and
I
think
when
you
begin
sampling,
the
the
that's
just
a
natural
consequence,
I
think
so
yeah.
The
the
fundamental
claim
here
is
that
most
people
can
do
their
work
with
a
with
a
with
a
trace
that
just
covers
part
of
the
system
and-
and
hopefully
that's
right,
but
I
can't
I
can't
promise.
F
Yeah
I
think
like
a
few
users
that
I
talk
to
they
imagine
something
else
they
imagine
like.
If
the
producer
is
not
sampled,
then
the
consumer
is
not
sampled
as
well,
and
if
the
producer
is
sampled,
then
the
consumer
is
simple,
so
you
always
have
the
full
picture.
So
if
you
sample
the
consumer-
and
you
send
this
data,
but
you
can't
correlate
to
the
producer,
then
it
gives
you
a
less
value
and
you
pay
for
something
that
doesn't
give
you
a
lot
of
value.
So
I.
B
D
This
I
see-
and
this
is
maybe
now
brings
us
back
to
some
technical
sampling
questions
and
I-
think
we
could
we're
about
to
run
out
of
time.
So
we
can
leave
some
open
questions
here.
D
The
what
you
just
described
is
what,
in
my
terminology,
would
be
a
non-probability
sampler,
so
you
could
say
I'm
going
to
sample
this
Trace.
If
any
of
my
spam
links
have
were
sampled
and
and
we
we
don't
have
that
sampler
today,
we
could
get.
We
could
write
that
sampler
today,
but
well,
my
vendor.
We
have
this
notion
of
a
spandometrics
pipeline
anytime,
we
see
a
span
like
and
we
have
probability
sampling.
We
can
count
it
and
draw
graphs
and
stuff,
and
it
makes
that
hard
for
us.
D
Yeah
sorry,
I
I
lost
my
transplant.
There
I.
B
Mean
I
can
add
something
here.
It
also
makes
it
hard
or
almost
impossible
to
control
the
same
thing
volume
that
you
get
it
because
when
you
have
like
a
span-
and
you
have
like
thousand
links
on
the
span
and
the
if
you
have
a
certain
proper
sampling
probability
for
each
of
them,
the
probability
that
one
of
them
is
same,
but
it's
very
high.
So
this
back
will
probably
end
up
having
sampled
all
of
the
time
and
then
also
it's
descendants,
and
that
will
basically
that
will
blow
your
sampling
volume
on
table
samples.
D
Sorry
that
now
I
remember
what
I
was
trying
to
say
and
in
the
sampling
seg.
We
have
discussed
this
question
a
little
bit.
It's
a
little
bit
academic
I
think,
but
but
there
might
be
an
answer
here.
So
the
idea
that
we've
started
with
is
consistent
sampling.
Is
this
idea
that
every
Trace
has
some
sort
of
like
random
truth
that
we
can
use
to
like
rank
traces
so
that,
when
we're
doing
rate
limiting
we
choose
the
same
traces
across
the
system?
Right?
D
The
question
and
I
think
that
everyone
has
a
kind
of
like
a
layperson's
notion
of
probability.
Sampling,
maybe
is
that
like
I've
got
10
links
coming
in
and
like
I
can
look
at
their
sampling,
probabilities
and
now
I
can
say:
okay
I'm
going
to
sample
myself,
and
this
was
my
probability,
but
there's
there's
not
really
a
clear
way
to
say
that
and
it
makes
it
hard
to
count
the
spam
that
you
are
deciding
to
sample.
But
there
is
a
question
here
and
I'm.
D
Looking
at
now,
my
sampling
experts
and
it's
something
like-
can
we
come
up
with
a
consistent
sampler
that
uses
the
links
and
is
more
likely
to
sample
the
more
links,
but
I
just
don't
know
how
to
set
the
probability,
which
is
what
I
want
to
do
and
I'm
not
sure
I
have
an
answer.
I
was
we
talked
about
this
a
week
or
two
two
times
ago,
something
like
if
I
always
had
exactly
the
same
number
of
parents.
D
B
D
I
I
think
there's
something
here:
I,
don't
think.
We've
really
discussed,
like
the
sampling
team,
can
take
care
of
the
the
logical
questions
and
the
and
the
sampling
proposal.
I
can
outline
that
and
we
know
how
to
count
these
things.
It's
just
a
little
tricky.
What
I
think
we
can't
do
the
sampling
people
here
is
like
is
like
actually
say
yes,
what
we,
what
we're
talking
about
here
would
work,
and
it's
there's
a
lot
of
compromises
in
this
there's
like
a
lot
of
different
ways
to
set
up
tracing
now
because
of
links
and
these.
D
If
these
are
reasonable
options
that
some
customers
and
some
vendors
want,
then
we
can.
Yes,
we
can
write
up
the
issue.
I
think
we
can
both
call
for
this
way
to
export
unsampled
spans.
We
could
also
talk
about
with
Miller's
proposal,
which
is
like
just
like
log
stuff
and
that's
another
way
to
make
an
easy
to
count,
I
think
and
then
we
can
also
make
us
I.
D
B
Yes,
because
I
mean
from
the
messaging
point
of
views,
the
roadmap,
especially
I,
think
the
sampling
decision
is
like
that
is
a
that
is
like
a
North
star
more
like
far
roadmap
point
for
us
to
have
like
a
good
solution
for
that,
but
allowing
links
after
span
creation,
that's
a
kind
of
that's
a
bit
of
a
blocker
for
us
on
the
short
term,
so
I
think,
at
least
in
that
regard.
B
It's
kind
of
good
for
us
to
hear
that,
from
your
point
of
view,
like
from
same
thing,
point
of
view,
there's
not
a
strong
blocker
there,
so
it
will
kind
of
then
probably
try
to
push
for
adding
links
after
span
creation
again
and
or
for
the
for
the
overall
sampling
sampling
questions
in
messaging,
yeah
I
think
we
will
have
need
future
discussions
to
kind
of,
but
but
I
think
that
was
great.
It
kind
of
helped
us
to
find
some
common
ground
and
understand
each
other
better
and
maybe
on
the
basis.
D
That
sounds
great
I.
Think
I'll
join
this
every
other
meeting
when
I'm,
not
in
the
sampling
Sig,
and
we
can
talk
about
this
continue
talking
about
this
I
I
you
can
see.
Carlos
is
here,
he's
going
to
help
us
prioritize
from
White
steps
perspective,
and
thank
you
very
much.
B
Awesome
so
thanks
Josh,
thanks
to
all
the
thanking
people
who
try
and
there's
a
thanks
to
the
messaging
people
and
I
thought
Charles,
she
put
the
link
to
his
deck
in
the
in
our
agenda,
so
maybe
also
for
the
messaging
people.
If
you
have
any
kind
of
ideas
or
questions,
you
can
add
comments
there,
and
we
will
then
definitely
revisit
this
discussion
in
the
in
the
future.
So
thanks
everybody.
D
B
F
F
Next
week
and
two
weeks,
they're
going
to
be
meetings
all
right,
that's
a
break
I.