►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sampling in the Observability Era - Steve Flanders, Splunk
In Cloud-Native environments, traditional monitoring techniques are not sufficient to understand system behavior and cannot effectively address availability as well as performance issues. This has led to the rise of observability and the importance of distributed tracing data which provides context and correlation missing from other data sources. Typically, distributed tracing data is sampled due to the amount of data it generates and concerns about whether all data generated is valuable. While sampling is often stated as a benign or normal thing, it can greatly impact observability. As a result, it is critical to understand what sampling is as well as how each sampling technique works and their associated tradeoffs. This talk will deep dive into what sampling is and at conventions as well as industry practices to address concerns including cost, performance, and overhead.
https://sched.co/cp93
A
Hello,
everyone
and
welcome
to
sampling
and
the
observability
error.
My
name
is
steve.
Flanders,
I'm
a
director
of
engineering
at
splunk,
I've
been
in
the
observability
and
monitoring
space
for
over
the
last
10
years,
working
in
logging,
distributed
tracing
and
open
source
technologies,
including
open
census
and
open
telemetry.
A
So
first
off
what
is
observability.
I
like
this
quote
from
baron
that
talks
about
how
monitoring
is
telling
you
how
a
system
works.
Usually
it's
some
sort
of
known
thing.
Maybe
it's
even
a
static
threshold.
You
get
alerted
on
where
observability
really
lets.
You
ask
any
question
that
you
want
about
your
environment,
to
figure
out
why
it's
not
working
it's
for
more
of
the
unknowns
and,
as
you
start
working
more
with
distributed,
architectures
more
cloud
native
stacks.
A
Now,
if
you've
heard
of
observability,
you've,
probably
heard
of
the
three
pillars
of
observability,
namely
logs
metrics
and
traces,
these
different
data
sources
have
some
similarities
and
some
differences.
They
all
usually
have
something
to
do
with
time,
but
the
way
that
they're
represented
are
different.
A
A
You
typically
get
your
red
metric,
so
request
errors
and
duration
again
with
associated
metadata,
and
this
is
often
used
for
problem
isolation
for
both
availability
and
performance
problems
and
we're
going
to
focus
primarily
on
the
tracing
aspect,
because
when
you
hear
the
word
sampling,
it's
usually
traces
that
you
are
sampling
to
be
fair
sampling
may
apply
to
other
data
sources
like
in
logs.
You
have
verbosity,
which
is
kind
of
a
form
of
sampling.
A
In
the
case
of
metrics,
it's
the
granularity
of
how
often
you
are
collecting
them
or
how
you
actually
roll
them
up
and
average
them.
That
is
a
form
of
sampling,
but
for
today
we're
just
going
to
focus
on
the
tracing
aspects.
Now
I
would
be
remiss
not
to
mention
the
open
telemetry
project.
Given
this.
This
is
a
cloud
native,
open
source
summit.
A
Open
telemetry
is
actually
looking
to
provide
infrastructure
around
instrumentation
data
collection,
api,
end-to-end
implementation,
basically
all
the
way
just
before
the
back
end,
so
it
doesn't
provide
a
back-end
solution,
but
it
provides
a
vendor
agnostic
way
to
send
to
the
back
end
or
back
ends
of
your
choice.
Now
I
bring
this
up
only
because
I'll
be
demoing
a
little
bit
later
to
show
how
sampling
kind
of
plays
into
this
world.
But
if
you're
interested
in
learning
more
about
this
project,
please
do
take
a
look
at
it.
A
Next
up
distributed
tracing.
So
what
is
distributed?
Tracing
I've
gone
ahead
and
taken
this
picture
from
the
yeager
tracing
documentation.
Jaeger
is
an
open
source
project,
also
part
of
cncf,
and
definitely
worth
taking
a
look
at
on
the
left.
You
basically
have
an
example
of
an
environment
where
every
circle
is
a
microservice
and
what
you're
basically
seeing,
is
a
call
of
an
end-to-end
request
throughout
the
system.
So
what
ends
up
happening
is
something
hits
service.
A
A
this
generates
a
piece
of
context
that
then
gets
passed
to
all
subsequent
downstream
calls,
for
example,
to
b
to
c
d
and
e.
How
that
ends
up
being
represented
when
it's
collected
is
more
of
a
waterfall
view
that
you
see
on
the
right,
where
a
is
kind
of
the
end
end
or
root
span
of
the
entire
transaction.
It
tells
you
how
long
its
duration
was
here.
A
We
can
also
tack
on
additional
metadata
and
because
we
have
context
and
correlation,
we
can
actually
identify
like
where
errors
are
occurring
and
why
they're
occurring,
which
really
helps
us
get
to
problem
isolation,
a
lot
faster,
which
is
super
critical
in
microservices
based
environments
that
are
typically
polyglot
where
every
service
may
be
owned
by
a
different
service
owner
and
from
an
on-call
perspective.
You
want
to
minimize
things
like
alert
storms
which
would
be
very
impactful
to
the
team.
A
A
In
addition,
you
have
the
when,
when
does
the
sampling
decision
actually
happen
and
depending
on
the
pipeline
that
you've
configured,
there
are
different
stages
in
which
it
could
occur
and
I'll
explain
that
more
in
depth?
In
just
a
moment,
you
have
the
where
so
within
a
trace,
you
have
spans
within
spans.
You
have
services
and
operations,
you
may
have
attributes
which
is
kind
of
that
metadata
or
key
value
pairs.
A
You
even
have
other
information
like
the
duration,
and
you
may
want
to
make
a
sampling
decision
based
on
these
different
pieces
of
the
telemetry
data
that
you're
collecting
the.
How
is
more
of
the
policy-based
way
to
make
a
sampling
decision
like
do?
I
want
to
do
it
based
on
some
rule,
or
some
rate
do
I
want
to
based
on
the
duration
or
a
probabilistic
way.
A
If
you
do
head-based
sampling.
What
that
basically
means
is
you
make
a
sampling
decision
on
the
initial
request?
So
if
the
a
service
here
is
the
service,
that's
hit
first,
that's
when
you're
going
to
decide
whether
or
not
to
sample
that
trace.
Now.
This
is
very
interesting
because
the
trace
is
not
completed,
so
you
don't
necessarily
know
what
happens
later
now
in
the
case
of
tail-based
sampling,
you
actually
wait
for
the
entire
end-to-end
request
to
be
completed
and
then
based
on
the
entire
request.
A
A
Now:
where
can
you
actually
make
a
sampling
decision?
Well
that
depends
a
lot
on
the
architecture
that
you've
rolled
out,
but
it's
not
uncommon
that
you
have
one
or
more
applications
and
those
applications
are
instrumented,
so
you
could
go
ahead
and
make
a
sampling
decision
in
the
instrumentation
itself.
You
probably
have
rolled
out
some
sort
of
agent.
Maybe
it
actually
exists
with
the
application
or
it's
more
of
a
host
based
agents.
You
have
one
per
host
with
multiple
applications
running
on
a
given
host.
Again,
you
could
make
a
sampling
decision
there.
A
You
could
have
some
sort
of
standalone
service
or
collector
or
aggregator
or
proxy.
Usually
this
is
like
per
data
center
or
per
region,
and
you
could
make
a
sampling
decision
there
and
then,
of
course,
the
back
end,
whether
you're
running
your
own
or
you're,
sending
to
a
third-party
sas
solution.
Sampling
decisions
can
also
be
made
on
the
back
end
so
really
anywhere
in
the
entire
pipeline.
A
You
can
go
ahead
and
make
a
sampling
decision,
but
what
you
can
sample
when
well
that
kind
of
matters
you
can
do
head-based
sampling
pretty
much
anywhere,
but
in
the
case
of
tail-based
sampling,
because
we
actually
have
to
construct
the
entire
trace
before
making
that
sampling
decision.
It's
typically
at
the
surface
or
back
end
level,
a
little
bit
more
nuanced.
A
Now,
how
does
tail-based
sampling
work?
So
I
mentioned
you
probably
want
to
do
this
at
the
service
or
the
back
end.
So
let's
say
here
I
again
have
my
topology
and
I
have
a
service
deployed.
So
I
have
three
collectors.
Satellites
proxies
whatever
you
want
to
call
them.
I
have
three
of
them
in
my
data
center
and
they
are
fronted
by
a
load
balancer
so
that
they
can
all
receive
the
traffic.
A
So,
let's
assume
that
a
request
comes
in
to
my
microservices-based
application,
a
gets
hit
and
it's
span
gets
sent
to
the
first
collector
and
then
b
gets
hit
and
it
gets
set
to
the
first
collector
as
well
c
gets
hit.
It
gets
sent
to
the
third
collector
d
gets
hit.
It
goes
to
the
third
e
goes
to
the
second.
Now
we
have
some
sort
of
maybe
round
robin
policy
base
here
on
the
load
balancer.
So
while
I
have
a
single
request,
the
spans
have
actually
gone
to
different
collectors.
A
Now
this
is
a
problem
for
tail-based
sampling,
because
I
need
to
actually
construct
the
entire
end
and
trace.
So
what
I
basically
need
is
trace
id
based
routing.
Now,
that's
typically
pretty
hard
to
do
in
the
load
balancer.
So
most
solutions
today
that
offer
tail
base
sampling
will
actually
handle
it
at
this
collector
level.
What
ends
up
happening
is
like
collector.
Two
here
would
say:
hey
I
realize
that
the
trace
id
is
already
being
stored
and
number
one
I'm
going
to
actually
pass
this
span.
A
It
depends
right,
like
probabilistic
type
rules,
you're,
typically
going
to
see
in
more
of
a
head-based
sampling
solution,
adapt,
adaptive
and
rule-based
things
are
typically
more
seen
on
the
tail-based
side
rate.
Limiting
may
be
able
to
apply
to
a
given
service,
but
may
not
be
applicable
to
a
given
operation.
So
only
certain
aspects
of
the
span
information
can
be
leveraged
and
then
policies
really
depend
on
the
service
or
the
backend
capabilities,
whether
that's
a
vendor
or
an
open
source
solution.
A
These
vary
greatly
upon
like
what
is
possible
from
a
configuration
perspective,
and
this
is
really
just
the
tip
of
the
iceberg.
There
are
more
nuance
here
as
well,
but
again
the
goal
is
to
focus
on
the
primary
themes
around
sampling,
to
give
you
a
better
understanding
of
the
topic
with
that
I'd
like
to
jump
into
trade-offs.
A
So,
let's
start
with
head-based
sampling
again.
Some
of
the
pros
of
this
approach
are
it's
very
easy
to
configure
it
and
to
manage
it.
In
addition,
it's
great
at
actually
reducing
the
amount
of
data
that's
being
sent,
which
is
one
of
the.
Why
reasons
of
why
you
do
sampling
anyway,
that
happens
at
the
the
root
service
based
level?
That's
where
you
make
the
sampling
decision
now
the
cons
are
well.
I
didn't
actually
wait
for
the
entire
request
to
complete.
A
So
I
don't
know
if
I'm
making
an
educated
or
a
smart
sampling
decision,
I'm
just
making
one
very
early
on
another.
Very
big
problem
is
that
if
I'm
making
a
sampling
decision
at
the
beginning,
then
I
probably
won't
have
accurate
red
metrics,
because
I'm
not
looking
at
all
of
the
calls,
I'm
not
actually
collecting
all
that
data
and
the
net
result
is
really
incomplete.
Observability
not
only
have
I
reduced
the
data,
and
I
probably
don't
have
representative
sets
for
the
things
that
I
care
about.
A
Even
my
red
metrics
are
not
a
hundred
percent
accurate
because
I'm
not
analyzing
all
that
data,
so
in
general
I
mean,
if
you
can
avoid
this
model.
You
typically
want
to
at
a
minimum.
You
want
to
have
full
fidelity
metrics.
You
have
accurate,
slis
or
red
metrics,
with
accurate
alerts
that
you're
leveraging
here
and
typically
with
head-based
sampling.
You
can't
do
that
now.
There
can
be
some
temporary
reasons
why
you
might
want
to
enable
head-based
sampling.
A
Maybe
it's
controlling
the
data
or
cost,
but
you
just
need
to
know
that
it
really
can
impact
your
observability
and
typically
is
not
the
solution
you
want
to
head
down.
So
let's
again
use
a
representation
of
head-based
sampling.
Here,
as
I
mentioned,
it
gets
made
at
the
parent
or
root
level
span.
So
here
it
would
be
the
a
service.
Well,
what
happens
if
further
on
I
decide
I'm
not
going
to
sample
this
given
trace
and
then,
when
the
d
service
gets
called,
it
actually
generates
an
error.
A
Well
because
I
made
a
sampling
decision
when
a
was
originally
called,
and
I
decided
not
to
sample
the
fact
that
an
error
was
thrown
in
d,
doesn't
matter,
I've
now
lost
that
data
that
really
impacts
my
observability.
I
can't
alert
on
that
scenario
like
if
someone's
on
call
they
can't
actually
pull
up
that
trace.
That's
a
huge
impact
to
my
ability
to
ask
any
question
of
my
system
and
to
really
troubleshoot
perform
problem
isolation.
A
Tail
based
sampling
on
the
other
side
actually
provides
full
fidelity
metrics,
and
it
allows
you
to
reduce
the
data
based
on
anything
available
in
the
telemetry
data
that
you're
collecting,
so
that's
very
powerful.
The
cons
are
that
it's
typically
hard
to
manage
and
it
can
be
quite
expensive
to
actually
run
these
collectors.
I'm
talking
about
primarily
the
customer
side.
A
Now,
at
the
end
of
the
day.
Another
con
that
I
wrote
is
that
you
have
to
analyze
all
the
data
anyway,
like
I
mentioned
all
the
compute
and
potentially
network
traffic,
that's
involved
with
that.
That's
actually
the
primary
cost.
So
if
you
already
are
going
to
take
that
hit
of
sampling
all
that
data,
then
then
why
are
you
doing
tail-based
sampling
in
the
first
place
now
there
can
be
good
reasons
for
it,
but
it's
definitely
a
question.
A
You
should
ask
yourself
so
when
you're
thinking
about
this,
if
you
have
to
sample
and
we'll
talk
more
about
whether
you
do
or
not
in
just
a
second
tail-based
sampling
is
going
to
be
better
than
head-based
sampling,
because
you
now
have
full
fidelity,
accurate
metrics.
You
now
have
flexibility
in
actually
being
being
able
to
choose
what
you
sample
and
not.
There
are
some
potential
cons,
but
the
flexibility
and
choice
here
is
quite
powerful.
A
One
thing
you
do
need
to
consider,
though,
is
you
have
to
factor
in
both
capex
and
opex
when
you're
determining
the
cost
of
a
solution
running
these
satellites
or
collectors
on
your
side,
scaling
them
up,
making
sure
that
they're
they're
upgraded
and
handling
the
network
traffic?
That's
associated
with
them,
that's
all
part
of
the
cost
of
leveraging,
whatever
solution
that
you
are
and
then
one
final
thing,
I
didn't
write
this
as
a
con,
but
most
tail-based
solutions.
Don't
really
have
enterprise-y
features
like
encryption
between
the
satellites
or
some
sort
of
dispatch
queue.
A
It's
usually
all
in
memory.
This
isn't
necessarily
a
big
deal.
It
comes
down
to
really
business
requirements
like
a
lot
of
people,
don't
treat
tracing
data
as
containing
much
or
any
pii.
It's
not
necessarily
the
end
of
the
world.
If
you
lose
that
data
like
it's
not
ideal,
if
you're
having
an
incident
or
an
outage,
but
at
other
times,
maybe
it's
not
as
critical.
If
some
of
that
data
is
lost,
just
something
to
keep
in
mind
when
you're
actually
evaluating
these
solutions
as
well.
A
So
let's
again
go
back
to
the
tail-based
sampling
example
and
how
it
works.
As
I
mentioned,
there
are
a
few
different
considerations
here.
So
let's
assume
that
I
have
my
end-to-end
app
deployed
in,
I
don't
know
I'm
in
aws,
so
I'm
in
a
single
region
and
it's
going
to
be
deployed
in
multiple
availability
zones.
Azs
because
I
want
to
have
high
availability
in
case
an
ac
goes
down,
so
these
satellites
may
be
deployed
in
a
single
az
or
across
multiple
for
redundancy
reasons.
A
A
It's
even
more
complex
if
I
actually
have
an
end-to-end
request
that
spans
multiple
different
regions-
and
there
are
valid
use
cases
for
that.
In
that
case,
I
might
have
my
services
in
region
one
and
region
two.
Well,
I
have
to
have
this
set
of
satellites
in
a
single
region.
It
doesn't
matter
if
it's
in
one,
two
or
even
three,
but
all
the
traffic
needs
to
go
to
a
single
region
so
that
I
can
actually
collect
a
single
trace
and
end
well.
A
That
map
now
means
I
might
have
cross
region
traffic
network
wise
that
I
need
to
pay
for
as
well
and
then,
as
I
mentioned
earlier
at
the
end
of
the
day,
it's
really
all
the
overhead
of
all
the
traffic
needs
to
be
able
to
be
processed
by
these
satellites.
The
amount
of
compute
power
memory
that
this
requires
can
be
extensive,
depending
on
the
solutions
that
are
available,
there
may
even
be
disk
related
as
well.
That
is
a
cost
of
leveraging
a
tail-based
sampling
solution.
A
So
now,
let's
go
back
to
the.
Why?
Why
would
you
sample
in
the
first
place-
and
these
are
some
of
the
common
reasons
that
come
up?
One
of
the
biggest
is
just
that
scaling
is
hard
like
we're
not
talking
about
a
small
amount
of
data
here
distributed
tracing
information
is
typically
significantly
more
than
metrics
and
it's
usually
on
par
with
logs,
maybe
a
little
less
in
some
cases,
maybe
a
little
more,
but
it's
pretty
extensive
and
so
the
back
end
required
to
actually
process
and
store.
A
This
information
is
quite
a
bit
and
then
the
front
end
to
be
able
to
query
it
and
display
it
quickly.
That's
also
a
hard
problem.
It's
a
big
data
problem.
Now
this
really
comes
down
to.
Are
you
rolling
your
own
like?
Do
you
have
your
own
back
end?
Is
it
open
source
if
you
built
something
custom
and
you're
just
trying
to
scale
it?
Is
that
actually
your
core
ip?
Is
that
where
you
want
to
be
investing
your
efforts?
Are
you
going
to
buy
instead
into
a
vendor
solution?
A
What
do
they
offer
right
that
there's
trade-offs
to
the
decision?
The
next
point
is
just
around.
Not
all
data
is
equal
like
if
I
have
a
trace
that
contains
an
error.
That's
probably
much
more
valuable
than
a
trace
that
does
not
contain
an
error.
Same
applies
with
things
like
duration,
like
if
I
have
an
outlier,
if
I'm
like,
above
my
p90
or
p95
or
even
p99,
I
probably
want
to
keep
that
data.
I
want
to
understand
why
I
want
to
have
more
samples
of
that
versus,
if
they're,
all
the
same
exact
average
duration.
A
Do
I
really
need
every
one
sure
I
need
them
from
a
metrics
perspective,
but
do
I
need
the
individual
traces
and
then
maybe
like?
I
want
production
to
to
not
sample.
I
want
development
to
sample
heavily
just
from
a
cost
perspective
and
from
an
availability
like
if
dev
is
down.
Maybe
it's
not
the
end
of
the
world.
A
In
my
environment
again,
this
comes
down
to
business
requirements
and
the
problems
you're
trying
to
solve,
but
really
the
biggest
reason
that
comes
up
around
sampling
is
cost
and
when
you
think
of
costs,
you
need
to
think
of
many
things.
The
processing
aspects,
the
network,
egress
costs,
the
storage
costs
of
containing
all
that
data
and
storing
it
for
whatever
retention
period
you
care
about
the
whole
run
it
versus
buy
it
like.
Are
you
rolling
your
own
back
end?
Are
you
rolling
your
own
solution
to
instrumentation,
or
are
you
leveraging
open
source?
A
Are
you
doing
some
open
source
in
some
vendor
and
then
finally,
the
cost
of
downtime
like
if
I
sample-
and
I
don't
have
the
data
that
I
need,
if
I
don't
have
the
observability?
If
I
can't
answer
the
questions,
I
need
to
what
does
that
cost
to
my
business,
and
am
I
willing
to
actually
pay
that
so
there's
a
lot
of
trade-offs
and
decisions,
but
this
is
why
sampling
is
a
real
thing
in
distributed
tracing
now
I
want
to
kind
of
warn
about
the
marketing
hype
and
now
full
disclosure.
A
I
work
for
a
vendor,
so
this
applies
just
as
much
the
company
I
work
for
as
any
other
vendor
out
there,
any
other
marketing
people
out
there,
but
you
need
to
be
aware
of
some
things
when
it
comes
to
sampling
like
you'll
hear
sampling
is
fine
because,
like
you
have
all
the
data
for
real-time
investigation,
what
does
that
mean?
It
usually
means
you
have
it
for
a
very
small
window
of
time.
We're
talking
minutes.
Maybe
five
minutes,
maybe
15
max
sure,
that's
great.
A
A
Another
one
is
for
just
around,
like
you
have
to
sample,
because
it's
just
too
much
data.
A
lot
of
this
comes
out
of
like
the
dapper
paper
that
came
from
google,
but
one
of
the
things
you
should
ask
is
like
am:
am
I
working
at
a
fang
company
where
there
is
just
so
much
data
that
I
absolutely
have
to
do
something
about
it?
Most
of
us
are
not
in
that
category
and
even
at
millions
of
spans
per
second,
it's
not
as
much
of
a
problem,
as
you
might
think.
Now.
A
Eventually,
you
will
hit
some
threshold
where
you
need
to
start
considering
it,
but
not
every
company
out
there
is
a
google
company,
so
you
need
to
take
that
with
a
grain
of
salt
as
well.
Are
you
going
to
manage
your
own
observability
back
end?
Are
you
going
to
use
open
source?
Are
you
going
to
use
a
vendor
solution?
A
All
of
that
kind
of
dictates
the
amount
of
data
that
you
can
collect
and,
of
course,
the
cost
associated
with
that
and
then
finally,
like
what
have
you
done
to
instrument
your
applications
and
if
it
is
so
verbose,
if
you
do
have
so
much
traffic,
why
what
problems
are
you
trying
to
solve?
Is
that
necessary?
Did
you
instrument
too
much?
A
Do
you
have
too
much
metadata,
like
is
the
instrumentation
that
you
provided
valuable
or
not,
because
that
might
be
a
better
problem
to
go
solve
than
whether
or
not
you
sample
the
data
and
then
finally,
just
around
hey
our
solution's,
cheaper
because,
typically
of
sampling,
again,
you
have
to
look
at
the
whole
ownership
thing.
Tail-Based
sampling
doesn't
come
for
free.
That
cost
comes
from
somewhere
either
it's
on
your
end
or
it's
on
the
vendor
end,
but
there
is
a
cost
associated
with
it.
A
If
you
sample
you're
going
to
lose
some
amount
of
observability,
how
much
is
acceptable,
typically
head
base
without
the
full
fidelity
metrics?
That's
not
going
to
be
good
enough
for
most
environments,
but
tail-based
sampling,
where
I
may
not
have
every
trace,
but
I
do
have
fully
accurate,
metrics
and
slis.
Usually
that
is
sufficient
and
then
are
your
specific
business
requirements
and
desired
outcomes
being
met
that,
at
the
end
of
the
day,
that's
really
the
most
important
thing.
A
A
So
what
we
basically
have
is
an
open,
telemetry
collector
and
in
that
open,
telemetry
collector.
We
are
going
to
point
to
three
different
jaeger
instances,
so
jaeger's
distributed
tracing
back
end
and
what
I've
done
is
I've
configured
three
different
pipelines
in
the
open,
telemetry
collector.
So
basically
we're
going
to
receive
100
of
the
traffic
and
we
are
going
to
send
it
to
a
jaeger
instance.
A
We
are
going
to
use
a
probabilistic
sampler
and
send
a
subset
of
that
traffic
to
a
second
jaeger
instance,
and
then
we're
going
to
leverage
tail
based
sampling
and
we're
going
to
send
it
to
a
third
instance.
This
also
kind
of
show
you
the
pros
and
cons
of
the
of
the
different
approaches.
Now,
from
a
configuration
perspective,
the
probabilistic
sampler,
the
head-based
sampling,
is
configured
to
sample
one
percent
of
the
traffic.
Again
this
is
customizable,
so
I
could
say
five
percent.
I
could
say
zero
point.
A
One
percent,
I
could
say
fifty
percent,
it's
really
configurable
to
your
needs,
tail-based
sampling.
What
we're
going
to
do
is
we're
going
to
tell
it
that
we
have
a
couple
policies.
One
is
if
it
contains
an
attribute
of
error
equals
true,
then
we're
going
to
sample
it
and
the
next
one
is.
If
we
have
an
http
status
code,
that
is
between
400
and
599,
again,
we
are
going
to
sample
it.
So
this
gives
me
kind
of
choice.
These
are
just
some
examples.
A
You
might
have
different
rules
or
policies
or
different
sampling
things
that
you
want
to
leverage.
This
may
configure
to
meet
your
needs,
but
hopefully
this
will
give
you
an
idea
of
how
it
actually
works
in
practice.
Now,
on
the
flip
side,
I
mentioned
this
is
a
docker
compose
file,
so
we
have
three
jaeger
instances
running.
We
have
an
open,
telemetry
collector
running
and
then
we
basically
have
a
load
generator.
That's
passing
traffic
to
the
collector.
That's
basically
everything
here.
This
is
based
on
the
open
symmetry
collector
demo,
that's
in
the
github
repository
today.
A
So
let's
go
ahead
and
fire
up
the
jaeger
instance
that's
receiving
all
of
the
traffic.
So
I'll
just
pick
one
of
the
services
and
you'll
see
that
there's
traffic
coming
in
here.
The
cart
service
is
the
one
that
I'm
looking
at,
particularly
so
tons
of
traces
are
coming
in.
That's
great.
I
will
pick
the
second
one
here.
This
is
doing,
probabilistic
sampling
so
again,
I'll
pick
the
cart
service
and
you'll
see,
there's
actually
less
traces
significantly
less
and
that's
because
we're
only
sampling,
one
percent
of
them
and
then
the
final
one.
A
This
is
the
tail-based
sampling.
That
is
only
capturing
errors.
So
in
this
case
you'll
see
again
cart.
Service
only
cart
service,
traces
that
contain
errors
are
showing
up.
You
won't
see
any
spans
or
traces
that
contain
non-errors,
because
that
wouldn't
meet
the
tail
based
sampling
rule.
So
again,
three
different
ways
of
collecting
no
sampling,
error-based
sampling,
probabilistic
sampling
at
a
one
percent
rate,
very
easy
to
get
started
very
easy
to
kind
of
configure
in
your
environments,
and
then
you
can
solve
more
of
the
scale
problems
or
the
different
policies
that
you
care
about.
A
The
rules
you
want
to
write
for
making
these
sampling
decisions
so
some
key
takeaways
from
this
presentation.
One
is
definitely
understand
your
business
requirements
and
the
desired
outcomes
every
environment's
a
little
bit
different.
What
are
you
trying
to
solve
for?
What
are
the
pain
points?
You
have
what
visibility
and
observability
do
you
really
need?
A
Second
understand
your
sampling
options,
understand
their
impact
to
both
cost
and
observability,
and
compare
that
against
your
business
requirements
and
desired
outcomes
and
decide
what
is
right
for
you
and
then.
Finally,
beware
of
the
marketing
hype.
There's
a
lot
of
a
lot
of
information
out
there.
It
can
be
hard
to
kind
of
navigate
all
the
noise
there's,
also
like
traditional,
more
solutions
versus
cloud
native
solutions
and
the
thought
processes
are
a
little
bit
different.
A
And
with
that?
Thank
you
so
much
for
joining
me.
This
is
really
just
an
introduction
to
different
types
of
sampling,
things
that
you're
going
to
come
across
specifically
for
distributed
tracing.
I
would
love
to
talk
to
folks
more
about
it
feel
free
to
reach
out
to
me
directly
and
thanks
so
much
for
joining.
B
A
Go
ahead,
so
the
first
one
is:
will
cloud
native
observability
be
a
topic
that
will
have
like
a
training
course
and
certification?
I
think
long
term.
That
wouldn't
be
too
surprising.
In
fact,
right
now
in
the
cncf
they
actually
started
a
sig
or
special
interest
group
on
observability,
and
some
of
these
topics
are
being
discussed
like
what
is
the
best
way
to
start
promoting
and
getting
more
educated
about
observability.
A
Some
of
the
initial
thoughts
are
around
more
blogs
and
just
other
ways
of
getting
material
around
observability,
but
long
term.
Sure
like
this
is
an
important
topic.
A
lot
of
the
projects
in
cncf
are
observability,
based
jaeger
being
a
big
one,
fluent
fluent
bits
in
there.
So
yeah.
I
think
long
term
that'll
happen.
I'm
not
aware
of
any
immediate
plans
for
that,
though,
next
question
your
recommendation
to
end
users
that
want
to
build
their
own
solutions
for
observability
yeah,
so
there's
start
with
cncf
or
start
with
open
source
projects.
A
There's
some
good
stuff
out
there,
like
jager,
is
a
very,
very
scalable
product,
very
easy
to
get
started.
They
actually
have
decoupled
the
architecture
very
well.
So
it's
easy
to
scale
up
individual
components
as
the
the
amount
of
data
that
you're
collecting
increases
and
they
can
support
more
streaming
based
systems
if
you
need
it
from
a
logging
perspective.
Fluent
bit
is
also
very
efficient
at
collecting
this
data.
A
It
has
rich
parsing
rules,
so
actually
get
the
data
in
the
format
that
you
care
about
prometheus
again
great
metricking
tool
to
to
collect,
in
more
cloud
native
environments,
very
common,
becoming
kind
of
the
standard
way
to
collect
metrics
today
compared
to
other
other
mechanisms,
so
you
can
definitely
start
with
the
open
source
tools
and
then
you'll
probably
start
building
on
top
of
them.
Like
hey,
I
have
prometheus,
and
now
I
want
to
have
high
availability
or
I
want
to
scale
this
to
a
significant,
a
very
large
scale.
A
How
do
I
do
that
and
again
there
are
open
source
solutions
out
there
to
to
get
you
started
so
from
a
getting
started
perspective.
It's
really
easy
if
you're
looking
for
a
single
tool
that
provides
all
the
different
data
sources
and
a
way
to
kind
of
quarry
against
them,
that's
not
as
common
today
and
as
you
get
to
more.
Like
enterprise
features
like
I
care
about
massive
scale
or
I
care
about
end-to-end
encryption
or
I
care
about
very
rich
pii
or
personally
identifying
information
being
extracted.
A
That
can
be
a
little
bit
harder,
but
there
are
solutions
out
there
for
both
open
source
and
and
commercial
depending
on
your
needs,
and
then
final
one
is
around
istio
and
asking:
if
is
there
any
solution
to
get
end-to-end
request?
Training
working
without
span
context
propagation,
as
this
will
make
it
easier
and
motivate
more
customers
to
kind
of
take
advantage
of
this
or
more
end
users
take
advantage
of
this.
Yes,
let
me
break
this
down
for
people
that
are
not
aware.
So
istio
is
a
service
mesh.
A
Basically,
how
it
works
is
you
deport,
deploy
proxies
to
every
one
of
your
applications
and
all
of
your
traffic
between
applications
get
routed
across
these
proxies
so
because
there's
a
proxy
in
between
you
can
actually
inspect
that
traffic
and,
as
a
result,
one
common
question
is
well
hey.
If
I'm
inspecting
this
traffic
anyway,
can
I
get
my
distributed
tracing
information
from
that
inspected
traffic
and
the
answer
is
of
course.
Yes,
you
definitely
can
now
I
need.
I
need
to
take
a
step
back
to
answer
the
question
with
distributed
tracing.
A
It
works
via
something
called
context.
Propagation.
I
talked
about
this
a
little
bit
in
the
presentation.
Basically
you,
if
I'm
thinking
about
like
rpc
or
remote
protocol,
like
type
things
like,
let's
think
of
restful
services,
it
basically
means
I
need
to
pass
headers
across
each
one
of
my
requests
so,
for
example,
the
trace
id.
I
have
to
pass
that
to
every
single
call
I
make
over
the
network.
Well,
what
happens
if
my
application
doesn't
actually
accept
or
pass
these
headers?
A
Well
then
context
propagation
is
broken
and,
as
a
result,
my
distributed
tracing
is
broken.
That's
a
problem,
so,
while
istio
or
any
service
mesh
can
provide
traffic
about
everything
can
provide
trace
information
about
everything
that
passes
through
the
proxy.
It
can't
guarantee
that
the
headers
that
are
being
passed
through
the
proxy
that
the
application
actually
accepts
and
propagates
to
its
downstream
calls.
So,
unfortunately
the
answer
is
no.
There
is
no
easy
way.
A
Next
question
is
google's:
hipster
is
not
supporting
open
telemetry.
Is
there
any
way
to
get
the
hipster
demo?
Okay,
so
yeah,
so
the
so
open
symmetry
is
the
joining
of
two
different
projects:
open
tracing
and
open
census.
Open
census
has
something
called
the
hipster
demo
application.
So
it's
basically
a
poly
polyglot
micro
services,
application
that
was
instrumented
with
open
census
and
it
was
used
as
a
demo
environment
to
kind
of
show
off
the
power
of
open
census.
A
We
are
actually
using
it
for
open
symmetry
as
well,
so
open
symmetry
merges
open
census
and
open
tracing
together.
It
actually
supports
the
open
census
format.
So
the
fact
that
the
hipster
demo
app
is
sending
open
census
is
totally
fine.
Now,
I'm
assuming
the
question
is
just
around
well:
hey:
it
doesn't
support
open
telemetry's
format,
which
is
called
otop
and
that's
true.
It
doesn't
and
the
reason
for
that
is
because
otop
was
just
finalized
less
than
a
month
ago.
A
So
it's
very
new
and
we
were
waiting
for
the
tracing
spec
to
become
finalized
and
waiting
for
the
otop
protocol
to
the
first
version
of
it
to
be
rolled
out.
That's
now
done
so
I
think
you'll
actually
see
the
hipster
demo
app
be
instrumented
with
open
symmetry
going
forward,
but
there's
no
reason
why
you
can't
use
it
with
the
open
symmetry
project
today,
because
the
open
symmetry
project
does
support
open
census.
So
if
you
have
the
collector,
for
example,
you
can
receive
and
export
open
census
format,
so
the
same
demo
application
can
be
used.
A
I
know
that
I
still
use
the
hipster
demo
app
today
when
I'm
showing
off
open
telemetry,
there's
also
a
question
about
how
do
you
get
this
tracing
hipster
demo
application?
I
don't
have
the
link
readily
available,
but
I'm
pretty
sure
if
you
just
go,
google,
the
google
microservices
demo
application
it's
on
github.
I
think
if
you
just
do,
google
hipster
microservices
you'll,
probably
find
it,
but
on
github
there
is
a
repository
that
basically
lists
many
different
microservices.
It
shows
off
all
the
primary
languages,
so
you
have
like
net
java
javascript
python.