►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Practical and Useful Latency Analysis using Istio and OpenCensus - Varun Talwar, Stealth Startup & Morgan McLean, Google (Intermediate Skill Level)
Imagine debugging and trying to find the cause for latency spikes for some customers in a microservices environment with hundreds of services, deployed on kubernetes. Imagine you are trying to do hotspot analysis of which service and service instances are in request path for those select requests and what is causing an increase in p95 or p99 latency. We will show in this session how Istio Proxy - Envoy integrated with OpenCensus can help achieve this very easily with minimal effort from your end.
About Varun
Product Management lead for gRPC and Istio at Google
About Morgan
Morgan Bauer works on Open Source at IBM in San Jose as part of the Cloud Division. Morgan is a maintainer on the core Docker Engine and also a founding contributor of the Kubernetes Service-Catalog.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
Morgan
MacLean
I
work
at
Google,
I'm,
the
product
manager
for
open
census,
as
well
as
a
number
of
our
stackdriver
products,
so
we're
here
today
to
talk
about
two
highly
related
topics:
open
census,
which
is
a
tracing
and
application
metrics
framework
that
we
publish
from
Google
and
it's
open
source
and
then
also
sto
and
envoy.
And
what
we're
going
to
focus
on
is
the
convergence
between
these
so
being
able
to
apply
tracing
and
metrics
and
and
gathered
other
types
of
signals
from
applications
that
are
running
inside
of
envoy.
B
We're
gonna
start
with
a
bit
of
a
basic
background
on
service
development.
So,
if
you've
been
to
other
tracing
talks
before
you've
probably
heard
this
part,
but
hopefully
you'll
still
have
a
few
insightful
things.
So,
as
everybody
is
very
aware
of
the
world,
is
moving
from
large
monolithic
code
bases
to
more
highly
distributed
systems
to
commonly
the
term
micro
services
is
used
to
describe
this,
and
so
we
wanted
to
give
you
an
example
just
to
really
sort
of
make.
It
really
apparent.
B
Maybe
then
a
database
guys
call
them
response
gets
sent
back
to
the
person
who
called
in
and
that's
you
know,
useful,
and
we
historically
have
had
many
many
tools
that
allow
developers
to
inspect
this,
whether
it's
you
know
traditional
debuggers
running
your
application
in
a
test
environment,
doing
just
plain
old,
static
code
analysis
and
your
ID
of
choice,
logging
system
level,
metrics,
etc.
And
these
are
things
that
you
know
the
whole
industry
is
pretty
used
to
I.
Think
you
know
most
developers.
B
Probably
almost
everybody
here
know
that
if
you
show
up
to
a
job
at
a
new
company-
and
they
don't
use
like
basic
things
like
whether
it
be
like
source
control
or
like
a
creative
logs
or
things
like
that,
you
should
you
know-
probably
walk
out
the
door
and
people
understand
us.
But
as
we
move
into
the
wild
wild
world
of
micro
services,
there
haven't
historically
been
the
same
kind
of
tools
available
right
and
so,
if
we
think
of
distributed
tracing
which
allows
you
to
track
a
request
as
it
hops
across
multiple
systems.
B
This
isn't
just
useful
for
latency.
Analysis
like
it
is
at
companies
like
Google
and
others
who
care
very
deeply
about
their
latency.
This
is
actually
how
you
understand
how
your
app
works
in
some
ways.
In
addition
to
the
latency
benefits,
it's
almost
like
looking
at
a
stack
trace
right
and
without
a
distributed
trace
with
micro
services.
It's
almost
like
trying
to
debug
a
problem
without
even
having
access
to
a
stack
trace
in
a
traditional
environment
which,
like
no
one,
would
want
to
be
in
that
world
and
you
know
distributed
tracings
up
there
with
you
know.
B
Other
technologies
like
profiling
and
debugging
that,
while
they
had
a
you,
know
ton
of
applications
and
uses
in
your
traditional
model.
If
the
code
bases,
we
need
to
take
a
different
approach
when
it
comes
to
micro
services,
and
this
is
what
open
census
aims
to
target
so
double-clicking
on
distributed
tracing.
This
is
a
screen
shot
of
Zipkin
that
I'm
sure
many
people
are
pretty
familiar
with
at
this
point,
but
it's
a
open
source
distributed
tracing
backend
along
with
its
own
tracers
distributed
tracing,
gives
us
a
number
of
things.
B
It
gives
us
the
topology
of
our
application,
so
you
understand
which
services
call
which
others
for
a
given
request.
It
gives
you
a
review
into
how
those
requests
are
handled
and
a
rough
view
of
how
much
time
is
spent
on
each
component
of
the
request,
and
it
lets
you
understand
where
the
system
is
performing
poorly,
where
errors
are
occurring
and
also
gives
you
correlation
for
other
signals.
So
one
really
common
thing
with
distributed.
B
Open
census
also
provides
app
level
metrics,
which
includes
you,
know:
latency
metrics,
RPC
stats
at
every
level
of
the
sack,
as
well
as
custom,
metrics,
so
I've,
given
you
a
bit
of
background,
a
distributed,
tracing
and
and
again
like
those
of
you
who
are
already
familiar,
probably
bored
to
tears
already,
but
it's
not
necessarily
an
easy
problem
to
solve,
and
this
is
where
open
census
really
comes
in.
So,
ideally,
you
want
traces
and
stats
to
be
automatically
collected
for
every
request
that
gets
routed
through
each
of
your
microservices.
B
This
is
actually
not
that
simple
I
think
men,
single
men
just
had
a
talk
on
open
tracing
which,
which
is
targeting
the
same
sort
of
set
of
challenges,
and
the
reason
you
see
projects
like
these
is
that
for
distributed
tracing,
you
need
to
have
plugins
or
insights
into
every
single
web
framework.
Rpc
framework,
storage,
client
library,
AP,
API,
client
library,
basically,
every
single
thing
that
can
interact
with
or
generate
rpcs
to
get
that
data
out
automatically.
B
You
can
do
this
yourself
as
a
developer,
but
it's
a
royal
pain
and
most
companies
and
most
developers
won't
bother.
So
we
need
a
better
way
of
making
this
automatic
there's
also
a
conversation
around
whether
libraries
so
like
things
that
you
link
to
your
app
versus
agents
that
sit
and
like
do
monkey
patching
or
something
else
to
get
this
data
out,
which
of
those
is
more
useful
agents,
can
be
very
slick.
You
run
them
in
the
background
they
automatically
directed
your
app.
B
There
are
some
challenges
of
that,
which
is
why
open
census
is
distributed
as
a
set
of
libraries,
the
biggest
just
being
that
we
want
to
provide
an
API
to
developers,
but
also
that
they
can.
It
can
be
managed
through
source
control.
So
if
you
want
to
add
open
census
to
your
app,
if
you
want
to
update
the
version
of
it
or
so
on,
you
just
do
that
through
source
control
make
a
new
deployment
you
don't
have
to
mess
around.
If
anything,
in
the
images
that
you're
using
so.
A
What
does
open
census
really
provide?
It
wasn't
obvious.
Open
census
is
a
single
set
of
libraries
for
metrics
and
traces.
So
it's
I
think
that's
one
key
sort
of
takeaway.
It
does
both
it's
not
just
tracing
and
it
intends
to
be
holistic
in
terms
of
correlating
with
logs
and
profile
over
time.
So
what
is
it
today?
It's
a
set
of
libraries
in
few
languages.
The
bold
ones
are
where
you
have
metrics
and
tracing
available.
The
rest
are
coming.
A
It
already
has
integrations
with
webinar
PC
frameworks,
so
GRP
see,
for
example,
the
plumbing
is
already
built
in
with
open
census
spring,
and
some
of
those
integrations
with
other
frameworks
is
coming
a
lot
of
the
client
libraries.
So
a
lot
of
Google
client
libraries
are
being
instrumented
with
open
census.
What
does
that
really
mean?
A
So
if
you're
a
user-
let's
say
you
use,
you
know
data
store
library
from
from
Google
cloud,
then
you
would
have
Plumbing's
for
open
census
already
built
in
such
that
when
you
are
using
it
as
part
of
your
set
of
micro
services,
you
get
metrics
and
distributed
tracing
in
a
back-end
of
choice
already
there
without
you
needing
to
do
work
to
instrument
that
and
there's
more
integrations
with
of
open
census.
Libraries
happening
so
I
think
is,
is
the
one
that's
already
done
and
there's
more
that
community
is
doing
in
terms
of
integrating
more
libraries,
we're.
B
A
As
a
developer,
you
get
these
libraries
and
you'll
get
api's
for
interacting
with
some
concepts
that
exist
in
open
census.
So
there's
a
few
concepts
of
trace
spans
and
tags,
and
we
will
touch
upon
some.
So
just
you
know
span
is
an
individual
sort
of
call,
there's
a
parent-child
relationship
between
a
set
of
spans.
So
let's
say
a
request
comes
in
from
a
user,
it
goes
through
a
bunch
of
services,
there's
a
parent
span
and
then
a
child
span.
If
it's
calling
five
other
services,
that's
like
five
other
spans
and
they
collectively
form
a
trace.
A
Con
these
are
basic
tracing
concepts
and
there's
API
is
for
interacting
them
in
open
census
as
a
developer.
The
one
other
concept
here
is
exporters.
So
when
you
have
these
metrics,
you
want
to
get
them
in
to
export
them
into
a
system
of
choice,
so
that
is
another
set
of
integrations
happening
with
open
census,
so
stackdriver
yeah.
B
We
support
driver
Prometheus
for
stats
Zipkin
for
traces
Yaeger
for
traces
signal
FX
for
stats.
There's
a
big
list.
That's
on
the
website,
there's
number
more,
that's
the
ones
I
can
sort
of
name
off
my
head,
but
the
the
kind
of
the
the
nice
thing
about
open
census
is
that
all
of
these
things
are
baked
into
like
the
main
library
that
you
download.
So
when
you
link
open
census
to
your
app,
you
choose
the
exporter
that
you
want
to
use.
B
It
will
automatically
gather
as
many
traces
bans
and
stats
as
possible,
and
it
will
send
those
to
your
back-end
of
choice.
You
can
also
use
multiple,
so
we
suspect,
like
a
really
common
use
case,
we'll
be
using
Prometheus
for
stats
and
then
Zipkin
or
Jager
for
tracing.
But
if
you
want
to
go
use
every
single
exporter
under
the
sutton,
you
are
absolutely
able
to
do
that.
This
is
important
for
some
companies,
where
you
had
different
teams
that
have
used
different
tracing
or
stats
backends,
and
you
can
continue
to
support
that
scenario.
Foe,
consensus.
A
All
right
so
one
other
very
useful
concept
that
exists
with
open
census,
and
it's
common
at
you
know
places
like
Google
for
any
service
developer
to
basically
go
to
a
service,
URL
and
say
slash
Tracy
or
/,
VAR
z,
or
slash
RPC,
Z,
and
that's
of
extremely
extremely
useful
concept
to
be
able
to
go
to
any
service
and
get
a
pretty
standardized
set
of
real
time,
unsampled
metrics.
So
you
are
having
some
problems
this
actually,
the
your
first
behavior
is
okay.
A
Let
me
check
the
/r
PCC
page
and
look
at
the
list
of
what
was
the
last.
You
know
few
are
PCs
what
were
their
latency
times?
What
were
the
error?
Samples
and
you
can?
You
can
actually
click
on
these
to
get
to
the
individual
requests,
and
you
can
look
at
the
trace,
IDs
and
span
ID
for
every
request,
for
that
was
generated
for
every
incoming
request
right
so
again,
extremely
useful
concept,
which
is
built
in
with
as
part
of
open
census.
So.
B
That
the
open
census,
client
libraries
each
host,
it's
optional,
but
they
host
these
pages.
They
have
a
little
web
server
built
it,
and
so
you
can
go
there
and
get
just
a
firehose
view
of
all
the
spans
and
stats
that
are
going
through
a
system.
Traces
are
generally
sampled,
often
that,
like
a
very
low
percentage
like
1%
or
fewer,
this
gives
you
data
to
everything
that
doesn't
even
get
sent
to
your
back-end.
So
if
you're
debugging
a
critical
production
issue,
the
EZ
pages
are
incredibly
useful
and
this
is
built
into
open
census.
A
So
imagine
again,
you're
doing
the
analysis,
but
you
say:
ok,
I
want
to
focus
on
I
want
to
get
a
subset
of
latency
or
some
other
metric
or
set
of
traces
for
a
subset
of
requests,
which
is
only
client
equal
to
Android,
or
only
for
user
agent
call
requests
that
happened
or
forgiven,
IP
is
and
so
on.
So
you
can
add
tags
which
is
basically
dimensions
in
your
open,
send
in
your
code
and
you
can
basically
define
your
own
dimensions
and
get
your
metrics
by
those
dimensions.
A
B
So
I
can
talk
about
this,
so
this
does
seem
like
a
somewhat
self-serving
best
slide,
but
I
can
explain
so
at
Google
and
other
not
just
Google,
but
other
large
web
services.
Companies
we've
been
using
these
technologies
for
a
long
time.
These
are
basically
how
we've
we've
been
able
to
manage
reliability
and
performance
at
extreme
scale
and
how
we've
been
able
to
reduce
our
root
cause
analysis
time
for
errors
and
keep
our
latency
very
low,
and
so
at
Google.
B
So
at
Google,
for
example,
our
equivalent
to
G
RPC
internally
is
called
stubby
stubby
effectively
just
collects
traces
automatically.
So
if
you
write
a
service,
even
if
you've
done
no
extra
effort,
you
can
get
distributed
traces
and
those
automatic
application
level
metrics
from
your
service
in
part
by
open
sourcing,
open
census,
which
is
kind
of
boring,
Lee,
just
called
census.
Inside
of
Google,
we
were
a
little
less
creative
with
the
name.
B
We
are
trying
to
expose
this
same
technology
to
developers
everywhere.
Obviously
we
have
you
know
an
offering
in
the
market
with
stackdriver,
but
to
be
blunt
like
open,
like
stackdriver,
is
not
the
reason,
we're
open
sourcing,
open
census,
it's
actually
funded
from
an
entirely
different
division
in
the
company.
It
is
mostly
just
so
weak.
B
Our
own
customers
can
be
more
successful
on
GCP,
but
also
so
that
customers
who
call
in
to
our
api's
and
want
to
get
traces
and
metrics
or
or
have
problems
with
those
api's
of
a
way
of
getting
it,
and
so
we're
trying
to
externalise
this
throughout
the
industry
so
that
getting
traces
and
metrics
becomes
really
ubiquitous
so
that
you
know
when
we
are
debugging
a
customer
issue,
or
even
other
cloud
vendors
or
other
api
vendors,
you're
debugging
customer
issues.
They
can
do
it
much
more
effectively.
A
A
A
So
when
you,
when
you
do
have
envoys
sitting
next
to
services
or
sidecars,
we
you
would
imagine
like
this
configuration
for
envoy
stats,
for
what
needs
should
be
measured.
So
whatever
the
metrics
are,
so
you
need
latency
error
count.
Those
are
configured
as
part
of
sto
configuration
which
drives
on
by
configuration.
You
can
get
application
level
as
well
as
on
roi
level,
tracing
into
an
ex
back
end
of
choice
through
exporters,
so
notice
that
you
can
get
in
process
application
level,
metrics
and
tracing
through
a
library
approach.
A
Whereas
for
you
know
in
Envoy,
that's
a
little
bit
trickier
to
do
that.
There's,
so
you
can
get
both
in
a
world
that
you
have
open,
sensor,
libraries
and
envoys.
The
other
piece
that
exists
in
is
tier
for
those
of
you
familiar
with
this
to
you
has
a
piece
called
mixer,
which
is
an
aggregation
point,
which
is
basically
also
the
policy
injection
point,
and
you
can
imagine
something
like
an
open
census,
adapter
to
mixer,
to
enable
sort
of
the
exporters
to
be
you
know,
twenty,
basically,
your
backends,
so
let's
say
you
use
Zipkin
or
Jager.
A
Those
can
be
integrated
through
a
mixer
adapter
mixer,
basically
is
a
central
service
for
those
familiar
with
this
do,
and
basically,
every
envoy,
which
is
sitting
next
to
every
service,
can
report
metrics
to
mixer,
which
can
write
to
the
opens
through
open
census,
adapter
to
the
tracing
back
noise.
That.
B
Has
a
couple
of
benefits
so
when
you're
using
mixer,
when
it's
aggregating
your
traces
and
metrics
with
open
census,
it's
nice
because
the
exporters
are
configured
in
mixers.
So
it's
you
know
you
configure
it
once
effectively,
but
also
all
of
each
of
your
notes,
isn't
sending
data
directly
to
your
back-end
they're,
sending
them
to
your
mixer
right.
B
So
you
don't
necessarily
need
those
nodes
connected
to
the
internet
if
you're
using
a
back-end,
that's
hosted
somewhere
else,
which
can
be
nice
for
security,
depending
on
how
your
applications
laid
out
and
you're
also
decentralizing
a
lot
of
traffic
through
one
place
longer-term.
It's
also
going
to.
B
Let
us
do
probably
some
pretty
interesting
things
where,
for
example,
we
might
do
like
subsampling
on
the
mixer,
so
you'll
collect
more
traces
from
each
nodes
tend
into
the
mixer
we
might
pick
and
choose
which
ones
are
more
interesting
to
then
get
sent
to
your
back-end
for
our
archival
purposes,
and
so
it
opens
up
a
number
of
really
cool
possibilities.
Once
we're
sending
this
data
to
the
mixer
aggregating,
it
they're
working
on
it
there
and
then
sending
it
to
back-end.
A
Yeah
and
and
you
could
do
aggregation
and
other
kinds
of
sort
of
post-processing
at
the
mixer
layer
and
then
the
other
thing
which
is
in
in
sto
you
application.
It's
not
fully
automatic,
like
the
application,
does
have
to
forward
headers
for
every
request
to
Envoy,
but
in
a
world
of
open
census,
libraries
they
could
do
that
job
instead
of
you
doing
their
job.
So
that's
another
sort
of
ease-of-use
things
when
they're.
A
A
Why
do
I
need
these
two
together,
so
there's
I
mean
there's
a
few
things
that
are
obvious,
which
is
a
bunch
of
features,
so
you
know
features
which
open
sensor
Springs,
which
is
the
/z
pages,
which
is
the
ability
to
create
custom
tags.
Those
are
just
features
that
don't
exist
without
open
census
right,
the
other
is
correlation.
So
a
big
value
in
tracing
is
being
able
to
correlate
that
for
a
given
request.
C
A
A
So
you
can
use
so
if
you
integrate
with
open
census,
you
get
app
level
instrumentation
and
tracing,
whereas
for
additional
spans
you
can
use
the
built-in
1y
tracer.
So
it
would
look
something
like
this
and
again
this
is
not
done,
but
planned
to
be
done.
You
would
have
an
open
census.
Library
integrated
with
your
actual
code,
probably
have
on
low
end
sitting
next
to
it.
If
it's
part
of
this
do,
and
each
of
them
can
be
sending
metrics
and
trace
data
to
the
back
end
that
you've
configured,
it
could
be.
B
A
Mean
just
wanted
to
show
what
it
looks
like
to
make
it
a
little
bit
real.
So
if
you
do
use
open
census,
so
there's
a
so.
This
is
a
simple
hello
world
code
for
using
open
census,
go
library
so
notice
you
just
all
you're
doing
is
just
basically
using
the
open
census,
your
PC
library,
you
are
right
and
exporter,
so
there's
a
concept
of
exporter
with
open
census,
which
is
where
do
I
want
my
metrics
and
traces
to
be
exported.
So
you
register
them.
A
In
this
case,
Prometheus
exporter
is
being
registered
for
metrics
and
stackdriver
exporter
is
being
registered
for
trace.
So
it's
a
simple
API
call
to
do
that
and
then
there's
a
bunch
of
predefined
metrics.
So
you
say,
okay
for
open
census,
I
want,
in
my
view,
metrics
I
want
round
trip.
Latency,
PU
and
I
want
error,
count
for
you
right
and
those
should
be
registered
and
that's
about
it
right.
There's
if
there's
a
whole
bunch
of
others
that
you
could-
and
this
is
the
one
that
gives
you
the
Z
pages.
A
So
there's
a
basically
starts
up
a
simple
HTTP
server
which
can
show
the
/zi
HTML
page
that
we
were
showing
in
the
slides
before
so
that's,
basically,
simple
open
senses
go
library
into
a
hollow
word
code.
Okay,
so
when
you
do
that,
you
actually-
and
in
this
case
we
registered
to
stack
driver
for
phrase
and
metrics.
A
B
A
Hello
method,
G,
RPC
method
on
client
and
server
so,
and
if
your
this
is
just
randomized
on
sort
of
request
sizes,
that's
why
these
it's
all
over
the
place.
But
these
are
graphs
you
can
get
sort
of
without
you
doing
any
work,
and
in
this
case,
and
it
could,
you
could
get
it
in
the
backend
that
you
desire
so
including
error,
count
round-trip
latency,
which
we
registered
in
our
code.
The
question
response,
bytes
and,
of
course,
counts
and.
B
A
Right
so
that's
metrics,
and
then
you
can
looking
at
tracing.
So
this
is
a
we
registered
stackdriver
trace.
So
this
is
looking
at
a
bunch
of
tracings,
so
you
have
your
first.
This
is
your
actual
trace
with
a
sort
of
one
span.
Just
you
sent
hello
world.
It
took
so
many
milliseconds,
so
many
bytes
and
it
received
and
what
so
often
this
is,
you
know
in
a
complicated
call
flow
graph.
A
This
is
becomes
an
extremely
useful
tool
to
say
which
service,
which
method
in
that
service
took
how
much
time,
and
where
is
the
bottleneck,
so
it's
it
becomes
a
very,
very
useful
tool
in
in
complicated
scenarios.
Also,
when
you
have
you
can
go
look
at
any
given
specific
requests
at
a
given
time.
You
can
look
at
simple
things
like
how
many
RPC
is
what
duration
it
was,
and
this
is
coming
from
open
senses.
Co-Agent.
A
B
We
we
have
a
lot
of
things
that
are
coming
forth,
both
for
open
census
as
well
as
open
census
with
sto.
So
we
we
look
at
sort
of
the
title
of
this
talk
is
do
an
open
census
and
the
things
that
are
focused
on
that
we're
completing
our
support
for
envoy
and
then
we're
gonna
be
making
some
pretty
big
investments
into
new
features,
that'll
be
accomplished
by
the
mixer.
B
So
that's
the
aggregation
and
and
perhaps
like
subsampling
another
things
that
I
talked
about
earlier
when
I
talked
when
we
think
about
just
open
census
on
its
own,
which
includes
open
census
and
on
boy,
but
just
the
open
census
really
running
in
any
environment.
We
have
a
lot
in
the
pipeline.
The
first
is
just
more
integrations.
So
right
now
we
have
you
know
Mike
MongoDB
and
a
number
of
other
databases,
we're
building
integrations
for
various
web
frameworks.
B
We've
I
think
we
already
shipped
G
RPC,
but
for
spraying
and
and
you
know
other
other
web
frameworks
across
various
languages,
so
you
can
get
those
automatic
traces
and
perform
correlations
completely
autonomously.
Next,
we're
also
finishing
our
library
support
across
all
eight
languages.
So
right
now
we
have
tracing
and
metric
support
for
c++
go
and
java.
B
Next
we're
longer-term
we're
gonna
be
focusing
on
client-side
instrumentation
for
mobile
and
web
apps,
so
today,
most
distributed
tracing
solutions,
focus
on
back-end
tracing,
so
tracing
requests
as
they
hop
between
your
micro
services,
but
that
doesn't
actually
give
you
a
good
visualization
of
client-side
latency
or
like
the
causes
of
the
latency
of
your
customers
are
experiencing.
For
example,
if
you
get
a
customer
who
complains
that
your
website
is
slow-
and
you
know
it's
just
not
fast
enough,
it
might
be
that
they're
just
on
a
really
slow
internet
connection,
in
which
case
there's
not
much.
B
B
B
Basically,
any
tracing
system
aren't
that
useful,
because
they're
describing
the
average
customer
request
so
useful
for
understanding
topology,
but
not
for
chasing
down
the
root
cause
of
an
issue
with
more
intelligent
sampling.
More,
the
higher
percent
of
your
traces
will
actually
be
interesting
and
actionable.
B
Finally,
our
big
long-term
ambition
for
open
census
is
it
for
it
to
be
the
library
that
you
use
to
capture
like
basically
all
of
your
applications
telemetry.
Today
we
do
traces
and
metrics
and
tags
longer-term,
we're
looking
at
other
types
of
telemetry,
whether
those
be
logs
profiles,
errors
anything
else
and
we'll
be
adding
those
into
open
census
as
we
go
forward.
I
think
that's
it.
B
C
B
B
So
there's
there's
a
number
of
things
that
we're
doing
so
just
to
ensure
that
the
trace
is
complete.
There's
the
trace
context,
working
group,
which
we
actually
kicked
off
about
a
year
ago,
just
to
make
sure
that,
like
any
third
party
services,
don't
strip
any
trace
headers,
though
in
terms
of
getting
metadata
out
of
that,
there's
a
few
different
opportunities
that
we're
looking
at
in
terms
of
maybe
propagating
some
kind
of
metadata
along
for
the
context
that
would
at
least
give
you
like
the
network
time
versus
like
the
compute
time.
B
For
example,
beyond
that
we've
been
sort
of
ping-ponging
around
the
idea
of
like
sort
of
standardized
API
is
for
tracing
that
could
send
data
like
sort
of
Web
API
is
they
can
send
data
to
a
back-end?
But
we
don't
have
anything
that
we're
like
working
on
like
right
now,
for
that
it's
a
pretty
tricky
problem,
especially
when
you're
dealing
with
so
many
different
vendors
of
different
appliances.
C
B
As
you
see,
adoption
of
like
open
census
through
other
cloud
vendors,
which
is
something
we're
working
on
at
least
when
you're
in
their
environment,
you
might
get
more
metadata
back
because
they
control
that.
But
in
terms
of
like
embedding
this
support
into
like
third
party,
like
switches
from
I,
don't
know
Cisco
or
someone
that's
a
whole
separate
conversation
yeah.
C
B
A
B
B
Yes,
so
the
question
was
about
resource
consumption,
for
open
census
and
just
in
general,
capturing
telemetry,
so
in
general,
the
the
resource,
consumption
of
open
census
is
effectively
zero
or
negligible.
This
is
part
of
the
reason
that
we
sample
traces,
for
example,
if
you
don't
sample
traces
that
becomes,
you
know
like
the
resource.
Consumption
becomes
more
obvious
and
more
measurable,
but
no
like
in
general,
open
census,
as
well
as
just
you
know,
speaking
just
from
from
Google
and
stackdriver
any
of
our
telemetry
collection
technology
aims
to
be
like
basically
no
impact.
B
Frankly,
technologies
like
this
aren't
necessarily
that
useful.
If
they're,
going
to
start
running
up
your
bill
or
making
your
application
perform
more
slowly.
Right,
like
the
goal
of
these
is
that
you
can
run
them
in
production
safely,
with
no
impact
and
like
for
reference
like
internally
at
Google.
We,
like
this
stuff,
is
all
there
and
it's
running
and
we
get
both
minimal
impact
from
it,
and
also
that
the
value
we
extract
is
huge.
So
it's
definitely.
B
B
That's
a
good
question:
yes,
so
open
tracing
defines
an
API
for
collecting
traces
from
and
there's
a
community
around.
That
open
census
defines,
of
course,
an
API
for
tracing
metrics
and
other
other
telemetry
and
also
has
the
implementation.
The
API
is
for
open
census
and
open
tracing
while
similar
are
not
identical.
As
many
people
often
point
out
I
believe
we
had
a
pull
request
from
the
community
that
added
Civ
not
changed
the
existing
open
census
API,
but
added
support
for
open
tracing
as
well
I
think
we
either
accepted
that
or
we're
about
to.
B
So
while
the
API
is
at
the
moment,
you
know,
we've
had
a
lot
of
conversations
with
ban
in
the
open
tracing
community.
At
the
moment
it
doesn't
seem
like
the
api's
are
going
to
converge,
but
we
do
have
compatibility
for
it
and
if
it's
I
I
don't
know
if
the
pull
request
is
like
done
or
not,
but
it's
in
review
and
it's
gonna
happen.