►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Network Observability with Envoy - Nic Jackson, HashiCorp
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
The service mesh is built from two components the data plane and the control plane. The data plane handles the traffic from your service and is responsible for securing traffic, reliability patterns and network observability. The control plane is responsible for managing the configuration of the data plane, service catalog, x509 certificates, and service to service authorization. To effectively observe our system, we must understand the statistics emitted from these two components to build reliable and robust systems. In this talk, we will take a deep dive into the statistics output by key features of Envoy when used as a service mesh data plane. We will investigate network listeners, reliability patterns like timeouts, service discovery, authentication and how we can leverage these statistics to build a rich picture of the health of our system.
https://sched.co/Nrpg
A
Okay,
Neha
Java,
Oh,
Jackson
Nick
and
my
Chinese
is
terrible.
I
try,
I,
have
a
duolingo
subscription
I've
been
trying,
maybe
next
year
so
I
I
apologize,
but
it's
a
pleasure
to
be
here.
I,
always
love
coming
to
to
China
I,
especially
love
coming
to
to
kukuana
in
China,
and
it's
wonderful
to
see
you
all
and
today
I
want
to
talk
to
you
about
network
observability
with
with
envoy.
A
Wen
my
clicker
okay,
so
I'm
a
developer
advocate
I
work
at
a
company
called
hash
eco
and
I've,
been
playing
around
with
service,
mash
and
envoy
and
and
sort
of
the
whole
observability
sort
of
sphere
for
about
12
months.
Maybe
18
months
now
and
I
found
that
there
are
some
concepts
which
are
pretty
difficult
to
grasp.
A
My
background,
I've
used
metrics
for
sort
of
basic
statistics
for
about
five
years
and
tracing
is
a
slightly
different
concept
and
we'll
take
a
look
at
that,
and
very
very
briefly.
I
want
to
look
at
logging
and
that's
kind
of
more
so
why
this
is
actually
really
important
in
addition
to
all
of
the
other
stuff.
Alright.
A
A
We've
changed
the
way
that
we're
managing
services
when
I
running
kubernetes
were
running
multi-talented
were
running
smaller
services,
and
you
know
these
things
cause
problems
as
well
as
solving
problems.
How
do
we
get
traffic
east-west?
How
do
we
deal
with
things
like
network
perimeter
and,
most
importantly,
for
this
talk?
How
do
we
understand
what's
actually
going
on
inside
of
that
environment.
A
So
the
market
trend
has
been
this
move
right
and
I
wrote
a
book
on
micro
services
a
couple
of
years
ago
and
I'm
gonna
publish
a
new
edition
and
updating
that,
hopefully,
this
yeah,
but
one
of
the
things
that
that
I
see
is
a
real
benefit.
Is
this
movement
towards
smaller
functions
of
work,
smaller
instances,
smaller
application
instances
container
based
services
rather
than
huge
monoliths
and
there's
a
number
of
reasons?
A
So
the
benefits
of
this
as
I
kind
of
said,
is
that
productivity
thing
you
know,
micro-services
I
see
are
a
developer
tool.
The
the
benefit
is
is
to
to
us
who
are
working
on
the
systems
and
coding
them
more
so
than
it
than
a
business
tool,
but
there's
also
a
business
benefit
in
that.
If
we
can
ship
code
quicker,
we
can
get
benefits
to
the
end-users
quicker,
but
there's
a
cost
as
well,
because
you're
deploying
these
multiple
instances
of
services
do
you
need,
like
many
many
different
load,
balancers
and
different
components
of
em
structures.
A
I'm,
not
gonna,
go
into
all
of
that
today,
but
just
some
terminology
before
we
begin
just
in
case
anybody
is
kind
of
not
familiar,
but
so
I'm
gonna
be
talking
about
envoy
and
envoy
as
a
component
of
a
service
match
so
composer
service
mesh
generally
built
into
two
two
main
components:
a
centralized
and
distributed
control
plane
and
the
data
plane.
So
the
data
plane
is
where
all
of
your
traffic
is
flowing.
Traffic
doesn't
flow
independently
from
a
service
instance
to
another
service
instance.
A
A
A
A
So
observability
comes
from
control
theory,
it's
an
engineering
term
in',
and
it's
kind
of
a
measure
of
how
well
internal
states
of
a
system
can
be
inferred
from
knowledge
of
external
outputs
like
what
well.
What
we're
trying
to
say
is
that
if
we
measure
internal
inputs
and
outputs,
we
can
determine
an
overall
health
of
a
system
based
on
those
we
can
infer.
A
And
what
does
it
encompass
so
observability?
This
is
kind
of
the
key
thing
for
me
that
observability
is
not
just
about
metrics
or
envoy.
Statistics
on
the
observability
encompasses
all
of
this,
so
things
like
your
Envoy
statistics,
application
statistics,
kubernetes
statistics,
tracing
logging,
health
checks,
business
analytics
right,
don't
don't
forget
this,
but
if
you
think
about,
if
you
want
to
measure
the
performance
of
a
system
based
on
its
external
inputs,
then
what
about
sales
like
if
a
system
isn't
functioning
correctly,
you're
gonna,
see
decrease
sales
potentially
or
decrease
traffic
or
customer
interaction?
A
Those
things
are
business,
metrics,
they're,
not
necessarily
system
properties,
but
observability
tries
to
encounter
for
all
of
those.
Well
we're
going
to
be
looking
at
is
just
a
small
part
of
that
and
we're
going
to
look
at
envoy
statistics
and
tracing
mainly
so
again.
We've
got
when
we're
thinking
about
observability.
We've
got
to
think
about
internal
and
external
instrumentation,
so
we're
gonna
be
thinking
about
things
like
probes,
they're,
on
Nagios,
looking
at
disk
space.
All
of
these
affect
the
behavior
of
the
system.
We're
gonna
look
at
things
like
health
checks.
A
A
So
when
you
want
to
understand
matrix
with
envoy,
you've,
you've
really
got
to
first
understand
the
architecture
of
envoy,
because
if
you
look
at
the
documentation,
olive
oil,
it's
just
thousands
and
thousands
of
different
metrics,
and
where
do
they
come
from
right?
So
there's,
ultimately
these
five
component
parts.
So
we
have
things
like
listeners,
listener
filters,
rudos
clusters
and
ultimately,
the
control
plane
so
bless
break
those
down.
A
A
It's
gonna
have
a
minimum
of
sort
of
to
form
for
the
control
plane
and
one
for
the
internals,
but
this
is
kind
of
a
another
key
term
and
differentiating
between
the
understanding
of
what
is
downstream
and
what
is
upstream
so
downstream
is
a
response
and
a
request
which
comes
from
an
end-user
client,
so
an
external,
so
I
could
be
a
downstream
source
when
I'm
making
a
request
to
your
website.
Another
service
can
be
a
downstream
root
source.
A
This
is
something
which
is
made
to
another
service
and
it's
important
to
understand
the
difference
between
these
two
when
we
start
getting
into
matrix,
because
a
failure
in
an
upstream
doesn't
necessarily
mean
a
failure,
response
code,
return
to
your
downstream
service,
customer
or
user,
and
that's
because
of
the
reliability
patterns
which
are
potentially
implemented
somewhere
in
the
filters
there,
a
cluster.
What
a
cluster
is
in
terms
of
envoy,
it's
a
collection
of
endpoints.
A
So
we're
going
to
look
at
this
when
we
start
looking
at
some
of
the
metrics,
but
the
key
thing
is
around
observability.
You
need
to
understand
from
where
a
metric
or
a
statistic
comes
from
right
and
being
able
to
add
additional
bits
of
metadata
to
a
metric
is
incredibly
beneficial
when
you
start
to
build
up
your
your
dashboards
and
your
alerts,
the
other
important
and
I
think
absolutely
essential.
A
Configuration
feature.
Is
this
one
here
which
is
using
all
default
tags?
Because
if
you
look
at
the
role
metrics
or
the
raw
statistics
in
terms
of
Envoy,
they
can
be
incredibly
long.
You'll
have
things
like
HTTP
listener,
192
168
underscore
8080
dot
downstream,
underscore
C
X
underscore
blah
blah
blah
blah
right.
A
It
becomes
incredibly
difficult
to
manage
those
and
build
up
dashboards
when
you
have
sort
of
dealing
with
something
as
complex,
whereas
if
you
use
this
configuration
element,
what
will
happen
is
envoy
is
going
to
extract
things
like
the
the
listener
name
or
the
cluster,
and
it's
going
to
put
it
into
a
tag.
So
it
makes
selection
of
metrics
much
much
easier
where
we're
going
to
kind
of
go
with
with
metrics.
Something
again,
something
important
to
to
point
out
is
that
one
boy
has
Prometheus
metrics.
A
It
has
a
the
ability
to
expose
a
Prometheus
endpoint
metric
same
endpoint,
but
there's
a
couple
of
caveats
with
it.
So
you
really
need
histograms
and
they
are
only
available
in
envoy
1.10
and
the
other
key
thing
which,
when
I
last
checked
the
changelog
I,
don't
think
it's
a
change
in
110
is
that
the
Prometheus
metrics
are
exposed
on
the
admin
endpoint.
A
There's
a
couple
of
ways
around
that
you
can
sort
of
set
an
internal
envoy
route
just
to
to
kind
of
loop
back
and
expose
only
that
specific
Prometheus
metrics
point,
but
I
believe
that
that's
going
to
be
either
kind
of
modified
or
the
admin
endpoint
in
or
vuoi
is
going
to
be
an
authenticated
endpoint
at
some
point,
so
using
metrics,
so
you've
got
a
number
of
different
ways
in
which
you
can
kind
of
emit
metrics
from
on.
For
you,
the
simplest
is
stats.
A
D
stats
knees
been
around
for
a
long
long
time,
originally
created
by
Etsy.
It's
a
push
based
metrics
format,
which
use
the
UDP
protocol.
The
key
thing
around
stats
D
is
it
doesn't
support
metadata
and
we're
going
to
kind
of
look
at.
Why
that's
important
you'll
find
that
whichever
sort
of
metric
system
you're
using
they
tend
to
support
these
kind
of
four
things
of
counters,
gauges,
timings
and
and
sets
predominantly
the
first
three
of
the
most
the
most
used,
and
it
kind
of
looks
like
this.
A
A
A
So
now
what
you
can
start
doing
is
you
can
add
metadata,
so
I
can
start
building
up.
You
know
my
service
or
service
a
dot.
My
method
called
tags
and
I
can
start
putting
the
service
ID
into
the
metric
and
that's
really
important.
Because
again
you
need
to
be
thinking
about.
How
do
you
filter?
How
do
you
break
down
information?
It's
it's
generally,
not
an
entire
system
of
services
which
which
you're
going
to
cause
have
problems.
It
can
be
particular
instances.
A
Prometheus
great,
open
source
for
eight
CN
CF
project
takes
a
slightly
different
approach
so,
rather
than
you
pushing
metrics
to
the
service
with
Prometheus.
What
Prometheus
will
do?
It
will
pull
the
metrics
from
your
your
application
and
that
pull
is
done
over
HTTP
and
importantly,
it
supports
metadata.
A
So
again,
those
key
things
there
counters
simple
things
like
number
of
requests
want
to
see
that
break
down
over
time
gauges.
What
is
the
current
CPU
valve?
You
know
levels,
something
which,
which
is
kind
of
a
fixed
value
and
in
histograms
his
friends,
really
really
essential
for
being
able
to
deal
with
timing
data
because
again
I
don't
want
to
be
looking
at
averages.
A
I
need
to
be
looking
at
at
things
like
quartiles
and
percentiles
inside
of
my
my
data
matrix
format,
pretty
simple,
straightforward
again:
here's
an
example
of
of
a
non-void
metric,
and
you
can
see
here
with
envoy
HTTP
connection
manager.
Prefix
I've
got
the
name
of
my
connection
manager.
Now,
if
I
wasn't
using
tags,
that
would
have
been
part
of
the
name,
so
that
use
tags
is
a
really
really
important
feature.
A
So
all
those
things
when
you're
choosing
a
format,
you've
kind
of
got
to
sort
of
take
into
all
of
those
things
into
account.
If
you're
using
Prometheus,
which
I
guess
a
lot
of
people
do,
then
you
know
you
can
kind
of
just
go
straight
with
with
Prometheus
but
think
about
how
you're
accessing
those
metrics
and
that
you're
accessing
them
in
a
secure
way
because
of
the
admin
endpoint
and
envoy.
But
ultimately
you
know
you
can
also
send
data
out
to
dog
stats,
D,
Zeta
dog
and
things
like
that.
A
But
you
need
to
be
thinking
about
metadata
and
metadata
needs
to
be
more
than
just
the
service
instance.
You
potentially
want
to
be
able
to
capture
things
like
the
kubernetes
know.
Do
you
want
to
be
able
to
capture
the
pod
name,
the
deployment
sort
of
ID
as
much
information
as
you
can,
which
will
help
you
in
the
event
that
you
need
it?
A
So
things
like
downstream
connections
and
I've,
just
kind
of
pulled
out
some
of
the
key
key
elements
here
that
there
are
a
huge
number.
But
but
these
are
some
of
the
things
that
I
like
to
to
kind
of
monitor
and
again
don't
forget
that,
because
I
need
to
be
able
to
get
that
metadata,
what
I
want
to
kind
of
build
that
up
into
pruning
Thea's
query.
A
So
I
can
look
at
something
like
the
connection
total,
so
I'm
gonna
about
that
a
counter,
so
I
want
to
be
able
to
see
maybe
a
bar
chart
or
a
graph
of
of
the
number
of
requests
with
with
count
as
I
tend
to
like
to
see
those
rather
than
a
rate.
I
want
to
see
the
number
of
requests,
so
I
tend
to
use
increase
instead
of
a
rate
there,
but
I'm
an
either
way
breaking
it
into
a
thirty
second
bucket.
A
If
you're
seeing
a
lot
of
connections
created
and
destroyed,
it
can
be
an
indication
that
potentially
there
is
something
in
the
system
which
is
not
allowing
persistent
connections.
Persistent
connections
gives
you
speed.
That's
why
we
kind
of
use
connection,
pooling
and
decision
connections
there's
also
some
some
key
metrics
around
connections,
which
I
think
are
really
good
for
alerts
and
the
kind
of
the
key
one
which
I've
highlighted
here
is
sss
ssl
fail
verifying
o
cert.
A
A
A
So
what
about
requests
so
requests
you've
got
to
be
thinking
about
layer,
seven
and
again
we're
gonna
break
up
request
into
downstream
and
upstream
and
remember
that
difference
a
downstream
is
to
me
back
to
your
your
end,
consumer.
The
upstream
is
something
which
is
happening
internal
in
an
envoy.
So
a
downstream
sorry,
an
upstream
failure
does
not
necessarily
result
in
a
downstream
failure
because
of
the
retry
patterns
and
the
reliability.
But
downstream
is
really
important
because
that's
what
your
end
user
sees
failures
on
the
downstream
means
at
your
and
user
is
suffering
again.
A
A
So
that
gives
me
the
capability,
because
it's
extracted
that
response
code
as
a
tag,
a
non-void
understands
the
HTTP
protocol,
so
it
understands
HTTP
response
codes
and
I
can
see
there
that
Envoy
will
inject
the
tag
response
code
class
and
it
allows
me
to
build
up
a
differentiation,
so
I
can
say
well
I'm
only
interested
in
errors
or
I'm
only
interested
in
things
which
are
on
not
errors,
because
a
404
is
in
an
hour,
probably
not
a
4
or
4
could
be.
You
know
it's
not
an
internal
error.
A
Looking
at
those
kind
of
things
on
a
chart,
I
can
actually
plot
them
and
I
can
stack
them
as
a
bar
if
I
want
and
I
can
kind
of
seen,
the
the
various
different
response,
codes
or
I
can
split
them
up,
but
I've
got
that
that
flexibility
to
be
able
to
do
that
now
request
errors.
This
go
hand
goes
without
saying,
but
this
is
really
really
important
to
be
monitoring,
request
errors
and
you
should
potentially
have
alerts
as
well.
Around
request
errors.
I
know
it
happens,
but
it's
something
which
is
undesirable.
A
So,
for
example,
if
I'm
monitoring
my
request,
errors,
I
start
a
new
version
of
my
pod
and
all
of
a
sudden
I
go
from
zero
requests
errors
to
a
whole
bunch
of
them
now
I
can
configure
an
alert
on
that
which
is
looking
at
a
percentile
increase.
That's
giving
me
early
warning
that
the
new
version
that
I've
just
deployed
is
not
working
as
as
requested
timing.
Timing
isn't
is
incredibly
important.
We
want
to
be
looking
at
things
like
timing
and
timing.
A
But
what
about
G
RPC?
So
GRP
sees
HTTP,
write,
HTTP
yeah,
but
it's
not
well,
it's
kinda.
It
uses
it
as
the
transport,
but
Jia
PC
doesn't
honor
things
like
HTTP
status
codes
with
ERP,
see
what
you're
doing
is
you're
encoding
the
status
code
into
the
the
protobuf
response.
So
when
it
comes
to
building
your
matrix,
I
could
have
a
failed
request,
which
is
an
internal
server
error.
A
So
to
enable
that
it
does
have
to
be
specifically
enabled
we
use
this
feature,
which
is
an
HTTP
filter
in
envoy,
and
it's
the
HTTP
one
bridge
filter.
Now,
if
you
look
at
the
documentation,
the
documentation
is
is
pretty
clear
on
what
this
thing
does,
but
when
it's
easy
to
misunderstand
so
a
lot
of
people
look
at
the
HTTP
one
bridge,
filter
and
say:
well,
it's
a
method
of
translating
an
HTTP
request
into
a
G
RPC
request,
and
it
does
that.
A
But
it
also
allows
envoy
to
understand
response
codes,
the
methods
that
the
encoded
data
inside
of
the
protobuf
and
to
report
that
into
your
statistics,
so
I
can
use
things
like
this
and
again,
you
can
kind
of
see
here.
I've
got
envoy
closer
to
your
PC.
So
this
is
a
metric,
a
statistic
and
then
I've
got
the
G
RPC
response
code
so
zero.
That's
it
that's!
The
equivalent
of
an
HTTP
200
I've
got
a
response
code
five
here.
This
is
kind
of
the
equivalent
of
a
404
and
G
RPC
status
codes.
A
But
again,
building
that
up
into
to
a
chart
I
can
build
something
which
is
really
really
rich.
I
can
filter
on
that
status
code,
but
I'm
also
seeing
the
method
call
so
I
can
actually
isolate
those
independent
requests,
which
is
really
really
nice
and
useful
to
see
I've
just
lumped
them
all
into
a
single
chart
here,
but
you
can
kind
of
break
them
up
as
you
as
you
need
to
errors
again,
I'm
using
a
different
metric
butter
in
the
same
way
as
I'm
dealing
with
HTTP
requests.
A
We
can
kind
of
see
here
that
we've
we've
got
some
some
retries
and
we've
got
some
arrows.
I've
applied
the
the
retry
policy,
so
I
can
see
that
envoy
again
is
is
putting
my
my
retries
there.
Now
these
retries
wouldn't
result
in
a
downstream
error,
but
they
are
manifesting
themselves
as
an
upstream
error.
A
Timeouts,
we
need
to
monitor
timeouts.
So
a
timeout
in
the
system
is
a
big
red
flag.
I
tend
to
put
alerts
on
on
timeouts
and
whether
you
have
that
go
straight
through
the
pages
UT
or
not
like
that.
That's
up
to
you,
but
I
think
a
timeout
was
a
big
red
flag
and
you
need
to
be
to
be
carefully
sort
of
monitoring
this.
This
information
and
if
you
kind
of
implementing
those
reliability
patterns,
you've
got
to
be
thinking
about,
like
monitoring,
outlier
injection,
each
action.
So
what
does
outlier
rejection
do
and
like
as
a
hatten?
A
If
envoy
receives
a
number
of
status
codes
from
an
endpoint
which
results
in
a
failure,
it
will
remove
that
endpoint
from
the
cluster
temporarily.
So
here,
I'm
monitoring
those
endpoints
have
been
ejected.
I
might
have
one
particular
service,
which
is
a
little
bit
sort
of
flaky,
but
envoy
will
remove
it
temporarily.
Then
you
can
see
the
gap
because
envoy
is
gonna.
Try
it
again.
A
A
Let's
see
so
the
way
that
kind
of
the
the
authorization
work
so
can
two
components,
authentication
and
authorization
authentication
is
done
through
the
an
TLS
process.
Client-Side
certificates,
Luther
ization,
is
a
facet
of
the
control
plane.
So
it's
an
additional
layer.
It's
I
have
this
presented
entity.
Is
it
allowed
to
connect
to
me
this
is
a
metric
that
you
really
want
to
be
monitoring
inside
of
your
control
plane
because
you
kind
of
gotta
be
thinking
about.
Why
would
you
get
these
failures?
You
should
never
really
get
a
lot.
A
A
A
Authorization
is
going
to
slow
down
your
service,
so
you
don't
want
to
see
many
authorization
requests
at
any
period
of
time,
authorization
failure.
This
is
a
massive
red
flag,
so
you've
got
to
be
thinking
about
authorization
failures.
Why
are
you
getting
those
there's
two
kind
of
predominant
reasons.
The
first
one
is
that
you've
got
miss
configuration,
so
you
haven't
actually
explicitly
allowed
service
communication
between
two
services
or
it
could
be
somebody
who's
probing
around
in
your
system
and
trying
to
do
something
that
they
shouldn't
an
aunty
tracing.
A
So
tracing
is
interesting,
I'm
kind
of
growing
in
my
appreciation
for
it,
as
I
said
it's
something.
That's
different
to
me
and
I
just
took
this
from
the
open
tracing
website,
but
it's
a
nice
description
and
the
thing
that's
interesting
about
tracing
is
when
you're
looking
at
a
formants,
so
the
configuration
of
tracing
an
envoy
requires
that
you
set
up
a
cluster
and
you've
got
to
set
up
the
cluster,
because
envoy
needs
to
know
where
it
is
going
to
send
the
the
tracing
data
to
the
spans.
A
Your
control
plane
should
handle
this
for
you.
But
it's
interesting
to
know.
I
think,
and
you
also
need
tracing
configuration
because
there's
a
number
of
different
drivers
which
envoy
can
use
it
can
use
a
zip
contriver
which
is
kind
of
open
tracing.
There's
some
new
stuff
in
110,
which
allows
pluggable
open
tracing
and
it'll
also
support
open
census.
A
I
believe
you've
got
light
step,
but
when
you
start
to
look
at
a
tracing,
so
here's
an
example
of
an
HTTP
POST
I
can
actually
see
visually
the
difference,
kind
of
the
paths
or
the
different
sort
of
network
hops
that
have
gone
through
my
service.
So
I've
got
internal
from
the
in
downstream
to
an
upstream
listener
and
then
I've
got
a
G
RPC
service.
A
What
you've
kind
of
got
to
be
reticent
of
when
you're
thinking
about
tracing
is
that
an
external
call
all
tracings
have
a
parent-child
relationship
and
if
this
is
an
example
for
Zipkin,
but
you've
got
to
be
thinking
about
forwarding
the
headers,
because
if
you
receive
a,
if
you
make
an
internal
call
from
your
code,
you've
got
to
forward
on
spine
IDs
and
the
tracing
IDs
for
it
to
be
registered.
As
part
of
that
that
graph.
A
The
open
tracing
in
a
G
RPC
there's
some
great
frameworks
which
which
work
as
middleware
to
help
you
do
this,
but
essentially
that's
what
they're
doing
they're
taking
the
parent
information
and
they're
adding
it
to
the
child
span
in
order
for
that
that
nice
chart
to
work.
Lastly,
on
logging,
why
is
logging
useful
if
you've
got
all
of
these
wonderful
metrics?
A
Well,
when
I
was
kind
of
putting
together
a
presentation
and
working
on
my
demo,
I
couldn't
get
my
tracing
to
work
and
my
metrics
weren't
telling
me
anything,
but
when
I
looked
in
my
my
logs
I
could
clearly
see
that
my
logs
I
wasn't
sending
the
spans,
so
you've
got
to
think
about
all
of
those
things.
Envoy
is
gonna.
Give
you
some
great
information
around
access
logs.
A
A
A
Yeah,
so
what
what's
my
preferred
way
to
send
tracing?
Is
it
directed
by
the
sto
mixer,
so
I
think
it's
still
mixer
and
I
correct
me
on
this,
but
I
believe
there's
gonna
be
some
changes,
because
when
you
send
everything
through
with
your
mixer,
one
of
the
performance
bottlenecks
in
sto
is
is
with
mixer
and
I.
Believe
that
there's
going
to
be
a
separation
from
mixer
as
a
centralized
component
in
an
up
and
coming
release,
so
you
can
absolutely
send
them
direct
to
Zipkin
or
you.
A
A
So
these
are
the
sorry.
This
is
a
little
bit
misleading,
I've
I've
actually
named
the
G
RPC
methods
put
Katan
exists
because
the
the
service
in
question
here
is
is
actually
storing.
Storing
some
data
or
it's
retrieving:
yeah
yeah.
Sorry,
not
HTTP,
yeah,
yeah.
Sorry
in
retrospect,
they
those
are
terrible
names,
but
no,
they
are
they're.
Actually
yup.
We
see
methods,
not
not
HTTP
methods.