►
From YouTube: Pushing Envoy Beyond the Edge - JP Simard, Lyft
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Pushing Envoy Beyond the Edge - JP Simard, Lyft
Over the last decade, infrastructure has been moving away from monolithic centralized servers and increasingly towards end users, with a focus on Edge Computing to run code as close to the people who are accessing it, wherever they are in the world. Envoy Mobile has pushed Envoy beyond the edge, all the way to your fingertips, unlocking a world of possibilities (and challenges) by being able to run Envoy on every node in the network chain from app to service and back again. Come learn how we’ve adapted Envoy to run as native embedded libraries for iOS and Android that feel right at home no matter the platform ecosystem; how we narrowed and in some cases exceeded the performance gap with established mobile networking libraries; what mobile-specific use cases or problem areas we discovered along the way, and how we solved for them; how Envoy’s rich observability tools helped us roll out safely to billions of requests a day; and how this is just the beginning of pushing mobile networking to the next level.
A
All
right,
hello,
everyone,
it's
not
that
big
of
a
room
so
I'm
gonna
tone
down
the
mic
is
super
loud
too.
We've
been
hearing
so
far
and
you're
probably
going
to
hear
quite
a
bit
more
about
how
networking
is
moving
closer
and
closer
to
the
edge
right.
Get
closer
to
your
clients,
push
it
as
far
as
you
can
well
today.
I
want
to
talk
about
pushing
it
just
a
little
further
than
that
and
actually
running
Envoy
on
your
end
user
or
client
devices
themselves.
With
a
little
project.
That's
called
Envoy
mobile.
A
My
name
is
JP
I
work
at
Lyft
on
this
thing
called
Envoy
mobile.
So
the
idea
here
is
that
we
can
take
everything,
that's
great
about
Envoy,
that
you
know,
love
that
you
run
on
the
server
side
and
we
can
embed
it
to
run
it
in
mobile
apps
to
run
it
on
iOS
on
Android
and
to
run
it
to
power
the
networking
capabilities
of
those
apps.
A
A
It's
just
not
a
Target
platform,
and
one
of
the
main
differences
is
that
we
have
a
number
of
constraints
by
nature
of
running
in
these
highly
sandboxed,
highly
locked
down
mobile
environments
right
where,
as
an
application
developer,
you
ship
a
binary
to
your
end
users
and
that
ends
up
running
as
a
single
binary
single
process
most
of
the
time,
and
so
you
can't
just
run
your
networking
client
as
a
sidecar
and
spawn
a
process.
As
your
app
is
running
and
query
requests
off
to
it.
A
You
have
to
run
it
as
part
of
your
applications
process,
sharing
memory
sharing,
CPU
Cycles.
All
of
that
we
built
Envoy
mobile
as
a
polyglot
library,
and
what
I
mean
by
this
is
that
there
are
apis
for
various
languages
and
platforms
that
you
might
want
to
use.
We
have
apis
for
C,
plus
plus
Swift
kotlin
Objective
C
Java
python,
although
you
don't
really
run
that
one
in
apps
and
the
idea
here
being
that
we
want
to
make
the
experience
of
making
Network
requests
for
application
developers
as
idiomatic
and
Native
feelings
possible
on
those
platforms.
A
A
Why
is
this
useful
to
build?
Well,
there's
a
lot
of
what
you
probably
know
and
love
about
Envoy.
That
would
be
hugely
beneficial
to
bring
in
a
mobile
application
context,
and
so
that
was
sort
of
one
of
the
main
ideas,
and
one
of
the
things
that
Envoy
is
quite
known
for
is
its
Rich
set
of
observability
capabilities,
there's
just
a
rich
set
of
stats
that
you
can
emit
by
just
nature
of
using
the
library
and
what's
nice
about
this?
A
Is
that
especially
on
mobile,
where
you
have
very
limited
resources,
you
have
to
be
very
careful
in
how
you
instrument,
and
you
Monitor
and
You,
observe,
what's
happening
on
your
end
users
devices
they
might
be
in
a
constrained
environment
both
from
CPU
or
memory
perspective.
They
might
be
in
an
intermittent
connectivity
State.
A
Another
reason
is
that
Envoy
has
all
of
these
support
for
newer
networking,
Technologies
and
I.
Think
we're
actually
going
to
hear
some
talks
later
today
about
some
of
what
those
are
and,
and
some
of
the
recent
developments
in
Envoy
Upstream
there
and
what's
nice
about
this-
is
that
on
certain
platforms
like
on
Apple's
platforms,
the
networking
library
and
the
networking
engine
ships
with
the
operating
system?
A
So
if
you
go
back
and
you
target,
you
support
your
end
users
still
using
an
OS,
that's
a
few
years
old
they're
not
going
to
have
access
to
everything.
That's
happened
since
then,
and
what's
nice
about
being
able
to
ship
our
own
networking
library
with
our
apps
is
that
we
can
go
and
back
Port
those
and
support
all
of
our
users,
not
just
the
ones
on
the
bleeding
edge.
A
A
Debuggability
is
another
big
one
that
bring
Envoy
to
mobile,
gets
us
where
on
alternate,
Network
engines.
Those
are
closed.
Source
they
ship
with
the
OS.
You
can't
step
through
the
code
necessarily
to
understand
deep
down,
what's
happening
when
you
have
a
request,
that's
timing
out
or
you
have
something
that
you're
just
trying
to
observe
and
by
having
full
access
to
the
source
code
and
building
it
in
and
loading
it
into
the
IDE
so
that
you
can
step
through
the
code,
set
break
points
really
introspect
and
understand.
A
A
A
So
we
published
a
blog
post
on
lyftengineeringblogenge.lift.com.
You
can
read
more
about
it
there
and
to
give
you
a
sense
of
what
we
now
have
now
that
Envoy
Powers,
the
vast
majority
of
network
requests
in
the
Lyft
apps.
Is
we
get
these
real-time
dashboards
and
statistics
on
what's
happening
at
scale,
and
this
might
seem
very
simple
to
you.
This
is
table
Stakes
when
you're
building
any
sort
of
service
eye
product.
But
what
you
have
to
understand
is
that
the
mobile
landscape
is
years
behind
when
it
comes
to
this
kind
of
level
of
observability.
A
And
if
you
were
to
build
this
with
any
of
the
existing
networking
engines,
you'd
probably
either
be
losing
out
on
a
lot
of
data
or
you'd
be
degrading.
The
user
experience,
and
so
the
fact
that
Envoy
has
such
a
strong
focus
on
rich
and
optimized
observability
out
of
the
box
means
that
we
get
a
lot
of
this
stuff
more
or
less
for
free
and
then
stats
get
added
Upstream
in
the
main
Envoy
project.
We
can
take
advantage
of
those
and
and
add
them
to
our
observability.
A
So
I
want
to
paint
a
picture
here
a
little
bit
and
just
talk
about
the
road
to
Bringing
Envoy
to
mobile
and
the
the
obstacles
or
the
challenges
that
we
had
along
the
way
and
envoys.
This
highly
deployed
widely
used
product
right.
So
it's
it's
already
been
quite
heavily
optimized
and
works
quite
well
right,
so
you
would
think
you
just
plug
that
into
an
app
and
start
using
it
to
make
requests
and
everything's
beautiful.
A
Well,
it
turns
out
that
the
environments
in
which
mobile
applications
run
tends
to
be
quite
different
than
a
server
application,
so
to
paint
in
very
broad
Strokes.
Here
in
a
server
environment,
you
tend
to
have
a
very
stable
internet
connection,
you're
directly
wired
into
the
backbone.
You
have
a
wired
connection
to
the
network.
It
tends
to
be
quite
fast
if
it's
not
you'll,
probably
change
vendors
right.
You
have
control
over
who
is
providing
this
service.
If
you
don't
like
it,
if
there's
an
outage,
you
go
and
open
a
ticket.
A
You
work
with
your
provider,
you
work
through.
It
is
not
the
status
quo,
it
is
not
the
normal
level
of
operation.
You
tend
to
have
a
certain
level
of
reliability
that
you're
expecting
out
of
your
infrastructure,
and
you
have
control
right
where,
if
you're
on
a
provider
you're
on
premises-
and
you
don't
like
any
of
the
quality
of
service
that
you're
getting,
you
have
some
amount
of
agency
in
terms
of
moving
that
over
I'm,
not
saying
it's
simple,
but
you
do
have
that
level
of
control.
A
You
contrast
that
to
mobile
and
mobile
networks
are
inherently
unreliable.
Not
only
do
you
not
control
the
carrier
side
and
the
provider
side
of
it,
you
also
don't
control
where
the
device
is
physically
located.
You,
don't
you
don't
control
the
health
of
the
device
that
your
application
is
running
on
you
don't
control
how
how
constrained
it
is
you
you
have
a
wide
amount
of
variability
in
terms
of
the
quality
and
the
performance
envelopes
of
devices
that
your
users
are
running.
A
You
have
a
lot
of
carriers,
and
this
is
even
more
true
if
you
operate
outside
of
just
North
America,
which
is
primarily
where
Lyft
operates.
Your
users
can
configure
their
device
in
many
ways
and
you
don't
really
have
the
option
to
say.
Oh,
your
configuration.
Isn't
really
one
that
we
like
and
so
we're
not
going
to
support
you?
Everything
has
to
work.
A
A
The
top
is
around
98
success
rate,
and
what
this
is
measuring
is
the
success
rate
by
carrier
and
by
networking
library
that
we've
measured
with
one
of
the
lift
apps
over
the
span
of
30
days,
and
what
this
helps
illustrate
is
that
not
only
is
there
a
wide
amount
of
variability
across
carriers,
but
also
the
top
performing
device
or
operating
system
is
not
consistent.
Neither
is
the
top
performing
networking
Library,
and
this
is
helpful
to
make
the
point
that
we
don't
control
where
what
carrier
our
users
are
running
their
devices
on.
A
All
right,
so
we've
talked
a
little
bit
about
mobile
constraints.
Here's
some
more
devices
have
a
finite
amount
of
memory
that
you
are
competing
with
the
rest
of
the
application
for
right.
They
have
unreliable
connectivity.
We
touched
on
that.
There
are
many
carriers,
you
have
to
support
them
all
user
configurations.
You
can
install
a
HTTP
proxy
to
run
on
your
phone.
You
know,
VPN,
you
can
configure
your
DNS
settings
to
point
to
something
that
returns
garbage
data
half
the
time.
A
Users
are
frequently
moving
between
networks
and
between
interfaces
and
physical
radios
on
the
device
as
well.
Where-
and
this
is
especially
true
of
lyft's
use
cases
where
you
might
be
requesting
a
ride
as
you're
exiting
your
condo
building
entering
the
elevator,
leaving
your
Wi-Fi
right
and
those
requests
still
have
to
perform
as
best
as
they
possibly
can,
or
you
might
be
trying
to
unlock
a
scooter
between
large
buildings.
A
This
is
a
fun
one.
The
operating
system
can
kill
your
app
effectively
at
any
moment,
but
typically
when
it's
misbehaving-
and
generally
you
don't
want
to
be
the
reason
why
the
operating
system
is
killing
you
right.
So
we
need
to
make
sure
that
the
networking
layer
is
not
responsible
or
is
not
overly
hungry
for
these
resources.
A
Another
tricky
thing
with
with
mobile
deployments
is
that
you
can't
SSH
into
a
production,
environment
and
troubleshoot
why
something
might
not
be
working
well,
you
really
need
to
rely
on
your
observability
and
more
often
than
not
in
the
mobile
space.
That
just
means
that
you
have
very
little
insight
into
what's
happening
and
that's
yet
another
reason
why
the
observability
angle
of
envoys
hugely
helpful
to
us
from
the
developer
experience
point
of
view.
A
Another
concern
on
mobile
is
binary
size
because
we
ship
a
single
binary
to
all
of
our
users.
We
need
to
really
be
careful
how
much
heft
we're
adding
to
that
binary,
especially
if
they're
updating,
Weekly
right
and
you
don't
want
to
balloon
the
application
size.
Just
because
you
wanted
to
add
some
observability
into
your
networking
layer.
A
A
This
meant
that
if
you
were
trying
to
connect
to
an
ipv4
address,
those
users
just
couldn't
connect
at
all,
and
so
what
we
ended
up
figuring
out
that
we
needed
to
do
is
on
Android.
We
now
Force
IPv6
addresses.
So
if
it's
an
ipv4
address,
we
map
it
to
an
equivalent
IPv6
address
so
that
the
the
connection
can
be
made.
Otherwise
it
it
can't
even
connect-
and
this
was
certainly
not
something
that
we
intuitively
knew.
A
So
we
always
Force
IPv6
on
Android.
For
that
reason,
on
iOS
it
also
that
OS
also
has
its
own
quirks,
and
what
we
found
is
that
the
envoy
uses
quite
a
few
and
both
Envoy
and
Envoy
Mobile
use
quite
a
few
C
plus
plus
static
State,
C
plus
Constructors,
and
what
iOS
in
particular
does
when
an
application
is
shutting
down
and
it's
being
terminated
is
that
it
might
go
and
free
up
some
of
those
static
resources
which
means
that,
if
something's,
still
accessing
that
it'll
crash.
A
Necessarily,
it
was
adding
quite
a
bit
of
noise
to
how
our
mobile
device
developers
look
at
our
overall
observability
for
crash
rates
and
we're
saying
oh
well,
this
this
thing
isn't
stable
at
all
it's
crashing.
Well,
it
turned
out
that
it's
it's
fine.
It
just
happens
to
crash
when
you're
shutting
it
down,
and
you
know,
we've
been
working
through
some
of
these
one
by
one
guarding
against
some
of
the
state
that
might
disappear
But.
A
Ultimately,
it's
still
still
something
that
can
happen,
and
this
is
just
a
cost
that
you
have
to
pay
if
you're
working
in
any
sort
of
jvm,
interacting
with
with
native
code,
is
that
you
do
have
to
cross
the
jni
and
that
can
it
can
be
costly,
but
also
it's
just
cumbersome
more
often
than
not.
So
we
try
to
minimize
how
often
we
cross
that
barrier.
A
You
know,
50,
plus
Megs,
in
size
it
it's
it's
a
good
enough
trade-off
in
our
case,
at
least
for
Lyft
IDE
integration.
So
this
is
something
that
that
we
worked
on
with
some
open
source
projects
like
the
bazel
rules,
xcode
proj
set
a
rule
set
where
you
can
run,
build
and
run
Envoy,
and
you
have
full
access
to
all
of
the
sources.
A
A
c
plus
in
this
case
there's
some
Objective
C,
plus
plus
Objective
C
Swift.
You
can
step
through
those
break
points,
use
lldb,
and
this
has
been
extremely
helpful
in
terms
of
understanding
how
how
things
work
under
the
hood,
where,
if
you
do
run
into
an
edge
case,
that
you
try
and
understand
you
have
access
to
to
to
that
same
thing
for
Android
Studio.
A
So
some
of
the
things
that
the
team's
been
working
on
recently
so
earlier
I
mentioned
that
Envoy
mobile
Powers
98
of
lyft's
mobile
network
requests,
so
there's
a
two
percent
holdout
that
we're
keeping
there
part
of
that
is
to
continue
running
comparison.
Experiments
with
the
the
previous
networking
Stacks
that
we
were
shipping
with.
But
another
reason
is
that
we
do
have
a
number
of
gaps
where
we
still
don't
offer
full
support
for
certain
configurations
and
those
are
generally,
they
fall
into
two
buckets.
A
One
is
vpns
and
one
is
HTTP
proxies
that
users
can
install
and
run
on
their
phones.
So,
on
the
VPN
front,
we
recently
added
support
for
vpns
on
Android
this
required
moving
from
one
DNS
resolver
to
another,
that's
backed
by
a
different
system,
call
that
allows
us
to
operate
in
the
background
in
low
power
mode
and
and
supports
vpns,
so
that
that's
now
done.
We
have
work
in
progress.
A
So
one
of
the
main
remaining
gaps
that
we
still
have
is
HTTP
proxies
on
iOS.
A
lot
of
the
code
that
was
written
for
proxy
support
on
Android
will
be
portable
here
and
will
work,
but
there's
a
few
things
that
still
need
to
be
done,
such
as
parsing
and
interpreting
pack
files
and
within
the
next
few
months.
We
should
have
that
up
and
running,
so
this
is
really
the
last
big
piece
of
core
functionality.
The
last
main
rollout
Gap
that
that
currently
we
require
delegating
to
to
to
the
alternate
Legacy
stacks.
A
Another
thing
that
we
want
to
experiment
with
is
quick,
quick
and
HTTP
HTTP
3..
So
this
is
something
that
unformobile
does
support
at
the
moment,
but
we
haven't
run
any
production,
experiments
and
so
there's
probably
some
stuff
that
we'll
uncover
once
we
start.
Turning
that
on
and
running
some
production
experiments,
but
the
fact
that
Envoy
has
rich,
quick
and
HTTP
3
support
does
mean
that
we
do
get
a
lot
of
that
more
or
less
for
free
and
actually
that's
something
that
I
I
want
to
reiterate.
A
I'll
I'll
brush
through
this
pretty
briefly,
because
I'm
running
a
long
time,
but
at
a
very
high
level
what's
going
on
here,
is
that
you
have
different,
Frameworks
and
and
products
that
we
that
we
produce,
such
as
Frameworks
for
iOS
Frameworks
for
Android,
a
common
C
and
C
plus
layer
that
these
bindings
delegate
to
and
then
ultimately
that
calls
out
into
Envoy
and
recently,
we've
also
been
putting
more
love
into
not
just
the
Swift
and
kotlin
apis.
But
First
Class,
C,
plus
plus
apis
that
also
call
into
this
Common
Core.
A
So
mostly
three
you
can
think
of
it
in
three
pieces.
You
have
sort
of
the
user-facing
apis.
You
have
the
the
middle
core
Bridge
layer
that
then
calls
into
the
C
plus
native
code
and
Envoy
I
will
skip
through
this,
but
I'm
happy
to
dive
into
more
details.
I'll
just
stop
on
this
slide
real
quick,
because
what
are
the
ways
where
we
interoperate
between
kotlin
Java
Swift,
Objective
C?
A
So
that's
sort
of
how
we
can
communicate
across
languages,
and
this
illustrates
it
a
little
bit
where
you
have
the
envoy
engine
at
the
upper
left,
that's
configured
with
those
higher
level
languages.
That
goes
and
translates
things
into
an
Envoy
set
of
Concepts,
which
effectively
calls
back
into
that
middle
layer,
which
then
invokes
the
the
the
the
C
functions
that
are
passed
in
converted
into
top
level
language
lambdas.
B
A
At
the
very
top,
so
just
briefly
over
the
last
few
years,
there's
over
a
dozen
folks
who've
been
contributing
to
the
project.
I've
been
working
on
it
for
the
last
year,
but
ultimately
some
folks
have
been
working
on
it
since
long
before
that,
and
so
I
just
wanted
to
call
that
out.
We
have
contributors
from
within
Lyft
from
within
Google,
but
also
from
elsewhere
in
the
community
and,
of
course,
every
time
that
you've
contributed
something
to
Upstream.
A
B
A
Yeah,
that's
a
great
question.
So
to
repeat
the
context
is
from
the
perspective
of
a
room
full
of
server
Engineers,
how
what
sort
of
opportunities
does
this
unlock
from
from
a
server
perspective?
Well
for
one,
if
you
have
any
sort
of
expertise
within
your
your
server
team
in
terms
of
how
Envoy
Works
how
to
configure
filters
or
stats
that
domain
knowledge
applies
on
the
mobile
side
now
as
well.
So
your
networking
team,
your
server
networking
team,
can
now
collaborate
more
closely
with
your
client
networking
team.
A
If,
if
you
adopt
Envoy
mobile
and
you've
already
adopted
Envoy
on
the
server,
so
that's
one
another!
Is
that
like
I
mentioned?
If
you,
if
you
do
any
sort
of
optimizations
or
bug
fixes
in
code,
that
happens
to
be
shared
between
Envoy,
mobile
and
Envoy
proxy
on
the
server
side
that
ends
up
in
benefiting
mobile
clients
as
well.
So
in
terms
of
sharing
the
code
base,
that's
helpful.
A
I
mentioned
compatibility
and
consistency
earlier
where,
if
you
have
a
filter
that
operates
bilaterally,
then
it
you
can
be
running
the
same
filter
on
your
server
as
you
are
on
the
client,
and
you
have
a
high
degree
of
confidence
that
they're
going
to
be
compatible
because
they're
they're
using
the
exact
same
code
right
so
a
common
example.
There
is
compression
and
decompression
where
you
know
that
you
know
you're
unlikely
to
run
into
an
edge
case
where
one
broadly
compressor
compresses
things
one
way,
but
it's
incompatible
with
an
alternate
implementation
right.
B
Answer
it
as
well,
I
I
just
wanted
to
say
real,
quick
that
as
server
Engineers,
we
tend
to
be
very
excited
that
you
know
we're
99.99
success
rate,
but
the
reality
is
that
the
client
might
be
100
broken
right.
I
mean
that's
like
a
pervasive
problem,
so
I
just
wanted
to
add
that
as
server
Engineers,
we
need
to
care
what
is
happening
on
the
client,
because
we
can
be
serving
100
success
and
there's
broken
Json
or
something
and
the
client
is
a
hundred
percent
broken.
B
So
just
wanted
to
add
that
we
should
care
more
about
the
end
to
end
problem
here
and
I.
I.
Think
as
JP
was
saying
that,
with
the
shared
Library,
we
can
participate
more
right,
there's
just
more
there's
more
shared
knowledge,
because
I
think
historically
at
most
companies
there's
these
silos.
There's
the
client
people
and
the
server
people
and
I
think
that's
not
super
productive.
So
I
just
wanted
to
add
that
color.
A
No,
that's
some
great
color
matte
and
just
briefly
on
that,
what
we
discovered
is
when
we
do
look
at
server-side
success
rate
versus
success
rate.
From
the
perspective
of
the
client,
we
have
seen
some
surprises
there,
where
the
the
numbers
didn't
match
up,
and
so
we
had
additional
insight
into
where
to
invest
in
terms
of
bridging
that
Gap
I,
don't
know
how
we're
doing
on
time,
I'm
happy
to
take
more
questions.
A
Yeah,
okay,
so
the
question
is
that
the
audience
member
noticed
that
there
were
different
success
rates
across
different
networking
libraries
and
across
different
carriers
and
operating
systems,
and
in
some
cases
Envoy
mobile
is
underperforming
or
rather
URL
session
or
okay.
Cpu
is
outperforming
Envoy
mobile
because
of
the
observability
that
we
now
have.
We
are
able
to
discover
these
gaps
where
previously,
okay-
well,
maybe
we're
just
on
a
single
networking
engine
per
operating
system,
but
we
still
didn't
have
as
much
of
that
visibility
into
the
disparity
both
from.
B
A
Operating
systems
or
within
carriers,
or
sometimes
even
geographically,
where
carriers
end
up
sometimes
a
b
testing,
Network
configurations
geographically,
and
so
you
end
up
having
someone
in
Detroit
that
has
a
very
different
networking
experience
than
someone
in
Manhattan,
for
example.
So,
yes,
there's
still
some
areas
where
it's
some
specific,
some
specific
areas
where
the
alternate
networking
engines
are
outperforming
what
we
can
do
with
Envoy
mobile,
but
because
we
have
that
added
observability.
A
We
know
the
areas
where
we
can
go
and
dig
into
and
optimize
and
potentially
fix
some
issues
and
our
barrier
to
fully
rolling
out
at
least
to
98
percent
was
to
outperform
globally
right
or
to
match
or
outperform
globally,
and
so
we
looked
at
a
number
of
top-level
business
metrics
to
make
sure
that
we
weren't
taking
any
step
back
there.
Even
though,
in
some
specific
areas
we
might
be
underperforming
other
areas,
we're
outperforming
the
the
ideas
that
overall
we
we
didn't
take
a
step
back,
one
more
okay,
one
more
yes,.
A
Right,
so
the
question
is:
how
can
other
companies
go
and
perform
some
of
the
same
measurements
and
experiments
that
we
did
and
ultimately
this
is
one
of
the
reasons
why
we've
invested
in
offway
Mobile
is
because
of
the
observability
that
we
can
get
out
of
it.
So
in
many
ways
what
you're
asking
is,
how
can
you
get
the
same
level
of
observability
with
the
alternate
stacks
and
the
truth?
A
Is
it's
very
hard
to
to
accomplish
that,
so
what
we
ended
up
doing
is
we
knowingly
degrade
the
experience
for
some
users
on
the
alternate
networking
stacks
on
URL
session
or
okhgp
in
order
to
add
additional
instrumentation
right,
because
that
ends
up
competing
for
resources.
So
we
know
that
that
there's
a
hit
there
and
we
sample
a
small
percentage
of
user
traffic
that
we
randomize
in
order
to
compare
with
the
same
type
of
observability
overhead
on
Envoy
mobile
right,
because
you
can't
necessarily
compare
Envoy
Mobile's
optimize
stats
to
other
stats.
A
That
would
otherwise
be
adding
overhead
because
that's
not
comparing
Apples
to
Apples.
So
we
have
the
same
sort
of
heavyweight
instrumentation
on
both
ends
and
then
we'll
run
at
scale.
Public
experiments
and
sometimes
take
multiple
weeks
of
gathering
data
in
order
to
understand
how
both
perform
okay.
Thank
you
very
much
for
your
time.