►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Lyft Used Envoy to Rethink Microservice Development - Matt Grossman, Lyft
To scale Lyft’s engineers' productivity, the Lyft developer experience team pivoted away from costly and hard to maintain custom individual environments, and rebuilt the development environment around a shared staging environment.
The goal was to enable Lyft developers to run multiple versions of the same service (both in Kubernetes and on their laptop) without stepping on each other's toes. To do this the team designed a scriptable ingress proxy as well as used custom Envoy filters within Lyft’s service mesh to route traffic to these isolated instances, inject and propagate custom metadata, and offload the traffic to custom developer tools. The end result gives engineers special networking debugging superpowers during development.
In this talk, you’ll follow Lyft’s progression designing and developing these different components. You’ll deep dive into the custom Envoy filters and how they combine tracing, Original DST clusters, a custom xDS control plane, and local tooling to build this new developer experience. We will also cover realized benefits to Lyft’s engineering productivity as well as problems encountered along the way.
A
Thank
you
everyone.
My
name
is
Matt
Grossman
and
I'm
here
from
Lyft
to
talk
about
how
we
have
used
Envoy
to
rethink
microservice
development,
so
I,
actually
I
don't
work
with
JP
over
on
Envoy
mobile
I.
Don't
work
with
Matt
on
other
parts
of
envoy
I'm,
actually
part
of
the
developer
experience
org
at
Lyft
our
goal:
we're
we're!
Not
networking
we're,
not
observability.
A
So
if
we
Rewind
by
like
two
years,
this
is
what
our
change
pipeline
used
to
look
like
when
a
developer
had
some
modification
that
they
wanted
to
get
out
to
prod
in
development.
Each
user
would
have
their
own.
What
we'd
call
a
one
box,
which
was
basically
just
a
very
large
ec2
instance,
and
this
would
run
the
entirety
of
the
lift
system.
They
would
change
whatever
amount
that
they
want
in
their
services
is
completely
sandboxed
in
when
they
were
done
and
they
felt
confident
in
their
change.
A
They
would
create
a
pull
request,
get
that
plus
one
by
a
teammate
and
then
take
it
out
to
our
staging
and
production
kubernetes
clusters,
so
notably
the
development
and
staging
and
production
run
on
very
different
Stacks.
Here
development,
our
one
box
is
totally
separate
versus
our
kubernetes.
Staging
production
were
quite
similar
other
than
just
like
a
couple
changes,
of
course
on
with
the
data
flowed
like
Etc,
so
just
to
zoom
in
a
little
bit
more
about
what
one
box
was
like
I
said,
really
large
ec2s.
A
A
That
worked
for
a
really
long
time,
especially
as
we
moved
from
a
monolith
to
microservices
at
first.
We
didn't
have
that
many,
but
eventually
we
ran
into
problems
one
of
the
problems
with
this
type
of
development,
where
each
user
got
their
own
ec2
was,
it
would
scale
by
End
by
services.
So,
of
course,
this
lift
grew.
We
got
a
lot
more
Engineers,
which
means
we
needed
a
lot
more.
One
boxes
to
give
out
to
everyone.
A
A
Another
large
bomb
that
we
ran
into
with
one
box
was
like
I
said
it
was
very
different
than
our
kubernetes
staging
and
production
environments.
So
we
have
these
issues
with
environment
drift
where
something
would
work
great
in
development,
but
then
it
would
get
deployed
out
to
staging
or
production
and
we'd
run
into
some
sort
of
strange
issues
due
to
the
system
that
was
built
on
top
of.
A
Sometimes
the
other
problem
would
happen
too,
where
someone
would
put
out
a
change
not
realized
until
later
that
it
was
broken
in
development,
and
then
this
caused
issues
with
unclear
ownership.
So
our
team,
the
developer
experience
team
owned
one
box
so
when
something
would
go
wrong
in
development,
but
it
was
working
in
other
environments,
they
would
kind
of
point
fingers
at
us,
but
we're
not
experts
on
every
single
person
service.
A
So
we
would
kind
of
have
to
rely
on
the
service
owners
and
they,
of
course,
wouldn't
really
feel
responsibility
themselves
because
they
weren't
sure
what
they
did
wrong
and
it
seemed
working
to
them.
So
a
combination
of
these
changes
or
these
problems
is
what
led
to
this,
unfortunately,
normal
workflow-
that
a
lot
of
developers
would
end
up
doing.
First,
they
would
provision
one
of
these
one
boxes.
They
would
run
a
command
by
sshing
into
it.
That
would
spin
up
a
whole
bunch
of
different
containers
and
oftentimes
that
could
take
over
an
hour.
A
Sometimes
some
of
those
container
starts
would
fail,
so
they
would
end
up
having
to
reprovision
it
and
then
eventually
they
would
just
give
up
and
deploy
and
test
and
staging
where
they're
a
lot
more
confident
that
things
worked
as
expected.
This,
of
course,
was
a
problem.
This
is
not
a
great
developer
environment
and
because
of
that,
we,
you
know
decided.
We
need
to
kind
of
go
back
to
the
drawing
board
and
figure
out
a
better
arrangement.
A
So
that's
what
led
us
to
the
solution
where
we
are
sharing
a
developer
environment.
So
in
that
last
step
that
I
said
what's
so
wrong
with
deploying
to
staging
if
we
ended
up
doing
that,
staging
is
not
so
bad.
It's
a
it's
a
real
environment,
it's
very
similar
to
production
staging,
has
a
bunch
of
great
advantages.
Actually,
it's
like
I,
said
very
production-like.
A
They
would
get
paged
about
it
and
you
know
if
they
would
peeve
a
bunch
of
people,
so
of
course
they're
going
back
and
fixing
it.
So
those
are
advantages,
but,
of
course,
staging
also
had
some
problems.
If
you
wanted
to
use
it
as
a
developer
environment,
one
of
them
was
like
I
said.
A
broken
service
in
staging
might
mean
many
transitive
dependencies
are
also
broken.
They
you
deploy
something
bad.
A
If
teammates
wanted
to
collaborate
and
work
on
the
same
service,
but
maybe
they
had
their
own
branches
that
they
wanted
to
test,
it
wasn't
super
straightforward.
Perhaps
they
would
have
to
kind
of
take
turns
and
say:
hey
I'm
testing
this
in
staging.
Can
you
wait
an
hour?
I
gotta
wrap
this
up,
make
sure
it
works
as
expected,
and
so
of
course,
that
slows
down
Edge
velocity
and
then
lastly,
deploying
something
to
staging
was
kind
of
more
complicated
than
testing
something
in
development.
A
That
gives
us
a
lot
of
the
advantages
of
a
shared
environment
for
development,
but
it
fixes
a
lot
of
the
problems
that
we
had
with
isolation,
so
that
was
our
overall
goal,
and
this
is
kind
of
like
a
very
high
level
picture
of
what
staging
overrides
looks
like
first
users
make
requests
to
the
core
staging
environment,
but
they
have
their
own
instance.
What
we
call
an
offloaded
pod
that
contains
just
their
changes,
so
when
they
are
done
with
their
changes,
they
don't
have
to
fully
merge
it.
A
They
just
need
to
get
it
up
and
make
a
container
image
build
from
it.
They
can
deploy
it
to
the
staging
mesh
and
from
there
it
kind
of
lives
in
staging
just
like
anything
else,
but
notably
it
does
not
receive
traffic
from
any
other
people
who
are
testing
and
staging.
So
this
fixes
a
lot
of
those
problems
of
teammates
stepping
on
each
other's
toes.
A
The
next
part
of
staging
overrides
is
this
override
metadata?
Now,
of
course,
we
don't
want
random
strangers
and
staging
designers
product
who
are
testing
to
hit
your
potentially
buggy
code,
but
we
do
want
you
to
hit
it.
So
you
can
make
sure
you're
confident
so
to
do
that
we
had
to
embed
metadata
within
the
request
that
kind
of
informs
us
how
to
do
this
type
of
routing
and
make
you
get
offloaded
to
your
or
get
sent
to
your
offloaded
pod.
A
In
this
case,
the
fraud
service
and
that's
kind
of
like
the
last
part
that
brings
us
all
together,
is
what
we
call
routing
override
and
sometimes
called
just
overriding
override
based
development.
Where
once
we
know
that
this
request
has
the
metadata
that
says
we
want
to
do
a
test
version.
We
need
to
actually
act
on
that
and
Route
kind
of
dynamically
and
appropriately
to
the
Pod
that
you
specified.
A
So
those
are
kind
of
like
the
main
components
that
make
up
the
system
they
call
staging
overrides.
First,
is
these
unregistered
offloaded
pods?
The
way
we
do
that
is
we
use
an
XDS
control
plane
that
we
have,
and
we
have
some
like
very
simple
logic
that
does
some
endpoint
Discovery
service
exclusion.
A
The
next
part
is
that
propagated
override
metadata
for
that
we
use
envoys
distributed
tracing
technology.
They
have
built
in,
as
we
just
saw
from
University
of
Illinois.
They
have
a
concept
in
there
called
baggage
that'll
dig
into
a
little
bit
further.
That
lets
us
propagate
this
data
and
then,
lastly,
are
the
actual
overrides
that
act
upon
this
data,
which
are
a
custom
filter
that
we
wrote
in
Envoy,
as
well
as
a
cluster
called
the
original
destination
cluster
or
original
DST.
A
So
in
the
next
couple,
minutes
I'll
just
kind
of
talk
about
all
these
different
pieces
and
and
how
they
work
and
how
we
kind
of
all
combine
them
together.
To
do
this,
so
first
is
offloaded,
pods
I
would
say
this
is
probably
the
more
straightforward
part
to
do
this.
We
created
GitHub
bot
for
all
of
our
users,
who
are
ended
up
putting
up
pull
requests.
They
could
just
type
slash
offload
and
once
their
container
image
had
finished
building
they
would
end
up.
A
It
would
start
a
new
deploy
a
new
deployment
in
our
kubernetes
staging
cluster,
and
this
deployment
that
had
the
Pod
inside
would
be
labeled
specifically
with
this
offloaded
deploy,
equals
true
label.
So
that's
really
all
we
have
to
do
on
the
deploy
side
to
modify
it
and
kind
of
wire
that
up
with
some
amount
of
bots,
but
then
in
our
control,
plane,
we're
processing
these
Kate's
events
and
sending
out
EDS
to
all
of
our
sidecar
mash
and
telling
them
like
hey
here's
a
routable
instance
of
the
pay
service.
A
A
So
this
on
its
own,
simply
deploying
a
pod
into
staging,
which
is
the
same
environment
that
others
are
comfortable
testing
in
actually
provided
some
benefits
and
people
just
wanted
to
SSH
into
a
pod,
run
a
script
and
kind
of
have
confidence
that
what
they're
doing
was
working.
They
didn't
need
any
Ingress
traffic,
but
of
course
the
the
power
comes
when
you
actually
can
choose
to
route
to
that
pod,
and
to
do
that,
we
needed
to
use
some
of
this
override
metadata.
So
here's
kind
of
just
an
example
of
what
this
metadata
would
look
like.
A
A
So
let's
dig
a
little
bit
more
into
that
override
metadata.
It
doesn't
have
to
be
very
complicated.
We
technically
we
have
some
protobuf,
but
you
know
really
just
a
Json
blob.
That
looks
like
this.
If
you
have
some
set
of
overrides
that
you
want
to
apply
throughout
the
duration
of
the
request,
you
just
say:
hey
this
service
in
particular,
rather
than
going
to
the
Blessed
version,
that's
running
on
staging
go
to
this
IP
colon
port
instead.
A
A
The
same
for
what
we're
doing,
we
were
able
to
embed
a
special
field
within
the
header
called
baggage,
which
a
lot
of
the
time
the
metadata
from
your
spans
within
a
given
Trace,
are
kind
of
local
only
to
that
span,
and
then
they
don't
also
get
propagated
to
any
further
upstreams
in
the
call
graph.
But
baggage
is
special
in
that
it
is
information
you
embed
once
and
it
gets
propagated
across
process
lines.
So
for
us,
that's
super
useful,
because
all
we
need
to
do
is
really
early
in
the
request
line.
A
So
now
that
we
have
the
metadata
that's
been
propagated,
we
have
our
offloaded
pods.
We
need
to
build
the
networking
magic
that'll,
actually
act
upon
that
metadata
and
send
us
to
the
Pod
we're
interested
in
so
very
high
level.
Overview
of
envoy
lift
pretty
standard
stuff.
Each
service
is
deployed
alongside
a
sidecar
Envoy,
it's
a
container
within
a
pod
for
each
service,
and
this
container
handles
all
Ingress
to
the
service
as
well
as
egress,
which
historically
provides
all
the
benefits
you
would
expect
out
of
envoy.
You
know
the
observability
super
simple
for
applications.
A
You
don't
need
to
think
much
about
networking,
but
for
our
specific
use
case,
it's
great
that
we
knew
every
service
was
deployed
alongside
these
sidecars
because
it
really
gave
us
a
hook
that
we
can
Implement
and
kind
of
intercept
all
of
the
routing
that
normally
happens
so
to
do
that,
we
used
envoys
HTTP
filter
subsystem,
so
we
modified
and
created
our
own
filter
that
then
we
injected
into
every
single
one
of
the
sidecars
that
is
deployed
at
Lyft
and
basically,
what
we
need.
This
HTTP
filter
to
do
is
mutate.
A
How
we
end
up
doing
that
routing
and
how
we
choose
what
our
Upstream
endpoint
is
just
before
we
were
about
to
go
to
whatever
was
normally
on
staging,
so
at
a
really
high
level.
The
filter
logic
is.
We
extract
the
information
outside
of
the
baggage,
which
is
generally
just
service.
Ip
pairs
like
I,
showed
before
we
see
if
any
of
the
overrides
within
that
baggage
match,
where
we
are
about
to
route
to
by
the
time
this
HTTP
filter
is
run.
We
already
know
where
Envoy
intends
ongoing.
A
So
here
is
a
very,
very
high
level,
C
plus
plus
overview
of
what
the
filter
looks
like
I'm,
not
going
to
walk
you
through
it
entirely
today.
If
you're
interested,
you
can
go
download
the
slides
later
and
check
out
all
the
code,
but
I'm
going
to
walk
you
just
through
the
real
key
components
that
kind
of
make
it
tick.
The
first
part
is
what
we've
been
talking
about,
which
was
because
Envoy
has
first
class
support
for
tracing.
All
we
have
to
do
is
when
we
Define
our
HTTP
filter.
A
We
say
we
get
all
these
callbacks.
That
gives
us
access
to
things
like
the
current
Envoy
span
and
then
from
there.
All
we
have
to
do
is
get
baggage
and
pass
it.
The
key
that
you
want,
so
you
can
imagine
having
posted
our
override
metadata
within
the
baggage
overrides
key.
We
just
call
this
get
baggage,
bring
it
out
and
then
parse
it
into
whatever
data
structure
that
you
want
to
represent
it
in.
A
A
You
know
you
can
imagine
if
we
were
sending
it
to
the
pay
cluster.
Originally,
we
want
to
go
to
a
different
cluster
instead.
So
what
we
end
up
having
to
do
is
take
that
existing
cluster
and
wrap
around
it
in
this
thing
called
the
delegating
route
that
we
kind
of
had
to
subclass,
and
when
we
do
that,
we
can
provide
the
cluster
that
we
want
to
go
to
instead.
A
A
So
what
we
want
to
do
is
send
it
to
this
other
cluster
called
an
original
destination
cluster,
so
the
original
destination
cluster
sometimes
called
the
original
DST
has
a
special
header,
a
configuration
called
use,
HTTP
header,
and
when
that
is
enabled,
if
a
request
gets
sent
to
this
cluster
with
this
header
x,
dash
Envoy,
Dash
original
DST
host,
which
you
see
at
the
bottom,
with
the
IP
address,
colon
Port
Envoy
will
take
the
IP
address
and
Port
of
that
header
and
route
to
that.
Instead
of
you
know,
wherever
it's
going.
A
Instead
of
that,
so
coming
back
to
the
filter,
you
can
imagine
what
we
had
to
do
was
when
we
overwrote
the
cluster
that
we
were
doing
our
routing
to
in
the
filter.
We
send
it
to
the
original
destination
cluster
instead
and
then
we
also
add
in
a
new
HTTP
header
and
that
HTTP
header
is
the
envoy
original
DST
host,
and
we
also
just
add
in
that
IP
address
that
we
kind
of
extracted
out
of
the
baggage
earlier,
and
that's
it
not
too
hard.
A
Of
course,
a
lot
of
error
handling
and
other
things
like
that
that
we
didn't
end
up
doing
as
a
real
high
level
overview.
Our
steps
there
were
take
out
the
overrides
from
the
baggage
from
the
tracing
header.
Take
that
route
wrap
it
in
this
delegating
route
and
make
it
so
that
for
the
most
part,
all
attributes
about
this
route
are
exactly
the
same
as
the
one
Envoy
has
already
chosen,
but
except
the
cluster
name
for
the
cluster
name.
We
want
to
send
it
to
the
original
destination
cluster.
A
A
You
know
a
year
and
a
half
or
something
to
Pivot
from
this
one
box
system,
and
we
did
this
a
while
ago-
and
we
were
really
happy
where,
where
this
got
us,
you
can
see
here,
you
know
developers
can
keep
sending
requests
to
staging
just
like
they
normally
would
developers
or
product
managers
data
science.
Anyone
who's.
Just
trying
to
test
something
see
how
something
looks
all
the
while
they
are
kind
of
blissfully
unaware
that
there's
all
of
this
testing
happening
around
them
because
it's
not
affecting
their
requests.
A
A
So
now
I'd
like
to
talk
a
little
bit
more
about
some
of
the
stuff
that
we
have
done
on
top
of
it
as
where
we
kind
of
want
to
go
next.
Now
that
we've
built
this
base
system,
so
one
thing
I
kind
of
hand
waved
over
at
the
beginning
is
how
do
we
actually
get
that
baggage
onto
the
request
in
the
first
place,
and
the
kind
of
we
call
that
baggage
attachment
tooling?
A
To
do
that?
We
ideally
don't
want
our
developers
thinking
about
how
do
I
manually
craft,
this
IP
address
Json,
blob
and
tuck
it
onto
every
single
request.
We
kind
of
want
that
to
be
abstracted
away
from
them.
So
there's
a
couple
different
ways
that
we
thought
about
this,
but
ideally
the
way
we
want
our
customers,
which
are
other
developers
that
lift
to
think
about
it,
is
I,
have
a
pull
request.
A
I've
deployed
this
pull
request
into
staging,
given
that
this
was
the
pull
request
of
the
changes,
give
me
the
baggage
and
attach
this
baggage
with
the
routes
me
to
it.
I
don't
really
care
about
any
other
details,
besides
that,
so
we've
actually
worked
on
a
couple
different
ways
to
baggage
attachment.
These
first
two
are
kind
of
examples
of
if
we
add
another
filter
up
front
at
the
edge
only
living
on
the
edge,
not
necessarily
in
the
mesh,
we
could
basically
have
it
decode
some
information,
either
from
a
header
or
from
the
host
name.
A
So
the
proxy
we
have
is
our
own
in
homegrown,
proxy
we'd
love
to
open
source
it
someday-
and
maybe
we'll
get
to
that.
Originally
we
created
this
progress.
This
scriptable
Ingress
proxy
for
mobile
Engineers
to
mock
out
backends
so
rather
than
pointing
their
mobile
apps
at
staging,
they
would
point
it
at
this
proxy
and
then
they
could
write
all
sorts
of
scripts
against
it
that
let
them
you
know
if
the
back
end
hadn't
completed
it
yet
return
back.
A
Some
dummy
Json
data
or
something
like
that
and
that
kind
of
allowed
the
product
Engineers,
who
are
both
mobile
and
back-end
to
work
concurrently
after
they
kind
of
agreed
upon
a
contract
of
what
the
response
body
would
look
like.
So
this
is
what
we
ended
up
using
is
we
just
tell
our
developers
to
point
their
browsers
at
their
own
proxy,
and
that
is
what
they
do
today
right
now:
they
they
say
they
Point
their
app
to
say.
Let
me
talk
to
this
scriptable
proxy.
A
This
proxy
has
all
of
the
information
it
needs
to
know
once
I've
configured
it
and
it
will
attach
the
baggage
before
it
ends
up
sending
it
off
to
the
rest
of
staging.
So
we
because
we
already
have
this
proxy,
which
was
very
fortunate
for
us.
We
kind
of
took
what
we
already
had
and
took
the
easiest
path
to
getting
this
working
for
our
users,
so
we
just
kind
of
defined
like
a
helper
type
script.
A
So,
given
that
we've
built
all
of
that
so
far,
we
still
think
there's
a
lot
more.
We
could
do
with
this
General
overrides
driven
development
so,
like
we
already
have
this
proxy.
That
knows
how
to
interact
with
baggage.
What
if
we
Chuck
that
proxy
into
the
middle
of
the
call
graph,
as
opposed
to
one
of
these
offloaded
pods?
Now
we
have
this
wildly
configurable
and
scriptable
solution
that
lets
us
kind
of
play
with
and
Fiddle
with
requests
before
they
actually
go
to
their
upstreams.
It
could
work.
A
A
Another
thing
we've
been
exploring
is
what,
if
we
redirect
the
traffic
to
laptops,
rather
than
just
going
to
offloaded
pods
that
have
been
deployed
to
staging.
If
we
embed
something
in
the
metadata
that
lets
you
redirect
to
your
own
laptop,
you
could
do
this
much
faster
PR,
iteration
cycle.
You
don't
have
to
push
up
a
pull
request.
You
don't
have
to
wait
for
container
image
build
and
you
don't
have
to
wait
for
a
deploy.
A
You
can
just
hit
it
straight
to
your
own
computer
and
another
extension
that
we've
been
playing
with,
and
we
have
some
of
this
and
we
just
keep
adding
more,
is
what
else
could
we
include
in
this
baggage?
For
example,
what
if
we
include
something
that
said
like
log
level
debug
and
that
will
very
temporarily
only
for
the
duration
of
your
request-
increase
the
verbosity
of
all
the
logs,
so
you
know
something
that
might
not
be
showing
up.
You
just
need
to
add
a
little
bit
more
metadata,
and
this
is
sufficient
optionally.
A
This
will
make
them
feel
like
they
have
all
the
knobs
and
levers
they
have,
so
they
can
reproduce
all
their
Arrangements
make
sure
that
they
feel
confident
that
they've
tested
everything
comprehensively
all
the
different
arrangements
before
they
end
up,
sending
it
to
real
staging
and,
more
importantly,
to
production,
and,
of
course,
when
we
do
this,
we
want
to
make
sure
it's
fully
isolated.
We
want
to
make
sure
that
developers
aren't
stepping
on
each
other's
toes
when
this
happens
and
our
team,
the
developer
experience
is
responsible
for
giving
really
good
tooling.
A
Overall,
these
are
really
good
results
compared
to
our
last
iteration
of
our
developer
environment
with
one
box
provisioning,
a
one
box
would
take
an
hour
or
sometimes
more
or
it
just
would
never
work
versus
getting
a
pull
request
built
and
pushed
up
to
staging
takes
10
minutes.
If
that
also
there's
a
lot
more
parity
now
between
our
staging
and
production
environments
than
our
development
was
with
our
other
environments-
and
this
is
great
for
us,
because
the
infra
is
more
similar,
so
changes
we
make
in
production
often
end
up
benefiting
staging
as
well.
A
So
we
just
got
a
lot
more
for
free
and
there's
also
a
lot
more
functional
parity.
So
that
way,
developers
who
push
something
see
it
work
in
staging
overrides
and
then
push
it
to
production
are
a
lot
more
confident
that
it
will
work
as
they
expect
it
to,
and
then
lastly,
we
think
this
is
just
a
great
new
framework
that
has
kind
of
changed
the
way
our
team
is
working.
We're
thinking
about
what
else?
A
Can
we
let
our
developers
do
to
make
sure
that
they
feel
confident
in
their
changes
before
it
goes
to
production
and
we're
kind
of
just
getting
started
on
all
the
different
things
where
we
can
enable,
as
far
as
some
challenges
that
we
ran
into
when
we
were
doing
this
context,
propagation,
as
we
talked
about
in
the
last
chat,
is
challenging.
We
need
to
make
sure
that
header
now
it
contains
the
trace
ID,
but
also
contains
these
routing
rules.
A
So
if
you
have
a
service
that
you're
trying
to
override
real
deep
into
a
call
graph
and
somewhere
along
the
way
that
header
gets
dropped,
then
it
won't
end
up
being
routed
to
the
override,
because
it'll
not
have
that
metadata.
So,
thankfully,
at
Lyft
we
have
a
whole
bunch
of
libraries
that
we
maintain
for
a
common.
You
know:
go
python
typescript
that
we
can
kind
of
inject
our
own
little
code
into
there
and
make
sure
that
it
does
the
propagation
there's
another
General
problem
with
data
isolation.
A
We
are
sharing
the
same
staging
environment,
so
you
can
imagine
if
something
is
really
broken
in
staging
and
starts
paging
and
sending
a
lot
of
stats
that
might
actually
page
to
on-call.
So
we've
had
to
do
work
there
to
sort
of
Ensure.
It
gets
sent
to
like
a
different
stats
namespace,
so
it
doesn't
trigger
all
these
alarms
that
have
already
been
set
up.
This
has
been
challenging
because
it's
a
generally
new
paradigm,
we
kind
of
had
to
do
a
whole
year-long
flow
of
reteaching
people.
A
How
this
is
supposed
to
work,
because
it's
very
different
from
the
the
one
box
you
know
everyone
gets
their
own
sandbox
type
development.
And,
lastly,
if
we
could
redo
this,
we
think
there's
a
lot
better
ways,
even
within
Envoy
versus
original
DST
that
we
could
do,
but
there's
also
other
really
cool
Tech
now,
like
telepresence,
is
another
cncf
tool
that
does
this
very
similarly,
and
you
know
we
would
probably
reassess.
Of
course
we
had
the
envoy
expertise
for
filters
in-house
and
we're
glad
to
work
on
that.
A
A
A
So
I
think
that
originally,
that
is
what
our
Trace
header
was
used
for
was
Telemetry,
and
in
fact
we
when
we
saw
that
we're
like.
Oh
can
we
really
trust
it,
because
our
distributed
tracing
had
some
sort
of
flakiness
to
it,
but
it
actually
forced
us
to
reinvest
it
and
it
doesn't
really
to
the
way
we
see
it
is
yes
that
metadata
is
used
for
Telemetry,
but
we
think
it
can
be
used
for
more
and
it's
worth
investing
in
because
it
can
be
used
for
more.
So
that
was
why
it
was
there.
A
Originally,
it
was
great
for
us
because
it
was
already
partially
implemented.
It
was
already
mostly
propagating,
but
finding
another
use
case
where
it
actually
could
like
influence
product
flows
was
enough
for
us
to
be
like
okay,
let's
really
go
in
here.
Let's
instrument:
where
are
the
services
that
are
dropping
traces?
How
do
we
go
fix
them?
How
do
we
page
them?
How
do
we
make
sure
that
they
don't
introduce
new
changes
that
end
up
dropping
this
header?
A
A
Yeah,
the
question
is:
does
anyone
want
one
box
back,
which
is
you
know
our
big
honking
ec2
instance
so
separately?
Alongside
with
this,
where
we
got
rid
of
one
box
there's
still,
this
was
kind
of
our
solution
for
end-to-end
testing
to
make
sure
that
you
felt
like
something
worked,
you
know
involving
multiple
Services
kind
of
integration
testing,
but
one
box
also
handled
the
use
case
of
local
development.
So
real,
quick,
just
my
own
service.
Let
me
run
unit
tests.
A
Let
me
really
quickly
iterate
on
it,
so
we
do
when
we
deprecated
one
box.
We
also
invested
heavily
in
getting
a
really
good
local
development
experience
as
well,
and
by
doing
that,
we
we
use
like
a
bunch
of
some
other
tools
like
tilt
is
one
of
them
for,
like
basically
running
Docker,
compose
locally
on
your
own
machine,
you're,
only
limited
to
really
running
your
service
and
we've
kind
of
cracked
down
on
you
know.
A
You
can't
have
this
crazy
Services
spend
dependency
on
your
own
laptop,
so
I
think
a
combination
of
those
two
which
is
the
really
good,
fast
local
Arrangement
and
then
like
a
much
more
comprehensive
and
trustworthy
end-to-end
experience
in
staging
kind
of
covers
all
the
gaps.
There's
still
people
who
do
want
something
like
that,
and
you
know
there's
ways
you
can
kind
of
do
it,
but
for
the
most
part
it
seems,
we've
gone
to
Oregon
board.
A
Definitely
a
lot
of
like
evangelism
internally,
though
that
we
had
to
do
you
know
that
was
certainly
one
of
the
challenges
here,
which
is
just
this
new
paradigm.
It's
like
we
had
to
clot
away
from
some
people
effectively
because
they
were
much
more
confident
in
it
or
maybe
they
weren't
set
up
super
well
in
staging,
like
most
services,
are.
A
A
It's
not
great,
but
there
are,
like
we've,
been
chatting
a
lot
with
the
networking
team
about
how
we
can
make
it,
so
they
can
like
test
a
new
Shaw
of
like
our
Envoy
binary
on
like
just
one
pod
or
something
so
like
we've
like
been
looking
into
little
Integrations
with
infra
teams
here
and
there
they're.
All
just
kind
of
you
know
way
less
the
same
in
the
way
that
all
these
product
teams
are
way
more
like
similar,
like
stateless
services.
So
it
was
way
more
easy
to
uniformly
introduce
this
for
them.
A
But
then
our
deploys
team,
we
kind
of
have
like
a
one-off
thing,
that
sort
of
works
with
this
system
and
requires
a
little
bit
of
cube,
cuddle
port
forward
and
a
couple
other
hacks
like
that.
But
it
can,
you
know,
should
we
choose
to
invest
it?
We
have
gone
at
working
with
some
other
info
components.