►
Description
The Mechanics of Deploying Envoy at Lyft - Matt Klein, Lyft
The idea of the "service mesh" is becoming very popular in microservice design circles. However, the mechanics of deploying one into an existing infrastructure are far from simple. In this talk we will cover the logistical details of how Envoy was developed and deployed incrementally at Lyft, focusing primarily on the evolution of service mesh configuration management. We will also discuss why high level systems such as Istio are likely to be the main mechanism by which most customers ultimately get access to the technology.
About Matt Klein
Matt Klein is a software engineer at Lyft and the architect of Envoy. Matt has been working on operating systems, virtualization, distributed systems, networking, and making systems easy to operate for 15 years across a variety of companies. Some highlights include leading the development of Twitter’s C++ L7 edge proxy and working on high-performance computing and networking in Amazon’s EC2.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Thanks
everyone
for
coming
I'm,
gonna,
let
you
know
that
I'm
getting
over
a
super-pac
cold.
So
if
my
voice
does
weird
things
or
if
I
start
hacking
uncontrollably,
that's
why
so?
What
we'll
be
talking
about
today
is
actually
how
we
built
and
deployed
envoy
at
at
lyft-
and
this
is
gonna-
be
a
talk
where
I'm
not
trying
to
sell
you
something
we're
actually
going
to
talk
about
the
practicalities
of
building
something
and
deploying
something
very
complicated
into
a
very
large
existing
infrastructure
and
I.
A
Think
it's
a
useful
case
study,
because
you
know
we
come
to
conferences
like
these
and
we
learn
about
all
the
really
amazing
technology
that
people
are
building.
But
very
often
we
don't
talk
about
it
in
a
construct
that
lets
people
understand.
You
know
you're
coming
from
some
existing
technology
and
you
want
to
adopt
this
new
technology.
That's
a
very
hard
problem.
It's
not
as
easy
as
saying
we're
going
to
run
envoy
or
run
sto
or
run
or
run
kubernetes.
That's
just
not
not
the
way
that
it
works.
A
A
I
think
everyone
here
probably
has
a
good
idea
of
of
what
that
is
a
day
behind
envoy
is
that
the
network
should
be
transparent
to
applications
and
when
network
and
application
problems
do
occur,
they
should
be
easy
to
determine
what
the
source
of
the
problem
is
and
what
we
find
these
days
when
people
try
to
deploy
micro
service
architectures
is
that
the
the
network
is
not
transparent.
It's
in
fact
a
huge
thorn
in
people
sides,
and
it
is
very,
very
difficult
to
understand
where
problems
are
occurring.
Is
that
problem
in
your
application?
A
Is
it
in
the
virtual
infrastructure
that
you're
using?
Is
it
in
some
container
runtime
right?
It's
like
almost
impossible
to
know
and
what
I
think
a
lot
of
companies
actually
find
when
they
try
to
move
from
a
monolith
to
a
micro
service
architecture?
Is
they
they
want
the
agility
gains
of
actually
moving
towards
that
architecture,
but
then,
when
they
go
to
actually
accomplish
that
architecture
when
they
go
to
actually
accomplish
that
architecture,
you
know
it
becomes
very
problematic
because
people
don't
trust
those
services
and,
in
fact,
lift.
A
What
we
saw
is
that
we
had
our
monolith
and
actually
all
you
know
what
it's
fine
and
what
we
saw
is
at
lift.
You
know
when
we
started
to
roll
out
our
Microsoft's
architecture.
We
found
that
developers
they
they
didn't
trust.
The
network
like
they
were
seeing
long
tail
problems
and
they
didn't
understand
how
to
debug
them.
They
saw
long
p99
latency
and
they
would
say
you
know
we
can't
do
this
this
networking.
We
can't
do
this
RPC,
so
we're
just
gonna.
Go
ahead.
We're
gonna
write
those
features
back
back
in
that
monolith.
A
So
we
we
see
that
problem
where
we
want
to
move
towards
this
architecture,
but
we
don't
have
the
tools
to
allow
people
to
write
business
logic
and
not
focus
on
all
the
faults
that
are
actually
happening
so
from
a
service
mesh
perspective.
This
is
a
very
simple
diagram
of
what
the
service
mesh
does,
and
the
idea
here
is
that
we
have
a
couple
of
different
services.
We
have
a
sidecar
proxy
that
is
co-located
next
to
every
service.
The
service
sends
all
of
its
requests
or
traffic
to
its
local
proxy.
A
That
proxy
does
all
of
the
things
that
we
all
know
and
love
around
service
discovery,
low,
balancing
rate
limiting
observability,
and
you
know
it's
gonna
pick
a
target.
It's
going
to
send
that
request
to
the
other
side.
The
other
side
will
deliver
it
to
the
service
and
then
it'll
all
come
back,
and
the
idea
here
is
that
each
individual
service
only
knows
about
its
local
proxy.
It
doesn't
have
to
actually
understand
anything
about
the
underlying
network
topology,
and
it
also
allows
it
to
be
the
case
that
your
service
can
be
written
in
any
language.
A
It
doesn't
matter
what
language
it
is
high
school
goes.
C++
Java
doesn't
matter
that
service
can
talk
to
its
proxy
on
localhost.
It
can,
you
know,
have
something
some
magic
happen:
timeouts
whatever
and
then
the
response
comes
back
and
it
and
it
comes
back,
and
it
really
does
make
that
network
transparent
in
a
way
where
people
can
focus
on
business
logic,
and
they
don't
have
to
worry
about
what
the
network
topology
actually
looks
like
so
from
an
envoy
perspective
and
I'm
gonna
go
through
this
superfast
Envoy
is
a
proxy.
It's
a
self-contained
proxy.
A
It
is
not
a
library,
so
it's
an
out
of
process
architecture.
So
when
I
showed
you
that
last
side,
let
last
slide
on
voice
it's
under
the
service
and
it
basically
receives
all
of
those
calls
and
those
calls
go
out
and
they
actually
come
back
envoy
is
written
in
C++
11.
So
it's
very
fast
as
well
as
very
productive.
Fundamentally
envoy
is
an
l3
l4
filter
architecture.
So
what
that
means
is
that
bytes
come
in
and
bytes
go
out,
so
we
can
support
multiple,
multiple
protocols.
A
We
can
do
rest,
we
can
do
Redis,
we
can
do
MongoDB.
We
can
do
my
sequel.
This
is
going
to
be
problematic
and
because,
obviously,
we
do
lots
and
lots
of
h1
and
h2
and
rest
and
all
of
these
things
we
do
a
lot
of
filtering
up
at
l7.
So
we
can
operate
on
message
boundaries,
so
headers
body
as
well
as
trailers
envoy,
is
an
h2
proxy
first.
So
we
can
proxy
from
H
1
to
H,
2
and
H
2
back
to
H
1.
A
We
spend
a
lot
of
time
doing
service
discovery
as
well
as
active
and
passive
health
checking,
so
active
health
checking
is
the
idea
that
I'm
going
to
periodically
send
a
ping
or
a
health
check
request
to
some
other
service.
Passive
health
checking
is
outlier
detection.
So
that's
the
idea
that
I
can
monitor
traffic
that's
happening
within
my
data
plane
and
and
I
can
go
back.
Sorry,
folks
and
and
I
I
can
go
back
and
I
can
see
if
there's
consecutive
500s
or
if
a
particular
hose
has
some
type
of
success
rate.
A
That's
out
of
out
of
some
variants
from
an
advanced
load
balancing
perspective.
We
can
do
things
like
timeouts
and
retries
and
circuit
breaking
and
shadowing
and
all
types
of
fun
stuff
and
we'll
talk
about
it
a
lot,
but
when
I
give
talks
about
service
mesh
and
envoy,
the
most
important
thing
is
really
observability.
A
They
all
do
service
discovery.
They
all
do
time
outs,
they
all
do
rate
laminating.
They
have,
they
all
do,
circuit
breaking
and
from
from
ops
perspective,
running
the
same
software
on
the
edge
and
doing
TLS
termination
and
running
the
same
proxy
on
your
internal
service
to
service
network
has
huge
operational
gains.
It's
just
it's
kind
of
like
a
core
tenet
of
systems.
Engineering.
Let's
software,
you
can
run
less
less
software.
You
have
to
understand
it's
it's
vastly
simpler,
so
this
is
lift
about
five
five
years
ago,
and-
and
this
is
you
know,
almost
a
standard.
A
Lamp
stack
right
like
this
is
the
way
that
every
every
startup
starts.
We
have
clients,
we
we
go
to
the
internet.
You
know
we
have
the
AWS
TLB.
Obviously,
now
AWS
has
their
advanced
load
balancer,
but
you
know
most
cloud
providers
still
have
a
fairly
basic
load.
Balancing
system
lift
was
originally
built
on
a
PHP
Apache
monolith
and
our
single
data
store
was
MongoDB,
and
you
know
this
is
supposedly
simple.
A
There's
no
there's
no
services,
but
what
I'd
like
to
point
out
really
briefly
here,
is
that,
even
even
in
this
basic
architecture
that
we
have
here
there,
there
still
is
networking
right,
like
you
still
have
the
load
balancer,
you
still
have
the
database,
you,
you
still
have
your.
You
know,
monolithic
service
and
even
with
this
basic
architecture,
trying
to
understand
what's
actually
going
on
here
from
did
the
client
have
a
problem
on
the
Internet?
Was
the
Amazon
load
balancer
crapping
out
did
PHP
have
a
buged
at
MongoDB?
Do
something
bad?
A
It's
actually
really
hard
to
know.
So,
even
in
this
very
simple
state
at
lyft,
we
were
already
having
a
lot
of
operational
problems,
trying
to
understand
how.
How
is
this
thing
working?
How
like?
How
is
it
actually
happening
and
we're
gonna
go
into?
Why
we
built
on
boy
and
how
we
deployed
it,
but
I
actually
want
to
show
this,
because
it's
instructive
again
to
see
that
you'd
actually
don't
need.
You
know
hundreds
of
services
to
make
something
like
Envoy
useful.
It
can
be
useful
with
only
very
few
services
so
fast
forward
to
today.
A
This
is
lifts
architecture.
Today,
roughly,
we
still
have
clients
still
have
internet,
we
have
a
set
of
TCP
load,
balancers,
so
very
basic
edge
load
balancers
and
in
there
we
come
to
our
front
edge
on
voice.
So
we
run
a
fleet
of
edge
ingress
proxies
which
do
l7,
parsing
and
TLS
termination,
and
things
like
that.
We
still
have
our
monolith.
I
have
five
years
later
and
probably
for
many
more
years.
We
are
decomposing
that
monolith
and
we
have
lots
and
lots
of
services
written
and
go
and
written
in
Python
and
we
run
envoy.
A
We
have
a
hundred
percent
coverage
at
lyft,
so
envoy
runs
co-located
next
to
every
service
at
lip
and
has
hundreds
of
services
and
then
to
our
back
ends.
We
have
MongoDB
still,
we
have
dynamo
DB
we're
actually
starting
to
use
spanner
from
Google
and
we
obviously
do
a
whole
lot
of
stats
and
tracing
and-
and
you
know
a
whole
whole
bunch
of
other
stuff
and
again.
The
important
takeaway
here
is
that
we
run
envoy
on
each
and
every
node.
So
every
hop
that
we
have
from
ingress
to
egress.
A
Okay,
so
that
gives
you
kind
of
the
overview
of
what
happened
so
like
I
was
saying
in
the
intro.
You
don't
go
from
what
we
had
before
to
what
we
actually
have.
Now
you
don't
do
that
overnight.
You
can't
just
automatically
roll
out
envoy
on
hundreds
of
services.
It
doesn't
work
that
way.
So
what
we're
gonna
do
is
I'm
actually
gonna
talk
you
through
kind
of
how
and
why
we
develop
envoy
and
we're
gonna
talk
about
the
incremental
steps
that
we
took
to
actually
roll
it
out
and
talk
about.
A
You
know
the
things
that
are
developers
that
live
found
most
useful
because
as
you're
starting
to
think
about
rolling
out
some
of
these
proxies
a
lot
of
it
is
actually
selling
it
to
your
internal
teams
and
a
lot
of
times.
They
want
to
know
what.
What
are
we
going
to
get
out
of
this
right,
like
is
this
actually
useful
and
I
know
that
the
teaser
here
is
that
it
wasn't
easy.
It
was
a
lot
of
work.
A
Okay,
so
what
we
did
is
we,
we
started
with
the
edge
proxy
and
actually
talked
about
this
briefly,
and
almost
every
internet
application
has
some
avenge
proxy.
You
have
clients
on
phones,
you
have
to
get
back
to
your
back-end
services
and
from
a
fundamental
perspective.
If
your
app
is
not
healthy
at
the
edge,
it
doesn't
really
matter
what
you
do
internally
so
having
having
that
observability
at
the
edge
location
is
super
important
and
we
could
have
a
whole
long
conversation
about
how
it's
important
to
extend
that
observability
all
the
way
out
to
the
client.
A
But
that's
not
not
this
talk,
but
from
the
envoy
perspective,
it's
critical
that
we
get
good
stats
out
to
the
edge
and
even
to
this
day,
if
you
look
at
the
state-of-the-art
cloud
solutions,
state-of-the-art
being
Google's,
load,
balancers
and
Amazon's
elastic
load,
balancer
that
the
alb
product,
they're
terrible
I
mean
like
they're,
still
really
terrible.
They're
black
boxes,
where
you
don't
actually
understand
what's
going
on
the
observability
coming
out
of
them,
is
still
fairly
bad.
A
A
Here's
an
example
of
one
of
the
dashboards
that
would
actually
provide
people
at
at
lyft
and
what
you're
seeing
here
is
you're,
seeing
on
a
per
host
basis
here,
our
our
our
front
edge
nodes
and
you're.
Seeing
stats
like
the
number
of
connections
per
second,
the
number
of
requests
per
second,
but
then
on
the
bottom
row,
you're,
seeing
latency
on
a
per
back-end
service
basis
or
you're,
seeing
the
500s
on
a
per
back-end
service
basis,
and
this
is
a
tiny
portion
of
the
metrics
that
we
actually
show
people.
A
But
the
idea
here
is
that
you
go
from
a
situation
in
which
you're
trying
to
go
into
s3
and
pull
down
logs,
and
then
you
know
and
kind
of
understand.
What's
going
on
and
the
dashboards
and
you
know
cloud
watch
are
super
bad
to
a
situation
where
we
have
like
excellent
visibility
and
we
can
much
quicker
pinpoint
what
are
problematic
services
or
what
are
causing
problems
for
our
customers.
So
again,
it's
all
about
observability,
observability
observability.
A
So
now
now
we're
on
our
web
scale
slide.
And
if
this
was
a
45-minute
talk,
we
would
stop
here.
We
would
watch
watch
the
MongoDB
use
web
scale,
video,
but
it's
not
45
minutes.
So
I
would
encourage
all
of
you
if
you
have
not
seen
the
video
and
the
little
asterisk
thing
down
there
to
go.
Watch
it
right
now,
because
it
is
the
funniest
video
that
you
have
ever
seen
so
lift
probably
at
this
point
is
one
of
the
biggest
MongoDB
installations
in
the
world.
A
I'm
guessing
like
we
have
a
lot
of
MongoDB
and
MongoDB
has
been
a
thorn
in
our
side
for
a
a
long
time,
and
one
of
the
things
that
you'll
find
with
MongoDB
even
in
the
most
recent
releases
is
MongoDB
has
a
very
poor
connection
handling
model
and
by
that
it
uses
a
thread
per
connection
and
high-performance
networking.
That
is
a
big
no-no,
and
what
that
means
is
that
MongoDB
will
das
itself
very
easily.
A
If
MongoDB
does
something
bad
it'll
start,
you
know
kind
of
refusing
connections,
then
clients
will
reconnect
and
then
you'll
have
like
a
thousand
threads
and
then
the
thing
will
just
shut
down
your
and
your
you
know
basically
in
this
death
spiral.
So
what
one
of
the
things
that
we
realize
that
we
could
do
pretty
quickly
with
this
proxy,
so
we're
already
running
it
on
the
edge.
Now
we
have
our
PHP
monolith.
We
realized
that
we
could
use
envoy
actually
on
the
PHP
monolith
to
collapse.
A
Connections
from
PHP
to
so
basically
monitor
and
rate
limit,
all
of
the
connections
that
are
coming
into
MongoDB
from
the
client
side
of
things
and
hey.
While
we're
here,
we
can
actually
parse
the
B
song
and
we
can
suck
out
amazing
stats
about
latency
and
tables
and
query
patterns
and
multi
gets
and
scatter
gets
and
and
and
those
types
of
things,
and
you
know
we
can-
we
can
do
this
for
all
services.
So
now
we're
also
doing
at
this
point
we're
doing
on
Python.
A
So,
instead
of
rewriting
all
of
this
rate-limiting
and
all
of
this
great
functionality
that
we
built
in
every
driver,
we
write
it
once
a
non-void.
We
point
all
of
the
applications
at
envoy
envoy
talks
to
and
lo
and
behold
we
have
no
more
outages
due
to
death
spirals.
So
our
current
stack
still
looks
exactly
like
this,
so
we
now
talk
to
from
PHP
Python
Ango,
all
of
that
traffic
runs
through
envoy
and
via
the
Envoy
filter
chain
that
I
talked
about
before
we
can
change
different
functionality.
A
We
can
build
in
stats,
we
can
build
in
rate-limiting,
we
can
build
in
all
of
this
cool
functionality
and
we
get
that
for
free.
The
applications
don't
have
to
do
anything,
we
just
roll
it
out
and
it
and
it
just
works
so
once
we
did
that.
So
this
is,
you
know,
we're
now
going
about
a
couple
years
and
we
have
envoy
running
at
the
edge.
We
have
envoy
running
on
our
on
our
monolith,
we're
still
using
internal
load
balancers
from
Amazon
to
actually
do
service
discovery.
A
So
obviously,
when
the
edge
has
to
talk
to
the
backend
service,
we
have
to
find
those
back-end
services.
So
services
boots,
they
come
up.
They
they
basically
register
into
the
end
of
the
load.
Balancer
and
then
you
know
the
load
balancers
is,
is
going
to
come
and
actually
pick
that
back
and
what
you
find
is
that
again,
like
these
load
balancers
these
cloud
products
they're,
you
know
they're
getting
better,
but
there's
still
there
are
black
boxes
and
that
extra
hop
from
an
observer
from
it
from
a
kind
of
an
Operations
perspective.
A
When
you
have
failure
now
you
have
this
giant
thing
in
there
that
you
don't
know
what
the
problem
is,
and
you
have
to
figure
out.
Well,
it
could
have
been
a
bad
you'll,
be
instance.
It
could
have
been
the
ec2
Network.
It
could
have
been.
My
application.
I
have
no
idea
so
you're
you're,
putting
this
big
thing,
that's
in
the
traffic
path,
a
single
point
of
failure
between
any
two
endpoints,
and
that
just
makes
understanding
what's
happening
in
that
network
topology,
so
much
more
complicated.
A
So
so
what
we
found
here
is,
but
even
with
this,
even
with
the
with
the
ELB
in
the
middle,
because
we're
running
envoy
on
one
side
and
we're
running
envoy
on
on
the
other
side,
we
still
get
really
amazing
stats.
We
still
get.
Although
the
egress
stats
from
the
edge
nodes,
we
still
get
all
of
the
ingress
stats
on
the
PHP
nodes.
We
can
do
things
like
buffer,
so
that
you
know
we
can
deal
batter
it
with
PHP
Apache
traffic
handling.
A
You
know
so
we're
still
getting
lots
of
benefits,
but
the
the
next
stage
here
is
that,
because
of
what
I
was
saying
before,
we
would
like
to
get
rid
of
those
internal,
though
balancers
those
internal
load
balancers
they
make
debugging
harder.
There
are
a
single
point
of
failure.
You
just
do
not
want
them;
they
don't
serve
any
any
good
purpose,
so
we
would
like
to
do
Direct
Connect,
and
at
this
point
obviously,
if
we're
going
to
do
Direct
Connect,
we
need
service
discovery
and
I'll.
Take
a
little
detour.
A
Here
is
if
you
look
around
the
industry
the
way
that
most
people
at
most
kind
of
internet
companies
have
done
service
discovery
historically,
is
they've
done
them
using
systems
like
zookeeper
or
console
or
something
along
those
lines,
and
these
are
great
systems
they're
fine.
But
what
you'll
find
is
that
these
are
all
fully
consistent
systems.
They
all
use
leader
election.
They
all
require
care
and
maintenance,
and
they
all
go
down
a
lot
and
you'll
find
that
most
big
companies
have
teams
that
are
dedicated
to
keeping
zookeeper
and
similar
systems
running,
especially
at
at
scale.
A
So
when
we
started
with
envoy,
we
made
the
kind
of
explicit
decision
that
you
know
service
discovery
for
networking.
It's
not
a
consistent
problem.
It's
an
eventually
consistent
problem
systems,
scale
up
and
scale
down
and
things
but--and
die
right.
It
doesn't
have
to
be
consistent.
It
just
has
to
converge,
so
we
made
the
very
explicit
decision
that
we
would
go
through
and
we
would
build
an
eventually
consistent
system
and
that
system
which
we
still
use
today
is
about
200
lines
of
Python
and
a
cron
job
and
the
cron
job
runs
on
boot.
A
Basically,
it
runs
every
minute.
We
have
a
service
that
writes
into
into
dynamodb,
and
then
we
basically
have
a
sweeper
on
that
table
or
if
a
service
doesn't
check
in
in
X
minutes,
say
10
minutes,
we
just
delete
it.
The
system
is
dead,
simple.
We
literally
have
not
edited
the
code
in
over
a
year
and
we
have
had
zero
outages
and
because
of
all
the
caching,
it's
eventually
consistent,
but
because
we
cache
in
the
service,
because
cash-in
envoy,
and
because
of
this
eventually
consistent
process
that
the
cron
jobs
right
into
this
system.
A
We
have
run
tests
where
we
have
deleted
the
entire
Dynamo
table
from
under
the
service.
Everything
keeps
going,
we
obviously
can't
scale
up,
but
everything
is
fine.
We
make
the
Dynamo
table
again.
Everything
starts
checking
in
again
and
we're
totally
fine,
so
I
would
encourage
everyone
to
think
very,
very,
very
carefully
about
whether
you
really
need
a
fully
consistent
system
when
eventual
consistency
would
probably
be
fine.
A
So
let's
talk
a
little
bit
about
how
developers
at
lyft
actually
actually
use
envoy.
So
at
this
point
you
know
we've
we're
basically
building
the
mesh
we're
not
fully
rolled
out,
but
you
know
we're
building
this
mesh
we're
actually
getting
a
lot
of
value
from
it,
and
I
actually
mentioned
this
this
morning
during
a
panel
but
you'll
you'll
hear
a
lot
of
servers.
Mesh
talks
and
people
like
me
will
get
up
here
and
say
that
the
service
mesh
is
magic.
A
It's
like
totally
amazing
and
it
is
really
awesome,
but
there
there's
a
little
there's
a
little
thing
that
we
don't
tell
you,
and
that
thing
is
that,
in
order
to
get
full
value,
especially
around
tracing,
you
have
to
propagate
context,
and
that
is
required.
There's
no
way
around
that.
You
must
have
application
code.
That
actually
runs
because
you
have
some
header
or
some
piece
of
data
that
has
to
get
propagated
from
your
ingress
point
to
your
egress
point
so
that
you
can
join
those
traces
up.
A
So
what
we
did
at
lyft
is
we
wrote
a
very
thin
client
in
Python.
We
also
have
one
now
in
Ingo.
This
is
you
know
we
call
a
Envoy
client,
it's
very
simple.
What
you're
seeing
here
is
an
example
of
someone
calling
a
service
called
switchboard.
The
idea
here
is
that
they
don't
need
to
understand
how
envoy
works.
They
want
to
make
a
request
to
a
service.
They
use
a
client,
they
say
what
the
service
name
is.
A
They
may
set
some
options
that
you
don't
see
here
around
timeouts
or
retries
or
various
other
things,
and
then
this
client
is
about
a
hundred
lines
of
code
and
it
calls
down.
It
knows
what
port
envoy
is
actually
running
on.
It
actually
goes
through
and
it
makes
the
call
and
a
base
of
get
the
response
and
unpacks
it
and,
most
importantly,
it
propagates
IDs.
A
So
if
we
know,
for
example,
in
Python
that
we
have,
you
know
an
X
request,
ID
or
an
X
ot
span
context
header
that's
come
in
and
it's
on
the
greenlit
local
thing
or
whatever
it
is
in
Python.
You
know
we'll
take
that
state
and
will
propagate
it
to
the
other
side,
and
that
makes
it
really
easy
for
developers
and
go.
A
A
But
if
the
big
service
people
are
happy,
it's
a
lot
easier
to
get
these
small
service
people
happy,
and
what
we
found
is
that
people
were
quite
happy
like
we
are
suddenly
saying
Oh
like
we're
not
having
all
these
problems,
because
we
do
more
intelligent,
load,
balancing
or
hate.
When
a
problem
does
happen,
we
can
actually
debug
it.
Isn't
that
awesome!
So
now
what
we
did
is
we
entered
the
slog
period
and
during
this
period
we
actually
had
a
spreadsheet
and
the
spreadsheet
was
updated
on
a
daily
basis.
A
We
had
a
program
that
would
scrape
out
all
the
services
and
figure
out.
Are
they
using
envoy?
Are
they
not
using
envoy?
We
would
build
a
Google
spreadsheet
every
day
and
we
would
go
through
and
we
would
burn
that
spreadsheet
down,
and
that
was
a
combination
of
just
you
know,
carrot
and
saying
like.
Could
you
please
do
this
it'll
work
better
right
like
wouldn't
this
be
great?
We
went
in
and
did
some
of
the
work
ourselves
to
be
perfectly
honest
with
you.
I
was
expecting
to
have
this.
A
Take
much
more
of
a
stick
approach
like
I
thought
we
were
gonna
have
to
go,
get
in
there
and
kind
of
get
management
to
go.
Make
this
happen.
To
be
honest,
it
was.
It
was
a
lot
of
carrot,
like
people
loved
what
they
were
getting
like.
They
love
the
features
they
love
the
observability.
They
loved
the
fact
that
they
just
no
longer
had
to
worried
about
how
certain
portions
of
the
system
actually
worked.
A
So
in
hindsight,
it
was
a
long
slog
but
I
think
what
you'll
find
is
that
most
developers,
most
people
that
write
business
logic
that
actually
run
on
a
service
mesh.
They
don't
want
to
go
back
and
in
fact
it's
been
really
interesting
because
lyft
when
I
joined
was
about
80
developers.
It
was
quite
small
we're
over
500
now
and
you
know
when
I
joined,
there
was
people
that
knew
the
old
way
and
there
was
people
that
knew
the
new
way.
A
A
A
So
we
did,
then,
is
what
everyone
does
they
bring
in
some
templating
code
and
we
brought
in
Jinja
so
and
and
what
we
actually
developed
is
a
system
which
we
called
config
gen,
which
we
still
use
to
this
day,
certain
parts
of
and
what
that
is.
Is
you
basically
realize
that
a
lot
of
the
Envoy
configurations
they
become
the
same
like
there's
a
lot
of
commonalities
and
how
they're
doing
it's
mostly
the
same
with
a
few
variables
that
end
up
being
different.
A
So
we
ended
up
building
a
whole
system
that
would
effectively
take
a
set
of
templates,
a
set
of
services
and
a
set
of
variables
and
use
Jinja
iterate
over
them
and
build
an
entire
set
of
configurations,
and
it's
at
this
point
that
we
really
realize
with
Envoy
that
any
complex
envoy
deployment,
the
the
configuration
is
going
to
be
handwritten
our
sorry.
It's
going
to
be
machine
generated
somehow
there's
just
no
way
around
that
you
can
do
it
using
template.
You
can
use
it.
A
You
know
with
something
like
SDO
pilots,
but
something
is
going
to
do
it
like
no
one
is
going
to
sit
around
and
hand
make
configurations
for
hundreds
of
different
services.
It's
just
not
realistic.
So
at
that
point,
what
we
did
is
we
moved
to
a
hybrid
system
and
the
way
that
the
hybrid
system
actually
works
is
we
did
a
combination
of
build
time,
template
generation
and
then
what
we
would
do
is
we'd
actually
ship
the
templates
to
each
box
and
the
Jinja
files,
and
we
would
actually
run
the
tempo
generation
code
on
the
hosts.
A
So
what
we
do
at
lyft
is
every
service
that
lyft
has
a
manifest
and
the
manifest
says
things
like
these
are
the
tests
that
run
you
know
and
for
networking
these
are
the
services
we
talk
to.
These
are
the
circuit
braking
settings,
blah
blah
blah
blah
blah,
so
what
we
did
is
we
would
take
in
this
manifest
data
on
the
node,
take
it
in
with
the
ginger
templates
and
generate
the
final
envoy
configuration
on
the
node.
A
So
now
we
have
a
mix
of
central
template
generation
with
deploy,
but
then
the
the
manifest
information
gets
joined
on
the
host
during
salt
run
because,
as
you
can
see
from
here,
we're
not
yet
a
kubernetes
native
shop,
we're
still
using
VMs
and
salts
and
all
types
of
legacy
infrastructure,
so
one
of
the
goals
of
ongoing
has
always
been
to
be
kind
of
what
I
call
a
universal
data
plane.
The
idea
is
that
we
have
a
set
of
api's.
A
All
of
the
configuration
so
you're
kind
of
now
seeing
this
transition
period
we've
gone
from
hand,
configurations
to
machine
generated
Jinja,
but
now
we're
moving
towards
very
complex
and
servers,
and
you
see
that
with
sto
Pilate
and
there's
actually
people
that
are
that
are
writing
all
types
of
different
solutions
and
in
the
v1
API
we
have
a
couple
of
different
api's.
We
have
service
discovery,
service,
cluster
discovery,
service
route,
discovery
service,
listener,
discovery
service.
A
The
idea
here
is
that,
ultimately
we
allow
every
aspect
of
on
valais
that
would
be
reasonably
reconfigured
to
be
configured
by
a
remote
management.
Server.
You're
not
required
to
use
these.
So
a
goal
of
envoy
is
that
you
take
the
complexity
that
you
need.
If
you
want
handwritten
configs
use,
handwritten
can
fix
if
you
want
STS
only
use
STS
only
if
you
want
the
whole
enchilada
use
it
and
it
lifts.
A
Currently,
we
use
SDS,
RTS
and
CTS
will
be
using
LDS
soon,
and
this
is
roughly
what
it
looks
like
today
and
what
you're
going
to
see
in
all
of
the
other
talks
that
you'll
have
times
here,
is
that
this
looks
a
lot
like
SDO
pilot,
which
is
not
very
surprising,
because
these
things
have
evolved
in
in
parallel
and
the
way
that
this
works
is
that
in
envoy
today,
at
lift.
We
have
our
legacy
hundred
line,
Python
service
discovery
thing.
A
We
have
a
new
service
called
envoy
manager,
which
is
which
is
which
is
like
pilot,
and
we
are
now
taking
in.
You
know
all
this
information
from
our
legacy
infrastructure
from
our
cron
jobs
from
our
manifests,
we're
merging
them
together
and
studying
that
configuration
to
on
voice
so
over
time.
The
config
that
we
ship
to
every
host
is
getting
smaller
and
smaller
until
it
becomes
a
unified,
bootstrap
config
that
runs
on
every
host,
never
changes
and
everything
else
comes
from
those
management
servers.
A
So
that's
that
progression,
but
the
idea
is
to
understand
that
you
can
take
still
any
of
this
complexity
from
end
to
end
and
and
you
can
build
very,
very
cool
systems
on
that
and
we're
actually
now
beginning
our
kubernetes
migration.
So
you
know
this.
This
will
get
a
lot
more
complicated
as
we
have
our
legacy
infrastructure
and
then
bringing
kubernetes
and
juggle
and
drug
all
of
it,
so
that
will
be
very
exciting.
A
So
thank
you
very
much
I.
Thank
you
for
thank
you
for
dealing
with
my
voice.
It's
very
shaky.
You
can
talk
to
me
on
Twitter
I.
Think
I'm,
almost
out
of
time
might
have
time
for
one
question
or
so
or
two
questions.
My
plug
lift
is
hiring.
We
have
lots
of
good
jobs.
So
if
you
want
to
talk
to
me,
please
talk
to
me
and
we
we
love
building
the
overall
on
bike
community.
So
please
join
us
on
our
viperox
cio.
So
thank
you
and
I'm
happy
to
take
questions.
A
Currently,
no
I
I
think
that
lambda
is
or
just
generally
functions
and
envoy
is
going
to
be
a
very
interesting
thing
over
the
next.
You
know
probably
two
three
years
mainly
because
functions
at
lambda.
They
require
networking
yeah
and
it's
actually
networking
is
way
more
complicated
in
that
environment
because
things
come
up
and
come
down
very
quickly.
So
the
answer
is
no
not
right
now,
but
I
think
that's
going
to
be
a
big
area
of
investment.
A
Yep,
so
the
question
is
at
lyft:
what
does
our
obstacle?
Ike
there's?
Actually,
if
you
look
at
the
monitor,
Amma
talks,
there's
a
couple
of
talks
to
talks
from
from
from
lift
that
go
into
that
entire
thing,
but
briefly
say
that
we
use
a
company
called
wavefront
for
stats.
We
use
a
company
called
light
step
for
tracing.
We
run
our
own
logging
system,
which
is
elastic,
search
and
cabana.
A
Yep
so
so
the
question
is
unvoyage
a
system
called
run
time,
which
is
like
a
decider
or
a
feature
flag
system.
It's
pretty
common
that
many
companies
have
these
things.
We
just
have
a
git
repo
that
basically
every
time
it
gets
checked
into
it
gets
sprayed
out
to
to
all
hosts
on
boy.
Does
a
file
system
watch
what
one
more
question
yeah.
A
A
Yep
yeah
I
mean
so
we
we
are
sorry,
so
the
question
is:
will
lift
move
more
towards
sto?
I
answer
is
absolutely
yes,
I
mean
like
we
obviously
have
had
envoy
for
longer
than
there's.
This
do
and
that's
kind
of
what
I
was
talking
about
before.
Is
that
it's
one
thing
if
you're
like
doing
this
from
scratch,
but
we
have
a
giant
system
to
migrate
like
hundreds
of
services,
millions
of
requests
per
second
like
it's,
not
something
that
we
can
just
you
know
basically
move.
A
So
the
goal
for
us
is
going
to
be
to
move
closer
to
pilot
I,
I,
don't
know
at
what
time
we
would
use
mixer
that
that's
a
separate
conversation,
but
the
cool
thing
about
SDO
is
that
the
components
are
actually
separate.
So
you
know
one
can
use
pilot,
but
not
mix
or
one
can
use
kubernetes
ingress,
but
not
something
else.
So,
yes,
the
goal
is
to
move
towards.