►
From YouTube: Replacing NGINX with Envoy in a Traffic Control System - Mark McBride, Turbine Labs, Inc
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Replacing NGINX with Envoy in a Traffic Control System - Mark McBride, Turbine Labs, Inc (Advanced Skill Level)
In late 2017 Turbine Labs migrated the proxy at the center of their traffic control product from NGINX from Envoy. In this talk Mark will outline the decision making criteria for embarking on this migration and the steps they took to make this transition transparent. He will also walk through the engineering effort, from packaging, deployment, monitoring, and committing new features back to the Envoy community. At the end he will describe the benefits of the new system, and talk about future improvements.
About Mark
Mark McBride is founder and CEO of Turbine Labs, building products that help teams manage Envoy at scale. Previously, Mark was services engineer lead at Nest Labs and Google, where he was responsible for the development of Nest’s server infrastructure that makes it possible for Nest customers to connect with their homes from wherever they are, and as an early developer on Twitter’s streaming API, delivering thousands of messages per second in real time to millions of users. During his time at Twitter, Mark managed developer productivity and led the web delivery, developer tools, and infrastructure test teams; he also worked with a variety of deploy pipelines and led development of some of Twitter’s early service migrations, which grew into a suite of tools used to migrate of millions of requests per second from legacy services to modern replacements.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
This
talk
is
about
how
we
moved
the
core
of
our
traffic
management
product
from
an
engine
X
based
proxy
core
over
to
envoy
over
the
course
of
about
six
months,
starting
last
late
last
summer
and
finalizing
very
late.
Last
year,
beginning
of
this
year
so
background
I
was
a
Twitter
from
2009
to
12,
when
Twitter
was
moving
a
lot
of
things
off
the
rails,
monolith
on
to
JVM
services
and
then
from
big
JVM
services
into
smaller
JVM
services
and
a
lot
of
what
became
sort
of
proto
cloud
native.
A
This
is
about
migrations,
and
so
we
want
to
kind
of
set
the
stage
here,
for
why
would
you
even
do
a
migration?
These
things
often
fail
they're,
really
hard.
This
talk
was
rated
as
advanced
skill
level,
which
in
hindsight
I
was
a
little
nervous
about
it
because
technically
this
stuff,
isn't
it's
not
rocket.
Science.
A
Socially
migrations
are
really
hard
to
pull
off
and
having
gone
through
a
bunch
of
pretty
big
ones,
they're
difficult,
not
in
the
same
way
that
doing
like
distributed
consensus
is
hard,
but
they
are
difficult
in
the
sense
that
you
have
to
get
a
lot
of
people
to
buy
into
a
lot
of
stuff.
You
have
to
do
risk
a
lot
of
things
and
from
a
process
perspective
and
a
political
perspective.
They're
much
more
difficult
than
a
lot
of
the
deeper
technical
challenges,
I
think
so
why
would
you
do
them?
A
They
are
one
of
the
only
ways
to
scale
ibly
get
tech,
debt
fixed
and
so
will
Larsen
who's.
A
manager
at
stripe
has
a
great
post
that
lays
out
the
case
that
most
systems
are
really
only
designed
to
grow
an
order
of
magnitude
beyond
their
initial
design
constraints.
Then
you
have
to
start
looking
at
other
systems
to
get
you
to
that
next
level
and
a
lot
of
people
don't
do
that
soon
enough
and
end
up
kind
of
grinding
out
that
last
little
bit
and
it's
painful.
A
So
this
talk
is
about
making
your
envoy
migration
easier,
but
a
lot
of
the
practices
and
the
principles
apply
to
general
migrations
and
one
of
the
nice
things
about
envoy
as
a
fluid
proxy
in
front
of
things
is,
it
makes
all
kinds
of
migrations
easier.
It's
an
extra
benefit.
A
disclaimer
I
didn't
do
much
of
the
work
outlined
here,
my
name's,
not
on
here
but
I'm
here
to
more
or
less
do
a
read
out
an
analysis
of
all
the
stuff
that
our
great
engineers
did
along
the
way
turbine
Labs.
A
The
company
we
started
after
coming
out
of
Twitter
and
nest
is
a
traffic
management
application.
That's
designed
to
give
a
lot
of
the
practices
we
built
at
Twitter
and
put
those
into
the
hands
of
the
wider
engineering
population
proxies
are
the
technical
core
of
that
and
so
we're
in
a
pretty
unique
position
as
being
a
vendor
selling
stuff,
but
also
a
consumer
proxy
tech
and
kind
of
connoisseurs
of
the
differences
between
H,
a
proxy
engine,
X
envoy
various
things
so
interesting
place
to
be.
A
We
have
lots
of
opinions
this
on
board
migration
sums
up
a
lot
of
them
so
step.
One
here
and
sort
of
kind
of
sensing
inception
of
the
company
was
we
are
going
to
use
a
proxy
technology
that
exists
at
the
time
in
late
2015,
the
reasonable
options
who
are
engine
X
and
H
a
proxy
we
built
the
product
on
top
of
engine
X.
This
is
our
early
plan
straight
out
of
2015
and
comments
on
this
slide.
It's
a
very
bottom.
This
is
kind
of
a
train
wreck
and
it
was
really
so
about
the
slide.
A
This
is
throughout
the
stock.
I'm
not
saying
engine
X
is
a
bad
piece
of
software
by
any
means
it
served
us
very
well.
This
was
mostly
about
the
graphics
that
I
did,
but
also
as
we'll
go
through
the
kind
of
rough
spots
of
the
architecture
we
built
it
was.
It
was
pretty
high
friction
for
us
trying
to
deal
with
the
cloud
native
type,
dynamic,
high,
cardinality,
frequently
changing
environment.
A
This
is
the
initial
product
we
built
and
notice.
It
looks
almost
exactly
the
same
as
this,
but
we
had
somebody
better
do
pictures.
So
it's
not
a
trainwreck
ish
circles,
and
this
worked
so
we
built
a
lot
of
stuff
on
engine
ax
and
we
got
the
system
working.
It
was
great.
You
could
do.
Bluegreen
deploys
Canaries
header
based
routing,
it
was
super
cool.
So
why
move
and
I
think
if
you're
looking
at
practice
today,
a
lot
of
you
have
who's
using
engine
X
on
kubernetes
and
good-good
H
a
proxy.
A
Do
you
say
to
H
a
proxy
or
how
a
proxy
Joey
ends
now?
Okay,
I,
don't
know
how
to
pronounce
it.
So
there's
a
bunch
of
things
we
looked
at
in
our
architecture
as
we
were
considering
this
moved
envoy.
At
about
the
same
time
we
started
our
company,
Matt
Klein
had
left
Twitter
to
go
to
lift
and
one
of
his
first
tasks.
A
The
the
first
thing
is
that
the
base
routing
primitives
are
not
are
not
the
greatest,
so
engine
X
has
locations,
you
can
kind
of
route
traffic,
but
this
notion
of
probabilistic
releases
like
five
percenter
traffic
to
a
thing
and
95%
to
the
other
and
gradually
shift
that
across,
like
a
sliding
blue-green
deploy
like
wasn't
in
there
and
extending
it
meant
shipping
a
custom
module.
So
that's
like
strike
one.
We
had
to
ship
a
custom
engine
X
build
to
get
our
functionality
out
to
people
which,
from
an
adoption
perspective,
is
kind
of
a
scary
thing.
A
The
other
thing
is
that
the
upstream
constructs
they
had
weren't
super
full-featured
in
that
a
lot
of
systems
expect
an
upstream
to
really
be
a
host
important
to
wait
and
that's
good
from
for
some
things,
but
with
modern
systems
you
get
a
lot
of
labels
and
metadata
and
tags
etc
on
instances.
So
you
know
more
about
that
workload
than
just
where
it
lives,
and
we
wanted
to
be
able
to
route
based
on
the
version
of
software
deployed
or
what
get
branch
it
came
from
or
whether
this
was
a
QA
or
a
product
system.
A
Another
thing
is
that
observability
was
important
for
us
from
the
get-go
when
you're
trying
to
do
routing
across
multiple
versions
of
stuff.
You
want
to
know
what
the
behavior
differences
are
between
multiple
versions
of
stuff
and
so
getting
the
good
stats
out
of
any
proxy
really
at
that
time
meant
having
a
log
tailor
that
would
parse
the
access
logs
and
give
you
the
stats
and
send
them
off
to
your
system,
so
you
could
parse
them
and
so
having
these
like
necessary
agents.
A
Co-Located
with
in
genetics,
meant
that
we
had
to
ship
this
whole
ball
of
stuff
out
to
all
of
our
customers
and
internally,
it
wasn't
as
simple
like
lightweight
sidecar.
Here's
the
whole
proxy
operationalized
in
this
like
very
non
idiomatic
docker
single
process
system
and,
like
you,
should
just
install
this
it'll
just
work,
don't
worry
about
it,
it
did
work,
but
people
still
worried
about
it
and
then
the
last
limitation
was
engine.
X
was
really
at
the
time
built
for
TCP
load,
balancing
in
HTTP
1,
which
was
you
know
that
covers
a
majority
of
traffic.
A
So
this
is
what
we
actually
can
build
with
envoy,
and
so
this
is
a
much
cleaner
break
of
there's
envoy
and
there's
management
stuff
and
we'll
go
through
the
reasons
why
why
that's
possible
and
whether
it's
a
better
solution
both
for
us
as
a
vendor
for
people
as
a
consumer
for
operators?
You
know
for
everybody,
everybody
concerned,
so
the
benefits
we
looked
at
as
envoy
had
well.
A
These
are
the
benefits
envoy
has
now
we'll
go
through
how
we
got
there
more
sophisticated
routing
when
you're
working
in
the
service,
mesh
style,
environment
or
even
at
the
edge
there's
just
a
lot
more.
You
can
do
now
with
modern
systems
that
have
tags
labels
more
information
about
workloads
than
you
had
before.
Changes
are
easier
to
get
in.
This
is
just
like
a
non-functional
feature,
but
our
ability
to
get
changes
made
directly
to
envoy
has
been
much
better
than
our
ability
to
push
a
patch
to
nginx,
and
it's
not
really
a
knock-on
engine
X.
A
So
much,
but
it's
an
older
project,
there's
more
inertia
there.
It's
hard
to
make
fundamental
changes
upstream.
Definitions
are
much
much
richer.
Envoys
notion
of
an
endpoint
has
arbitrary,
GRP
c
structs,
so
you
can
shove.
Whatever
you
want
in.
There
is
an
endpoint
provider
which
is
great.
The
whole
config
via
service
design
is
pretty
novel
for
a
proxy.
So
for
people
not
as
familiar
with
envoy,
you
can
still
configure
it
with
files,
but
it
also
defines
G
RPC
interfaces
that
will
deliver
config,
and
so
you
implement
the
service.
A
You
point
envoy
at
it
and
it
will
say,
hey
give
me
my
clusters
give
me
my
listeners
give
me
my
routes
and
if
they
change,
let
me
know,
and
so
your
entire
Envoy
fleet
is
connecting
to
a
management
service
getting
config,
which
means
you
have
a
central
point
of
control
for
Envoy
config
changes,
which
is
great.
The
other
thing
that's
pretty
recent
is
the
ability
to
stream
logs
over
G
RPC
as
well.
A
So,
instead
of
having
to
run
a
log
tailor,
you
implement
a
log
sync
and
you
can
do
all
your
processing
central
lights
as
well
and
there's
a
wide
range
of
protocol
supported,
so
not
just
HTTP
one
through
two
I
guess,
that's
all
they're
up
to
now,
but
there's
Redis
dynamo,
it's
a
couple
of
others
and
adding
new
new
protocols
is
pretty
straightforward.
So
this
is
all
good.
A
We
looked
in
in
mid-summer
as
like
sounds
great,
but
features
aren't
necessarily
enough
to
kind
of
bet
a
company
or
production
site
on
so
you
get
into
like
the
non-functional
requirements
of
a
migration
I'm
gonna.
Do
this
thing
technically,
it
seems
better.
There's
a
lot
of
features
I
want,
but
if
it's
only
gonna
be
up
fifty
percent
of
the
time,
that's
not
great,
and
if
it's
not
supported,
that's
not
great,
and
so
we
went
through
these
as
the
is
a
qualitative
aspects
of
an
envoy
migration.
A
So
how
supported
is
it
is
a
lightweight
if
we're
going
to
have
people
deploy
this
aside
cars,
it's
important!
Is
it
fast
enough
if
you're
going
to
introduce
this
as
a
hop
between
every
service?
Call,
you
don't
want
it
to
be
super
slow
and
then
predictable.
Latency
is
also
pretty
important.
So
the
way
to
experience
a
Twitter
working
with
proxies
written
on
the
JVM
and
for
the
most
part
performance
is
very
good,
but
you
have
surprising
outliers
when
there's
full,
GCS
and
stuff
like
that.
That
can
be
not
good
for
your
long
tail
agency.
A
So
with
envoy
now,
I
look
at
the
last
release,
so
between
one
six
and
one
five
on
Envoy,
so
71
committers
on
the
project,
it's
maintained
by
pretty
big
companies
and
us
so
lift
Google,
Apple
and
us
with
eleven
people.
It's
kind
of
an
outlier,
and
so
these
aren't
people
that
commit
these
are
people
who
are
maintainer
x'
of
the
project.
50%
of
the
commits
actually
greater
than
50%
came
from
people,
not
in
those
orgs.
So
there's
a
pretty
diverse
set
of
people
committing
the
other
nice
thing
about
usage.
A
Is
that
it's
not
just
use
it
lifts
it's
fairly,
unique
that
it
sprang
from
lift
open
sourced
already
and
full
production.
There
is
no
real
incubation
period
where
it
wasn't
prod
ready,
but
the
things
Google
is
doing
with
envoy
are
very
different
from
a
use
case
perspective
and
lift
and
I
put
Verizon
is
doing
with.
It
is
very
different
than
what
lift
it's
using
it
for
so
there's
a
pretty
broad
set
of
use
cases
it's
applied
to
that
made
us
more
comfortable.
A
So
if
anybody
follows
Matt
the
Envoy
author
on
Twitter,
you
know
he
has
strong
opinions
about
C++,
still
being
cool
enough
to
write
software
in
it's
controversial,
apparently
so
I
mean
the
benefit.
C++
gives
envoy
like
not
directly.
You
can
write
bad
code
in
any
language,
but
it
has
a
pretty
small
footprint
resource
wise
that
makes
it
very
suitable
for
running
in
a
sidecar
manner.
There's
no
garbage
collection,
which
means
it
has
a
very
predictable
latency
profile,
and
it's
fast
enough
like
if
you
stack
it
up
against
Engine,
X
or
H
a
proxy.
A
Now
it's
not
going
to
win
a
benchmark
that
you
contrived,
but
for
the
flexibility
and
other
like
non-functional
or
even
other
functional
requirements.
It
has
it's
fast
enough
to
trade-off
for
the
additional
management
capabilities
you
get,
and
so
our
process
last
summer
was
really
watching.
That
curve
of
support
community
speed,
footprint
and
saying
like,
is
this
the
right
time
for
us
to
move
from
engine
X
to
envoy
and
that
all
converged
mid
to
late?
A
But
we'll
go
through
go
through
this,
so
things
you
consider
after
you've
decided.
This
is
a
good
idea,
so
Engine,
X
and
H
a
proxy
have
a
lot
of
features.
They've
been
around
for
a
long
time
and
you
probably
don't
use
all
of
them,
but
you
need
to
make
sure
that
there's
there's
some
way
for
you
to
ensure
that,
like
your
envoy,
deploy
is
going
to
match
up
except
ibly,
with
your
engine,
X
and
H,
a
proxy
deploy
and
the
way
we
did
that
is.
A
We
started
off
with
a
higher
level
of
abstraction
for
config,
and
so,
instead
of
our
source
of
truth
being
engine
X
comp
distributed
everywhere,
we
had
a
data
model
that
led
us
to
find
the
attributes
we
needed
in
an
in
genetics
comp
file
and
that
allowed
us
to
come
up
with
an
equivalent
envoy,
config
and
match
those
things
up.
And
so,
when
we
rolled
out,
we
knew
that
envoy
would
behave
identically
to
the
nginx
based
system.
A
That
was
there
before
the
other
thing
is
there
just
may
be
straight
feature
gaps
that
you
can't
you
can't
address,
and
so
you
really
have
had
three
choices:
contribute
to
core
highly
recommended
push
to
stuff
back
to
the
open
source
project.
So
everybody,
everybody
wins
envoy-
has
a
really
extensible
filter
mechanism.
So
very
much
in
the
vein
of
like
working
with
with
Metis
or
other
filter
based
network
pipelines.
You
can
implement
your
own
stuff
pretty
easily.
A
There
has
Lewis
support
now,
so
you
can
script
it
up
for
prototypes
and
then
the
higher
level
abstraction
thing
we
did
and
our
approach
was
when
we
found
gaps,
we
just
worked
to
commit
them
back
to
open
source
envoy,
which
was
great.
It
was
pretty
easy
to
get
full
request
through
there's
high
code
review
standards,
but
that's
a
good
thing,
not
a
bad
thing,
and
so
we
committed
pretty
pretty
substantial
things
like
the
ability
to
build
on
a
Mac,
not
in
a
docker
container.
A
Just
super
useful,
subset
load,
balancing
arbitrary
meditate
on
endpoints,
some
pretty
big
things
we
got
in
place,
I
think
fairly
quickly.
The
next
thing,
their
operational
parity,
is
definitely
a
thing,
I
think
for
a
lot
of
migrations.
People
go
through.
It's
like
I.
Have
this
running
on
my
laptop.
This
is
fantastic.
A
We
should
just
go
to
prod
and
the
more
seasoned
slash
older
people
and
the
team
will
be
like
you
know
how
you're
gonna
deal
with
logs
right,
like
oh
yeah,
that's
a
thing,
and
so
on
voice,
pretty
good
about
most
of
the
stuff
like
getting
along
shout
pretty
standard
metrics.
It
has
an
admin
server.
That
gives
you
a
pretty
wider
range
of
stats.
If
you're
used
to
dealing
with
some
of
the
other
proxies
and
trying
to
get
internal
proxy
stats
out
envoy
is
much
more
approachable.
A
Let's
figure
out
to
do
alerting
packaging
and
deployment
are
still
pretty
new
for
envoy
there's
a
docker
image
and
not
much
else,
but
it's
not
that
hard
to
roll
your
own.
At
this
point
and
the
process
management
is
a
thing
too,
that
it
doesn't
come
with
like
a
built-in
systemd
integration.
It
is
a
docker
container,
so
you'll
have
to
figure
that
out
too.
A
Let's
plan
for
success
and
when
I
flip
the
switch,
it's
gonna
be
great,
then
it
isn't
so
one
of
the
things
we
really
pioneered
at
Twitter
was
incremental
rollout
of
all
the
things,
because
I
never
worked
so
sliders
are
better
than
switches
in
almost
all
cases
when
you're
thinking
about
this
migration.
So
how
can
I
do
X
percent
of
this
thing
as
a
test?
How
will
I
roll
it
that
quickly
when
it
fails?
How
easy
is
it
and
how
many
people
can
do
the
ramp
up
ramp
down?
A
A
It's
not
that
you
shouldn't.
You
know
it's
not
just
about
collecting
those
things
and
observing
them.
But
having
like
having
fallback
points
that,
if
latency
gets
this
bad,
we
will
roll
this
thing
back.
It's
much
easier
to
know
that
stuff
upfront,
then,
during
the
migration,
when
people
are
worried
and
emotions
run
a
little
hotter
and
you
have
to
have
an
argument
with
somebody
to
roll
it
back,
because
you
think
50
milliseconds
is
fine
and
they
think
it
should
be
30.
So
having
those
set
points
to
find
upfront
makes
the
whole
process
quite
a
bit.
A
So,
with
all
that
in
place,
our
first
step
and
also
on
the
incremental
note,
was
we
didn't
go
sort
of
full
envoy
from
the
beginning
like
if
you
look
at
this
picture
of
phase
1,
it
just
looks
exactly
like
the
old
picture.
We
had
it's
just
instead
of
the
management
agent,
we
implement
the
next
es
server
and
instead
of
what
we
still
had
a
log
parser.
So
there's
a
new
log
parser
that
parses
envoy
logs,
but
it's
still
a
log
parser.
The
nice
thing
about
this
is
operationally.
It
is
the
same
thing.
A
There's
a
ball
of
a
container.
We
ship
this
one
has
envoy
the
last
one
had
nginx,
nobody
knew
the
difference
and
it
was
really
just
a
bump
on
the
container
version.
Like
change
your
deployment
to
use
this
new
thing
and
you'll
have
envoy
tada,
and
this
worked
after
a
lot
of
work
along
these
things.
So
our
process
kind
of
beginning
to
end
to
get
to
phase
1
was
a
lot
of
like
filling
the
feature
gaps
we
had,
and
this
wasn't
that
envoy
just
had
a
pour
feature
set.
A
It's
just
that
we'd
built
a
lot
of
stuff
into
engine
acts
that
we
didn't
want
to
give
up.
So
we
kind
of
looked
at
what
all
had
we
added?
What
did
envoy
have-
and
there
were
the
beginnings
of
a
lot
of
those
features
that
we
just
fleshed
out
implementing
an
XDS
service
which
has
been
something
we
push
on
for
for
quite
a
long
time,
because
there's
several
of
the
XDS
services
and
varying
levels
of
difficulty
and
implementing
them
updating
the
log
parser.
A
This
was,
like
surprisingly,
a
reasonable
amount
of
work.
I,
don't
remember
why,
but
it
was
packaging
deployment.
So
this
is
a
lot
of
engine.
X
has
a
pretty
you
know
straightforward,
like
configure
make
we've
entered
it.
We
built
a
binary
shoved
it
into
a
container
with
envoy.
It
is
easiest
we
found
to
get
the
shipped
Envoy
container
yank
the
binary
out
of
it
plop
it
into
your
container
and
go
or
you
can
just
extend,
extend
their
container
this
one
actually
wasn't
too
bad
did
a
lot
of
testing
the
fun
thing
about
proxies.
A
Is
they
are
very
hard
to
test
with
a
live
traffic
workload,
because
unless
you're
doing
really
sophisticated
capture,
we
play
a
production
traffic
at
scale
the
best
place
to
test
these
things
is
actually
in
production
which
can
be
nerve-wracking
but
sliders,
be
switches,
be
prepared
to
rollback
we
didn't
actually
have
to
with
this.
We
did
a
lot
of
testing
outside
production
and
it
happened
to
work,
but
we
were
prepared
after
that,
canary
new
proxies,
so
we
had
an
e
lb
for
our
site.
A
We
have
a
bunch
of
containers
behind
that
that
we're
running
engine
X
we
upgraded
one
of
them
to
run
on
voi.
Looked
at
stats
across
the
whole
fleet.
See
if
latency
is
the
same
resource
usage
is
the
same.
Error
rates
are
the
same
run
all
of
our
qualification
tests
with
our
app
to
make
sure
behavior
was
the
same.
The
400s
for
a
requester,
still
four
hundredths
404s
to
four
fours.
This
was
all
good
and
then
fully
rolled
out
new
proxies.
A
This
whole
process
took
us
like
from
the
time
we
headed
well
from
the
time
we
started.
The
time
we
rolled
out
was
probably
five
months.
A
lot
of
that
was
spent
in
steps
one
and
two
six
and
seven
were
pretty
smooth
I
think
we
kept
their
canary
running
for
a
week,
or
so
there
were
really
no
issues.
Then
we
decided
to
fully
roll
them
out.
A
That
was
great,
it's
been
very
smooth
and
then
once
we
had
that
transparent
swap
in
place
like
the
ergonomics
of
deploy
were
exactly
the
same
with
Engine,
X
and
H
a
proxy.
Then
we
took
that
next
step
of
taking
advantage
of
the
XDS
services
to
really
fully
separate
that
management
playing
from
envoy
itself.
So
this
is
where
we
ended
up,
and
we
are
much
much
happier
the
envoy
we
we
run
now
or
the
week
we
can
run.
You
know
run
further.
People
is
just
stock.
A
Open-Source
bring
your
own
pointed
at
our
thing
and
it'll
get
itself
configured,
and
this
means
that
we
don't
have
to
ship
our
own
container
of
Envoy.
We
don't
have
to
worry
about
patches,
somebody
else
has
to
which
is
better
for
us.
Somebody
else
worries
about
it,
but
it
also
means
that
people
can
use
their
own
build
pipeline,
their
own
packaging,
their
own
deployment,
their
own,
etc
and
just
point
to
the
management
server
and
get
benefits.
A
The
other
thing
that
we've
used,
that's
pretty
new
is
the
access
log
service,
which
is
great,
so
we
don't
even
have
to
have
a
telemetry
sidecar
on
envoy,
we
just
say:
hey
ship,
your
access
logs
to
us,
we'll
take
that
and
forward
it
to
STATS,
dear
influx
or
prometheus
or
whatever,
and
you
don't
have
to
worry
about
anything
other
than
just
deploying
envoy
and
getting
it
configured
properly.
This
has
been
really
great,
really
great.
A
And
that's
where
we
are
today
fully
deployed
envoy
XDS,
it's
really
good.
So
here's
lessons
learned
and
after
after
six
months
of
this
stuff
and
I
would
say
six
months
of
this
one
at
Twitter
for
a
long
time.
Our
job
was
basically
doing
these
things
over
and
over,
and
there
was
like
a
little
checklist
of
here's
how
we're
going
to
move
some
slice
of
the
rails,
monolith
onto
the
JVM,
and
so
these
these
things,
like
these
approaches,
apply
to
lots
of
different
kinds
of
migrations.
A
The
same
approach,
D
risk
incremental
know
when
you're
gonna
have
to
roll
back,
know
that
you're
going
to
know
when
you
have
to
roll
back
and
just
be
prepared
to
do
that
several
times
along
the
way.
The
nice
thing
is:
if
you
get
that
as
a
common
practice,
then
the
refactoring
of
the
system
becomes
quicker
and
I.
Think
your
developer
productivity
and
the
ability
you're
able
to
get
out
of
your
system
increases.
A
The
other
thing
we
learned
here
like
a
third
bullet
cluster
management
and
listener
management
for
Envoy,
was
interesting
for
us
where
cluster
management
is
almost
always.
You
have
some
sort
of
truth,
some
source
of
truth
that
knows
about
clusters
and
endpoints
and
you're,
really
just
mapping
that,
from
whatever
the
native
implementation
is
into
envoy,
speak
so
in
kubernetes
you
do
coop
control
get
pods.
You
can
convert
that
to
an
EDS
response
pretty
easily
and
it's
really
just
glue
code.
A
So
none
of
this
is
rocket.
Science
like
none
of
this
is
like
deep
system
stuff.
You
don't
have
to
read
like
ACM
papers.
Really,
but
it's
a
long
project,
it's
a
high-risk
project.
It's
hard
to
keep
people
excited
along
the
way
to
let
you
keep
doing
the
thing
to
roll
it
out
and
a
lot
of
migrations.
Gonna
get
bogged
down
with
three
months
and
with
management.
There's
somebody
else
saying
like
I,
don't
know
that
we're
actually
ever
gonna
finish
this
and
I
don't
see
the
benefits.
A
If
we
do
so,
it
seemed
to
be
risky
for
different
reason,
and
so
that's
another
reason
that,
like
incremental
ISM,
showing
like
quick
wins,
the
ability
that
you're
still
moving
forward
is
great.
The
other
thing
is
a
lot
of
the
stuff
we
built
along
the
way,
I
think
are
pretty
useful
outside
our
project,
so
we'll
actually
be
open
sourcing.
This
ball
of
management
Bastion
over
the
next
few
weeks.
So
this
is
a
drop-in.
You
know,
implementation
of
c.d.s
and
EDS
that
will
bridge
console
ec2
things
there
into
cluster
discovery.
A
So
you
don't
have
to
write
your
own
if
you're,
using
those
environments,
it'll
also
read
from
JSON
or
EML
files,
which
is
probably
easier
than
writing
your
own
cdf
service.
It
has
an
ALS
implementation
that
forwards
to
the
big
metrics
backends
yeah,
that's
all
the
all
the
big
ones,
I
think
and
then
we'll
set
up
a
simple
LDS
and
RDS
config.
To
give
you
like
a
fully
working
envoy.
A
If
this
LDS
and
RDS
isn't
sophisticated
enough,
you
can
use
a
combination
of
static
and
dynamic
config
to
do
customization
and
then
the
upgrade
path
to
our
product,
which
does
much
more
sophisticated
routing
and
listener
management,
is
just
provide
an
API
key
and
it'll
use
us
as
the
canonical
store
instead
of
LDS
and
Rd.
Has
the
other
thing
we'll
open
source
is
what
we
call
envoy
simple
right
now
to
run
envoy.
You
need
a
bootstrap
config
that
you
provide,
so
you
almost
necessarily
have
to
stand
up
your
own
container
to
get
it
configured.
A
This
does
a
very
simple
go
template
file
that
lets
you
configure
the
stuff
via
environment
variables.
It
gets
you
to
a
head
start
on
rolling
out
an
envoy
managed
by
XDS
in
production,
so
pointed
XDS,
configure
the
admin
server
configure
the
log
level
and
go,
and
so
this
is
a
pretty
dramatic
speed
up
from
trying
to
figure
out
the
Envoy
packaging
stuff.
A
A
B
A
It
depends
on
the
type
of
change
short
answer.
Yes,
why
not?
There
are
a
wide
variety
of
production
changes
that
envoy
helps
roll
out.
We've
seen
people
use
it
from
migrating
from
system
a
to
B,
where
a
is
usually
not
kubernetes
and
B
is
kubernetes,
but
being
able
to
load
up
endpoints
that
span
multiple
different
runtimes
and
being
able
to
gradually
shift
traffic
from
like
an
ec2
console
runtime
over
into
kubernetes
is
really
useful.
We
use
if
our
own
software
releases
to
always
do
Bluegreen
deploys
for
those
like
incremental
Bluegreen.
B
A
Is
this
a
question
is
how
are
you
on
using
invoice
checks
versus
kubernetes
liveness
probes?
That
is
like
a
whole
talk,
I'm,
not
a
big
believer
in
health
checks
in
general,
which
is
probably
a
pretty
hot.
Take
the
the
downside
with
health
checks
is
that
they
tend
to
never
be
full
fidelity,
so
health
check
is
like
a
I'm.
Listening
on
a
port
I
can
accept
traffic,
which
is
cool,
but
does
not
mean
that
the
service
behind
it
is
working
at
all
and
so
on.
Voice
supports
some
pretty
interesting.
A
Liveness
constructs
internally
like
if
it
starts
receiving
error
responses
from
a
from
an
endpoint.
It
will
eject
it
from
the
pool
and
then
retry
again
later
to
see
if
it's
healthy
and
so
I
think
in
most
cases,
I'm
more
in
favor
of
not
even
using
the
health
checks
at
the
cluster
level
and
on
void,
but
using
outlier
detection
and
it's
sort
of
built
in
version
of
health.
C
A
So
so
in
void
does
not
do
static
file
serving
right
now,
which
is
which
is
a
big
thing,
that
engine
X
does
very
well
at.
There's
also
I
think
a
larger
overall
list
of
modules
for
engine
X,
particularly
around
mail
handling
and
some
other
things
so
I
think
short
answer
is
yes,
there
are
cases
where
you
would
want
to
use
engine
X
I.
Think
envoy
is
catching
up
very
rapidly
on
almost
all
fronts
and
surpassing
it
on
a
number
of
others.
A
B
A
So
the
question
is
I
briefly
mentioned:
test
in
production.
Have
I
ever
had
to
do
some?
Our
original
product
was
very
much
a
test
and
production
product.
We
are
strong
believers
in
test
and
prod
and
that's
like
a
whole
other,
probably
conference,
but
the
our
core
thesis
is
that
going
to
production
is
another
form
of
test
and
like
if
test
production
makes
you
feel
dirty,
then
using
validation
and
production
is
probably
a
little
better.
A
But
it's
the
same
thing
like
you're
deploying
new
code,
the
chance
of
there
being
bugs
in
it
is
greater
than
zero,
and
you
should
tool
up
to
be
able
to
detect
those
and
find
them
and
respond
to
them.
And
a
lot
of
the
things
like
the
incremental
blue-green
release
are
exactly
a
testing
prod
I'm,
going
to
put
a
thing
out,
and
it's
not
that
I
just
leave.
It
I
want
to
observe
metrics
like
the
differences
between
a
test
and
a
control,
and
if
the
test
looks
good
relative
to
the
control,
then
I
can
proceed.
A
Yes,
so
the
question
is
I
said
it
was
fast
enough
and
how
does
it
compare
to
other
proxies
I?
Think
the
general
stance
in
the
envoy
slack
whenever
it's
come
up.
Is
that
there's
no
good
standard
benchmark
to
use
right
now,
which
may
be
I,
don't
know
there
have
not
been
people
complaining,
I
think
the
generally
accepted
Delta
is
somewhere
around
10%,
but
it's
hard
to
come
up
with
the
benchmark
for
a
given
workload
at
you
know
a
client
or
production
site
to
do
an
apples-to-apples
comparison.
A
You
know
eBay
uses
it
at
the
edge
now
for
a
lot
of
their
ingress
traffic.
They're
kind
of
a
big
deal
so
I
think
it
works,
and
I
think
that
there
was
no
real
inclination
to
look
at
replacing
like
their
Hardware
load.
Balancers
was
something
different,
so
I
think
that's
a
pretty
big
like
vote
of
confidence
and
there's
other
companies
that
are
doing
like
pretty
high
scale
things
where
they
care
about
density
from
a
sidecar
perspective.
It's
interesting
that
fast
enough
is
really
about.