►
From YouTube: Running Envoy at the Edge - Derek Argueta, Pinterest
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running Envoy at the Edge - Derek Argueta, Pinterest
Migrating infrastructure can be hard, but we want to help you accomplish it. In this talk, we will walk you through the story of how we operationalized and deployed Envoy as the edge load balancer for Pinterest, and all the fun learnings along the way. We will demonstrate how we leverage Envoy's powerful stats subsystem to build confidence in the load balancing tier and aid debugging for all engineers. We will also discuss some of the upstream contributions we have made and why we needed them, as well as cover the custom extensions we wrote to accomplish on-par feature parity with our previous load balancing system. You will learn how we debug, profile, and develop on Envoy. Lastly, we will lay out the vision for Envoy in the service mesh use-case at Pinterest and provide information about how we plan to get there.
A
Hi
everyone
I'm
Derek
I,
am
a
site
reliability
engineer
at
Pinterest,
where
I
work
on
the
traffic
engineering,
team
and
I'm
here
to
talk
about
our
edge
deployment
of
envoy
when
I
first
got
to
Pinterest
rolling
out
on
boy
as
our
load
balancer
in
HD
router
was
one
of
the
first
things.
I
did
sounds
really
fun
journey.
If
we
started
working
that
out
in
about
March
of
this
year
and
as
of
summer,
our
load
balancers
are
a
hundred
percent
running
on
boy,
so
I'll
request
hitting
Pinterest.
A
We
go
through
on
volume,
a
bit
of
agenda,
so
we
talked
about
about
the
migration
process.
In
particular,
I
want
to
spend
a
book.
This
talk
talking
about
the
subset
load,
balancer
and
some
sophisticated
routing
scheme
that
we
did
for
serving
the
website
duty.
Some
interesting
requires
that
we
adopted
from
our
previous
load
bouncer
it's
a
bit
of
background
as
to
our
stack,
so
we
run
well.
We
use
multiple
seeding
vendors
in
front
of
our
edge
deployment,
just
interleaved
into
one
large,
coherent
Network.
A
Those
went
to
classic
load,
balancers
serving
HTTP,
sorry
Larry,
seven
HTTP,
which
would
go
to
a
fleet
of
varnish
hosts,
which
did
the
path
based
routing
the
complexity,
balancing
all
those
sorts
of
things.
So
this
is
how
we
looked
at
this
point
last
year.
This
stack
service,
well,
I,
think
has
been
this
way
since
maybe
2012,
but
as
time
progressed,
we
started
running
into
many
issues
and
commuting
tech,
tech,
debt
that
came
time
to
rethink
how
we
serve
Pinterest.
A
So
some
of
the
reasons
we
want
to
migrate,
our
classic
Gilbey's
we're
facing
scaling
issues
during
peak
traffic
of
the
week
we
start
experiencing
intermittent
connection,
failures
or
connection
timeouts
that
we
just
cannot
debug
cannot
find
out.
Why
and
it
turned
out
that
our
LB
was
hitting
capacity
limits.
The
options
80s
gave
us
were
to
either
shard
our
traffic
amongst
multiple
I/o
bees
or
adopt
their
new
network
load,
balancing
products,
their
layer
for
load
balancer.
That
sounds
awesome,
but
it
doesn't
support
a
less
termination
which
the
classic
EO
bees
were
doing
for
us.
A
So
that
meant
pushing
till
us
back
a
layer
in
our
stack,
which
would
be
great
but
varnish
does
not
support
TLS
termination
out
of
the
box.
The
varnish
way
to
do
till
extermination
is
to
run
another
proxy
in
front
of
Arnage
and
Terman
TLS
and
then
proxy
the
plaintext
to
varnish.
So
if
we're
gonna
operationalize
an
entirely
new
proxy,
maybe
if
you
just
rethink
using
there
are
a
number
of
other
issues
too.
So
varnish
is
a
great
caching
proxy,
no
doubt
about
that.
A
Fastly
uses
it
to
run
their
CDN,
but
the
path
that
we
were
serving
was
all
dynamic
traffic.
We
weren't
really
caching
anything,
so
we
weren't
really
using
varnish
for
its
strengths
and
what
is
meant
for
for
the
more
there
were
some
other
goodies
that
we
wanted
out
of
our
edge
to
the
balancer
dynamic
upstreams,
which
we
kind
of
hacked
onto
varnish.
With
this
scheme,
where
we
would
reload
or
regenerate
configuration
and
reload
varnish.
Every
time
we
detect
a
change
in
zookeeper.
A
There's
this
brittle
group
Goldberg
machine
that
when
it
broke,
was
terrifying
to
deal
with
the
observability
wasn't
great.
When
varnish
did
break,
it
was
very
hard
to
figure
out.
Why
and
lastly,
when
we
were
looking
for
a
new
proxy,
we're
sorry
yeah
those
reasons
to
migrate,
so
we
were
in
the
market
for
new
proxy
and
one
of
the
main
options
we
were
looking
at
was
open
rusty,
which
many
of
you
might
know
is
an
entry
next
module
that
embeds
Lua
scripting
into
nginx
runtime.
So
you
get
a
script,
a
bullet,
bouncer,
dynamic,
behavior
great.
A
We
were
talking
to
some
other
internal
teams
at
Pinterest,
specifically
our
service
framework
team.
This
is
the
team
that
writes
networking
code
in
each
language.
You
know
the
problem
that
on
boys
trying
to
solve
and
they're
like
hey,
there's
this
cool
proxy
that
we
want
to
use
as
a
service
mesh.
Maybe
you
guys
want
to
check
it
out
for
your
edge
proxy
they're
gonna
replace.
So
we
took
a
look
and
it
checked
all
our
boxes
that
we
needed
out
of
an
edge
proxies
voice,
Hillis
termination,
its
high
performance.
It
has
a
great
extension
model.
A
B
A
Were
ready
for
new
proxy,
we
talked
a
tomboy
that
was
a
great
fit
for
our
stack,
so
we
won
with
it
some
logistics
of
how
we
started
for
an
envoy.
We
don't
fort
the
project,
any
chord
changes
we
need.
We
try
to
contribute
through
the
open-source
process.
We
run
envoy
and
docker
containers.
We
do
use
hot
restart
across
containers
that
was
fun
to
get
working
and
works
really
well.
And
lastly,
our
control
plane
for
our
initial
on
WA
rollout
runs
as
a
sidecar
on
our
edge
hosts.
A
It's
a
bit
of
detail
about
the
control
plane
that
we
run
at
Pinterest.
We
have
a
default
service
discovery
setup
where
everything's
backed
by
zookeeper
when
a
host
comes
up,
they
registers
in
a
fumarole
node,
and
we
have
a
daemon
on
every
host
at
Pinterest.
That
will
read
that
do
to
keep
your
configuration
and
we'll
write
it
to
a
file
on
the
machine.
So
if
our
communication
with
zookeeper
breaks
or
never
partitions,
we
have
a
local
cache
copy
available,
sounds
very
similar
to
how
Envoy
caches
eds
intent.
A
So
the
second
part
is
the
routes,
clusters
and
listeners
which,
for
our
edge
deployments,
don't
change
very
often
so.
For
those
three
services,
we
decided
to
use
static,
json
files
on
the
machine
when
we
deploy
changes
to
those
it's
just
a
matter
of
swapping
the
files
and
on
boys
configure
to
watch
those
files
for
updates
for
EDS.
The
problem
was
simply
taking
those
cached
files
of
the
zookeeper
data
and
fitting
in
to
API
that
Amelie
understands.
A
So,
to
get
complete
feature
parity
with
varnish,
we
had
a
list
of
things
we
had
to
satisfy.
Most
notably
there
were
so
the
routing
schemes
and
request
manipulation
and
that
kind
of
comes
out
of
the
box
with
the
Envoy.
There
wasn't
any
work
to
be
done
there.
One
of
the
more
sophisticated
features
we
had
to
do
is
CDN
request
signing.
So
since
we
use
multiple
CD
ends,
the
CD
ends
will
sign
the
request
with
an
H
Mac
and
forward
it
to
our
proxies
and
the
proxies
and
code.
A
The
HTML
gonna
verify
that
ok,
yes,
this
request
came
from
CDN
since
we
used
multiple
CD
ends.
This
was
the
best
solution
we
had
in
a
simpler
world.
We
could
have
an
IP
list
of
known
IPS
for
the
CD
ends
and
to
say
have
a
security
group
that
says
it
has
to
come
from
these
IPS,
but
we
don't
live
in
a
simple
world.
Unfortunately,
so
this
is
an
adaptable
scheme
to,
however
many
CD
ends
we
on
board
in
the
future.
A
We
also
have
to
do
bot
detection,
our
SEO
or
SEO
team
with
imp
interests
are
heavy
clients
of
our
team.
They
want
to
know
when
is
Googlebot
crawling
when
is
Facebook
Bach
crawling.
Are
we
giving
them
five
hundreds
or
we
getting
to
hundreds?
So
we
did
a
bit
of
legwork
to
write
up
a
filter
that
builds
off
of
the
IP
tagging
filter
to
identify
bot
traffic
pretty
reliably,
since
it's
an
edge
proxy.
A
We
have
to
do
access
control
recognize
whether
the
requests
to
pit
envoy
is
an
internal
request,
whether
it
can
access
certain
services
and,
lastly,
what
I'm
gonna
be
talking
about
for
the
bulk.
Half
of
this
talk
is
fronting
an
AP
deploy
at
Pinterest,
so
that
Pinterest
we
have
one
service.
That
is
a
be
deployed.
That's
our
website,
dr.
pepper,
intercom,
I'm
gonna
dive
into
the
Okhrana.
Then
a
bit
after
we
yeah
there
were
a
couple.
A
Migration
hiccups
along
the
way
that
ran
into
envoy
has
the
configuration
to
handle
X
44
for
you
and
detect
client
IP
addresses,
which
is
great.
If
you
know
how
many
trusted
hops,
you
have
in
front
of
you
I'm
in
a
guard
case.
We
have
multiple
scenes
and
they
make
no
guarantees
about
how
many
trusted
hops
there
are.
So
we
adapted
our
CN
request,
signing
mechanism
to
also
proxy
the
trusted
client
IP
address.
A
Sometimes
the
trusted
top
would
be
one
for
some
of
the
C
ends
at
b2,
so
we
couldn't
rely
on
a
static
number.
Therefore
trusted
hops.
We
also
learned
the
hard
way
that,
on
point
by
default,
is
conservative
about
circuit
breaking.
When
we
were
first
running
envoy
and
it
was
pumping
about
1,000
RPS
compared
to
varnish,
which
was
pushing
out
six
or
seven
thousand,
we
were
wondering
what
we're
doing
wrong.
It
turns
out
that
by
default,
the
upstream
circuit
breaker
limits
at
1024.
A
If
you
don't
specify
anything,
and
since
we
live
in
in
HTTP
one
world,
it's
one
request
to
one
connection,
so
we
were
effectively
limited
to
1,000
requests
upstream
until
we
realized
that
we
were
missing
a
configuration
option,
hog
restart
across
containers
took
a
bit
of
work
to
get
going
since
for
har
we
starting
each
shared
memory
and
shared
networking
space.
That
was
pretty
easy
to
do
once
you
have
the
doctrine
configurations
to
share
the
memory,
space
and
network
space
and
then
along
the
way
we
learned
a
lot
of
services.
A
Don't
quiet,
respect,
HTTP
the
way
you
want
them
to
respect
HTTP,
we
found
a
service
that
was
using
a
library
whose
header
parser
was
still
case
sensitive,
so
when
they
should
be
in
front
and
lower
case
all
the
headers
that
service
broke.
There
was
another
fun
bug
where
an
older
version
of
finagle
or
some
buggy
version
that
we
were
using.
If
the
response
was
empty,
it
would
put
too
content
length
headers
in
the
response
which,
if
you
read
the
art
of
C,
it
says
you
can
either
collapse
them
into
one
header
or
just
call.
A
It
request
invalid
and
from
Akito
garnish
was
perfectly
fine
with
it.
But
when
Envoy
jumped
in
front
with
nodes
HTTP
parser,
it
was
started
rejecting
all
those
requests
which,
mostly
were
health
check
requests.
So
these
services
just
started
failing
health
checks
for
no
reason
that
was
a
ton
of
fun
to
debug,
not
really
so
just
a
bunch
of
really
fun
interesting
ways
that
things
can
break.
When
you
replace
a
north-south
proxy,
you
learn
the
hard
way
that
you
have
a
lot
of
inherent
dependencies.
A
You
don't
know
about
so
it's
critical
to
try
to
get
as
close
on
par
functionality
as
you
can,
even
if
you
think-
and
this
isn't
gonna
make
a
difference-
someone
downstream
is
depending
on
that
I
think
they
call
it
Hiram's
law
of
sovereign
sneering
that
anything
that
is
observable.
People
will
depend
upon
so
getting
everything
as
close
as
possible
is
critical.
Okay,
so
jumping
into
AP
deploys
so
with
the
website.
Dub
dub,
dub,
dub,
fitter
comm.
A
We
have
this
requirement
that
when
a
user
starts
the
recession
of
interest,
they
continue
to
see
the
same
version
throughout
their
browsing
session.
Pinterest
is
the
kind
of
application
where
people
don't
check
it.
Every
five
minutes
they
typically
sit
down,
pour
a
glass
of
wine
and
start
scrolling
through
boards
of
interior
design
and
gardening
in
recipes.
It's
very
meditative
therapeutic
process,
so
we
want
make
sure
we
don't
disrupt
that
throughout
their
browsing
session.
So
once
the
user
starts
their
session,
we
want
to
keep
serving
in
the
same
version.
A
A
It
goes
to
one
of
the
versions
and
the
user
can
still
access
the
old
version
through
a
request
header
that
the
browser
sets
or
our
JavaScript
client
sets,
so
that
B
version
is
still
available
and
then
once
their
session
ends
and
it
come
back
on
to
Pinterest,
then
they
gave
up
to
the
new
version
that
has
been
launched.
So
we're
running
these
two
versions
at
all
times,
which
has
been
fine
and
then
okay,
so
to
route
to
these
different
versions.
We
use
these
subsets
load
balancer
to
do
so.
A
We
attach
the
version
as
an
endpoint
metadata,
and
the
request
sends
the
version
header
that
comes
in,
and
so
we
wrote
a
little
glue
filter
called
a
header
to
metadata
filter.
If
you
have
seen
this
in
in
the
quorum
wondered
what
the
heck
is
this
for.
We
use
this
to
solve
the
problem
of
matching
request,
headers
to
endpoint
metadata.
A
Say
that
up,
however,
we
put
on
point
front
of
this:
a
B
cluster
and
something
everything
happened.
Cpu
usage
jumped
up
to
a
hundred
percent,
almost
a
hundred
percent
and
the
worst
part
about
this.
This
was
the
last
deploy
group
we
were
trying
to
get
upon
point
out.
We
had
like
three
successful
deploy
groups
go
out.
This
was
the
fourth
one,
the
main
one
DUP
DUP
DUP.
A
Here
we
go
and
CPU
skyrocketed,
so
we
had
to
take
it
out
and
do
a
bit
of
homework
into
what's
going
on,
so
we
ran
perf
top
on
Envoy
while
was
fronting.
This
deploy,
which
perf
top
is
a
wonderful
way
to
get
to
know
any
piece
of
software,
and
we
saw
this
this
function.
This
update
function,
taking
up
86%
of
CPU
time,
and
if
you
look
three
lines
above,
that
is
I,
think
says:
cluster
update
membership
or
up
to
membership
cluster
uh-huh,
okay!
A
So,
what's
going
on
here,
so
we
did
a
bit
more
diving
and
the
the
caveat
of
having
the
a/b
deploy
worked
away
was
where
we
have
two
versions
on
every
machine
meant
that,
however
many
web
hosts,
you
have
double
that
and
that's
actually
how
many
endpoints
envoy
has
so.
However,
big
the
fleet
was
double
that
and
that's
on
the
endpoints
and
a
fleet
of
that
size
is
constantly
experiencing
health
check,
churn
ho
start
dying,
hosts
are
launching
auto
skill
and
kicks
in
a
deploy,
happens
that
causes
health
checks
to
turn
and
at
that
scale.
A
What
you're
doing
when
you
update
the
health
check
is
that
Envoy
has
to
not
only
take
the
cluster
membership,
but
when
used
the
subset
load
balancer,
it
also
needs
to
update
all
of
its
subsets
and
keep
track
of
who
was
in
what
subset
he
was
still
alive,
etc.
So
this
constant
continual
churn
was
really
pecking
the
CPU,
and
so
we
had
to
come
up
with
a
better
way
to
do
this.
Downsizing.
The
fleet
is
not
an
option,
unfortunately,
so
I
forgot
how
to
how
to
make
this
work.
A
So
my
teammate
rubble
came
up
with
this
process
that
we
call
health
check
coalescing.
The
idea
here
is
to
batch,
deliver
updates
of
health
check
membership,
so
I
think
it's
not
by
default
now,
an
envoy,
but
essentially
you
can
configure
envoy
to
say,
check
the
membership
status
every
one
second
or
every
three
seconds
rather
than
having
it
constantly
happen,
because
we
didn't
need
sub
second
granularity
on
cluster
membership.
We
just
needed
a
general
update
at
least
every
couple
of
seconds,
this
dramatically
fixed
our
CPU
problem
and
we
were
ready
to
go.
A
A
So
these
are
the
constraints
that
we
had
to
abide
by.
In
our
new
deploy
scenario,
we
still
had
to
pin
users
to
a
certain
version
that
couldn't
change
the
model
that
we
were
going
for
was
what
we
call
staged
deploys
staged
base
deploys.
Essentially
you
divide
up
your
clusters
into
stages,
so
canary
staging
fraud,
the
usual
one
thing,
you're,
all
probably
familiar
with,
and
each
stage
should
have
a
virgin
write
and
handling
this
stuff
by
default.
An
envoy
is
pretty
easy
to
do.
A
Okay,
cool
not
getting
head
actually
so
the
way
that
we
solved
this
is
we
have
the
end
points
all
we
have
ets
served
which
has
the
stage
and
version
those
lot
go
up
into
zookeeper
and
the
sidecar
process
will
pull
this
down
and
writing
directly
ideas.
We
then
came
up
with
something
called
a
route
map.
A
So
that
way,
the
subset
lab
on
site
can
use
matching
and
also
this
route
map
which
gets
consumed
by
a
filter
rewrote
so
the
filter
that
we
wrote
does
is
whenever
request
comes
into
envoy,
if
there
it'll
try
to
sign
it,
a
routing
ID
if
one
does
not
exist,
this
is
just
unique
identifiers.
So
that
way,
when
this
trip
this
user
comes
back,
we
can
send
them
to
the
same
version
and
stage
without
having
the
flat
between
anything.
A
The
B
filter
then
collects
the
capacity
measurements
from
the
route
map
which
I
forgot
to
mention
is
injected
by
runtime
data.
So
this
the
sidecar
as
I
mentioned
it's
a
sidecar.
It
sits
on
the
host,
so
it
has
access
to
on
point
to
directly
pass
it
runtime
values.
So
the
filter
will
read
those
runtime
values
for
the
capacity
and
then
compute.
Where
should
this
request
go
and
then
at
once
all
those
computations
are
done.
A
It's
just
a
matter
of
setting
the
metadata
on
the
request
and
then
letting
the
request
go
on
its
merry
way
onto
the
subset
load,
balancer
to
get
matched
and
then,
of
course,
since
we're
having
a
filter
in
here,
it's
a
great
place
to
put
in
some
custom
metrics
so
that
we
can
see
what
the
decision
split
is.
How
many
requests
come
in
that
don't
have
a
routing
idea
at
all,
just
some
good
metrics
to
keep
track
of.
A
A
All
we
had
to
do
was
set
the
metadata
to
any
flexibility
that
we
needed
so
as
I
mentioned.
So
we
had
to
go
to
this
custom
solution.
So
envoy
has
a
feature
in
it
called
subset
localities.
We
can
set
the
weights
per
subset,
which
works
great
if
you
want
to
do
traffic
distribution
on
a
request
by
a
quest
basis.
A
But
as
mentioned,
we
have
this
constraint
of
pinning
not
just
requests
to
a
certain
version,
but
users,
so
we
found
that
the
localities,
the
subset
locality
weights,
didn't
really
work
for
us
because
it
would
shard
traffic
and
could
potentially
flap
users
between
some
requests
go
to
certain
stage.
Some
requests
go
to
a
different
stage.
I
wanted
to
keep
that
screens
cohesive
for
our
users.
A
Lastly,
my
problem,
favorite
part
about
envoy,
is
how
many
metrics
it
emits
I.
Did
a
quick
little
measurement,
I
think
it's
something
like
1300
metrics
out
the
box
for
one
of
our
deployments.
It's
fantastic,
my
favorite
thing
about
it.
These
are
some
things
that
we
learn
on.
We
have
the
basics
like
high
CPU
disk,
those
those
come
from
just
a
tea,
collector
module,
that's
querying
the
system
metrics,
and
then
we
also
learn
a
variety
of
failure
cases.
These
are
very
helpful
for
diagnosing
issues
in
production
when
things
break.
A
Why
they're,
breaking
and
fantastically
Hong
call
is
very
quiet.
Envoy
is
very
level
if
you
software,
as
Matt
mentioned,
what's
great,
is
seeing
other
people's
reaction
to
how
much
observability
we
have
so
envoy
by
default
has
metrics
on
a
per
cluster
basis.
That's
something
that's
practically
new
to
our
edge
tier.
So
the
fact
that
we
can
see
spike
in
five
hundreds,
exactly
which
upstream
cause
those
five
hundreds
that
has
been
freely
revolutionary
for
a
series
and
we
even
had
a
little
hot
boy.
I'm
gonna
be
calling
to
slack.
A
So,
looking
to
the
future,
so
we
have
on
play
running
100%
of
our
edge
deployment.
We
are
looking
at
the
service
miss
scenario.
Our
case
is
a
little
bit
interesting
because
we
are
a
thrift
shop
and
not
only
our
third
shop,
but
we
have
multiple
dialects
of
thrift
that
run
within
Pinterest.
So
if
we
reckon
out
how
to
get
all
the
standardized
and
figure
out
how
to
get
envoy
and
way
that
can
support
all
those
permutations
of
functionality
is
gonna
be
a
fun
challenge.
A
We
are
looking
at
the
possibility
of
integrating
memcached
in
some
way,
maybe
it
as
a
network
filter
and
we're
keeping
an
eye
on
the
my
sequel
and
Kafka
filters
very
closely,
as
we
are
heavy
users
of
those
systems
and
with
envoy
at
the
edge
we're
looking
at
moving
a
lot
of
our
authentication
authorization
functionality
out
to
the
edge
and
doing
everything
there.
That
requires
looking
doing
lookaside
request
other
services
in
thrift
so
we'll
so
we
need
both
out
there
at
foundational
work,
but
rogue
decided
to
push
that
in
this
coming
year.
A
B
A
B
A
It
was
mostly
operability
and
reliability,
so
he
asked
what
was
the
what
was
the
end
user
impact
of
running
on
boy
at
the
edge,
so
there
wasn't
much
in
terms
of
user
facing
things
like
users
didn't
necessarily
know
like
Pintrest
are
doing
Neela
balancer.
It
was
mostly
for
our
own
engineering
delight
in
that
we
had
better
observability
things
broke
less
often
when
they
did
break.
It
was
really
easy
to
debug.
A
B
A
B
A
Great
question,
so
we
have
two
different
deployments,
our
main
two.
So
yes,
if
we
expose
on
weight
directly
to
the
internet,
most
of
our
deployments
are
fronted
by
a
CDN
who
implements
the
request,
signing
words
to
verify
that
requests
are
going
to
the
CDN
so
for
the
most
part
stretch
of
traffic,
we
do
have
one
deployment
that
does
face
the
OP
internet
yeah.
A
On
a
secured
respective,
how
we
decided
it
yeah,
sir,
from
security
perspective
for
the
internet
facing
one.
Are
you
talking
about
yeah,
so
the
interface
facing
service
just
has
a
history
of
not
being
fronted
by
CDN
for
a
number
of
reasons
that
it's
a
logging,
endpoint
and
so
yeah.
There
were
the
security
of
that
has
been
like
before
my
time.
This
is
kind
of
how
that
service
is
always
run.
A
B
A
Great,
so
the
question
was
how
routing
changes
were
expressed.
So
in
the
previous
world,
with
varnish,
we
were
having
VCL.
It
was
kind
of
up
to
traffic
engineers
to
make
the
changes
on
behalf
for
the
people,
because
those
route
tables
grew
so
complex
and
there
goes
Auto
luck
again
and
so
with
the
change
to
envoy.
A
Essentially,
what
that
migration
consisted
of
is
I
would
take
the
VCL
and
just
do
an
enumeration
of
what
are
all
the
features
that
weren't
read,
doing
and
implement
them
into
on,
which
is
a
checklist
and
now
that
the
routes
are
expressed
as
JSON
files
is
very
easy
for
other
engineers
to
go
in
and
understand,
what's
happening
and
make
the
changes
themselves.
So
we
for
now
writing
changes.
It's
just
up
to
engineers
to
submit
a
pull
request.
Who
will
review
it?
Ok
it
and
deploy
it.
B
A
The
D
yeah
we
we
have
that
as
a
hard
set
sized
group,
so
it's
susceptible
to
toss,
but
we
have
firewall
rules
to
prevent
that
and
yeah
I'm
not
too
familiar
with
that
deployments.
I
can't
speak
to
it
in
great
detail,
but
we
have
done
a
bunch
of
failure
testing
with
envoy
in
terms
of
what
happens,
if
runs
ICP,
or
not
a
memory,
auto-scaling
or
strains,
etc.