►
From YouTube: How to DDOS yourself with Envoy (and other tales of migration horror) - Ben Plotnick + John Billings
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How to DDOS yourself with Envoy (and other tales of migration horror) - Ben Plotnick, Yelp + John Billings, Yelp
Yelp has been working with service meshes since the introduction of SmartStack in 2014. After reaching the limits of the SmartStack featureset, we were eager to upgrade to Envoy. With years of experience operating a service mesh at scale, we figured that the migration would be a walk in the park… It turns out the park is larger than we thought.
In this session, we’ll discuss our experience migrating Yelp’s hundreds of services to using Envoy, the migration challenges we faced, and the solutions that we’ve come up with including:
* Transparently moving clients between SmartStack and Envoy using client library feature toggles
* Transforming Envoy’s grpc access log messages to HAProxy-formatted syslog messages in order to harness our existing log processing pipeline
* Using docker-based acceptance testing to confidently push control plane changes directly to production
* How to avoid DDOSing yourself with Envoy’s health checks
A
Okay,
hello,
everyone,
I'm
John:
this
is
my
colleague
Ben.
We
work
at
Yelp
and
we're
both
in
the
engineering
effectiveness
group
and
we're
going
to
be
talking
to
you
about
how
we've
started
using
envoy
at
Yelp.
So
I
want
you
to
imagine
you
just
got
up
in
the
morning
you're
staring
yourself
in
the
mirror.
Bleary-Eyed
as
you
brush
your
teeth.
Maybe
you
went
to
bed
a
little
too
late.
Last
night
you
watched
one
episode
too
many
of
the
Great
British
Bake
Off.
What
are
you
thinking
about?
A
So
let's
talk
about
develop
a
penis.
What
do
developers
care
about
I
think
they
want
to
quickly
ship
bug-free
code
and,
if
they're
doing
that
this
hopefully
means
that
they
can
solve
customers
problems
on
your
land,
what
a
developers
not
so
interested
about?
Well
often,
you
really
don't
care
about
setting
timeout
values.
People
come
to
me.
They
say
John
should
I
be
using
200,
milliseconds
or
400
milliseconds,
I,
just
don't
know
and
I'm
like
I,
don't
really
know
either
like
I
wish.
You
could
just
ship
your
code,
but
yeah.
Let's
figure
this
one
out.
A
A
So
with
that
in
mind,
let's
look
at
our
first
service
mesh
architecture
at
Yelp,
so
we
were
using
this
up
until
about
2014
I
call
this.
Let's
just
manually
configure
a
che
proxy
and
at
that
time
most
code
lived
in
a
Python
monolith
on
the
left
and
we
also
had
a
handful
of
services
on
the
right
search
service
and
geocoding
service
shown
here
and
our
monolith
communicated
with
these
services
using
a
single
redundantly
provisioned
h,
a
proxy
instance
data
center.
So
let's
return
back
to
that
and
develop
a
happiness
question
what's
working
well
here!
A
What's
not
working
so
well,
ok,
so
this
is
an
email
from
infrastructure.
Engineer
saying
that
they've
just
manually
changed.
Yes,
the
H,
a
proxy
config
to
add
some
new
servers
and
they're
going
to
reload
that
H
a
proxy
instance
which
is
going
to
cause
some
user
visible
errors.
Sorry,
so
why
is
this
gonna
make
developers
unhappy?
Well?
Firstly,
developers
are
unhappy
any
time
they're
waiting
on
another
person
to
do
something
for
them,
because
this
slows
them
down.
A
Let's
look
at
another
source
of
unhappiness,
so
there's
a
ticket,
that's
documenting
an
issue
whereby
we're
hitting
capacity
limitations
on
our
single
H
a
proxy
instance-
and
this
is
causing
20
plus
seconds
of
latency
in
one
of
our
data.
Centers
so
like
well
quickly
go
on
to
our
other
data
center,
and
why
does
this
make
developers
unhappy?
Well
again,
they
were
writing
code
trying
to
solve
customers
problems
and
then
suddenly,
every
single
service
is
alerting
and
so
they're
all
drawn
away
from
what
they
were
doing
and
have
to
focus
on
these
issues.
A
Okay,
so
it
was
pretty
clear
to
us
that
our
manually
configured
centralized
service
mesh
was
making
developers
a
little
unhappy
and
I'd
heard
about
finagle,
in
particular
the
zookeeper
based
service
discovery
and
client-side
routing
and
I
thought
this
could
solve
these
problems
for
us,
the
problem
being
with
this,
that
finagle
only
works
on
a
JVM
and
we're
running
lots
of
Python
services.
So
what
I
do
is
I
prototyped
an
in
process,
python
version
of
finagle
and
shipped
it
to
production,
and
the
website
fell
over
because
python
does
not
work
very
well
with
threading.
Okay.
A
We
started
thinking
about
building
a
sidecar
process
to
perform
service
registration
and
then
in
October,
2013
Airbnb
announced
smart
stack,
which
does
this
for
us,
so
we
evaluated
it
and
we
found
that
it
did
promise
a
big
improvement
over
our
first
service
mesh.
So
we
started
using
it
and
we've
been
successfully
using
it
for
the
past
four
years.
A
So
for
those
of
you
who
haven't
seen
it
before
I'm
just
going
to
give
a
quick
overview
of
the
smart
stack
architecture
and
then
Ben
is
going
to
talk
about
how
he
changed
it
shortly.
So
let's
talk
about
the
control
plane.
First,
so
at
the
top
right,
you've
got
this
nerve
process,
that's
health
checking
a
couple
of
service
instances
and
if
they're
healthy,
it
registers
those
in
zookeeper
and
top
left
on.
Another
host.
A
You've
got
this
synapse
instance
and
all
it's
doing
is
getting
these
push
based,
notifications
from
zookeeper
and
configuring,
a
local
H
a
proxy
instance
and
then
over
on
the
data
plane.
You've
got
clients
and
their
interface
to
the
network
is
just
H
a
proxy.
That's
all
they're,
talking
to
an
H,
a
proxy
is
routing
to
whatever
service
instances.
It's
been
configured
with,
and
this
is
the
basic
design
and
it
works
really
well
I'm
going
to
say
that
again
it
works
well.
A
So
in
particular
it
allows
us
to
reliably
get
bits
across
the
wire,
no
more
waiting
on
humans
to
configure
those
load,
balancers,
no
more
alert,
floods.
Every
time,
a
load,
balancer
config
changes
once
you've
figured
out
how
to
do
zero,
downtime
H,
a
proxy
restarts,
which
is
another
story
and
no
more
scaling
problems
to
a
first
approximation.
A
So,
let's
get
back
to
that,
what
makes
developers
sad
we've
kind
of
solved
most
of
the
problems
there,
so
we
had
to
think
a
little
harder.
Maybe
our
job
was
done,
maybe
not
so
we
about
a
year
ago
we
started
talking
to
developers.
We
asked
them
what
was
slowing
them
down
and
we
came
up
with
a
list
and
I'm
gonna
show
you
our
particular
list.
A
Maybe
your
list
is
different,
so
here
a
bunch
of
things
that
we
got
from
discussions
with
our
developers,
along
with
some
scores,
if
you
went
through
the
same
exercise,
you'd
probably
come
up
with
a
different
list
with
different
scores
and
let's
just
focus
on
the
things
on
the
left,
the
descriptions
so
developers
said
that
they'd
really
quite
like
some
standardized
circuit
breaking
retrying
and
time
outs.
They
were
tired
of
having
to
figure
out
those
values
themselves.
They
wanted
us
infrastructure
engineers
to
take
care
of
those
problems
for
them.
A
A
So
now,
let's
zoom
in
on
one
of
these,
don't
dynamit
request,
routing
and
we've,
given
it
a
score
of
70,
so
I'm
just
going
to
tell
you
how
we
reach
that
score.
So
basically,
it's
a
weighted
sum
of
factors,
some
positive
and
some
negative
and
there's
nothing
special
about
these
factors,
but
we
felt
that
they
were
important
to
our
organization
and-
and
we
also
wanted
to
be
a
little
principled
about
this
exercise.
So,
let's
start
with
the
positive
factors:
what's
good
about
dynamic
requests,
routing
well,
developer
happiness,
we
score
three
out
of
three.
A
We
think
there's
a
big
win
for
developers
from
this,
and
there
are
also
high
benefits
from
developer
velocity
and
also
a
little
bit
positive
on
stability
and
operational
costs.
But,
as
all
of
us
know,
whenever
we
make
a
change,
there's
also
some
downsides,
so
we
tried
to
capture
those
as
well.
So
what
are
the
negative
effects
to
the
business?
Well,
we
thought
it
was
pretty
risky
if
we
were
to
implement
this
thing.
Potentially
we
could
bring
the
website
down
and
we
also
thought
it
was
difficult.
So
that
was
how
we
arrived
with
our
score.
A
So
we
identified
a
bunch
of
problems
and
then
we
went
looking
for
ways
that
we
could
solve
them
and
we
realized
that
a
lot
of
these
problems
could
be
addressed
by
building
a
smarter
service
mesh.
So
we
evaluated
the
available
options
here.
We
firstly
considered
extending
H
a
proxy
we'd
already
written
some
custom,
Lua
script
plugins
to
do
fault
injection.
We
thought
we
kind
of
knew
it
pretty
well,
maybe
we
could
even
start
hacking
on
the
C
code.
We
seriously
considered
this.
A
B
Thanks
John,
so
we
knew
at
this
point.
We
knew
that
we
wanted
to
go
to
envoy.
There
were
a
lot
of
features
that
our
developers
were
going
to
benefit
greatly
from,
but
we
knew
that
there
was
a
lot
of
risk,
so
John
showed
you
this
slide
earlier.
We
know
that
there
are
some
things
that
developers
like
and
developers
don't
like,
but
we
actually
missed
one
here
and
that
is
that
developers
hate
getting
woken
up
in
the
middle
of
the
night
because
their
service
mesh
broke
so
infrastructure
development
is
inherently
risky.
B
That's
there's
no
getting
around
that
not
with
any
cloud
native
technologies.
It's
just
risky
most
developers,
I
would
venture
to
say
would
probably
rather
have
you
know
to
change
anything
than
to
break
it
by
upgrading
to
the
newest
shiny
thing.
But
we
also
know
that
envoy
is
gonna,
make
their
lives
better.
So
how
do
we
rectify
these
two?
And
we
took
a
page
from
that
from
the
software
development
world
and
try
to
apply
the
agile
technique.
B
So
I'm
gonna
talk
for
a
little
bit
about
how
we
rolled
out
envoy,
using
agile
techniques
and
I'm
gonna,
specifically
talk
about
incremental
development,
the
idea
of
cut
points
and
CI
and
CD.
So,
let's
first
talk
about
incremental
development,
so
John
showed
you
this
smart
stack
architecture
before
and
honestly
oftentimes.
The
temptation
is
to
just
rip
this
whole
thing
out
and
replace
it
with
the
thing
that
everybody
else
is
using.
B
So
maybe
that's
kubernetes
with
this
teo
or
whatever,
but
that's
really
disruptive,
and
that
makes
developers
unhappy
and
we
don't
want
to
do
that,
so
the
new
architecture
that
we
came
up
with,
which
was
an
incremental
improvement
over
this-
is
this
so
I'm,
just
gonna,
flip
back
and
forth
to
see
if
you
can
spot
the
difference,
they're,
old
and
new
old,
new
and
yeah
it's
just
this.
We
basically
replaced
one
part
of
the
control
plane
and
we
replaced
the
proxy
with
Envoy.
So
our
long
term
goal
is
to
go
to
this
ideal
infrastructure.
B
You
know
kubernetes,
st
or
whatever,
but
we
need
to
make
incremental
improvements.
So
I'm
gonna
take
a
quick
detour
to
talk
about
our
control
plane.
I!
Don't
want
to
go
into
too
much
detail
because
I
feel
like
it's
very
similar
to
other
control
planes,
but
why
did
we
write
our
own?
We
could
have
used
an
existing
one.
We
could
have
used
Java
control
plane
or
go
control
plane,
but
we
didn't
we're
more
comfortable
in
Python
and
we
like
flat
files.
So
this
is
it
it's
pretty
much
pulls.
B
This
mesti
thing
is
less
than
a
thousand
lines
of
code.
It
pulls
service
registration,
data
from
zookeeper
pulls
some
manifest
e
type
of
stuff
from
flat
files
writes
out
some
flat
files
in
XC
s,
format
not
void,
reads
it.
It's
really
really
simple
and
we
do
plan
on.
You
know
going
to
some
G
RPC
system
or
whatever,
but
but
this
is
the
the
simplest
incremental
improvement
that
we
could
have
made,
and
at
this
point
we
had
some
sort
of
control
plane.
We
could
have
tried
to
get
to
a
feature
parity
with
our
existing
system.
B
We
actually
have
a
ton
of
features
in
our
existing
system
we
built
in
like
fault,
injection
and
chaos,
testing
and
all
that
stuff.
In
our
existing
system
we
could
have
gone
to
feature
parity,
but
that's
really
risky.
So
the
idea
of
a
tracer
bullet
comes
comes
into
play
here,
and
a
tracer
bullet
is
getting
a
narrow
use.
Case
solved
end
to
end,
so
you
might
be
familiar
with
this
page.
This
is
kind
of
like
a
business
page
on
Yelp
and
if
we
were
to
try
to
migrate
all
of
this
traffic,
this
is
pretty
complex.
B
There's
a
lot
of
things
going
on
here
and
if
we
were
trying
to
try
to
get
all
of
this
through
envoy,
it
would
have
taken
a
very
long
time
and
would
have
been
very
risky.
So
what
we
did
instead
was.
We
just
picked
one
little
part.
This
little
ask
the
community
part
and
within
a
quarter
using
incremental
development,
we
were
able
to
get
this
up
and
running
routing
through
envoy.
So
we
did
a
very
simple
thing
and
to
end
very
quickly
and
as
a
bonus.
B
So
some
key
takeaways
about
incremental
development
make
small
changes.
Rather
than
large
overhauls,
you
shouldn't
ever
break
anything.
Well,
that's
a
pretty
broad
statement,
but
each
version
should
be
usable
and
yeah.
Don't
try
to
solve
every
use
case
immediately
if
you
can
use
tracer
bullets,
and
so
the
next
thing
I
want
to
talk
about.
Is
the
idea
of
this
cut
points,
but
first
I'm
going
to
take
a
quick
digression
to
client
libraries
and
you'll
see
why,
in
a
second,
so
to
talk
to
a
service
at
Yelp?
You
can
just
use
curl.
B
That's
our
developers
like
that,
but
you
can
also
use
a
client
library,
and
this
is
generally
what
people
use
so
a
client
libraries
generated
from
an
API
spec.
We
have
a
client
libraries
in
different
languages,
I'm
sure
this
is
a
familiar
concept:
here's
the
actual
code
example.
If
you
want
to
talk
to
an
example,
happy
hour
serve
you
instantiate
it
and
you
make
an
RPC
call
just
like
that.
We've
done
a
huge
amount
of
work
to
get
people
on
client
libraries
and
we
have
a
similar
one
for
databases.
B
We
have
this
in
a
few
different
languages,
but
I
know
what
you're
thinking
isn't
envoy
supposed
to
replace
client
libraries
or
maybe
replace
a
lot
of
it.
So
it's
all
about
balance.
Thick
client
libraries
have
a
lot
of
problems.
Sorry
finagle!
You
need
to
maintain
multiple
implementations.
As
these
client
libraries
start
to
live
longer,
they
start
to
get
more
features
and
you
have
to
maintain
that
over
multiple
languages
and
that's
a
lot
of
work.
B
Upgrading
everyone
is
a
nightmare.
You
have.
We
have
hundreds
of
services
and
upgrades
for
our
client.
Libraries
take
a
very
long
time.
I'm
sure
many
of
you
have
experienced
this.
This
was
actually
a
huge
driver
for
us
to
move
to
Envoy.
We
wanted
to
make
our
client
libraries
simple,
so
that
we
could
have
so
that
we
could
control
a
lot
of
that.
B
However,
you
still
need
some
sort
of
thin
client
libraries
and
there's
some
useful
things
that
you
can
do
their
context.
Propagation,
for
example,
say
if
you
want
to
do
Zipkin
tracing,
injecting
that
context
into
the
wire
is
really
useful
or
doing
the
client
library
and
setting
X
envoy
headers,
for
example.
It's
something
that
you
can
abstract
away
in
the
in
the
client
library
things
like
data
marshalling
as
well,
and
if
people
don't
use
them
directly
at
least
they
can
serve
as
a
reference
implementation.
B
And
the
last
thing
is
that
client
lives
are
a
great
cut
point.
So
a
cut
point
is
I,
don't
know
if
it's
a
common
term,
but
it's
a
term
that
we
use,
meaning
that
it's
a
place
where
you
can
have
do
a
little
bit
amount
of
work
and
have
a
huge
amount
of
impact.
So
since
we
had
already
done
the
work
to
migrate
people
to
client
libraries,
this
is
a
great
place
where
we
can
do
some
migrations.
B
So
it's
literally
this
like
this
is
a
simplified
example,
but
it's
not
very
many
more
lines
of
code
in
reality,
which
is
that
we
change
the
client
library
to
have
a
little
feature
toggle
there.
That
says,
basically,
if
the
future
toggle
is
enabled
route
through
Envoy
otherwise
route
through
a
chat
proxy-
and
this
is
great
for
a
number
of
reasons.
B
Firstly,
the
developers
don't
even
know
that
they're
routing
through
Android
they
have
no
idea
that
we've
switched
them
over
to
use
envoy
or
agent
proxy
I
mean
we
do
tell
them
we're
not
that
evil,
but
it
also
enables
us
to
do
phase
deployment
which
we'll
talk
about
in
a
little
bit.
And
finally,
it
allows
us
in
in
bad
scenarios
to
shut
off
envoy
routing
for
everybody
as
a
kill
switch.
B
What
other
place
cut
points
are
useful
is
for
things
that
developers
indirectly
use
such
as
observability
tools.
So
this
is
our
existing
log
processing
pipeline.
We
have
a
che
proxy
that,
logs
to
some
log
processor-
and
this
thing
spews
out
a
bunch
of
structured
logs,
for
example,
we
extract
Zipkin
tracing
from
here.
We
do
some
alerting.
We
send
logs
to
Splunk
it's
pretty
complex
and
pretty
used
pretty
well
used
by
developers.
B
B
Erp
C
based
access
logs,
but
this
can
be
risky
as
well
because
the
mainly
because
the
existing
code
is
large
and
kind
of
unwieldly,
so
the
option
that
we
went
with
is
to
actually
have
this
separate
thing
that
munches
the
Envoy
logs
and
makes
them
look
like
a
g8
proxy.
So
the
log
processor
thinks
that
it's
consuming
H
a
proxy
logs
but
in
reality
they're
coming
from
envoy-
and
this
allows
us
to
support
the
old
system
and
gradually
move
away
from
it
and
the
the
developers
don't
even
know
the
difference.
They
see
the
logs.
B
They
see
the
Zipkin
traces,
they
see
the
logs
at
Splunk
and
they're
happy.
This
is
kind
of
like
a
branch
by
abstraction
or
maybe
branch
by
abstraction
in
Reverse,
so
yeah
takeaways
for
cut
points
find
places
where
you
can
do
a
lot
of
have
a
lot
of
impact
for
a
little
bit
of
a
amount
of
work
and
use
shims
for
an
abstraction
layer
and
also
feature
toggles
are
amazing.
B
So,
lastly,
I'm
going
to
talk
about
continuous
integration,
continuous
deployment,
so
I
don't
know
about
you,
but
this
is
kind
of
how
I
knew
infrastructure
development.
You
want
to
make
a
change.
You
make
an
upgrade.
You
find
a
host
that
maybe
there's
some
people
on
it
or
maybe
not.
You
run
some
upgrades
on
it.
B
You
run
some
manual
tests,
maybe
you
break
things
and
then
you
deploy
to
prod
and
you
pray
and
sometimes
you're
lucky,
and
you
claim
that
you're
really
smart
and
other
times
you
get
unlucky,
and
you
say
that
the
gods
have
smitten
you,
but
I'm
gonna
claim
that
infrastructure
is
not
an
exception
to
good
software
development
practices.
We
have
automated
tests.
We
were
used
to
this.
So
why
not
write
automated
tests
for
our
system?
This
is
real
code
from
our
control
plane
tests.
B
We
run
end-to-end
tests
using
docker,
compose
and
I'd
venture
to
say
that
I'd
be
willing
to
bet
that
if
zookeeper
went
down
right
now
in
our
environment,
then
we'd
still
be
able
to
route
through
envoy
because
we've
tested
it.
In
this
example,
we
bring
up
a
bunch
of
services
at
zookeeper
and
envoy,
and
then
we
start
killing
them
and
making
sure
that
we
can
still
route
through
it.
So
this
gives
us
a
lot
of
confidence
when
we
deploy
and
as
a
bonus,
you
can
use
these
end-to-end
tests
in
your
github
issues
on
Envoy.
B
So
when
you're
struggling
to
explain
what
exactly
is
happening
rather
than
trying
to
explain
it
in
words,
just
point
to
some
docker
compose
scripty
thing
and
say:
hey
just
run
this.
This
is
exactly
what
I
did
here,
but
no
testing
discussion
would
be
complete
without
the
test
pyramid
make
sure
you
still
have
integration
unit
tests
and
to
end
tests
are
still
slow.
So
you
can't
get
around
that
sorry,
okay,
yeah,
takeaways,
right
automated
tests
doctrine
compose
is
amazing
for
running
end-to-end
tests
run
against
master.
B
If
you
can
stomach
it,
we
during
heavy
development
will
deploy
master
of
envoy
up
to
once
once
per
week.
It's
great
quality
I'm,
pretty
confident
in
it,
and
the
last
point
is
about
phase
deployment.
So
phase
deployment
is
the
idea
that
you
want
to
roll
out
to
kind
of
safer
environments
first
and
then
do
a
gradual
rollout,
and
this
is
really
important
and
I'll.
Tell
you
a
story.
B
We
did
have
tests
for
this
and
we
actually
did
roll
it
out
to
a
staging
environment
first,
but
because
distributed
health
checking
is
open
squared.
It
only
failed
when
we
scaled
it
up
to
a
larger
phase.
So
this
is
not
the
only
time
it
happened.
We
had
John,
also
DDoS
our
environment
I'm,
not
the
only
one
that
did
this,
but
because
we
were
able
to
deploy
in
phases,
the
damage
was
minimal
and
we
were
able
to
roll
back
quickly
and
remember.
That
envoy
is
part
of
pretty
much
everything
when
you
roll
it
out.
B
So,
if
you
break
something
bad
things
will
happen,
developers
will
get
unhappy,
so
just
to
recap:
yeah
apply
incremental
development
everywhere,
identify
cut
points
where
you
can
kind
of
do
a
little
bit
of
work
and
have
a
lot
of
impact
and
use
CI
CD,
so
yeah.
In
summary,
this
is
amazing.
This
is
awesome
that
we
get
to
work
on
this.
No,
no
okay,
focus
on
your
users.
B
Yeah
the
question:
the
question
was
about
this:
this
kind
of
branch
by
abstraction
thing,
with
the
access
logs
and
yeah.
The
long
term
goal
is
to
actually
move
over
to
using
this
new
system.
So
we
have
to
support
these
two
systems
in
parallel,
but
yeah.
Eventually,
we
won't
be
using
a
chip
proxy
at
all
and
we'll
have
only
this
new
access,
log,
Munzer
or
whatever
will
actually
do
more
things.
We're
originally
going
to
call
it
envoy,
2h
a
proxy,
but
that
will
become
a
useless
name
very
quickly.
So
we
just
called
it
bike
shed.