►
From YouTube: Hardening Envoy - Alyssa Wilk, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Hardening Envoy - Alyssa Wilk, Google
As Envoy picks up momentum both as a standalone proxy and the signature proxy of CNCF it becomes more and more important to improve Envoy's reliability and stability without reducing developer momentum.
This talk will cover some of the recent enhancements made both to the Envoy binary to improve reliability and decrease attack surface, work done to fuzz both Envoy and Envoy's third party dependencies to reduce the likelihood of queries of death sneaking into the code base, and committed and planned changes to the Envoy test framework to improve end to end coverage and better test production-like scenarios.
A
A
A
Well,
first
off
again,
lyft
originally
open-source
envoy
there
behind
AWS,
so
they
have
AWS
as
a
front
line
of
defense,
but,
as
we've
heard
today,
there's
more
and
more
people
using
envoy
as
an
unprotected
edge
proxy
and
this
kind
of
increases
your
attack
surface,
because
it's
different
attack
modes
and
and
more
interesting
targets
for
attacking
and
then
also
as
envoys
adopted
as
a
cloud
proxy.
You
have
to
worry
about
the
case
of
untrusted
and
possibly
malicious
upstreams,
as
well
as
a
control
plane.
As
an
attack
surface.
A
You
know
people
doing
conflicts
of
death,
so
some
of
the
work
that
you
know
I
and
my
team,
my
larger
team
at
Google,
as
well
as
the
own
voice
communication,
doing,
is
trying
to
address
this
through
a
combination
of
software
improvements,
improving
resilience
against
external
attacks
and,
hopefully,
accidental
internal
attacks
as
the
less
test
improvements.
So
improvements
to
envoy
integration,
test,
the
integration
tests
framework,
our
Cir,
continuous
integration
through
adding
fuzzing
and
then
coming
soon
and
adding
load
testing
so
starting
with
resilience
against
external
attacks,
because
of
course
this
is
where
things
get
really
fun.
A
My
starter
project
and
envoy
was
trying
to
close
a
trickle
attack
vector
and
what
the
trickle
attack
is
essentially
envoy
eagerly
buffers
from
upstream.
It
really
wants
to
have
data
ready
for
the
clients
to
read
as
soon
as
a
client
is
available,
you
know
has
available
bandwidth
descendant,
but
the
problem
is:
is
that
means
with
very
few
resources
with
a
botnet,
and
you
know,
opening
a
number
of
connections
that
have
large
responses.
A
You
can
pretty
easily
cause,
could
cause
envoy
to
early
buffer
and
buffer,
more
and
buffer,
more
and
then
crash
out
of
memory,
which
is
obviously
not
optimal.
So
my
startup
ratchet
and
envoy
was
rewriting
bits
of
the
data
plane
based
on
a
design
by
Harvey
and
that
Cline
to
add
in
memory
buffer
limits
on
every
buffer
down
from
the
network
connection
buffer
up
to
individual
HT
filters
and
basically,
as
buffers
fill
up,
they
would
signal
the
source
of
incoming
data
to
back
off.
A
You
take
a
combination
of
TCP
congestion,
control
and
h2b
to
flow
control,
so
coming
back
to
this
case
of
a
fast
upstream
and
a
slow
downstream.
Again,
if
you
have
this,
you
know
botnet
aimed
at
envoy,
and
the
downstream
is
reading
really
slowly,
eventually,
the
TCP
socket
buffer
to
downstream
and
Philip,
but
a
network
connection
would
fill
up
and
then
envoy
would
signal
the
router
to
either
4-h
to
stop
acknowledging
data.
A
So
the
peer
would
stop
sending
data
for
that
stream
or
4-h
to
be
one
one
or
TCA
proxying
simply
stop
reading
from
the
TCP
socket
and
let
TCP
do
its
thing,
and
this
is
great,
but
it
is
woefully
insufficient.
All
it
does
is
make
your
attacker
change
their
attack
mode.
So
the
next
thing
we
had
to
do
was
downstream
circuit.
Braking
again,
you
can
have
an
upper
bound
on
how
much
memory
each
stream
can
take,
and
so
then
your
attacker
just
switches
to
opening
many
many
streams
until
again
envoy
crash.
A
Is
that
a
memory
or
they
keep
opening
new
connections
until
your
system
runs
out
of
file
descriptors
and
again
on
boyhood
crash,
and
this
sort
of
attack
mode
has
a
really
high
potential
for
cascading
failure?
You
can
imagine
it's
not
even
a
malicious
botnet
right.
If
you
run
your
own
voice
hot,
you
could
just
have
an
unusually
high
traffic
spike
and
your
traffic
spikes
you're
on
boys
are
running
hot.
Some
on
way,
instance
goes
over
its
limit
it
crashes
and
then
your
fronted
load
balancer
ends
up
spreading
the
rest
of
that
load.
A
That
used
to
go
to
that
envoy
to
the
rest
of
your
cluster
and
then
one
by
one.
The
on
ways
get
knocked
over
and
beyond
that.
It
can
be
really
hard
to
recover
that
from
this
type
of
failure,
because
even
if
a
traffic
spike
is
excited
as
your
envoys
come
up,
it's
like
a
game
of
whack-a-mole
right.
They
come
up,
they
sinkhole
the
incoming
traffic
and
they
go
right
back
down
again
and
then
again,
this
is
even
more
dangerous.
When,
coupled
with
envoys
prior
lack
of
keepalive
timeouts
on
boy
had
timeouts
on
the
request
path.
A
If
you
had
the
buffer
filter
your
running
without
the
buffer
filter,
it
was
pretty
easy
for
again
a
malicious
attacker
to
send
a
bunch
of
post
requests
and
then
have
malicious
use
of
more
system
resources
than
you
want
them
to
and
again
there
was
also
a
connection.
League
bug
which
I'll
get
into
later,
which
would
again
allow
attackers
to
potentially
use
more
system
resources,
and
we
want
them
to
so
Elijah
and
my
greater
team
implemented
downstream
circuit.
A
Braking
and
now
you
can
configure
file,
descriptive
limits
and
memory
limits
and
essentially,
as
you
approach,
those
FD
limits
envoy
will
stop
accepting
new
connections
as
you
get
new.
Those
memory
bounds
on
voice
stops
accepting
your
streams.
So
now
this
combined
with
flow
control
gives
you
a
bound
on
system
resources
right.
A
You
may
not
serve
everything
that
you
want
to
serve
right,
where
you're
not
accepting
you're
going
to
have
high
latency
or
timeouts
and
where
you're
rejecting
streams
are
going
to
have
five
hundreds,
but
having
some
number
of
five
hundreds
is
certainly
better
than
having
a
system-wide
cascade
failure.
So
that's
resilience
against
external
attacks.
Moving
on
to
internal
attacks,
one
of
the
areas
Google's
put
a
lot
of
work
into
is
protecting
our
back
ends
our
up
streams
from
our
proxies
right.
A
It's
really
really
easy
for
a
high
performance
proxy
to
proxy
enough
requests
that
you're
gonna
take
out
your
backends.
So
one
of
the
things
we
put
a
lot
of
work
into
is
improving
on
voice
load
balancing.
So
the
first
thing
we
did
is
give
unvoyage
concept
priorities
in-house
at
Google.
We
tend
to
have
primaries
and
failover
clusters.
So
essentially,
if
something
happens,
and
you
have
traffic
routing
to
your
primary
your
primary
up
streams
and
then
they
get
overloaded
or
you
forget
to
change
your
config
and
you
drain
something.
A
You
want
that
traffic
to
gradually
shift
to
the
fail
overs
with
no
interruption
for
user
web
browsing
on
Voisine
more
flexible
than
this,
because
it
allows
an
unbounded
number
of
priorities
essentially
priority.
Zero
is
your
primaries
priority.
One
is
kind
of
concept
of
failovers,
but
you
can
have
P
equals
to
P
equals
three.
You
know
as
many
as
you
want
to
have
and
again
the
algorithm
is
used
to
shift
this
traffic
over
there's
been
a
lot
of
thought
put
into
them.
A
They've
worked
at
Google
for
well
over
a
decade,
making
sure
that
as
you're
up
streams
go
unhealthy,
you
shift
traffic
over
really
gradually
and
shift
it
back
gradually.
So
you
don't
have
query,
is
kind
of
sloshing
back
and
forth.
In
a
way
that
could
overwhelm
your
up
streams
and
cause
outages
in
the
first
place,
there's
been
some
work
put
in
to
improve
fairness.
So,
for
example,
an
earlier
iteration
of
one
of
the
Envoy
load,
balancers
I
had
a
classic
load
balancing
problem
or
didn't
have
random
offset
as
lift
discovered
that
they
just
may.
A
Essentially,
what
you
can
have
is
a
coordinated
fleet
of
envoys,
sending
requests
to
upstream
number
one
and
then
upstream
number
two
and
taking
things
out
as
I
go
along
the
line,
and
this
is
fairly
easily
fixed
with
random
offset,
but
there's
still
a
little
bit
of
work.
That
needs
to
be
done
if
you
have
up
streams
with
very
disparate
weights
and
that'll
happen
over
the
next
year
and
then
finally,
there's
been
a
lot
of
work,
put
into
flexibility
and
load.
A
So
the
folks
at
list
upstream
they're,
pretty
validation,
suite
so
essentially
as
people
add
new
envoy
config,
you
can
annotate
it
with
bounce,
so
put
a
limit
on
how
much
memory
a
given
string
can
take
or
put
a
limit
on
how
many
entries
or
repeated
field
can
have,
and
this
allows
us
to
do
very-
very
low
costs.
At
best
effort,
avoidance
of
poison
can
fix
right.
Everyone
should
absolutely
validate
customer
config
before
they
littered
onto
envoys.
A
We
learned
this
internally,
unfortunately
do
overdo
a
particularly
memorable
adage
like
with
Envoy
internal
at
Google,
we
tend
to
roll
out
features
separate
for
the
binary,
so
we
configured
them,
we
flag
guard
them.
We
roll
out
the
binary
and
the
way
push
a
config
to
turn
these
features
in
case
they're
buggy.
What's
really
important
is
that
all
of
these
features
are
well
back
safe.
If
you
roll
out
a
config,
it
doesn't
work
the
way
that
you
want.
A
You
need
that
roll
back
to
actually
reinstate
prior
behavior,
and
we
had
one
particularly
memorable
instance
where
that
didn't
work
right,
we
rolled
config
out
it
didn't
work
the
way
one
in
she
or
operations
team
like
you,
do
rolled
it
back
and
then
the
outage
continued
right
and
then
you
said
around
dinner
angle.
Was
it
actually?
They
could
think
that
caused
the
problem
with
something
else,
and
then
the
only
way
to
guarantee
the
prior
behavior
is:
do
a
sleep
ride,
restart
right,
resetting
every
TCP
connection
resetting
every
stream.
A
It's
really
not
enjoyable
and
I've,
caught
bugs
in
review
visually
that
were
sent
up.
You
know
for
review
and
upstream
envoy
that
were
not
rollback
safe.
So
in
house,
let
me
have
this
particular
outage.
We
added
best
practices
that,
when
we're
turning
on
a
new
feature
via
config,
we
do
reload
a
rollback
test,
so
essentially
an
integration
test
where
you
send
a
request
through
get
a
response
back
and
then
use
the
internal
equivalent
of
XDS
to
change
the
conflict.
A
To
turn
the
feature
on
send
another
request
through
and
back
make
sure
that
the
behavior
has
changed
and
then
again
and
one
line
of
code
ruled
that
back
and
make
sure
that
the
original
behavior
is
reinstated,
super
important
and
getting
into
that
there's
a
whole
slew
of
improvements.
We've
made
to
the
Envoy
integration
test
framework,
and
this
is
where
I
tend
to
geek
out
of
it.
I
love
integration
tests.
This
dates
back
to
when
I
started
at
Google
11
years
ago.
A
I
was
a
terrified
niggler
and
I
was
very
well
aware
that
any
code
I
checked
in
could
not
you
know
not
just
break.
You
know
Google
ads,
that
break
all
of
Google
right.
So
I
invested
a
lot
of
time
in
the
GPS
integration
test
framework
and
I've
been
trying
to
bring
these
things
one
by
one
over
nominally.
A
So
the
first
thing
I
did
again
starting
out
envoy
I,
basically
made
it
really
easy
to
do
what
most
people
want
to
do
when
they're
doing
testing
is
send
a
request
and
get
a
response
right,
we're
testing
a
proxy.
There
should
be
pretty
simple,
so
it's
now
90%
fewer
lines
of
code
for
testing
than
happy
path.
We
also
added
utilities
to
test
to
end
end
configuration
so
again,
most
apartment
features
an
envoy
are
conflict,
guarded
people
put
them
in
filters,
people
put
them.
You
know
as
fields
on
the
HTTP
connection
manager.
A
A
Here's
my
filter,
config
I,
just
want
to
tweak
this
particular
field
I'm,
making
it
really
easy
people
to
write
integration
tests,
because
I've
seen
multiple
multiple
PRS
or
people
did
comprehensive
unit
tests
envoy
has
a
really
high
bar
for
code
coverage,
but
when
they
ask
people
to
add
integration
tests,
it
turns
out
that
the
unit
test
did
it
test.
The
exact
stream
lifetime
that
actually
happens
in
production
right,
so
these
unit
tests
ended
up
finding
issues
that
weren't
caught
the
integration
test
down
here,
she's
at
work
up
by
the
unit
tests.
A
We
made
it
a
lot
easier
to
do
a
cross
particle
testing.
So
again,
if
you
want
to
test
full
mesh
of
HTP
one
one
downstream
edge
to
downstream
h1
upstream
h2
upstream,
you
can
do
it
just
by
subclassing
the
appropriate
class,
and
this
is
really
important,
as
people
are
messing
around
in
the
G
RPC
and
adding
trailers
and
making
sure
that,
as
you
do,
these
upgrades
and
downgrades
from
H
to
B
1
to
H
2
that
things
are
working
as
they
expect.
A
We
added
an
automated
upstream,
and
you
know
this
both
helps
again
make
the
happy
path
of
sending
requests
and
response
easier,
but
it
also
ends
up
fussing
timing
conditions.
We
found
out
at
Google
a
lot
of
the
time
when
you
have
crashing
bugs
it's
due
to
a
specific
timing
where
you
disconnect
at
just
the
wrong
moment,
or
you
reset
a
stream
at
just
the
wrong
moment.
A
So
having
the
timing
of
the
client
and
the
timing
of
the
upstream
completely
decoupled
and
tests,
and
allows
you
to
kind
of
Fuzzy's
time
and
conditions
and
hopefully
catch
them
over
repeated
CI,
runs
and
then
finally,
an
improved
debug
ability,
sorry
I'm
skipping
that
we
made
it
a
lot
easier
to
debug
the
integration
test,
so
moving
things
out
of
the
closure
base.
So
you
can
use
gdb
Michael
on
my
greater
team.
A
Did
a
lot
of
work
to
change
the
he
failure
mode
from
infinite
timeouts
to
fast
fail,
timeouts
telling
you
what
went
wrong
and
then
again
changing
it,
so
that
a
lot
of
the
test
failures
inform
you.
What
you've
done
wrong
and
point
you
to
documentation
on
how
you
might
be
able
to
fix
it,
and
so
all
of
these
integration
test
improvements
are
kind
of
have
their
effect
multiplied
by
improvements.
We've
made
a
CI.
A
So
now,
every
PR
that
goes
out
for
review
for
Envoy
gets
run
through
address
sanitizer
thread
sanitizer
on
the
undefined
behavior
sanitizer
and
clanked
ID,
and
these
regularly
catch
issues
before
they're
actually
checked
in
I.
Can't
tell
you
the
number
of
peers
I've,
seen
that
passive
release
built
and
failed.
One
of
the
sanitizers
I
didn't
want
to
admit
how
many
peers
of
my
own
have
passed
the
release
valve,
but
failed.
One
of
these
sanitizers
they're
really
important.
They
catch
a
lot
of
bugs.
Also
the
increased
sirens
do
catch
real
production
bugs
on
their
own.
A
We've
had
two
issues:
I
can
remember
offhand
that
were
caught
by
CI
by
these
increased
runs
because
the
more
people
we
have
developing
on
envoy
the
more
times
our
CI
gets
executed.
So
the
to
remember
offhand,
there
was
a
backup
bug,
a
race
condition
between
L
7
and
L
4,
where
essentially,
a
user
was
doing
a
post.
The
upstream
sent
in
early
400
close
a
connection.
An
envoy
detected,
the
connection
close
before
it
had
read
that
400,
so
it
treated
as
a
server
error
and
translated
a
400
and
form.
A
These
are
what
they
done
wrong
into
a
500.
You
know,
and
Google
takes
five
hundreds
pretty
seriously.
This
actually
had
been
caught
and
filed
by
sto,
but
it
was
caught
by
CI
as
well,
and
if
we'd
had
more
CI
runs
see,
I
would
have
caught
it
first
and
there
was
another
issue
in
this
connection.
League
I
mentioned
earlier,
and
this
was
an
issue
where
a
classic
errno
bug
where
envoy
write
to
a
close
socket.
The
kernel
wouldn't
form
envoy
that
the
wrote
failed.
A
We'd
logged
the
err
no
and
the
very
act
of
logging,
the
air
overrode
it
if
the
login
flash,
very
rare
race
condition,
maybe
one
in
five
thousand
times,
but
our
CI
caught
it.
We
debugged
it
and
fixed
it,
and
this
had
been
a
bug
that
had
been
an
envoy
since
it
was
written
right
and
that's
now
fixed
and,
along
with
you
know,
all
the
CI
improvements.
We
now
have
detailed
instructions
for
reproducing
and
deflating
test
failures,
there's
essentially
a
magic
magic,
Basel
invocation.
A
You
can
run
that
brings
it
from
one
in
five
thousand
to
one
in
five
or
so
to
make
it
a
lot
easier
to
debug
so
again,
I'm
a
wax
on
about
integration
test.
The
problem
with
integration
tests
is,
they
only
actually
catch
issues
you
think
to
test,
and
this
is
where
fuzzing
comes
in
so
Harvey
again
on
my
team
has
but
a
ton
of
work
in
on
boy
fuzzing.
A
We
now
have
14
buzzers,
coming
covering
a
combination
of
untrusted
code
like
our
each
each
to
be
at
h2
code
X,
as
well
as
internal
fuzzing,
like
convict
buzzing,
and
these
run
as
part
of
our
CI,
but
they
also
run
independently
in
cluster
of
us.
Harvey's
got
a
ton
of
work
here.
It's
over
80
PRS
and
over
35
bug,
fixed
I
started
by
going
through
git
log
to
try
to
find
exactly
what
fixes
are
were
and
I
gave
up
and
asked
Harvey,
because
it
was
just
too
many
35
plus,
and
these
are
serious
bugs
right.
A
These
are
bugs
that
could
crash
Envoy
that
could
cause
deadlock.
That
would
fail
internal
code
assertions
essentially
undefined
behavior.
That
would
allow
buffer
overflow
and
goodness
only
knows
what
I'm
gonna,
let
you
know
a
malicious
person
could
do
with
that.
So
it's
really
really
important
and
the
fuzzing
is
also
an
important
part
of
envoy
security
story
right.
A
We
have
a
really
good
team,
a
maintainer
is
doing
code,
reviews
for
envoy
and
paying
a
lot
of
attention
to
what
goes
gets
checked
in,
but
we
also
import
libraries
that
other
people
write
right
and
the
maintainers
don't
pay
a
whole
lot
of
attention
as
to
what
goes
into
HTTP,
parser
or
ng
HTTP,
for
example.
So
imagine
someone
getting
some
malicious
code
at
MGH,
B
and
then
envoy
imports
that
right.
A
We
need
some
way
of
detecting
that
and
the
fuzzer
is
again
throwing
some
combination
of
crafted
and
random
data
and
Envoy
are
the
sort
of
thing
that
would
suss
this
sort
of
issue
out
along
with
flensing.
Again,
we
are
working
now
on
load
testing,
so
Harvey
and
Josh
Marantz
are
working
with
an
external
company.
We
amp
to
try
to
have
an
upstream
envoy
load
testing
harness,
and
this
is
super
important.
A
It's
hard
to
figure
out
if
something's
gonna
be
a
CPU
aggression
just
by
doing
static
code
analysis,
so
having
an
external
load
testing
harness
where
someone
can
run
a
load
test
on
a
clean,
pr4
master
and
then
run
with
their
high
risk.
You
knows.
Maybe
this
is
causing
CPU
problems
PR
and
see
what
the
actual
production
impact
is.
A
It's
a
lot
nicer
than
saying:
hey
guys
at
lift
you
mind,
smoke
testing
this
for
us,
which
is
our
current
best
practice
and
again,
you
know,
there's
a
lot
of
things
that
you're
you're
only
gonna
suss
out
with
like
load
testing.
You
know
finding
issues
with
downstream
circuit,
braking
or
finding
things
where
different
streams
are
interacting
better.
This
is
a
sort
of
thing
that
you
end
up
finding
through
load
testing
and
then
finally,
as
much
as
I
hate
to
admit
it,
which
is
why
I
didn't
include
it
in
my
index
slide.
A
The
really
last
line
of
defense
here
is
adoption.
You
know,
while
production
is
the
last
place
you
ever
want
to
cache.
I
choose
it's
bound
to
happen.
What
are
the
weirder
bugs
I've,
seen
in
my
time
at
Google
was
an
issue
where
it
was
only
for
users
whose
data
was
transparently
proxied
coming
from
one
country
running
a
specific
version
of
OS
X
right
you're,
not
gonna
catch
a
sort
of
thing,
with
load
testing
you're
not
likely
to
catch
up
as
buzzing.
A
This
is
the
sort
of
thing
you
catch
in
prod,
and
the
good
news
here
is
that
envoys
now
used
by
dozens
of
companies
we've
got
tens
of
thousands
of
different
deployments.
Tens
of
millions
of
QPS
I
listed
a
bunch
of
people
from
the
graduation
proposal
and
from
people
who
are
speaking
at
envoy
con,
but
as
an
envoy,
senior
maintainer
I
can
tell
you,
there's
a
lot
more
people
than
this
using
envoy.
I
just
don't
know
who's
actually
admitting
it.
Nowadays.
A
Envoys
has
amazing
uptake
right
and
all
of
these
different
deployments
are
going
to
catch
different
bugs.
So
the
more
users
envoy
has
the
faster.
These
issues
are
gonna
get
found
and
the
larger
our
community
is
the
faster
it's
going
to
get
fixed.
So
in
closing,
I
want
to
thank
all
of
you
who
are
here,
who've,
already
deployed
on
Glee
or
even
Envoy
curious,
because
you're
going
to
end
up
being
a
part
of
her
reliability
story
and
I,
especially
want
to
thank
all
the
people.
Who've
contributed
to
the
online
codebase.
A
A
A
So
the
question
of
this
is
how
much
of
running
envoy
has
good
each
balls.
I
can
tell
you,
as
I,
have
not
personally
deployed
on
Foyet
scale,
I'm,
not
really
sure
I
heard
earlier
today
that
someone
did
have
a
less
and
sensible
default
on
on
how
many
up
streams
I
think
most
of
the
defaults
are
good
and
again
not
everyone
needs
all
of
these
features
right.
If
you're
not
regularly
attacked,
you
may
not
need
the
downstream
circuit
breaking,
but
for
certainly
the
big
industry
players
regularly
could
hit
my
botnets.
A
So
they
need
that
sort
of
thing
implemented
and
I
would
say.
The
nice
thing
is:
is
that,
because
envoy
has
our
little
config
if
you
hit
a
plate
where
all
of
a
sudden
you're
hit
by
your
first
thought,
net
you're
like
oh,
my
god,
what
do
I
do?
You
know
you
can
read
up
on
this
and
then
deploy
that
config,
rather
than
not
really
having
a
choice.
Other
than
doing
code
fixes,
which
is
going
to
have
a
much
longer
managed
raishin.