►
From YouTube: Fuzz Testing of Envoy - Adi Peleg & Teju Nareddy, Google
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Fuzz Testing of Envoy - Adi Peleg & Teju Nareddy, Google
How can we be assured of the correctness and safety of the many Envoy parsers and state machines in the presence of untrusted or adversarial input? While developers cover main scenarios using tests, complex edge cases may be missed. Adversaries may be able to exploit these cases to trigger denial of service attacks, access Envoy process memory remotely, or trigger remote execution of malicious code. Fuzzing is an automated software testing technique that provides randomized input to the system under test (SUT). Some tests may use a variety of sanitizers to check for violations of memory safety, check for invariants expressed as assert statements or abnormal program terminations or timeouts. Other tests may compare behavior of different SUTs to the same input. This talk will include an overview of different fuzzers in Envoy, the OSS-Fuzz infrastructure for running fuzz tests, some bugs fuzz tests discovered, and examples of creating specific fuzz tests for ESF components in Envoy.
A
A
B
B
A
A
A
A
A
B
A
B
A
B
B
B
This
functionality
is
split
over
two
components:
the
config
plan
and
the
data
plane
in
the
config
plane,
the
library
parses,
the
input,
access,
log
format
string
into
a
list
of
internal
formatters
than
in
the
data
plane.
Each
formatter
runs
on
every
single
request
response.
Each
formatter
extracts
information
from
the
headers
trailer
stream
info
to
create
the
final
string.
B
There's
three
main
steps:
we
follow
to
create
a
fuzzer
and
Envoy
proxy.
First,
we
Define
the
fuzz
input
schema
next,
we
write
the
CC
fuzz
test
and
finally,
we
add
in
the
initial
Corpus,
let's
walk
through
these
one
by
one,
so
first
husband
foot
schema.
The
input
schema
essentially
indicates
to
the
fuzzing
engine.
The
types
of
data
to
generate
our
goal
is
to
make
an
input
schema
that
captures
the
true
input
space
of
the
library
under
test.
B
B
On
produce
here,
so,
let's
consider
what
the
inputs
for
the
access
logger
Library
are.
The
most
evident
input
is
the
axis
slug
formestering.
We
want
the
fuzzing
engine
to
Generate
random
format,
strings.
To
indicate
this
we
add
in
a
string
field
into
the
protocol
buffer
message
now
in
the
fuzzing
engine
runs,
each
iteration
will
generate
a
random
string.
A
B
A
plain
old,
empty
string:
all
of
these
are
valid
inputs.
The
axis
logger
should
handle
now
that
only
fuzzes
the
config
path
of
the
axis
logger.
We
also
want
to
fuzz
the
data
path
to
do
so.
We
need
the
access.
We
need
the
buzzing
engine
to
create
the
request
and
response
information,
such
as
the
request,
headers
response,
headers
response
trailers
in
the
Stream
info.
We
add
all
these
fields
into
the
fuzz
input
schema
notice
here
that
these
fields
are
references
to
other
protocol
buffer
messages
under
the
test.pose
package.
A
B
B
B
B
B
Once
we
have
all
that
request
and
response
information,
we
pause
the
data
path
we
Loop
over
the
formatters
and
we
call
the
format
method
on
each
one.
Passing
in
the
request.
Headers
response,
headers
response,
trailers
and
stream
info
do
note
here
that
we're
not
checking
the
final
output
of
the
format
function.
Remember
this
is
a
pause
test.
All
we're
trying
to
do
is
generate
as
many
valid
inputs
as
possible
and
pass
them
to
a
library
under
test
we're
not
trying
to
verify
the
correctional
the
correctness
or
the
functional
behavior
of
the
axis
logger.
B
B
So,
for
example,
in
the
first
input
schema,
we
indicated
to
the
fuzzer
that
we
wanted
to
Generate
random
access
log
strings,
but
we
never
told
the
pausing
engine
what
a
valid
access
log
format
directive
looks
like,
so
the
fuzzing
engine
will
have
to
do
a
lot
of
trial
and
error,
Brute
Force
to
find
realistic
inputs,
realistic
values.
We
can
optimize
this
by
adding
in
the
initial
Corpus.
B
B
Valid
HTTP
Response
Code
into
the
stream
info.
Similarly
I
added
another
example
with
Upstream
local
address
directive
and
then
I
filled
in
a
valid
ipv4
address
and
a
valid
port
number
into
the
stream
info.
With
these
inputs,
the
fuzzing
engine
can
generate
new
inputs
via
mutation,
and
it
will
generate
new,
realistic
inputs,
for
example.
This
is
a
possible
mutation.
The
fuzzing
engine
might
do
when
creating
a
new
input,
so
it
could
take
the
two
axis
log
formatting
directives
and
append
them
together
to
one
string.
B
It
could
copy
over
the
Response
Code
information,
but
flip
a
single
bit
resulting
in
an
invalid
HTTP
code
and
it
might
even
just
completely
mutate
the
address
putting
in
special
characters
in
the
address
zeroing
up
the
port
value.
So
this
input
looks
nonsensical
to
us
as
developers,
but
it's
still
a
valid
input.
The
fuzzing
engine
will
pass
this
to
your
fuzzer
and
your
fuzzer
should
ensure
the
access
logger
Library
can
handle
us
gracefully
without
any
crashes
and
without
any
undefined
Behavior.
A
A
A
B
A
B
One
key
point
I
want
to
stress
is
that
we're
not
trying
to
fuzz
every
single
Target
in
Envoy
proxy.
Adding
a
new
fuzz
test
has
some
computational
cost
and
a
little
bit
of
Maintenance
burden.
Instead,
we
prioritize
what
targets
to
fuzz.
We
prioritize
based
on
two
main
attributes:
the
traffic
type
and
the
complexity
we'll
break
those
down
traffic
type
is
essentially
asking
what
type
of
actor
is
your
code
exposed
to,
for
example,
if
your
library
is
completely
on
the
config
path,
it's
probably
lower
priority
to
fuzz,
because
we
trust
our
configs.
B
We
trust
the
person
deploying
Envoy
or
we
trust
the
xgs
server
we're
getting
our
configs
from.
If
your
library
is
on
the
data
path,
it's
worth
further
breaking
down
the
trust
level
by
your
connection.
So
if
you
have
envoid
applied
deployed
as
the
Gateway,
perhaps
you
have
untrusted
Downstream
clients.
If.
A
B
Have
Envoy
in
a
multi-tenant
model,
perhaps
you
have
untrusted
Upstream
clients
or
if
your
library
makes
out
any
other
external
service
calls.
Do
you
trust
that
interaction
with
that
external
service?
It's
really
important
to
consider
that,
because
you
want
to
focus
on
fuzzing
inputs
that
are
untrusted
inputs
from
malicious
actors
that
are
trying
to
break
your
deployment.
B
Please
reference
the
envoy
threat
model
if
you're
unsure
what
type
of
actor
your
code
is
rated
for.
A
good
example
of
this
is
the
grpc
Json
transcoder.
The
transcoder
is
a
HTTP
filter
that
has
both
decoder
and
encoder
paths.
If
you
look
at
the
envoy
threat
model,
the
transcoder
is
classified
as
follows:
robust
to
untrusted
Downstream,
but
assumes
trusted
Upstream.
B
So
with
that
classification,
we
know
that
all
the
malicious
clients-
or
we
assume
all
the
malicious
clients,
are
on
the
downstream
side.
When
we
were
adding
fuzz
tests
for
the
transcoder,
we
focused
on
just
us
testing
the
downstream
we
still
added
fuzz
tests
for
the
Upstream
encoder
code
paths,
but
we
really
cared
about
the
code
coverage
on
the
downstream
or
decoder
code
paths
that
was
able
to
reduce
our
fuzz
work
in
half.
For
the
transcoder
going
back
to
prioritization
the
other
attribute
to
Prior
test
bias
complexity.
B
B
First
of
all,
we
have
utility
fuzzers
and
Envoy.
These
are
fuzzers
that
fuzz
parsley-like
libraries
such
as
the
access
logger
walkthrough
we
just
did.
These
are
pretty
easy
to
write,
very
efficient,
very
fast
and
high
signal.
We
recommend
that
a
hundred
percent
of
utilities
have
fuzzers
in
our
repo.
The
second
type
of
fuzzer
is
configuration.
Fuzzers
he's
focused
on
fuzzing,
just
the
config
service
config
surface,
like
the
server
initialization
and
the
xgs
inputs.
B
These
are
also
easy
to
write
but
consider
the
threat
model.
They
might
be
lower
priority
because
we
trust
our
inputs.
We
Trust,
our
configs
things
get
more
interesting
with
the
third
fuzzer.
These
are
HTTP
data
plane
fuzzers,
these
various
portions
of
the
data
plane
from
HTTP
codecs
to
routing
header,
processing,
they're
pretty
complex,
and
they
come
in
a
lot
of
different
styles.
When
we
were
writing
some
buzzers
in
this
category,
we
found
that
they
were
not
very
accessible
to
the
open
source
community.
B
A
B
Http
filter
we
created
a
framework
to
simulate
decode
and
encode
calls
to
the
filter
under
test,
and
we
also
added
helpers
to
generate
headers
trailers,
HTTP
data
grpc
data.
These
are
the
same
helpers.
You
saw
in
the
walkthrough
with
this
new
category,
any
open
source,
contributor
or
any
filter
maintainer
can
add
their
own
fuzz
tasks.
B
B
B
It's
important
that
you
that
the
input
space
matches,
if
you
don't
have,
if
your
fuzzing
engine
isn't
generating
correct
inputs,
you
probably
won't
be
triggering
edge
cases
in
your
underlying
Library.
You
can
fix
that
by
you
know
expanding
out
your
fuzz
input
schema,
but
when
you
do
that
you're
making
your
inputs
much
more
complex,
this
makes
it
hard
for
the
fuzzing
engine
to
generate
new
values.
B
When
you
do
that,
you
have
a
loss
of
efficiency.
Efficiency
is
the
fuzz
rate.
How
many
times
a
second,
the
fuzzing
engine
can
generate
a
new
value
and
feed
it
to
your
instrumented
code.
Efficiency
is
also
important.
If
you
don't
have
high
efficiency,
fuzzing
engine
doesn't
have
time
to
explore
new
input
space
new
state
space.
You
want
to
maintain
high
efficiency.
B
You
can
usually
improve
efficiency
by
adding
in
some
domain-specific
knowledge,
for
example,
some
config
validation
checks
or
some
short
circuiting
in
your
fuzzer
to
reject
uninteresting
inputs
early
before
they're
passed
to
your
instrument
to
code.
When
you
start
adding
in
all
these
checks,
you
end
up
with
higher
maintenance
burden.
Now
your
fuzzer
is
scattered
with
tons
of
tiny
checks.
You
have
a
lot
of
code
complexity
and
if
you
ever
need
to
change
your
library
under
chest,
you
also
need
to
change
your
fuzzer.
You
have
to
make
sure
the
assumptions
between
the
two
match.
B
So
there's
no
one
solution
that
fits
every
single
funder.
Our
best
advice
is
to
start
by
writing
a
fuzzer,
that's
easy
to
maintain
and
efficient,
but
maybe
not
so
realistic.
Let
the
fuzzer
run
on
OSS
fuzz
for
a
few
days
and
then
later
measure
the
code
coverage.
If
you're
unhappy
with
the
code
coverage,
you
can
look
at
ways
to
increase
the
fuzz
input
schema
while
trying
to
trying
to
deal
with
the
loss
and
efficiency
by
adding
in
some
optimizations.
A
B
A
B
A
The
filters
interface
of
any
HTTP
filter-
these
fuzzers
are
also
part
of
the
envoy
repository,
so
you
can
look
it
up
on
the
envoy
repo
from
GitHub
there's,
also
a
third
type
of
fuzzers,
those
that
Target
extensions
that
are
not
part
of
the
envoy
for
onward
proxy
repo.
These
puzzers
can
be
executed
by
the
OSS
infrastructure
if
the
project
that
hosts
them
has
integrated
me,
those
as
possible,
but
how
can
a
community
help?
B
A
A
B
A
A
We
would
like
to
thank
Harvey
to
azra
Ali
and
Matt
Klein
for
all
their
efforts
in
setting
up
the
envoy,
fuzzing
libraries
and
infrastructure,
and
of
course
there
are
many
to
the
rest
of
the
envoy
Community
for
their
contributions
in
developing
new
fuzzers,
addressing
bugs
and
making
Envoy
more
secure
and
robust
in
the
future.
We're
looking
into
integrating
the
envoy
CI
with
cluster
fuzzlite
that
executes
fuzzers
when
a
PR
is
submitted
and
attempts
to
find.
B
A
A
B
B
A
Is
what
we
call
the
signal
to
noise
ratio
right
so
and
to
address
them?
We
need
to
understand
a
specific
which
puzzer
we're
talking
about,
for
example,
if
we're
talking
about
the
conflict
plane,
which
complicates
fuzzles,
which
the
the
input
space
is
very,
very
large,
it's
harder
to
just
take
some
input
and
say:
okay,
let's
run
this
also,
if
you
look
at
the
thread
model,
usually
the
the
the
Comfort
plane
is
considered
to
be
thrusted
and
so
we're
reducing
the
amount
there
now
detecting
it,
detecting
it's
easy.
A
B
A
B
B
B
B
A
So
with
these,
when
you,
whenever
you
fix
these
kinds
of
bugs,
you
don't
see
the
excitement
as
in
you
know,
there
was
an
issue
and
everything
crashed
and
then
you're,
like
okay
I
found
the
the
fixed
rate
right.
This
is
before
everything
you
know
your
system
correct.
So
it's
you
don't
get.
The
excitement
of
you
know
I
fixed
the
bugs
in
production,
because
the
you
just
prevent
the
bug
from
going
into
production.
That's
the
idea.
A
B
Yeah
and
just
going
a
little
deeper
into
that,
if
you
looked
at
our
cve
list,
if
you
looked
closely
at
it,
you'd
notice,
all
the
CVS
we
listed
were
2019.,
so
I
assume.
The
reason
that
happened
is
because
that's
when
we
first
started
adding
in
fuzzers
and
we
caught
all
those
bugs
in
production
now
that
we
have
our
fuzzers
running,
we're
not
catching
any
production
issues,
we're
catching
these
bugs
earlier
and
fixing
them
before
they
make
it
out
to
production.