►
From YouTube: From Microbenchmarks to HTTP2 Load-testing: 5 Performance To... Joshua Marantz & Otto van der Schaaf
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
From Microbenchmarks to HTTP2 Load-testing: 5 Performance Tools and Techniques to Improve Envoy Scalability - Joshua Marantz, Google & Otto van der Schaaf, We-Amp B.V.
As Envoy scales with traffic growth, service complexity, and processor-count, to achieve our performance goals we need an increasing array of tools. We need tools to help visualize latency, throughput, memory, CPU-load, and thread contention.
Some of these tools already exist, such as kcachegrind and Google’s performance benchmarking library. Others needed to be built, such as a new OSS L7 load-tester based on the Envoy networking stack, that is capable of driving HTTP2 traffic through proxies.
In this talk, we’ll discuss these tools and how we’ve applied them to find and fix bottlenecks in Envoy, and help us make decisions about how to improve the system and its usage.
A
So
man,
every
talk
has
been
a
great
lead-in.
Everybody
is
talking
about
how
awesome
the
observability
is,
and
that's
exactly
our
observation
as
well,
that
you
can
kind
of
put
Envoy
next
to
every
system
within
your
service-oriented
architecture,
and
you
get
a
ton
of
visibility,
and
one
thing
I
really
want
to
point
out
is
that
it's
a
very
kind
of
extensible
system.
A
So
the
big
question
for
us
is:
how
does
it
scale-
and
you
know
the
needs
of
the
company
that
originally
built
it
lift,
are
not
exactly
the
same
as
some
of
the
needs
of
the
companies
that
have
been
adopting
it
and
will
do
so
in
the
future,
and
one
thing
that
really
interested
us
is
to
explore
what
happens
when
you
have
a
lot
of
tenets
or
different
applications
sharing
the
same
proxies?
This
might
happen.
A
So
one
of
the
first
things
that
I
found
when
I
started
to
work
with
envoy
with
the
large
number
of
clusters
was,
there
was
some
pretty
severe
quadratic
behavior
in
parts
of
the
system.
This
kind
of
pathological
situation
was
actually
very
easy
to
spot.
In
fact,
just
by
hitting
ctrl
C,
while
in
the
debugger
you
not
only
know
exactly
where
the
problem
is,
but
have
all
the
context,
you
need
to
understand
how
to
solve
it
and
solving
it
was.
A
You
know,
took
some
time,
but
was
not
really
all
that
it
was
relatively
straightforward
and
got
us
a
20x
speed-up,
bringing
the
startup
time
from
over
10
minutes
to
22
seconds,
but
it
wasn't
really
very
satisfying
to
have
to
take
22
seconds
just
to
start
up
approximately
10,000
clusters.
So
we
move
forward
once
again
at
22
seconds.
The
problem
was
pretty
easy
to
spot
in
the
debugger,
but
profiling
tools
are
really
helpful,
as
well.
In
particular,
call
grind
helps
you
identify
like
how
many
times
the
regular
expression
library
was
being
called
10
million.
A
And
of
course,
we
don't
know
why.
You
know
we
still
don't
know
what
the
best
way
to
solve
the
problem
is.
We
also
don't
want
to
just
drop
the
regular
expressions.
They
were
providing
a
lot
of
that
really
great
stats
functionality
like
being
able
to
take
kind
of
a
flat
list
of
stats
and
add
some
structure
to
it,
which
is
really
good
for
some
of
the
stats
visualization
tools
like
Prometheus
and
so
on.
So
we
knew
the
problem
was
regular
expressions
about
how
to
solve.
It
was
not
completely
clear.
A
So
this
brings
us
to
our
second
method
of
performance
improvements
called
annotations,
so
here
we've
put
together
a
really
small
library
of
macros
and
support
functions
and
use
them
to
annotate
the
source
code,
where
we
thought
the
problem
might
lie
and
in
particular,
the
tag
extraction,
regular
expressions.
So
the
way
that
this
works
is
that
you
you
first
of
all,
have
these
macros
that
don't
you
know
they
compile
to
nothing
if
you're,
not
enabling
performance
annotations.
A
But
when
you
do
you
can
the
perf
annotation
macro
kind
of
initiates
at
heimer,
then
there's
a
record
macro
which
records
the
time
and,
in
particular,
there's
kind
of
two
bits
of
annotations
that
you
want
here.
One
of
them
is
a
static
one
which
identifies
in
this
case.
Did
the
code
go
on
the
match
path
or
the
miss
path,
and
the
other
one
which
was
really
important
was
what
exactly
was
the
pattern
that
it
was
trying
to
match
when
it
took
a
lot
of
time?
A
So
once
you
have
all
all
that
you
can
dump
out
the
results
into
a
table
and
you
get
something
that
looks
like
that
and
then
suddenly
the
clouds
kind
of
parted,
because,
first
of
all,
we
noticed
that
almost
all
of
the
high
cost
operations
were
mismatches.
So
basically
we
were
spending,
so
three
of
them
were
more
more
than
the
others
that
were
spending
over
three-quarters
of
the
initialization
time.
A
Basically,
finding
nothing
finding
you
know
tags,
you
know
finding
stats
that
didn't
match
a
tag,
structure
that
we
were
looking
for,
and
only
one
of
the
kind
of
significant
time
consumers
which
took
about
point
four
seconds
was
actually
a
match
that
delivered
any
value
so
diving
into
what
the
regular
expressions
actually
looked
like.
This
was
an
example
of
a
regular
expression,
Bush
extracted
some
cluster
information
and
some
SSL
cipher,
and
we
noticed
that
this
regular
expression
must
begin
with
a
literal
string
cluster
and
must
also
include
another
literal
string,
SSL
dot
ciphers.
A
A
So
when
we
reallocate
the
code,
we
first
of
all
have
to
split
the
system
into
an
initialization
phase
where
we
build
kind
of
a
prefix
match.
So
all
of
the
you
know
all
of
the
regular
expressions
that
might
match
cluster
dot
will
be
put
into
a
list
and
those
are
the
only
ones
that
we
want
to
evaluate,
and
then
we
can
also
do
the
skip
of
the
Reg
ax
eval
when
the
required
substring
is
not
in
there.
So
this
brings
up
a
very
different
story.
Now.
A
A
The
question
is:
do
you
stop
here,
and
the
answer
is
yes,
we
actually
stop
there,
but
that's
kind
of
good
enough,
but
but
you
know,
for
the
purposes
of
this
talk,
I
said
well.
Why
don't?
We
just
keep
going
a
little
bit
further,
because
that
point
four
seconds
looks
like
it
might
be
low-hanging
fruit.
So
now
we
can
dive
into
the
regular
expression
library.
A
Actually,
the
other
thing
that
happened
in
the
meantime
was
that
Matt
added
re
support,
Envoy,
which
kind
of
enabled
this
to
occur,
which
is
the
other
the
other
kind
of
gating
item
for
this
particular
optimization.
So
the
first
thing
that
we
want
to
do
is
is
look
deeply
at
how
well
the
regular
expression
library
is
performing.
We
know
that
student
Rugg
X
is
not
the
best
implementation
for
our
purposes.
It
has
a
lot
of
platform
variants.
It
actually
allocates
memory
during
the
matching
phase,
at
least
the
way
that
we
were
calling
it
in
some
cases.
A
So
it's
time
to
bring
in
micro
benchmarks,
because
if
you
want
to
make
a
decision
like
moving
from
one
library
to
another,
you
want
to
do
so
with
real
data,
so
micro,
benchmarking
helps
measure
the
performance
of
a
very
small
and
specific
piece
of
code.
Google
has
a
micro,
benchmarking
library
which
I'm
really
familiar
with,
but
also
there's
tons
of
other
alternative
micro
benchmark
libraries
that
you
can
find
on
the
web.
So
these
are
really
good
at
many
phases
of
development,
including
when
you're
first
writing
code.
A
So
the
this
is
the
exact
regular
expression
that
was
in
place
and
I
kind
of
looked
at
that
and
thought
it
wasn't.
The
most
intuitive
way
to
write
it.
So
I
wrote
it
a
different
way
and
we
can
try
that
one.
We
can
try
it
with
with
multiple
regular
expression
library
see
which
one
works
at
best,
and
this
is
what
we
came
out
with.
A
So
the
the
starting
point
was
that
it
took
about
1.3
microseconds
to
evaluate
that
regular
expression
on
a
candidate
set
of
of
inputs,
and
actually
that's
a
pretty
small
number,
and
if
that
was
being
done
once,
then
you
would
be
done,
and
that
would
be
fine.
But
this
was
being
done
a
million
times
for
that
one.
You
know
per
regular
expression,
I
mean
per
yeah
for
each
regular
expression.
A
So
it
was
worth
optimizing,
just
switching
the
regular
expression
library
to
use
a
string
view
rather
than
a
string
reduce
the
number
of
allocations,
and
that
was
kind
of
a
very
easy
way
to
get
five
percent
improvement
and
then
switching
to
that
alternate
pattern.
I
thought
that
was
gonna
make
things
better
withstood
reg
X,
but
it
made
it
substantially
worse.
I
was
surprised
at
that.
I
still,
don't
really
know
why
that
is,
but
with
re2
everything
was
much
better.
A
It
brought
it
down
to
a
microsecond
for
the
original
form
and
to
less
than
half
of
a
microsecond
for
the
alternate
version
that
I
wrote
so
now
that
that,
basically
more
than
cut
in
half
the
time
that
we
spent
on
that
on
the
longest
regular
expression
improved
the
overall
start
up
envoy
by
10
percent,
really
for
just
changing
row,
get
1
reg
X.
That
seemed
worth
it.
A
So
so
far,
we've
been
talking
about
startup
performance,
but
proxies
spend
a
lot
more
time
moving
packets
around
than
they
do
initializing.
So
we
need
also
application
load
test
infrastructure
and
when
you're
building
a
really
high
performance
system.
When
you
have
a
really
well
performing
proxy,
like
Envoy,
you
also
need
a
really
high
performance
piece
of
test
infrastructure
in
order
to
adequately
measure
it.
You
want
one
with
really
low
jitter.
A
You
know:
building
load,
balancers
and
improving
on
voice
control,
plane
and
other
things,
and
we
really
wanted
to
bring
in
somebody
to
help
us
evaluate
what
the
next
step
should
be.
That
had
a
lot
of
experience
with
those
with
this,
so
we
contacted
auto
vendor
chef,
who
will
take
us
through
the
load,
generator
yeah.
B
So
we
were
looking
at
something
that
we
could
modify
and
improve,
and
for
that
we
will
need
a
code
base
that
would
be
well
tested
like
invoice
and
with
a
licensing
scheme
that
would
work
for
us.
So
we
evaluate
that
ten
existing
tools
and
based
on
the
earlier
requirements
we
were
able
to
narrow
them
down
to
two
w
RK
and
40
o
so
well.
40
L
has
some
great
technical
properties.
What
we
didn't
find
was
h2
support
or
a
community.
B
4008
kind
of
like
a
different
set
of
trade-offs.
We
also
didn't
find
the
community
we
were
looking
for
whatever.
Also
some,
the
the
the
the
latency
is
reported,
were
kind
of
like
less
accurate
than
compared
to
W
or
K
and
control
measurements
in.
We
could
pinpoint
those
two
and
even
request,
release,
timings
and
possibly
garbage
collection
and
go
scheduler
and
fairness
issues.
B
B
That
would
allow
us
to
piggyback
upon
a
networking
stack
that
is
being
invested
by
by
a
lot
of
people
across
a
lot
of
companies.
Concretely,
HTTP
3
support
is
in
development
as
we
speak,
and
once
that
lanced
and
nighthawk
would
basically
get
that
for
free,
and
we
also
decided
to
make
two
visualization
agnostic.
So
we
would
exploit
existing
visualizations,
including
for
TOS
that
new
tool
is
called
Night
Out.
B
It
supports
closed
loop
mode
where,
when
I
requested
issues,
another
one
will
not
be
issued
until
the
reply
from
the
server
has
been
observed,
and
this
mode
strives
to
provide
a
consistent
stress
on
the
surface
throughout
time
in
closed
loop
mode,
Nigel
will
report
how
much
time
it
was
spent
blocked
waiting
for
the
surfer
to
respond
in
open
loop
modes.
A
low
generator
can
send
up
of
multiple
asynchronous
requests
without
waiting
for
the
server
to
respond.
This
is
more
like
how
browsers
work,
especially
in
HTTP,
to
an
open
loop.
B
We
strive
to
provide
a
cost
of
work
load
to
the
server
throughout
time
within
configured
resource
limits,
so
in
open
loop,
there's,
no
blocking
diagram,
and
instead,
if
the
server
can't
keep
up
with
the
load,
we
may
observe
local
connection
pool
overflows,
meaning
that
the
request
is
skipped
in
this
case.
From
the
surface
point
of
view,
we
can
use
nighthawk
to
collect
accurate
data
and
export
it
to
4tl
to
visualize
the
long
tail.
B
B
B
B
Next,
we
look
at
in
comparison
of
HTTP
1
versus
HTTP
2
and
surprisingly,
we
observe
HTTP
one
to
be
faster,
then
HTTP,
2
by
5
microseconds
and
that's
kind
of
unexpected,
because
HTTP
2
is
a
binary
protocol
and
you'd
expect
that
to
be
faster.
So
we
didn't
take
into
this.
But
this
is
like
an
interesting
shell
case
of
whites.
A
Thanks
Otto,
okay,
so
to
finish
up
with
the
last
technique,
I
wanted
to
describe
something:
that's
maybe
somewhat
more
speculative
and
crazy,
which
is
using
fuzzing
as
a
performance
tool.
So
fuzzing
is
not
intended
to
be
a
performance
tool.
It's
intended
to
find
potential
security
problems,
but
it
also
finds
among
those
denial
of
service,
Valle,
'nor
abilities,
which
are
really
a
flavor
of
performance
issue
where
the
data
controlled
outside
the
system
can
influence
its
performance.
A
Here
you
can
imagine
using
this
table
kind
of
the
way
that
a
day
trader
would
use
a
stock
screen
to
pick
out
potential
interesting
investments,
so
in
this
case
a
fuzzer
that
runs
very
slowly
less
than
two
evaluations
per
second
and
it
reports
a
lot
of
timeouts
failures
due
to
time
out,
and
it
doesn't
crash
too
much.
I
should
say
that
crashing
in
fuzzers
actually
is
not
that
unexpected.
It
doesn't
mean.
There's
a
security
problem.
A
Necessarily
or
a
problem
with
the
code,
usually
you're,
running
fuzzers,
on
a
subsystem
with
preconditions
and
when
those
preconditions
are
violated,
you
may
get
an
assertion,
so
it
doesn't
mean
that
there's
a
problem,
but
it
means
that
the
fuzzer
runs
really
slowly,
because
the
assertions
take
a
long
time
to
execute
even
when
they
pass
and
if
they
fail.
It
really
takes
a
long
time
for
the
fuzzing
infrastructure
to
recover,
and
you
waste
a
lot
of
your
your
time
kind
of
rebooting,
your
system,
the
fuzzing
infrastructure
itself,
is
slow
as
well.
A
So
if
you're,
finding
like
less
crashes
than
the
other
ones
like
7%
and
a
bunch
of
timeouts,
maybe
it's
a
performance
bug.
So
you
get
this
automated
notification
that
you
had
a
fuzz
timeout,
and
this
happened
to
be
an
area
that
we
were
a
lot
about
anyway,
which
is
the
Envoy
HP
header
class.
And
then
you
can
pull
up
a
flame
graph
from
that
and
there
it
actually
shows
the
problem
pretty
clearly
that
there
was
an
N
squared
issue
with
updating
account
when
new
when
new
headers
arrived.
A
Similarly,
it's
almost
always
harder
to
find
these
problems
than
it
is
to
fix
them.
The
fix
was
relatively
straightforward.
This
resulted
in
a
CVE
for
an
N
squared
sensitivity,
but
the
main
point
is
that
the
fuzzer
helped
us
kind
of
find
this
problem
and
it
can
find
other
performance
related
problems
as
well.
So
this
is
not
a
slam-dunk.
A
A
So
next
steps
I
think.
The
big
news
here
is
that
we've
put
a
ton
of
work
we
meeting
mostly
auto,
but
also
the
people
that
have
been
revealing,
Harvey
and
others,
and
we've
evolved.
What
I
think
is
the
bones
of
a
really
useful
performance
tool
that
works
it
in
with
a
modern
architecture,
on
top
of
the
Envoy
Network
stack
speaking
HTTP,
and
it
will
inherit
the
ability
to
speak,
quick
and
HTTP
three
as
well.
A
A
Echoing
Matz
earlier
comments,
I
think
that
if
we,
if
we
think
this
kind
of
tool
is
valuable,
we
would
love
to
have
more
people
participate
in
this
and-
and
please
join
us-
it's
github
envoy,
proxy
nighthawk,
so
fuzz
testing
for
performance
is
more
of
a
hope.
I
hope
something
gels
out
of
this
I
think
that
it
would
be
potentially
useful
to
find
performance
problems
using
this
infrastructure
and
I
think
its
goals
are
also
really
aligned
with
the
needs
of
the
fuzzing
for
security.