►
From YouTube: Are You Sure About Your Mesh Performance? - Otto van der Schaaf, Sunku Ranganath & Mrittika Ganguli
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Are You Sure About Your Mesh Performance? Details matter! - Otto van der Schaaf, Red Hat, Sunku Ranganath & Mrittika Ganguli, Intel Corporation
Service Mesh performance characterization has been an elusive aspect of understanding the impact of mesh in production. While few studies have been published, mesh performance generally is highly influenced by runtime environment, hardware settings, test tool & methodology used to benchmark. Based on various tests performed on Envoy, this presentation aims to shed light on: -Performance characterization methodology of Envoy for deterministic throughput & latency -Gaps in benchmark tools - disconnect between L2 to L7 optimizations for load generation & features WIP to address these gaps, e.g. with Nighthawk -Common pitfalls in measurements -Usual culprits for lack of consistency in benchmarks -Impact of scaling Envoy on latency & hardware utilization -Share benchmark results & common bottlenecks using Envoy sandboxes -Customizing Envoy for telco grade performance with hardware offloads
A
Hi
everyone,
I'm
sun,
kuranganath,
I'm
a
network
software
engineer
in
intel,
working
on
enabling
service
mesh
for
5g
and
edge
environments
topic
today,
I'm
presenting
title
are
you
sure
about
your
mesh
performance
details
matter
along
with
myrthika
and
otto
here's,
the
legal
disclaimer
high
level
agenda?
We
start
off
looking
at
some
of
the
important
aspects
to
keep
in
mind
while
measuring
your
mesh
deployment.
A
We
then
share
the
some
of
the
results
and
observations
with
the
performance
studies
we
have
done
on
bare
metal
and
virtualized
environment.
Otto
would
then
talk
about
some
of
the
benchmarking
pitfalls
around
tooling
and
test
environment
and
method.
I
would
end
the
talk
sharing
some
of
the
ongoing
work.
A
So
if
you're
someone
looking
to
understand
your
mesh
deployment,
you
would
sort
of
setting
up
your
cluster.
Your
service,
mesh
application
under
test
pick
a
benchmarking
tool.
Configure
your
transactions
per
second
or
protocol
of
interest
test
duration
connections
then
run
the
test
and
capture
the
results.
However,
if
you
closely
look
at
this
picture,
what's
missing
right
so
turns
out
quite
a
lot
for
the
purposes
of
this
talk.
A
A
So,
depending
on
how
you
configure
the
number
of
hops
between
your
source
and
destination,
that
your
mesh
performance
has
a
huge
impact.
For
example,
your
load
balancer,
your
api
gateways,
ingress
controllers
firewall.
All
of
these
add
a
good
amount
of
latency
impacting
overall
performance
of
your
mesh,
not.
A
That's
your
hardware
settings
related
to
your
bias,
settings
power
management,
new
awareness,
type
accelerators
or
your
operating
system.
Settings
type
of
networking
stack.
You
would
use
kernel
versus
user
plane
stack
all
the
type
of
tuning.
You
would
do
across
your
l2
and
three
layer
right
and
your
load
generator
settings.
All
of
these
have
a
good
amount
of
impact
on
your
mesh
performance,
so
for
the
ultimately
what's
important
is
to
have
a
method
to
have
a
consistent
results
across
a
repeated
number
of
test
cycles
right.
So
that's
the
crucial
part.
A
So
we
started
off
looking
at
performance
of
envoy
leveraging
the
unvoiced
friend
proxy
sandbox
available
part
of
the
online
source
code.
Well,
if
you're
someone
new
looking
to
understand
service
mesh
with
envoy
front
pro,
the
sandboxes
provided
are
really
great
way
to
understand,
which
is
where
I
was
just
a
few
short
months
ago.
A
So
in
this
example,
front
proxy
sandbox
provides
a
simple
enviro,
acting
as
a
load
balancer
at
the
front
service.
Saying
two
services,
as
I
was
one
service,
two,
each
of
them
having
a
simple
flask
app
and
on
y,
has
a
cycle
process
in
a
in
a
docker
container,
so
type
of
test
we
do
is
scale
the
number
of
service
one
end
points
from
either
anywhere
from
one
to
100.
Look
at
the
amount
of
transactions
resolved
clear,
latency
change,
the
traffic
generator
settings.
A
So
to
give
you
an
idea
of
the
results.
So
if
you
look
at
here
on
the
x-axis,
you
have
a
number
of
on
app
or
on
y
microservices
from
anywhere
from
two
to
hundred
and
y
axis.
Here
one
side
you
have,
the
transactions
result.
Other
side,
you
have
latency,
you
could
see
from
the
bar
graphs
of
with
the
one
core
versus
two
versus
four
course,
especially
you
have
when
you
have
a
higher
container
count
about
hundred
at
a
thousand
dps
input.
A
Second,
if
you
look
at
the
cpu
utilization
across
the
is
four
cores
and
when
still
cross
1000
tps
input
rate,
your
cpu
utilization
for
your
flask
cap
and
cycle
proxy
goes
close
to
100
right
anything
over
1000
right,
so
reaching
100,
which
is
which
is
a
lot.
So
essentially,
we
found
that
number
of
cores
have
a
significant
impact
on
overall
mesh
performance,
isolated
course
and
core
pinning
are
necessarily
helpful,
which
is
what
some
of
the
telco
deployments
do.
A
We'd
use,
obs,
dpdk
right,
so
some
of
the
top
coated
plywoods
use
either
obs
ppdk
or
sioe
type
of
scenarios.
In
this
example,
we
have
two
polema
drivers
servicing
the
data
plane
traffic
and
the
idea
is
to
understand
the
impact
of
istio
in
this
type
of
deployment.
Right
and
ford.
Io
client
would
talk
to
nginx
web
server
running
in
a
pod,
along
with
on
voice
site
called
proxy.
A
So
to
give
you
a
brief
view
of
the
results,
all
right
so
for
the
vm
to
vm
case
where
ford,
io
and
vm
versus
engine
export,
another
vm
under
two
different
hosts,
you
could
see.
Adding
istio
adds
about
three
times
the
latency
across
different
tps
rate
and
at
about
10
000
dps,
where
without
steel
you
could
resolve
10
000
dps.
It
still
adds
in
about
46
performance
degradation
and
with
respect
to
the
case
where
ford
io
is
on
the
master
host.
A
Reaching
the
engine
x
bar
in
the
vm
istio
adds
about
four
times
the
latency
and
about
11
11
performance
degradation
about
10,
000,
dps
rate
in
hyper
traffic.
You
could
see
anywhere
from
32
to
60,
depending
on
the
case
right.
So,
essentially,
what
we
found
is
there's
a
good
amount
of
performance
impact,
adding
istio
in
a
virtualized
environment,
and
we
could
see
the
across
the
stack
starting
from
cpus.
A
In
this
case
we
have
10
and,
of
course,
made
available
for
vm
right,
so
a
cpu
number
of
course,
type
of
course,
calico
config
or
respective
mtu
connection
rate
right
or
your
on
y
config,
with
respect
to
number,
of
course,
versus
number
of
online
worker
threats
right.
So
there
are
a
lot
of
variables
across
the
stack
that
we
could
look
into
or
tune
into
a
few
observations,
so
hardware
tuning,
a
lot
of
things
could
be
done,
for
example,
for
our
management
policies.
Simple
thing
to
do
is
turn
off
all
power
management
options.
A
However,
we
found
and
depending
on
the
peace
states
and
c-state
conflict
conflict,
have
a
good
impact
on
dail
latencies,
enabling
hyper
threading
improves
performance.
Core
isolation
isn't
necessarily
helpful.
For
example,
some
of
the
cases
we
described
earlier
with
bare
metal
tests
right
impact
of
tuning,
a
vm
for
new
locality
or
pinning
queue
threads
on
on
the
isolated
course
have
a
as
an
impact
on
tail
latencies.
A
Otto
would
touch
upon
some
of
these.
In
his
part
also,
we
found
some
of
the
l2
l3
traffic
generators,
leverage,
rfc
2544,
but
for
l7
benchmarking.
We
haven't
found
loads
in
later
leveraging
in
standardized
way
right,
so
we
could
see
overall,
optimizations
could
be
done
from
a
load
generator
perspective
across
the
stack,
etc.
A
In
fact,
we've
been
discussing
a
lot
of
this
work
as
part
of
cncf
service.
Mesh
working
group,
which
is
part
of
a
sig
network
and
the
the
project
we've
been
talking
about.
A
lot
of
these
aspects
is
service
mesh
performance.
In
fact,
this
project
has
been
submitted
to
cncf
to
be
considered
as
a
sandbox
project.
Essentially,
smb
provides
a
standardized
way
of
running
these
mesh
performance
measurements.
A
Looking
at
the
vendor
neutral
way
of
specifying
mesh
deployment
patterns,
your
environment
infrastructure
capturing
a
test
results
essentially
in
a
automated
way,
for
example,
leveraging
measuring
right,
and
so
this
is
where,
in
fact,
I
met
otto
for
the
first
time
and
we
had
a
quite
a
few
fruit,
fruitful
conversations
after
that.
So
with
that
pass
it
on
to
otto.
C
C
C
In
that
test,
wrk2
was
used
to
compare
latencies
across
the
two
proxies
and
envoy
seemed
to
add
two
or
three
times
as
much
latency.
At
some
point,
I
figured
that
sanity
checking.
These
lit
numbers
was
where
I
warranted,
so
I
set
up
a
reproduction
and
next
to
wrk.
I
used
ford,
io
and
nighthawk
to
measure
latencies,
while
doing
so
wrk
reproduced
mean
latencies
in
the
right
ballpark,
but
fortio
reported
latencies
about
twice
as
high
and
nighthawk
would
in
turn
report
latencies
in
order
of
magnitude
lower.
They
couldn't
all
possibly
be
right.
C
In
the
discussion
that
fallout,
we
learned
that
the
reported
measurements
had
been
obtained
by
executing
on
a
virtual
machine,
whereas
my
reproduction
was
on
a
tuned
bare
metal
machine.
Still,
that
did
not
explain
the
divergence
between
nighthawk
and
the
other
tools
to
understand
what
happened
there.
We
need
to
dig
into
the
subtleties
around
request,
release,
timings
and
connection
reuse.
C
C
The
same
connection
and
queries
per
second
parameterization
can
be
expressed
in
alternative
ways.
For
example,
a
load
gen
might
depend
on
internal
or
external
http
libraries,
which
will
not
prefetch
connections.
This
could
in
turn
mean
that
only
a
single
connection
is
actually
involved,
where
one
might
not
easily
think
that
four
would
have
been
involved.
C
C
Connection
pools
tend
to
use
the
most
recently
used
strategy
when
picking
a
free
connection
from
the
available
ones,
depending
on
the
specifics
of
what
is
being
tested.
A
single
or
a
few
large
pools
behaving
like
this
may
or
may
not
be
desirable
when
realistically
trying
to
simulate
browsers.
It
is
probably
better
to
have
lots
of
small
pool
instances
that
behave
like
this
when
simulating
a
downstream
proxy
one
or
only
a
large
few
of
these
pools
may
reflect
reality.
C
C
C
Load
generators
diverge
here
and
how
they
handle
this
closed.
Loop
load
generators
may
just
block
and
release
when
a
connection
frees
up
some
track
time
being
blocked.
Others
will
try
to
use
math
to
correct
for
missed
requests
in
histogram
output,
open
loop
generators
may
report
connection
and
stream
overflows
regardless
of
closed
or
open
loop
methodology.
C
C
C
C
C
C
C
C
C
To
a
phenomena
called
catastrophical
cancellation
when
using
a
naive
approach
and
through
that
tooling
can
produce
distorted
numbers
when
comparing
baselines
of
different
servers
and
meshes
or
even
across
load
generators.
It
is
good
to
be
aware
that
diverging
tls
cipher
send
or
session
reuse
characteristic.
C
So
when
it
comes
to
reporting
results,
it
is
common
for
a
tooling
to
produce
a
single
histogram,
sometimes,
however,
being
able
to
dive
a
bit
deeper
is
useful.
This
example
shows
a
single
globally
aggregated
histogram
on
the
left.
It
is
under
its
underlying
data,
comes
from
measurements
across
multiple
distinct
workers
which
run
on
different
threads.
C
The
workers
all
use
their
own
connection
pool
the
plot
shows
a
broken
knee,
which
is
kind
of
odd
on
the
right.
We
visualize
the
output
once
more,
but
now
we
also
include
the
perverter
plots,
those
all
look
as
regular
latency
plots,
and
now
it's
much
easier
to
see
how
the
aggregated
view
obtained
its
odd
shape.
C
C
B
C
Has
some
mitigations
throughout
this
today,
but
your
mileage
may
vary
depending
on
the
technology.
You
are
running
tests
against
unless
you
are
specifically
testing
this
aspect
of
a
system,
it
might
be
good
to
explicitly
limit
connection
reuse
to
allow
for
balancing
opportunities
when
running
into
this.
C
C
Load
generators
aside,
following
the
earlier
observations,
it
might
be
good
to
define
load
tests
a
bit
more
narrowly
than
just
queries
per
second
over
a
number
of
connections.
There
seem
to
be
many
implicit
choices
across
load
generators
and
from
the
outside.
It's
not
always
obvious
what
those
are
talking
about
test
definitions.
C
Google
recently
contributed
an
adaptive
load
controller
feature
to
nighthawk.
That
feature
makes
it
easy
to
perform
sla-based
testing
to
get
answers
to
questions
like
how
many
queries
per
second.
Can
my
system
sustain,
given
these
resource
constraints
under
set
latency,
while
observing
at
most
x
errors?
B
You
thanks
auro
as
a
continuation
of
the
bottleneck
analysis.
We
would
like
to
propose
some
next
steps.
Let's
look
at
the
ingress
connections
on
the
left
hand,
side
of
the
diagram
here
that
can
be
distributed
among
multiple
worker
threats,
the
label,
one
so
worker
threads
that
are
waiting
for
traffic
versus
threads
that
have
a
traffic
to
microservices
that
are
active,
but
disbalanced.
B
So
in
summary,
achieving
a
qps
of
more
than
50k
and
latency
of
less
than
five
milliseconds
in
the
current
setup
is
one
of
our
goals.
This
performance
within
a
cni
plus
proxy
in
a
virtualized
environment
is
a
formidable
goal.
We
are
targeting
if
you
want
to
join
some
of
our
analysis.
Let
us
know
connect
with
us
and
we
should
be
able
to
achieve
some
of
these
goals
and
present
in
a
future
session.
Thank.