►
Description
Service Mesh Performance Community Meeting - Sept 2nd, 2021
Agenda:
- Explanation of adaptive load controller as a capability by Otto van der Schaaf
- Distributed performance testing and adaptive load control with Nighthawk by Navendu (Slides - https://docs.google.com/document/d/1gAHLDSMpWQoYEh8VP38xyrD8CVb4-L-6HXxFkSzVJoo/edit#)
- GetNighthawk architecture review by Rudraksh - (Slides - https://docs.google.com/presentation/d/1arewKYZuhkv7NGRZKx0w-7bQdS-ho-YSYJF0X32qfHo/edit#slide=id.g89c4ba9d58_2_6)
Join the community at https://layer5.io/community
Find Layer5 on:
GitHub: https://github.com/layer5io
Twitter: https://twitter.com/layer5
LinkedIn: https://www.linkedin.com/company/layer5
Docker Hub: https://hub.docker.com/u/layer5/
A
All
right,
fair
enough:
hey
everybody!
Welcome
to
the
service
mesh
performance
community
meeting.
This
is
a
well
a
cncf
project.
Now
service
mesh
performance
has
been
a
project
for
about
a
year.
It
has
had
fits
and
starts,
and
various
people
involved
over
time
and
we've
been
fortunate
that
the
the
community
around
it
has
been
congealing
into
and
beginning
to
harden
the
charter.
A
A
A
And
actually
so
showing
up
is
part
of
contributing,
it's
just
being
active.
Being
here.
Sometimes
we've
had
an
individual
in
the
the
measuring
community,
he's
been
there
for
a
year
and
a
half
fairly
consistently
shows
up
hasn't
said.
One
word
on
the
calls.
I've
spoken
with
him,
he's
asked
for
mentorship
and
other
things
privately,
but
but
just
shows
up
to
calls
to
hang
out,
and
so
that's
great,
so
he
that
counts
is
what
I'm
saying
anyway.
A
A
Clearly,
we
don't
take
ourselves
that
seriously,
although
I'll
say
this
before
people
start
to
introduce
that
the
work
here
is
actually
taken
quite
seriously,
there
are
more
people
watching
what
goes
on
here
than
most
of
you
are
probably
aware
of.
There
are
a
number
of
other
well
service
mesh
projects
that
are
nervous
about
the
potential
research
being
done
here.
Some
of
them
are
nervous
in
a
good
way.
B
B
A
Good
so
I'll
I'll
add
to
this
because
it's
unfair
to
have
pablo
do
this,
which
is
which
is
to
say
pablo
well,
might
have
a
higher
iq
than
me,
which
you
know.
That's
that's
saying
a
lot.
You
know.
Actually
what
I
mean
to
say
is
pablo's.
I've
done
some
work
in
this
area
before
I'm
really
pleased
that
he's
here,
because
there's
a
number
of
things
that
we
want
to
do
around
with
respect
to
the
application
of
data
science.
A
A
I
was
just
talking
to
him,
but
there's
a
few
folks
who've
been
interested
in
some
of
the
initiatives
because
it
does
have
to
do
with
some
machine
learning
or
some
some,
some
algorithms
that
we
want
to
get
into,
and
so
I'm
thankful
that
pablo
has
something
of
a
background
or
some
amount
of
experience
toward
these
use
cases
and
can
maybe
show
some
of
the
well
I'll
speak
for
myself.
He
can
show
us
show
me
the
way
in
terms
of
how
we
might
how
we
might
create
some
interesting
things.
Some
interesting
analysis.
B
C
C
C
A
Nice
awesome
yeah,
I'm
stoked
I'm
pumped
yeah
that
jared
there's
a
cup
between
I
I
hadn't
even
considered
it,
but
it
was
just
kept
thinking
about
smi,
conformance
so
much
and
then
but
yeah
very
clearly.
Smp
and
the
two
are
related
smi
and
s
p,
there's
a
diagram
showing
that
overlap.
Actually
we
should.
We
should
bring
that
up
at
some
point
but
yeah
that
yeah
yeah
good
good,
good,
good,
a
nearby
you've.
D
D
I
had
developed
something
on
the
vassam,
but
it
was
very
difficult
to
for
me
to
stitch
with
the
docker
kubernetes
stuff,
because
I
personally
have
no
background
on
the
containers.
I'm
still
learning
on
that
side,
so
it
was
quite
tough
and,
as
I
told
you
that
I
am
very
much
interested
in
a
pure
cc,
plus
plus
kind
of
environment
where
there
are
not
other
things
to
be
taken
care
of.
So
so
I
I'll
be
willing
to
contribute
here.
D
E
A
That's
good,
actually,
one
thing
to
highlight
that
suhani's,
been
involved
in
is,
is
interactive
labs
that
will
highlight
that
allow
people
to
get
their
hands
on
smp
the
oh.
This
is
a
good,
it's
a
good
reminder.
So
on
the
smp
spec
site,
we
don't
currently
have
any
interactive
labs,
although
there
is
an
interactive
lab
already
for
smp.
Specifically,
I
it's
not
up
here
and
it
can
be
easily
made
to
be
up
here.
A
G
G
Can
I
yeah
please
so
hello
everybody?
My
name
is
zenbin
abdulmus.
I
come
from
kerala
india
and
today
marks
my
two
week
anniversary
at
commercially,
so
I
joined
it
two
weeks
back
and
I've
been
trying
to
contribute
towards
the
front
end
as
of
now
but
trying
to
shift
towards
the
back
end,
because
that's
what
I
like
more
yeah,
that's
that's
all
from
myself.
A
A
It's
really
important.
It's
really
important
for
folks
to
be
able
to
connect
it's
like
that's
a
half
of
half
of
being
part
of
the
project.
So
there's
a
couple
of
we're
fortunate
that
I
think
there
are
five
or
six
surface
mesh
performance.
Maintainers
s,
p
maintainers,
I
don't
quite
recall,
but
there's
about
four
of
them
on
the
call
today,
and
so
otto
van
der
schaff
is
on
the
call
currently
at
red
hat,
we'll
hear
from
him
and
just
actually
probably
first
up.
A
A
A
And
the
vendor
poder
cat
is
on
the
call
with
cars
and
street
vista
was
on
the
call
good,
okay,
hey
with
that.
Let's
so
one
thing
that
jared
was
noting.
Is
that
there's
a
lot
of
stuff
going
on
that's
intentionally?
True,
there's
a
lot
of
things
to
to
cover
we're
going
to
talk
about
a
couple
of
projects
today,
both
s
p
and
get
nighthawk
and
maybe
even
spend
a
lot
of
our
time
toward
toward
nighthawk.
A
I
I
So
first
I
want
to
back
off
a
little
bit.
I
I
do
need
to
sort
of
well
give
a
heads
up.
I
haven't
been
able
to
prepare
this
as
much
as
I'd
like
to,
because
there's
like
zero
documentation
for
this
right
now
and
I
know
actually,
I
was
under
the
impression
that
there
would
be
at
least
some
from
the
design
documents
around.
That
would
be
useful
in
like
a
presentation,
but
that's
from
previous
gig
that
I
worked
at
and
that's
not
cleared
for
publishing
yet
so.
I
I
I
I
The
adaptive
load
controller
is
like
an
automates
like
iterating,
searching
that
optimal
request
for
second
demand
according
to
a
traffic
profile.
While
seeking
like
convergence
on
that-
and
you
know
it
gives
you
like
the
current
maximum
amount
of
requests
per
second,
that
your
mesh
or
system
is
able
to
sustain
it,
charges
that
for
you
and
that's
like
what
the
stock
implementation
that
comes
with
nighthawk
out
of
the
box
does,
and
then
it's,
I
say,
a
stock
implementation.
I
I
I
And
yeah
before
I
can
do
that
so
before
I
share
my
screen,
there's
also
one
other
thing
that
I
wanted
to
point
out.
I
mentioned
it
in
the
meeting
notes,
but
this
was
built
in
a
collaboration
with
the
with
with
a
team
within
google,
and
it
is
very
likely
that
one
of
them
will
be
presenting
in
way
more
depth
than
I'm
doing
right
now
on
this
system.
F
Any
questions
while
you're
sharing
this.
I
think
it
would
really
help
to
kind
of
understand,
open
loop
versus
closed
loop,
the
main
differences
and-
and
you
know
how
we
test
this,
because
some
of
my
colleagues
internally
have
been
leveraging
using
nighthawk
and
that
the
details
or
understanding
of
like,
what's
the
main
difference
between
those
is
still.
You
know
this
lack
of
clarity
there
that
really
helped
it
briefly
go
through.
I
Basically,
the
the
pacing
at
which
a
load
generator
releases
request
is
influenced
by
the
speed
of
the
server.
So
I
I
think
that's
that's
the
key
thing
and
in
in
open
book
mode,
there's
no,
no
correlation
between
whatever
the
server
does
and
this
the
frequency
at
which
a
load
generator
releases
requests.
F
I
F
F
I
F
I
I
Results
are
actually
going
to
make,
and
this
is
specific
to
closed
loop
mode
in
open
loop
mode,
you'll
be
seeing
overflows,
it
will
just
skip
so
in
open
loop
mode.
It
probably
makes
much
more
sense
to
just
configure
these
resources
to
not
be
as
constrained,
because
in
either
cases
you
don't
want
to
see
blocking
time
and
you
don't
want
to
see,
pool
overflows
and
that
sort
of
thing
in
your
test.
Executions,
probably.
F
Okay
got
it.
The
reason
I
ask
is
for
something
like
rfc,
two,
five,
four,
three,
two
five,
three
three,
I
think
yeah,
maybe
wrong
with
the
numbers
but
yeah
anyways,
the
layer,
three
performance
testing,
the
rfc
for
performance
at
layer.
3
testing
suggests
that
you
follow
the
binary
search
algorithm
where
it
starts
sending
traffic
100,
and
you
see
the
losses,
and
you
know
at
overflows
like
what
you
suggest
here.
F
You
can
scale
back
50
if
that
succeeds,
go
up
to
75
that
succeeds
and
then
midpoint
like
75
plus
12.5.
So
it's
so
it's
a
kind
of
like
a
binary
search
algorithm
until
it
yeah
you
it.
You
come
down
to
the
the
parameter
that
you
set
for
the
testing,
which
is
zero
loss
or
0.1
percent
loss
or
sort
of
99.9
percent
success.
Success
rate,
something
like
that
right.
F
So
until
you
reach
that
rate,
there's
a
binary
search,
algorithm,
just
curious
how
you
know
what
kind
of
algorithm
you
follow
here
and
seems
like
it's
mentioning.
You
have
a
histogram
of
this
loss
rate
or
the
lack
of
connections
that
you're
looking
at
here
and
and
at
some
point,
which
is
what
the
adaptive
nature
that
you're
sharing
here
is
that
it
will
adjust
accordingly.
I
I
That
is
that
and
yeah.
So
when,
when
I
say
resources,
it's
actually
one
of
either
connections
or
maximum
streams,
that's
what
I
think
that's
from
the
top
of
my
head,
good!
Okay,
thank
you,
but
there's
more
because
basically
this
translates
into
envoys.
What's
it
called
sure
I'm
not
lost
for
the
term,
but
it
it.
It
translates
into
an
enfi
concept
and
I
actually
think
that's
where
the
term
resource
also
comes
from,
but
yeah.
I
agree
that
it's
not
it's!
It's
a
too
generic
of
a
term.
F
I
All
right,
I
made
a
little
note
of
that
so
well,
okay,
now,
as
for
the
code
code
of
the
adaptive
load
controller,
I
think
the
most
or
the
best
entry
point
to
start.
You
know
glossing
over
how
it's
stitched
together,
so
to
speak.
It's
actually
this.
It
resides
in
well
in
in
the
android
proxies
like
nighthawk
repository
and
then
there's
a
directory
called
source,
adaptive
underscore
load
and
that's
like
where
everything
all
everything
implementation,
wise
lives
for
the
adaptive
load
controller.
I
I
I
I
A
About
that,
if
I
could
insert
a
question
about
that
sure,
and
if
I,
if
I'm
jumping
ahead
just
ignore
it-
is
part
of
that
communication
that
you,
the
through
that
jrpc
channel,
is
that,
with
each
potential
generated,
load
and
subsequent
analysis
of
that
load,
with
each
potential
step
through
what
might
be?
What
might
be
something
of
a
recursive
analysis?
A
A
Like,
inter
in
terms
of
injecting
or
in
terms
of
the
analysis
that
can
be
done
in
terms
of
controlling
when
to
step
versus
not
like
score
like
so
there's
boy,
I'm
gonna
confuse
a
lot
of
things,
probably
and
throw
it.
But
if
you're
you
know
more
or
less
performing
analysis
after
you've
run
a
generated
load
and
you're,
you
have
a
scoring
function.
Maybe
for
that
particular
output
of
that
particular
analysis.
A
I
So
you're,
are
you
like
referring
to
a
pause
between
different
steps?
Yeah,
that's
that's
a
good
question
and
I'm
really
sorry,
but
I
I
can't
answer
that
on
the
spot.
I
don't
know
and
actually
I'm
kind
kind
of
pessimistic.
I
don't.
I
think
it
will
just
immediately
go
to
the
next
step.
It's
not
a
thing
right.
I
You
know,
ideally,
that
that
was
the
original
idea
and-
and
it
still
is
with
horizontal
scalability
the
way
nighthawk
implements.
That
is
that
it's
like
transparent
to
either
cli
or
in
this
case
the
adaptive
load
controller.
So
well,
I'm
gonna
make
sure
that
we
actually
don't
require
any
changes
here.
I'm
sorry,
I'm
kind
of
seeing
this
and
it
comes
to
my
mind.
I
just
want
this
to
work
when
horizontal
scalability
lands
so.
I
A
Here's
here's
one
it's
high
level
as
well,
which
is
for,
if
I
can,
if
I
can
use
the
term
so
so
we're
looking
at
a
reference
implementation
of
well
I'll,
try
to
use
these
terms
consistently
and
maybe
we'll
through
our
use
together-
maybe
we'll
solidify
some
of
these.
But
this
so
actually
can
you
say
auto.
This
is
the
this
is
a
reference
implementation
of
an
adaptive
load
controller,
or
this
is
a
reference.
I
I
I
You
know
we
need
to
check
like
this
interface.
We
either
we
need
to
propose
something
to
make
that
a
little
bit
less
specific,
because
now
it's
you
know,
there's
stuff
in
there
that
ties
it
specifically
to
nighthawk,
like
you
know
this,
this
proto
message
and
that's
not
that's
optimal
in
light
of
what
we
want
here.
I
guess
so
that
needs
looking
into
so
you
know
I
haven't.
You
know
looks
in
that
much
detail
in
it
that
I'm
gonna.
I
I
Because
in
it
you
know,
I
think,
when
you
look
at
the
adaptive
controller,
this
adaptive
load
system
when
you
zoom
out
it's
not
what
it
does,
isn't
really
specific
to
what's
nighthawk
it
shouldn't
be
at
all
or
a
load
generator.
It's
just
reconfiguring
something.
You
know
and
analyzing
results,
and
you
know
if
you
can
abstract
away
from
all
the
inputs
and
outputs
and
it's
a
pretty
generic
system,
and
it
could
do
what
you
want.
A
There
is,
if
we
go
back
to
the
community
meeting
minutes
for
a
moment,
just
sort
of
one
noting
the
time
and
then
two
understanding
the
state
of
this
particular
spec
that
it's
probably
not
appropriate
to
go
through
this
spec
at
the
moment.
But
but
what
would
be
really
nice
as
people
familiarize
with
that
reference?
Implementation
of
the
adaptive
load
controller
is
that
we
begin
to
articulate
a
few
questions
that
we
would
like
to
see.
If
we
can
answer
and
to
otto's
point.
A
Some
of
those
questions,
at
least
from
my
perspective
are
well
are
about
helping
people
run
well
in
context
of
a
service
mesh.
Specifically,
it
doesn't
have
to
be
limited
to
a
service
mesh.
I
think,
and
that's
actually
part
of
what
we've
been
discussing
about
the
charter
of
service
mesh
performance
itself,
whether
or
not
to
the
whether
or
not
the
focus
should
be
constrained
to
this
particular
niche
or
not.
That
was
the
subject
of
conversation
last
meeting
yeah.
A
Is
that,
like
some
of
the
questions
that
you
can,
I
think
we
should
articulate
some
of
these
questions
fairly
well
see
if
the
existing
adaptive
load
controller
framework
that
system
can
is
well
positioned
to
be
able
to
answer
these
things.
Some
of
the
like
an
example
question
that
some
of
you
have
probably
heard
me
say
before
is
like
it
follows
on
what
otto
had
just
articulated.
Is
that
a
lot
of
times
people
will
will
grab
a
nighthawk
or
grab
nighthawk?
A
To
answer
a
question
like
to
understand
and
better
right
size,
maybe
their
infrastructure
for
either
financial
savings
or
for
to
guarantee
a
certain
slo
to,
and
there
are
any
number
any
number
of
considerations
to
understand
there.
One
of
those
is
about
latency
and
trying
to
achieve
a
certain
latency
or
guarantee
a
certain
latency,
without
necessarily
overpaying
for
it
or
without
necessarily
not
having
your
infrastructure.
F
Yeah,
but
one
thing
that
would
help,
in
my
opinion,
is
having
a
way
to
understand
when
the
the
scaling
happens
or
scale
scaling
in
a
sense
when
this
adaptiveness
so
to
say,
changes
either
in
form
of
ability
to.
I
think
that,
in
the
code,
you
have
a
way
to
log
log.
These
changes,
however,
in
terms
of
what
kind
of
granularity
would
you
be
able
to
modify
it
or
provide
it
a
parameter
to
change
it
right,
so
something
along
those
lines
would
be
really
help.
A
F
So
from
that
perspective,
if
I
have
a
way
to
control
my
adaptiveness
or
if
I
have
to
wait
to
understand,
that's
the
adaptability
so
to
say
right,
so
that
would
kind
of
provide
me
a
deterministic
mechanism
to
say,
okay
under
these
constraints
or
if
I
have
like
10x
the
tcp
connections.
Just
then
the
first
test,
the
this
the
load
scale,
the
load
changed
in
this
format
versus
5x
the
connection
so
the
first
test,
the
load
changed
and
in
so
and
so
sequence,
whether
it's
time
or
whether
it's
the
number
of
connections.
I
F
Right
so
so
I
guess
the
the
adaptability
in
the
closed
loop
mode
here
is
that,
depending
on
the
back
pressure,
so
if
I
have
a
100
tcp
connections
and
10
000
hdp
queries
per
second,
I
might
have
so
many
losses
based
on
which
the
nighthawk
load
would
change
versus
you
know
say
if
I
have
a
80
tcp
connections,
and
and
on
top
of
that
I
have
10
000
hdb
queries
per
second.
My
load
from
nighthawk
would
change
right,
because
driver.
G
F
Failure
rates,
or
whatever
other
parameter
you
consider
for
to
change
the
load,
would
be
different
and
unless
I
have
a
way
to
kind
of
measure
or
understand
or
or
know,
the,
how
much,
how
nighthawk
is
changing
the
load.
My
two
tests
are
not
similar.
My
failure
rate
that
I
measure
or
the
queries
per
second
success
rate
is,
is
not
necessarily
similar,
because
the
load
being
generated
is
different
in
in
the
two
cases
obvious
choice.
F
I
Nighthawk
is
able
to
quantify
so
it's
able
to
quantify
how
much
time
it's
spent
waiting
on
the
server
in
closed
loop
mode.
So
it
can
quantify
that
for
you,
it
can
also
quantify
error
rates,
but
so,
but
having
said
that,
that
is
something
that
and
you
could
like
it
would
be
probably
be
a
small
tweak
to
the
adaptive
load
control
system
to
write
an
extension
to
consume
that
times
spent
being
blocked
and
consider
that
in
its
input,
while
it
you
know
computes
its
new
configuration.
G
F
So
to
say
that
that's
definitely
a
great
option.
If
you
can,
the
other
side
of
it
is
the
the
from
what
you
mentioned.
You
can
quantify
this
changes
so
that
I
think
it's
a
really
good
first
step
to
have
a
clarity
either
in
the
logs
or
in
the
output
results
to
say:
hey,
here's,
the
back
pressure,
here's
how
much
I
changed
just
enough
yeah
that
that
would
be
a
really
great
first
step.
I
I
think
prometheus
or
something
like
that,
so
you
can
track
it
over
there
as
well.
So
well,
it's
probably
this
visual
inspection,
which
is
not
optimal,
but
you
can
you
know
you
with
that.
You
can
have
a
way
to
at
least
inspect
like
what
happened
during
the
adaptive
load
control
sessions
to
see.
Well,
if
something
funky
was
going
on
with
respect
to.
A
I
A
F
I
F
Great
great,
thank
you
so
another
future
topic
from
my
side
is
about
test
cases
and
what
kind
of
testing
we
could
do
in
in
terms
of
overall
mesh
testing
internally
being
going
through
some
of
the
scalability
tests
or
scale
out
scale
scale
up
kind
of
test.
So
possibly
it's
something
I
can
share
at
some
point.
A
Good
I'll
try
to
tie
up
the
notes
here
it
doesn't
by
the
way.
Just
on
these
calls.
I
know
we're
the
project
in
the
community
around.
It
is
somewhat
young
and
so
we've
yet
to
hit
just
a
consistent
cadence,
but
we're
we're
getting
there
and
this.
Today's
call
is
really
great
there's
a
couple
of
folks
who
weren't
able
to
make
it
that
I
know
are
interested.
A
I
bring
up
the
cadence
and
things
to
say
that,
for
my
part-
or
there
can
be
other
hosts
of
this-
call
like
this
isn't
and
that
doesn't
need
to
be
me
and
yeah
we'll
otto
you'll
it'd
be
great.
If
I
know
eric
is
something
of
a
linchpin
for
part
of
that
next
discussion,
we'll
also
hound
jacob
a
little
bit
to
see.
If
no,
I
know
he's
not
one
for
meetings.