►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
gRPC Performance; Tuning Applications and Libraries - Noah Eisen, Google
gRPC C++ team has been working on performance for over a year now. In this presentation we will share the story of our journey, sharing insights on tuning applications that use gRPC as well as optimizing the library itself. All concrete examples will be from gRPC, but the high level concepts will be interesting to anyone who has worked on performance. We will focus on: - benchmarking + microbenchmarks + synthetic benchmarks + application benchmarks + cutting down noise on benchmarks - tooling + flamegraphs + latency breakdowns - concrete optimizations + tuning the threading model of gRPC apps + high performance network polling systems
To learn more: https://sched.co/GrYz
A
A
lot
of
content
to
get
through
I
want
to
make
sure
this
time,
so
my
name
is
Noah
Eisen
I'm,
an
engineer
on
the
G
RPC
team
I've
been
working
there
for
about
two
and
a
half
years
now,
with
the
focus
on
C++
and
for
the
past
year
and
a
half
I've
been
focusing
on
performance.
So
this
talk
is
on
GC
performance.
We're
going
to
talk
about
tuning
both
libraries
like
the
internals
of
the
GBC
library,
and
also
tuning
applications
that
use
G
RPC.
A
So
before
we
get
further
into
this,
I
want
to
talk
about
intended
audience.
It's
really
meant
for
anyone
who
has
an
interest
in
performance.
The
examples
are
going
to
come
from
GBC
library
and
geography
applications,
but
all
of
the
concepts
are
very
high
level,
and
so
they
should
be
interesting
to
anyone
who
has
worked
with
performance
or
who
cares
about
performance.
A
So
I
want
to
talk
and
also
on
the
slide.
I,
don't
have
it
in
text,
but
I
kinda
want
to
talk
about
like
goals
of
this
presentation,
like
definitely
some
general
not
to
be
specific
ideas
about
performance
best
practices
is
something
I
want
people
to
walk
away
with
and
then
also
near
the
end.
I
kind
of
want
this
to
become
a
way
that
people
are
more
comfortable
starting
conversations
about,
like
maybe
specifically,
geography
performance
like
how
to
open
a
github
issue
and
wedding
kind
of
information
to
put
in
there.
A
That
would
make
it
more
actionable
from
our
end.
Okay,
so
I'll
start
with
the
jerbs
the
overview
in
case
people
aren't
as
familiar
with
G
RPC
I
want
to
do
a
quick
show
of
hands
how
many
people
have
heard
of
G
RPC,
okay,
cool?
That's
what
I
figured,
how
many
people
have
actually
written
Jer
BC
code
and
among
that
who
has
worked
with
the
Java
stack?
A
Okay,
how
about
the
go
stack
and
then
how
about
the
C++
or
rap
language
stack?
Okay,
cool
cool,
okay,
so
I'll
do
an
overview
of
your
PC
and
then
I'm
going
to
talk
about
kind
of
like
the
crux
of
this
presentation,
which
is
tooling
benchmarks
and
data
I'm,
going
to
keep
coming
back
to
those
three
word:
you're
gonna
hear
them
again
and
again
and
again
applied
throughout
the
talk
in
different
ways.
A
So
then
we're
gonna
go
through
some
examples
in
the
GRP
c
library
itself
and
then
near
the
end,
we'll
talk
about
govt
applications
and
again
apply
those
same
concepts
so,
starting
with
the
GOP
overview.
Google.
Has
this
rich
history
of
taking
the
awesome
technology
internally
and
making
it
available
to
the
open
source
so
at
Google
internally,
there's
a
container
management
system
called
Borg.
That
was
open
sources,
kubernetes
I
assume
most
people
are
familiar
with
that
because
we're
at
coop
con.
A
Similarly,
the
internal
build
system
at
Google
is
called
blaze
that
was
open
source
as
basil.
Recently
tensorflow,
the
machine
learning
library
exists
internally
at
Google,
and
it
was
also
made
available
to
the
open
source
community
and
then.
Lastly,
the
internal
RPC
solution
for
Google
was
stubby,
but
Google
is
replacing
it
with
GRDC,
which
is
built
in
the
open
source
and
currently
migrating
internal
use,
cases
to
use
GBC
more
and
more
so
that's
kind
of
like
the
origins
in
history
of
Djerba,
see
in
case
you
aren't
familiar
g-o-p
C
stands
for
G
RPC
remote
procedure.
A
Call
it's
a
recursive
definition:
I
promise,
Oh
G
every
C
is
a
high-performance,
open
source,
standards-based
general-purpose,
polyglot,
feature-rich
RPC
framework.
Each
of
those
bullet
points
could
deserve
an
hour-long
talk
by
itself.
I've
taken
the
high
performance
one
and
put
it
in
bold,
because
that's
what
we're
going
to
talk
about
today
and
it's
actually
developed,
there's
a
big
teen
of
Google
that
is
working
on
it
and
some
open
source
contributors.
We'd
love
them
work,
but
there
are
definitely
some
and
it
is
production
ready
today.
A
So
that's
the
brief
overview
of
G
RPC,
a
generic
stack
for
G
or
PC
looks
kind
of
like
this
there
at
the
top
and
the
green
you
have
a
user
application,
that's
the
code
that
is
actually
calling
into
the
GRDC
api's.
So
we
don't
own
that
the
first
entry
point
into
our
library
is
a
layer
of
generated
code,
and
that
is
API
specific.
So
you
feed
some
generating
process,
a
proto
file
which
has
your
service
definitions
and
it
will
spin
up
that
generated
code
layer.
A
The
generated
code
is
then
going
to
hook
into
the
core
library.
That's
where
main
functionality
and
main
features
are
implemented,
and
then
some
are
lower
down
at
the
bottom
is
a
transport-
and
this
is
a
library
or
set
of
system
calls.
That
knows
how
to
put
bytes
onto
a
wire,
and
this
is
going
to
look
different
from
stack
to
stack,
so
I'm
gonna
briefly
run
through
each
of
our
language
2x,
because
we
have
three
separate
language
stacks.
First
off
go
I
just
wanted
to
call
it
some
differences
in
the
bottom.
A
They
in
some
ways
depend
on
the
XNA
HTTP
to
library.
This
is
mostly
for
h2
framing
and
then
on
the
bottom.
They
will
be
calling
to
TCP
Java
stack.
Also,
the
main
differences
to
call
out
are
in
the
transport.
You
know
the
bottom
Java
depends
on
two
libraries,
Medi
and
ok
HTTP.
Okay,
HTTP
is
used
for
mobile
clients.
Java
also
has
a
completely
G
RPC
own
in
process
transport
to
work.
A
In
a
case
where
you
have
gypsy
in
the
same
process,
the
C++
deck
looks
a
little
bit
different,
so
starting
from
the
bottom
and
going
up
at
the
bottom
is
also
a
TCP,
but
the
C
library
has
implemented
its
own
HTTP
implementation.
That's
called
C
HTTP
that
allows
us
to
write
an
h2
implementation
that
uses
your
PC
data
structures
and
we
can
more
heavily
optimize
it
because
it's
more
of
our
own
code.
We
can
move
faster
above
that
there's
the
GOP
C
core.
A
Again,
that's
the
bulk
of
the
functionality
on
top
of
the
core
is
a
thin
sea
surface
and
the
sea
surface
is
what
supports
are
wrapped
languages.
So,
above
that
you
can
see
Python
Ruby,
there's
also
a
node
Objective
C
PHP.
Anything
that
wasn't
mentioned
before
is
supported
by
our
sea
surface
and
then
that
kind
of
allows
the
rap
language
is
to
share
most
of
the
functionality
and
we
implement
it
in
core
and
they
just
get
that
feature
for
free
and
then
above.
That
is,
of
course,
layer
generated
code.
Ok,
that's
it
for
the
overview.
A
I
want
to
talk
about
the
key
points,
as
I
mentioned,
tooling
benchmarks
and
data.
These
the
points
we're
gonna
keep
coming
back
to
and
if
there's
three
words
that
you
remember
walking
out
of
this,
it
should
be
tooling
benchmarks
and
data
and
you'll
hear
them
a
lot
so
starting
with
tooling.
In
order
to
optimize
code,
you
need
to
know
where
to
look.
I
have
confused
Jackie
Chan
asking:
where
am
I
microseconds
going
you
don't
know
you
cannot
possibly
start
to
perform
optimizations
and
so,
in
my
mind,
tooling
kind
of
narrows
the
problem.
A
Scope
so
tooling,
narrows
I,
know
that's
kind
of
cryptic
we'll
get
back
to
what
that.
What
I
exactly
mean
by
that
and
the
last
thing
I
want
to
call
out
before
I
dive
deeper
into
tooling.
Is
that
it's
important
that
there
is
no
there's,
no
perfect
tool?
There's
no
one
tool.
That's
going
to
show
all
the
hotspots
optimize
all
the
places
where
your
code
is
slow,
there's
no
one
tool
to
rule
them
all
or
something
which
is
why
it's
important
to
use
many
tools
and
we'll
talk
about
that.
A
So
diving
into
a
few
important
kinds
of
tooling
that
have
been
very
important,
important
to
GRDC
I
want
to
start
with
latency
traces,
so
just
know
they're
like
show
of
hands
kind
of
thing.
Who
here
has
used
a
tracing
system.
This
could
be
like
distributed
tracing
or
just
in
process.
Okay,
cool,
so
experience
yeah.
So
then
this
might
look
very
familiar.
Tracing
involves
adding
annotations
directly
to
the
code.
A
Gr
PCC
has
our
own
timer
system
that
does
tracing
so,
as
you
can
see,
that's
actually
what
it
looks
like
GPR,
timer
scope
and
you
give
it
a
string
value
and
what
that'll
do
it'll
log
a
timestamp
and
that
string
and
it'll
allow
for
some
post-processing.
So
at
the
simplest
form,
this
could
actually
just
be
writing
to
a
log
file
and
then
have
a
post
processor
that
reads
in
the
timestamps.
It
could
get
a
little
more
fancy.
A
These
annotations
could
instead
write
some
in-memory
data
structure
if
you
want
less
overhead,
but
anything
latency
tracing
maintains
ordering
and
gives
you
kind
of
a
this
happens,
and
this
happens-
and
this
happens
view
of
your
code
and
that's
a
great
way
to
show
hotspots
on
the
left.
I
have
actually
a
screenshot
of
some
internal
visualization
tool
and,
as
you
can
see,
with
an
arrow,
this
is
just
capturing
kind
of
all.
A
The
things
that
happen
before
we
actually
go
to
the
send
message:
syscall
and
I've
linked
to
our
gob
c
implementation
of
these
timers
in
our
github
repo.
If
anyone's
curious,
but
as
I
mentioned,
there's
no
perfect
tool
and
tracing
has
its
strengths
and
it
has
its
weaknesses
and
it's
important
to
understand
those
to
use
it
more
most
effectively.
So
for
tracing.
One
of
the
biggest
weaknesses
is
that
the
tracing
system
itself
can
add
a
lot
of
overhead.
A
A
Also
tracing
works
very
well
in
low
thread
environments.
Multiple
threads
are
happening
at
once.
Tracing
can
get
a
little
Messier.
This
comes
from
a
very
single
threaded
benchmark.
We
have
so
tracing
in
itself
is
not
enough.
Another
tool
that
geo
PC
has
focused
on
CPU
profiling,
again
kind
of
show
of
hands
like
who
here
is
familiar
with
profiling
or
is
like
looked
at
charts
like
this
okay
I
figured.
This
would
be
a
little
more
popular.
So
this
is
the
flame
graph
again
taking
of
a
real
GBC
benchmark.
A
A
Profiling,
like
any
tool,
will
have
its
strengths
and
have
its
weaknesses
so,
like
a
strength
of
profiling
would
be
you
don't
actually
need
to
go
and
annotate.
Your
code,
profiling
kind
of
works
for
free
out
of
the
box.
The
kernel
is,
actually
you
know,
asking
each
core
400
times
a
second
or
some
frequency.
What
function
is
running
on
you
right
now,
it'll
value
at
that
data
and
generate
this
chart.
A
So
if
a
function
is
hot,
it
will
show
up
in
an
in
a
profile
which
is
great
because
it
can
kind
of
be
the
fastest
way
forward
and
fastest
tool
to
get
started
on,
and
then
it
also
tends
to
be
pretty
low
overhead,
depending
on
like
with
what
frequency
it's
profiling.
But
again,
every
tool
has
weaknesses.
A
So
lastly,
I
also
wanted
to
call
out
some
other
tools
that
are
worth
mentioning.
Lock.
Contention
measuring
tools
just
mentioned
hard
to
do
lock,
contention
with
profiling,
so
Val
gram,
you
trace
things
like
that.
Jus
bc
has
also
had
very
good
luck
with
custom
counter
code.
So
we've
implemented
some
infrastructure
in
our
library
that
will
actually,
in
a
certain
kind
of
build
hook
into
the
allocation
system,
the
atomic
system
and
run
counters
and
then
log
that
information.
A
But
the
bottom
line
about
tooling
is
that
it's
you
need
to
obtain
an
arsenal
of
tools
and
then
that's
not
enough,
because
that
Arsenal
needs
to
continuously
grow
and
people
should
be
experimenting
with
new
tools
trying
to
figure
out
what's
cool.
How
can
we
look
at
our
stack
in
another
way
and
then
use
tools
together
where
one
is
weak
use
another
one
or
when
a
strong
focus
on
its
strengths,
and
that
really
gives
like
a
performance
team,
a
great
sense
of
where
the
time
is
being
spent
where
the
microseconds
are
going?
A
So
that's
tooling,
let's
move
on
to
benchmarks.
So
in
order
to
optimize,
you
need
to
know
how
to
measure
so
now
I
have
confused
Fry
asking.
Is
that
really
an
optimization
I'm,
not
sure
like
I
think
it
was
faster,
but
I
mean
if
you
can't
prove
it,
it's
a
little
bit
harder
to
say
so
benchmarks,
then
we're
tooling
narrow
scope,
I
kind
of
see
it
as
benchmarks,
widen
scope
of
impact
and
again
I'll
get
back
to
that
and
make
that
a
little
more
clear.
A
Some
other
diagrams,
like
tooling
benchmarks,
come
with
many
different
kinds,
so
micro
benchmarks
raise
your
hand
if
you
have
written
a
micro
benchmark
in
any
language
with
okay,
fewer,
so
microphones
folks
were
awesome,
they're
great
for
narrow
scope
optimizations.
So
this
is
a
real
snippet
of
from
GBC
code
base.
A
It's
benchmarking:
air
creation,
geo
PCC
core
has
a
struct
that
represents
air
and
status
and
in
this
tight
loop,
it's
going
to
create
it
and
then
delete
it,
and
that's
gonna
run
something
like
five
five
point:
five
million
times,
I
think
yeah
5.5
million
times
until
the
benchmarking
system
gets
a
good
sense
of
how
long
that
operation
took
and
it
will
output
data.
So
it's
telling
us
that
creating
and
deleting
an
error
is
about
120
mana
seconds
and
micro
benchmarks
are
awesome
in
a
very
narrow,
specific
use
case.
A
They
can
show
changes
to
small
sub
components
of
your
code,
but
this
is
again
a
point
that
I'm
going
to
circle
back
on
and
back
on.
All
of
these
kinds
of
tools
have
their
strengths
and
have
their
weaknesses,
and
so
micro
benchmarks
have
the
weakness
of
being
narrow
in
scope.
When
in
a
program,
would
it
ever
be
realistic
to
create
5.5
million
errors
and
then
delete
them
immediately?
Never
so
it's
gonna
show
pretty
unrealistic.
A
Cache,
behavior,
behavior
you're,
not
gonna,
see
multi-threaded
scenarios
working
well,
so
micro
benchmarks
are
good
for
showing
directional
changes
of
sub
components.
As
far
as
looking
at
absolute
numbers
from
micro
benchmarks-
probably
not
as
useful
but
to
show
you
know
to
show
an
optimization
for
something,
some
type
module
they're
great.
A
So
another
kind
of
mesh
mark
that
we've
focused
on
is
synthetic
benchmarks
and
what
I
mean
by
synthetic
benchmarks
is
anything
that
is
written
above
the
API.
So
this
is
a
program
and
for
gr
pcs
use
case.
This
is
a
program
that
uses
the
JIRA
PC
library
does
some
amount
of
work
and
then
outputs
some
data
about
that
work.
A
So
I've
thrown
up
an
example
of
GR
pcs
synthetic
benchmarking
system,
which
is
highly
configurable.
It
involves
a
driver
sending
out
configuration
files
to
a
client
and
server,
and
then
the
client
and
server
will
talk
to
each
other
with
some
configuration
of
messages
and
then
output,
data
about
the
latency
and
the
RPC
counts,
and
then
for
our
example.
Our
configuration
file
looks
like
that
on
the
right.
It
is
very
configurable.
You
can
configure
the
number
of
servers.
A
The
number
of
server
threads
number
of
clients
request
response
size,
and
that
is
very
useful
for
us
to
kind
of
experiment
with
a
wide
array
of
RPC
topographies,
and
maybe
we
have
a
problem
when
there's
very
large
messages
with
only
one
client
in
one
server.
Maybe
it's
when
there's
five
clients
and
five
servers
and
uneven
messages.
Small
requests,
large
response,
it's
completely
configurable-
and
this
has
been
awesome,
but
I
want
to
make
a
point
about
this-
that
this
is
just
an
example
of
our
synthetic
benchmarking
system.
A
It's
not
the
using
synthetic
benchmarking
system,
so
another
example
of
a
synthetic
benchmark
might
be
if
you
were
to
write
a
server
and
a
client
and
the
client
does
some
fixed
amount
of
work
like
a
thousand.
Are
pcs-
and
you
just
use
time
like
the
system
called
time
on
it,
to
see
how
long
that
takes?
That's
also
an
example
of
a
synthetic
benchmark.
It's
anything
written
above
the
API.
That's
doing
work
for
the
sole
purpose
of
benchmarking,
for
the
sole
purpose
of
timing.
A
It
or
getting
counts
and
I
want
to
talk
about
one
more
benchmark
type
and
that's
application
benchmarks,
and
this
is
kind
of
a
subtle
differentiation.
So
I
want
to
talk
about
this
because
benchmarking
applicable
application
benchmarks
are
also
synthetic
benchmarks.
But
the
difference
in
my
mind
is
that
these
are
written
with
another
team's
API.
So
this
is
gonna
exercise
your
stack
in
a
new
way.
It's
actually
gonna
exercise
your
stack
in
the
exact
way
that
this
other
team
exercises
your
stack.
A
So
to
make
this
more
clear,
the
example
is
tensor
flows,
RPC
bench
also
yeah,
so
I
know.
This
only
applies
to
libraries
because,
like
if
you're,
not
a
library,
no
and
probably
be
using
your
API,
so
this
asterisk
there,
but
the
example
that
I
chose
is
tension,
flows,
RPC
bench.
This
is
a
benchmark
written
via
tensor
flow
api's,
but
it's
written
in
a
way
that
it's,
it's
very
RPC
intensive,
so
I
kind
of
like
drew
up
a
little
bit
of
the
configuration
it
has
30
workers,
one
of
them
starts.
A
It
sends
an
RPC
to
all
the
other
ones.
They
all
talk
to
the
other
ones
in
one
full
mesh
step
and
then
all
the
results
come
back
to
the
first
worker
and
all
of
the
actual
touchflo
work.
Doing
that
it's
doing
is
pretty
lightweight
this.
This
stresses
the
RPC
system
heavily,
and
so
they
created
this
benchmark
and
we
we
found
it-
and
this
was
so
useful
to
us
because
it
exercised
our
stack
in
a
new
way
with
more
of
a
realistic
RPC
typography
for
tensorflow.
So
that's
that's
application
benchmarks.
A
Lastly
data.
So
we
talked
about
tooling,
we've
talked
about
benchmarks,
I
want
to
talk
about
data,
and
this
kind
of
comes
a
little
bit
of
a
backstory.
So
there
was
a
time
in
the
JBC
performance
effort
where
we
had
a
lot
of
engineer's
working
on
it,
bringing
numbers
to
the
table
with
various
benchmarks,
various
environments-
and
it
was
a
lot
of
motion.
A
The
the
clip
art
I
have
here
I
heard,
was
kind
of
opaque,
that's
the
Tower
of
Babel,
but
kind
of
so
you
could
find
that,
if,
like
all,
your
engineers
are
speaking
different
languages
about
like
different
numbers
and
different
metrics,
it's
impossible
to
make
progress.
So
that's
the
data
point
18
bits
so
another
one
of
my
crappy
diagrams.
A
This
is
back
to
tooling
narrows
the
scope
of
the
problem
until
it
becomes
manageable
to
solve,
and
then
benchmarking
widens
the
scope
to
demonstrate
the
impacts,
and
all
this
together
gets
packaged
up
as
data
as
a
way
to
like
discuss
and
talk
about
and
show.
Optimizations
and
I'll
come
back
to
this
in
a
case
study
which
will
hopefully
make
it
more
clear.
A
Ok,
so
that
is
we're
kind
of
the
high
level
general
library
application
agnostic
points.
Don't
worry,
you'll
hear
them
again,
they're
coming
back.
They
will
be
in
this
section
in
the
next
section
as
well.
But
those
were
this
kind
of
the
crux
of
this
talk
so
now
we're
gonna
dive
into
the
GBC
library.
Talk
about
some
specific
examples
from
there.
A
And
before
I
even
get
into
the
text
of
this
slide,
I
kind
of
want
to
make
a
point
of
these.
These
three
principles,
tooling
benchmarks
and
data-
are
awesome
to
have,
and
it
would
be
so
great
if,
when
you
started
the
project,
your
tooling
and
benchmarks
were
in
a
perfect
place
and
they
were
they
were
amazing.
But
realistically
that's
not
how
it
happens.
Things
need
to
get
built
in
to
get
built
fast
and
realistically,
your
benchmarking
system
will
come
up
a
little
bit
later
than
the
actual
core
functionality.
A
Gbc
found
this
to
be
the
case,
and
there
are
some
things
that
you
know
we're
known
to
be
slow.
We
went
and
we
fixed
them.
We
got
all
the
low-hanging
fruit
off
the
table
and
then
it
came
time
to
say:
okay.
Well,
where
is
the
next
big
spot
for
optimization?
And
there
wasn't
really
a
clear
answer,
but
at
the
same
time,
there's
three
kind
of
important
facts:
I
want
to
call
out,
that's
features
can
cause
small
regressions
and
these
small
regressions
might
be
below
your
margin
of
detection
and
I.
Have
that
italicized?
A
Because
I
want
to
talk
about
what
that
means
below
margin
of
detection.
If
you
have
no
benchmarking
system
that
can
detect
regressions,
then
by
definition
every
regression
is
below
the
margin
of
detection
or
if
you're
benchmarking
system
is
in
place,
but
it
doesn't
have
coverage.
Maybe
you
have
rich
benchmarks
on
insecure
scenarios,
but
then
a
regression
slips
into
a
secure
pathway
like
that's
missed
or
if
the
benchmark
is
noisy
and
it
can't
catch
regressions,
then
that's
another
way
that
these
Russians
can
like
slip
into
the
codebase
regardless.
A
The
result
is
consistent,
slow
degradation
and
performance,
and
so
at
this
point
kind
of
injury
history,
we
asked
ourselves
the
question
of
how
do
we
reverse
this
process?
The
answer,
of
course,
was
like
better
benchmarks,
better
tooling,
more
data,
so
we
focused
on
a
new
benchmark.
We
called
it
the
minimal
RPC.
This
was
a
specific
scenario
of
our
synthetic
system
with
that
highly
configurable
system.
So
we
configured
it
to
ping
pong
back
and
forth,
one
byte
payloads,
very
small.
A
We
did
no
security,
we
did
no
stamps
or
tracing,
and
then
we
started
focusing
on
the
median
latency
between
two
computers
running
in
the
same
rack,
and
we
knew
this
number
was
about
70,
microseconds
and
everyone.
The
team
knew
this
number
and
we
could
start
iterating
on
this
number
and
demonstrating
change
using
this
scenario
and
looking
at
data
coming
from
this
scenario.
A
A
We
ran
that
two
minute
benchmark
for
30
times
then
took
the
median
so
throw
some
resources
at
it,
get
the
noise
down,
and
then
you
have
a
minor
granularity
with
which
to
measure
changers,
so
I'm
gonna
run
through
a
quick
case
study.
This
was
from
a
latency
trace.
This
is
the
visualization
tool
I
showed
earlier.
We
would
kind
of
look
at
chunks
of
time
together
and
ask
ourselves
is
anything
surprising
or
taking
more
time
than
we
think
it
should.
So,
in
this
case
we
looked
at
this
Apple
filter.
A
It
was
just
doing
an
access
control
check
on
an
outbound
RPC
and
it
was
taking
about
a
microsecond
you
can
say
from
here,
so
that
seemed
a
little
bit
sketchy.
The
next
step,
of
course,
is
to
go
to
another
tool,
so
this
is
a
flame
graph
kind
of
NER
it
in
I'm,
gonna
zoom
it
in
one
further.
So
take
a
second
to
look
at
that
top
layer
and
like
see
if
you
can
see
any
things
that
look
kind
of
out
of
place.
There's
there's
like
three
really
good
answers
here.
Actually,
there's.
A
If
there's
a
lot
of
things
going
on
so
I'm
gonna
pause
her
like
two
more
seconds
so
I'm,
not
gonna
pause
have
to
keep
going
okay.
Firstly,
there's
do
it
as
though
there's
a
flag
green
going
on,
and
it's
not
just
any
flag
read
it's.
It's
actually
a
very
long
flag,
great
it's
taking
a
long
time,
and
this
is
all
happening
in
the
the
per
RPC
hot
path.
So
this
is
kind
of
sketchy
because
flag
reads:
probably
aren't
going
to
change
over
the
life
of
the
by
an
area.
A
This
is
something
that
could
be
cached
somewhere.
Other
good
answers
would
be
all
the
string,
manipulation
stuff
going
on
over
here,
again,
probably
not
needed
for
something
to
like
happen.
Every
time
the
RPC
goes
through
and
there's
a
mutex
and
that
one
I
mean
maybe
could
be
replaced
for
the
tommix,
but
that's
a
little
bit,
but
mainly
was
the
flag
read.
This
was
a
quick,
quick
fix.
We
saw
this
data,
we
went
to
the
code.
A
We
saw
the
problem
cache
the
flag,
reduce
the
overhead,
so
this
was
a
good,
a
good
story
for
that,
and
so
now
I
want
to
take
that
crappy
diagram
again
with
some
examples
and
I
think
it'll
be
more
clear,
but
we
started
with
a
very
wide
problem.
Optimize
gr
PC
makes
your
PC
faster.
That
is
so
wide
in
scope
that
it's
almost
impossible
to
give
to
an
engineer
and
they'll.
Do
it
so,
then
we
use
the
tooling
and
we
got
that
a
little
bit
more
narrow
in
scope.
A
Now
it's
at
this
number
that's
great
and
then
the
next
question
is
okay
to
this
impact.
Our
synthetic
benchmarks
should
this
impact
an
actual
scenario
using
gypsy
API
and
then
to
get
even
wider
to
this
impact
applications
using
geodesy
and
all
that
comes
together
package
with
the
data
and
the
code
change
and
that
that
becomes
a
very
clear,
consistent
story
for
progressing
and
making
optimizations
to
the
code
base.
A
So
the
Java
team
has
made
a
lot
of
contributions
to
Nettie
and
okay
HTTP
because
they
said
atop
that
library
overhead
in
that
library
will
be
a
it's
attributed
to
the
RPC
layer
and
if
there's
things
that
can
be
faster
between
them,
that's
great.
Similarly
go
has
made
some
contributions
to
X,
Net,
HTTP
library
and
in
Google.
The
C
team
has
worked
extensively
with
Google's
tcp
team
to
tune
some
of
the
tcp
options
to
make
it
more
more
optimized
for
the
RPC
use
case.
So
you
can
always
go
deeper.
You
can
always
go
lower.
A
A
So
tuning
gypsy
applications,
tuning
applications
that
use
g
RPC
and
this
part
was
a
little
impossible
to
keep
language
agnostic,
so
I
have
kind
of
a
slide.
I'm
gonna
run
through
that
has
some
language
by
language
call
outs.
But
to
start
with,
like
the
obvious
thing,
which
is
always
good
to
start
with
the
obvious
all
language
facts
are
going
to
benefit
from
reducing
allocations,
reducing
your
copies,
using
syscalls,
reducing
contention
how
to
figure
out
how
to
reduce
those
answer
is
probably
tooling
in
benchmarks.
A
Bruising
allocations
is
going
to
be
especially
important
for
the
go
stack
and
the
Java
stack,
because,
by
nature
of
being
garbage
collected,
there
gonna
be
hurt
way
more
by
excessive
allocations.
So,
specifically
I'm
just
gonna
run
through
each
language
stack,
but
the
Java
stack
using
the
async
API
tuning
thread,
pools.
A
Not
the
out-of-the-box
solution
is
uses
niño,
and
these
are
all
things
like
quick
things
to
try
things
that
I,
you
know,
aren't
going
to
be
always
the
right
solution,
or
else
they
would
be
the
default.
But
if
you
have
good
benchmarks
in
place,
trying
them
and
then
experimenting
is
a
good
idea
for
the
C++
stack.
This
above
all
else
use
the
async
API,
which
is
way
more
highly
tuned
than
our
sink.
Api
I.
Don't
have
it
in
the
text
because
it's
pretty
experimental,
but
there
is
a
new
callback
based
API
coming
to
C++.
A
It
sits
in
the
code
base.
Now
it
needs
more
tuning,
but
that's
like
what
kind
of
like
what's
next
look
out
for.
We've
got
a
lot
of
feedback
that
our
async
API
is
not
that
easy
to
use,
and
so
this
is
kind
of
an
effort
to
fix
that
and
make
it
easier
to
write,
high-performance
li+,
plus
G
or
C
applications
for
suppose,
plus
tuning
the
threading
model
and
tuning
the
number
of
completion.
A
Going
to
run
through
a
case
study
from
the
application
side
now
so
I
distributed
tensor
flow
needs
a
network
layer.
This
involves
many
tensor
flow
processes
running
on
different
machines
in
the
open
source
they
use
G
RPC.
At
some
point,
it
became
a
goal
and
a
project
to
improve
attention
flow
over
G
or
PC
performance.
This
was
kind
of
a
cross
team
effort
between
our
team
and
their
team,
so
the
first
step
was
to
learn
some
of
Thames
flows
tooling.
A
This
is
some
of
the
times
flow
logging
system,
which
is
in
their
open
source
repo,
and
this
is
a
visualization
of
that
tooling.
This
is
showing
actual
communication
between
two
GPUs
that
are
going
to
the
network.
The
purple
rectangles
show
our
pcs
that
are
happening
and
we
can
see
their
lengths
and
we
can
see
that
they're
kind
of
stacking
up
on
each
other
and
being
very
uneven,
and
this
is
like
not
really
doesn't
good.
You
kind
of
want
to
see
consistent
performance
across
program
execution,
so
with
new
tooling
under
our
belt.
A
The
next
step
was
to
go
through
some
benchmarks.
I
already
mentioned
RPC
bench
from
our
perspective
from
govt
seed
libraries
perspective,
RPC
bench
was
an
application
benchmark
because
it
was
using
the
tential
API
is
not
gypsies.
Aps
from
tetra
flows,
team
RPC
bench
was
a
synthetic
benchmark,
because
this
is
a
program
that
they've
written
with
their
own
api
is
for
the
purpose
of
benchmarking
so
like
what
one
team's
application
benchmark
is
another
team's
synthetic
benchmark
is
so
we
use
that.
A
But
then
we
also
went
even
higher
and
we
we
use
kind
of
what
from
tensor
flows
team
would
be
an
application
layer
benchmark.
We
use
benchmarks
that
people
had
written
using
the
tension
flow
api's
themselves,
so
this
was
real
attention,
flow
training
tasks
and
I've
kind
of
drawn
out
what
the
typography
looks
like
they
are.
A
The
parameter
servers
are
going
to
be
way
more
bottlenecks,
and
that
was
the
case
in
certain
cases
that
that
the
parameter
servers
were
kind
of
the
slow
point
for
this
execution,
and
so
that
needed
some
better
threading
to
support
kind
of
like
overloaded
parameter
servers
and
also
putting
serialization
in
a
thread
pool
instead
of
the
the
thread
that
was
doing.
Do
you
ever
see,
network
work
was
very
I've
linked
to
those
changes
if
you're
curious.
A
A
This
could
also
come
in
the
form
of
very
informative
bug
reports,
but
with
those
focus
on
tooling
and
benchmarks,
you
know
a
bug
report
that
mentions
GRC
performance
not
being
up
to
par,
but
it's
not
actionable
is
not
super
helpful,
but
one
that
comes
with
here's,
some
tooling
here's
some
data,
here's
some
diagrams.
We
found
here's
an
exact
benchmark
that
you
can
run
on
your
machine.
Then
it
become
something
that
like
we
can
engage
with
and
and
it's
actionable
and
so
yeah
we
love
contributions
from
open
source.
A
A
Definitely
so
the
question
was:
have
we
ever
run
into
a
case
where
observing
the
program
caused
a
degradation
in
performance?
Absolutely
that
was.
That
was
something
that
bit
us
with
like
some
tracing
tools
which,
like
as
I
mentioned,
can
have
higher
overhead.
So
some
of
these
yeah
some
of
these
toolings,
if
they're,
enabled
the
actual
performance
numbers
get
way
out
of
whack
because
there
may
be
logging
or
they're
writing
some
in
memory
data
structure,
doing
something
slow,
yeah.
Definitely,
and
that's
something
like
if
it
try
to
understand.
B
B
A
Yeah,
that's
a
good!
That's
a
good
question!
That's
gonna!
I,
so
caveat
I'm,
not
a
go
expert,
more
of
a
C++
sack
person,
but
I
think
that
comes
in
using
goroutines
to
paralyze,
more
work,
so
maybe
using
a
separate,
go
routine
to
do
the
serialization,
whereas
separate
and
then
a
different
one
to
do.
The
interaction
with
the
GBC
API
can
be
something
you
look
into,
but
yes,
that
is
I'm
not
totally
sure
having
not
had
too
much
familiarity
with
go.
B
A
So
spot-check
we're
here
but
I
think
the
answer
is
not
yet
right.
Now
the
traffic
does
go
through
a
translation
step,
I
think,
eventually
the
goal
of
like
I
mean
this
was
one
of
the
goals
of
buildings
you
ever
see
in
the
open-source
and
using
it
internally
was
for
that
communication
to
happen
with
less
translation.
I,
don't
know
right
now,
if
there's
any
like,
like
direct
communication,
gypsy
client
outside
of
Google
going
to
inside,
but
that
is
definitely
like
a
goal
of
the
use
case
like
that
yeah
Oh
many
translation
steps.
A
Yeah,
so
this
I've
got
read
some
repeats
and
questions,
but
this
is
asking
about
struck
packing.
Yes,
this
is
looking
into
kind
of
like
the
internal
struck.
The
core
uses
like
to
represent
a
transport
which
are
these
huge,
huge
trucks
and
both
doing
better
packing
and
also
making
sure
that
kind
of,
like
the
hot,
the
hot
items
in
that
struct,
occur
at
the
beginning
of
the
cache
line
where
it's
going
to
be
easier
and
that
also
things
that
will
be
touched
together
up
here
in
the
same
cache
line.
A
A
So
for
the
synthetic,
oh
right
so
sub-channels,
the
question
was
very
I
should
keep
remembering
to
repeat
the
question.
The
question
was
about
the
benchmarks:
are
they
doing
multiple
sub
channels
or
one
sub
channel
or
top-level
channel
for
everything?
I've
talked
about,
there's
nothing
that
involved
actual
load
balancing
so
like
we
never
involved
the
load,
balancing
pathways
of
round-robin
II
over
sub
channels.
They
were
all
doing.
One
client
channel
equals
one
connection
underlying.
There
are
some
cases
where
we're
using
multiple
client
channels
which
each
have
their
own
connection
over
them.