►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
gRPC Deep Dive: Prevent Your Service From Overtaking Itself - Lidi Zheng, Google
In any distributed system, it is very common to have mismatched processing power on sending (client) and receiving (server) side. This can result in failures or excessive buffering of messages on either side leading to out-of-memory situation. Fortunately, gRPC has a flow control mechanism that transparently throttles the traffic to protect both services. In this presentation, we will dive into how does networking protocols (like TCP, HTTP, gRPC) control traffic, and how gRPC flow control facilitate your usage of the bandwidth between your services.
https://sched.co/Uafi
A
So
shall
we
get
started
so
hi,
everyone,
I'm
Livi
from
Google-
and
this
is
my
github
account
and
today
I
want
to
share
a
topic
that
I
personally
find
interesting
during
our
daily
routine,
of
helping
people
debugging
their
services
and
I
hope
this
hobby
will
help
you
to
build
a
more
reliable
service
like
afternoon.
It
is
done,
and
today
we
have
three
agendas.
We
are
going
to
answer
the
three
questions:
what
is
flow
control,
wise
flow
control,
important
and
how
GRC
solves
it
before
we
dive
into
the
product
itself?
A
So
what
is
flow
control
that
gives
it
a
definition?
Flow
control
is
the
mechanism
to
throttle
the
traffic
in
order
to
protect
the
endpoints
that
are
under
resource
constraints,
so
for
people
who
are
sitting
here,
if
you
have
unlimited
budget
sorry,
this
talk
is
not
for
you,
and
also
some
of
you
may
ask
like
why
do
I
want
to
slow
down
my
traffic?
A
Do
you
know
how
much
effort
I
put
into
it
to
make
it
fast
so
well
to
answer
that
we
need
to
realize
flow
control
is
all
about
scaling,
because
first,
it's
common
to
see
computational
power
difference
between
server
and
client
in
micro
service
architecture.
It
is
common
to
see
a
database
node
so
be
more
than
a
hundred
different
clients,
and
it's
also
common
to
see
a
web
server
is
serving
tens
of
thousands
of
devices.
At
the
same
time,
also,
the
infrastructure
has
evolved
a
lot
in
the
past.
A
Luckily,
we
have
docker
in
2013,
we
have
kubernetes
in
2014,
and
this
is
the
fifth
cocoon.
So
we
are
putting
more
and
more
fancy
stuff
into
our
architectures,
like
service
mesh,
select,
proxies
and
various
stuff.
So
every
single
request,
nowaday,
involving
a
large
number
of
networking
components
and
neglecting
flow
control,
may
cause
outages
in
one
of
them
and
you
will
be
extremely
hard
to
debug
also
to
make
things
worse.
A
People
love
to
maximize
the
concurrency
of
their
application
by
using
buffering
caching
or
message
queues,
which
I
agree,
a
boost,
the
performance
a
lot,
but
also
put
a
lot
of
pressure
on
the
own
memory
resources.
So
in
a
word,
if
you're
just
writing
unit
tests
or
building
toy
projects,
you
don't
need
to
worry
about
flow
control.
But
if
you
are
serious,
you
should
pay
attention
to
it
to
better
demonstrate
the
effect
I
draw
this
diagram.
Imagine
you
have
a
busy
application.
A
You
can
see
it
as
a
database
node
or
machine
learning
worker
that
is
doing
some
heavy
lifting
work
and
cannot
keep
up
with
the
sender.
So
what
will
happen?
Is
the
message
will
be
puff
up
in
the
busy
application
and
heel
it
out
of
memory?
So
if
the
application
is
that
that
which
means
no
one
is
reading
the
buffer
out
of
kernel
and
then
the
kernel
will
start
start
to
drop
incoming
packets,
then
the
message
will
start
to
pop
up
on
the
standards
I.
Eventually,
the
standard
application
is
also
out.
A
So
the
idea
of
flow
control
actually
is
very
simple.
So
when
the
message
is
buffered
up
in
the
receiver
application,
all
you
need
to
do
is
find
out
a
way
to
signal
the
sender
please
slow
down
and
the
sender
will
respect
this
command
and
righto
is
traffic.
So
that's
taken
another
look
at
this
and
flow
control.
So
imagine
you
have
a
sequence
of
services.
A
A
That's
their
work
nope,
because
the
push
back
message
is
not
sent
from
the
end
receiver
to
the
original
set
original
sender
of
the
request,
and
then
you
will
have
some
fun
time,
debugging
like
which
one
is
the
one
to
blame,
so
the
valid
solution
of
flow
control
has
to
be
from
end
to
end
next,
let's
talk
about
the
challenges
of
implementing
flow
control,
so
first
in
need
to
be
performant
because
flow
control
is
in
beta
pass.
We
do
not
want
a
protection
mechanism
that
will
slow
down
the
traffic
by
30%.
A
It
is
unacceptable
and
we
want
fairness
between
our
pcs,
which
means
we
do
not
want
a
malicious
client,
sending
a
malicious
request
and
blocks
and
entire
service,
and
also
we
want
algorithm
to
throttle
based
on
performance,
so
we
can
utilize
all
the
original,
proper
and
networking
resources.
Last
and
most
importantly,
we
want
a
flow
control
mechanism
that
can
deploy
you
from
end
to
end
next.
If
you
still
remember
your
networking
system
from
your
undergrad,
you
probably
will
remember
remember
this.
There
is
a
similar
concept
to
flow
control,
which
is
called
congestion
control.
A
They
are
similar
by
serving
different
purpose.
The
flow
control
the
goal
of
flow
control
is
protocol.
Node
is
the
node
inside
the
network,
while
congestion
control
is
protecting
the
network
itself.
The
trigger
condition
for
flow
control
is
when
the
receiver
is
slower
than
the
sender
and
the
trigger
condition
for
congestion
control
is
low,
bang
waste,
lawsy
network
and
in
real
world
for
most
of
the
time.
A
The
flow
control
and
congestion
control
are
working
together
to
protect
your
cluster,
but
if
only
congestion
control
is
like
kicking
frequently,
maybe
is
a
good
indicator
that
you
should
contact
your
cloud
provider
in
necks
since
we're
talking
about
congestion
control.
Let's
take
a
closer
look
at
it
like
how
you
works
and
why
it
is
not
the
best
solution.
A
Quicks
I
know
here.
If
you
are
a
Linux
user,
you
can
try
to
type
in
the
command
below
to
check
your
active
TCP
congestion,
control,
algorithm,
ok,
back
to
the
topic,
so
the
common
algorithms
are
Reno.
Big
cubic
I
want
IP
to
dip
into
them,
and
the
general
strategy
is
very
simple:
it
will
increase
the
sending
rate
if
a
key
is
received
and
you
will
decrease
the
sending
speed
if
a
kiss
missed,
which
means
the
packet
has
been
dropped
by
switches
routers
up
here.
A
In
this
way,
we
the
TCT,
can
achieve
flow
control
by
doing
following
steps
when
the
path
is
filled
up
in
your
application
stop
reading
from
the
kernel
buffers,
then
the
receiver
will
drops
any
further
incoming
packets,
and
in
this
way,
does
the
memory
pressure
of
the
receiver
will
not
get
worse.
Then
the
receiver
is
getting
protected,
job
done,
but
it
except
there
are
two
drawbacks.
First,
the
center
is
observing
a
lot
of
packet
packet
loss,
which
means
it
was
think,
maybe
I'm
in
a
really
crappy
Network.
A
A
It
cannot
differentiate
like
which
task
is
which
so,
when
push
back
all
of
the
task
is
going
to
be
pushed
back,
which
means
a
misbehaving
task
is
influenced
other
normal
tasks
you
are
using
and
it
will
be
helpful
also
in
data
in
data
center
case
that
port
in
your
service
mesh
may
be
not
responding
to
your
control
panel
due
to
congestion
control,
so
how
to
jab.
He
sees
of
this
problem.
This
is
the
GFC
solution
which
is
HTTP
to
flow
control.
The
algorithm
itself
is
highly
similar
to
token
packets.
A
You
can
imagine
the
flow
control
window
as
a
bucket
for
tokens,
and
the
sender
will
try
to
grab
the
amount
of
tokens
as
as
many
as
the
size
of
the
data
frame.
If
the
tokens
is
not
sufficient,
the
sender
will
have
to
wait.
On
the
other
hand,
read
the
receiver
can
replenish
tokens
by
sending
window
update
frames,
which
effectively
means
I
am
ready
to
read
more
bytes.
A
As
you
can
see
here,
this
algorithm
is
highly
performant,
because
for
the
most
of
the
time,
the
sender
is
not
blocked
by
flow
control,
and
you
can
stand
at
ease
full
capacity,
also
HDTV,
to
provides
fine
grain
structure
which
enables
per
stream
flow
control
and
per
connection
flow
control,
and
the
most
important
feature
of
this
algorithm
is
the
per
stream
flow
control,
which
means
enables
the
receiver
to
push
back
on
their
misbehaving
RPC
instead
of
all
of
them.
Next,
it
also
provides
frame
priority,
which
means
the
the
push
back
only
work
on
beta
frame.
A
A
A
A
A
During
our
real
world
deployment,
we
found
an
issue
with
window
size,
imagining
the
network
as
a
pipe,
and
it
takes
time
for
data
to
flow
from
one
side
to
the
other.
If
we
set
the
window
size
too
small,
which
means
when
the
data
is
floating
from
one
side
to
the
other
side,
the
sender
is
idling
not
doing
anything
but
waiting
for
the
window
up
a
frame
and
this
phenomena
will
get
worse.
A
When
your
take
when
your
sender
and
receiver
is
not
in
the
same,
not
in
the
same
data
center,
maybe
is
why
is
in
East
Coast
when
in
West
Kauai
in
Europe?
Why,
in
North
America?
What
we
want
here
is
we
want
the
window
size
to
be
as
big
as
possible
to
utilize
all
the
networking
resources,
but
we
don't
want
it
to
be
too
big
that
it
can
overflow
either
side.
A
The
RPC
solution
here
is
the
PDP
estimator.
So
what
is
P
DP
DP
is
bandwidth
delay
product,
which
means
the
amount
of
data
they
can
be
interested
in
the
network,
and
the
computation
is
easy.
You
can
measure
the
bandwidth
times
the
round-trip
time
which
effectively
you
can
get
the
volume
of
the
network,
which
is
the
flying
bytes
on
the
wire
here,
and
this
algorithm
will
measure
the
PTP
and
instead
the
initial
window
size
to
PDP,
and
by
doing
this
you
can
intelligently
avoid
triggering
flow
control.
A
So,
let's
recap:
what
were
the
challenges
we
had
before
so?
First,
we
need
a
flow
control
mechanism
that
is
performant
well
Jam,
pcs
performance
and
we
want
fairness
between
our
pcs,
HDTV
flow
control,
supports
multiplexing
and
throttle
based
on
performance.
We
got
PDP,
estimator
and
last
flow
control
from
end
to
end.
A
Gbc
has
building
flow
control,
which
means
by
adopting
geodesy
you're,
protecting
you're
already
protecting
your
cluster.
So
probably
soft
everyone
is
happy,
and
next,
let's
take
a
look
at
how
to
flow
control,
actually
works
in
inside
of
actual
code.
So
this
is
a
snippet
from
geophysical.
You
can
see
this
is
streaming
hall
and
when
you
are
trying
to
sell
something,
the
same
API
will
block
and
here.
Thus
the
message
is
accepted
by
the
flow
control
window.
A
A
The
first
one
is
it:
there
is
a
flag
call,
is
ready
to
check
whether
the
pier
is
ready
to
receive
your
message
and
we,
you
can
also
set
already
handler,
which
means,
if
there
is
an
opening
whenever
there
is
an
opening
in
flow
control,
it
will
invoke
your
callback
and,
even
though
flow
control
is
great
in
most
of
a
scenario.
Some
of
you
may
still
think
it
doesn't
fit
into
my
scenario.
I
just
want
buffering
whatsoever.
Does
that
mean?
I
cannot
use
G
RPC.
A
A
You
can
just
simply
ignore
the
Israeli
flag,
but
you
may
cause
indefinitely
buffering
here,
and
maybe
it
will
cause
out
of
memory
in
your
application,
so
use
waste
caution
so
take
today
we
have
the
three
question
like
first:
what
is
flow
control
flow
control.
Is
the
mechanism
to
sort
out
the
traffic
to
protect
endpoints
and
second,
why
is
flow
control
important,
because
fast
senders
may
cause
excessive
buffering
on
both
sides
and
how
do
you
actually
sauce
it?
Gpc
provides
easy-to-use
building
and
flow
control.
Okay,
that's
all
the
content
of
this
talk.
A
B
A
C
So
there
we
go
so
the
the
API
in
Java
is
fully
async.
That
means
it's
not
going
to
block,
and
that
means
that
there's
only
a
couple
different
ways
that
you
can
interact
with
it
in
order
to
deal
with
flow
control.
One
way
is
we
could
throw
an
exception
or
something
like
that.
If
you
know
you
made
us
unhappy,
there's
there's
a
couple
others
for
variations,
but
lots
of
times
you
might
need
to
do
you.
You
get
like
a
batch
of
messages
all
at
once,
or
something
like
that.
C
C
Ready
will
still
be
operating
during
that
period
of
time,
so
you
can
send
a
bunch.
You
know
you
already
have
those
in
memory
there
now
just
queued
whenever
they
actually
get
sent
and
whenever
there's
more
memory
available,
then
you'll
still
be
able
to
get
an
on
ready
callback
and
continue
sort
of
to
have
your
nice
proper
flow
control,
and
so
it
allows
you
to
have
a
little
bit
more
bursty
behavior.
This
is
the
primary
reason
that
the
the
infinite
buffering
is
there
just
because
we
didn't
because
of
the
way
the
API
is
fully
async.
C
E
C
D
C
C
It's
quite
surprising
how
many
networks
it
is
a
good
number
for,
because,
as
networks
tend
to
get
a
faster
as
in
less
latency,
they
tend
to
get
bigger
as
far
as
pipes,
and
it
just
so
happens
that,
like
Monta
Tom
on
atera
lee
64
K
is
just
sort
of
how
those
two
things
even
out.
The
main
case
that
that
doesn't
work
as
well
is
whenever
you've
got
big
expensive
pipes
like
between
big
data,
centers
or
especially
across
continent.
C
Those
tend
to
be
very,
very
long
and
because
they're
so
important,
they
tend
to
be
pretty
big
and
fat
as
well,
and
so
you
may
need
to
if
you're
going
over,
some
of
those
pipes
increase
the
default
window
size.
So
the
the
bdp
measurement
figures
this
out
for
you,
so
you
don't
necessarily
need
to
normally,
but
for
like
a
notation
there,
you
would
get
to
figure
it
out
and
you
can
end
up
monitoring
your
network.
C
If
you
know
the
throughput
and
if
you
know
the
latency
just
with
ping,
you
multiply
those
together
and
that's
pretty
appropriate
size.
But
but,
like
64
K
I
said,
is
what,
in
my
default,
so
in
these
networks
might
eat
four
megabytes
or
eight
megabytes.
It
really
scales
quite
high
on
that
high
end,
which
is
why
we
don't
know.
Typically
you
want
to
use
all
that
memory.
We
want
to
use
something
less
than
that.
That's
more
appropriate
to
the
actual
network
you're
using.
C
A
Is
recurring
then
estrada
ring
is
occurring.
We
have
one
use
case
that
well,
the
people
was
confident
they
turn
off
the
flow
control
before
and
after
like
because
cut
because
they're
confident
about
what
they're
doing
and
when
you
put
it
back
home,
they
can
observe
a
more
than
50%
decrease
of
meadow
see
there.
C
Will
know,
though,
that
like
here
it's
it's
not
necessarily
bad
whenever
it's
happening
it's
it's.
Some
of
this
is
normal,
especially
for
burstiness
behavior,
it's
very
normal
for
like
if
you're
gonna
send
4
megabyte
message
or
something
like
that,
you're
going
to
get
sort
of
temporarily
throttled,
but
it's
not
actually
harming
anything,
because
that's
just
the
way
the
network
walk
rate
and
stuff.
So
there
you
have
to
be
a
little
bit
cautious
to
figure
out
what
you're
actually
looking
at,
if
you're
looking
at
sort
of.
C
So
the
the
the
TCP
flow
control
will
still
interact
there,
yeah
because
that's
just
what
HP
one
used,
and
so,
if
you're,
let's
say
doing
streaming
without
like
soda
streaming
or
something
like
that,
TCP
will
interact
fine
and
that
can
get
converted
fine
into
HB
to
flow
control,
and
things
like
that.
It
depends
a
little
bit
if
you're
talking
about
like
just
doing
lots
of
requests
all
at
once
or
more
of
a
streaming
situation.
Oh.
C
Yeah
yeah
also,
whenever
you're
going
through
things
like
that,
every
proxy
sort
of
everything
in
the
in
the
middle
will
have
its
own
flow
control,
and
so
you
might
need
to
configure
some
of
the
proxy
settings
to
be
larger.
So
so
we
said
like,
for
example,
this
64k
I
mentioned
proxies,
might
have
that
as
well,
and
you
might
need
to
increase
it
or
if
you
see
their
buffering
a
lot,
you
might
decrease
it,
and
so.
C
Flow
control
is
sort
of
point-to-point
and
then
you
add
them
all
together,
nice
or
to
get
in
the
end.
So
I'm
basically
saying
that
there's
existing
flow
control
there
and
it'll
just
work.
Fine
like
you,
because
HP
one
has
flow
control
via
TCP,
and
then
it
gets
converted
into
HP
to
flow
control,
and
if
it
gets
converted
into
something
else,
you
can
go
as
many
times
as
you
need
a
sort
of
point-to-point.
So
it
should
all
just
be
fine.
B
C
A
So
the
question
is,
if
actually
to
every
every
obviously
will
appear
open
as
a
stream.
So
if
you
open
up
an
Andre
RPC,
does
that
means
this?
100
stream
is
a
flow
control
separately
or
all
together.
So
in
most
of
the
case,
there
are
flow
control
separately,
which
means
if
you
are
sending
a
large
amount
of
unary
requests.
You
can
still
overload
your
server
in
that
way
and
the
her
connection
flow
control
actually
is
quite
similar
to
what
we
have
for
TCP
congestion
control.
C
Yeah
and
the
short
answer
is
both
I
like
and
it
does
depend
on
the
implementations
you're,
using
especially
many
of
your
mixing
proxies
and
stuff.
There
is
a
connection
level
of
flow
control,
there's
also
stream
level
flow
control,
but
connection
flow
level
flow
controls
a
little
bit.
I'm
gonna
say
not
dangerous
I'm,
not
necessarily
saying
that
it
might
be
a
little
temperamental
because
I
you
can
have
the
possibility
that
you
end
up
starving
streams
because
of
some
streams
that
are
very
active
and
they
just
steal
all
the
flow
control
from
the
others.
C
So
you
there's
a
couple
different
ways:
it
can
be
tuned,
there's
a
couple
different
things:
it's
it's
a
little
bit
more
complicated
and
in
some
ways
it's
more
to
protect
your
more
your
entire
processes
memory
and
not
as
much
this
particular.
This
particular
connection.
You
might
bring
it
down
for
multiple
connections
at
a
time
or
a
couple
different
things.
It's
a
very
much
nuanced
conversation,
I.
Think
a
lot
of
the
implementations,
I'm,
saying,
GRC
and
outside
of
G
RPC
are
a
little
bit
more
simplistic.
C
F
When
HTTP
three
comes
I,
think
quick
is
kind
of
the
basis
and
we
are
thinking
we've
seen
benefits
on
mobile
site.
So
that's
where
we're
trying
to
use
it
and
then,
as
our
experience
grows,
definitely
yeah.
We
have
it
on
mobile
platforms,
quick,
not
exactly
HTTP
three
but
will
become
hope
for
the
HTTP
three.