►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Using gRPC for Long-lived and Streaming RPCs - Eric Anderson, Google
Support for long-lived RPCs and streaming RPCs is a core benefit of using gRPC. While such RPCs have fundamental advantages, they also have inherent complications versus "normal" single-request, single-response RPCs. Learn when it is advantageous to use these more advanced features, potential gotchas, and ways to address them.
To learn more: https://sched.co/GrWo
A
I'm
Erik
Anderson
I'm
on
the
GRC
Java
team,
I'm,
the
tech
lead
for
that
I
work
at
Google
and
I'll
talk
some
about
using
gypsy
for
long
lived
and
streaming
RBC's,
so
a
little
bit
of
sort
of
who
I
hope
is
coming
so
that
you
get
the
most
out
of
it.
The
talk
I
hope
you
have
used
G
RPC
before
you,
don't
necessarily
have
to
have
used
Allah,
but
maybe
made
a
service.
You
made
a
couple
clients
you
sort
of
played
with
it
and
then
you're
sort
of
interested
in
more
advanced
use
cases.
A
So
here
for
long
lived,
our
pcs.
That
can
probably
mean
little
different
things
to
different
people.
Some
people
think
one
second
is
a
really
really
long:
RPC
I'm,
not
too
particular
of
myself,
but
generally.
Basically,
if
it's
lasting
minutes
or
hours
or
days,
that
would
be
long,
something
that's
order
of
magnitudes.
More
than
you
normally
talk
about
and
where
it
really
starts
crossing
the
lines
starts
mattering
for
some
of
the
issues
we'll
get
into
like
load,
balancing
and
things
like
that
because
of
how
it
sort
of
impacts.
A
Others,
parts
of
the
system
sort
of
some
of
the
use
cases
that
the
main
one
is
something
sort
of
resembling
a
hanging
get
if,
if
you've
used
up
other
systems,
but
basically
you
can
start
an
RPC
and
have
it
just
sit
there
waiting
for
some
event
to
happen.
If
you
weren't
you,
and
so
it
may
last,
quite
a
while
before
that
event
actually
happens.
The
benefit
of
this
is
like
the
alternative.
Is
you
do
an
RPC?
Hey?
A
It
hasn't
happened
yet
and
it's
like
no
and
then
you
wait
and
you
sleep
awhile,
and
then
you
try
again
so
going
ahead
and
doing
long
live
RBC
reduces
the
total
number
of
RPC,
as
you
do
well.
Also,
as
soon
as
the
event
happens,
you
already
have
RPC
outstanding.
You
can
go
ahead
and
be
notified
that
it
happened
so
watches
and
notifications
is
sort
of
what
I'm
talking
about
here.
Whenever
you
start
mixing
it
in
with
streaming
and
stuff,
there's
more
more
things
you
can
do
with
it.
A
That's
at
least
enough
to
give
you
an
idea
of
why
you
might,
but
we've
not
done
very
much
here.
It's
very
simple.
We've
just
extent
expense
extended
the
length
of
time
that
an
RPC
lasts,
and
we
do
notice
a
couple
of
issues
that
you
should
at
least
be
aware
of.
The
first
one
is
load
balancing
load,
balancing
decisions
are
made
on
a
per
RPC
basis.
A
So
that
means
that
if
you
have
an
RPC
at
the
moment,
that's
created
that
you,
you
get
some
load
balancing
decisions
made,
but
those
are
going
to
last
for
the
rest
of
the
lifetime
of
that
RPC.
And
so,
if
you
bring
up
new
backends-
and
you
just
have
a
lot
of
long-lived
rpcs
you're
like
why
aren't
hitting
my
backends
getting
any
new
traffic,
and
that
is
because
all
of
your
back
ins
are
all
the
clients
are
connected
to
old
backends
and
they
just
haven't
issued
a
new
RPC.
A
Yet
the
other
thing
some
people
will
use
Mac's
connection
age,
it's
pretty
useful
for
elf,
for
proxies
and
Mac's
connection.
Age
will
start
shutting
down
a
connection
whenever
it
reaches
a
certain
age,
but
it
by
itself
does
not
kill
the
connection
or
killing
me
at
the
our
pcs
on
that.
So
it's
a
waits
for
those
our
pcs
to
finish.
Well,
it's
the
time
they
take
a
couple
seconds
max
and
so
everything
will
sort
of
clean
up.
A
But
whenever
you
start
introducing
long-lived
our
pcs,
these
connections
might
last
for
a
while
so
sort
of
the
pathological
case.
There
is
each
of
these
old.
Our
pcs
has
their
own
TCP
connection,
which,
if
you
don't
have
too
many,
that's
not
a
big
deal,
but
some
people,
if
you're
doing
like
lots
and
lots
of
these,
you
might
start
noticing
that
the
other
thing
is
network
failures
happen
like
it's
network.
It
happens
just
because
your
PC
sports
long-lived,
our
pcs,
does
not
mean
you're
guaranteed
to
get
two
months
of
time.
A
For
this
one
RPC,
this
is
your
cherished
IVC.
You
never
wanted
to
end.
So,
just
whenever
you're
thinking
about
long-lived
RPC
as
well.
Yes,
they
can't
have.
You
can
get
quite
long
time
for
some
of
them
just
be
aware
that
you
can't
you
shouldn't
assume
that
you'll
always
get
that
amount
of
time
things
go
things
go
wrong.
Maybe
servers
are
coming
up
and
down.
So
you
should
still
expect
at
some
points.
A
You'll
get
less
than
the
whole
time
that
you
would
hope
for
and
whenever
the
failures
happen,
TCP
will
will
get
TCP
connects
and
those
will
kill.
The
calls
is
we
don't
migrate
from
one
TCP
connection
to
another
or
anything
like
that.
The
other
sort
of
more
interesting
thing
is
network
failures.
Whenever
they
happen,
you
don't
know
instantly.
In
fact,
if
there's
no
TCP
writes,
you
may
not
even
know
because
there's
exponential
back-off
sand
play
the
network
itself
is
unreliable.
So
it's
it's
not
just
some
golden
Oracle.
A
That
tells
you
so
that
that
you
need
to
be
careful
of,
and
because
these
are
long-lived
RBC's
deadlines
by
a
sort
of
very
nature.
If
you're,
using
that
a
lot
for
unary
calls
stops
being
quite
as
useful
for
long-lived
RBC's,
you
can't
still
specify
it,
but
it'll
be
much
larger
than
you
would
normally
have,
and
it
serves
less
of
a
purpose
than
you
may
be
sort
of
familiar
with.
A
So
there's
a
couple
different
things,
my
pull
asterisks
on
the
ones
that
basically,
are
the
issues
that
we
get
to
live
with
the
other
things
we
can
maybe
do
something
to
combat
it.
But,
like
network
failures,
happen,
yes,
I'm,
sorry
I
do
not
have
any
solution
stuff,
but
some
of
the
other
things
we
can
maybe
improve
so
for
the
load.
Balancing
easiest
thing
I
would
suggest
is:
have
the
server
occasionally
close
the
RPC.
A
If
you
need
your,
if
you
really
wish
that
over
30
minute
period,
all
your
clients
were
rebalanced
well,
go
ahead
and
close
the
our
pcs
after
30
minutes
and
then
that
will
over
the
period,
get
them
all
reconnected
and
let
the
load
balancer
make
new
decisions.
I'm
also
I
mentioned
max
connection
age
before
well.
A
Sort
of
sibling
configuration
is
max
connection,
age
grace,
which
is
a
grace
period.
For
how
long
you?
Let
these
are
pcs
last,
and
so
that's
just
a
duration
like
if
you
say
five
minutes,
then
after
the
max
connection
age
is
triggered.
You
get
our
pcs
gets
another
five
minutes
to
complete
and
then
they'll
go
ahead
and
get
cancelled.
So
this,
even
if
you
don't
use
it
all
the
time,
this
setting
might
still
be
usable
as
a
backup
like
you
still
are
going
to
have
other
mechanisms
to
close
our
pcs.
A
A
But
you
may
find
wait
for
ready
to
be
useful,
so
wait
for
ready,
I'll
say
normal
are
PCs
whenever
you
issue
them
and
let's
say,
there's
no
connection
because
someone
tripped
over
the
wire,
obviously
you're,
not
it's
not
gonna
work,
you're,
not
gonna,
get
connected
to
the
other
side,
and
so
those
are
pcs
fail
immediately
gr
casinos
that
are
gonna
fail.
You
issue
the
RBC
NGO
bruselas
he's
like
yes,
it
failed,
however,
with
wait
for
ready.
A
Whenever
gr
PC
knows,
the
RPC
would
fail
immediately
because
there's
no
connectivity
like
we
we've
been
unable
to
make
a
connection.
The
port
is
closed
or
those
sorts
of
things
will
just
not
send
the
RPC
and
we'll
go
ahead
and
wait.
We'll
continue,
retrying
to
connect
and
things
like
that,
and
then
once
connectivity
is
re-established,
the
the
RBC
will
go
out,
and
so
that
can
sort
of
reduce
the
amount
of
exponential
back-off
you
do
manually
or
the
moment.
A
The
connection
is
alive
again,
the
RBC
will
go
out,
and
so
you
get
actually
a
little
bit
reduced
latency
and
though
those
it's
not
that
big
of
a
deal
but
I
thought
it
would
just
at
least
give
a
little
plug
for
it.
So
that's
that's
basically,
long-lived
our
pcs
in
many
ways,
they're
sort
of
simple
in
some
ways
they
still
impact
things
but
sort
of
moving
on
to
streaming.
Our
pcs
streams
by
their
sort
of
nature
are
just
zero,
too
many
messages.
A
Normally,
whenever
we
talk
about
normal,
our
pcs,
we
call
those
unary.
Just
they
have
one
in
streaming
case,
it'll
be
zero
too
many.
Each
of
those
messages
is
ordered.
So
if
you
receive
a
later
or
message
you
already
received,
the
previous
ones
and
streaming
is
independent
in
each
direction
so
that
the
ordering
is
independent,
whether
or
not
you
have
straight
stream
itself
is
independent,
so
sort
of
how
that
looks
is
so.
We've
got
client-to-server.
The
the
client
would
send
some
method,
plus
some
headers
and
some
things
like
that.
A
It'll
send
a
message
and
then
half
post,
and
so
this
is
just
normal
unary.
This
is
sort
of
the
normal
stuff.
People
used
to
half-closed
doesn't
even
normally
matter
because,
what's
the
point,
there's
only
one
message
and
then
the
server
server
responds
back
with
some
some
headers
of
its
own.
It
sends
back
a
message
and
their
sins
back
to
status
code,
which
was
whether
it
was
okay
or
not.
Hopefully
we
you
know
generally,
we
hope
it's
okay,
so
this
is
just
a
normal
unary.
A
We're
going
to
go
really
advanced
and
streaming
is
just
adding
more
of
this
messages
not
too
much
there.
The
big
thing
to
notice
is
the
the
headers
and
the
trailers.
That's
where
metadata
goes.
So
metadata
is
only
at
the
beginning
at
the
end
of
the
RPC
you're
not
going
to
get
those
in
between
the
messages
but
other
than
that
is
it's
probably
what
you
would
expect.
A
Also
half-closed
becomes
more
important
here,
because
that's
an
actual
signal
to
the
server
that
is
not
going
to
send
any
more
messages,
and
you
can
use
that
and
so
how
this
ends
up.
Looking
in
the
normal
sort
of
Preda,
buff
IDL,
so
I
said
that
streaming
is
independent.
The
normal
unary
RPC
is
just
the
normal
RPC.
You
get
a
single
request.
You
send
a
single
response,
you
know,
then
you
can
end
up
doing
clients
streaming
client
streaming.
The
client
sends
multiple
messages
and
then,
after
that,
the
server
does
a
response.
A
Bi-Directional
is
a
little
interesting
because
it
has
some
emergent
properties
as
everything,
but
by
I.
M
is
what
we
call
half
duplex
and
that
there's
only
one
sender
at
a
time.
It's
and
even
by
de
I
can
be
this
way,
but
basically
you've
got
the
client
send
and
then
the
server
responds
and
it's
sort
of
always
that
that
sort
of
mix
with
by
de
you've
got
two
streams
going
in
the
same
going
in
opposite
directions.
A
They
can
happen
at
the
same
time,
and
so
that
opens
up
full
duplex
and
that's
basically
gets
you
something
Kenda
TCP,
but
instead
of
it
doing
raw
bytes
back
and
forth.
You're
gonna
go
ahead
and
do
messages
which
can
be
quite
convenient.
The
if
you're,
really
familiar
with
TCP
I'll.
Note
that
my
the
TCP
semantics
are
a
little
bit
off
here,
because
close
is
different
between
the
two
but
a
lot
of
people
don't
know
the
full
TCP
closed
semantics.
So
most
people
are
probably
not
that
bothered
by
it,
because.
A
A
A
And
so
sort
of
going
into
the
half
duplex,
and
here
I'm
talking
about
client
streaming
server
streaming
and
a
little
bit
of
the
bye
die.
I
will
note,
though,
that
really,
whenever
we're
talking
about
by
day
a
lot
of
people
get
excited
about
it
because
it
provides
full
duplex,
you
can
sort
of
in
some
ways,
just
assume
that
whenever
someone's
talking
about
bi-directional
streaming,
it's
full
duplex
but
I
mean
technically,
it's
not
understand
required.
A
So
this
is
a
lot
of
client
streaming
and
service
streaming
stuff,
so
is
streaming
itself,
has
lots
of
use
cases
you
can
use
it
just
for
latency
or
memory
reduction.
So
an
example
that
sort
of
provides
both
is
sort
of
like
speech
to
text
API.
So
you
need
you've,
got
a
one-minute
clip
so
you're
recording
someone's
issuing
stuff
right
now
and
at
the
end,
you
like
a
transcript
of
what
and
you
can
either
wait
for
the
full
minutes
to
be
recorded
and
then
send
it,
and
then
the
server
will
process
it.
A
And
then
the
response
comes.
But
you
could
even
chunk
it
up
into
smaller
pieces.
Let's
say
five
seconds
or
10
seconds
and
you
send
those
as
as
it
happens
in
real
time.
The
server
could
be
processing
those
as
if
they
happen
in
approximately
real
time
and
then
at
the
last
10
second
chunk.
The
server
receives
that
processed.
A
That
and
Sall
ready
done
most
of
the
processing,
and
so
then
it
goes
ahead
and
sends
back
the
response
with
the
nice
pretty
text
that
knew
exactly
what
you're
saying
and
so
that
that
actually
reduced
latency,
because
the
the
server
didn't
need
to
wait
before
receiving
everything
before
it
was
able
to
do
some
processing
and
it
also
reduced
over
the
overall
amount
of
memory
that
both
the
client
and
the
server
needed,
because
it
neither
side
actually
needed
the
full
one
minute
of
recording
in
memory
at
one
particular
time.
You
can
also
do
some.
A
You
get
some
little
benefits
because
there's
a
separation
now
between
the
response
and
the
end
of
the
call
normally
with
a
unary
response,
the
server
responds,
and
now
it's
given
up
like
it
not
given
up,
but
it
has
no
more
opportunity
to
say
anything
more
about
that
RPC
with
streaming.
Now
you
can
respond
and
then,
if
you
want,
you
can
go
ahead
and
respond
again
and
respond
again.
I'm
gonna
bring
it
up
again
again,
but
watches
are
a
good,
easy
use
case.
Where
you
can
see
how
this
could
be
useful.
A
You
can
say
I'm
interested
in
some
topic
and
it
can
tell
you.
Oh
hey,
here's
what's
happened
on
that
topic
and
then,
if
something
else
happens,
it
can
go
ahead
and
send
that
to
you
immediately,
you
don't
need
a
wait
for
any
RP
RBC
to
request
more
information
and
that
can
reduce
latency
and
just
reduce
the
amount
of
churn
of
the
our
pcs.
A
Another
nice
benefit
of
streaming.
Is
you
get
flow,
control
or
pushback?
This
is
where
the
if
a
cinder
is
sending
faster
than
the
receiver
can
consume,
it
it'll
go
ahead
and
slow
down
the
sender
to
whatever
the
rates
the
receiver
can
process
it,
and
this
can
be
really
useful
for
like
bulk
uploads.
So
normally,
whenever
you
upload
you
like
me
to
do
chunks,
and
so
you
upload
a
chunk,
you
wait
for
the
response.
Okay,
that
one's
good.
A
You
then
go
ahead
and
send
another
chunk,
and
you
wait
for
the
response
that
one's
good
each
time
is
those
that
you
wait.
That's
latency
that
was
lost
bandwidth
that
you
could
have
had.
Instead,
you
can
actually
send
one
after
the
other
in
the
stream,
if
you're,
sending
too
fast
you'll
be
slowed
down
to
whatever
the
network
can
handle,
and
so
you
don't
end
up
having
to
incur
those
delays
where
you
end
up
waiting
at
the
end,
and
so
that
can
save.
A
That
means
you
don't
need
to
tune
the
chunk
size
as
much.
You
don't
want
one
bytes,
but
you
also
don't
want
you
know
one
gig,
but
anything
sort
of
in
a
medium
range
will
do
pretty
reasonably
well,
whereas
before
with
the
sort
of
chunked,
and
then
you
wait,
you're
sort
of
encouraged
to
have
really
large
chunks
sizes
so
that
those
wait
times
if
you're
on
a
high
latency
link
or
things
like
that,
they
don't
occurs
much.
They
don't
they're,
not
as
big
of
a
penalty,
and
basically
some
of
this
is
just
pipelining.
A
So
streaming
because
you've
got
all
these
messages,
they're
ordered
they've
got
implicit
state.
You
can
associate
state
between
these
different
messages,
so
some
simple
things
so
that
that
is
all
these
are
piece
messages
are
going
to
the
same
back
end.
A
stream
is
tied
to
the
back
engine,
so
you
know
the
same
back
end
is
receiving
everything.
It
knows
about
the
things
that
came
before
and
you
can
you
make
that
you
can
use
that
to
your
advantage.
A
Also,
the
the
call
lifetime
itself
has
been
expanded.
Before
is
only
the
request
response
now,
you
can
do
a
lot
more
things
within
the
call.
So,
for
example,
if
you
want
to
have
a
transaction
where
you
end
up
sending
multiple
requests
which
or
each
query
and
or
update
or
things
like
that,
the
the
you
can
either
use
the
call
lifetime
itself
for
managing
the
transaction
or
think
so
if
the
call
gets
killed,
the
transaction
gets
killed.
But
you
now
have
this.
A
Just
because
that's
how
the
system
has
to
work,
and
then
one
sort
of
interesting
thing
is
is
because
all
the
messages
are
ordered,
you
can
actually
like
provide
a
full
state
at
the
very
beginning
and
then
any
like
just
deltas
of
changes
that
are
happening
after
that
point
and
that
could
be
really
useful,
watches
again
it
pretty
easy,
but
the
sort
of
deltas
and
things
like
that.
You
end
up
having
to
do
quite
a
lot
of
infrastructure.
A
If
you
want
to
do
that
in
normal,
our
pcs
in
Ian's
area,
our
pcs
you're
gonna,
have
to
have
some
tokens
or
recovery
tokens.
So
what
was
the
previous
state
you
saw?
But
in
this
case
you
can
just
the
server
can
just
say:
here's
the
current
state
and
then
as
changes
happen,
you
send
it
out,
and
so
there's
much
less
synchronization
between
the
client
and
the
server
and
protocol
is
sort
of
simpler,
but
there's
complexities,
there's
issues
with
with
streaming
and
a
lot
of
this.
A
It's
it
it's
not
for
free,
there's
complexities
involved,
so
it
makes
sure
you're
getting
something
out
of
the
streaming
or
the
long-lived.
Our
pcs,
don't
do
it
just
because
Wells
gypsy
provides
streaming.
Obviously,
I
want
to
use
it
because
it's
the
greatest
feature
in
the
world
unary
our
PCs
I'm,
not
trying
to
burn
them
at
all
they're,
actually
really
great
most.
A
One
thing
to
be
aware
of
is
that
gypsy
has
flow
control,
mention
that
for
the
streams,
but
that
has
very
large
buffers
relative
to
what
some
people
may
consider
large
and
certain
circumstances.
So
we
see
some
streaming,
our
pcs
that
have
messages
that
are
5,
bytes,
large,
well,
64
K
seems
really
really
large
whenever
you're
talking
about
5
byte
messages
and
the
the
flow
control
is
point-to-point.
A
So
if
you
add
new
nodes
into
the
middle
with
new,
so
you've
got
a
new
proxy
that
that's
gonna,
be
an
l7
proxy
in
the
middle,
it
can
add
its
own
buffers,
and
so
you
can.
You
can
very
easily
be
looking
at
megabytes
of
buffers
that
all
vary
depending
on
your
network
and
some
of
actual
concrete
implementations,
but
just
be
aware
that
it
could
be
large
and
I
already
mentioned
some
of
the
API
complexity.
It's
both
the
complexity
of
the
system
itself.
It
can
do
more.
That
means
you
need
to
be
aware
of
more
Korth.
A
More
cases
before
is
real,
simple,
but
also
the
API
is
themselves
in
this.
In
the
languages
we
we
have
optimized,
or
we
do
think
that
most
our
pcs
are
unary.
Those
are
the
ones
that
we
try
to
make
as
easy
as
possible.
Streaming
is
a
more
advanced
use
case,
and
so
it's
intended
for
you
to
be
able
to
get
some
of
that
power,
but
it
also
means
that
you
incur
some.
Some
complexity
and
that'll
vary
a
little
bit
per
language
that
you're
you're
sort
of
interacting
with,
but
be
aware.
A
Whenever
you're
doing
streaming
very
often
your
normal
retries
like
if
you've
got
sort
of
a
framework
level
retry,
it
stops
making
as
much
sense.
So
you
may
end
up,
especially
with
server
streaming.
You
may
end
up
having
trouble
and
needing
to
implement
reach
right
at
inside
the
application
itself,
as
opposed
to
any
sort
of
framework
level,
something
that's
automatically
doing
it,
and
that's
because
you're
gonna
need
to
have
some
specialized
logic
in
order
to
retrieve
just
the
new
things
that
you
wanted
and
not
have
to
reissue.
A
The
full
RVC
so
just
be
aware
that,
because
it
can
be
a
little
saddening
whenever
you
realize
that
and
tracing
stat
tracing
and
stats
systems
are
not
necessarily
all
universally
expecting
streaming
and
so
it'll,
it
can
mean
maybe
get
muddled
a
little
bit
or
might
be
missing.
That'll
vary
per
system
and
I,
don't
know
all
of
the
systems
out
there
on
how
well
they
do
here,
but
be
aware
that
it
may
be
a
little
harder
to
investigate
what's
happening
in
the
system,
so
the
things
that
we
can
actually
control.
A
A
It's
not
queuing
up
at
all
within
G,
RPC,
you're,
doing
you're
sort
of
preventing
the
the
remote
side
from
sending
it
in
the
first
place,
and
the
benefit
of
that
is
that's
also
in
the
end.
You
do
there's
some
some
complexities
there
in
that,
at
times
you
have
to
decide
how
many
do
you
want
to
be
outstanding
at
a
time
you
get
to
tune
that,
but
you
know
that
it's
it's
the
this
to
get
that
sort
of
features.
A
It
may
not
work
out
as
well,
but
in
plenty
of
cases
it
will
be
a
pretty
close
approximation,
and
so
you
may
not
be
that
bad
off,
but
now
we're
getting
into
full
duplex
streaming.
This
is
the
really
advanced
stuff.
This
was
like
it's
suppose,
to
be
an
intermediate
talk,
a
lot
of
the
streaming.
You
can
do
intermediate
pretty
well
at
the
end
of
doing
a
full
bi-directional
protocol.
A
I
would
consider
you
advanced,
that
is
that
the
advanced
stuff
we're
doing
with
GRDC,
and
that's
because
you
basically
have
tcp
with
messages
you
can
sort
of
do
whatever
you
want.
So,
whenever
you
can
do
whatever
you
want,
you
can
do
whatever
you
want
and
that's
you
then
the
time
potentially
be
your
own
worst
enemy,
but
you
can
make
any
custom
protocols.
You
have
a
full
escape
hatch
to
do
whatever
you
need
and
as
I
mentioned
before,
you've
got
application
level
flow
control.
A
Transactions
can
work
pretty
nicely
here,
because
you
can
actually
be
issuing
requests
or
queries
and
then
receive
the
response
for
the
queries
and
that
all
happens
in
conjunction
on
these
streams
and
then,
if,
let's
say
the
client
dies,
it
crashes,
the
RPC
itself
will
get
canceled
on
the
server,
and
so
the
server
knows
to
cancel
the
RV's
cancel
the
transaction.
You
can
sort
of
do
a
couple
sort
of
fancy
things
around
those.
You
can
also
do
live
reconfiguration,
since
things
are
all
you've
got
the
the
organized
state,
one
messages
Falls,
the
other.
A
You
can
say
oh
I'm,
interested
in
five
different
topics
or
my
watch
and
then
later
you're
like
oh
actually,
a
sixth
one
would
be
great
and
a
little
bit
later,
like
yeah,
get
rid
of
that
third
one
and
you
can
change
it
on
the
fly
and
you're
not
needing
to
tear
down
the
old
one,
bring
up
a
new
one
and
figure
out.
What
do
you
do
in
the
middle
point,
because
there's
a
race
or
anything
like
that,
and
you
can
also
do
bulk
uploads
so
before
I
mentioned
both
downloads
for
just
a
half-duplex
case.
A
The
reason
uploads
are
different
is
because
you
really
need
resumption,
typically
with
with
large
bulk
uploads,
and
that
means
as
you're
uploading.
You
need
resumption
keys
to
come
back
to
you,
and
so
in
that
case
the
client
would
remember
the
last
resumption
key
that
it
received.
If
something
goes
wrong,
it
will
go
ahead
and
create
a
new
RPC
and
new
and
just
resume
based
on
that
key.
A
This
still
has
some
benefits,
though,
because
you're
not
having
to
use
that
key
continually.
So
normally,
if
you're
doing
bulk
uploads
you're
using
the
keys
constantly
it's
a
very
normal
common
case,
in
this
case
with
bulk
uploads
streaming,
you
sort
of
delegated
it.
It
happens,
but
it's
not
happening
constantly.
It's
doesn't
need
to
be
quite
as
optimized
as
normal
and
then
sort
of
off
to
the
side.
A
So
full
duplex
we
get
a
couple
more
issues.
Steps
tracing
systems
may
be
overly
simplistic.
That
looks
a
little
bit
similar.
It
becomes
a
bigger
problem
with
folding
Plex.
We
have
API
protocol
complexity.
You
are
dealing
with
two
streams
that
infinite
events
are
happening
on
that's
complex
like
whenever
it
like.
We
talk
about
threading
and
things
like
that.
Your
code
itself
will
become
more
complex
because
you're
doing
something
more
complex
and
I
would
not
underestimate
how
much
impacts
that
can
have
at
this
point,
you're
almost
guaranteed
to
have
application
level
retry.
A
A
There's
flow
control,
there's
pushback,
there's
all
these
things.
That
means
you
actually
open
yourself
up
to
deadlocking.
Yes,
your
RBC's
can
deadlock.
Beware,
that's
not
something
you
you
might
think
about.
So
just
beware
be
a
conscious
because
you
you,
you
can
go
and
do
all
these
other
our
pcs
for
so
long
and
not
have
to
think
about
that,
and
if
you
do
full
by
die,
then
our
full
duplex,
then
basically
breast
conversion
is,
is
thrown
out.
There's
really
no
hope
for
that,
given
the
the
rest
ecosystem.
A
So
a
lot
of
these
those
we
couldn't
do
very
much
about,
but
as
far
as
the
the
deadlocks
the
best
way
to
handle.
That
is
make
sure
that
any
point
in
time
at
least
one
side
is
receiving.
If
you
do
that,
then
everything
sort
of
can
flow
through
the
system
and
it'll
be
okay.
But
if
you
ever
get
to
a
point
where
both
sides
are
sending
in
the
nobody's
receiving,
that
is
a
potential
for
deadlock
also
know
just
because
you
tested
it
once
and
it
didn't
that
lock
doesn't
mean
anything.
A
I
mean
there
were
buffers
in
the
system.
Those
can
change
based
on
live
network
behavior,
and
so
maybe
you
had
one
megabyte
buffer
whenever
you
tested
and
it's
been
like
that
for
a
while
and
then
the
buffer
sort
of
diminished
and
it
became
64k,
and
that
would
be
enough
for
a
lot
of
people
to
maybe
know
the
deadlock
in
that
case,
but
it
didn't
happen
before,
and
so
now,
let's
go
ahead
and
plug
the
two
together
long
live
plus
streaming,
so
we
have
load
balancing
before
before
it
was
mainly
CPU.
A
A
So
it
was
sorry,
sorry
it
wasn't
CPU
before
it
is
memory
before
it's
just
sort
of
sitting
there
consuming
some
resources,
but
probably
not
too
active
on
the
server.
Now
we're
talking
about
streaming.
There's
lots
of
stuff
happening
all
the
time,
but
we're
talking
about
the
transaction
case.
For
instance,
all
these
messages
are
implying
quite
a
lot
of
load,
and
so
you
have
both
memory
and
CPU
now
so
at
the
moment
you're
doing
long-lived
streaming.
A
Our
pcs
definitely
definitely
watch
out
for
load
balancing
before
with
just
the
long-lived
our
PCs
load
balancing
comes
into
play,
but
it's
not
as
dire
and
then
again
chasing
and
stat
systems,
maybe
overly
simplistic
before
I
said
that
if
you're
doing
streaming,
it's
mostly
fine
because
you
can
just
sort
of
assume
all
the
messages
are
one
big
message
here:
we're
spreading
out
the
messages
over
time.
It's
really
really
spread
out
and
so
that
that
sort
of
extra
delays
become
very
noticeable
and
make
it
much
harder
to
debug.
A
If
you
don't
see
those
in
your
step
systems,
but
really,
as
far
as
solving
the
load
balancing
issue,
it's
gonna
be
a
lot
like
what
I
had
said
before
with
the
long-lived
RPC.
It's
nothing
really
changes.
Occasionally
you
should
go
ahead
and
clean
up
our
pcs.
Let
them
go
ahead
and
finish:
let
them
go
ahead
and
make
another
load
balancer,
but
that's
gonna
be
basically
yeah,
and
so
then
that's
what
we've
got
I'm
very
eager
for
QA.
A
If
I've
got
my
email
and
jitter
up
there,
sorry
github
I'm
on
Gator
as
well.
If
you
send
me
an
e
mails,
please
send
a
link
to
GRP
cio
as
well.
That's
the
mailing
list,
because
there's
lots
of
people
who
can
answer
stuff
and
then
shameless
plug
that
erp
at
google
is
hiring.
You
can
come
talk
to
me
and
I'll
get
you
some
contacting
for
me.
B
A
I
should
have
mentioned
that,
although
honestly
I
can
apply
it
in
many
different
cases,
not
just
I,
guess
that
mainly
impacts
long-lived
are
pcs,
and
so
there
is
a
configuration
setting
max
concurrent
streams
that
comes
from
the
HTTP
2
spec
GRC.
Has
it
as
infinite
right
now
for
all
all
of
our
main
implementations,
but
lots
of
proxies
will
go
ahead
and
choose
something
less
than
that,
and
so
like.
If
you're
using
some
services
with
some
Google
API
services,
you
might
see.
100
100
is
pretty
common,
as
its
specifies
the
the
it
would
be.
A
How
does
the
spec
put
it?
Basically,
it
should
not
be
something
less
than
100
or
something
sort
of
wishy-washy
like
that,
and
so
a
lot
of
a
lot
of
proxies
want
something
really
small,
and
so
they
chose
100,
yes,
anytime,
you're,
doing
lots
of
concurrent
RBC's
and
especially
long-lived
if
you're
doing
something
on
the
order
of
hundreds
or
thousands
of
these,
you
could
run
into
that
problem
and
then
the
the
requirements
that
you
open
up,
multiple
channels
or
get
your
PC
to
open
up
multiple
connections.
A
A
So
the
idea
was
that
if
you
already
have
a
message
might
as
well,
tlg
or
if
you
see
about
it,
so
we're
not
going
to
get
too
upset,
but
if
you're
not
watching
the
flow
control
and
if
you're
not
looking
at
say,
if
you
don't
ask
your
PC
hey,
are
you
ready
for
more
then
you'll
just
end
up
buffering
a
lot
locally,
but
it'll
it'll
bubble
up
differently
in
each
language,
so
that
does
you.
You
do
need
to
look
into
that,
but
it'll
vary
yeah.
B
C
A
I
think
in
general,
I
I
wouldn't
do
it
as
much
on
the
wire
level.
I
would
look
more
at
the
events
that
are
coming
higher
in
the
which
messages
are
being
received.
A
lot
of
stuff
can
happen
on
the
wire
and
you
don't
necessarily
need
to
be
an
HTTP
2
expert
to
follow
along
if
things.
If
you
get
that
deadlock
issue,
then
that
is
something
you
could
look
at
and
like.
Oh,
the
window
size
is
different
in
this
case
versus
ask
case.
A
There
are
some
debug
flags
you
can
put
in
to
see
all
the
HTTP
2
frames.
Unfortunately,
we've
got
all
very
per
language,
so
at
times
you
can
do
generally
I
try
to
avoid
needing
to
suggest
someone
to
do
that,
because
that's
quite
involved
but
I
think
Oh
commonly
just
at
the
G
RPC
layer.
What
the
api's
provide,
if
you
just
logged
out
yourself
or
I,
turn
on
the
frame
logging
and
you
sort
of
ignore
a
lot
of
the
HTTP
to
stuff
you
can
get
pretty
far,
but
tracing
and
Status
are
also
great
or
tracing.
There's.
D
My
questions
related
to,
if
you
have
a
service
somewhere
and
you
can't
invoke
a
normal
unary
RPC
to
it
like-
let's
say
it's
behind
a
NAT
and
you
can't
punch
a
hole
through
that
net
is
using
a
streaming
RPC
an
acceptable
pattern
that
you've
seen
used
for
getting
around
that
sort
of
thing.
So
you
can
listen
for
requests
in
the
opposite
direction,
so.
A
It's
this
is
really
the
you
mentioned
listening
to
request
in
the
opposite
direction.
That's
I
can't
use
case
that
comes
up
pretty
frequently.
It's
not
everyone
having
to
deal
with
it,
but
whenever
you're
needing
to
deal
with
it
really
needing
to
deal
with
it,
and
so
this
is
basically
the
core
problem.
Is
the
client
and
server
are
reversed
who's
requesting
who
to
do
what
the
client
can
access?
The
the
the
real
client
can't
access
the
real
server,
and
so
you,
you
sort
of,
have
to
do
some
hole
punching.
A
D
A
A
We
would
have
one
RPC
that
would
be
full
duplex
PI
die,
and
then
we
end
up
just
using
that
as
a
carrier
for
a
bunch
of
bytes,
and
then
it
ends
up
layering
HTTP
to
engineer,
if
you
see
on
top
of
that,
and
that
gets
you
a
pretty
good
inversion
that
I
I
ran
out
of
some
time.
As
with
that
installed
a
little
bit
but
I
think
there's
been
a
quiet.
There's
a
issue
open
for
that
that
it
receives
a
reasonable
amount
of
pings,
because
I
requested
people
to
ping
it.
A
So
there
has
been
some
look
at
that.
It's
totally
feasible
and
I've
got
a
proof
of
concept
that
technique
that
we
get
totally
works.
The
question
is
more
just
protocol
details
of
hoping
to
do
there,
there's
a
couple
decisions
to
make
and
that's
something
that
a
cup
would
be
helpful
to
have
a
little
bit
of
community
involvement
with,
because
you
basically
need
the
client
to
register
with
a
server
and
exactly
what
information
it
provides.
How
fancy
or
simple
is
that
there's
that's
most
of
where
the
therm
aining
design
questions
are.
Thank
you
only.
A
So
yes,
the
the
streaming
and
long-lived
are
PCs
are
supported
in
all
the
clients.
It
is
something,
though,
that
whenever
you
start
using
share
PC
web
those
sorts
of
things
that
you
may
not
get
so,
for
example,
by
die
you
don't
you
don't
get
full
by
die,
full
duplex
streaming
and
your
PC
web,
but
all
the
other
I'll
say
native
implementations,
so
the
the
main
nine
they
support.
All
this
thanks.