►
From YouTube: gRPC Community Meetup: 3.23.21
Description
On March 23, 2021 the gRPC community held their monthly meetup. Animesh Chaturvedi, Distinguished Engineer at Palo Alto Networks, presented "Large Scale Ingestion using gRPC". Cortex Data Lake Ingestion Service has the requirement to ingest millions of requests per second while acknowledging each request to guarantee no data loss at low latency. Cortex Data Lake clients are typically long-lived and send data continuously. With these requirements, gRPC quickly emerged as the technology of choice to build Cortex Data Lake Ingestion Service.
A
And
we
are
off
welcome
everybody.
This
is
the
grpc
community
meetup.
My
name
is
maria
cruz,
I'm
a
program
manager
in
the
world,
open
source
programs
office,
and
we
are
going
to
start
this
meetup.
I
wanted
to
have
a
general
question
to
the
community
to
ask:
if
there's
any
updates
for
the
community
before
we
jump
into
the
demo,
anybody
or
any
working
groups
that
may
have
updates
now
would
be
a
good
time
to
share
them.
A
C
B
All
right
so
good
afternoon,
folks,
thanks
for
inviting
me
to
this
grpc
meetup.
This
is
sort
of
a
talk
that
I
had
given
at
istio,
khan
and
maria
reached
out
and
then
romina
reached
out
to
me
to
say:
ask
me
to
come
and
share
our
experience
at
palo
alto
networks
using
grpc.
So
here
I
am
and
thanks
for
inviting
me
again.
B
B
Pretty
much
came
up
at
the
top,
given
that
it
supported
a
bunch
of
different
communication
modes
and
in
order
to
support
a
high
level
of
ingest
several
hundred
billion
events
per
day.
The
cost
was
also
a
big
consideration,
so
it's
technically
possible
to
build
a
system
which,
even
based
on
http
rest,
which
can
scale
to
that
level.
But
it's
going
to
be
a
lot
more
expensive
and
costly.
B
So,
as
a
result,
we
started
looking
at
grpc
and
giving
a
high
level
overview
of
what
our
data
lake
is
and
how
our
customers
use.
This
is
sort
of
a
high
level
architecture.
Our
customer
buys
a
virtual
data
lake
by
going
to
an
onboarding
portal
at
palo
alto
networks
when
they
buy
a
particular
virtual
data
lake.
We
basically
provision
a
bunch
of
systems
behind
the
scene.
For
this,
this
is
a
completely
multi-tenant
pipeline.
So
once
a
customer
has
bought
a
virtual
data
lake
of
certain
capacity,
we
provision
the
back-end
storage.
B
B
B
B
B
B
So
when
a
client
starts
sending
data,
they're
gonna
write
to
their
request
stream
and
on
that
they'll
implement
on
next
on
error
and
on
completed
so
for
outgoing
messages.
This
stream
observer
is
provided
to
application
by
the
grpc
library
and
the
client
will
start
writing
through
that
observer
for
incoming
messages
on
the
server
side.
The
application
implements
the
stream
observer
interface
and
passes
it
to
the
grpc
library
for
receiving
data.
B
B
So
the
pros
and
cons
are
are
listed
here
and
with
bi-directional
streaming.
The
message
throughput
is
higher
and
latency
is
lower,
which
met
our
requirements,
and
our
firewalls
are
typically
connected
all
the
time,
so
it
did
make
sense
to
go
with
the
load
with
the
bi-directional
streaming.
From
that
perspective,
now,
having
long-lived
streams
and
multiple
messages
per
stream,
using
bi-directional
streaming
does
transfer
some
of
the
responsibility
of
flow
control
etc
to
the
underlying
application,
and
that's
where
we
ran
into
some
challenges
which
I'm
going
to
cover
in
in
future
in
the
next
few
slides.
B
There
were
several
issues
and
one
of
which
was
grpc.
So
I'm
gonna
talk
a
little
bit
about
the
other
issues
and
stay
focused
more
on
the
grpc
side,
so
at
high
load.
We
basically
ran
into
some
issues
in
terms
of
the
infrastructure
that
we
were
deployed,
which
is
a
cloud
provider.
We
use
google
cloud,
so
we
had
to
do
some
tuning
on
the
gke
node
kernel
side.
There
were
implementation
issues
in
the
istio
side.
B
We
were
also
one
of
the
early
adopters
of
stu
at
that
time,
and
that
decision
was
driven
mainly
because
we
were
using
grpc
grpc.
Since
we
had
already
selected
grpc,
we
were
looking
for
proxies
that
supported
grpc,
natively
and
envoy
was
the
only
proxy
at
that
time
in
beginning
of
2019
that
supported
grpc.
B
So
we
went
for
grp's,
envoy
and
stu
at
that
time,
so
what
we
found
out
was
basically,
we
needed
a
large
number
of
pods
in
order
to
scale,
and
we
had
some
challenges
in
controlling
controlling
the
back
pressure
and
the
flow
control,
and
that's
where
we
found
out.
We
had
to
do
some
work,
so
first
we
had
to
exert
flow
control
and
the
way
this
gets
done
in
grpc
is,
as
I
had
mentioned
in
one
of
the
earlier
slides.
B
Maybe
let
me
go
jump
back
to
the
slide,
so
the
client
and
the
server
implement
the
stream
observer
interface
right.
So
on
the
server
side,
if
you
want
to
exert
flow
control,
the
way
you
will
do
that
is
delay,
reading
you're
on
next
and
that
way
the
buffers
in
between
will
start
getting
filled
up.
If
the
server
is
not
reading
from
this
channel,
the
it's
basically
will
slow
the
responses
down
to
the
client.
B
Okay,
so
exert
flow
control,
and
this
is
another
important
point
was
we
had
we
wanted
to.
We
are
deployed
in
kubernetes
and
we
wanted
to
leverage
kubernetes
horizon
pod,
auto
scaling
and
the
long-lived
default
long-lived
grpc
connection
hindered
our
ability
to
use
horizontal
power,
dot,
auto
scaling,
and
I
will
cover
that
in
just
a
bit,
so
we
have
to
implement
a
connection
expiration
policy
in
grpc,
which
is
which
is
actually
configurable
via
just
some
configuration
parameters.
You
do
not
need
to
make
any
code
changes
now.
B
It's
not
just
enough
to
exert
flow
control
on
the
server
side
on
the
client
side.
Also,
you
need
to
do
a
bunch
of
work
to
be
able
to
adhere
to
those
flow
control
messages
and
for
that
in
2019,
the
version
of
grpc.
I
don't
remember
the
exact
version
we
were
using
at
that
time.
If
we
look
at
the
documentation,
the
the
flow
control
capabilities
were
marked
experimental.
So
there
are
two
ways
to
do
it.
B
You
have
on
ready
handler,
and
so
this
is
basically
when
the
underlying
channel
is
ready
it
will,
you
will
register
a
call
back
and
you
will.
You
will
basically
get
a
notification
and
then
you
can
start
sending
data
on
that
stream
and
another
issue
we
ran
into,
which
was
more
of
a
manifestation
of
how
we
had
implemented
our
client.
B
We
were
creating
large
number
of
streams
and
connection
which
was
overwhelming
our
istio
in
istio
and
our
upstream
applications,
because
there
were
simply
very
large
number
of
streams
and
each
stream
carries
data
on
it
and
has
some
capacity
in
terms
of
memory
and
if
you
have
very
large
number
of
streams
you're.
Basically,
your
parallelization
is
is
too
much
to
handle
and
it
was
not
really
the
way
it
was
designed
to
be
used.
B
B
So
as
in
our
pipeline,
we
had
grpc
clients,
they
were
talking
to
an
api
gateway
that
api
gateway
was
a
proxy
which
would
proxy
the
request
to
our
upstream
service,
and
we
were
we
are.
We
were
using
stun
envoy,
and
so
there
was
a
side
car
also
injected
as
part
of
our
our
workload.
So
if
you
look
in
in
this
diagram,
there
are
multiple
buffers
on
the
left
side
in
the
orange
box.
Is
the
grpc
client
neti
buffer?
B
It
is
important
to
tune
these
buffers
as
per
your
application
need
and
in
our
deployment
envoy,
which
is
being
used
as
the
gateway
and
the
sidecar
per
grpc
stream.
Buffer
value
was
very
large.
It
was
256
mb,
so
imagine
that
you
have
large
number
of
streams
coming
from
the
clients
and
being
processed
by
these
ingestion
pods.
B
Okay,
so
so
I'll,
let
me
start
one
more
time,
so
basically,
these
buffers
you
need
to
you
need
to
control
how
large
these
buffers
should
be,
so
that
you
can
communicate
the
back
pressure
to
the
client
quick
enough.
So
we
ended
up
tuning
these
buffers
down.
The
the
client
buffer
were
are
basically
java,
netty
controlled
and
they
were
they
are
by
default
at
1
mb.
B
One
other
challenge
we
ran
into
was
our
load
varies
from
in
a
ratio
of
one
is
to
two
for
min
to
max
what
that
means
is
during
the
night
time
our
traffic
is,
let's
say
one
during
the
daytime.
It
will
be
twice
and
it
is
at
our
scale.
Cost
is
very
important,
so
we
wanted
to
leverage
kubernetes,
hpa
or
horizontal
pod,
auto
scaling
to
add
our
ingestion
workload
pods
and
as
as
more
traffic
comes
in
the
basically,
we
wanted
to
add
additional
pods.
B
This,
however,
does
not
work
well,
if
you
have,
if
the
load
distribution
across
different
grpc
stream
is
non-uniform
and
what
we
found
was,
there
was
a
wide
range
of
variations
across
these
these
pods
in
terms
of
how
much
cpu
utilization
they
were,
they
were
working
with,
and
there
were
like
hot
spots.
Some
pods
were
if
they
somehow
happened
to
get
streams
that
were
that
had
huge
amount
of
data,
they
would
run
very
hot
and
sometimes
run
out
of
memory,
and
we
could
not
really
leverage
kubernetes,
hpa.
B
Kubernetes
hpa
relies,
on
average,
of
across
all
your
pods
to
basically
add
additional
parts,
but
if
the
variation
across
the
pauses
is
very
high,
it
may
not
even
rise
up
to
an
average
value
which
is
whatever
threshold
you
have
set
to.
So
it
is
important
to
narrow
the
band
for
a
a
more
controlled
and
uniform,
uniform,
auto
scaling.
B
So
in
order
to
do
that,
we
had
to
control
the
grpc
connections.
If
the
connections
are
long
lived,
they
would
basically
get
pinned
to
a
specific
upstream
pod
and
start
sending
data
to
it,
which
would
record
result
in
hotspots.
So
we
had
to
control
the
grpc
connections
and
for
that
in
case
of
java
grpc
library,
this
is
done
by
using
two
configuration
server
builder
options
max
connection
age
and
max
connection
age
grace.
B
So
what
this
means
is
the
server
will
allow
the
connections
to
go
up
to
max
connection
age
and
it
will
also
allow
it
will
not
allow
new
streams
to
be
created,
but
it
will
allow
existing
streams
to
be
to
continue
until
that
grace
period
option.
So
we
played
with
bunch
of
different
options
to
arrive
at
based
on
different
load,
arrive
at
values
that
would
work
with
us
for
us
and
after
that
period
the
connection
gets
expired
and
and
for
the
auto
scaling
portion.
B
So,
after
all
of
these
changes,
we
basically
were
able
to
run
our
pipeline.
We
currently
process
few
hundred
billion
events
in
a
day
and
we
have
been
growing
very
quickly
in
last
year
or
so,
and
for
since
this
one
year
we
really
didn't
have
to
touch
our
pipeline
hpa
kicks
in
grpc
works
beautifully
and
we
are
able
to
able
to
continue
to
continue
to
grow
and
and
and
be
horizontally.
Scalable.
B
C
B
B
C
B
That
was
the
main
one
and
we've
we
got
thrown
off
because
it
was
marked
experimental.
We
didn't
know
that
we
could.
We
could
use
it
if
it
was
not
marked
experimental.
I
would
say
it:
it
was
pretty
much
up
to
the
it
was
good,
but
because
of
marking
it
as
experimental,
we
were
not
sure
we
could
leverage
it
and
maybe
maybe
it
made
sense,
because
not
everybody
runs
at
that
scale
and
it
was
working
fine
for
us
until
until
we
cross
like
10
million
events
a
day.
E
B
B
Now
this
test
was
done
when
we
had
pretty
much
zero
pods,
so
you
can
see
that
the
first
part
that
came
up
took
in
about
10
000
requests
per
second
or
actually
11
000
requests
per
second,
and
each
of
these
colors
are
indicating
a
new
part.
That's
that
joined
the
the
pool
of
pool
of
pods
in
this
deployment
right,
so
you
can
see
the
first
few
pods
they're
going
to
run
hot.
B
They
are
handling
quite
a
bit
of
requests
per
second,
their
cpu
utilization
will
would
be
high
and
the
hpa
or
the
average
cpu
utilization
across
pod
would
be.
In
that
case,
I
think
our
threshold
was
70
would
rise
up.
Kubernetes
would
add
an
additional
pod.
A
new
pod
will
come
up
and
you
can
see
that
over
time
these
parts
started
the
the
load
across
the
pods
started
averaging
out
and
come
closer
to
about
five
to
six
k.
E
B
B
So
it
really
depends,
so
there
is
one
more
problem
which
I
did
not
cover
in
this
slide,
but
actually,
maybe
maybe
I
can
talk
about
it
so
in
our
pipeline,
while
we
handle
the
grpc
load,
balancing
issue
with
connections
and
by
tuning
the
stream
duration,
what
it
also
means
is
the
data
that
we
receive
in
our
data
lake
is
coming
from
multiple
tenants
right
so
over
time,
a
given
pod
will
be
handling
data
for
multiple
tenants
right
at
any.
Given
time
you
can
have
data
with
multiple
tenants
coming
in
or
upstream
is
kafka.
B
So
if
I
go
back
to
one
of
the
earlier
slide
building
a
data
lake,
so
once
the
data
comes
to
the
injection
service,
it's
all
multi-tenant.
This
is
this
is
where
our
grpc
server
lives.
These
pods
are
scaling
up
and,
as
these
streams
get
terminated
and
new
streams
get
created,
these
pods
will
end
up
pretty
much
seeing
any
tenant
in
the
system
and,
as
a
result,
we
will
create
a
large
fan
in
for
kafka.
B
B
C
B
B
It
really
depends
on
how
much
time
an
upstream
service
is
processing
the
request,
if
upstream,
is
not
doing
much
work,
your
throughput
would
be
much
higher
and
it
also
depends
on
whether
you
need
acknowledgement
or
not.
Acknowledgement
means
that
if
the
client
has
sent
a
request,
the
server
needs
to
acknowledge
and
only
then
the
clients
will
send
next
set
of
requests.
This
ends
up
tying
resources
on
the
server,
as
well
as
the
client
side
and
limits
throughput,
which
is
how,
in
our
case,
that
was
our
requirement.
B
We
could
not
afford
a
data
loss,
so
we
had
to
ensure
that
the
client
and
server
are
acknowledging.
But
if,
in
your
use
case,
if
it's
let's
say
a
workload
which
requires
more
of
an
analytical
workload,
in
that
case,
you
may
you
may
get
by
by
only
doing
client
streaming
and
server
doesn't
need
to
acknowledge.
In
that
case,
your
throughput
will
be
much
higher.
C
About
the
load
balancing,
so
it's
mostly
like
connection
based
right.
It's
a
long-lived
connection.
So
once
the
connection
is
gracefully
shut
down
by
the
server,
how
does
the
client
know
it
has
to
connect
to
a
next
set
of
like
ip
addresses?
Did
you
implement
any
load
balancing
on
the
client
side
so.
B
So
that
is
actually
handled
by
the
grpc
library
itself.
We
did
not
end
up
needing
to
do
do
anything
about
it.
All
you
needed
was
the
fqdn
of
the
server
and,
if
the
server
in
in
this
case
the
server
terminating
the
connection,
the
client
will
basically
would
need
to
had
a
connection
pool,
and
it
will
basically
pick
another
connection
out
of
connection
pool
and
start
sending
on
that.
One.
The.
C
B
C
B
C
B
I'm
not
directly
exp,
I
didn't
have
experience
running
it
on
aws,
but
we
were
using
a
google
cloud
and
the
nlb
we
didn't
need
to
do
any
configuration
there.
Basically,
the
the
the
client
was
given
an
fqdn
and
that
fqdn
would
map
to
a
virtual
ip
behind.
That
would
be
a
bunch
of
let's
say
the
gateway,
pods
or
envoy
pods,
and
it
didn't.
We
didn't
have
to
do
any
any
other
tuning.
B
B
So
grpc
client
is
a
java
client.
We
also
had
like
one
of
these
application
is
a
java
client
and
the
firewall
grpc
client
is,
is
go
based
and
there
are
subtle
differences
between
the
go
client
versus
the
java
client
java
is
fully
asynchronous
for
go
the
go
grpc
library
is
blocking,
and
that
required
us
slightly
different
tuning
in
terms
of
these
stream
buffers.
B
B
B
B
Yes,
that's
correct,
so
most
of
our
requests
going
in
a
stream
are
below
1kb
and
it
really
depends
on
how
quickly
you
want
to
sort
of
communicate
back
to
the
client
that
they
need
to
slow
down.
So
if
you
can
and
and
the
thing
is,
if
you
do
not
slow
the
client
down
or
don't
provide
them
the
signals
and
if
you
have
large
load,
your
server
can
blow
up
right.
So
it's
it.
B
E
Thank
you,
I
mean
lakshmi
here.
I
have
a
quick
question,
so
I
was
when
I
was
trying
when
read
your.
I
read
your
the
article
and
they're
very
nice
and-
and
I
was
trying
to
implement
it
in
my
company,
but
I
could
not
make
it
work
in
the
in
the
data
flow.
The
client
side,
especially
the
client
side.
Do
you
have
any
like
like
remove
all
your
proprietary
stuff?
Do
you
have
any
sample
program
which
you
can
put
in
the
github?
Is
it
possible.
B
E
B
E
Yeah
to
control
the
flow
streams,
we
were
using
the
semaphore
and
the
semaphore
type
to
control
the
flow
control
on
the
front
and
the
reading
itself,
but
the
observer
and
the
stream
observer
on
the
data
flow.
We
are
not
sure
how
that
how
that
exactly
implemented.
I
tried
to
get
it
in
the
grpc
examples.
Also,
there
was
no
not
such
thing
for
the
data
flow.
So
that's
right!
Okay,
if
you
can.
B
E
C
C
D
B
We
actually
didn't
upgraded
it
just
once
we
haven't
done
it
yet.
So
maybe
it's
a
good
time
for
for
me
to
go
back
and
read
up
on
the
release,
notes
for
grpc
and
try
it
out
in
in
our
devs
and
box
is:
are
there
any
specific
features
added
in
last
18
months
or
so
that
are
significant
enough
for
us
to
pick
it
up.
D
So
the
b2b
change
will
actually
be
disabled
whenever
you
use
the
flow
control
window.
Api.
Let's
see,
is
that
right,
there's
two
there's
two
flow
control
window
apis
now,
one
of
them
disables
it
and
then
one
of
them
just
sets
it
as
that's
the
default
value.
But
I
don't
think
in
in
your
environment
in
most
sort
of
data
center
environments,
you
don't
need
as
large
of
a
buffer
size.
So
I
don't
think
it
would
impact
things
too
much.
D
You
might
notice
that
maybe
the
buffer
has
increased
from
64k
to
128k
or
something
like
that
and
you
squeeze
out
a
little
bit
more
performance,
but
I
don't
really
expect
it.
I
think
I
think
mainly
the
improvements
have
been
around
some
air
handling,
just
making
better
error
messages
and
small
bug
fixes
that
sort
of
those
accumulate
over
time.
C
B
D
The
flow
control
just
applies
to
the
streaming
calls
with
unitary
calls.
You
have
to
do
other
sort
of
more
classic
approaches
of
either
fail.
The
request
and
have
the
clients
do
some
sort
of
back
off,
and
you
may
even
tell
the
client
how
how
long
to
sleep
or
you
not
do
a
target,
but
you
you
add,
in
extra
latency
and
delay
to
slow
down
the
client
a
little
bit,
but
those
those
are
pretty
classic
for
pretty
much
any
system.
They
don't
relate
specifically
to
grpc
as
much.
C
C
B
A
All
right
before
I
let
you
go
and
in
case
there
aren't
any
other
new
questions.
I
wanted
to
direct
your
attention
to
the
icon.
That
is
right
next
to
the
chat
box,
which
is
three
shapes.
There's
a
poll
active
there.
So
if
you
could
give
us
your
feedback
through
the
poll,
that
would
be
great
and
I'm
launching
another
poll
right
now,
so
these
these
surveys
really
help
us
to
to
improve
the
meetup
every
time.
So
thank
you
for
taking
a
couple
of
minutes
to
to
fill
this
out.
A
So
if
you
would
like
to
show
your
implementation
of
grpc,
please
do
get
in
touch
with
a
the
google
group
that
we
have
kind
of
launching
on
the
name
right
now,
but
I
think
it's
here,
io
at
google
groups.com
checking
that
yes,
that's
the
email
address.
So
please
email
there
with
your
presentation
idea
and
we
can
also
feature
your
implementation
and.
A
A
I
just
received
the
the
email
address
on
the
on
the
chat
box.
So,
if
you
can,
let
us
know
there,
we
will
be
happy
to
also
feature
your
work
and,
as
I
said
before,
this
recording
is
going
to
be
available
on
the
grpc
youtube
channel
and
once
again,
thank
you
so
much
animesh
for
presenting
and
yeah
everybody
else
also
for
engaging
and
for
bringing
your
questions
and
comments
to
to
this
event.
It's
really
what
makes
the
community
meet
up
thrive.
So
thank
you.