►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lightning Talk: More Intelligent Global Rate Limiting - Tianyu Xia, Google
Effective traffic management is the key to allow customers to ensure that their micro-services and overall architecture are highly available and highly reliable . How to design the rate limiting system for distributed systems handling millions of requests per second at Google scale? How to make the rate limiting system smarter to ensure fair sharing between various clients and handle the service surge in a particular region? This talk will go into details about the design of this more scalable, intelligent and performant rate limiting service and how your own service can benefit from it.
A
A
A
So,
first
the
background,
so
why
we
need
to
read
limiting
particularly
effective
traffic
management
is
a
key
to
allow
customers
to
ensure
that
their
micro
surveys
and
over
architecture
is
highly
available
for
when
any
particular
client
from
exhausting
service
resource
and
a
highly
reliable,
be
resilient
to
misbehaving
clients
from
overloading
the
service
in
Envoy.
It
can
delegate
the
retainmentization
to
external
service.
A
A
B
A
A
So
a
few
things
to
highlight
here:
first,
as
you
can
see
from
API
interface,
it
is
using
unary,
grpc
mode,
video
client
sender,
single
request,
get
a
SQL
response
back
like
a
normal
function,
call.
Secondly,
Envoy
query:
integrating
server
for
every
incoming
custom
request,
certainly
so,
on
the
left
hand
side
the
clock,
the
custom
request
is
blocked,
while
waiting
for
the
response.
A
B
A
A
B
A
A
So
first
Imperial
slice
I
mentioned
unit.
Grpc
is
not
good
for
performance.
Here
we
switch
to
bytext
streaming
mode.
It
provides
the
persistent
connection
between
client
and
the
server
basically
because
inside
stream
mode,
so
the
client
and
server
can
send
arbitrary
number
or
message
back
and
forth
over
long
live
stream,
so
why
this
is
good
for
performance.
A
First,
let's
look
at
the
retail
limiting
server
side,
so
this
is
thickness
actually
have
helped
us
to
make
a
full
use
of
server-side
functionality.
For
example.
Caching
we
can
avoid
the
cache
Miss
on
the
cost
of
Cashmere,
so
I.
Imagine
that
repeatedly
hitting
different
backend
server
will
increase
the
chunks
of
cache
Miss,
so
the
written
request
will
have
to
wait
for
the
data
to
be
available
and,
besides
that,
on
the
server
side,
it
can
also
help
to
reduce
the
synchronization
between
different
backhand.
A
So
next
is
our
quota
based
stable
approach,
so
what
is
quarter
based?
Basically,
we
group
The
Client
into
each
quarter
bucket,
why
we
are
doing
that
so,
firstly,
because
richer
limits
can
be
satisfied
in
various
degree
of
granularities
by
default.
Our
clients
are
equal,
but
you
can
also
group
the
client
to
so
that
you
can
allocate
more
of
your
capacity
to
hybrid
clients,
for
example,
if
you
have
a
production,
client
and
a
developer
client,
you
may
want
to
allocate
more
quarters
to
your
production
client.
A
Secondly,
it
can
help
to
Associated
response,
with
request
so
implied
eye
streaming
mode.
The
order
of
response
requests
are
not
guaranteed,
so
basically
the
client
and
the
server
can
really
write
in
whatever
order
they
want,
but
for
vitamin
client
it
is
required
to
know
which
response
correspond
to
which
request
so
that
they
can
apply
the
recommendization
properly
so
quarter
bucket
with
the
bucket
ID,
as
identify
will
serve
as
a
bridge
between
the
server
and
the
client
to
establish
the
mapping
between
the
request
and
the
response.
A
So
next
is
how
we
group
them.
Physically,
we
generate
the
bucket
ID,
either
statically
or
diamantically.
So
let
me
use
a
diagram
below
to
explain
this.
So
basically,
on
the
left
hand,
side
is
a
configuration
on
the
right
hand.
Side
is
a
general
debugged
ID
for
the
static
method,
so
the
key
and
value
from
the
configuration
will
be
used
as
they
are
for
the
dynamic
method.
A
A
So
next
it's
about
how
we
make
it
a
staple
push.
The
answer
is
pretty
simple:
we
are
using
cash,
so
we
leverage
the
thread:
local
storage
on
void
to
catch
a
response
from
reading
the
server
by
using
cache.
It
can
first
avoid
the
returning
queries
to
the
server
and
also
it
can
improve
the
latency
on
the
client
side
if
there
is
already
valid,
according
in
the
cache.
A
So
last
is
about
our
report.
Reply
reply
pattern
with
subscription
model,
so
first
and
so
basically
for
retaining
server
using
client
is
a
periodically
reported,
it's
called
usage
to
the
server
and
for
the
written
server,
and
it
will
send
back
assignment
once
it
has
collect
enough
reports
to
make
the
decision
and
under
the
sufficient
model
here
means
that
the
first
report
from
the
client
will
serve
as
an
indicator
to
the
server
that
okay,
the
client
is
subscribed
to
receive
future
updates
from
the
server.
A
So
what
what's
the
benefit
is
this
model
provides?
Basically,
you
can
provide
a
more
intelligent,
rhythmatization
so
right
now
the
routine
server
can
adjust
the
quarter
assignment
based
on
real-time
usage
report
from
a
client.
A
single
example
below
requires
assign
per
use.
So
there's
a
like
a
social
media
website,
News
website
and
a
shopping
website
on
Black
Friday.
There
might
be
a
spike
of
internet
usage
on
the
shopping
website.
A
So
yeah,
so
let
me
summarize
the
design
and
wrap
it
up.
So
first,
this
design
is
more
scalable.
Basically,
we
have
this
staple
infrastructure,
which
is
called
the
cache
you
can
avoid
returning
queries
to
the
server
and
also
yeah.
This
design
is
more
intelligent
because,
right
now
the
written
server
can
adjust
the
assignment
based
on
the
real-time
client
usage
report.
Last
but
not
least,
the
design
is
more
performant
for
some
server
side,
because
it
can
make
full
use
of
server-side
functionality
and
avoid
the
heaviest
connection
between
different
server
back-ends.
A
A
A
For
the
service,
so
basically
our
design
is
not
don't
really
care
about
the
client's
information
so
because
the
client
is
grouped
with
a
quarter
bucket.
So
what
we
clear
is
about
the
each
book
called
the
buckets
so
I
think
our
design
doesn't
need
to
synchronize
between
different
clients
yeah.
If
I
understand
your
question
card.
A
Yeah,
so
that's
a
good
question,
so
basically
we
have
a
predefined
configuration
even
on
the
so
if
there
is,
for
example,
there's
no
assignment,
so
we
have
a
predefined
role
instead
of
a
querying
server
and
also
in
the
cache
side
in
the
server
side,
as
you
mentioned,
we
can
like
pre-load
the
ultimatization.
Instead
of
waiting
for
the
data
to
be
available
right.