►
From YouTube: Context Aware Traffic Routing for Stateful Services using Envoy - Renana Yacobi & Piaw Na, Niantic
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Context Aware Traffic Routing for Stateful Services using Envoy - Renana Yacobi & Piaw Na, Niantic
Service infrastructure should be invisible to application APIs but frequently the implementation leaks into the APIs, injected in the form of URL prefixes or message headers custom fields. In some cases this is undesirable due to the sensitivity of the information. Examples of such cases include PII and company IP. In these scenarios a message is sent to the server, which makes a routing decision based on the information inside the message. At that point a reroute request is send to the client that redirect its calls to the assigned backend, In this presentation, Renana Yacobi, Server Core Infrastructure Lead @ Niantic, presents a context aware routing, using the power of Envoy custom filters and partial message parsing to use information extracted from the message payload to route the traffic while supporting legacy APIs.
A
Thank
you
for
coming
to
the
talk
at
the
nearly
the
conference.
As
randy
said,
the
title
is
context
aware:
traffic
routing,
here's
the
agenda
for
the
talk,
we're
gonna
talk
a
little
bit
about
ourselves,
explain
to
you
what
the
motivation
you
might
have
for
doing,
context-aware,
routing
discuss
the
issues
involved,
talk
about
the
implementation,
look
at
results
and
then
talk
more
about
our
prototype
and
we'll
definitely
have
time.
At
the
end
for
questions.
A
A
A
A
We
actually
peek
back
multiple
protocol
buffers
into
a
single
request
and
when
the
monolithic
application
is
broken
up
into
microservices,
one
thing
that's
useful
to
do
is
you?
Can
hey
now
take
each
of
these
individual
protocol
sub
messages
in
that
protocol,
buffer
and
route
them
to
different
services
simultaneously?
A
A
Looking
at
a
graph,
you
can
see
some
interesting
characteristics
so,
for
instance,
for
the
large
messages,
the
memory
is
much
more
spiky
and
we
speculate.
This
is
because
of
the
heap
allocator.
You
would
get
a
message.
You
would
spend
some
time
like
defragging
your
free
list
or
allocating
free
lease
and
breaking
it
up,
and
so
that's
that's
where
you
see
these
like
up
and
downs
on
the
memory
use
and
then
on
the
cpu.
A
Okay,
let's
see
if
we
can
improve
it
to
be
able
to
like
peek
inside
the
actual
message
you
actually
without
actually
having
the
protocol
buffer
definition,
you
need
to
decode
the
protobuf
wire
format
turns
out.
There
are
only
five
objects
that
are
typically
inside
a
protocol
buffer
and
one
of
them
has
been
deprecated,
so
you
don't
even
have
to
think
about
it.
So
that
makes
fall
the
key
the
most
the
most
important
one
is
the
variant:
that's
what
you
have
on
the
right
side
of
a
protocol
buffer
definition
when
you're
specifying
a
field.
A
A
The
fixed,
64
and
fixed
32-bit
fields
are
self-explanatory,
they're
usually
used
for
doubles
or
floats,
and
the
interesting
bit
from
our
point
of
view
is
the
length
delimited
field
right
and
those
are
used
for
both
strings
and
embedded
protocol
buffer
messages
that
are
inside
the
overall
big
protocol.
But
so
our
insight
is,
when
you
see
one
of
these
lengths
of
limited
views,
you
can
try
to
unpack
it,
treat
it
like
a
message
and
if
you
fail,
then
it
was
a
string.
A
If
you
succeeded,
then
you
can
peek
inside
and
say.
Oh
you
know,
let
me
take
a
look
at
what's
inside
and
in
this
example,
that's
on
the
screen.
What
we
do
is
we
have
an
envelope
that
contains
four
four
strings
and
as
an
exercise
this
this
last
string
envelope
id
is
used
to
inject
a
unique
string
that
lets
you
say.
Oh,
this
particular
thing
is
one
of
these
envelopes
that
can
be
used
for
routing
and
the
other
three
fields
app
id
server
id
player
id
could
easily
be
the
content
on
which
you
do
your
routing.
A
A
few
interesting
observations.
First
of
all,
you
the
envelope
can
be
retrofitted.
So
if
you
had
an
existing
protocol
buffer,
you
know
where
the
payload
is
already
used
up.
Like
the
first
10
views,
you
can
use
the
12
field
to
inject
an
envelope.
If
you
needed
to
do
that,
when
you
have
something
like
an
envelope
id,
that's
a
unique
string,
that's
used
to
identify
the
envelope.
A
What
you
can
do
is
you
can
change
the
contents
of
that
string
in
order
to
indicate
the
version
number
of
the
of
the
envelope
that
you're
doing
so,
you
could
indicate
a
backwards,
incompatible
change,
for
instance,
if
you
needed
to
change
the
type
of
envelope
you
had
and
lastly,
as
previously
stated,
you
can
actually
use
a
combination
of
the
data
in
the
envelope
to
do
a
routing.
You
don't
have
to
depend
on
just
one
field
or
another,
so
we
implemented
this
partial
deserialization,
and
this
is
what
it
looks
like
for
small
messages.
A
We
used
about
the
same
number
of
calls,
but
the
amount
of
memory
used
has
been
reduced
by
more
than
50
percent
right,
and
this
this
this
makes
sense.
Well,
you
know
for
a
small
message
envelope.
You
know
it's
it's.
Even
the
the
envelope
is
not
as
big
as
half
a
megabyte,
so
you
you're
only
deserializing
as
much
as
you
need
to
do
routing
and
then
the
rest
of
the
protocol
buffer
is
just
handed
off
for
large
envelopes.
A
A
A
This
is
the
interesting
chart.
What's
happened
in
the
large
message
case
is
that
you
no
longer
see
the
jiggle,
because
you
you're
no
longer
fragmenting
your
heap
as
much
as
you
used
to
both
the
cp.
Both
the
cpu
on
both
sides
look
pretty
much
the
same,
and
but
you
know
in
in
this
particular
case
we're
pretty
happy
about
the
performance
improvement.
We
got.
B
So
now
we
would
like
to
focus
regarding
context
that
we're
routing
for
the
case
of
our
stateful
services,
and
what
does
it
mean
for
niantic,
so
in
niantic,
when
you
play
pokemon
go
a
mobile
client
is
when
it
connects
for
the
first
time
is
assigned
a
server.
That
server
is
where
the
player
state
is
being
maintained
and
from
that
point
on,
we
want
the
request
to
be
as
fast
as
possible,
more
in
the
area
of
milliseconds
than
in
the
area
of
seconds.
B
So
how
does
it
look
when
a
client
connects
for
the
first
time
it
sends
a
handshake
request.
The
server
then
goes
into
a
storage
where
the
state
of
the
player
is
being
stored
and
it
asks
the
storage
to
assign
the
player
to
himself.
The
storage
then
check,
is
it
possible?
Is
it
the
player
already
assigned
a
server?
If
it's
assigned
the
server,
then
it
will
return
the
id
of
the
server
it's
already
assigned
to
and,
if
not,
it
will
say,
yeah
sure
your
client.
B
The
servers
themselves
to
make
sure
that
everything
works
correctly
are
talking
with
the
storage
where
the
player
state
is
stored,
and
in
that
storage
we
have
mechanism,
of
course,
for
timeouts
when
a
client
is
not
connected
for
a
long
time
and
locks.
Of
course,
to
make
sure
the
client
is
assigned
only
a
single
server,
all
this
flow
allah
adds
a
lot
of
latency.
Now,
it's
true
that
this
happens
usually
only
on
first
connect
or
after
a
long
time
that
the
client
haven't
connected.
B
B
We
take
the
client
to
server
mapping
and
put
it
closer.
So
in
this
case,
for
example,
the
client
to
server
mapping
is
part
of
the
memory
of
the
running
invoice.
Every
message
that
gets
in
we
pull
the
context
in
this
case
the
client
id,
and
then
we
check.
If
there
is
already
a
map,
then
we
already
know
where
to
route
it.
If
not,
then
we
take
just
the
selected
load,
bouncer
algorithm,
and
then
we
apply
it.
Take
whatever
invoice
proposed
to
us
and
ask
the
mapping
to
update
accordingly.
B
The
invoice
holds
the
client
to
server
mapping
and
the
servers
themselves
don't
need
to
communicate
with
the
storage
to
make
sure
that
they
are
talking
with
the
right
the
right
client
now.
One
of
the
things
that
is
important
to
note
here
is
that
we
guarantee
100
correctness,
and
why
is
that
important?
B
So,
first
of
all,
most
of
you
already
noticed
that
this
works
only
if
you
have
one
invoice,
because
if
it's
in
memory,
what
will
happen
with
the
rest
of
them?
You
need
a
way
now
to
synchronize
this,
so
our
proposed
solution
was
to
add
a
service
called
the
state
coordinator.
This
is
where
the
origin
of
your
data.
This
is
your
source
of
truth.
This
is
where
the
client-to-server
mapping
actually
lives.
Each
envoy
have
a
cache
of
that
one.
B
Now,
whenever
a
new
client
connects
whenever
a
client
connects,
sorry,
then
involve
will
check.
Do
I
have
it
in
the
cache?
If
I
have
it
in
the
cache
great,
I
just
map
it
to
where
it
should
go.
If
I
don't
have
it
in
the
cache,
then
invoi
will
go
and
ask
the
state
coordinator
to
assign
a
server
now.
One
of
the
things
that
the
state
coordinator
also
have
is
the
ability
to
monitor
which
servers
are
alive.
B
If
a
server
is
started,
he
can
reroute
clients
and
have
a
lot
of
logic
for
supporting
that,
and
then
it
will
reassign
a
new
backend
now
the
moment
that
it
have
a
new
backend
assigned.
It
means
that
your
your
origin
is
already
updated.
It
will
reply
with
the
server
that
the
android
needs
to
route
to,
and
then
the
mechanism
where
we
update
the
the
cache
on
each
invoice
is
very
similar
to
the
cosmos
db
change
feed.
B
I
will
not
try
to
start
to
explain
that
one,
because
it's
very
complicated,
but
there
is
a
link
here
and
we
will
uploaded
the
update
presentation
with
this.
But
the
way
that
we
are
updating
is
that
whenever
and
change
is
happening
on
the
client
server
mapping,
we
just
stream
the
change
itself
that
allows
us
to
do
fast
updates
and
allows
the
invoice
to
be
updated
as
much
as
possible.
So
getting
back
to
the
subject
of
100
correctness.
B
So
in
this
case
we
moved
a
lot
of
the
computation
and
the
logic
that
the
state
that
was
maintained
in
the
back
end
and
just
shifted
it
closer
closer
to
where
actually
it's
needed,
which
is
where
we
are
routing
messages.
It
definitely
improved
performance
and
then,
when
we
started
to
handle
larger
messages,
we
saw
things
are
costing
us
more
and
take
a
bit
more
time.
And
then
we
started
investigating
the
message:
partial
messages
organization
to
help
with
large
messages.