►
From YouTube: Decentralized Routing For a Sharded Application On Service Mesh - Vinay Gonuguntla & Pankaj Sikka
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Decentralized Routing For a Sharded Application On Service Mesh - Vinay Gonuguntla & Pankaj Sikka, Intuit
Speakers: Pankaj Sikka, Vinay Gonuguntla
Intuit has a few monolithic applications that are scaled horizontally by sharding. It is common practice to use a dedicated frontend layer to route requests to a specific application shard based on request attributes. The attributes used for shard determination are non static and include tens of millions of users and thousands of services. Hence maintaining a static mapping of these attributes to shards is not feasible. For maintainability and separation of concerns, a dedicated look up service could be used to store and retrieve this shard information. Currently at Intuit, the sharded routing is performed in a central API Gateway using this lookup service. However, as we move our monolithic applications to service mesh, our next step was to decentralize the sharded routing to happen on a client’s service mesh proxy. A service mesh that utilizes a client side proxy like Envoy does not have an out of the box support for lookup based dynamic routing to the destination shards. This session discusses and demos how Intuit uses WASM to extend Envoy in a service mesh to provide decentralized routing for a sharded application.
A
So
we
are
software
engineers
at
Intuit
and
we
work
on
the
service
mesh
platform
at
into
it.
So
let
me
go
over
the
agenda
for
today,
we'll
be
talking
about
a
little
bit
about
what
service
mesh
is
and
how
it
is
set
up
at
Intuit
a
little
bit
about
the
starter
application
and
how
routing
is
done
on
a
starter
application,
our
design
and
some
challenges
we
faced
we'll
go
over
the
demo
as
well
as
some
future
investment.
A
A
If
you
look
at
the
cncf
portfolio
into
pretty
much
uses
most
of
the
products
there,
we
also
are
a
big
open
source
contributor.
We
contribute
to
more
than
75
plus
projects
and
some
of
the
projects
we
maintain
and
our
open
source
are
Argo.
Kiko
Admiral
and
new
approach,
so
let
me
go
into
a
little
bit
of
the
scale
at
which
Intuit
operates.
A
Intuit
has
more
than
245
plus
kubernetes
clusters
running
in
production.
We
spend
about
16
000
name
spaces
in
those
clusters.
At
Peak
we
recorded
more
than
77
000
nodes
are
running
in
production
with
more
than
7
million
pods,
and
we
have
2000
plus
unique
Services
which
are
running
among
these
clusters,
and
these
services
are
a
mix
of
both
end
user
Services,
as
well
as
internal
services.
A
So
next
I
want
to
just
walk
through
service
mesh.
How
many
of
you
here
know
what
service
mesh
is
I?
Think
most
of
you
do
so
I
can
go
a
little
fast
on
this
section
is
an
infrastructure
layer
to
facilitate
service
to
service
communication.
I
think
the
most
popular
approach
right
now
is
to
use
the
sidecar
to
intercept
traffic
for
Ingress
and
egress
communication
through
the
application.
This
allows
us
to
do
few
cool
things
with
by
adding
a
sidecars
or
using
service
mesh.
A
Some
of
them
are
security
by
providing
Mutual
TLS
observability
by
taking
a
snapshot
of
the
entire
system
and
finding
the
bottlenecks
routing
by
providing
things
like
Canary
or
traffic
splitting,
and
also
reliability
by
automating
retries,
some
of
the
most
popular
service
mesh
offerings
out.
There
are
istio,
linkery
and
console.
A
We
had
to
use
istio
as
our
service
mesh
platform.
So
there's
a
little
bit
about
istio,
since
most
of
you
are
familiar,
I'll
just
I'll
go
through
this.
Pretty
quick
istio
has
or
many
services
are
mainly
divided
logically
into
two
parts:
control,
plane
and
data
plane.
The
data
plane
part
is
where
the
service
or
the
application
lies,
and
this
is
where
the
communication
happens
and
to
basically
maintain
or
configure
these
proxies.
We
have
a
control
plane
section
and
here
I'm,
just
mentioning
two
important
components
in
istio.
One
is
pilot
and
others.
A
Galley
pilot
is
responsible
for
configuring.
The
proxy
and
Galley
is
mainly
for
user
configuration
management
and
it
listens
to
your
requests
and
sends
the
instructions
to
Pilot.
So
at
Intuit
we
have
a
multi-cluster
istio
setup.
So
here
I'm,
just
representing
like
three
clusters
with
istio
installed
individually.
A
Is
your
control
plane
installed
individually
in
each
of
these
clusters,
now
think
of
a
communication
say
from
service
C
to
service
B,
which
is
within
the
cluster
and
istio
is
responsible
for
that
now,
for
example,
if
you
think
of
communication,
say
from
service
a
to
service,
C
or
service
C
to
service
d,
this
would
require
some
sort
of
a
multi-cluster
setup.
We
use
Admiral
as
our
open
source
control,
plane
component,
which
manages
different
istio
resources
among
these
clusters
and
allows
communication
between
different
Services
here.
B
Thanks
vinay
thanks
for
the
introduction,
so
before
I
dive
into
the
details
a
little
bit
about
the
traffic,
the
way
it's
handled
out
into
it
right,
so
all
the
North
South
traffic
at
Intuit
is
handled
through
into
it
managed
API,
Gateway
and
internally.
We
are
going
through
this
journey
of
moving
all
these
internal
application
traffic
to
service
mesh.
B
Basically,
what
we
had
to
do
was
solve,
for
which
particular
Shard
request
needs
to
go
based
on
the
request
context
before
we
dive
into
the
details,
I
think
it's
I'll
touch
upon
why
a
sharded
application
is
required
and
what
charted
application
is
so
think
of
a
normal
web
application,
which
is
fronted
by
API
Gateway
and
has
a
bunch
of
users
that
are
making
requests
to
it
as
the
number
of
users
grow.
There's
some
kind
of
some
need
of
some
scaling
on
the
web
application.
One
approach
to
do
so
is
sharding.
B
Where
the
web
application
splits
its
data
into
multiple
shards
and
API
Gateway
the
way
it
makes
the
routing
decision
to
these
shards,
it
could
do
so
by
maintaining
a
static
list,
but
as
the
number
of
users
grow,
even
further
static
list
is
no
longer
a
viable
approach
and
there's
a
need
for
a
better
solution
at
Intuit.
What
we
do
is
we
have
a
external
lookup
service.
That
has
the
logic,
and
it
can
you
know
it's
using
the
basically
the
same
back
end
as
these
web
applications.
B
So
it
knows
where
the
data
is
charted
and
it
also
has
an
algorithm
to
determine
The
Shard
where
it
needs
to
go
to
so
Gateway
makes
a
call
to
this
lookup
service,
and
then
it
knows
what
chart
it
needs
to
Route
it
to
to
make
this
lookup
small,
more
efficient.
We
are
also
using
a
distributed
cache.
So
one
example
of
an
application
that
does
that
at
Intuit
is
QuickBooks
Online,
which
is
an
accounting
software.
More
than
150
000
companies
use
it
and
there
are
more
than
5
million
customers
across
the
globe.
B
B
So
here
the
routing
decision
can
be
made
on
the
online
proxy
or
one
approach
is
to
let
this
request
go
through
API
Gateway
as
well,
but
then
that
introduces
an
additional
hop
between
the
service
to
service
communication
and
usually
what
the
services
that
talk
to
each
other
a
lot
are
co-located
within
the
same
cluster,
probably
also
within
the
same
VPC,
and
then
this
call
adds
Network
latency
by
going
out
of
the
VPC
and
back-end
and
so
on
right.
So
we
can
avoid
that
by
using
service
mesh.
B
So
the
approach
we
took
was
to
implement
the
routing
logic
within
the
proxy
itself,
but
before
we
go
there,
I
want
to,
you
know,
go
through
the
goals
we
had
for
this
approach,
so
as
the
shards
are
maintained
by
by
the
services
themselves,
we
did
not
want
the
client
to
be
aware
of
them.
These
decisions
need
to
be
transparent
to
the
client
services
as
well,
so
we
did
not
want
to
have
any
changes
or,
if
any
changes
we
wanted
them
to
be
very
minimal.
B
Then
also,
as
I
spoke
about
the
Qbo
use
case,
which
shards
its
data
based
on
the
company
IDs,
we
have
to
handle
that
and
also
other
applications,
identity
that
have
millions
of
users.
They
may
chart
their
data
based
upon
a
user
location,
multiple
criteria,
so
we
our
solution
needed
to
handle
all
those
use
cases
as
well.
B
So
when
imagine
I'll
go
back
to
the
Qbo
example.
When
the
data
in
a
particular
Shard
you
know
grows
beyond
a
certain
threshold,
there
could
be
a
requirement
of
dividing
that
data
into
multiple
for
the
shards,
and
this
needs
to
be
supported
in
near
real
time
as
well.
So
we
don't
want
to
cause
any
disruption
to
the
ongoing
traffic
and
the
routing
decisions
still
need
to
be
made
correctly
right,
as
I
mentioned
earlier,
service
owners
control
the
routing
configuration
we
wanted
this
to
be
transparent
to
the
client
side,
so
no
changes
on
the
client
side.
B
We
wanted
to
give
this
control
to
the
service
owners,
so
we
went
out
and
looked
at
the
existing
solutions
that
the
service
mesh
had
to
offer.
Now,
since
we
were
using
istio
service
mesh
I'll,
look
at
istio
examples
here,
it
still
provides
a
custom
resource
called
virtual
service,
which
defines
a
set
of
traffic
rules
that
apply
when
a
given
host
is
at
rest.
In
this.
B
In
this
example
right,
whenever
a
request
is
made
to
demo.greeting
dot
mesh,
these
traffic
rules
will
apply
and
each
traffic
rule
defines
a
protocol
and
the
matching
criteria
in
this
example
Company
ID,
1
and
2.
Those
requests
are
routed
to
short
one,
whereas
the
request
for
company
ID3
are
routed
to
chart
2..
This
is
very
a
basic
example
of
a
virtual
service
and
but
then
the
virtual
service
did
not
work
for
us
due
to
some
limitations
due
to
our
use
cases.
So
there
were
maintainability
problems.
What
I
mean
by
that
is.
B
If
we
have
to
incorporate
a
routing
logic
in
the
virtual
service,
there
has
to
be
some
kind
of
coupling
between
the
sharding
logic
and
it
has
to
go
to
the
virtual
service,
which
needs
to
be
created
into
multiple
clusters
where
the
client
services
exist.
So
this
is
a
lot
of
overhead
and
then
also
the
size
of
the
virtual
service
object
itself
can
grow
a
lot
which
is
again
a
management
problem,
as
I
mentioned
earlier.
B
We
need
near
real-time
updates,
imagine
a
new
company
getting
added
and
we
need
that
charting
info
reflected
in
the
virtual
Service.
This
is
this
is
not
possible.
With
the
virtual
service
approach,
we
have
to
update
the
virtual
service
every
time
a
new
company
gets
added
across
all
the
shards
again
right
across
all
the
client
clusters
and
then,
if
the
data
can
move
among
the
shards
as
well
virtual
Service.
As
you
are
aware,
it's
a
static
mapping,
so
we
cannot
make
that
decision
in
the
virtual
service.
B
So,
due
to
these
limitations,
we
started
designing
our
solution
and
I
will
walk
through
that
approach.
What
we
took
in
this
diagram
we'll
see
a
request
that
is
originating
on
the
this
is
this
block
is
the
client
that
is
making
a
request
to
the
to
a
sharded
application
divided
into
three
different
shards.
B
So
when
an
HTTP
client
makes
a
request
to
the
sharded
application,
the
on-way
proxy
intercepts
this
request,
as
always
so
onward.
Proxy
has
a
filter
chain
and
istio
provides
a
custom
resource
called
on-way
filter
that
gives
the
capability
to
extend
the
on
web
proxy
and
we
have
built
the
logic
in
an
HTTP
filter
or
HTTP
Envy
filter
that
applies
to
all
the
sidecar
outbound
requests.
B
So
what
this
on
my
filter
does
is,
as
if
you
remember
the
API
Gateway
use
case,
the
same
lookup
service
is
called
to
ask
where
a
particular
request
needs
to
be
routed
to
based
on
the
request
context
and
the
lookup
service
responds
with
the
DNS
of
the
chart
that
it
needs
to
go
to
then
the
within
the
onboard
filter.
We
are
able
to
Route
the
request
to
that
chart.
Now
this
creation
of
the
on-way
filter.
We
have
automated
this
through
Admiral,
which
is
an
open
source
tool
under
the
HTO
ecosystem,
an
airspace
that
Intuit
manages.
B
B
So
this
is
how,
with
this
solution,
this
is
how
the
dynamic
routing
identity
looks
like
this.
Typically
Source
service
and
the
destination
service
live
in
different
clusters
and
for
an
admiral
lies
on
top
of
all
the
Clusters.
It
watches
all
the
clusters
for
the
custom
resources
it
defines
and
it
creates
istio
resources
in
the
in
the
in
all
the
Clusters
where
it
needs
to
to
make
the
mesh
like
the
life
of
the
mesh
operators
and
developers
easier.
So
for
this
Dynamic
routing
use
case,
we
introduced
a
new
custom
resource
called
routing
policy.
B
Routing
policy
defines
just
the
config
that
needs
to
go
to
the
onward
filter,
so
the
onward
filter
can
be
created
with
that
config
now
Admiral
watches
for
it,
and
it
has
a
mapping
of
all
the
Clusters
where
a
particular
Services
dependencies
exist.
So
it's
able
to
create
the
on
my
filter
and
the
source
source
cluster
as
well.
So
this
is
how
the
routing
policy
gets
translated
to
the
on
by
filter.
Now
we
understand
so
this.
B
This
is
basically
the
Manifest
of
a
routing
policy
on
the
right
and
how
Admiral
translates
it
to
a
onward
filter
on
the
left.
If
You
observe
the
config
section
is
pretty
much
the
same.
It's
just
that
you
know
it's
rearranged
to
fit
the
on-way
filter,
spec
and
the
onward
filter
that
we
create
is
also
a
HTTP
filter
that
matches
all
the
requests
that
are
for
the
sidecar
outbound
context.
B
Now
now
we
understand
the
solution.
There
were
some
challenges
associated
with
it
that
I'll
walk
through,
so
we
use
workload
selectors,
which
are
provided
by
on
by
filter
custom
resource
to
match
the
client
workloads
where
a
particular
filter
needs
to
apply
to
so
because
the
workloads
filter
can
only
match
one
workload
as
of
now.
So
what
it
meant
for
us
is
that
if
a
given
service
has
10
different
clients
in
a
cluster,
we
have
to
create
10
different
on-way
filters,
because
the
or
operation
was
not
supported.
B
We
were
not
able
to
use
that.
We
also
use
a
tiny,
go
and
wasm
for
creating
a
Runway
filters
due
to
the
limitations
that
tinygo
has.
It
does
not
expose
full
language
feature
set.
We
could
not
use
an
internal
cache
within
the
onward
filter,
so
we
had
to
create
an
external
caching
service
to
make
the
calls
to
the
lookup
service
more
efficient.
B
Also,
the
the
logic
where
the
envoy
filter
makes
a
call
to
the
lookup
service
and,
and
then
it
also
retries,
based
on
you
know.
If
the
data
is
present
in
a
chart
or
not
that
logic
had
to
be
pre-built
in
the
onward
proxy,
we
we
did
that,
because
so
so
anytime,
we
need
to
update,
update
that
logic
right.
We
have
to
rebuild
the
onward
proxy
and
then
make
sure
the
client
workloads
are
rotated.
A
A
So
let
me
switch
the
screen
here.
So
this
is
a
kiali
dashboard.
Cali
is
an
observability
dashboard
which
shows
exactly
what
a
traffic
is
running
within
a
service
mesh
and
within
our
cluster.
So
here
you
don't
see
any
connections
because
we
don't
have
traffic
live
traffic
yet,
but
you
can
see
all
the
components
that
I
described.
We
have
all
the
the
three
shards
the
client,
the
router
and
cache
and
the
lookup
service.
So
let's
go
to
the
configuration
first
I
hope
it's
visible
to
everyone.
So
this
is
basically
the
routing
policy.
A
We
are
using
the
same
spec
what
pankaj
explained
about.
So
we
have
a
cache
section
here
and
we
have
a
host
section
here.
So
host
section
is
basically
the
URL
we
are
going
to
use
from
the
client
which
gets
overwritten
to
one
of
The,
Shard
urls
and
on
the
left,
I'm
doing
a
watch.
So
let
me
rerun
The
Watch,
Once,
More,
oops,
so
I'm
doing
a
watch
on
Automotive
filters
on
the
client
cluster
here.
A
A
So,
if
you
see
here,
I
mean
pretty
much
the
same
configuration
it's
copied
over
from
the
routing
policy
to
the
online
filter.
One
thing
you
can
notice
the
workload
selector
workload
selector
applies
only
to
the
client
here
and
not
to
any
other
workloads.
So
this
is
determined
by
Admiral
by
maintaining
a
map.
A
A
So
I'm
just
doing
a
slash
company
and
using
the
same
URL
as
as
one
we
configured
before
so
now.
First
I
will
search
for
something
in
the
9000
range
and
we
expect
the
request
to
go
to
Shard
one
and
the
lookup
service.
So
here
you
see
a
log
showing
up
in
lookup,
Service
First.
It
looks
up
the
where
the
data
for
9000
is
and
then
routes
it
to
the
short
one.
So
rerunning
the
request.
Multiple
times
you
see
it
just
goes
to
Shard
one
because
we
cache
the
data
inside
our
inside
our
service.
A
Similarly,
if
I
try
a
request
say
in
the
8000
range,
we
expect
it
to
go
to
Shard
2
and
you
see
a
a
login
lookup
Service
as
well
as
Chart
2
and
repeating
the
request
use
oops
I
just
did
it
for
it,
yeah
repeating
the
request.
You
just
see
it
going
to
chart
too.
Similarly,
I
can
also
do
a
request
on
chart
3,
which
is
a
7000
range
company
and
yeah.
You
can
see
that
showing
up
there
so
now
we
can
see
this
a
little
bit
better
in
the
kiali
dashboard.
A
So
here
you
can
see
client
talking
to
Shard
One,
Shot,
Two
and
short
three,
and
you
see
some
unknown
connection
happening
to
router
and
lookup
right
now.
Kiali
does
not
have
a
ability
to
show
onoi
filter
calls,
so
it
shows
it
up
as
unknown,
but
basically
the
filter
is
making
a
call
to
Route
router
and
lookup,
getting
the
sharp
DNS
sending
it
to
the
or
rewriting
it
for
the
client
and
the
client
communicates
to
the
multiple
shards
located
here.
A
So
now,
let's
go
back
to
the
slides.
I
just
want
to
talk
a
little
bit
about
the
future
work
we
planned
for
this.
We
want
to
also
add
client-side
rate,
limiting
using
a
similar
approach
of
creating
on
my
filter,
and
you
know
at
the
client
side
and
rate
limiting
it
there
and
the
services
can
control
that.
Also
we
wanted
to
work
with
the
istio
community
and
see
if
we
can
enhance
the
workload
selector
I
just
saw
before
we
have
to
create
one
workload
selector
per
client.
A
If
you
could
find
a
way
to
add
one
workload,
selector
per
cluster
and
apply
it
to
all
the
clients.
We
would
want
to
explore
that
approach.
Also,
we
use
tiny
go
as
our
wasm
language,
and
we
did
that
because
of
more
familiarity
with
golang.
But
tinygo
has
a
limited
feature
set
and
we
wanted
to
look
at.
Maybe
C,
plus,
plus
and
rust
to
you
know,
enhance
the
entire
solution
to.
C
D
D
A
A
E
B
So
the
sharding
logic
we
don't
need
to
you
know
for
for
our
solution.
We
don't
really
need
to
think
about
what
the
logic
is
for,
sharding
that
the
applications
are
using.
We,
as
you
know,
mesh
operators
mesh
platform
owners.
What
we
do
is
we
provide
them
the
capability
to
have
that
to
have
to
have
their
charging
logic,
reflect
in
the
lookup
service
and
from
mesh
standpoint.
We
just
make
a
call
to
the
lookup
service
and
it
Returns
the
chart.
B
That
is
there
and
it
could
be
the
sharding
logic
itself
could
be
depending
upon
use
case
right,
for
example,
Qbo
which
charge
its
data
based
on
the
company.
Ids
can
do
so
using
the
company
information
and
the
request,
and
it
can
identify
where
a
request
goes
to,
depending
upon
the
request,
context
right
and
then
another
like,
for
example,
other
teams
are
generated
that
chart
their
data
based
on
customer
location,
for
instance,
they
can
just
return.
You
know,
get
the
request,
context
based
on
the
origin
and
stuff
like
that
they
can
return.
A
F
B
Yeah
so
I
just
want
to
clarify
it's
not
like
a
drawback
of
virtual
service
itself,
but
you
know
the
kind
of
use
cases
that
we
were
trying
to
solve
for
with
virtual
service.
We
could
not
use
it
for
so
to
your
question,
do
we
use
controllers
to
update
the
virtual
Services?
As
I
mentioned,
we
already
have
Admiral
that
that
is
like
a
management
plane
for
the
sdo
control
plane.
We
already
had
that
in
place,
and
that
was
the
easiest
route
to
take
and
it
could
potentially
update
the
virtual
services.
B
A
Question
so
first
thing
on
latency,
so
initially
we
were
doing
a
lookup
for
every
request
that
definitely
added
couple
of
milliseconds
of
latency.
So
that's
when
we
introduced
our
cache,
we
still
have
subsequent
latency,
but
it
was
acceptable
for
our
solution
and
for
CPU
usage.
Yes,
we
do
encode
a
little
bit
more
whenever
you
add
a
new
Envoy
filter,
but
it
was
negligible
to
any
other
solution.
We
were
using
there.
B
The
routing
policy,
as
I
said,
the
service
owner
controls,
how
the
sharding
happens,
and
then
the
routing
policy
happens
on
the
destination
cluster,
where
the
service
is
running.
So
the
service
owner
defines
the
routing
policy,
as
they
also
dictate
the
sharding
logic.
So
that's
the
config
that
gets
translated
to
the
onward
filter.
B
Yeah
Admiral
knows
where
a
particular
services
dependencies
are
there's
another
custom
resource
that
Admiral
defines
it's
called
dependency.
That
is
how
you
know
when
a
service
gets
created,
there's
also
a
dependency
custom
resource
they
Define,
which
tells
them
what
all
clients
it
needs
to
apply
to
and
each
workload
that
is
running
in
our
multi-cluster
setup
has
an
identity
associated
with
it
and
it's
Unique
per
service.
So
Admiral
knows
what
all
clusters,
the
clients
of
that
application
are
running
in
and
that's
how
it's
able
to
you
know
create
the
envoy
filter
in
those
clusters
right.
E
A
B
A
B
H
A
I
mean
have
you
tried
one
more,
but
it
wasn't
really
for
this
use
case.
It
was
the
authent
based
onward
filter,
but
it
wasn't
really
for
this
use
case,
but
it
you
can
do
an
external
call
to
an
authentication
Service
and
get
and
get
it
back,
but
that
didn't
give
us
that
doesn't
give
you
a
metadata
back.
It
just
gives
you
a
200,
okay
and
forwards
the
request,
so
that
kind
of
didn't
work
for
our
favor.
So
yeah,
that's
right.
A
I
I
I
A
Yeah
using
I
mean
a
very
good
question
so
for
this
solution,
right
I'm,
not
sure
if
this
is
what
you
want
to
hear,
but
for
this
solution
we
pre-built
the
envoy
proxy.
We
we
have
a.
We
just
do
a
copy
of
the
awesome
binary
into
the
envoy
proxy
image
and
we
just
reference
it
in
our
Envoy
filter
yeah.
So
this
is
the
the
reason
for
this
is
we
used
1.10
and
Below
istio
when
we
designed
the
solution-
and
that
was
the
approach
we
took
but
wasn't
plugins
would
give
you
that
opportunity
right.