►
From YouTube: xds-relay: Performance initiatives for control plane management - Jessica Yuen, Jyoti Mahapatra
Description
xds-relay: Performance initiatives for control plane management - Jessica Yuen, Jyoti Mahapatra
In this talk, presenters will share their experience running Envoy and Lyft’s control plane at scale. They will explore the challenges of operating Lyft’s service mesh to be reactive to Kubernetes’ dynamic infrastructure and evolving xDS versions. This talk is a deep dive into a new open source project, xds-relay, that the Lyft team has developed to bring their solutions to the greater community. xds-relay is a lightweight caching, aggregation, and low latency distribution layer for xDS compliant clients and servers. At scale, xds-relay reliably distributes xDS protos to thousands of xDS clients over gRPC. Join Lyft’s journey as the presenters share how Lyft envisions the future of control planes. The presenters will cover a range of topics including pluggable xDS transformations, automatic endpoint subsetting, API driven configurations, and State-of-the-world to Delta xDS conversion.
A
A
I'm
jyoti
and
I'm
here
with
just
today,
we
both
work
in
the
networking
team
at
lyft.
We
are
the
maintainers
of
the
xts
relay
project,
as
well
as
the
new
maintenance
on
the
go
control
plane.
Our
contact
information
is
mentioned
on
the
slide
and
would
be
very
happy
to
connect
and
engage
further
after
the
talk.
A
Everything
has
a
history
and
we
are
not
different.
The
networking
team
at
left
has
been
maintaining
the
lift
edge
and
service
mesh
over
numerous
years.
The
lift
compute
architecture
has
evolved
from
vm
based
asgs
to
kubernetes
initial
adapters
of
kubernetes
slowly
tripled
into
the
match.
As
the
platform
became
more
robust,
the
adoption
became
more
widespread
in
order
to
maintain
one
way
of
doing
things.
The
leadership
also
made
a
mandate
to
move
all
services
to
communities
by
the
end
of
williamson.
A
A
A
A
Briefly
describe
a
bit
if
about
the
before
and
after
model
of
the
of
how
service
discovery
works
left,
we
have
a
homegrown
control
plane
based
on
the
door.
Control
plane
subscribes
to
endpoint
updates
from
vm
based
discovery,
service
and
kubernetes
api
server.
It
also
subscribes
to
cluster
updates
from
a
relatively
low
flux,
s3
files,
the
control
plane,
is
connected
to
the
on-voice
height
cards
via
grpc.
A
A
A
A
A
The
interesting
part
starts
next,
it
is
a
control,
plane's
responsibility
to
understand
the
update
pack,
the
information
in
the
grpc
object
and
send
that
across
the
dependent
sidecars,
this
process
needs
a
grpc
payload
to
be
serialized
into
network
bytes
and
sent
serialization
is
a
cpu
intensive
operation,
and
we
noticed
that
as
more
and
more
services
adopted
communities,
the
control
plane
cpu
was
through
the
roof.
When
everything
scaled
up
and
down
the
scaled
down
characteristics
is
very
similar
to
scalar
and
suffers
from
the
same
symptoms.
A
A
A
One
of
the
first
approach
was
to
move
the
control
plane.
Infrastructure
took
abilities
so
that
it
could
scale
quickly
and
proportionally
with
other
services.
We
could
also
scale
up
or
pre-scale
the
existing
legacy
control
plane.
Before
known
events,
control,
plane,
parts
were
deployed
in
the
two-way
disk
clusters,
so
they
could
scale
proportionally.
A
We
changed
the
bokeh
channels
in
the
go
controls
name
from
unbuffered
to
buffer
and
buffer
channels
of
length,
one
for
the
state
of
the
world
eds.
The
last
update
wins
if
one
of
the
sidecars
network
was
getting
flow
control
and
the
pro
control
plane's
channel
was
blocked
from
making
progress.
We
could
keep
the
keep
overriding.
The
latest
update
in
the
buffer
channel
to
maintain
correctness.
A
We
performed
flame
graph
analysis
and
fixed
a
few
wasteful
relationships
both
in
go
control,
plane,
library
and
in
our
own
private
control.
Plane
instantaneous
in
point
update,
was
cost
for
everything,
so
we
make
sure
that
the
rate
limit
and
batch
kubernetes
end
point
updates
to
a
configurable
interval
say
tens
of
seconds
envoy
is
eventually
consistent,
so
this
worked
out
fine
for
now,
although
we
slowed
down
the
membership
convergence
and
made
the
control
plan
response
times
flow.
A
B
B
B
Lastly,
we
want
to
build
xcs
relay
in
a
way
that
the
components
operate
in
a
plug-and-play
manner,
we'll
get
into
a
lot
of
ambitious
goals.
We
have
surrounding
a
general
relaying
type
component
later,
but
we
understand
that
a
feature
rich
set
can
be
bloated
for
a
company
that
wants
to
just
run
lightweight
components.
B
B
So,
on
the
slide
you
can
see
the
workflows
numbered
one
all
the
way
to
six
when
discovery
requests
first
make
their
way
into
xcs
relay
they
go
through
an
aggregator
component.
This
is
the
component
that
takes
the
rules
we
mentioned
in
the
previous
slide
and
translates
these
requests
into
unique
discovery
keys.
B
So
at
lyft
we
run
a
group
of
control
plane
servers
on
each
kubernetes
cluster,
with
services
distributed
across
each
of
the
clusters
and
our
control
plane.
Cluster
was
scaled
up
to
run
on
multiple
nodes.
In
order
to
support
the
number
of
online
clients
we
had
at
lyft
with
this
new
architecture,
we're
running
a
group
of
excess
relays
on
each
kubernetes
cluster,
allowing
us
to
scale
down
our
control,
plane,
server
and
refocus
the
control,
plane
core
logic
on
pre-processing
and
generating
new
xcs
responses.
B
B
So
lyft
runs
all
of
its
infrastructure
on
a
giant
vpc,
but
we
envision
other
interesting
topologies
for
running
axis,
relay
on
hybrid
clouds
and
multiple
vpc
infrastructure
setups.
So,
for
example,
here's
a
topology
where
we
run
a
cluster
of
xts
relay
on-prem.
So
on
this
data
center
in
another
data
center,
physically
close
to
the
onboard
services
while
hosting
the
control
plane
server
on
a
centralized
vpc
for
faster
services,
discovery.
B
B
B
So
we
have
two
bootstrap
files
here
for
the
onward
clients
and
they're.
Quite
simple:
one
of
them
has
the
node
id
on
the
client
one.
The
other
one
has
the
node
id
over
client
two.
They
both
run
on
slightly
different
ports.
One
runs
on
port
nineteen
thousand,
the
other
one
runs
on
port
9001,
but
everything
else
is
the
same.
B
So
it's
going
to
end
up
generating
keys
that
look
like
staging
underscore,
eds
or
staging
underscore
cds
and
how
that
works
is
via
these
fragments.
So
for
the
first
fragments
we
look
at
these
request,
types
that
fall
under
lds,
cds,
eds
or
rds,
and
we
apply
this
regex
match
and
replace
operation
on
the
node
cluster.
B
So
because,
in
our
armor
bootstrap
files
we
have
the
node
cluster
set
to
staging.
This
will
actually
always
result
in
staging,
as
the
first
cache
key
fragment
for
the
second
fragment.
It's
just
a
static
constant.
So
if
it's
of
type
listener,
then
we
append
lds
as
the
second
string
fragment
with
cds.
B
B
So,
as
you
can
see,
the
response
contains
the
version
of
the
latest
snapshot
that
was
generated
by
the
server.
In
this
case,
it's
95
197
more
seconds
have
passed
since
I've
ran
this
query.
So
that's
why
we're
not
seeing
the
latest,
but
we
can
also
see
that
both
onward
clients
are
cash
in
the
requests.
B
So
since
these
clients
fall
under
the
same
aggregation
rule,
if
we
were
to
observe
the
stats,
we'd
also
know
that
there's
only
one
grpc
stream
being
made
to
our
upstream
server.
Despite
there
being
two
onward
clients,
we
can
also
validate
that
the
online
clients
have
received
valid
cds
information
by
querying
on
voice
admin,
endpoint.
B
B
You
can
see
it
very
slightly
because
we've
generated
a
new
snapshot
since,
but
if
we
quickly
run
both
in
sequence,
you
notice
that
their
cluster
information
is
the
same
and
this
example
is
available
on
our
github.
If
you
would
like
to
try
it
out
for
yourself,
please
let
us
know
your
thoughts
thanks.
B
B
Another
one
that
we're
particularly
excited
about
is
endpoint
subsetting,
so
in
a
topology
that
we
mentioned
earlier,
it
might
not
be
a
deal
for
xcs
clients
to
be
aware
of
clients
running
in
a
different
vpc
or
data
center,
similar
to
the
aggregation
rules.
We're
looking
at
creating
rule
based
configuration
where
operators
can
use
xcs
relay
to
send
back
a
subset
of
the
eds
information
rather
than
all
endpoints
that
exist
from
the
control
plane.
Response.
B
B
And
this
list
goes
on
in
the
interest
of
time.
We
won't
be
able
to
cover
them
all
today,
but,
as
you
can
see,
there's
plenty
of
directions
that
we
could
and
want
to
take
this
project.
So
we're
always
looking
for
contributors
and
love
talking
to
people
curious
about
control,
plan,
implementations
and
their
use
cases
please
reach
out
to
one
of
us
directly
or
visit
the
envoy
slack
here.
B
B
B
B
I
think
the
part
that
we
want
to
implement
in
exclusive
relay
is
if
it
can
be
defined
based
on
rules
that
anyone
might
have
to
live
in
xts
relay.
But
if
there
is
too
much
custom
logic
that
should
be
generated
from
service
metadata
based
on
your
company's
logic,
then
it
should
fall
in
the
control
plane.
A
A
Yeah
in
case
the
control
plane
is
not
present
momentarily,
we'll
always
continue
to
receive
configuration
the
htc
delay.
All
this
depends
on
the
upstream,
but
if
the
upstream
is
not
present,
we
have
pre-tries
and
back
off
mechanisms
built
in,
so
that
we'll
keep
trying
until
the
controller
comes
back,
but
it
will
keep
serving
contribution
from
the
cash
to
the
sidecar,
so
the
fleet
will
continue
operating
and
there
will
be
no
disruption
until
the
back
in
the
total
comes.
A
B
B
A
In
case
the
back
end
control
plane
comes
back,
will
it
invaded
the
caching
delay?
So
yes,
when
the
back-end
control
thing
comes
back,
it
will
be
a
new
stream
from
exchange
delay
to
the
back-end
control
plane,
as
if
everything
is
fresh.
So
whatever
the
version
updated
version
there
is,
it
will
reflect
in
the.