►
Description
xDS transport and versioning evolution - Harvey Tuch, Mark D. Roth
Envoy’s xDS APIs are the foundation for its control plane ecosystem. We are in the process of evolving them towards the Universal Data Plane API (UDPA), supporting clients beyond Envoy (e.g. Google’s gRPC libraries). We also continue to improve support for versioning in xDS and are following on from last year's introduction of major versions with minor/patch versions.
In the first part of this talk, we will dive into UDPA. We will focus on the next steps in the xDS transport protocol evolution. We will provide an introduction to a new URI-centric resource naming scheme and how this will allow for transport simplifications and elimination of technical debt in both Envoy and the control plane. We will also cover advanced use cases, such as federation, caching, control plane scalability and reliability wins.
In the second part of this talk, we will provide a recap on Envoy's existing API versioning story and discuss the implementation of minor/patch versioning for xDS resources. This incremental strategy is the plan-of-record for xDS, managing the trade-off between Envoy/xDS technical debt and control plane complexity/implementation cost.
A
Hello
oneboycon,
I'm
harvey
tuch
and
I
work
at
google.
I
I'm
the
tl
of
the
envoy
platform
team
and
today
I'm
going
to
be
co-presenting
with
mark
roth,
a
colleague
of
mine
who
leads
the
grpc
c,
plus
efforts
at
google
and
we're
going
to
be
talking
about
the
future
of
xds,
in
particular,
some
of
the
changes
around
versioning
and
xts
transport,
which
are
going
to
be
landing
over
the
coming
year.
A
A
All
we
had
was
polling
based
rest
for
xds
delivery
and
that
had
significant
issues
with
performance
on
control,
plane,
scalability
and
also
for
latency
of
updates
and
the
versions
xds
transport
for
essentially
a
version
pub
sub
scheme
which
we
introduced
since
then
has
brought
about
huge
improvements,
and
it's
working
incredibly
well
today,
and
so
we
want
to
evolve
this
forward
incrementally
without
breaking
the
world
and
without
affecting
anyone
who's
using
it
today.
But
at
the
same
time
we
want
to
address
some
of
these
limitations,
which
I'm
going
to
talk
about.
A
So
what
are
these
limitations?
So?
The
first
of
these
is
that
there's
actually
issues
around
cash
ability
of
xds,
and
so
this
impacts
our
xcss.
We
want
to
scale
it
up
to
many
many
different
endpoints,
and
this
could
be,
for
example,
an
onboard
mobile
as
depicted
on
this
slide,
where
we
might
have
millions
or
tens
or
hundreds
of
millions
of
endpoints,
and
we
need
a
good
story
for
how
xts
resources
can
be
distributed
and
fanned
out
and
there's.
A
This
project
called
xds
relay,
which
it's
just
checking
out,
which
basically
is
building
the
first
steps
of
this.
But
to
do
this
kind
of
caching
and
relay
there
are
other
issues
and
let's
go
through
some
of
these.
The
first
is
that
the
xcs
resource
things
that
exist
today
are
just
opaque
strings
and
they're,
not
really
a
unique
cache
key,
because
they
don't
include
a
bunch
of
useful
contextual
information
which
includes
things
like
node
id
and
metadata,
which
are
actually
provided
in
the
xds
stream,
but
out
of
band
with
the
xts
resource
name.
A
A
The
next
major
sort
of
next-gen
use
case
is
federation,
and
this
is
when
you
want
to
have
multiple
control
planes
managing
xds,
and
today
we
have
some
limited
supports
for
having
more
than
one
control
plane
with
envoy.
But
it's
at
a
very
coarse
granularity
and
there's.
You
can't
like,
for
example,
delegate
a
single
route
configuration
to
a
different
server
and
this
kind
of
thing
so
in
this
world
we
want
to
think
about.
Well,
we've
got
multiple
xds
servers.
This
might
be
in
different
clusters
might
be
in
different
clouds
might
be
on
premise
and
so
on.
A
So
today,
as
I
mentioned
before,
we
kind
of
lack
some
of
the
support.
We
have
no
explicit
notion
of
what
an
authority
is
and
sort
of
what
a
node
in
this
graph.
That's
where
a
drawing
looks
like
resource
names-
I
don't
know
they
are
opaque
and
non-cachable,
but
they're
also
global
and
there's
no
way
to
qualify
them
by
specific
authorities
or
have
each
authority
manage
its
own
resource
name,
space,
there's
no
support
for
redirections
or
failover
alternatives,
and
our
existing
conflict
sources
are
pretty
coarse-grained.
A
A
But
as
we've
gone
to
sort
of
stretch
and
test
the
parameters
of
xds
transport
and
its
use,
it
became
apparent
that
there
really
are
issues
here
which
need
some
mitigation.
And
then
we
want
to
do
this.
To
simplify
control
planes,
make
things
easier
to
implement,
make
it
more
robust
and
reliable,
and
there
are
all
these
surprises.
I
mentioned
before
the
weird
treatment
of
collections
for
lds
and
cds.
There's
also
the
fact
that
we
have
multiple
wire
protocols
effectively
now
for
state
of
the
world
and
delta.
A
We
have
this
for
issue
with
cylindrical
aliases,
which
we
usually
do,
especially
with
the
hds.
So
our
proposal
and
I
have
links
to
the
proposal
in
the
final
slide
in
this
talk.
It
goes
about
systematically
addressing
all
of
these
and
we're
gonna
hand
it
things
over
now
to
mark
who's,
gonna
talk
about
one
of
the
core
concepts
which
really
is
fundamental
to
this,
and
that
is
an
idea
of
a
structured
resource
name
which
takes
the
form
of
a
url
and
has
this
udpa
schema
so
I'll
have
things
over
to
mark.
At
this
point.
B
So
the
the
key
to
addressing
the
limitations
that
harvey
spoke
about
is
the
this
new
naming
scheme
for
xcs
resources
in
which
resource
names
are
represented
as
a
udpa
uri.
So
so,
let's
talk
about
the
anatomy
of
a
udpa
uri.
The
the
first
part
of
the
uri.
That's
interesting
is
the
authority.
This
indicates
who's
authoritative
for
the
resource,
which
is
not
necessarily
the
same
as
the
xds
server.
That's
used
by
the
client.
B
If
you've
got
for
scalability
reasons,
you
might
have
sort
of
a
a
local
caching
xts
proxy.
You
know
in
each
data
center
that
that
has
clients
and
you
might
have
those
clients
access
the
data
through
the
local
caching
proxy,
even
though
the
authority
is
still
some,
you
know
centralized
single
point,
that
is,
that
is
centralized
across
the
entire.
You
know
global
infrastructure,
the
authority
acts
as
a
global
name
space
for
resource
names.
B
So
you
know
if
you
have
resources
in
you
know,
if
you
have
an
authority
that
has
a
bunch
of
resources
in
it,
you
can
feel
free
to
organize
them.
However,
you
want
and
you're
not
going
to
have
any
naming
conflicts
with
anything
that
anyone
any
resources
that
anyone
has
created
in
in
other
authorities.
B
The
authority
is
an
optional
part
of
the
uri.
So
if
you
have
a
non-federating
use
case,
just
a
you
know
a
bunch
of
local
servers
that
don't
interact
with
anything
else.
You
can
just
omit
the
authority
and
have
everything
in
sort
of
the
you
know
the
empty
string
as
the
authority
name
in
the
future.
The
authority
might
be
used
to
authenticate
the
resources
themselves
using
some
sort
of
signing
mechanism.
That's
not
something
we
actually
have
today,
but
it's
a
possible
future
direction
that
we
could
go
in
here.
B
The
next
part
of
the
uri
is
the
resource
type,
which
is
you
know,
listener
cluster
route
configuration
that
sort
of
thing
the
id
is
is
essentially
the
path
part
of
the
uri.
It
can
be
any
string
that
you
want
it's
totally
up
to
you
as
an
authority
owner
as
to
how
you
want
to
lay
out
your
your
resource
names
context.
Params
are
similar
to
query
params
in
an
http
uri.
B
They
provide
a
way
to
serve
multiple
variants
of
the
same
resource
and
we'll
talk
more
about
that
in
a
minute
and
the
final
part
of
the
uri
is
the
directive.
This
is
similar
to
the
fragment
in
an
http
uri.
It's
a
a
directive
for
how
the
client
should
interpret
or
use
some
part
of
the
resource,
but
it's
never
actually
sent
to
the
server
it's
only
interpreted
on
the
on
the
client
next
slide.
B
Please.
So
here
are
some
examples
of
udpa
uris.
First,
we
have
an
example
of
a
fairly
basic
uri.
It's
got
authority,
xds.example.com
and
so
envoy
would
look
in
its
bootstrap
file
to
determine
which
server
to
query
for
this
authority.
The
resource
type
here
is
listener
and
the
id
is
service
sidecar
and
again,
this
can
be
any
path
that
you
want.
This
is
just
sort
of
you
know
one
arbitrary
example
of
what
you
could
put
in
here.
B
The
second
example
shows
the
use
of
context
params.
In
this
case,
the
context
param
is
node
type
equals
front
end.
Note
that
the
context
params
are
actually
part
of
the
resource
name
they're
part
of
the
unique
identity
of
the
resource
so
from
the
xds
perspective,
two
resource
names
that
vary
only
by
content
params.
If,
for
example,
you
had
another
one
that
was
node
type
equals
backend
from
the
xds
perspective,
that's
actually
a
different
resource
from
the
human
perspective
in
practice.
B
Usually,
if
you,
if
you
have
a
resource
that
that
varies
only
based
on
the
context,
params,
that's
usually
two
variants
of
the
same.
You
know
the
same
basic
resource
that
it's
telling
the
client
to
do
the
same
thing,
but
in
slightly
different
ways
for
slightly
different
scenarios
context
frames
do
actually
come
from
different
places,
not
just
from
the
ui
uri.
There
are
some
additional
sources
of
params
that
can
get
added
on
when
requests
are
made
to
the
grp
to
the
xcs
server.
Excuse
me
and
we'll
talk
more
about
that
in
a
minute.
B
The
final
example
here
shows
use
of
a
directive.
In
this
case.
There's
a
directive
called
alt
which
specifies
a
fallback
resource
so
when
the
primary
resource
is
doesn't
exist
or
the
server
can't
be
reached,
the
client
can
sort
of
know
to
fall
back
to
this
alternative
resource
next
slide
all
right.
So
let's
talk
more
about
context
parameters,
as
mentioned
earlier.
The
purpose
is
to
allow
servers
to
provide
multiple
variants
of
the
same
resource.
This
is
the
key
to
making
all
kinds
of
information
a
first
class
part
of
the
cache
key
for
resources.
B
B
This
is
a
key
part
of
replacing
the
wildcard
queries
in
lds
and
cds
that
harvey
mentioned
earlier.
The
idea
here
is
to
use
these
context
params
to
encode
things
that
used
to
live
in
the
node
message
only
so,
instead
of
the
server
looking
at
the
client's
node
identity
to
decide
which
resources
to
return,
the
client
can
explicitly
ask
for
what
it
wants
in
a
flexible
way.
B
Next,
the
the
second
source
of
of
context
parameters
are
the
ones
encoded
in
the
uri
themselves.
So
this
is
like
the
example
I
showed
earlier
of
node
type
equals
front
end.
It
can
have
any
prefix,
there's
no,
no
restrictions
on
on
what
the
the
content
is
and
it
is
applied
after
the
node
identity
param,
so
it
can
actually
override
them.
So
you
know
you
could
actually
specify
something
in
the
uri,
that's
udpa.node.something
and
it
would
override
whatever
comes
from
the
bootstrap
file.
B
B
These
client
features
are
resource
specific,
so
a
client
feature
that
that
is
specific
to,
for
example,
eds
will
only
be
added
to
requests
for
eds
resources,
and
this
prevents
unnecessarily
polluting
caches,
with
duplicate
copies
of
the
same
resource,
which
is
what
would
occur
if
you
know
you
know
if
the
same
cds
resource
was
accessed
by
two
different
clients
that
had
different
values
for
a
client
feature
that
is
relevant
only
to
eds
right.
You
wouldn't
want
to
have
to
make
two
copies
of
that.
B
So
this
avoids
that
next,
the
final
source
of
context
parameters
is
per
resource
type
attributes
these
use
the
prefix
udp
dot
resource
just
like
client
features.
These
are
not
user
defined,
they're
added
automatically
by
the
implementation
and
they're
defined
for
each
resource
type
as
needed.
The
one
concrete
example
here
is
for
vhds.
We
plan
to
use
this
to
allow
the
client
to
to
request
the
specific
virtual
host
that
the
client
wants.
B
B
A
list
collection
is
a
resource
that
contains
a
collection
of
resources
of
a
particular
type.
The
collection
resource
itself
will
have
its
own
type.
So,
for
example,
a
resource
of
type
listener
collection
is
a
resource
that
contains
a
collection
of
listeners.
A
resource
of
type
cluster
collection
is
a
resource
that
contains
a
collection
of
clusters,
as
indicated
by
those
examples.
This
is
another
key
part
of
replacing
the
wildcard
queries
for
lds
and
cds.
B
The
other
type
of
collections
that
we're
going
to
support
are
glob
collections.
These
are
an
alternative
to
list
collections
for
cases
that
need
additional
scalability.
So
the
way
this
works
is
the
client
requests
a
resource
where
the
last
component
of
the
id
is
star
and
it
works.
Basically
like
a
simple
shell
glob
pattern,
the
server
will
return
all
resources
in
this
quote
unquote
directory
the
client.
Will
then
know
what
resources
in
the
response
match
the
this
request
by
virtue
of
them
all
being
in
this
requested
directory?
B
B
B
Next,
let's
talk
about
different
ways
that
you
can
use
udpa
uris
to
address
various
federation
scenarios.
There
are
three
different
ways
for
a
resource
from
one
authority
to
sort
of
delegate
to
resources
in
another
authority.
First,
the
delegation
can
happen
at
the
normal
handoff
point
between
resources.
You
know
we
know
that
listener
resources
can
tell
the
client
what
route
configuration
to
request
via
rds,
so
that
rds
resource
can
actually
be
in
a
different
resource.
Sorry,
a
different
authority
than
the
listener
resource
was
so
that's
one
way
of
sort
of
handing
off
next.
B
The
second
type
of
authority
delegation
is
redirection,
so
this
is
similar
to
an
http.
Redirect
the
client
will
request
a
specific
resource
and
the
server
then
tells
it
to
use
a
different
resource
instead
and
that
resource
can
be
in
a
different
authority.
So
this
is
a
way
for
one
authority
to
sort
of
say
I
want
to
delegate
you
know
this
particular
resource
to
some
other
authority
next.
So
the
final
form
of
authority
delegation
is
having
a
list
collection
that
includes
references
to
resources
in
a
different
authority.
B
So
in
this
example,
we
have
a
listener
collection
in
the
authority
xds.example.com,
but
it's
redirecting
inside
of
it.
It's
referencing
inside
of
it
resources
in
other.com
and
mumble.com
next
slide.
Please
now
a
different
type
of
authority
handoff
is
a
failover.
This
is
where
we
use
that
alt
directive
that
we
talked
about
earlier,
but
in
this
case
the
alt
directive
points
to
a
resource
in
a
different
authority
than
the
original
resource
was.
B
A
A
We've
got
better
support
for
collections,
we've
eliminated
a
bunch
of
technical
debt
and
we're
now
ready
to
go,
I
think
for
things
like
federation,
so
this
is
like
all
pretty
exciting
now
in
terms
of
the
implementation
roadmap.
We've
just
started
to
do
this.
We
have
the
first
three,
I
guess
items
in
that
roadmap
underway,
but
at
very
early
stages.
We
planning
q4
landing,
probably
one
through
four,
at
least
in
this
list,
and
then
I
think
once
we've
landed,
that
which
is
basically
support
for
the
the
core,
udp
urls
and
glob
collections.
A
In
the
second
part
of
this
talk,
I
plan
on
looking
at
xts
versioning.
I
know
this
has
been
a
source
of
considerable
friction
in
the
envelope
community
in
the
past
year.
So
I'm
hoping
with
the
supply
that
I
present
has
sort
of
reflected
some
of
the
feedback
that
we've
received
and
is
providing
us
path
forward
to
achieving
the
best
properties
of
versioning
while
still
being
thoughtful
and
mindful
of
the
costs
that
control
point
operators
bear
when
I'm
implementing
some
of
these
schemes.
A
We
decided
to
stop
the
clock
right
there
because
it
turned
out
that
there
is
just
too
much
control
playing
operator
developer
pain
which
I'll
go
into.
I
have
a
mini
retrospective
in
the
next
few
slides,
but
that's
essentially
where
the
situation
we're
in
today
we're
not
getting
rid
of
major
versions,
but
we're
not
going
to
issue
one
per
year
and
we're
going
to
come
up
with
a
more
incremental
approach
towards
versioning
which
I'll
explore
in
the
next
few
slides.
A
So
a
quick
retrospective,
the
good,
the
bad,
the
ugly
well
good
things
is
major
version
did
solve
our
breaking
change
problem
within
major
versions.
We
built
sophisticated
tooling
in
onboard
to
automate
most
of
the
upgrades,
so
it
was
a
lot
smoother
than
it
would
have
been
if
we
did
it
manually
and
there
were
a
bunch
of
technical
improvements
in
v3
over
v2,
which
were
made
possible
by
the
ability
to
break
the
api
over
the
version
bump
what
worked
badly?
Well,
we
had
a
fair
bit
of
code
chain.
A
That's
not
too
bad,
though
we
didn't
finish
basil
integration.
So
there's
this
generated
api
shadow
thing
that
you
may
be
familiar
with.
If
you're
a
onboard
developer,
you
need
to
unfix
format,
for
we
didn't
have
at
all
parts
of
v3
ready
to
go
on
day
zero
and
there's
some
complexity
around
handling
versioning
in
envoy.
Now
the
really
ugly
parts
are
control
playing
developers
and
operators
had
a
lot
of
pain
and
they
didn't
have
the
same
tool
in
libraries
that
we
had
in
envoy.
A
A
Okay,
now?
What
are
the
implications?
Well,
if
you're,
a
control,
plane
operator,
you're
gonna,
have
to
state
a
policy
and
support
one
around
which
minor
versions
you
support.
If
you're
a
control,
plane
developer,
you
need
to
support
a
range
of
minor
versions
and
support
negotiation
of
features
and
version.
Xcs
client
developers
need
to
pretty
much
do
the
same,
and
anyone
who
wants
to
is
willing
to
put
in
the
work
of
being
an
api
shepherd
can
join
in
the
process
of
deciding
you
know
what
gets
deprecated
and
when
to
help
guide
these.
A
A
B
Hello,
hello,
all
right,
so
should
we
tackle
the
one
about
the
collections,
including
collections
like,
I
think,
that's
something
we
could
support.
I'm
not
sure
we'd
have
to
look
at
the
use
cases
and
make
sure
there's
no
implications,
but
it
seems
on
on
just
first
thinking
about
it.
It
sort
of
seems
like
we
could
do
that.
A
Yeah
for
list
collections,
sure
I
mean
we
already
have
the
xds
client
going
back
and
forth
to
the
server
multiple
times
when
fetching
a
collection,
and
so
this
is
essentially
just
adding
a
bit
of
hierarchy
there.
So
there's
no
reason
we
can't
support
it.
I
guess
we
will
just
need
to
understand
the
use
case
for
that.
First,
because
it's
I
mean
like
looking
at
the
existing
proposal.
We
already
have
a
fair
bit
of
conceptual
complexity,
around
collections
around
where
these
list
collections,
and
we
also
need
these
glob
collections
for
performance
and
scalability.
B
B
A
A
It
seems
that
would
be
a
good
place
to
also
attach
certificate
information
as
well,
and
the
xcs
resource
objects
themselves
are
now
in
a
wrapper
object,
and
that
is
where
essentially,
the
signature
would
go,
but
this
scheme
has
not
been
fully
designed
yet
we're
it's
really
just
the
stage
at
which
we
understand
how
you
would
probably
go
about
building
this
and,
I
think,
like
anyone's
actually
interested
in
helping
to
drive.
That
would
be
very
into
them,
because
I
think
that's
a
really
important
part
of
making
this
work
for
real.
B
A
A
Goes
back
like
the
original
point
of
this
idea,
like
a
disaggregated
control
plane,
so
today,
control
planes
are
like
it's
a
single
or
more.
I
think
service
is
probably
fine,
while
they're
simple,
but
you
can
think
about.
You
know
breaking
up
the
control
plane
and
even
amongst
those
who,
let's
say
we're
fronting
that
control
plane
with
go
control
plane.
They
probably
have
a
pretty
sophisticated
configuration
pipeline
behind
that
which
runs
in
a
different
service
and
that
kind
of
thing.