►
From YouTube: Lightning Talk: It’s 10pm, do you know what your proxy is doing? Understanding... - Isaac Diamond
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Lightning Talk: It’s 10pm, do you know what your proxy is doing? Understanding the xDS Protocol! - Isaac Diamond, Stripe
The Envoy xDS protocol forms the backbone of the Envoy v2 discovery APIs but understanding the the details can be pretty challenging. Remembering the differences between the CDS, LDS, EDS, and RDS is tough, let alone figuring out how they work under the hood! The documentation for the xDS protocol can be hard to understand for beginners and there are a few subtleties to consider while building an implementation. This talk explores the design of the xDS protocol through the lens of our experience building a management server implementation. Over the course of this talk we’ll cover the basics of the lifecycle of xDS gRPC connections and the way configuration updates are delivered to Envoy. Attendees will hopefully come away with a clearer understanding of the internals of the protocol that underlies Isito and Envoy’s Go Control Plane and will be equipped with an understanding of where to look to start building xDS implementations themselves!
A
Well,
everyone
it's
10:00
p.m.
do
you
know
your
proxy
is
doing
understanding
the
xes
protocol.
This
is
gonna,
be
a
talk
about
some
of
our
buoys
dynamic
configuration
mechanism
and
a
little
bit
about
how
we
use
if
it's
right,
hello,
everyone,
my
name
is
Isaac
I
work
at
stripe
on,
what's
called
the
orchestration
team,
so
we're
we're
responsible
for
the
kubernetes
deployment
that
we've
been
building
out,
as
well
as
our
envoy
service
mesh.
A
We're
really
excited
about
on
the
way
for
a
mutual
TLS,
we're
putting
it
between
effectively
every
service
at
stripe,
with
the
eventual
goal
of
securing
all
on
the
routes
between
them
for
security
and
reliability.
We
have
several
thousand
nodes
right
now,
which
we
all
manage
with
a
custom
management
server
that
we
I
wrote
in
about
5,000
ish
lines
ago.
So
it's
not
that
much
code,
but
it's
you
know
pretty
significant
and
then
finally,
we're
hiring.
A
So
if
you're
interested
in
work
at
stripe
as
to
mean
email
company
a
bit
of
terminology
before
we
dive
in
to
the
way
the
XTS
protocol
actually
works,
envoy
has
a
number
of
listeners
associated
with
it.
This
is
a
place
where
Anglais
can
listen
for
connections
and
then
proxy
requests
through
each
of
these
listeners
has
a
set
of
routes.
A
route
is
a
mapping
of
requests,
sort
of
metadata
to
a
location
that
we
should
send
traffic
each
of
those
locations
that
amelie
should
send.
A
Each
of
these
things
can
be
configured
dynamically
yeah.
What's
called
the
discovery
service
in
this
diagram,
I'm
blaze
talking
to
a
customer
discovery
service
or
a
discovery
service
which
I'll
just
refer
to
is
already
SCDS.
Yes,
yes,
these
things
are
all
modular.
You
can
kind
of
mix
and
match
them
and
build
the
many
different
binaries
run
them
in
different
processes,
run
them
across
different
machines,
which
is
what
we
do,
but
they
all
speak.
The
XDS
protocol,
which
just
the
X
stands
in
for
the
specific
resource.
A
It's
a
push
based
protocol
based
on
your
PC,
which
means
that
it's
sort
of
more
challenging
constructs
invalid
configuration
or
configuration.
That
is
totally
impossible
by
envoy
and
then
finally,
there's
an
acknowledgement
mechanism
built
right
into
the
protocol,
meaning
that
you
can
determine
with
sort
of
doing
the
right
thing
as
you
expected.
So,
let's
type
in
to
kind
of
what
one
of
these
streams
looks
like
this
is
a
timing
diagram
on
the
left
hand,
side.
We
have
envoy
the
right
hand
side,
we
have
a
management
server.
A
Let's
assume
that
this
is
a
cluster
discovery
and
went
on
boots
up.
It
makes
a
discovery
request
to
the
management
server,
letting
it
know
that
it
exists
and
providing
I
mean
metadata
that
it
needs
to
determine
the
right
clusters
to
send
back.
The
management
server
constructs
the
discovery
response
with
the
set
of
clusters
that
it
wants
envoy
to
know
about,
and
a
version
number
associated
with
it
and
then
passes
this
down
the
chair,
PC
stream,
and
way
sort
of
updates
its
internal
state
to
the
version
number
that
the
management
server
has
passed.
A
This
allows
a
management
server
to
know
that
everything
is
working
as
expected
and
it's
received
and
the
Envoy
on
the
other
end
has
received
the
the
discovery
response
that
the
management
three-percent
later
based
on
changes
in
infrastructure
or
just
after
a
time
period.
However,
you
construct
your
management
server.
The
management
server
can
spontaneously
send
an
update
to
envoy
letting
it
know
of
a
change
in
customer
information
or
a
new
cluster
is
been
added
to
do
this.
A
This
is
sort
of
a
happy
path
when
everything
goes
right,
but
it
can
also
go
wrong.
In
this
case.
We've
double
included
a
cluster,
so
cluster
1
is
listed
twice
in
Discovery
response.
I'm
boy
doesn't
really
know
how
to
process
this,
because
if
you
like
make
a
request
to
cluster
1,
it
won't
know
where
it's
in
that
traffic,
so
I'm
going
to
reject
receipt
of
this
discovery.
Response
I'd
respond
with
the
previous
version
number
you're
back
to
the
management
server.
A
This
indicates
to
the
management
server.
Then
it's
not
like
processable
had
also
logged
out
and
I
store
message,
a
stripe.
We
page
on
these
events,
because
it
means
that
envoy
is
probably
not
running
traffic
and
the
way
that
we
expect,
and
it
means
that
we
miss
configuring,
the
management
server.
So
we
want
someone
to
like
wake
up
and
take
a
look
at.
Oh
no
I
have
more
sites
the
XP
essence.
Modularity
can
also
lead
to
some
race
conditions.
If
you
run
the
discovery
services
across
different
processes
or
across
different
hosts
in
distributed
fashion,.
A
A
Envoy
in
this
case,
boots
up
and
makes
discovery,
requests
to
the
employee
discovery
service
and
the
CTS
simultaneously,
but
the
end
boy
discovery
service
responds
first
letting
envoy
know
that
some
cluster
of
maps
trigger
an
endpoint.
In
this
case,
somebody
is
like.
Oh
what's,
cluster
one
I
haven't
actually
been
told
about
that
yet
from
the
CTS
and
it
will
like
drop
traffic
or
have
a
sort
of
undefined
behavior
for
this
grade.
I
think
this
is
bad
because
it
means
that
you're
potentially
dropping
traffic
on
that
route.
A
For
that
period
of
time,
when
the
you're
waiting
for
the
city
asset
stripe,
we
actually
like
take
the
straight
off.
We
rely
on
sort
of
envoy
retry
logic,
as
well
as
application
level
return
logic,
but
if
you're
an
environment
where
you
care
about
sort
of
very
precise
traffic
routing-
and
this
is
potentially
a
problem-
one
way
that
you
can
get
around
this
is
by
coordinating
between
management
servers,
storing
sort
of
the
state
of
things
in
a
database
or
in
Redis
or
some
backing
store,
but
that's
like
fairly
expensive
and
even
better
there's.
A
This
newer
protocol
that
envoy
has
implemented
recently
called
the
aggregated
discovery
service.
The
aggregated
discovery,
service
or
ATS
allows
you
to
multiplex
Wiltord
resource
types
over
a
single
G,
RPC
stream,
so
envoy
will
make
multiple
discovery,
requests
for
different
kinds
of
resources
and
the
management
server
will
respond
with
those
resources
in
the
precise
order
that
it
wants,
which
means
that
you
can
guarantee
sort
of
more
precisely
ordered
traffic
updates,
because
onwe
will
only
ever
subscribe
to
a
single
ATS
to
the
given
time.
A
In
summary,
dynamic
config
is
really
awesome.
We're
really
excited
about
it.
It's
right
because
lets
us
change
for
the
configuration
of
envoy
across
the
fleet
without
needing
to
go,
modify
a
config
file
everywhere.
Go
right
out
a
new
template.
The
XTS
protocol
lets
you
push
updates
at
will.
You
can
push
updates
without
modifying
the
configuration
of
envoy
everywhere
as
long
as
you
just
send
a
new
discovery
response
in
the
stream
lets
you
monitor
for
malformed
or
unpossible
configuration.
You
can
wake
yourself
up
page
yourself.
A
If
you
miss
configured,
your
management
server
or
the
customers
are
being
sent
wrong
or
whatever,
and
then,
finally,
you
can
sequence
updates,
precisely
meaning
that
you
can
guarantee
the
traffic
awry
or
that
configuration
arrives
in
order
to
expectance
is
even
easier
with
the
a
great
discovery
service.
Thank
you
all
so
much
the
full
of
documentation
for
this
is
available
in
the
data
plane.
A
Api
repository,
as
well
as
that
part
above
jarrett,
pc,
specs,
there's
a
reference
implementation,
a
go
control
plane,
which
I
highly
recommend
using
as
opposed
to
writing
your
own,
which
is
what
we
did.
I
meant
silly
I'm,
a
console,
also
bundle
old
sort
of
high
quality,
ICS
implementations
and
then
SEO
is
like
a
pluggable
service
finish.
That
provides
a
lot
of
this
for
you
anyway.
Thank
you.
So
much
take
care.