►
From YouTube: Multi-cluster Made Reasonable: Envoy Service Mesh Control Plane - Ashley Kasim & Paul Fisher, Lyft
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi-cluster Made Reasonable: Envoy Service Mesh Control Plane - Ashley Kasim & Paul Fisher, Lyft
Lyft runs business critical microservices sharded across multiple independent Kubernetes clusters in an Envoy service mesh, enabling services to autoscale and serve rides in the face of Kubernetes cluster and backplane failures. Existing Envoy control plane solutions are complex and can be difficult to span multiple clusters. Ashley and Paul's talk covers the design and implementation of Dyplomat — Lyft’s simple cloud-agnostic open source Envoy control plane. Built on top of Envoy's Go Control Plane, Dyplomat bridges together multiple Kubernetes clusters and legacy infrastructure. The talk demonstrates how to scale a production service mesh to hundreds of microservices running across multiple clusters spanning thousands of nodes.
https://sched.co/ZesE
A
A
Okay,
so
those
of
you
that
might
not
be
familiar
with
lyft,
particularly
if
you're
in
europe,
so
we're
a
ride,
share
network.
That's
in
all
50
us
states,
toronto,
ottawa
and
we
recently
launched
in
vancouver
as
well
and
in
addition
to
the
car
ride
network.
We
also
have
a
scooter
and
bike
share
networks
in
the
u.s
largest
of
which
are
a
city
bike
in
new
york
city,
as
well
as
bay
wheels
and
the
san
francisco
bay
area.
A
A
A
So
our
ml
training
jobs,
trooper
notebooks,
all
of
our
cpu
workloads,
exist
there
and
they
were
the
first
to
migrate
over
there's
also
flight,
which
was
open
sourced
at
the
last
kubecon.
Us
there's
lots
of
documentation
out
there
about
flight,
but
that's
how
I
run
spark
and
hive
and
then
sort
of
the
largest
set
of
clusters
that
we
run
here
at
lyft
are
related
to
the
rideshare
platform
and
that's
about
600,
plus
stateless
microservices.
A
A
A
So
in
terms
of
timeline
of
kubernetes
at
lyft,
we
sort
of
started
the
containerization
effort.
If
you
will
in
making
use
of
docker
back
in
2015.,
we
had
a
homegrown
system
which
sort
of
looked
a
bit
like
a
docker
swarm.
It
wasn't
until
2017
that
we
started
investigating
options
for
running
kubernetes
on
aws
and
in
particular,
we
realized
that
it
was
necessary
to
have
a
networking
platform
that
could
interoperate
with
the
existing
legacy
stack
or
what
we
call
the
legacy
stack
today,
and
so
we
worked
aggressively
to
to
get
that
platform
in
place.
A
We
open
sourced
that
work
in
december
2017
and
that
it's
part
of
our
rcni
stack
amazon
also
released
their
open
source,
cni
stacker
on
the
same
time
and
the
feature
set
is
pretty
similar
between
the
two
and,
as
I
mentioned
before,
the
batch
and
ml
workloads
were
the
first
to
sort
of
migrate
over
we've
pretty
much
finished
our
migration
of
all
of
our
stateless
services
to
kubernetes.
That
was
work
that
was
done
last
year.
It's
all
the
tier
zero
tier
one.
A
All
the
other
core
services
run
on
kubernetes
that
lift,
and
this
year
we're
focused
more
on
stateful
migrations
combination
of
what
we
call
semi-stateful
and
stateful.
Semi-Stateful
are
things
that
just
have
a
fairly
low
tolerance
for
pod
disruption,
and
we
don't
aggressively
been
packed
there,
whereas
the
stateful
services
usually
have
like
local,
ssd
and
they're
they're
instances
where
we
can't
necessarily
terminate
the
nodes
without
there
being
impact
to
the
service.
A
A
We
use
ubuntu
on
the
developer
side,
so
it's
basically
like
a
fedora,
minimal,
shim
os
running
a
bunch
of
containers
for
the
actual
lift
services,
everything's
mutable
in
the
fedora
world
we
use
packer
and
in
the
core
s
world
we
use
ignition
and
all
that's
orchestrated
via
terraform.
So
I
think
I
mentioned
earlier
we're
in
aws.
We
use
lots
and
lots
of
ec2
and
ebs
and
s3
and
our
primary
build
outs
are
in
us,
east,
1
and
us
west
2..
A
A
And
related
to
that,
and
the
reason
why
we
can
sort
of
have
this
multi-cluster
environment
that
works
as
we
have
to
keep
well,
rather,
we've
chosen
to
keep
things
as
simple
as
possible.
So
there's
no
overlay
networks,
there's
no
bgp,
we
don't
use
nat,
we
use
very
limited
use
of
ingress.
We
don't
use
cubeproxy,
as
I
mentioned
before,
pods
can
communicate
with
pods
in
any
cluster
and
then
envoy
sort
of
binds
all
that
together.
A
A
As
mentioned,
a
lot
of
kubernetes
clusters
participate
in
the
mesh
and
we
have
one
vpcip
per
kubernetes
pod
and
then
our
control
plane
is
a
software
called
envoy
manager
which
actually
we'll
talk
about
here
in
a
bit
to
cover
our
cni
stack
and
this
sort
of
comes
with
trying
to
keep
things
as
simple
as
possible.
As
we
as
we
scale
the
clusters
out,
it's
a
very
minimal
design.
A
A
So
we
have
five
main
production
clusters,
one
per
az
for
the
for
the
stateless
side
and
that's
where
the
bulk
of
the
services
are
deployed
to.
So
when
you
deploy
its
service
at
lift,
it
ends
up
getting
deployed
across
all
of
those
core
clusters
and
there's
a
bit
of
an
abstraction
layer
where
end
developers
don't
have
to
really
be
concerned
or
worried
about
where
their
services.
B
B
B
So
there's
two
types
of
clusters
we'll
be
dealing
with
kubernetes
clusters,
which
are
a
collection
of
worker
nodes,
running
pods,
orchestrated
by
one
or
more
api
servers,
and
then
there's
envoy
clusters.
Onboard
clusters
are
a
collection
of
endpoints
comprising
a
service
in
diplomat.
We
have
mapped
an
envoy
cluster
to
a
kubernetes
service.
B
B
The
two
xds
services
relevant
to
diplomat
are
cds
or
cluster
discovery
service,
which
registers
onboard
clusters
and
eds
or
endpoint
discovery
service,
which
registers
envoy
endpoints
diplomat,
is
implemented
as
an
example
in
go
control
plane,
which
is
a
reference
open
source,
go
implementation
of
the
xds
apis
which
can
be
used
to
build
custom
envoy
control.
Planes
such
as
diplomat
or
envoy
manager.
B
B
B
Envoy
manager
is
also
fault
tolerant
or
we
have
deployed
multiple
independent
envoy
manager
replicas
across
these
n
clusters,
and
our
envelope
manager
deployment
can
auto
scale
with
load
environment.
Manager's
kubernetes
support
was
originally
implemented
using
endpoint
formers,
but
has
since
switched
to
using
pod
informers,
there's
going
to
be
a
link
later
to
a
talk
that
goes
more
in
depth
as
to
this
design
decision
later
later.
On
in
this
presentation,
this
currently
runs
a
single
service
mesh
where
all
source
pods
across
these
n
clusters
are
part
of
the
same
mesh.
B
So,
let's
take
a
closer
look
at
what's
happening
in
this
diagram
here.
The
service
foo
has
two
pods.
The
envoy
sidecar
in
each
of
them
is
going
to
discover
their
cluster
cluster
local
envoy
manager
via
dns,
where
em.service.culture.local
is
a
grpc
load
balanced
across
the
envoy
manager.
Replicas
in
that
cluster,
each
envoy
manager
is
then
talking
to
the
api
servers
of
kubernetes
clusters
zero
and
one,
and
this
means
that
each
of
these
envoy
managers
has
a
complete
picture
of
the
whole
state
of
the
world
across
the
clusters.
B
And
now
on
to
diplomat,
diplomat
is
essentially
a
minimal
envoy
manager,
implementation,
which
simplifies
the
eds
control
loop
and
strips
out
all
that
legacy
and
with
specific
implementation
to
make
it
more
generic
and
adaptable
to
the
needs
of
other
organizations
and
the
open
source
community
like
envoy
manager.
It
is
based
on
the
go
control,
plane
and
it
supports
multi-cluster,
and
currently
we
have
implemented
im,
authentication
and
aws,
because
that's
what
lift
use
the
off
implementation
is
extensible.
However,
so,
with
a
bit
of
code,
other
cloud
providers
could
easily
be.
B
B
B
The
informers
implement
an
a
hook
on
endpoint
update,
which
collects
the
new
endpoint
data
and
known
and
point
data
from
the
other
clusters
and
then
performs
an
update
to
the
diplomat's
snapshot.
Cache
insert
informing
the
service
pod
on
voice
sidecars
of
the
new
envoy
or
the
new
endpoint
via
the
envoy,
endpoint
discovery
service,
and
then
in
this
diagram
we
have
n
kubernetes
clusters
and
each
of
them
has
an
endpoint
informer.
B
B
B
The
first
is
the
beta
introduction
of
endpoint
slices
in
kubernetes
117,
which
provide
a
more
scalable
implementation
for
tracking
larger
numbers
of
network
endpoints,
and
then,
as
I
had
mentioned
earlier,
lyft
has
kind
of
gone
back
and
forth
between
watching
pod
and
endpoint.
Events
in
envoy
manager,
envoy
manager
is
currently
using
a
pod
informer,
but
we
are
aiming
to
move
back
to
an
endpoint
informer
in
the
future.
B
B
Finally,
the
envoy
upstream
community
has
an
open
issue
around
envoy
health
checking
being
unaware
of
kubernetes
pod
planned
removal.
This
is
one
of
the
main
issues
with
using
an
endpoint
informer
with
your
onvoice
control
plane.
Currently,
as
kubernetes
endpoints
are
updated
only
once
a
pod
is
torn
down,
so
you
may
not
be
able
to
update
eds
endpoints
fast
enough
to
prevent
envoy
from
seeing
failed
health
checks
from
the
draining
pods
and
thinking
that
the
service
is
in
fact
unhealthy.
B
As
you
can
see,
we
have
two
eks
clusters
in
this
account
in
each
of
them.
We
are
running
a
few
components:
a
front
envoy
proxy
service,
one
and
service,
two
services,
which
we
want
to
connect
using
our
service
mesh
and
implement
our
onboard
control
plane
that
implements
envoy
eds
for
kubernetes
endpoints.
B
B
B
As
you
can
see,
it
has
an
ip
address.
Now
we
can
delete
this
pod
and
the
kubernetes
deployment
recreates.
The
pod
and
updates
the
kubernetes
endpoint
object
with
a
new
pod
ip
address.
The
kubernetes
endpoint
informer,
implemented
by
diplomat,
sees
this
update
event
and
triggers
an
update
to
envoy
eds.
So
going
back
to
our
service
1
pod
in
the
demo
2
cluster,
we
can
again
curl
the
envoy
admin
endpoint
and
see
that
the
ips
for
service
2
have
been
updated
to
include
the
new
ip
and
the
updated
endpoint.
B
B
B
All
right,
hopefully,
this
was
valuable
in
providing
a
pattern
to
build
multi
kubernetes
cluster
envoy
meshes
and
the
corresponding
control
plane.
The
diplomat
source
code
can
be
found
on
lyft's
public
fork
of
go
control
plane
on
the
diplomat
branch
contributions
are
welcome,
so
please
check
it
out.
B
A
A
A
A
A
A
For
next
questions
related
to
what's
the
typical
network
overhead,
around
broadcasting,
xps
values,
I've
forgotten
what
the
current
time
frame
is
for
convergence.
We
can
follow
back
up
on
slack,
I
think
a
lot
of
our
health
checks
and
stuff,
it's
on
the
order
of
20-ish
seconds
or
so
for
health
checks.
But
it's
a
couple
like
multiple
round
trips,
but
we
can
follow
up
separately
about
that.
B
So
I
can
talk
to
the
next
two
of
them.
The
first
is:
does
diplomat
talk
to
non-trade
services?
This
is
currently
not
implemented.
However,
this
could
be
extended
by
adding
you
know.
Whatever
the
equivalent
is
that's
doing
discovery
for
your
non-kubernetes
infrastructure
and
then
the
second
is.
Can
we
use
diplomat
on
an
enterprise
setup
diplomat
is
an
example,
so
I
think
that
while
it's
a
good
basis
for
building
your
own
custom
control
plane,
it's
not
necessarily
like
tested
at
scale.
A
Yeah
and
then,
and
then
particularly,
we
also
have
the
original
sort
of
like
open
source
discovery
service
that
came
out
when
envoy
originally
came
out
and
that's
what
was
used
for
our
legacy
stack,
and
so
that
was
the
the
discovery.
Service
was
what
like
the
original
versions
of
of
envoy
manager
and
the
control
plane
didn't
actually
speak
kubernetes
at
all,
and
so
since
we
had,
since
all
the
pods
have
vpcips
the
early
versions
of
kubernetes.
A
Performance
issues
some
of
these
questions
related
to
have
we
have
we
had
any
performance
issues
watching
all
kubernetes
clusters
and
control
plane.
So
when
we
were
using
endpoint
informers,
no
for
pod
informers,
there's
also
got
mentioned
in
the
talk,
but
there
there's
a
prior
kubecon
talk
from
tom
and
lita
at
lyft.
A
A
In
that
sense,
there
were
some
issues
with
the
onboard
manager
in
terms
of
like
how
quickly
it
could
scale
up
and
keep
the
state
of
the
world,
but
most
of
those
issues
have
been
addressed
of
late.
It's
pretty
much,
I
think,
just
the
the
size
of
the
mesh
that's
concerned,
particularly
with
related
to
memory
usage
within
the
envoy
sidecar,
and
that's
the
reason
for
moving
out
to
the
to
the
split
mesh
approach.
B
So
the
next
question
is:
how
do
you
deal
with
new
deployments
for
each
cluster?
This
is
where
cvs
comes
in
and
a
simple
implementation
is,
you
know
you
annotate
the
deployment
and
then
you
use
an
informer
to
watch
for
new
employments
and
that
deployments,
and
then
you
generate
an
envoy
cluster
for
the
new
clusters
that
are
discovered.