►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Low Latency Multi-cluster Kubernetes Networking in AWS - Paul Fisher, Lyft
Lyft is migrating their entire service stack of hundreds of microservices to Kubernetes on AWS. A critical component to Lyft’s successful migration is their open source set of CNI plugins which implement a simple, fast, and low latency networking stack tying together multiple Kubernetes clusters into a flat network within AWS Virtual Private Clouds. Paul’s talk takes a deep dive into the design and implementation of Lyft’s multi-cluster Kubernetes platform from a network-centric perspective, including Envoy mesh integration and performance characteristics.
https://sched.co/Uaay
A
Okay,
so
in
terms
of
lift
scale,
so
we
have
millions
of
riders
per
day
more
than
30
million
riders,
more
than
2
million
drivers
and
we're
available
in
all
50
US
states,
Toronto
and
Ottawa,
and
hopefully
you
took
a
lift
from
the
airport.
And/Or
took
a
lift
to
get
to
coop
gun
and
there's
requests
definitely
went
through
kubernetes
as
part
of
as
part
of
getting
here.
A
So
in
terms
of
what
the
scale
of
criminais
looks
like
at
lift
today,
it's
sort
of
divided
into
three
main
areas
for
the
section
of
a
company
that
I'm
in
so
we
have
our
machine
running
workloads,
they're
whole
bunch
of
GPUs
they're,
lots
of
training,
jobs
that
we're
on
Jupiter
notebook
support
in
terms
of
scale.
It's
about
5,000,
plus
pods
and
10,000
cores
are
dedicated
to
that
effort.
A
Flight
I
won't
go
into
too
much
detail
on
flight
they're
a
whole
bunch
of
other
talks
at
coop
con
on
flight,
so
I
encourage
you
to
go,
see
those,
but
that's
our
distributed.
Workflow
orchestration
system
and
that's
kind
of
the
inverse
the
machine
learning
in
terms
of
scale.
It's
flip-flopping
the
number
of
pods
versus
number
of
cores.
They
didn't
run
what's
inside
cards
on
their
side
and
then
the
cluster
is
that
my
team
I
spend
most
the
time
focused
on.
A
And
that's
made
up
of
plus
stateless
microservices,
all
of
those,
so
we
have
our
attendant
clusters
that
exist
on
a
per
availability,
own
basis.
I'll
talk
about
that
more
in
detail
and
all
those
are
part
of
a
single
production
envoy
mesh
and
in
terms
of
like
the
the
scale
in
terms
the
numbers
we
use
HPA's
aggressively.
So
the
number
of
pods
that
we
have
and
the
number
of
ec2
instances
that
were
running
greatly
varies
based
on
time
of
the
day
and
and
load,
and
we
run
a
whole
bunch
of
sidecars.
A
This
talk
does
get
into
it,
but
a
lot
of
the
work
that
we've
done
has
been
related
to
sidecar
ordering
and
making
sure
that,
like
what,
what
a
lyft
service
developer
expects
to
be.
The
case
like
the
same
from
legacy
stack
holds
true
for
the
communities
world.
It's
about
1,000
containers,
plus
and
3,000,
plus
course
for
that
environment.
A
In
terms
of
the
timeline,
so
in
December
of
2015,
we
started
internal
docker
based
effort
at
the
time
to
sort
of
create
a
container
based
system
for
handling
our
development
and
continuous
integration
stacks.
But
all
of
our
production
side
was
basically
stateless
services
running
on
auto
scaling
groups
and
just
raw
ec2
instances.
A
So
it's
about
May
of
2017
when
you
started
investigating
options
for
running
kubernetes
label
us,
and
at
least
at
the
time.
We
didn't
feel
that
any
of
the
available
options
were
can
necessarily
like
work
at
the
scale
that
we
needed
for
them
to
and
we're
we're
too
complicated.
And
so
that
was
when
we
kind
of
went
back
to
first
principles
and
tried
to
figure
out
the
most
performant
and
the
best
method
for
mapping
abus
VPC
constructs
on
to
kubernetes
and
the
scene
I
stacked
specifically.
A
A
In
addition
to
sort
of
doing
the
plumbing
work
that
we
needed
to
do
for
the
the
stateless
services
and
inside
cars,
there's
a
lot
of
stuff
that
happened
as
part
of
needing
to
get
stateless
there,
and
so
this
year
has
been
focused
mostly
on
migrating
services
or
the
stateless
services
that
are
the
core
of
the
platform
overt.
If
you
were
Nettie's
and
that
does
include
tier
0
services,
so
the
tier
0
services
are
the
ones
if
that
service
is
down,
then
lifts
itself
is
hard
down.
A
So,
in
terms
of
what
our
production
environment
looks
like
today,
more
on
communities
114,
the
plan
is
to
move
to
116
by
the
end
of
the
year
that
works
starting
this
week,
probably
in
terms
of
the
operating
system
that
we
use,
we
tend
to
hold
back
one
release
from
the
current
production:
Fedora
release
minimal
small
OS.
We
use
cryo
as
our
container
runtime
environment,
so
it
gives
us
like
a
handful
of
good
things
in
the
sense
of
like
we
have
kernels
that
are
very
close
to
mainline.
A
Lots
of
bug
fixes
the
handle
container
as
well,
and
we
use
system
D
to
handle
all
the
secret
management
so
on.
The
flip
side
of
that
is
that
most
of
the
internal
developers
that
lift
don't
know
that
we
run
Fedora
as
sort
of
like
the
bootstrap
OS,
because
they
just
use
a
bun
to
and
sims
2
that
is
based
on
Bionic.
A
All
of
our
infrastructure
is
immutable.
In
the
community
side
we
build
a
Mize
from
Packer,
we
use
terraform
orchestration.
We
try
very
much
and
there's
other
talks
at
cube
con
as
well
that
sort
of
talk
about
how
we,
how
we
roll
the
clusters
and
how
we
maintain
that
that
state
yeah
so
well,
it's
entirely
an
AWS,
our
largest
build-out
is
us
East
one.
We
also
have
some
clusters
over
in
u.s.
West
too,
as
well
as
I
mentioned.
Before
we
have
crazy
clusters.
We
have
separate
clusters
for
staging
production.
A
This
lets
us
do
staggered
rollouts,
so
we
can
have
a
very
well-defined
process.
So
if
we
end
up
taking
a
change
to
say
a
staging
cluster,
you
know
it's
likely
that
we
only
packed
one.
Staging
cluster
is
opposed
to
all
staging
clusters
and
then,
in
the
case
of
production
as
well,
it's
not
likely
that
we
have
a
chain
change,
rollout
that
affects
all
abroad
and
then
of
course,
on
top
of
all
this,
we
run
our
stack
which
I'll
talk
about.
A
A
A
Ok,
so
in
terms
of
sort
of
the
network
options-
and
this
is
sort
of
almost
time-based
as
well
in
terms
of
how
CNI
plugins
sort
of
developed
and
what
was
available,
the
very
most
like
simplest,
very
straightforward,
which
is
what
I
think
a
lot
of
people
used
early
on.
It's
very
high
performant.
Is
you
just
take
a
slash,
24
per
node
and
you
just
pull
it
down
to
the
ec2
instance.
A
So
you
think
we
have
around
285
IPS
and
you
get
a
max
of
fifth-year
out
some
talk
about
that
a
little
bit
or
default
max
of
50
routes,
and
then
the
overlay
network
space
is
where
most
United,
plugins
I'd
say,
operate
or
miss
the
ones
are
used
in
production
and
that
sort
of
abstracts
away
or
tries
to
abstract
away
a
lot
of
the
underlying
network,
semantics
and
sort
of
lets.
You
operate
at
a
higher
level,
and
then
you
have
your
VC
VPC
native
networks,
of
which
the
little
stack
is
one
of
those
okay.
A
So
I'll
go
into
each
of
these
briefly,
so
we
have
a
slash
paper
per
ECT
note
as
one
option
so
previously
the
limit
on
this,
you
could
only
have
100
routes
per
VPC
back
in
2017.
We
first
start
looking
at
all
this,
which
meant
that
roughly
you
had
on
the
order
of
100
node
cluster
was
the
most
you
could
get
out
of
out
of
EPC.
Since
then,
abs
has
increased
limits
added
an
order
of
magnitude.
So
now
you
can
have
up
to
a
thousand
route,
so
you
can
have
a
much
larger
cluster.
A
A
You
turn
on
this
mic
and
held
yes,
good
people
hear
me:
okay,
okay,
sorry
about
that.
Okay
caught
a
gnostic
new
one,
it's
one
cluster,
sighs,
okay,
so
effectively
like
AWS
itself
and
the
V
PC
is
a
software-defined
network.
That
has
a
very
it's
like
it's
a
very
good
API.
It's
been
around
for
a
long
time,
but
it's
it's
very
stable
and
it
does
what
it
does
and
so
effectively
what
you're
doing.
A
If
you
run
an
overlay
network
is
you're
choosing
to
run
a
software-defined
network
on
top
of
a
software-defined
network,
so
you're
in
this
like
whole
other
business,
which
I
don't
know
that
people
necessarily
want
to
be
in.
If
you're,
you
know
paying
Abe,
you
asked
to
run
your
network
for
you
and
also
because
you
have
these
sorts
of
limitations,
you
might
have
issues
connecting
to
the
Envoy
mesh.
You
might
have
to
use
NAT
and
there's
just
like
lots
of
like
hard
edges.
A
If
you
will
I
guess
someone,
one
thing
is
like
a
lot
of
these
are
fairly
easy
to
set
up
in
terms
of
just
like
point-and-click
and
get
them
up
and
running.
But
it's
like
once.
You
have
a
problem.
It
can
be
really
complicated
to
the
bug.
What's
going
on,
okay,
so
next
category,
which
didn't
exist
at
the
time
when
we
started
writing
all
this
two
years
ago,
where
options
to
talk
to
the
EPC
network
natively.
So
all
this
is
from
our
view
at
least
it's
very
simple
and
straightforward.
A
There's
not
a
lot
of
code
that
exists
to
make
all
this
work.
Pods
receive
full
VPC
IP
addresses
and
they
get
full
connectivity
within
the
V
PC.
So
anything
else
in
the
V
PC,
whether
it's
Amazon
service
or
some
legacy
service,
you're
running
running
that
you
have
like
you
just
can
talk
to
it
and
it's
another
V
PC
IP
address
and
then
most
one
of
the
most
important
points
is
that
you
get
native
network
performance
or
you
should
get
like
near
native
network
performance
and
so
the
two
main
options
there.
A
Or
sack
so
it's
very
minimal.
There's
no
daman
sets
there's
pods.
There's
no
runtimes
is
a
handful
of
stateless.
Go
binaries,
we're
on
everything.
Production
with
cryo
data
dog
runs
their
production,
workloads
and
container
D,
as
I
mentioned
overlay.
The
two
interfaces
I'll
talk
about
that
that
live
inside
of
of
our
environment,
which
is
the
IP
VLAN
interface
and
an
unnumbered
point-to-point,
interface
and
kind
of
most
importantly,
is
like
you
don't
need
to
change
anything
about
your
V
PC,
so
you
don't
have
to
fiddle
with
like
asymmetric
routing.
A
You
don't
have
to
like
change
your
route
table.
It's
intended
to
just
work.
However.
Your
VP
is
C
is
currently
configured
and
from
our
perspective,
it's
basically
feature
complete.
We've
been
running
in
production
for
two
years
with
very
minimal
changes,
so
regarding
an
IP,
VLAN
you're,
not
aware
of
it,
there's
some
good
papers
about
it.
There's
a
lot
of
talks
been
presented
over
the
years
about
it,
but
it
was
created
by
Google
in
2014
it
shipped
with
Linux
318
the
primary
benefit
it
was.
A
It
was
essentially
designed
to
work
with
containers
and
it
avoids
going
back
and
forth
between
the
default
network
namespace
or
the
host
namespace.
So
with
a
lot
of
systems
for
getting
IPs
and
the
containers
or
internetwork
namespace
with
Linux
you've
got
to
effectively
have
a
virtual
Ethernet
device
or
like
some
sort
of
bridge
that
since
the
packets
back
and
forth
through
the
main
namespace.
So
in
our
case
we
can
take
a
TBS
elastic
network
interfaces
which
are
like
our
host
network
adapters
and
then
tie
those
directly
into
pods.
A
So
there's
very
little
overhead
I'd
argue
that
at
least
it's
probably
the
least
amount
of
overhead
that
you
can
have
in
order
to
in
order
to
get
IPs
and
to
into
pots,
and
then
in
particular,
I.
Don't
know
if
Google
like
knew
this
at
the
time,
but
IP
VLAN
Maps,
basically
perfectly
to
the
constraints
on
the
AWS
VPC
design,
particularly
in
our
case
your
own
IP,
VLAN
and
layer
2
mode,
and
that
just
it
works
out
really
well.
A
Ok,
I
mentioned
earlier.
It's
a
virtual
NIC,
and
so
the
number
of
ni
is
that
you
can
have
her
ec2
instance
is
based
on
the
instant
size.
So
right
now
that's
two
to
fifteen
and
then
also
the
number
of
IPs
you
can
have
per
II
and
I
also
varies
based
on
the
instant
size,
and
so
the
the
point
I
mentioned
earlier
about
needing
to
support
up
to
a
hundred
and
ten
pod
310
IPS
per
node.
A
A
But
again
this
is
more
for
a
production
environment,
not
necessarily
for,
like
you
know,
a
development
environment
in
terms
of
how
we
managing
eyes,
that's
all
managed
by
the
by
the
CNI.
So
we
reserved
the
beauty,
knife
or
the
kubernetes
control
plane
and
also
any
services
that
that
need
the
host
IP.
And
then
we
effectively
pull
down
in
eyes
as
we
need
to
and
then
IP
addresses
get
added
to
the
in
is
until
they're
full
and
then
once
the
eye
is
full.
A
We
make
a
new
en
I
and
we
add
IP
staff
and
there's
some
options
like
pre
allocating
eyes
on
boot.
If
you
want
to
just
to
save
the
overhead
from
that
by
default,
we
use
a
60
second
TTL,
so
there's
IP
addresses
get
get
reused
as
as
we
need
to,
and
that's
but
effectively
it
like
an
accurate
ease
environment.
It's
very
common
to
have
a
lot
of
pod
turn,
and
so
we
try
to
hang
on
to
those
IPS
when
when
we
can
and
just
reuse
them.
A
Okay,
so
if
you
end
up
coop
coddling
exec
into
into
pod
you'll,
see
in
addition
to
loopback
interface,
you'll
see
two
main
interfaces.
One
is
e
0,
which
is
your
IP
VLAN
interface,
and
so
that's
the
one
that's
tied
directly
back
to
the
you
know,
I
and
I
that
you're
attached
to
all
V
PC
traffic
goes
out
at
that
address
and
additionally,
it's
also
isolated
from
all
the
other
n
eyes
on
the
system
and
then
the
other
interface
that
we
have
since
you'll,
probably
going
to
have
traffic
that
is
not
directly
over
the
V
PC.
A
Is
we
have
an
unnumbered
p2p
interface?
So
this
is
effectively
a
high-speed
interconnect
back
to
the
host
namespace.
So
if
you
have
any
pods
with
host
networking
or
if
you
have
any
cupric
see
virtual
IP
addresses
any
sort
of
like
the
networking
magic
layer
that
kubernetes
does
all
this
traffic
goes
back,
goes
out
over
a
v-0,
and
so
there's
sort
of
this
I
think
it's
kind
of
a
neat
little
hack,
but
it's
specified
in
RFC
5309,
which
is
that
if
an
IP
address
doesn't
exist
for
an
interface,
then
an
IP
address
gets
borrowed.
A
So,
in
our
case,
this
particular
interface
doesn't
have
an
IP
address
at
all
on
either
side
and
it's
going
to
borrow
the
DPC
IP
address
from
each
zero.
And
so,
if
you
recall
back
when
I
was
talking
about
how,
like
everyone
else,
has
to
see
you
with
the
same
IP,
that's
what
you
see
yourself
as
that's
how
we
achieve
it
for
this
particular
interface
and
then
also
this
interface
gets
used
for
Internet
egress.
If
that's
something
that
you
need
to
do
in
your
environment,
which
we
do
so
how
many
Internet
us
works.
A
So
this
is
a
feature
of
ec2
hosts
where
you
can
have
private
IP
of
your
boot
network
adapter,
which
can
map
to
a
public
IP
and
then
any
traffic
that
gets
sent
out
that
gets
sent
out.
That
interface
ends
up
getting
added
by
by
a
TOS
in
order
to
talk
to
the
public
internet,
and
so
in
our
case,
we
because
we
can
like
get
to
the
public
internet
over
this
interface
and
it's
redundant
and
scalable
doesn't
involve
any
sort
of
like
single
points
of
failure.
We
end
up
having
our
pods.
A
A
A
We
also
have
a
staging
environment
that
looks
similar
to
this,
and
then
we
have
a
legacy
stack
as
well,
and
so,
as
I
mentioned
before,
it
lets
us
like
incriminating
rollout,
and
it
also
means
that
if
we
lose
a
whole
entire
cluster,
then
left
as
a
whole
stays
up.
So
services
end
up
deploying
to
all
of
the
core
clusters
and
HPA's
run
on
each
of
those
clusters
to
scale
services
up
and
down.
A
Okay,
so
pods
are
made
up
of
a
whole
bunch
of
containers.
I'm
not
going
to
go
too
much
in
detail
about.
Those
kind
of
the
important
thing
is
that
we
have
a
lift
service
which
is
Python
goer
JavaScript.
Most
of
our
services
are
in
Python
or
go,
and
then
we
have
an
envoy
sidecar,
but
they're
headed
for
other
side.
A
Our
legacy
stack
just
using
ec2
instances
with
all
the
scaling
groups
so
from
envoys
perspective
and
has
an
IP
and
that
just
becomes
part
of
the
mesh.
So
health
checks
it
like
everything
that
you'd
expect
would
work.
So
we
had
a
controller
that
was
on
all
of
our
port
clusters
and
that
would
watch
for
pods
to
come
up
and
then
register
those
and
to
open-source
our
open
source
service
discovery
layer.
And
so
since
then,
we've
revamp
the
control,
plane
envoy
managers.
A
So
in
terms
of
how
so
I
mentioned
before
about
pods-
and
we
have
like
all
these
sidecar
containers
and
one
of
those
containers,
it's
the
on
way
mesh
like
cart,
and
so
those
need
to
get
XDS
data
as
part
of
their
startup.
In
order
to
do
that,
we
run
a
headless
service
or
Envoy
manager
and
so
effectively
all
the
Envoy
managers
that
are
in
renée's
cluster.
A
Think
kind
of
one
of
the
other
were
sort
of
like
driving.
Factor
for
us
in
terms
of
design
is
to
avoid
complexity
on
the
network,
I
think
communities
as
a
whole
lets
you
there's
a
lot
of
magic
there,
which
I
think
is
great
for
development
environments
and
is
is
also
good
for
sort
of
like
setting
up
things
quickly
or
maybe,
like
smaller
production
environments.
A
But
if
you
have
envoy
up
and
running-
and
you
have
a
production
network,
you
want
to
try
to
avoid
any
technology,
you
don't
absolutely
need,
and
so,
if
you
need
to
technology,
please
feel
free
to
use
it.
But
I
guess
it's
important
to
understand
the
complexity
and
what
it's
doing
behind
the
scenes
to
make
that
happen,
because
if
it
breaks
it's
gonna
be
that
much
harder
to
bug,
because
at
that
point,
you're
gonna
have
to
understand
how
all
the
magic
works
and
in
terms
of
IPP
LAN,
it's
worked
really
well
for
us.
A
We
keep
latency
stats
relating
numbers
for
all
of
our
lift
services.
That's
all
exported
as
part
of
the
Envoy
metrics
P
95
and
P
99
has
been
same
across
this.
So
in
that
sense,
from
a
network
perspective,
the
migration
has
been
very
transparent
for
service
owners.
It
lifts
I,
think
also.
Another
sort
of
important
item
is
that
the
VP
C
continues
to
work
as
intended.
A
The
other
thing
is:
if
you
have
a
way
to
to
get
your
tears,
your
services
on
to
multiple
clusters,
I
would
say:
do
that
I?
Think
one
of
the
one
of
my
fears
or
kept
me
up
a
lot
at
night.
It
was
like
okay.
Well,
if
we
lose
a
cluster,
does
lift
go
down,
and
so
it's
very
important
to
me
that
that
we
can
lose
a
cluster.
The
goal
is
to
is
to
not
lose
a
cluster,
but
if
we
do
lose
a
production
cluster,
it's
no
different
than
losing
an
available
at
least
own.
A
If
anyone
else
wants
to
add
that
we're
more
than
welcome
to
set
patches
for
that,
and
one
particular
take
as
well,
which
I
think
was
like
a
lot
of
discussion
early
on,
is
how
to
have
network
policies
and
I.
Think
c9
chaining
is
probably
the
correct
approach
there
and
that
you
have
a
specific
stack.
That
is
not
your
core
stack,
which
is
responsible
for
handling
network
policy
enforcement,
so
in
general,
it
looks
like
selenium
should
be
pretty
easy
to
wire
up
with
our
stack,
so
we're
looking
up
into
doing
that
with
there
seen
I
chaining.
A
Another
thing:
that's
currently
under
development
is
using
traffic
control
to
restrict
bandwidth
based
on
CPU
count.
So
this
isn't
really
a
production
issue
for
us
right
now
or
has
not
been
just
because
of
how
large
of
the
instances
that
we
run,
and
so
when
we're
scheduling,
pods
against
nodes,
we
can
run
out
of
CP
memory
like
way
before
we
end
up
running
out
of
network
bandwidth.
A
A
D
A
So
you
effectively
tag
what
subnets
you
want
to
use
within
that
and
our
plug-in
will
burn
down.
Ip
addresses
at
an
equal
rate,
so
whatever
subnets
are,
are
allowed
for
it
to
use
it'll
make
sure
that
there's
get
burned
down
evenly
and
then,
additionally,
you
can
have
secondary
sliders
on
your
V
PC,
which
I
think
abs
introduced
within
the
past
couple
of
years.
But
it's
you
can
add
more
slash,
16
to
your
V
PC
and
you
can
assign
those
to
clusters
so
like
effectively.
E
F
A
How
much
does
it
lower
it?
Does
it
manage
of
all
negligible
and
we
actually
internally,
we
don't
have
really
good
stats,
unlike
P
triple
nine,
so
there's
probably
some
weight
definitions
on
like
P
triple
nine,
but
there's
a
handful
of
papers
out
there
that
cover
IP,
VL
and
Layton
sees
and
performance
testing
but
and
our
practice
for
what
we
care
about,
which
is
P
99.
It's
been
equivalent
hi.
A
A
So
we
have
a
very
sort
of
like
lightweight
I'd,
say
it's
sort
of
like
we'll
call
it
Federation,
but
it's
a
very
lightweight
stack
that
merges
together
all
the
clusters
and
that,
if
you're
a
lyft
developer
or
if
you're,
using
on
deployment
snack,
you
end
up
deploying
to
all
the
clusters
and
so
as
part
of
the
deploy.
You
can
determine
whether
or
not
your
application
failed
deploying
to
a
particular
cluster.
But
each
cluster
is
treated
independently.
A
With
other
cloud
providers
like
GCP,
I
know
in
this
case
like
it,
is
again
there's
not
a
lot
of
code
there
and
it's
heavily
dependent
upon
a
TBS
api's.
So
it's
very
much
depended
on
the
B
PC
I'm,
so
yeah
it
wouldn't
be
appropriate
for
a
different
cloud
provider.
Thanks
I
mean
having
said
that,
I
would
say
that,
regardless
of
what
cloud
provider
you're
using
you
for
production,
you
want
something
that
gives
you
this
like
one-to-one.
Mapping
of
a
VPC
IP
address.
A
I
A
H
A
D
A
Security
groups
are
on
a
Purina
basis
and
so
phenomenally
whatever
in
eyes
you
have
set
up
for
which
cluster
that
determines.
You
can
talk
to
you
at
the
80s
level,
so
our
pods
are
not
addressable
from
the
public
Internet.
So
there
is
a
way
for
our
pods
to
egress
to
the
public
internet
over
an
out,
but
that
doesn't
mean
that
someone
from
public
internet
can
access
those
pods
all.