►
From YouTube: Evolving Legacy Systems into Kubernetes at Lyft: A Hybrid Environment - Lita Cho & Jose Nino, Lyft
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Evolving Legacy Systems into Kubernetes at Lyft: A Hybrid Environment - Lita Cho & Jose Nino, Lyft
Before Kubernetes, Lyft has hundreds services that deployed as VMs in AWS. Moving over to Kubernetes is a big undertaking. This talk will go over how we leveraged Envoy and the go-control-plane (the service that hosts Envoy configuration and discovery API) to split traffic between our legacy infrastructure and our Kubernetes infrastructure. This talk will demonstrate how Lyft handled service discovery within our hybrid environment, various ways to control the traffic when Kubernetes issues came up, and how Lyft used Envoy as a sidecar container to keep in par with the legacy behavior. Lastly, this talk will discuss the various challenges we faced with this migration plan but why it was needed due to the size of Lyft’s legacy infrastructure.
To learn more: https://sched.co/GrS1
A
A
My
name
is
leta,
and
this
is
Jose
and
we're
going
to
be
presenting
to
you
a
case
study
of
how
lyft
is
migrating
over
to
kubernetes
specific,
specifically
on
the
networking
side.
So
we'll
be
talking
about
like
how
we're
handling
network
traffic
and
how
about
handling
service
discovery
in
this
new
kubernetes
world.
There's
many
parts
to
a
cooperate
migration.
So
there's
like
the
deployment
phase,
as
well
as
the
provisioning
stage
and
security
groups,
but
we
don't
have
time
to
cover
all
of
that.
A
So
it's
very
good
specifically
if
you'd
be
talking
about
the
networking
portion
of
it.
So
a
little
brief
intro
to
who
we
are
my
name
is
leta
I
really
like
boba,
so
that
icon
drives
with
me
very
much
I'm
in
the
networking
team
for
a
year
and
a
half
now
I
like
working
on
like
many
different
things,
but
mostly
on
like
operating
on
voice
and
lately
like
integrating
kubernetes
with
our
service
mesh
and
creating
the
networking
portion
of
them.
A
Creating
a
migration
plan
for
the
networking
portion
and
I've
been
that
live
for
a
long
time
like
a
four
and
a
half
years
and
half
of
my
time
at
lyft,
I've
been
in
product,
I
have
launched
lyft
line
and
then
the
second
half
has
been
in
infrastructure.
So
it's
been
pretty
great
to
be
able
to
take
all
my
learnings
from
product
and
bring
them
over
to
infrastructure.
A
A
So
we've
broken
up
this
talk
to
like
three
pieces,
so
first
we
want
to
give
you
some
context
of
why
I
love
doing
this.
Kubernetes
migration.
Migrations
are
expensive.
They
take
a
lot
of
engineering,
time
and
they're
and
they
take
up
a
lot
of
resources.
So
we
want
to
explain
why
we
decided
to
do
this
and
what
are
the
visit
business
benefits?
A
A
And
lastly,
we
want
to
share
with
you
some
lessons
learned
so
that
when
you
decide
to
do
your
own
kubernetes
migration,
you
could
take
these
lessons
with
you
and
take
some
of
our
learnings,
so
you
don't
have
to
relearn
them
twice,
so,
let's
dive
into
it
so
like
so
we're
gonna
dive
into
why
lyft
decide
to
do
this.
Migration
in
the
first
place,
cuz
yeah,
again,
migrations
are
expensive,
it's
easier
to
just
to
maintain
our
legacy
system.
So
what
made
us
decide
to
pay
this
cost?
A
Now,
at
the
size
that
we
are
and
move
over
to
kubernetes,
so
I'm
gonna
give
a
little
bit
of
context.
Business-Wise.
So
looked
it's
a
very
real-time
system
and
we
usually
can
predict
when
our
peak
events
are
from
historical
data.
But
sometimes
we
can't
always
do
that
right.
So
we
have
this
so
there's
a
particular
day.
I.
A
It
didn't
happen
historically
before
it
was
also
a
nationwide
thing
that
was
happen
to
be
very
popular,
but
we
didn't
have
any
historical
data
to
like
predict
that,
and
also
we
in
our
legacy
infrastructure.
It's
pretty
expensive,
like
you
know,
we
use
AWS
ec2
instances
and
to
launch
that
many
instances
it
takes
around
20
minutes
so
like
we
couldn't
like
ceiling
out
fast.
This
is
something
that
that's
that's.
A
Secondly,
is
we
like
that
criminals
gives
you
a
separation
of
concerns
between
your
infrastructure
code
and
your
application
layer?
So,
right
now,
in
our
legacy
world,
we
have
this
folder
called
ops,
folder
and
that's
where
all
your
orchestration
files
go
and
also
like
you,
run
a
lot
of
infrastructure
like
processes
on
your
ec2
box.
A
They
like
want
to
program
their
application
and
business
logic,
and
then,
once
a
month
or
once
every
like
two
quarter,
they
like
look
at
this
and
like
oh,
we
need
to
scale
I
forgot
how
to
like
I,
don't
know
what
these
terms
mean.
I
have
to
go.
Look
him
up.
So
there's
a
lot
of
context,
switching
happening
here
and
in
kubernetes
everything
all
that
info
like
in
kubernetes.
A
A
You
want
the
product
engineers
to
think
about
just
their
application
and
minimize
the
knowledge
of
infrastructure
things
that
they
need
to
need
to
do
and
reduce
that
cognitive
load.
Lastly,
the
biggest
reason
is
kubernetes
gives
you
this
immutable
infrastructure
due
to
the
fact
that
you
know
like
in
our
latest
legacy
infrastructure,
due
to
the
fact
that
what
is
it
ec2
instances
take
a
long
time
to
spin
up.
We
can't
deploy
like
rotating
aSG's,
because
then
a
deployer
would
take
a
really
long
time.
A
It
will
take
probably
an
hour
hours
or
so,
and
so
we
want
we
value
like
being
able
to
deploy
and
roll
back
very
quickly.
So
we
do
upgrades
in
place,
but
there's
some
downsides
to
that.
Then
one
main
downside
is
that
the
fact
that
your
scale-up
code-
it's
not
necessary.
This
lets
the
behavior
of
scale-up.
It's
not
necessarily
the
same
as
your
initial
deploy
and
with
communities
because
of
the
immutable
infrastructure.
You
get
the
same,
no
matter
what!
B
All
right
so
taking
into
account
the
gains
that
we
would
see
in
a
kubernetes
infrastructure,
let's
dive
into
what
we
have
done
from
the
networking
perspective.
So
unless
you're
starting
from
scratch,
you
will
end
up
in
this
hybrid
scenarios
where
you
might
have
negative
infrastructure
and
the
infrastructure
that
you're
moving
to
and
because
we're
real-time
systems
that
can
take
down
time.
B
So
before
we
start
with
where
we
are
today
and
where
we're
going?
Let's
take
a
look
at
our
network
topology
up
until
now,
envoy
is
a
project
that
came
out
of
lyft
and
it
is
our
networking
proxy
all
the
way,
from
the
edge
of
our
infrastructure
to
all
of
our
services
to
also
egress
out
of
our
infrastructure.
So
what
we
have
here
really
is
a
universal
network,
primitive
all.
B
So
before
I
go
into
talking
more
about
service
discovery,
I
just
want
to
do
some
novel
setting
and
briefly
discuss
what
what
service
discovery.
Yes,
so
in
elastic
infrastructure,
it
says
you
may
know
we
have
machines
coming
up
and
shutting
down
all
the
time,
and
so
it
is
hard
to
pinpoint
all
the
machines
that,
in
a
stable
manner,
all
the
machines
that
might
belong
to
a
logical
service.
So
what
we
do
in
service
discovery
is
be
able
to
pair
a
particular
host
or
particular
IP
addresses
to
a
logical
set
of
a
service
or
an
application.
B
So,
for
example,
in
the
left,
we
might
have
three
machines
that
are
running
a
Python
application.
All
this
machines
belong
to.
You
know,
service,
foo
versus
in
the
right.
We
might
have
again
three
machines
that
are
running
a
go
service
and
well
there's
three
different
machines
with
three
IPS:
they
all
belong
to
the
logical
service
bar.
B
B
So
the
desired
end
state
that
we
have
both
in
our
legacy
infrastructure
and
as
we
want
to
introduce
infrastructure
managed
by
kubernetes.
Is
this
this
simple
flow
of
requests
through
the
service
mesh?
So
we
want
request
to
flow
through
this
I
convoy,
regardless
of
what
application
it
might
be,
or
particularly,
what,
where
that
resource
might
be,
either
in
our
AWS
infrastructure
or
in
our
Cooper
various
infrastructure.
So
we
won.
B
So
really
the
goal
that
we
have
as
we
move
into
this
hybrid
infrastructure
is
from
the
perspective
of
product
engineers.
We
want
to
build
a
system
that
makes
it
easy
for
them
to
operate
the
network,
regardless
of
what
might
be
behind
the
scenes.
So
this
is
the
focus
that
the
network
and
team
has
had
ever
since
we
were
ever
since
we'd
begun
and
particularly
for
kubernetes.
We
want
to
make
this
new
infrastructure
accessible
and
easy
to
use
while
at
the
same
time
maintaining
lives
real-time
system
reliable.
B
So
in
particular
for
this
migration
we
have
three
three
goals
and
on
the
Left
we
have
the
goal
that
we
have
for
product
engineers.
This
change,
as
we
move
to
new
infrastructure,
has
to
be
transparent
to
them.
They
they
can.
We
can
burn
our
product
engineers
with
knowing
the
internals
of
our
infrastructure,
for
example,
when
appropriate
genera
wants
to
make
a
request,
a
service
foo
they
shouldn't
need
to
think
about.
Is
this
request
going
to
a
pod,
managed
by
kubernetes
or
to
a
virtual
machine
in
AWS?
The
request
should
be
transparent.
B
They
should
they
should
just
send
the
request
and
it
should
be
managed
by
by
the
proxy.
So
what
that's
the
goal
for
that
we
have
for
product
engineers.
We
also
us
infrastructure.
Engineers
have
two
particularly
important
goals:
to
keep
lift
services
and
platform
reliable.
The
first
one
is:
we
want
to
design
the
system
such
that
we
can
do
an
incremental
rollout
of
this
new
infrastructure
and
second,
while
we
try
to
have
the
most
reliable
infrastructure
and
release
it
in
safe
ways,
we
also
need
to
preserve
the
ability
to
rollback
when
problems
occur.
B
This
is
the
set
of
XDS
API
is
that
Android
has
described,
which
are
a
set
of
API?
Is
that
can
dynamically
configure
the
proxy
from
a
centralized
management
server?
So
here
we're
separate
the
actual
data
plane,
that
is,
that
requests
are
flowing
through
from
the
control
plane
that
might
configure
how
that
data
plane
is,
is
managing
requests
so
specifically
for
service
discovery
and
what
it
has
implemented
implemented.
B
The
eds
api,
which
stands
for
endpoint
discovery
service
endpoint,
is
just
a
specific
entity
that
responds
to
a
logical
set
of
load
balancing
entities.
So
specifically,
the
XDS
api's
are
a
set
of
G
RPC
API
is
that
from
a
central
control,
plane
can
configure
the
data
plane
and
it
separates
a
concern
of
configuration
from
actual
requests
person.
A
So
I'm
gonna
go
dive
into
like
the
hybrid
service
discovery
service,
discovery,
architecture
that
we
came
up
with
so
as
you
are
from
before.
We,
although
sidecar
on
voice
running
a
separate
processes,
are
registering
the
IP
address
of
that
machine
into
the
crowd
discovery
service.
But
now
we
also
have
these
kubernetes
clusters,
where,
where
they
we
have
multiple
clusters
and
there's
a
pot
pocket.
We
have
services
as
pods
and
they
are
charted
between
mini
clusters
and
we
design
it
that
way
so
that
if
one
cluster
will
go
down,
we
could
still
in
encrypted
entities.
A
We
can
still
serve
traffic
from
the
other
clusters
and
there's
a
kubernetes
api
server.
That
is
aware
of
the
state
of
the
cluster,
and
so
we
migrated
over
to
the
Envoy
xes
control
plane,
which
is
a
go
service
that
had
that
is
up
created
the
configuration
to
the
new
XDS
api's
and
it's
pulling
in
data
from
the
old
discovery
service
and
also
making
connections.
A
It
uses
the
kubernetes
watch
api's
to
get
this,
get
the
IP
addresses
of
each
pods.
That
of
that
cluster
and
then
converts
all
that
data
into
and
the
eds
configuration
and
sends
that
off
to
the
side,
car
envoys
and
now
that
now
that
I
have
that
data.
They
know
about
each
other
and
they
could
start
routing
traffic
to
each
other
and
know
here
the
note
here
that
in
our
communities
clusters
there's
no
ingress
controller,
the
pods
are
able
to
get
an
IP
address.
A
That
is
aware
in
our
V
PC,
and
so
that
means
that
the
legacy
machines
can
talk
to
that
pod
directly
and
there's
many
reasons
why
we
went
with
this
route.
There's
a
portion.
We
don't
have
time
to
go
over.
Why
we
went
that
way
today.
But
if
you
want
to
talk
about
how
we
did
that
and
why
we
did
that,
you
could
come
talk
to
me
or
my
colleague,
Paul
is
here
and
he'll,
be
happily
talk
to
you
about
how
we
made
that
possible,
but
yeah.
A
So,
at
the
end
of
the
day
that
we,
because
we
are
able
to
send
those
IP
addresses
out
to
all
the
sidecar
envoys,
they
now
form
that
mesh
like
like
now,
there's
no
difference
between
the
legacy
boxes
versus
the
kubernetes
boxes
and
they
could
have
servers.
This
can
be
split
between
kubernetes
then
and
the
legacy
infrastructure,
but
we
this
at
the
end
a
day
they
get
converted
and
merge
and
could
serve
traffic
to
each
other
so
like
this
is
great.
A
So
the
reason
why
we
went
with
this
approach
is
the
change
is
transparent
to
the
product
engineers.
They
could
still
write
application
code
and
resend.
A
request
and
envoy
will
figure
out
how
to
route
that
request
to
and
send
basically
I
get
a
response
back.
We
can
now
do
incremental
rollouts
because
we
can
using
envoys
xes
api's.
We
can
like
set
priorities,
so
we
could
put
a
lower
priority
on
the
kubernetes
clusters
or
we
can.
A
A
B
The
first
one
that
we
want
to
talk
about
is
eventual
consistency,
it's
at
friend
or
foe.
So
the
way
that
service
discovery
works
in
Envoy
is
it's
an
eventual
consistent
system
where
it's,
the
the
ability
of
a
particular
Envoy
to
route
traffic
to
another
envoy
is
based
on
on
this
matrix,
where
it
depends
both
on
the
data
that
the
discovery
system
is
sending
via
the
EDS
API,
but
also
based
on
the
health
check
status
between
one
envoy
and
the
other
one.
B
If
envoy
has
already
known
of
that
of
those
hosts
in
a
previous
deploy
of
the
discovery
system,
then-
and
we
will
still
will
still
have
that
host
in
its
local
knowledge
of
the
mesh,
even
if
they
discover
if
the
discovery
system
is
failing.
This
allows
us
to
more
rely
and
more
confidently
deployed
changes.
Our
discovery
system.
B
Why
I
say
it
might
also
be
faux?
Is
that
because
we
have
an
eventually
consistent
discovery
system
as
we
roll
out
changes,
changes
are
not
impacting
the
entirety
of
the
topology
at
once.
This
changes
take
time
to
roll
out,
and
so,
as
an
operator
when
you're
deploying
changes
to
service
mesh,
you
might
expect
that
metrics
changed
more
rapidly
that
they
actually
do,
and
so
you
might
have
to
be
able
to
monitor
and
think
about
problems
not
immediately,
but
in
in
the
long
term,
as
the
mesh
converges.
A
B
Well,
today,
we've
talked
about
service
discovery
and,
in
a
larger
sense,
the
network
team.
This
hybrid
infrastructure,
as
I
said
previously,
is
something
that
happens
for
for
any
type
of
of
migration
for
any
time
type
of
system
where
you're
moving
from
leg
see
infrastructure
to
a
new
infrastructure.
While
today
we've
only
talked
about
kubernetes
and
and
our
legacy
virtual
machines,
the
principles
here
of
maintaining
a
hybrid
infrastructure
can
apply
to
other
topologies
like
multi
cloud
or
multi
tenant.
That
is
all
that
we
had
for
today.
A
B
A
B
B
Active
health
check
in
envoy
works
with
with
certain
parameters.
We
have
it
such
that
you,
you
have
check
on
an
interval
right,
and
so
when
that
health
check
starts
failing
and
we
will
drain,
they
will
drain
the
connection.
So
it
will.
It
will
still
finish
the
requests
that
are,
you
know
in
flight,
but
it
will
not
accept
new
connections
to
that
particular
stream.
So
any
other
requests
that
might
have
happened
to
that
logical
cluster
will
be
load
balance
to
other
healthy
endpoints.
C
A
B
A
Right
so
the
question
I
believe
if
I
understand
it
is
that
how
do
you
deal
with
the
scaling
other
clusters
as
well
so
from
in
that's?
For
that
perspective,
we
I
think
provisioning
is
coming
up
with
a
plan
to
scale
up
like
make
sure
that
there's
like
a
pool
of
nodes,
so
there's
oh
they're,
always
going
to
be
a
resources
available
and
then
like
we,
and
basically
there
should
be
enough.
A
If
there's,
if,
as
more
pots,
spin
up
and
there's
less
nodes
available
for
resources,
they
say
they
start
scaling
up
so
we're
getting
ahead
of
it
right
by
a
like.
So
there
would
be
there
always
gonna,
be
a
pool
available
for
more
workloads
and
that's
provisioning
job
to
make
sure
that
that
they
they're
always
going
to
be
enough
to
enough
nodes
available
to
launch
more
pods
onto
the
kubernetes
cluster
and.
B
A
C
C
B
Yes,
so
the
question
is:
how
does
this
topology
of
the
hybrid
ecosystem
relate
to
the
front
proxy?
So
the
front
proxy
is
also
reading
discovery
or
EDS
data
from
the
XDS
control
plane.
So
it's
essentially
the
same
right.
The
front
proxy
has,
for
example,
a
logical
cluster
foo
and
within
that
logical
cluster
foo
we
might
have
endpoints
that
are
in
kubernetes
or
in
the
legacy
discord
or
in
the
legacy
system.
So
the
same
way
that
the
service
mesh
discovers
you
know
particular
hosts
for
particular
services.
The
front
proxies
the
season,
the
same
system.
B
B
The
question
is:
do
we
have
any
plans
to
use
SEO
as
our
control
plane,
I
have
a
full
talk
or
in
control
planes?
Basically,
my
basic
answer
here
is
control.
Planes
are
very
opinionated,
I
always
describe
them
as
a
service
in
three
parts
and
the
front
part
of
a
control
plane
is
munging.
You
know
your
do.
Infrastructures
topology
into
the
XDS
api's
for
lyft
in
particular,
because
you
know
we
have.
We
carry
the
history
of
Android
with
us
and
a
very,
very
opinionated
topology.
It
makes
no
sense
for
us
to
at
this
point.
B
D
B
So
the
question
is
in
in
this
environments,
where
we're
relying
on
health
check
into
actually
doing
traffic,
shifting
and
and
popping
endpoints
out
of
out
of
a
logical
service.
How
do
you
deal
with
Grayson
errors
where
an
endpoint
might
be
not
responding
correctly,
but
it
might
still
be
responding.
Health
checking,
that's
what
I
was
talking
about.
The
power
of
the
EDS
API
is
that
you
can
specify,
for
example,
load
balancing
priorities,
so
we
might,
as
network
operators,
open
up
an
API
in
our
control
plane
that
can
say,
okay,
mark
or
mark.
B
You
know
this
endpoints
as
unhealthy,
not
by
failing
health
check
on
the
on
the
machine
itself,
but
by
delivering
discovery
data
that
says
you
know
this
has
a
priority
of
you
know
negative
1000,
so
that
traffic
will
not
be
routed
there.
So
so
that's
one
of
the
ways
that
having
this
having
this
topology
has
to
resist
this
migration,
because
we
have
several,
you
know,
checks
and
balances
or
flags
that
allow
us
to
shape
traffic.
The
way
that
we
want
I
think
we
are
a
time.
Thank
you.
So.