►
From YouTube: Evolution of Envoy as a Dynamic Redis Proxy - Nicolas Flacco, Henry Yang & Mitch Sulaski
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Evolution of Envoy as a Dynamic Redis Proxy - Nicolas Flacco & Henry Yang, Lyft | Mitch Sulaski, Workday
This session will go over the evolution of Redis support in Envoy. Initially Envoy redis proxy only supported sharding to clusters of independent Redis nodes. Recent developments have enabled support for the open source Redis Cluster protocol as well as some unique features such as multicluster routing, flexible load balancing options, and traffic shadowing.
As the usage of Redis expanded different usage patterns emerged, requiring different availability, durability and consistency trade-offs. Henry and Mitch will discuss how the Envoy redis proxy was extended to support these new requirements in large scale environment(10+ Millions rps) at Lyft and Workday.
Traditionally migrating data between Redis clusters has been painful, requiring lots of application code changes and extensive monitoring. Now, it is possible to move traffic between Redis clusters with zero application code changes, using Envoy itself. Nicolas will discuss his experience at Lyft migrating Redis clusters.
A
Have
the
easy
part?
Tell
you
about
what
we're
gonna
actually
talk
about
in
this
talk?
First,
we
could
start
out
with
a
little
bit
about
why
envoy
and
then
Henry
is
going
to
jump
into
the
actual
evolution
of
redis
support
and
we'll
show
you
some
of
the
deeper
internals
of
what's
going
on
and
then
Nico
is
going
to
actually
cover
online
data
migration
and
as
a
real
production
use
case
and
then
we'll
end
up
a
little
bit
with
what's
next,
and
hopefully
we'll
have
time
for
Q&A.
A
One
of
the
things
that
envoy
is
to
do
is
support
complex
behavior,
that
we
want
to
implement
in
the
proxy
and
relieve
the
Redis
client
from
actually
having
to
to
implement
or
do
correctly,
and
this
actually
helps
quite
a
bit
and,
like
I
said
we
can
support
very
complex
operational
environments.
We
can
handle
dynamic,
configuration
changes,
we
can
implement
smart,
read
policies,
we
can
federate
over
multiple
clusters
using
things
like
prefix
routing,
and
then
we
can
do
zero
code
change
online
data
migrations
all
just
by
using
a
Envoy
as
a
single
control.
A
Point,
of
course,
as
you've
heard
from
all
the
talks
earlier
today,
we
also
get
observability,
which
also
helps
quite
a
bit
in
any
sort
of
operational
environment
and
I'd,
also
like
to
point
out
that
really
what
Envoy
provides
that
that
we
find
at
work
day
very
useful
and
folks
that
live
feel
very
useful.
Is
that
it's
flexible,
its
flexible
and
it's
flexible?
A
You
can
deploy
in
any
sort
of
environment
which
is
really
important
in
sort
of
complex
environments
where
your
movie
and
say
from
you
know
a
self-hosted
private
cloud,
implementation
of
your
own
set
of
jury-rigged
protocols
and
and
products
to
something
like
kubernetes
and
vastly
performance
on
voice,
worker
thread
architecture
and
the
ability
to
do
Retta,
spy
planning
and
a
variety
of
other
things.
The
focus
on
performance
is
conceived
in
to
the
architecture
and
it's
something
that
I
feel
like
you'll
you'll
see
in
Henry's
talks
when
we
start
going
through
the
internals.
B
All
right,
so
I
am
just
going
to
go
with
kind
of
rip
the
band-aid
off
approach
and
show
you
kind
of
what
what
the
envoys
are.
Reddy's
proxied
looks,
light
and
the
red
is
proxy
really
evolved
over
time,
and
this
is
where
it's
in
its
first
iterations
and
it
actually
would
be
an
interesting
case
to
look
that
are
to
extending
on
voids
as
a
proxy
for
other
stateful
services.
B
So
in
these
iterations,
if
we
look
under
the
hood,
you
can
see
there
are
a
lot
of
boxes
so
to
kind
of
look
at
it,
we
can
go
from
left
to
right
when
the
client
makes
the
connections
are
to
the
envoys
eye
card.
First
thing
we
see
is
the
filter
things
where
we
Dakotah,
that
is
reading
the
binary,
reddest
promo
code
from
from
the
connections,
and
we
decode
that
and
go
into
the
readies
filter.
B
B
So
because
a
single
command
can
spend
multiple
keys
and
those
keys
are
actually
stored
in
different
nodes,
so
we
need
to
split
them
to
create
different
requests,
that
we
can
then
route
to
the
correct
collaborators
nodes
and
then
once
we
get
the
response,
we
need
to
also
use
this
command
sweeter
to
aggregate
them
and
return
to
the
client
and
next
lap
is
where
we
have
the
connection
pool.
This
is
where
a
lot
of
the
interesting
thing
that
are
happening
as
far
as
proxying
is
concerned.
B
When
we
are
talking
about
proxying
to
a
cluster
of
nodes,
one
of
the
things
we
need
to
think
about
is
actually
what
is
the
cluster
or
a
very
related
question
is
how
do
we
do
service
discovery?
How
do
we
discover
and
update
the
membership
of
this
cluster
and
in
the
first
iteration,
a
cluster
is
very
simple:
it's
a
flat
list
of
nodes
that
we
are
interested
in.
It
then
the
connection
pool
knowing
what
the
holes
are.
Then
we
can
start
making
connections
and
sending
our
routing
requests.
B
So
in
this
case,
we
actually
can
just
use
the
other
box
Casta
implementations
for
envoys,
where
we
can
either
use
DNS
or
XPS
forecasted
discovery
service
and
in
in
any
environment.
So
we
actually
don't
need
to
change
that.
The
next
thing,
when
we're
talking
about
sending
traffic
to
multiple
nodes,
is
how
do
we
distribute
the
different
command?
And
this
is
where
the
low
balancer
comes
in
envoy,
come
out
of
box
with
a
large
number
of
load,
balancers,
randoms,
power
of
two
etc
etc.
B
However,
unlike
a
lot
of
other
proxies,
where
we're
talking
to
stateless
service,
things
like
randoms
or
round-robin,
actually
doesn't
make
sense
for
stateful
service,
because
we
need
to
make
sure
the
same
key
is
always
as
as
much
as
possible
route
it
to
the
same
Redis
note.
So
luckily,
we
have
kind
of
the
alpha
box
consistent,
hashing
load
balancer,
which
is
the
ring
hash,
which
is
the
Kodama
hashing
algorithms,
that
everyone's
very
familiar
with
consistent
hashing
ring
or
the
Maglev
hashing,
and
we
can
in
this
case
again
because
I'll
note
just
a
flat
list
of
nodes.
B
We
can
just
use
those
out
of
the
box
and
then
for
the
up.
That's
where
the
filter
chains
are
for
upstream,
so
each
guy
is
is
very
local,
so
we
can
guarantee
performance
without
doing
any
locks
and
then
is
the
rivers
of
our
downstream,
where
we
encode
what
we
have
understand
as
the
Redis
connections,
radix
command
and
request,
and
then
we
send
upstream
to
Redis
node
somewhere
it
could
be
in
kubernetes.
B
It
could
be
in
your
other
G's
or
yeah,
and
then
once
we
get
the
response,
we
travel
backwards
and,
of
course
we
also
have
to
extend
the
configuration
so
that
this
can
all
be
configured.
And
on
top
of
that
we
get
the
amazing
sets
framework
and
the
admins
endpoints
that
we
can
look
at
all
the
metrics
and
the
cluster
memberships
and
also
out
of
the
box
envoy,
provides
hot
restart
so
and
the
configuration
can
be
dynamic,
meaning
compared
to
what
these
was
kind
of.
B
So
this
looks
pretty
good,
but
in
real
life
failures
happens
and
we
need
to
know
how
to
handle
that
failure,
and
in
order
to
do
that
better,
we
want
to
have
both
pathy
passive
and
active
health
check
so
for
passive
health
track.
It's
actually
quite
simple.
We
just
make
sure
whenever
we
get
a
failure
in
the
client
that
we
register
with
the
Allied
detection.
That
is
provide
the
other
box
for
envoy.
B
But
what
about
for
active
health
check?
How
do
we
actually
know
a
radius
know
is
actually
healthy
before
it
gets
removed
from
the
discovery?
Well,
we
actually
have
to
implement
a
Redis
health
checker
again
we
leverage
the
same
Redis,
encoder
and
decoder
to
send
either
pink
or
exists,
exists,
requests
to
every
single
Redis
note,
so
you
might
be
wondering,
like
for
health
check,
isn't
a
successful
being
enough.
B
So
at
this
point
we
have
a
pretty
well
functioning
Redis
proxy.
It
even
survived
Network
partitions,
which
is
one
of
the
hard
things
to
do,
because
if
there
is
a
pairwise
network
partitions,
we
just
our
health
checker
would
detect
that
and
it
would
evict
it
from
the
node
and
it
would.
The
law
will
be
roughly
to
just
a
different
note.
So
what
more
can
we
want?
B
B
And
you
survive,
then
we
partition,
we
want
consistency
and
it's
it's
a
reasonable
request,
because
for
some
services,
where
consistency
is
more
important,
they
would
rather
evict
the
node
that
is
having
network
partitions,
so
that
everyone's
would
always
write
the
same
key
to
the
to
the
same
Redis
nodes
rather
than
having
kind
of
working
in
a
split
brain
situations.
And
secondly,
when
we're
dealing
with
stateful
services,
sometimes
having
more
predictable
failure.
B
Behavior
is
is
important
when
we
remove
a
node
from
a
consistent
hashing
ring
they
it's
it's
not
very
predictable
on
where
the
shards
would
be
landing
on,
and
it
is
possible
that
we
can
have
multiple
hotkeys
landing
on
a
new
node
and
also
we
want
to
be
able
to
say,
hey
like
exactly
what
the
performance
would
look
like
and
if
we
have,
which
is
kind
of
related
to
the
next
points
are
related
to
replications.
Is
we
want
to
be
able
for
on
a
replica
if
possible,
rather
than
just
hashing
to
the
rest
of
the
cluster?
B
Also
in
the
last
couple
of
years,
there
is
really
strong
community
support
and
momentum
going
towards
embracing
the
Redis
cluster
protocol
way.
It's
about
some
of
these
features
and
what
are
some
of
the
challenges
of
adding
these
well
prior
to
trying
to
implement
this
feature
on
voice
service
discovery.
Discovery
mechanism
is
Bill
as
part
of
own
voice
and
obviously
we
don't
want
to
be
ready,
specific
implementation
into
the
call
of
own
voice.
B
So
we
need
to
have
a
mechanism
to
extend
that
and
to
come
up
with
the
Redis
cost
the
protocol
we
have
a
more
complicated
topology.
Well,
especially
when
we
want
to
support
things
like
replications.
Now
we
don't
just
have
a
flat
list
of
note.
We
have
primaries
and
we
have
replicas,
we
have
concepts
of
slots
and
wear
and
shot
and
another
thing
that
will
come
with
the
Redis
protocol,
so
the
Redis
Perl
code.
B
Basically,
is
you
have
a
consensus,
a
gossip
promo
code
on
the
on
the
Redis
castle
itself
to
determines
what
the
topology
will
be
and
then
you
can
make
amends
to
it
to
say,
hey
like
tell
me
what
the
topologies
are
and
if
you
send
the
requests,
let's
say:
you're,
your
discovery
is
is
is
out
of
date.
You
send
commands
to
the
wrong
note.
B
B
Well,
that's
what
the
next
iteration
looks
like,
so
we
implemented
a
extensions
that
would
handle
the
raddest
custard
discovery
so
on
on
initializations.
We
would
get
some
list
of
seed
clusters
and
then
we
make
the
Redis
commands
to
those
to
call
the
cluster
slots
to
get
the
topology,
and
then
we
use
that
to
inform
the
connections
now
on
the
data
pipelines.
B
When
we
make
a
request
to
read
this
note
and
then
it
come
back
with
either
ask
or
move
basically
in
redirections,
then
we
will
just
resend
the
request
to
the
new
nodes,
and
we
are
asked
to
redirect
to
know
that
we
are
not
currently
aware
of
that
will
create
a
temporary
host
in
the
connection
pool
and
and
then
I
would
try
to
drain
those
once
they
discover
in
the
next
time.
We
we
do.
B
The
discovery
one
of
the
small
trade
off
of
doing
this
is
now
we
have
to
hold
on
to
the
request
for
longer
because
in
the
case
like
before,
we
will
get
the
request.
We
send
it
off
and
we
can
create
that
from
the
memory.
But
now
we
have
to
hold
on
until
we
get
either
error
or
a
failure
response,
because
there's
a
chance
that
we
need
to
resend
that
so
there
is
a
small
increase
in
memory
footprint.
B
So
at
this
point
we
are
not
making.
We
don't
need
to
make
any
changes
to
the
load
balancer,
and
this
is
what
the
open
source
readies
community
called
a
dumb
client.
You
could
actually
even
use
a
random
a
load
balancer,
because
what
we
rely
on
the
correctness
is
that
you
doesn't
matter
you
send
it
to
any
Redis
note,
and
if
you
get
it
right,
it
gives
you
the
answer
you
want
and
if
you
get
you
get
it
wrong.
B
It's
sent
back
a
redirection
to
point
you
to
the
correct
note,
but
obviously
I
mean
we
are
working
in.
Although
we
don't
want
anything
dumb,
so
we
will
find
a
way
to
to
get
around
that,
so
we
need
to
create
a
reddest,
specific
load
balancer
that
can
understand
this.
That
can
handle
these
protocols,
so
it
has
to
understand
the
the
cluster
topology.
It
has
to
understand.
B
Only
in
that
case,
you
say:
hey
I
want
to
make
sure
I'm
consistent,
always
I
can't
tolerate
in
any
inconsistency
and
the
replica
is
there
only
to
serve
fair
over?
Oh,
you
can,
you
can
say:
hey
I
want
to
send
to
the
master
if
it's
available,
it's
it's
healthy
and
if
not
I,
don't
want
to
wait
for
the
load.
Balance
I
just
want
to
send
the
require
green
request
to
replicas.
B
So
what
does
it
look
like
once
we
implement
this,
then
we
we
have
to
come
up
with
a
new
mechanisms
for
the
cluster
itself
to
provide
the
load
balancer
rather
than
the
load
balancer
being
for
its
configurations
on
the
listener
and
that
looked
pretty
straightforward.
The
Redis
Costner
would
update
the
Redis
load
balancer
with
all
the
shots,
and
it
actually
is
we.
We
now
don't
have
to
recompute
this
topology
mapping.
B
Similarly,
with
health
check,
we
find
that
in
practice
we
have
this
dilemma
whether
if
we
want
to
have
a
very
quick
failover
times
when
a
no
fail
or
when
we
add
a
note,
we
have
to
have
very
short.
If
we
only
rely
on
periodic
updates,
we
have
to
have
a
really
short
refresh
cycles
like
we
would
be
pinging
the
Redis
cluster
for
topology
up
there
all
the
time.
Oh,
we
have
to
tolerate
a
long
period
of
increase
our
redirections.
B
B
B
We
want
to
be
able
to
mirror
to
multiple
clusters,
maybe
cross
regions
so
or
maybe
we
want
to
have
on
the
manage
where
this
all
these
commands
goat-like.
We
can
have
a
share
Custer
with
different
prefix,
and
then
we
can
route
them
just
like
how
we
draw
HTTP
requests,
and
so
we
we
added
this
router,
where
we
can
wrap
a
request,
not
just
on
what
notes
in
the
classroom,
but
across
multiple
classes
and
Nico
Nico.
He
is
going
to
talk
about
a
very
interesting
case
where
we
use
these.
C
B
C
So
over
the
last
year,
we've
done
all
this
hard
work
and
the
real
world
application
is
migrating
millions
of
QPS
to
Redis
cluster.
At
lyft
we
have
hundreds
of
clusters,
something
like
40
million
QPS
or
more
during
peak
ordering
load
tests.
The
biggest
clusters
500
nodes
with
you
know
a
quarter
of
that
whole
load.
C
These
clusters
aren't
all
just
straight-up
caches.
Some
of
them
are
more
like
transient
data
where
there's
a
source
of
record,
and
you
can't
just
throw
away
to
data
and
rehydrate
from
you
know,
dynamo,
let's
say,
and
these
services
are
in
Python,
go
lots
of
different
languages
and,
like
Henry
was
saying
about
Redis
clients.
You
know
each
language
kind
of
has
its
own
implementation
and
with
Redis
cluster,
where
you
have
a
smart
client,
this
becomes
really
critical
because
you
want
your
topology
updates
to
be
the
same.
You
and
a
lot
of
that
behavior
to
be
consistent.
C
So
how
do
we
migrate
all
these
things?
You
know
zero
plan,
downtime
zero
outages,
zero
data
loss.
How
do
we
do
that?
And
the
answer
of
course
is
envoy,
as
you
guessed
traditionally,
Redis
migration
has
been
really
painful.
You
know
you
look
across
all
the
applications
that
might
be
using
a
given
Redis
cluster
and
like
a
regular
itís,
not
just
Redis
cluster,
and
you
know,
instrument
everything
do
right.
C
If
things
look
all
right,
then
you
do
read,
and
then
you
kind
of
hope
and
if
you're
lucky
it
works
well
otherwise
need
to
go
back
and
you
know
monitor
the
Redis
check
where
all
the
connections
are
coming
from
and
with
Envoy.
You
don't
need
to
do
this
anymore.
You
know
you
can
really
can
know
where
everything's
going.
If
you're
already
talking
to
Redis
be
Envoy,
you
have
consistent
stats.
You
don't
need
to
make
any
changes
to
the
application.
With
our
you
know,
cluster
support.
You
don't
need
to
a
cluster
at
his
client.
C
That's
really
the
key
part,
because
you'll
have
now
consistent.
Behavior
across
everything
envoy
manages
the
apology
with
the
stats.
You
know.
One
thing
owner
to
call
out
is
that
because
you
can
get
stats
in
the
upstream,
not
just
the
downstream
right,
you
know
can
traditionally
just
look
at
her
downstream
stats.
But
if
you
have
multiple
up
streams,
where
you're
mirroring
across
traffic,
you
can
just
kind
of
individual
per
upstream
request
stats,
and
then
we
also
have
some
special
rhetta's
command
stats.
C
So
you
can
get
kind
of
per
command
counts
and
really
get
more
confidence
that
things
are
all
in
sync
and
then
you
know.
Finally,
you
can
mirror
the
traffic
from
one
Redis
to
another
and
like
Henry
was
talking
about
you
know
we
have
prefix
routing
which,
if
you've
unique
prefixes,
allows
you
to
compose
and
decompose
traffic
from
different
services
which
becomes
very
powerful
later
on
soo.
C
Let's
now
talk
about
how
a
migration
looks
practically
speaking,
it's
a
four-step
process.
If
we
have
a
single
service
to
talks
to
Redis
via
envoy
like
we
do
it
lifts,
it's
really
straightforward.
You
know
assuming
you've
provisioned
your
new
Redis
you'll,
then
just
add
the
Redis
in
the
cluster
section
on
droid
config,
and
you
know
immediately
you'll
start
to
see
health
checks
come
in.
You
know.
If
you
go
and
read
us
Redis
and
monitor
or
the
connections
you'll
see
the
connections
and
you
don't
have
to
add
a
listener
for
the
new
Redis.
C
If
you
want
to,
you
know,
you
could
just
to
make
sure
things
are
up
and
running,
but
thistles
enough
to
get
you
started.
The
next
step
is
adding
a
mirror
policy,
so
this
is
where
we
essentially
just
tell
it.
You
know
you
can
shift
this
or
you
can
copy
this
much
traffic
over
to
new
Redis
and
it's
useful
not
just
for
arena
warming,
the
cache
but
also
load
testing
and
sizing,
and
it's
pretty
performant
for
that.
C
C
Then
we
essentially
can
mirror
for
a
while
in
our
max
TTL
we
might
new
in
a
bake
a
little
bit
longer
if
we
have
a
new
computer
environment
like
moving
to
kubernetes
or
a
new
cloud
provider
and
then
once
we're
there,
we
can
validate
our
data.
So
for
lift
migrations
were
chosen,
essentially
just
you
know
say:
given
a
reddish
shard
in
our
old
Redis.
C
Let's
get
all
the
keys
and
then
run
those
keys
through
envoy
for
old
new
and
compare
them
by
data
type,
and
if
we
have
a
under
a
certain
threshold
differences
we're
good
now
mirroring
has
one
case
where
it
won't
work,
and
this
is,
if
you
have
data
that
doesn't
have
a
TTL
like
you're,
using
Redis
as
a
queue.
So
you
know
there
you
need
to
do
some
sort
of
copying
su.
C
Finally,
you
can
essentially
switch
over
traffic
now.
One
thing
I
want
to
note
in
this
diagram
up
here
is
that
we
do
a
reverse
mirror
to
keep
the
data
in
sync
on
the
old
Redis.
So
we
can
bake
you
know,
especially
if
we're
moving
like
ElastiCache
or
a
new
environment.
You
know
just
let
it
sit
for
awhile
and
we
can
switch
back
in
emergency.
C
The
other
point
here
to
note
is
that
we're
not
doing
a
selective
read
really
like
you
might
do
normally
we're
just
in
the
listener,
we'll
just
say
you
know
pointer
to
do
cluster
and
then
change
your
mirror
policy
to
pointer
to
old
cluster,
but
we've
been
weaving
reading
and
writing
to
both
clusters.
The
whole
time.
So
it's
not
really
that
different.
C
We
have
stats
to
basically
show
everything's
working
and
then
we've
done
a
validation,
and
you
know
then
we've
baked
a
little
bit
longer
and
you
get
ready
to
turn
off
all
the
Redis
and
then
we
do
that.
We've
done
this
for
a
number
of
lift
services
and
it's
worked
pretty
well
and
we've
done
tens
of
clusters.
You
know
millions
of
QPS
and
we
haven't
had
an
outage.
Yet
it
makes
me
very
happy.
So.
C
Finally,
what's
next
we'd
like
to
support
some
more
commands,
we'd
like
to
support
new
data
structures
like
streaming,
we
wondered
implement
fault
injection
next
year.
Http
side
of
envoy
has
this,
but
Redis
does
not.
You
know
with
chaos
monkey
we
can
kill
a
node,
but
we
really
want
to
have
more
fine-grained
control.
We
also
want
to
do
stuff,
like
add,
could
consistent
retries
across
all
of
our
Redis
users.
Just
like
you
know
you
have
it
the
stats
and
then
we'd
like
to
add
rate
limiting
and
finally
compression
when
people
start
storing
big
blobs.