►
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Spanning the Globe with Envoy at Stripe - Dylan Carney, Stripe
Stripe operates compute infrastructure around the globe in order to provide low latency and high availability to its users, using Envoy to connect this infrastructure together.
To ensure reliability and operability, Stripe built an Envoy control plane, along with new tooling to manage it. New XDS services provide features like ramp-up of traffic to specific compute clusters, tiered failover, and per-customer routing of API requests. Stripe also built their blue/green deployments on top of this. Finally, they made improvements to make Envoy significantly more reliable in the face of issues like packet loss and head-of-line blocking, based on traffic patterns and behavior observed in production.
Attendees will learn details about these projects and get to hear war stories, cautionary tales, and valuable lessons learned during the process of building out a globe-spanning Envoy topology.
A
My
name
is
Dylan
Kearney
I'm,
an
engineer
on
the
traffic
team
at
stripe
and
I'm
here
today
to
talk
to
you
about
some
things
that
my
team
built
with
envoy
over
the
past
year
or
so
I'm,
going
to
talk
about
what
we
built
and
how
we
incorporated
envoy
into
what
we
built
some
of
the
challenges
we
faced
along
the
way.
Maybe
some
of
them
are
interesting
or
novel,
depending
on
your
experience
with
envoy
and
then
with
the
benefit
of
hindsight
and
looking
back
at
the
entire
project,
what
were
some
pros
and
cons?
A
Maybe
some
things
we
would
have
done
differently
so
with
that
before
I
dive
into
any
sort
of
technical
details,
but
just
start
rambling
on
about
HTTP
or
on
vallei
a
bit
about
my
employer
stripe.
Some
of
you
might
be
developers
in
the
audience.
You
might
have
integrated
with
stripe
at
one
point
or
you
might
work
for
a
company
that
uses
stripe.
I
know
I
saw
it
up
on
a
slide
earlier,
but
stripe
is
a
technology
company
that
builds
economic
infrastructure
for
the
internet.
So
what
does
that
actually
mean
another
way
to
think
about?
A
That
is
that
we
run
software
and
api's
to
accept
payments
and
to
allow
companies
to
manage
their
businesses
online.
So
a
large
part
of
that
is
removing
the
complexity
of
global
money
movement.
So
we
have
millions
of
customers
in
over
120
countries.
The
use
stripe
to
start
run
and
scale
their
business.
A
Now
I'm
a
software
engineer,
I'm
sure
a
lot
of
you
are
as
well
and
anyone
who's
been
in
the
software
engineering
or
computer
science
fields
for
a
while
knows
that
the
field
is
littered
with
problems.
Problems
to
work
on
some
of
them
are
maybe
a
little
more
interesting
than
others.
Depending
on
your
your
background,
we
have
problems
of
resource
constraints.
A
We
have
problems
of
taming,
overwhelming
complexity,
but
I
would
argue
that
having
millions
and
millions
of
people
all
around
the
world
who
want
to
use
your
software
all
day
every
day
with
many
nines
of
availability
is
one
of
the
greatest
problems
to
have
it's
fun
to
work
on.
It
turns
out
from
a
business
perspective,
it's
also
a
pretty
great
problem
to
have,
as
you
can
imagine,
so
that's
the
problem
that
I
want
to
talk
about
today
and
how
my
team
at
stripe
overcame
some
of
the
inherent
obstacles,
without
particular
software
engineering
problem.
A
Oh
sorry,
I
didn't
even
read
my
slide
here.
Another
way
of
saying
that
is
with
increased
reach
with
global
reach
comes
some
pretty
interesting
software
problems,
so
the
first
one
is
a
basic
tale
of
latency.
It's
a
fundamental
problem
when
dealing
with
networks.
Now,
if
we
look
at
the
time
it
takes
some
data
to
make
its
way
from
Seattle
where
I
live
and
where
I
work
to
where
we
are
today
here
in
San,
Diego
we're
looking
at
I,
don't
know,
depending
on
your
source,
about
36
milliseconds.
A
Now,
if
we
move
that
out
to
a
city,
I
was
recently
in
Mexico
City,
we're
now
looking
at,
say:
67
milliseconds,
give
or
take
a
few,
but
if
we
move
further
afield
and
look
at
parts
of
that
globe,
that
I
showed
earlier
much
farther
from
Seattle,
like
Mumbai
say,
were
up
to
a
pretty
appreciable
275
mm
seconds.
I
don't
need
to
beat
you
over
the
head
with
examples
for
you
to
get
the
point
that
some
places
are
much
further
away
than
others.
The
speed
of
light
is
a
constant.
A
It's
not
moving
any
faster
Mumbai
is
not
getting
any
closer
to
Seattle
last
time,
I
checked
so
one
one
way
you
might
think
of
to
combat
this.
It's
not
just
a
naive
solution,
but
one
that
actually
works
pretty
well
in
practice.
It's
to
move
some
of
your
computers
closer
to
where
those
users
are,
it
makes
intuitive
sense
and
it
makes
fundamental
physical
sense
as
well.
A
Now
the
significance
of
the
different
colors
on
the
arrows,
the
red
arrows,
represent
traffic
that
moves
over
a
connection
that
might
be
the
users
internet,
say
mobile
network,
their
home
broadband,
their
office
what-have-you,
whereas
Green
Arrow's,
represent
connections
or
traffic
over
connections
run
and
operated
by
stripe.
That
would
become
kind
of
important
in
a
second
here.
So
great
we've
established
points
of
presence
or
pops
as
I
sometimes
refer
to
them
in
places
all
around
the
world
they're
closer
to
users.
Well,
what
is
that
bias?
A
The
first
thing
we
can
do
is
drastically
reduce
the
TLS
negotiation
time,
so
any
good
API
is
going
to
require
that
you
establish
a
secure,
TLS
connection
before
doing
anything
else,
and
if
we
remember
real
quick
back
to
maybe
our
computer
networking
classes
we
took
in
school.
We
know
that
just
to
set
up
that
TLS
connection
before
we
can
do
anything
else,
we
have
to
do
a
bunch
of
round
trips.
So
between
the
center
and
the
receiver,
we
have
a
full
round
trip,
just
open
a
TCP
connection.
A
A
A
How
does
the
Envoy
help
with
this
right?
I
just
talked
about
TLS
negotiation
over
a
user's
internet
connection.
Well,
after
that,
TLS
connection
is
established.
We
still
might
have
to
do
something
like
this.
We
still
need
to
get
that
users
API
request
all
the
way
from
wherever
they
happen
to
live
through
one
of
our
points
of
presence,
all
the
way
back
to
where
our
home
data
center
lives.
We
might
need
to
hit
the
database
for
that,
and
maybe
those
are
only
geographically
located
in
one
place
now
again,
there's
nothing
about
this
or
sorry.
A
The
the
part
that
envoy
can
help
us
with
is
making
that
leap
from
two
geographically
distant
points:
Japan
Ireland
those
places
aren't
getting
any
closer
to
the
US,
no
matter
what
fancy
tricks
we
wish
we
could
pull.
So
in
order
to
talk
about
the
Envoy
portion
of
this
I'd
like
to
zoom
in
on
the
most
interesting
part
of
this
topography,
which
is
again
the
links
between
geographically
distant
locations.
So
we
can
ignore
the
users
internet.
We
can
ignore
our
database
and
focus
on
these
links
now
what
we
at
stripe.
A
Do
the
second
request:
wait
for
it
to
complete,
etc
with
HTTP
two
we
can
multiplex
these
and
perform
them
more
quickly.
So
the
great
thing
about
setting
up
this
pretty
basic
envoy
use
case
is
by
having
envoy
and
manage
the
connection
on
both
sides.
Those
connections
which
are
pretty
expensive
to
establish
you
can
imagine.
We've
got
to
do
em
TLS
between
faraway
places.
Those
connections
now
outlive
any
application,
restarts
on
either
side.
So
at
stripe
we
deploy
many
times
a
day,
dozens
of
times
a
day,
depending
on
the
day.
A
If
we
have
to
deploy
to
any
of
those
points
of
presence
and
restart
some
servers,
we
don't
have
to
tear
down
and
reestablish
these
MT
last
backhaul
connections,
as
we
call
them
to
kind
of
steal,
a
metaphor
from
the
trucking
industry.
Those
connections
stay
established
and
quote-unquote
warm
so
that
any
incoming
customer
requests,
whether
it
be
to
Japan
Ireland,
the
US
what-have-you
lands
on
one
of
those
connections
and
gets
sent
back
to
our
home
datacenter.
A
A
Another
thing
I
want
to
touch
on
real
quickly
is
something
we
built
with
great
help
from
envoy,
which
is
our
Bluegreen
deployments.
So
a
quick
perimeter
if,
if
you're
not
familiar
with
what
Bluegreen
deployments,
are
conveniently
I've
duplicated
our
home
data
center,
we
made
one
of
them
blue
one
green.
There
are
two
identical
sets
of
infrastructure
and,
typically
in
the
steady
state,
you
have
a
hundred
percent
of
your
traffic
on
one
copy.
Let's
say
blue.
A
That
leaves
you
free
to
deploy
new
software
to
your
other
copy,
the
green
copy
and
then
shift
traffic
at
your
leisure
to
test
things
out,
make
sure
everything
looks
good.
There
are
no
bugs
and
you
can
gradually
ramp
traffic
to
your
new
software
version.
Maybe
something
like
this
until
eventually
you're
free
to
deploy
back
to
blue.
So
that's
blue,
green.
In
a
nutshell.
A
The
other
nice
thing
that
gets
you
is
that,
since
we're
using
our
voice-
Software
Defined
Networking
for
this,
rather
than
like
changing
weights
on
a
DNS
entry
or
changing
DNS
entries
altogether,
we
don't
have
a
long
tail
of
traffic
that
refuses
to
shift
like
you,
sometimes
get
with
DNS
clients,
love
to
cache,
DNS
entries
and
not
resolve
them.
When
you
want
them
to
so
the
joke
we
use
at
stripe
as
we
use
envoy
Sdn,
not
DNS,
to
shift
this
traffic.
It's
easy,
its
quick.
A
So
these
are
all
elements
of
what
we
think
of
as
our
custom
control
plan
that
we
built
at
stripe
using
on
voice.
So
we
have
custom
XDS
services,
like
I,
said
they
give
us
control
over
routing
traffic,
routing
traffic,
ramping
they
power
all
of
our
Bluegreen
deployments.
Instead
of
DNS
and
its
myriad
issues
and
using
the
priority
and
locality
features
of
our
voice,
configuration
we've
been
able
to
establish
different
tiers
of
failover
infrastructure
so
through
both
active
and
passive
health
checking.
A
We
allow
envoy
to
failover
to
different
classes
of
infrastructure
if
it
detects
problems
much
faster
than
any
developer,
with
their
own
cool
little
Co
I
would
manually
be
able
to
shift
traffic
in
an
incident,
and
we
can
do
cool
things
like
prefer:
a
nearby
infrastructure
versus
infrastructure,
further
away,
which
is
the
locality
piece
of
that
again,
because
we
have
developer
tooling
built
up
on
this.
We
can
manually
remediate
incidents
pretty
easily.
We
can
shift
traffic
off
of
defective
Hardware,
bat
deployments,
etc.
A
We
can
also
and
do
take
advantage
of
on
Veliz
header
based
routing
and
things
like
that,
to
not
only
route
specific
requests
to
particular
infrastructure
but
requests,
say
per
customer
or
per
account
and
I
believe.
There's
the
talk
coming
up
later
today
about
some
dynamic
routing
that
I'm
I'm
excited
to
hear
about.
A
Of
course,
it
wouldn't
be
a
comprehensive
talk
if
all
I
did
was
talk
about
the
great
things
we
built
and
none
of
the
challenges
we
experienced
along
the
way
so
I
mentioned
earlier
that
HTTP
2
allows
you
to
multiplex
many
requests
onto
a
single
TCP
connection.
So
what
we
have
here
I
know:
there's
no
y-axis
and
there's
no
numbers
on
here.
A
Now
we
run
multi-threaded
envoy
on
the
server
we
have
one
worker
thread
per
core
I
think
you
know
how
this
story
goes
envoy
defers
to
the
kernel
for
certain
things,
like
figuring
out,
which
worker
thread
will
accept
an
incoming
connection,
and
we
we
found
in
our
initial
configuration
that
busy
threads
tended
to
win
so
connections
with
many
requests,
tended
to
accept
new
requests
so
with
this
knowledge
in
hand
and
again
going
back
to
HTTP
2,
which
is
handling
data
for
multiple
requests.
At
the
same
time,
that's
known
as
a
stream
in
HTTP
2
parlance
here.
A
A
So
why
does
any
of
that
matter?
Who
cares
if
more
requests
are
on
one
connection
or
the
other
as
long
as
they're
succeeding
right?
Well,
as
we
know,
or
we
should
constantly
remind
ourselves,
the
network
is
not
reliable.
That's
a
fundamental
fact,
and
we
experience
especially
when
going
to
these
geographically
distant
locations,
bad
Network,
condition
things
like
TCP
packet
loss,
so
again,
quick
primer
packet
loss
and
TCP
retransmits
or
what
happened
when
data
gets
lost
in
transit.
So
if
we
send
three
TCP
segments,
the
receiver
only
gets
two
of
them.
A
Eventually,
the
fact
has
to
be
communicated
the
head
and
get
one
retransmitted.
This
happens
all
the
time.
Tcp
is
pretty
good
at
dealing
with
this,
but
again
what
happens
in
practice
when
packet
loss
and
retransmit
start
to
occur?
Well,
I
apologize!
If
this
is
a
tiny
bit
hard
to
read
in
the
back,
but
what
we're
looking
at
here
is
a
blue
line
going
steadily
up
into
the
right,
and
that
represents
TCP
retransmits,
and
this
is
from
our
production
infrastructure,
and
so
we
have
cumulative
retransmits.
A
That
means
things
are
having
to
be
resent
over
and
over
by
the
operating
system.
We're
experiencing
some
sort
of
network
difficulty
and
the
green
line
is
a
measure
of
the
kernels
right
Q.
So
the
data
backing
up
that's
unable
to
be
sent
over
the
wire
during
that
time
of
retransmits.
So
this,
this
kind
of
makes
intuitive
sense
right,
like
retransmits,
are
happening.
The
kernels
like
backing
up
for
the
customers.
That
means
increased
latency,
x'
timeouts
things
that
are
not
great.
So
this
next
chart
I've
Chet
I've
kept
the
green
line.
The
kernel
right
queue.
A
A
So
one
thing
we
did
as
we
change,
which
congestion
control
algorithm.
We
were
using
on
the
hardware
that
I
diagrammed
earlier
we
used
a
new
congestion
control,
algorithm
called
bbr.
This
is
something
that
came
out
of
Google
research,
I
believe
and
what
the
silly
chart
here.
Attempts
to
kind
of
show
is
that
bbr
attempts
to
maintain
higher
throughput
in
the
face
of
packet
loss,
retransmits,
network
instability.
A
A
We
made
some
other
tweaks
once
we
set
this
up,
so
in
addition
to
changing
the
congestion
control
algorithm,
we
encountered
this
HT
to
be
head
of
line
blocking
problem.
Well,
HTTP
is
supposed
to
solve
head
of
line
blocking,
but
it
has
its
own
head
of
line
blocking
problem.
It's
just
one
based
on
TCP,
if
TCP
isn't
able
to
send
all
the
packets
there's
nothing.
A
higher
level
protocol
like
HTTP
2
can
do
to
fix
that.
A
So
one
thing
we
did
is
we
increase
the
number
of
upstream
TCP
connections
and
what
we
found
is
that
that
distribution
of
requests
per
connection
got
a
month
got
much
more
even
and
we
didn't
have
the
disproportionate
impact
when
one
link
would
experience
packet,
loss
or
retransmits.
We
did
this
by
telling
envoy
to
use
these
up
streams.
So
as
it
turns
out,
this
is
an
actual
Envoy
config,
as
you
can
tell,
but
andö
is
more
than
happy
to
open
twice
as
many
TCP
connections
to
fubar
striped
comm.
A
If
we
tell
it
to
so,
this
was
a
little
unexpected,
but
it
was
great
because
now
we
have
the
best
of
both
worlds.
We
have
HTTP
to
request
multiplexing,
so
we
get
more
requests
per
connection,
and
now
we
have
more
TCP
connections,
great
we're
getting
somewhere
we're
now,
sharing
that
request
burden
evenly
across
our
connections
a
little
bit
of
a
I
guess,
I'll
call
it
creative
solution.
It
felt
it
felt
weird
to
set
something
like
this
up
in
config,
but
again
envoy
is
more
than
happy
to
honor
it.
A
So
with
enough
custom
tweaks
like
this,
we
eventually
figured
out
that
we
could
manage
the
bandwidth
and
request
burden
across
all
of
our
geographically
distributed
links
and
the
infrastructure
became
stable.
So
this
is
kind
of
where
we
move
into
the
pros
and
cons.
Looking
back,
what
would
we
do
differently?
Part
of
it
to
talk
so
again
be
be.
Our
congestion
control
was
something
we
used
along
with
the
FQ
fair,
queuing
packet
scheduler.
These
are
all
things
I
don't
have
time
to
dive
into.
A
We
also
enabled
TCP
keep
Alive's
on
all
of
our
up
streams
depending
on
your
cloud
provider
and
what
middle
boxes
you
might
have
in
your
architecture.
This
becomes
pretty
important
to
not
have
connection
state
get
garbage
collected
out
from
under
you.
We
can,
if
you
find
me
later,
we
can
talk
all
about
the
fun
problems
that
come
with
that.
A
A
We
have
great
observability
now
between
all
of
our
network
links
and
our
data
centers
again
the
running
joke.
Any
conversation
at
envoy
about
Anglais
that
happens
at
stripe
usually
starts
with.
Oh
there's
a
metric
for
that,
and
indeed
there
literally
is
a
metric
for
almost
everything.
So
we
have
great
observability
now
again
fixes
get
propagated
to
all
of
our
network
links
between
data
centers,
roughly
at
once,
I
don't
want
to
say
at
once
whether
they
be
fixes
for
a
CVE
or
regular
Envoy
version
updates.
A
This
is
in
contrast
to
having
to
upgrade
and
migrate
all
of
your
applications
at
each
of
those
points
and
similarly
network
behavior.
So
you
find
a
better
configuration
a
better
way
to
do.
Retries
you
change.
You
want
to
change
the
cipher
Suites.
Your
TLS
connections
are
using.
You
can
change
that
behavior
across
your
network
topology,
without
having
to
go,
modify,
100
applications
now
I'm
a
software
engineer
again:
I
write
application
code
for
living.
Why
do
I
make
it
sound
like
changing
an
application?
Is
this
insurmountable
huge
pain
in
the
ass
task?
A
Well,
it's
because
teams,
not
only
it's
stripe,
but
every
company
I've
ever
worked
at
our
polyglot.
So
it's
stripe.
We
have
people,
writing
servers
and
applications
and
go.
We
have
a
ton
of
stuff
in
Ruby
we
have
some
Scala
thrown
in
there,
but
even
within
a
language
you
have
different
frameworks.
People
might
be
using
different
Network
clients,
you
might
have
different
runtime
versions,
etc,
etc.
All
of
those
are
gonna
have
different
network
behavior
defaults,
different
ways
to
configure
things
and
so
change,
making
a
sweeping
like
network
behavior
change
across
many
services.
Languages
frameworks.
A
A
Again,
I
can't
give
prose
without
given
some
cons.
So
what
are
some
some
cons
that
we
get
with
the
benefit
of
hindsight?
So
one
is
that
having
ongoing,
manage
both
sides
of
every
connection
between
every
data
center
adds
a
lot
of
moving
pieces
to
your
stack.
All
those
pieces
need
to
be
monitored.
They
need
to
be
make
their
way
into
your
network
diagrams.
They
need
to
be
upgraded
from
time
to
time
and
your
engineers
need
to
learn
about
how
they
those
pieces
work
and
how
to
operate
them.
A
So
they
need
to
know
like
where
the
knobs
are.
How
do
I
change
config?
How
do
I
go
about
manipulating
Envoy
in
practice,
things
like
timeouts
and
access
logs
and
vestiges
of
network
behavior
that
live
in
your
apps
still
need
to
be
kept
in
sync,
with
the
on
boy,
that's
kind
of
a
minor
point,
a
bigger
one
for
us
was
when
you
encounter
bugs
in
a
product
like
envoy.
It
might
not
be
in
the
sort
of
wheelhouse
or
domain
of
your
application
developers.
So
for
us
at
stripe.
A
A
You
know
I'm
not
on
call
right
now,
thank
goodness,
but
when
I
am
on
call,
I
need
to
know
how
to
operate
all
the
pieces
of
the
stack
I'm
responsible
for
whether
that
be
envoy
or
any
other
piece
of
software
and
I
see
that
I'm
running
a
little
low
on
time.
So
we
don't
have
time
to
dive
into
this
last
one.
But
it's
pretty
interesting.
A
If
you
want
to
find
me
after
the
actual
implementations
of
the
HTTP
protocols
themselves
might
differ
slightly
in
Envoy
than
they
do
in
other
pieces
of
your
stack,
whether
that
be
nginx
go
H
a
proxy
and
you're
thinking.
Well,
HTTP
is
a
standard
like
they.
Everything
works,
the
same
if
it's
codified
in
a
standard.
Well,
yeah,
most
the
time
and
again,
like
I,
have
some
fun
some
fun
stories
about
that.
I
picked
and
genetics
go
H
a
proxy.
These
are
common
at
stripe,
common
other
places.
A
So
I
guess
that's
a
lot
of
information.
I
apologize,
I,
just
kind
of
beat
you
over
the
head
with
BB
rtcp
retransmits
Anglais
configuration
XDS
sidecars
a
lot
of
jargon,
but
the
message
I'd
like
to
leave
you
with
to
kind
of
wrap.
This
up
is
that
we
built
a
network
topology
heavily
based
on
envoy,
and
we
did
indeed
span
the
globe.
A
We
used
Envoy
to
connect
those
millions
of
customers
in
those
120
plus
countries,
I'm
sure
I
left
some
of
them
off
of
this
diagram
and
we
allowed
people
to
transact
move
money,
start
run
and
manage
their
businesses
online
24/7,
not
many
nines
of
availability
and
they're
doing
so,
even
as
we
speak
today,
maybe
for
some
of
the
businesses
that
employ
all
of
you,
so
we
couldn't
have
done
it
without
a
lot
of
help
from
the
Anglais
community.
I
actually
just
met.