►
From YouTube: The Evolution of Twitter's Edge - Ryland Degnan, Twitter
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Evolution of Twitter's Edge - Ryland Degnan, Twitter
Each day, Twitter responds to hundreds of billions of requests from users around the world. Today, Envoy is the point of entry for 100% of these requests. This hasn't always been the case. Not long ago, requests to Twitter passed through an ancient and highly bespoke edge proxy that was created internally and predated the open-source Envoy. In this talk, Ryland will describe how Twitter runs Envoy at the edge at scale, some of the unique benefits that Envoy provides in Twitter's edge architecture, and highlight features that Twitter has contributed to Envoy to support the edge use-case. He will outline how Twitter's edge architecture has evolved over time as the number of users and services has grown, what the next steps are for Twitter's edge, and the role Envoy will play in the future.
A
A
A
You
know,
within
one
context,
coming
coming
back
in
a
different
context,
spawning
sort
of
a
popular
open
source
project
and
then
that
project
sort
of
being
returning
full
circle
to
you
know
an
organization
where
some
of
those
ideas
were
born
I'll
go
a
little
bit
more
into
that,
but
yeah.
If
you
have
any
questions,
why
open
source
is
easing
the
world.
This
is
an
awesome
example,
so
my
name
is
ryland.
A
The
tfe
team
at
twitter
is
a
team
that
now
is
responsible
for
handling
hundreds
of
billions
of
requests
a
day
filtered
through
envoy,
and
I
you
know
everything
I'm
going
to
talk
about
is
essentially
the
result
of
two
years
of
work
on
behalf
of
the
the
tfe
team.
So
I'm
not
gonna
take
credit
for
any
of
it,
but
I
also
wanted
to
mention
the
csl
team,
which
is
the
core
systems
library
team
at
twitter,
who
the
tfe
team
collaborated
with
to
sort
of
adopt
envoy
at
scale
within
twitter.
A
So
we
pride
ourselves
at
twitter
at
availability
at
our
availability
and
reliability.
So
if
you
remember
a
couple
of
weeks
ago,
when
facebook
and
instagram
went
down,
half
the
internet
seemed
to
be
burning,
twitter
stayed
up
and
we're
pretty
proud
of
that.
Given
there's
a
bunch
of
things,
sort
of
unique
challenges,
I
think
we
have
which
makes
that
pretty
hard
to
accomplish,
namely
the
you
know.
One
thing
is
the
scale
of
it:
so
200
million
daily
active
users
coming
in
wanting
to
use
twitter,
sending
millions
of
requests
a
second
and
to
make
matters
worse.
A
Those
users
are
geographically
distributed
right.
So,
despite
our
data,
centers,
primarily
being
located
in
the
u.s,
our
user
base
is
growing
much
more
rapidly
in
other
parts
of
the
world,
with
less
good
internet
connections,
and
yet
we
have
to
manage
to
make
a
service.
That's
both
you
know,
responsive
and
reliable
for
all
of
our
users.
Globally,
we
have
to
deal
with
unpredictable
load
spikes
for
a
variety
of
reasons.
Some
of
them
are
more
predictable
than
others.
A
But
once
in
a
while
there
will
be
a
hit
television
show
say
big
brother
brazil,
that
nobody
could
have
anticipated
how
popular
that
would
be
and
that
almost
took
out
twitter
and
brazil,
except
for
some
last
minute
heroics.
So
there's
always
sort
of
interesting
things
cropping
up
from
time
to
time,
some
of
them
some
of
the
load.
Spikes
are
not
organic.
Some
of
them
are
malicious
as
well.
So
twitter
has
a
large
unauthenticated
api.
A
A
We
just
don't
have
a
lot
of
control
over
that,
and
yet
we
managed
to
stay
up.
But
the
story
wasn't
always
that
good.
So
I
wanted
to
rewind
a
little
bit
about
sort
of
twitter
pre-2014
before
we
were
really
doing.
Much
of
anything
at
the
edge,
we
had
some
sort
of
points
of
presence
globally,
networking
pops,
but
we
really
weren't
doing
anything
like
terminating
tls
or
anything
fancy
at
the
edge
at
all,
and
this
started
to
become
more
and
more
of
a
concern.
A
A
So
this
issue
kind
of
came
to
head
prior
to
the
2014
world
cup
in
brazil,
where
we
just
realized.
It
was
going
to
be
a
massive
missed
opportunity
if
we
sort
of
if
the
service
went
down
in
brazil
or
was
unusable
during
this
time,
because
we
wanted
to
be
part
of
the
global
conversation
and
allow
people
to
tweet
from
the
world
cup
so
enter
the
twitter
streaming
aggregator
created
by
a
twitter
employee
called
matt
klein
in
2013..
A
The
name
kind
of
comes
from
it
was
originally
intended
from
a
for
a
different
use
case
for
the
twitter
sort
of
consuming
the
twitter
fire
hose.
So,
like
everything,
that's
happening
everything
that's
going
on
within
twitter.
We
offer
a
service
where
you
can
sort
of
get
a
stream
of
all
these
aggregated
events,
but
around
about
before
you
know,
early
2014.
A
It
was
a
huge
success.
We
stayed
up
during
the
world
cup
and
tsa
was
born
and
it
was
so
successful.
You
know
I
could
characterize
the
next
several
years
worth
of
edge
development
at
twitter
by
basically
expanding
on
this
idea,
so
this
went
from
sort
of
a
proof
of
concept
to
being
installed
in
many
edge
locations
around
the
world,
so
twitter
owned
pops
globally.
A
So
we
can
have
you
connect
over
http
one
or
whatever
protocol
to
your
local
pop
and
then
multiplex
over
sort
of
nice,
http,
2
connections
back
to
our
data
center
and
then,
furthermore,
it
gives
us
the
ability
to
shift
traffic
back
and
forth.
So
we
don't
necessarily
have
to
home
to
a
single
data
center
from
a
single
pop.
If
a
data
center
is
having
an
issue,
if
an
individual
service
is
having
an
issue,
we
can
redirect
some
or
all
of
traffic
from
each
pop
to
other
data
centers
where
the
service
is
currently
running.
Fine.
A
A
A
It's
not
always
rfc
compliant
and
the
main
author
left
for
lift,
which
you
know,
tends
to
put
a
damper
on
feature
development
in
some
ways,
but
it's
essentially
modern
hardware
evolved.
This
software
was
designed,
for
you,
know
a
maximum
of
say,
like
20
24,
core
machines.
At
the
time
we
have
much
larger
numbers
of
cores
available
to
us
right
now,
but
we
found
that
it
just
wasn't
able
to
scale
the
threading
model.
Isn't
quite
as
nice
as
envoy
is
now
there
were
some
mistakes
made
in
some
ways.
A
This
is
the
one
you
build
to
throw
away
before
you
get
it
right.
It
didn't
support
a
bunch
of
modern
protocols
that
we
really
wanted
to
use.
We
wanted
to
experiment
with,
say
http,
3
or
grpc,
since
tsa
was
lacking,
proper
trailer
support
that
made
that
difficult,
and
then
things
like
tls
1.3
would
have
been
a
huge
effort
to
retrofit
into
into
our
existing
proxy
tsa
and
then,
from
a
security
perspective.
A
Getting
this
up
and
running,
we
ended
up
forking
openssl
and
making
our
own
modifications
to
it,
which
became
a
challenge
to
you,
know
port
into
more
up-to-date
versions
of
open
ssl.
So
we
had
issues
and
that
would
it
required
lots
of
work
to
sort
of
port.
These
changes
to
modern
versions
of
an
sl,
which
was
you
know,
an
ongoing
nightmare,
there's
no
concept
of
a
control
plane,
it's
all
statically
configured
and
there's
not
really
the
ability
to
restart.
A
We
kicked
the
tires
a
little
bit
on.
You
know
various
options,
but
I'd
say
it
was
in
2019
that
we
sort
of
committed
to
envoy
as
a
good
solution
in
part,
because
it
was
so
similar
to
tsa
that
it
would.
It
was
pretty
straightforward
to
reintegrate
back
into
the
twitter
ecosystem
and
then,
secondly,
like
fortuitously,
because
the
csl
team
wanted
to
use
envoy
in
the
context
of
a
service
mesh,
and
so
that
gave
us
a
good
opportunity
to
combine
our
resources
on
things
like
setting
up
a
build
pipeline.
A
Writing
the
bones
of
a
control
plane
and
everything
like
that.
That
could
be
configured
both
as
a
service
mesh
and
as
an
edge
proxy,
and
so
we
started
development
of
what
we
called
a
project
called
t3
in
2019
and
starting
from
late
last
year,
and
into
this
year
we've
been
rolling
it
out
in
production
at
twitter.
A
A
A
So
if
I
zoom
in
a
little
bit
on
how
this
looks
so
at
twitter,
we've
built
our
own
internal
control
plane
based
on
scala.
The
reason
we've
done.
This
is
because
we
have
a
bunch
of
existing
libraries,
so
the
core
infrastructure
of
twitter
is
mostly
built
around
scala
libraries,
if
you
have
any
experience
with
finagle
or
anything
like
that,
so
we
have
essentially
fat
clients
that
interact
with
whole
bunch
of
systems
out
there.
So
service
discovery
feature
switches.
A
A
And
in
terms
of
the
data
plane
so
we've
we
took
envoy,
we
have
a
few
of
our
own
sort
of
modifications
of
the
core
envoy,
plus
a
bunch
of
filters
that
we've
implemented
and
we
build
it
all
together,
they're
all
c
plus
plus
filters,
and
then
we
sort
of
distribute
our
own
envoy
binary.
So
some
examples
of
stuff
we've
implemented
I'll
go
into
a
few
more
later,
but
twitter
specific
metrics
extensions.
A
We
use
a
globally
unique
connection
hash
just
so
we
can
correlate
things
later
on
in
our
data
pipeline,
so
we
can
see
exactly
what
happened
during
the
lifetime
like
of
an
individual
connection
and
then
some
other
things
were
important
to
us
were
the
mux
protocol,
which
is
a
rpc
protocol.
We
use
internally
at
twitter,
it's
the
rbc
multiplexing
protocol.
A
We
worked
a
little
bit
with
envoy
maintainers
to
implement
start
tls,
which
lets
you
start
with
an
unsecure
connection
and
then
upgrade
to
like
a
secure,
mux
connection
and
more
but
the
stuff
we
love
about
envoy
or
the
stuff
are
the
things
that
we
didn't
have
to
build
ourselves
and
would
have
taken
us
lots
of
time.
So
ipv6
was
a
huge
one.
A
A
This
turns
out
to
be
a
huge
win.
So
just
when
we
turn
this
on,
we
looked
pretty
closely.
We
gathered
a
bunch
of
client
metrics
and
we
notice
sort
of
38
drop
in
handshake
time,
and
that
translates
into
something
like
almost
30
drop
in
the
home
timeline,
pull
to
request,
pull
to
refresh
request
duration.
So
when
you're
scrolling,
twitter
and
you're
trying
to
refresh
the
screen,
it
translates
directly
into
an
improvement
of
the
user
experience.
A
Yeah
and
then
so
I
like,
as
like
the
last
speaker
mentioned
bot
mitigation,
is
a
huge
deal
to
us
at
twitter,
so
yeah
we're
constantly
flooded
with
malicious
requests,
either
trying
to
take
down
twitter
trying
to
find
interesting
unused,
usernames
trying
just
trying
to
you
know,
gather
data
for
whatever
purpose
and
prior
to
envoy.
We
had
sort
of
a
brain
dead
way
of
dealing
with
this
in
tsa.
A
But
of
course
that
has
a
negative
effect
on
legitimate
users
as
well.
So
we
looked
at
the
rate
limiting
options
that
were
available
in
envoy
and
initially
we
determined
that,
like
none
of
them
were
really
a
good
fit
for
what
we
wanted
to
do
here,
either
local
rate
limiting
or
global
rate
limiting,
and
the
reason
being,
is
that
they're
both
sort
of
assuming
a
fair
playing
field
or
lack
of
malicious
users.
A
So
one
user
could
connect
if,
if
you're
using
say
local
rate
limiting
in
a
token
bucket,
then
one
user
could
connect
and
instantly
drain
that
bucket
of
all
tokens,
thereby
denying
access
to
every
legitimate
user
who
comes
and
lands
on
that
instance,
and
if
you
were
to
use
global
array
limiting
that
would
be
sort
of
even
worse
at
the
edge.
So
what
we
wanted
was
a
way
to
rate
limit
users
per
connection,
but
not
deny
access
to
other
legitimate
users
who
happen
to
be
connected
through
that
same.
A
You
know
envoy
instance,
so
we
implemented
and
contributed
back
to
envoy.
This
is
available
now
a
per
connection
rate
limiting
filter.
So
this
allows
us
to
rate
limit
quotas
on
a
per
connection
basis,
and
we
can
so
we
can
sort
of
set
this
per
v
host.
We
can
say
you
know
you
get
x
amount
of
tokens
per
per
minute
and
if
you
exhaust
them,
then
we
just
started
returning
like
four
two
nines
or
redirect
you
to
somewhere
else,
and
that
was
that
ended
up
being
a
huge
win
for
us.
A
A
Libraries
will
have
a
particular
hash
and
so
that's
all
fed
into
a
live
pipeline
where
we
can
look
at
all
the
connection
hashes
that
are
flowing
through
the
system
and
flag
up
pretty
quickly.
You
know
a
botnet,
that's
suddenly
attacking
twitter
with
a
connection
hash
we've
with
the
tls
fingerprint
we've
never
seen
before.
A
And
then
sort
of
in
the
longer
term,
we
we
have
a
lot
of
ideas
we
want
to
explore
with
envoy
twitter
does
a
lot
of
caching
in
the
data
center,
but
not
a
lot
of
caching
at
the
edge.
So
that's
a
huge
opportunity
so
we'd
like
to
explore
sort
of
envoys
http
caching
filter,
so
we
can
cache
things
at
the
edge
both
like
materialized
views
of
api
objects
as
well
as
things
like.
You
know,
auth
tokens
and
pre-computed.
A
So
that's
something
that
we've
sort
of
already
prototyped
and
are
looking
to
expand
on
the
other
interesting
conversations
we're
having
around
envoy
mobile,
which
is
really
attractive,
having
your
edge
be
envoy.
Also
having
the
other
side
of
that
connection
be
envoy.
That
means
you
can
deploy
protocol
changes
and
things
in
lockstep
and
then
unify.
A
Typically,
we've
had
sort
of
totally
different
observability
stacks
for
like
mobile
and
backend
services.
That
would
unify
all
that
and
be
a
replacement
for
some
of
the
stuff
we've
built
in-house.
So
we
have
sort
of
a
mechanism
for
beacons
to
be
sent
out
to
different
potential
connection
points
to
twitter's
network,
to
allow
clients
to
intelligently
choose,
but
there's
some
really
cool
stuff
around
multi-dimensional
connection
methods
built
into
envoy
mobile.