►
Description
Improving performance of RPCs with envoy at Wikimedia - Giuseppe Lavagetto
Performance of remote procedure calls between services depend on a lot of factors, but when you start doing RPCs over a high latency network and/or using TLS (so when you have to perform RPCs across different datacenters, for example), the cost of establishing a connection is very steep. This is particularly problematic for environment which don't support persistent connection pools - one notable example being the PHP language, that we use to run MediaWiki. This talk will go through how Wikimedia introduced envoy in its mixed on-prem/kubernetes environment, and how that allowed to improve the performance, reliability and observability of its stack. Particular focus will be put on: the performance effects for our PHP applications running at scale, the operational problems adopting envoy allowed solving, and the challenges introduced by moving to use it.
A
Let's
start
with
the
introduction
who
we
are,
the
wikimedia
foundation
is
the
nonprofit
organization
that
runs
the
infrastructure
that
supports
wikipedia
and
its
sister
projects
like
quick
scenario
or
wikidata.
We
do
quite
a
lot
of
traffic
monthly.
We
get
one
21
billion
page
views.
This
is
the
data
from
last
august.
A
Now,
how
is
our
infrastructure
structured?
We
have
five
data,
centers,
three
of
which
are
just
caching
pops,
so
they
are
just
points
of
presence
that
only
hosts
the
caching
layer,
while
the
two
main
data
centers
are
here
in
blue
and
green
and
are
both
located
in
the
united
states,
one
in
virginia
and
the
other
in
dallas,
and
they
host
our
wall
application
stack
now
what
happens
when
somebody
makes
a
request?
A
When
they
want
to
snoop
on
the
data
of
your
users,
and
so
we
need
to
introduce
tls
in
front
of
all
of
our
application
now
we
could
have
asked
every
development
team
to
develop
to
add
tls
termination
to
their
application,
but
that
would
have
meant
asking
all
these
teams
to
become
somewhat
experts
in
configuring,
a
tls
stack
and
at
the
same
time
it
would.
It
meant
that
we
would
have
to
track
the
security
defects
in
multiple
application
stacks.
So
we
decided
pretty
early
we're
going
to
install
a
tls
termination
sidecar.
A
A
A
Now
the
logged-in
users
are
typically
the
editors,
the
people
that
add
infor,
add
content
to
the
wikis,
and
these
are
in
some
ways
our
most
valued
users,
because
we
are
the
ones
that
build
the
projects
that
make
them
successful,
and
these
users
already
pay
a
price
for
their
loyalty
by
always
being
sent
to
the
main
data
centers.
The
reason
for
that
is
that
when
you're
logged
in
you
can
change
the
interface
the
appearance
of
the
site,
so
we
can't
just
send
you
a
test
copy.
A
Now
we
didn't
want
our
introduction
of
encryption
between
services
to
add
another
penalty
that
was
used,
and,
apart
from
this
envoy,
has
been
designed
to
be
the
perfect
service
to
service
middleware.
So
it
has
a
series
of
characteristics
that
we
really
wanted
to
add,
one
of
which
is
its
observability
features.
It
adds
telemetry
within
all
services.
A
It
gives
you
the
ability
to
emit
tracing
data.
It
has
some
additional
things
that
you
want
when
you
build
a
true
microservices
infrastructure,
like
rate
limiting
and
circuit
breaking
built
into
the
proxy,
so
that
you
can
have
a
common
implementation
across
services
and,
finally,
it's
very
easy
to
configure.
A
A
A
But
we
wanted
to
have
the
same
data
structure,
defining
listeners
and
clusters
for
envoy
to
be
used
both
by
kubernetes
and
the
legacy
environment,
and
we
wanted
this
to
be
simple
and
easy
to
understand
to
any
sre,
even
if
they
have
no
prior
experience
with
envoy,
we
came
up
with
a
simple
yaml
data
structure.
Here
is
an
example:
that's
taken
from
our
configuration.
A
I
want
to
make
a
shout
out
to
the
android
apps
for
adding
mood
validate
to
the
server
so
that
it
allowed
us
to
easily
catch
any
errors
that
we
were
introducing
in
the
configuration
directly
in
the
continuous
integration
environment
and
some
investment
in
making
your
configuration,
continuous
integration
environment
check
with
the
configuration
is
doing.
What
it
expects
to
do
is
really
something
you
should
invest
in.
A
Now
all
transitions
that
happen
in
real
life
happen
with
some
struggles.
I
want
to
name
some
of
the
struggles
we
had
and
the
first
and
foremost
was.
We
use
a
level
four
load
balancing,
which
means
that
we
balance
connections,
ip
connections
and
not
requests
now
envoy
tries
to
funnel
as
many
requests
as
possible
as
possible
across
the
same
connection.
A
That's
one
of
his
strengths,
one
of
the
reasons
that
we
made
big
gains
later,
but
this
also
means
that
it
makes
it
very
bad
to
be
load
balanced
through
a
connection
based
load
balancer,
because
you
can
send
one
million
requests
across
one
connection
and
three
connect.
Three
requests
across
another
connection:
you've
balanced
these
connections
across
the
back
ends,
but
you
didn't
balance
the
requests,
which
is
what
you
really
want
to
balance.
A
So
what
we
did
was
just
to
limit
the
number
of
requests
that
you
can
send
over
one
single
connection
to
1000
by
default,
and
that
was
enough
to
make
the
problems
that
I
was
naming
or
basically
go
away.
We
also
had
another
problem,
which
was
we
did
see,
especially
for
the
applications
that
run
at
high
scales
or
thousands
and
thousands
of
requests
per
second.
A
But
we
had
some
mysterious
connection.
Timeouts
connection
failures,
sorry
happening
from
time
to
time,
and
we
treat
it
back
to
the
fact
that
application
servers
typically
define
them.
They
keep
a
lifetime
out
for
http
connections
so
that
they
can
basically
kill
a
connection.
That's
kept
alive
by
the
client
if
the
client
doesn't
send
any
data
out
of
that
connection
for
more
than
n
seconds
and
turned
out
that,
basically,
we
had
to
account
for
vetting
and
void
as
well.
A
So
just
keep
that
value
a
bit
smaller
in
on
the
invoice
side
than
on
the
server
side
and
all
these
errors
that
we
had
go
away.
And
finally,
since
we
didn't,
we
choose
to
not
go
with
the
sdoa
and
just
make
all
routing
through
envoy,
transparent
to
the
application
by
using
iptable
sorceries,
and
we
decided
to
actually
go
and
change
the
configuration
of
the
application
to
find
a
request
for
envoy.
Also
because
we
this
way
we
could
just
switch
one
back
end
at
a
time
if
we
need
to
well.
A
The
problem
is
that
sometimes
the
same
configuration
key
is
used
for
finding
the
upstream
servers
to
call,
and
also
to
output,
some
data
to
the
user
like
a
csv
url
for
css,
and
thus,
when
you
use
localhost
5603
in
a
configuration
for
our
mobile
application
service,
you
might
break
mobile.
Wikipedia,
like
you
did.
A
A
Now
it's
easy
to
dunk
on
php,
but
it's
also
undeniable
that
for
all
its
flows,
it's
very
successful.
It's
used
to
run
some
of
the
largest
website
investment
sites
in
the
world,
and
we
create
one
of
the
creators
of
hhvm
keith
adams,
as
written
has
argued
that
the
reason
for
success
for
php's
success
is
its
scoping
rules,
which
means
the
scope
of
any
of
the
execution
of
a
php
script
in
a
web
server.
Is
the
web
request
so
at
the
start
of
a
web
request,
you
start
with
basically
an
empty
scope.
A
You
have
nothing
besides
some
globals
and
the
request
variables
and
when
you
have
to
build
anything,
you
have
to
allocate
memory.
You
have
to
build
anything.
You
need
to
make
all
the
connections
you
need
and
at
the
end
of
the
request,
everything
the
memory
you
are
located,
the
file
descriptors
is
open.
Everything
is
run
away.
A
A
A
This
means
that
also
you
can
share
things
like
connection
pools,
and
this
means
that
whenever
your
php
application
has
to
call
other
services,
it
needs
to
create
a
new
connection
for
every
request
it
needs
to
make,
and
this
is
a
cost,
but
this
cost
is
even
bigger
if
your
connection
is
using
tls,
because
when
you
use
tls,
you
have
at
least
two
additional
round
trips
to
account,
for
I'm
saying
at
least
because
it
depends
on
a
series
of
factors,
but
it's
at
least
two
round
trips.
It's
one.
A
If
you
get
two
tls
1.3,
but
good
luck
using
tls
1.3
from
php,
then
there's
the
cost
of
establishing.
Actually,
with
this
connection,
from
the
computational
point
of
view,
you
have
to
exchange
a
certificate,
you
have
to
exchange
secrets.
You
have
to
recreate
for
every
request
the
session
tickets
on
the
server
side
because,
as
we
said,
shared
nothing
on
the
client
side.
A
Now
I
said
before
it's
a
bit
more
complex
than
saying:
share
nothing,
that's
because
php
extensions
are
written
in
c
and
they
can
bypass
the
shared,
nothing
behavior
of
php
and,
in
fact,
kulex
securely
extension.
The
cure
extension,
which
is
what
everybody
uses
to
make
remote
dhtp
calls
in
hhvm,
allows
you
to
predefine
shared
connection,
pools
for
specific
host
names,
remote
test
names
in
zen
php,
which
is
what
we
are
using.
A
This
is
not
possible
at
all.
There
is
no
way
to
do
that,
so
we
knew
that
introducing
tls,
especially
for
requests
that
went
cross
data
center,
who
made
us
pay
a
big
price,
and
we
wanted
to
test
how
much
and
to
do
that.
We
just
did
this
very,
very
small,
benchmark
of
cross
dc
performance
in
our
production
environment.
A
We
just
wrote
a
small
script
that
would
call
will
fetch
using
the
php
call
extension,
the
elastic
search
burner
page
from
the
other
data
center,
which
is
35
milliseconds
more
or
less
from
triple
way,
and
then
we
call
this
script
with
a
concurrency
of
100
under
three
different
conditions.
In
one
case,
we
pointed
the
script
to
fetch
the
data
from
directly
from
elasticsearch,
but
using
the
http
endpoint,
so
no
encryption,
we
got
720
requests
per
second
of
throughput.
A
And
finally,
we
configured
envoy
and
the
same
machine
to
manage
connections
to
elasticsearch,
and
we
made
the
php
script
connect
to
envoy
and
localhost,
so
the
connections
were
encrypted
but
mediated
by
envoy,
and
in
this
case
we
obtained
1050
requests
per
second,
which
is
more
than
double
that
we
got
with
the
recalls
and
much
much
better.
Even
the
direct
calls
with
no
encryption
now
take
these
numbers
with
some
discretion.
A
A
So
we
started
from
there
and
let's
see
what
happened
so
we
said
before
we
expect
to
get
gains
of
latency,
mainly
across
data
centers.
We
expect
to
get
a
reduction
in
the
cpu
and
network
use.
Let's
see
what
happened
when
we
transitioned
the
the
or
through
the
use
of
envoy
process
calling
session
store
from
session
store.
A
A
A
Okay,
this
is
all
looks
very
good
right
on
paper,
but
what
about
the
latency,
which
is
what
we
really
care
about,
because
that's
what
the
users
see
and
the
latency
of
both
the
services
and
media
was
reduced
significantly,
let's
see
how
so
this
is
the
latency
buckets
traced
for
the
session
store
service.
You
can
see
here
below
that
the
green
line,
which
is
requests
that
be
served
in
less
than
one
millisecond
doubled.
A
Basically,
at
the
time
of
the
transition,
at
the
same
time,
the
number
of
requests
that
took
over
10
milliseconds
almost
disappeared,
so
good
effects
on
the
latency
of
session
store,
let's
see
if
it
had
any
effect
on
mediawiki.
Let's
always
remember,
it
was
about
one-fourth
of
the
traffic
that
was
affected
by
the
transition,
because
only
one-fourth
of
the
traffic
was
served
by
was
using
session
store
at
the
time
and
at
the
time
of
the
transition.
A
You
can
see
here
that
the
number
of
requests,
the
percentage
of
requests
that
mediawiki
was
responding
to
and
that's
when
100
milliseconds
went
from
about
21
20
between
20
and
22.
It
went
to
be
between
27
and
25,
so
a
five
percent,
the
increase
in
the
number
of
responses
that
took
less
than
100
milliseconds,
not
super
shocking
as
a
result,
but
still
pretty
impressive.
A
If
you
keep
in
mind
that
we
did
no
optimizations
at
the
application
layer
now,
this
game
is
the
random
volume,
and
I
want
to
show
it
to
you
with
a
couple
of
more
graphs.
This
is
of
two
different
deployments
of
the
same
application.
It's
a
rest
gateway
to
a
to.
Basically,
one
getting
two
thousand
requests
per
second
from
mediawiki
and,
in
this
case,
recipe
where
the
usage
reduction
is
about
25
percent
and
one
where
it
gets.
13
000
requests
per
second-
and
in
this
case
the
cpu
research
reduction
was
40.
A
So
you
have
to
keep
in
mind
that
how
much
you
gain
by
introducing
something
that
does
persistent
connections
on
behalf
of
php
depends
on
the
volume
of
course,
that
you
make,
if
you're
in
in
the
range
of
making
hundreds
of
calls
per.
Second-
probably,
this
is
not
very
important
for
you,
but
still
there
are
other
things
that
we
gained,
that
are
appliable
even
to
a
small
scale,
and
one
is,
as
I
told
you
before,
php
runs
active
passive.
So
at
some
point
at
the
start
of
september,
we
switched
which
data
center.
A
A
A
A
It
was
maybe
not
reporting
statistics
about
visits,
but
still
it
was
serving
pages
to
the
users,
which
is
what
we
care
most
mostly
about.
Finally,
just
one
notice:
we
I
want
to
say
that
we
still
have
some
scratches.
That's
normal,
that's
expected
when
you're
transitioning
to
using
something
completely
new.
A
By
using
envoy,
we
were
able
to
encrypt
all
the
communications
between
our
services
to
add
a
lot
of
observability
to
them
and
we
were
able
to
grammatically
improve
the
performance
of
our
php
applications
and
really
anyone
running
a
php
application
at
scale
should
think
of
doing
something
like
this
well.
Thank
you
for
your
time.
I
hope
I
didn't
bore
you
too
much.
A
Goodbye,
hello,
everyone.
I
hope
you
can
hear
me
so.
First
of
all,
yes,
jake
is
asking
when
I'm
saying
circuit,
breaking
I'm
talking
about
the
concept
of
circuit
breaking
and
in
the
case
of
envoy,
in
terms
of
envoys
terminology,
it's
outlier
detection.
In
this
case.
Yes,
sorry,
I
should
have
been,
I
should
have
been
clearer.
I
realized
why
I
was
listening
now,
so
I
think
I
answered
in
the
chat
already,
because
I
knew
that
I
was
running
a
bit
late
with
the
time.
A
So
I
see
that
derek
was
asking.
If
the
ultrastarter
works
differently
than
swapping
config
files,
we
wanted
to
use
the
the
hotel
starter
anyways
for
things
that
are
not
running
kubernetes
because
we
don't
want
to.
We
want
to
racially
serve
all
the
requests
whenever
we
have
to
restart
and
buy
so.
At
that
point
it
was
natural
to
just
use
it
just
changing
files.
A
We
have
a
procedure
to
build
the
files,
basically
on
yeah
puppet
and
and
deploy
them
the
configuration
files
and
then
just
we
just
send
a
sector
with
a
cd
agp
to
together
starter
and
just
starts
again
and
any
a
new
envoy
to
run
the
new
configuration.
Also
from
what
I
remember.
If
the
configuration
is
wrong,
the
old
version
should
keep
working
from
what
I
remember,
but
maybe
it
changed
over
last
year.
A
A
I
didn't
we
didn't
experience
that,
but
probably
we're
not
running
we're
not
running
and
boy
at
the
edge
we're
just
running
it
between
the
services
internally,
so
every
envoy
is
really
doing
the
amount
of
requests
that
probably
could
trigger
that
every
invoice
running
at
pretty
low
scale.
If
you,
if
you
want
right
running
some,
maybe
a
thousand
requests
per
second
overall,
a
peak.