►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Building a Service Mesh From Scratch - The Pinterest Story - Derek Argueta, Senior Software Engineer, Tesla
2 years ago Pinterest began rolling out Envoy as the ingress load balancer for its dynamic web traffic. Since then they've expanded Envoy's footprint to include service-to-service HTTP and Thrift communication while supporting Kubernetes and non-Kubernetes deployments with a powerful homegrown control plane. Derek works on the Pinterest Traffic Team and led the initial rollout of Envoy for ingress load balancing. In this talk Derek will provide a comprehensive overview of how Pinterest's service mesh evolved from static configuration files to a dynamic centralized control plane, and explore some of the challenges in adapting a service mesh to legacy infrastructure as well as the benefits it has provided.
https://sched.co/ZfGp
A
A
A
One
of
the
first
projects
that
I
was
able
to
participate
in
on
the
traffic
team
was
replacing
our
http
proxy
inside
our
ingress
load.
Balancer,
the
ingross
load,
balancing
architecture
at
pinterest
consists
of
a
cloud
load
balancer
that
goes
to
a
set
of
http
proxies
that
can
do
more
rich
and
feature
complete
proxying
such
as
http
routing
on
headers
or
paths,
or
provide
rich
stats.
Observability.
A
A
We
looked
at
the
existing
proxies
available
and
did
a
future
comparison
of
what
they
all
provided.
Envoy
checked.
All
the
boxes
that
we
needed
for
an
edge
proxy,
but
we
were
particularly
excited
about
the
c
plus
api
that
allowed
us
to
do
whatever
we
wanted
inside
the
proxy
with
a
type
safe
and
fast
language,
along
with
the
strong
unit
testing
framework
and
awesome
build
support.
A
We
were
also
really
excited
about
the
open
source
community
around
envoy.
It
was
very
receptive
to
any
issues
that
we
had
and
people
were
willing
to
work
with
us
on
workarounds
or
to
encourage
us
to
contribute,
pull
requests
to
the
envoy
code
base
and
improve
the
proxy
better,
for
it
make
it
better
for
everyone
to
get
ready
for
envoy.
We
did
a
lot
of
operational
homework
to
make
sure
that
we
had
fast
and
reliable
deployments
well,
integration
with
our
ci
cd
and
monitoring
and
alerts.
A
We
wrote
plenty
of
run
books
around
envoy
to
make
sure
we
were
prepared
for
when
things
could
go
wrong
and
we
wrote
numerous
extensions,
both
internally
and
open
source,
to
make
envoy
feature
complete
and
provide
exact
parity
with
our
existing
proxy.
Our
objective
here
was
to
have
such
close
feature
completion
that
we
could
essentially.
A
A
We
started
with
the
java
service
framework,
since
java
is
one
of
the
most
commonly
used
languages
at
pinterest
and
that
migration
involved
a
lot
of
work,
including
bug,
fixes
and
libraries,
and
a
massive
coordination
effort
to
ensure
that
deployments
were
safe
once
the
completion
or
once
the
migration
for
java
was
completed.
We
looked
at
the
long
list
of
languages
that
we
had
ahead
of
us
python,
node
c,
plus
elixir
and
go
and
thought
man.
We
really
have
to
complete
this
all
over
again
for
each
of
these
languages.
A
That
sounds
like
a
lot
more
work
and
not
to
mention
the
intercompatibility
between
the
languages,
how
to
make
sure
that
python,
tls
libraries
play
nicely
with
the
c
plus
libraries
and
so
forth,
and
all
this
is
tls
over
thrift
as
well
to
add
another
layer
of
complexity,
rather
than
repeating
all
this
work
for
all
these
services.
Again,
we
started
looking
at
envoy
as
a
solution
to
terminate
tls
elsewhere
in
infrastructure.
A
The
way
that
we
have
completed
our
load
balancer
infrastructure
is
such
that
envoy
can
be
deployed
to
arbitrary
hosts
and
accept
arbitrary
configuration
using
environment
variables
that
are
set
by
our
deployment
tooling.
So
we
looked
at
doing
just
that.
We
played
around
with
taking
some
non-ingress
hosts
and
adding
them
into
the
onboard
deployment
group
as
an
experiment
to
say
what
happens
if
you
put
envoy
there
and
once
it
is
there,
what
does
that
configuration
look
like
and
play
around
with
the
different
settings
that
we
needed
to
successfully
terminate
tls
on
a
canary
machine?
A
Once
we
had
a
successful
configuration
going,
we
adjusted
our
ingress
load
balancers
to
recognize
this
new
deployment
environment
that
had
envoy
trimming
tls
and
using
envoy's
incremental
routing
or
sorry
weighted
routing.
We
gradually
shifted
traffic
from
non-envoy
tls
to
envoy,
with
tls,
and
eventually
we
got
something
like
this,
which,
if
you
squint
a
little
bit,
it's
kind
of
starting
to
look
like
a
mesh
right.
A
While
we're
adding
all
these
services
to
the
service
mesh,
our
control
plane
stayed
relatively
simple.
One
of
envoy's
shining
features
that
many
people
know
it
for
is
its
xds
api
that
allows
it
to
be
configured
over
http
or
grpc,
and
we
did
leverage
that
for
service
discovery
since
service
discovery
is
a
highly
dynamic
component
with
hosts
coming
in
and
out
of
service
discovery
all
the
time.
A
But
the
other
configuration
aspects
of
envoy
like
routes,
clusters
and
listeners,
didn't
really
change
all
that
often
for
us.
We
find
us
tweaking
them,
maybe
once
or
twice
a
week,
maybe
more
often
than
that
if
we
were
doing
an
active
migration,
but
those
all
stayed
relatively
static
once
they
were
in
place.
So
we
figured.
A
This
example
here
shows
the
ideal
base
case
for
a
service
owner
assuming
they're
using
all
the
defaults.
We
would
like
them
to
use
and
aren't
doing
anything
too
crazy
with
their
service.
You
can
ideally
just
import
some
macros,
define
your
ingress
and
then
define
the
cluster
you
want
to
talk
to
and
the
rest
just
gets
generated
from
there.
A
This
has
saved
us
many
times
from
bad
configuration
options,
even
just
typos
or
whatnot
that
get
post
into
a
pull
request
and
the
ci
tools
would
check
and
say
no,
this
isn't
going
to
work.
This
isn't
going
to
make
it
to
production,
and
so
we
can
fix
our
configurations
before
even
putting
it
into
the
pipeline.
A
At
this
point,
we
had
really
nailed
the
fundamentals
of
what
it
meant
to
package
and
deploy
envoy
in
a
generic
fashion.
We
had
the
ability
to
deploy
envoy
to
any
service
through
our
deployment
tool.
We
had
the
ability
to
configure
envoy
for
any
service
through
our
compiler
config
compiler
and
configuration
pipeline.
A
A
We
also
had
a
build
system
through
the
open
source
onboard
project
that
allowed
us
to
write.
Cpus
plus
extensions,
apply
patches
and
regularly
synchronize
our
envoy
build
with
upstream's
code.
We
were
often
syncing
our
upgram
code
up
to
twice
a
week,
deploying
new
envoy
builds
and
the
build
system
that
allows
us
to
do
that.
Helped
us
go
very
quickly.
A
A
One
of
my
favorite
use
cases
here,
one
of
the
first
ones
that
came
from
security
is
internal
web
envoy,
the
idea
being
that
pinterest
has
a
lot
of
internal
web
services,
websites
that
are
running
for
a
variety
of
reasons,
deployment
tooling
or
say
an
ml
tool
to
analyze
scoring
on
images.
Whatever
you
could
imagine.
A
To
do
so.
The
security
team
built
a
configuration
pipeline
that
enables
all
to
be
deployed
with
the
standard
and
well
accepted
security
defaults
for
web
services,
such
as
csrf
protection
course,
protection,
tls
termination
and
click
tracking
and
crosstalk
scripting
prevention.
Headers.
All
this
without
the
web
developers
having
to
really
do
much
except
define
their
service
in
the
envoy
configuration
jinja
language.
A
The
key
idea
here
being
that
internal
web
envoy,
while
being
a
really
cool
pattern
and
deployment
strategy,
is
really
just
another
node
in
the
mesh.
It's
not
a
unique
case
that
we
need
to
cater
to.
We
simply
need
to
write
the
configuration
dsl
for
it
and
then
treat
it
just
like
any
other
node
mesh
deployed
to
a
host
and
given
an
arbitrary
configuration,
and
this
now
runs
in
front
of
all
of
our
major
web
services,
including
open
source
software
such
as
fabricator
and
jenkins,
as
well
as
a
variety
of
internal
web
services.
A
Some
more
mesh
specific
use
cases
also
started
to
come
our
way
after
proving
the
success
with
internal
web
envoy.
The
web
infrastructure
team
had
a
particularly
interesting
case
where
the
deployment
patterns
used
for
the
web
app
were
really
difficult
and
fragile
to
work
with,
and
it
was
constraining
the
team
from
adopting
newer
infrastructures
such
as
kubernetes
by
analyzing
the
problem.
We
realized
that
this
is
really
a
routing
problem
and
if
we
could
write
an
envoy
filter
that
would
correctly
route
requests
to
an
arbitrary
destination
that
we
want
them
to
go
to.
A
Then
the
cyber
liability
team
came
to
us
and
said
well,
if
we're
deploying
envoy
everywhere,
then
perhaps
we
could
do
sli
monitoring
in
a
very
generic
way
for
http
and
thrift,
which
is
true
regardless
of
what
language
you
are
using.
We
could
deploy
all
in
front
of
your
service
and
have
envoy
monitor
for
success
and
error
rates
and
report.
Those
back
and
the
site
variability
team
can
then
compose
error
budget
reports
that
are
generic
to
any
application.
A
A
To
do
so.
We
also
wrote
an
album
extension
to
solve
this
problem
as
well,
and
so
we
started
to
accumulate
all
these
external
use
cases
and
start
to
really
become
a
platform
that
people
wanted
to
use.
We
often
did
not
have
to
ask
people
to
use
envoy.
It
was
more
often
that
people
came
to
us
asking.
Can
you
put
onboard
in
front
of
our
service,
and
I
would
like
to
reflect
on
how
did
we
get
here?
How
did
we
build
this
mesh?
A
Almost
seemingly
accidentally,
we
started
off
with
an
ingress
load
balancer
and
ended
up
with
envoy
being
deployed
everywhere
and
having
many
new
use
cases
that
we
had
never
foreseen
before,
and
I
like
to
believe
that
we
got
here
by
solving
business
problems.
First,
we
never
really
had
a
meeting
saying:
let's
build
a
service
mesh,
it
was
always
about
what
business
problem
is
it
that
we
need
to
solve
in
the
first
business
problem,
we
needed
a
more
reliable
and
better
observability
in
our
english
load
balancer,
and
so
we
deployed
envoy.
A
Then
we
needed
tls
termination,
so
we
took
envoy
as
is
and
deployed
it
elsewhere
and
then
very
custom
use
cases
came
in
like
cookie
monitoring
and
sli
monitoring,
for
which
we
could
write
extensions
using
this
powerful
c
plus
api
that
we
love,
and
in
that
process
we
began
delivering
incremental
progress
and
showing
business
value
at
each
step.
There
was
no
mass
time
where
people
were
waiting
for
a
new
mesh.
It
was
kind
of
incrementally
growing
and
accumulating
new
use
cases
that
were
very
specific
to
the
problems
that
pinterest
needed
to
solve.
A
In
this
process.
We
also
unified
the
traffic
and
service
framework
teams.
We
found
that
they
were
solving
very
similar
problems
in
regards
to
proxying
and
service
management,
and
so
we
decided
to
unify
those
efforts
in
following
the
idea
that
every
envoy
proxy
is
just
a
node
in
the
mesh.
The
ingress
load
balancers
from
the
mesh
perspective,
aren't
really
that
special
they're,
just
more
awful
proxies.
Getting
configuration
also
got
here
by
having
a
build
system
that
allows
us
to
write
these
powerful
envelope
extensions
and
customize
the
behavior
at
the
data
plane.
A
While
you
do
want
to
orchestrate
as
much
of
the
control
plane
layer,
it's
very
powerful
to
have
the
ability
to
manipulate
the
data
plane,
behavior
http,
cookie
monitoring
is
not
something
you
can
really
implement
as
a
control
plane
feature,
that's
really
something
that
has
to
be
done
at
the
data
plane
when
it's
running
the
traffic
through
it
and
lastly,
but
most
importantly,
we
got
buy-in
from
teams
outside
of
the
traffic
team.
It
wasn't
the
traffic
team
pushing
for
people
to
adopt
our
infrastructure.
A
It
was
a
lot
of
people
asking
to
use
traffic
infrastructure
and
that's
a
fantastic
position
to
be
in
as
an
infrastructure
team
and
throughout
all
this.
We
of
course
will
grow
and
incrementally
evolve.
The
mesh
to
handle
more
complex
use
cases
and
our
control
plane
is
increasingly
becoming
more
robust
and
centralized,
and
we
now
stream
our
configurations
over
grpc
instead
of
flat
files
deployed
to
hosts,
but
we
always
push
this
complexity
until
we
actually
need
to
do
it
and
folks,
instead
on
solving
business
problems.