►
From YouTube: What WePay Learned From Processing Billions of Dollars on GKE Using Linkerd - Mohsen Rezaei, WePay
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
What WePay Learned From Processing Billions of Dollars on GKE Using Linkerd - Mohsen Rezaei, WePay
WePay processes billions of dollars worth of payments each year. As the number of services that process payment requests grow in WePay’s infrastructure, so does the challenge of monitoring, debugging, and tracing call paths and service internals that run on GKE. This session focuses on how the Platform Infrastructure & Tools team at WePay utilized monitoring services like Prometheus and Grafana to migrate their ever growing infrastructure and all of their production traffic (REST and gRPC) to service mesh on top of Linkerd in 2018. In addition, we will show how we used Namerd to bring all services together using discovery, Linkerd to power the data plane where the payments are processed and parsed behind WePay’s payment APIs, and Prometheus and NewRelic to monitor all infrastructure services' and microservices' activities in our production environment.
https://sched.co/MPfM
A
B
Thank
you,
David
all
right,
thanks
everyone
for
joining
me
today,
Moss
and
Rose
I
am
the
software
engineer
at
we
pay
now
a
chase
company
super
excited
to
be
here
talking
about
what
we
did
last
year
to
essentially
integrate
service
much
with
our
infrastructure
and
migrate,
all
of
our
production
services
and
and
traffic
to
link
Rd
last
year,
so
to
kind
of
get
things
started.
The
who's
here
have
learned
have
heard
of
linker
D
before
and
who's
played
with
lincolni
before
and
who's
in
production.
B
B
B
B
Api
is
public
payment
API
is
that
our
payment
partners
use
to
be
able
to
synchronously
authorize
charge
of
payments
and
those
are
famously
receive
from
their
users
and
so
any
successful
payment
that
goes
through
the
system
and
successful
you
are
authorized,
then
there
are
captured
in
the
background
in
a
different
time,
so
as
a
to
be
able
to
provide
that
world-class
payment
api's,
you
know
we.
The
goal
is
to
provide
a
very
high
success
rate
for
all
the
payments
that
are
being
authorized
through
a
through
the
system.
So
we
want
to
be
able
to.
B
So,
a
few
years
ago,
this
whole
all
the
payment
api
is
were
backed
by
a
single
monolithic
application
that
all
the
synchronous
payment
requests
were
going
through
it
and
it
would
validate
all
the
payments
and
then
so.
We
wanted
to
make
sure
us
providing
a
highly
available
service.
One
make
sure
we
have.
We
have
internally
and
overall
hae
system.
So
we
made
use
of
a
few
monitoring
services
to
be
able
to.
You
know,
look
at
custom
metrics,
you
know,
create
custom
checks
and
look
at
application
performance
through
all
these
different
things.
B
So
as
our
monolith
became,
the
bottleneck
for
the
whole
infrastructure
and
our
engineering
teams
started
to
grow
larger
than
you
know,
being
able
to
collaborate
on
a
single
service.
We
started
refactoring
that
monolithic
application
into
micro
services
and
by
by
doing
that,
you
need
to
find
a
solution
to
be
able
to
host
them.
So
we
chose
the
managed
kubernetes
clusters
and
running
those
micro
services,
as
containers
to
be
able
to.
B
You
know,
serve
the
same
set
of
features
in
the
background,
and
so
these
these
micro
services
are
all
split
into
different
sub
networks
and
in
return
in
two
different
decayed
clusters
to
be
able
to
kind
of
focus
on
a
specific
set
of
logics.
In
the
background,
so
to
kind
of
continue
that
H
a
story,
we
also
use
the
same
monitoring
best
practices
and
applied
it
to
the
micro
services.
So,
overall,
we
can
continue
relying
on
the
same
monitoring
services
that
gave
us
the
good
h/h
story
all
until
until
this
point.
B
So
now,
if
we
kind
of
focus
a
little
bit
and
zoom
in
on
the
micro
services.
The
way
these
microsoft
says,
communication
layers
set
up
is
the
engine
axis,
fronting
the
containers
to
be
able
to
provide
the
TLS
termination.
And
then,
if
you
take
an
example
of
service,
a
calling
service
B
from
cluster
B
to
cluster
a
service,
a
is
configured
to
a
fully
qualified
name
for
service
B.
So
what
it
does
it
tries
to
resolve
the
four
that
fully
qualified
name
in
its
configuration
once
it
has
a
once.
B
So
we
can
kind
of
take
that
and
it's
an
open-ended
kind
of
a
world
for
metrics.
So
in
this
scenario,
Prometheus
collects
the
metrics
from
these
micro
services
directly
and
sends
who,
since
they
need
to
monitor
the
health
check
of
these
services
from
the
eyes
of
all
these
other
micro
services.
It
calls
you
know,
nginx
to
be
able
to
test
the
health
of
these
services,
so
we've
successfully
been
able
to
kind
of
develop
that
infrastructure.
That
gives
us
a
few
things.
B
You
know
end-to-end
monitoring,
we're
able
to
monitor
everything,
great
we're
able
to
tell
what's
going
on
if
something
goes
wrong
in
the
infrastructure.
Every
single
piece
in
the
infrastructure
is,
you
know,
configurable
upgradeable
independently,
and
it
doesn't
matter
where
they
live
in
it
in
that
environment
or
data
center.
So
and
certain
things
to
certain
point
are
automated
and
it
kind
of
gives
us
what
we
need
to
be
able
to
run
the
infrastructure
the
way
we
want
it,
but
the
setup
is
super,
complex
and
and
complexity
and
being
so
closely
integrated
with
these
micro
services.
B
It
doesn't
really
encourage
big
improvements
on
the
infrastructure
or
the
micro
services
that
are
using
the
infrastructure.
So
we
want
to
kind
of
take
this
opportunity
and
start
moving
toward
an
infrastructure
2.0
where
we
can
start
defining
our
infrastructure
in
code
with
an
idea
to
be
doing
that
and
through
being
able
to
build
a
communication
platform.
Where
you
know,
communication,
protocol
configuration
or
changes
are
easier.
A
lifecycle
of
that
layer
is
easier,
independent
of
how
many
services
or
where
the
services
are
running
and
be
able
to
make
use
of
more
modern
technologies.
B
Like
software
load
balancing
you
know,
you
might
need
to
use
specific
algorithms
in
your
software
load
balancing.
Then
you
know
round-robin,
for
example,
and
then
make
it
easier
and
take
some
of
the
responsibility.
That's
really
the
infrastructure
is
responsibility
from
the
micro
services,
so
they
can
focus
on
developing
the
product
more
than
being
concerned
about
the
infrastructure
layers
so
and
that's
essentially,
the
goal
of
sort
of
integrating
with
service
mesh
and
being
able
to
simplify
the
infrastructure
and
make
it
more
modern,
and
that
really
involves
three
major
steps.
B
First,
is
introducing
service
much
to
the
infrastructure
that
it's
only
a
level
of
complexity.
The
other
is
integrating
all
the
micro
services
that
are
not
on
service,
managed
and
being
able
to
start
moving
their
request
to
service
mash
and
that's
opposed
for
receiving
and
sending
requests,
and
also
doing
all
that,
while
we
keep
the
high
availability
at
the
same
level
with
the
same
set
of
requirements
and
at
the
same
time,
we
want
to
make
sure
that
this
new
infrastructure
is
also
highly
available.
B
So
kind
of
recap,
on
what
service
measure
is
a
typical.
You
probably
hear
this
every
and
all
the
mesh
tox
service,
the
typical
service
nation
infrastructure,
includes
two
major
components.
One
is
the
data
plane
which
is
backed
by
a
linker
D
in
this
scenario,
and
this
is
linker
d1
and
I'll.
Explain
why
and
then.
One
of
the
reasons
is
because
we're
using
named
our
D
for
service
discovery
and
name
where
D
gives
us
that
backbone
for
the
control
plane
in
linker
d
1x
world.
So
what
name
or
D?
B
Does
it
watches
the
events
that
are
going
through
the
API
servers
for
the
GK
clusters
and
since
we're
using
the
regional
GK
clusters?
Here
we
also
get
a
GK
master
cluster
or
API
server
cluster,
where
you
are
given
a
load
balancer
with
set
of
backends
for
different
zones,
so
you're
available
in
the
region
with
multiple,
depending
on
where
your
class,
which
zones
your
clusters
are
set
up
with.
B
So
we
also
add
that
layer
of
high
availability
to
be
able
to
discover
services
in
in
case
zone
C,
for
example,
Goes,
Down
and
you're
available
in
a
B
and
C.
You
can
still
use
a
B
to
be
able
to
watch
the
events
that
are
going
through
that
cluster
and
the
same
kind
of
watch.
Events
story
happens
for
all
the
cluster
that
that
are
in
that
environment
so
and
we
call
that
a
discovery
scope.
B
You
might
already
see
that
it's
simplified.
You
know,
services
making
communications
to
one
another
they're,
not
they
don't
need
to
resolve
anymore.
You
know
names.
The
linker
DS
already
have
a
mapping
of
all
the
service
graph,
and
so
they
can.
So
when
service
a
is
sending
a
request,
a
service
C
all
it
does.
It
sends
a
request
to
its
link,
ready
proxy
and
that
proxy
takes
it
over
fold
it
to
the
destination
privacy
in
cluster,
a
and
enact
forwards.
The
request
to
service
C
and
the
same
thing
happens
from
service
to
service
B.
B
So
so,
really
that
data
plane
layer
it
provides
the
tracing
so
now
that
we've
shifted
the
visibility
or
observer
observability
from
the
services
to
that
data,
layer,
data
plane
layer.
So
these
the
proxies
can
now
do
that
distributed
tracing
for
the
services
it
doesn't
matter.
What
service
is
doing?
What
call
all
these
prices
are
aware
of?
B
B
So
so,
since
we
have
both
kubernetes
and
non
kubernetes
services,
we
need
to.
You
know
we're
using
link
31
and
that's
the
only
way
we
can
get
an
on
kubernetes
connection
in
this
infrastructure
within
the
same
discovery
scope.
So
now
monolith
needs
to
talk
to
you
service
B,
since
it
has
a
linker
D
proxy.
It
sends
a
request
there
and
then
service
B
receives
it
from
the
proxy
that's
running
in
cluster
a
so
how
that
happens
is
with
a
semi
simplified
detail
that
you
see
next
to
name
or
D
and
so
for
rest.
B
It's
super
simple,
like
there's.
No,
the
service
discovery
already
handles.
What's
going
out
for
GRP,
see
it's
a
little
bit
simplified
here,
because
you
need
to
kind
of
go
through
the
G
RPC
header
configuration
to
be
able
to
kind
of
map
your
G
RPC
packages
to
the
actual
service
name.
That
name
ready
knows
as
a
service
name
I'm
not
going
to
go
into
detail
about
that.
But
I
encourage
you
reading
our
latest
blog
post
about
this.
B
It
goes
into
detail
about
how
we
configured
both
name,
most
well,
I,
guess,
name
or
D
and
linker
D,
but
at
the
same
time,
how
we
have
our
applications
use
this
configuration
to
be
able
to
send
both
rest
and
G
RPC
requests
at
the
same
time.
So
since
both
these
clusters
and
non-kin
kubernetes
services
are
in
the
same
discovery,
scope
then
service,
they
can
also
send
request
a
service
c
by
using
the
same
name
or
d
configuration.
B
So
I
kind
of
mentioned
this,
but
linker
DS
and
the
core
telemetry
pieces
of
name
or
d
exposed
more
than
a
thousand
metrics
points.
So
then
you
can
start
gathering
information
about.
You
know.
On
the
client
side,
you
can
look
at
things
like
latency,
whether
it's
client
latency
or
overall
latency
over
the
wire
between
service.
You
know-
and
then
you
know,
request
times
and
so
on
and
so
forth,
and
then
on
the
server
side
you
can
take.
B
And
you
know
if
you're
looking
at
a
sec
set
overall
success
rate,
that's
dropping,
then
we
can
put
that
next
to
all
these
other
dashboards,
where
its
showing
internals
and
overall
health
of
the
infrastructure
and
be
able
to
live
debug.
Anything
that's
not
really
visible
from
a
service
perspective
or
service
logs,
and
you
know
things
that
are
outside
this
dashboards,
so
some
examples
that
we've
been
able
to
easily
tell
you
know
by
migrating.
One
of
our
services.
B
B
So
all
these
kind
of
puts
us
for
high
availability
and
and
I
think
this
is.
This
was
the
most
important
and
most
I
guess
challenging
part
of
adopting
this
infrastructure
and
so
avoiding
all
the
internal
hell
checking
because
he
can.
You
know,
since
these
are
all
running
in
kubernetes,
you
can
do
a
lot
of
lightness
probes
and
a
lot
of
tricks
with
internals
to
be
able
to
have
kubernetes
watch
all
the
pods
as
they're.
B
You
know
doing
their
thing
and
restart
them,
or
you
know
the
report
back
but
externally,
because
that's
what
that's,
how
link
or
DS
are
seeing
service
discovery
and
named
or
D?
We
can
then
ask
questions
like
you
know,
if
name
or
D
at
all
can
respond
to
any
discovery.
Is
it
is
it
alive
at
all,
or
if
we
can
take
an
example
for
discovering
a
service
food?
B
We
can
ask
question
through
name
and
the
API
is
that
if
I
were
to
give
you
the
service
name,
can
you
discover
it
for
me
and
so
what
we
did
internally?
We
also
hooked
up
sensor
because
that's
the
one
that's
running
the
custom
checks
to
a
service
registry
that
can
automatically
then,
as
services
come
alive
based
on
that
certain
you
know
either
deployment
or
something
that
registers
the
service
of
the
system
be
able
to
dynamically
either
discover,
say:
I
want
you
to
be
able
to
make
sure
that
these
are
these.
B
So
the
other
part
of
H
a
is
data
plan,
and
this
becomes
more
challenging
because
I
mean
compared
to
the
Premesh,
because
now
we're
looking
at
request
only
going
internally,
so
we
can't
really
rely
on
external
custom
checks
from
sensor.
So
what
we
did?
We
implemented
a
probing
service
where
it's
idempotent,
you
can,
you
know,
live
anywhere
inside
outside
kubernetes
and
it's
able
to
handle
both
restful
and
g
RPC.
B
They
use
a
framework
player
for
monitoring
that
gives
all
of
our
micro
services
and
this
probing
service
a
layer
where
it
can
assume
that
any
service
that
comes
alive
in
the
system.
It
has
that
layer
and
we
can
ask
it
for
health
check.
We
can
say
now
service
X
comes
alive
in
this
infrastructure.
Can
we
route
to
it
from
probe
probing
service
to
wherever
it
lives?
In
this
discovery,
scope.
B
B
So
what
with
this,
what
we
achieve?
Basically,
we've
been
able
to
refactor
a
lot
of
things
outside
of
micro-services
and
being
able
to
kind
of
simplify
both
at
a
configuration
level
and
life
cycling
and
we're
able
to
get
the
load
balancing
user
using
linker
D
to
be
able
to
kind
of
tweak
all
the
algorithms
for
load.
Balancing,
for
example,
that's
a
very
specific
case
and
be
able
to
use
the
right
algorithm,
the
right
places
in
the
infrastructure,
and
since
you
know
the
services
now
have
less
responsibility
a
lot
of
times.
B
B
So
one
thing
that
we're
still
working
on
and
was
super
challenging
to
deal
with
was
distributed,
tracing
using
linker
D
and
the
reason
why
that's
challenging
is
because,
when
you
start
introducing
distributed
tracing
to
these
proxies,
then
they
start
becoming
bottom
legs
under
pressure.
So
what
we
had
to
do,
we
had
to
do
a
workaround
and
be
able
and
use
other
services
like
in
astana
to
be
able
to
give
us
if
we're
using
something
more
than
something
close
to
100%
sampling
rate,
then
we
can't
really
use
the
proxies
to
do
the
same
work
as
they're.
B
B
B
To
kind
of
put
that
in
picture
it's
it's
really
about,
you
know
if
you
have
all
these.
This
is
a
snapshot
of
you
know
a
couple
of
weeks
ago
from
our
service
graph
and
each
dot
here
is
a
service
is
not
a
pod,
so
each
service
multiplied
by
the
number
of
pods
they're
running
in
the
environment,
and
you
want
to
make
sure,
as
all
these
little
dots
go
in
between
these
bigger
dots,
then
we're
not
really
affecting
any
live
traffic
as
we're
migrating
or
upgrading
things
to
newer
versions
or
new
configuration.
B
B
You
know
horizontally
scalable
kubernetes
service,
where
we're
able
to
kind
of
give
enough
information
configuration
to
each
of
these
parts
to
be
able
to
discover
all
the
things
in
the
discovery
scope
on
their
own
so
and
these
pods
being
behind
the
load
balancer.
Then
we
can
kind
of
play
with
all
these
different
release
strategies.
For
name
of
D,
so
whether
it's
blue
green
that
works
or
canary
you
want
to
test
a
new,
you
know
version,
that's
you're,
not
sure
your
backward
compatible
or
not.
Then
you
can
do
canary
and
get
it
tested
through
that
traffic.
B
As
far
as
data
plane,
it's
the
upgrades
strategy
becomes
more
interesting
when
you
start
using
the
diamond
set
pattern
and
it's
easier
for
the
sidecar,
because
really
the
lifecycle
is
tied
to
how
the
micro
services
are
deployed
and
like
what
what
are
their
lifecycle,
because
each
part
will
get
restarted
on
their
own
schedule,
based
on
deployment
or
health
check
and
whatnot.
But
in
this
scenario,
if
we
take
an
example
and
use
this
as
the
with
the
rolling
upgrade
pattern,
node
a
is
the
first
node.
That's
triggered
to
be
upgraded.
B
So
when
that's
finished,
then
node
C
is
being
upgraded,
so
one
load
C
is
being
upgraded
and
service.
One
is
trying
to
send
a
request
to
service
two,
then
that
request
will
either
fail
or
a
wool
and
depending
on
the
race
condition,
will
either
fail
or
it
that
node
B
proxy
already
knows
that
note.
C
is
not
available
at
the
moment
and
it
will
send
a
request
to
node
a
and
really
that's
that's
a
responsibility
of
both
the
this
rolling
upgrade
configuration
and
the
services.
B
B
B
D
E
B
So
that's
one
of
the
other
things
were
working
with
the
community
and
really
with
the
Boyan
team
is
trying
to
bring
that
non
kubernetes
side
of
the
stories
to
link
it
to
they're,
really
great
at
getting
all
the
features
of
one
to
two.
But
since
of
focus,
it's
been
laser
focused
on
kubernetes
we're
trying
to
kind
of
get
everything
first
and
then
so
we're
not
really
losing
functionality,
we're
actually
at
least
we're
feature
parity
when
we
go
through
this
link
ready.
Thank.
E
F
B
B
And
that's
specific
to
the
practices
I'm
assuming
you're,
wondering
I
mean
there's
a
lot
of
testing
that
goes
into
migrations
to
or
upgrades
to,
new
versions.
That
really
depends
on
what
we
see
in
our
testing
environment
and
in
our
staging
environment
and
I,
guess
through
all
that
testing
and
making
sure
that
all
these
environments
are
in
sync
with
each
other.
We
are
able
to
tell
all
the
problems
in
advance
before
we
go
to
production.
Do.
F
B
And
patterns
yes,
so
we
have
a
lot
of
data
that
goes
in
our
testing
environment.
That's
copied
from
production
that
we
can
kind
of
simulate
what's
happening
in
production.
In
fact,
we
found
a
lot
of
the
like
problems
with
load
testing
in
our
testing
environment,
then,
before
we
actually
even
go
into
production
for
migration
to
all
the
services
to
it.
Nice.
F
B
D
G
Very
nice
stuck
so,
as
you
asked
the
beginning,
there's
a
lot
of
people
who
are
curious
of
our
link
ad,
our
service
meshes
in
general,
but
having
not
used
them
in
production,
so
from
a
company
has
being
using
them
in
production.
Apart
from
all
the
features
which
occurs
are
very
nice,
what
is
the
operational
overhead?
The
difficulty
of
introducing
a
have
you
had
any
problems?
Have
you
found
yourself
using
liquidy
so
far,
the.
B
So
we're
actually
having
a
talk
yesterday
in
a
workshop
and
apparently
for
distributed
tracing,
it's
not
going
to
be
a
problem.
That's
gonna
be
solved
using
the
practice,
because
when
you
put
these
prices
under
load-
and
they
have
just
you
know
so
much
resources
left
to
dedicate
to
distribute,
tracing
and
and
since
they're
their
stands,
and
you
know
you
need
to
do
100%
sampling
to
be
able
to
get
the
whole
picture
of
that
span.
B
Then
you
can't
really
rely
on
all
these
to
be
kind
of
shipping,
all
the
tracing
information
at
the
same
time
on
their
heavy
load.
So
you
can
do
lowered
and
you
know
5%
sampling
rate,
but
that's
really
like
debugging.
Something
live,
which
is
you
know,
linker
d2
gives
it
a
with
tap
and
all
the
other
great
new
features
for
debugging.