►
From YouTube: Evolving Systems Design: From Unreliable rpc to Resilience with Linkerd - Edward Wilde, Form3
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Evolving Systems Design: From Unreliable rpc to Resilience with Linkerd - Edward Wilde, Form3 (Intermediate Skill Level)
Form3 delivers cloud-based connectivity, payment processing, clearing and settlement services to a wide range of regulated financial institutions. Every transaction in fintech represents real money from real people. Protecting the integrity of service communication here is key. In this talk, Edward describes how Form3 built a payments platform that is reliable, safe, and resilient. He starts by looking at a previous generation system that suffered from unreliable messaging and poor tail latencies. He then covers modern design techniques used to solve those legacy problems. This talk examines how Linkerd is used and deployed to protect financial transactions from failure and latency. Warning this talk contains live demonstrations!
About Edward
Edward Wilde is the Platform Architect at Form3, leading the development of a novel distributed payment system allowing connectivity, payment processing, clearing and settlement services for a wide range of regulated financial institutions, including leading banks and fintechs.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
I
think
we're
good
to
start
hi
everyone
thanks
very
much
for
coming
to
my
talk:
evolving
systems,
design
with
linka
D,
my
name's
Edie
I
work
at
form,
3
I'm,
a
developer
there
and
we've
done
a
really
interesting
thing.
Over
the
last
18
months,
we've
integrated
our
platform
with
all
of
the
important
payments
infrastructure
in
Europe.
So
he
started
off
with
doing
that.
A
So
that's
been
a
lot
of
fun,
so
we've
dealt
with
everything
from
FLAC
files
with
fix
width
message,
formats,
all
the
way
up
to
cutting-edge
1990s,
soap,
XML
technology
and
the
reason
that
we've
done
all
of
that
painful
integration
is
so
that
our
customers
don't
have
to
so
they
can
rock
up
to
our
JSON
API
with
web
callbacks
and
cloud-based.
Cue
callbacks
do
one
straightforward
integration,
which
takes
roughly
two
weeks
and
they
can
achieve
as
much
as
we
did
in
the
last
18
months.
A
So
that's
the
kind
of
business
concept
behind
this
is
that
our
customers
are
usually
financially
regulated,
institutions
which
one
to
send
and
receive
payments
from
other
bank
accounts.
So
that's
the
nature
of
the
business.
It's
quite
an
exciting
business
idea
and
it's
is
proving
very
popular
and
we've
got
a
lot
of
demand
from
from
banks.
A
So
I
think
that
was
quite
a
cool
idea
and
they
got
some
really
brilliant
scientists
together
and
the
scientists
made
a
very
shrewd
observation.
They
thought
that
they
would
set
up
their
research
center
as
far
away
from
head
offices
as
physically
possible,
so
get
away
from
all
of
the
managers
and
concentrate
on
on
building
the
office
of
the
future.
It
was
very
interesting
in
three
to
five
years.
They
developed
amazing
technology.
They
started
off
with
the
laser
printer
just
to
keep
the
folks
back
in
New
Jersey
happy.
A
Then
they
developed
East
net
networking,
which
kind
of
makes
sense.
You
could
then
network
your
computer.
They
then
went
on
to
invent
small
talk
and
object,
oriented
programming
and
bitmap
graphics
and
the
first
personal
computer
that
had
a
graphical
user
interface,
which
I've
got
a
picture
of
here.
It's
the
Alto
star,
and
they
did
that
all
within
five
years,
which
is
pretty
amazing
when
it
took
us
18
months
to
integrate
a
payment
system.
I'm
impressed
with
that.
A
Another
book
which
I
really
enjoyed
reading
some
years
ago,
was
about
the
race
to
invent
the
next
generating
next
generation
operating
system
around
the
Windows
3.1
era.
Microsoft
realized
that
their
architecture
wasn't
going
to
cut
it.
They
embarked
on
a
joint
project
with
IBM
with
os/2
warp,
but
secretly.
A
So
my
story
is
nowhere
near
as
exciting
or
interesting
as
that
I'm
afraid.
But
it's
it's
all
about
day-to-day
software
development
in
small
teams
and
I've
worked
with
the
same
engineering
team
across
two
different
systems
and
I
want
to
show
you.
We
had
problems
with
reliability
of
service
to
service
communication
on
both
of
those
systems
and
some
of
the
techniques
that
we
use
to
make
that
more
reliable.
A
So
in
system
one
we
increase
the
reliability
and
we
reduce
some
of
the
latency
I
did
a
similar
thing
in
system
two,
which
is
the
current
platform
at
forum
3.
But
we
added
circuit
circuit,
breaker
technology
to
protect
services
under
under
stress,
but
we
still
got
quite
a
bit
more
work
to
do
with
tail
latency.
A
So
what
is
teh
latency
if
you
line
it
up
all
your
requests
from
fastest
to
slowest?
You
quite
often
see
that
you
get
this
hockey
stick
at
the
end
of
the
last
one
percentile.
So
you
can
see
it
here,
oops
like
shaking
Hauser,
and
you
might
think.
Well,
that's
not
really
a
problem,
because
it's
only
1%,
but
if
you've
got
a
million
users
that
are
making
one
request,
your
API.
That
means
that
a
hundred
users
are
going
to
see
the
worst
part
of
that
tail.
A
Latency
right
at
the
top
and
10,000
of
your
users
are
going
to
see
some
part
of
that
tail
latency.
So
it
does
turn
out
to
be
quite
a
big
problem
and
that's
amplified
if
your
page
view
has
several
calls
to
your
back-end.
So
if
you've
got
10
requests
going
to
your
back-end,
that's
no
longer
10,000
users,
it's
many
more.
A
So
this
first
system
that
we
worked
on
was
a
trading
application.
You
could
buy
and
sell
shares
commodities
and
FX
you've
probably
seen
one
of
these
systems
before
it
was
the
monolith.
It
was
the
worst
kind
of
monolith
the
dreaded
distributed
monolith
where
we
had
all
of
the
trappings
of
SOA
architecture,
but
we
had
all
the
disadvantages
of
a
single
shared
database
and
data
model.
So
it's
quite
hard
to
work
on
this
system,
but
nonetheless,
we
endeavored
the
basic
design
of
this
system
is
you've.
A
Breaker
architecture
as
we
talked
around
and
asked
people
what
what
was
this?
What
was
the
purpose
of
this
message
broker?
Because
we
can
see
in
the
code
that
it
was
really
doing
anything
useful
and
no
one
really
seemed
to
know
why
it
was
there.
So
that
was
a
shame,
so
we
decided
to
remove
it
in
the
lab
and
we
just
put
a
straightforward
HTTP
load
balancer
in
between
and
it
worked
a
whole
ton
better
we're
getting
much
lower
latencies
and
we
thought
well
we've
solved
this
problem.
A
Unfortunately,
about
a
week
later,
we
started
getting
these
timeouts
again,
so
that
was
a
bit
annoying,
so
we
reopened
the
project
and
we
noticed
that
there
were
very
rare
events
that
would
cause
these
timeouts.
So
there
were
a
lot
less
frequent.
Now
that
we've
got
our
load,
pallets
straight
three
load
bounds,
but
it's
one
in
a
million
or
something
would
timeout,
and
we
notice
that
these
correlated
sometimes
with
garbage
collection
events
in
the
in
the
processes
or
sometimes
strange
Network
activity
on
the
network
card.
It
would
just
get
saturated.
A
So
we
came
up
with
the
theory
that
if
we
just
retried,
the
next
request
would
have
succeeded
because
it
was
just
a
very
rare
event.
So
we
introduced
this
this
library
in
our
API
gateway,
which
we
very
imaginatively
called
the
retry
library
and
it
basically
just
retried
the
request
on
one
of
those
rare
events
and
that
approach
worked
very
well,
so
we
rolled
that
out
to
production
thinking.
A
A
So
we
then
started
to
learn
about
circuit
breakers
and
why
they
were
a
good
thing
which
sort
of
now
brings
us
on
to
the
second
system
of
his
form.
3.
We
learned
a
few
lessons
about
circuit
breakers
and
we
wanted
to
carry
those
forward
into
this.
This
new
system
and
we
thought
well,
there's
no
sense
us
writing
more
retry
libraries.
A
There's
a
slight
problem,
though,
is
it
was
tied
to
the
JVM,
which
was
something
that
we
we
weren't
interested
in,
so
he
started
looking
further
afield
and
came
across
linker
D
and
they
had
taken
the
finagle
library,
essentially
taken
it
out
of
process
and
they're
now
inside
the
request
flow
as
a
proxy
and
they've
added
features
on
top
of
finagle,
but
they've
got
that
core
4
Nagel
inside
it.
So
you
get
all
the
good
client-side
features
like
timeouts
retries
on
failures
and
timeouts
they've
implemented
that
important
circuit
breaker
pattern.
A
For
us,
it's
got
a
really
good
way
to
do
security
using
TLS,
it's
just
quite
straightforward,
and
it's
got
the
ability
to
very
easily
integrate
with
common
service
discovery
tools.
It
can
do
different
types
of
deployments
integrates
with
distributed
tracing,
and
here
I've
got
a
distributed
trace
for
you.
That
is
an
example
from
our
early
prototype
of
one
of
our
core
payments
api's.
It's
quite
hard
to
see
on
this
on
this
slide,
but
essentially
this
call
stack
will
enable
you
to
look
at
what
micro
services
are
calling.
A
What
Microsoft
is
in
at
the
top
here
you
can
see
that
we're
calling
the
payment
API,
but
that
is
in
turn
calling
the
user
API
to
check
that
you're
allowed
to
carrying
out
that
function.
But
for
some
reason
the
user
API
is
then
calling
a
whole
bunch
of
other
services,
and
we
hadn't
realized
that
until
we
started
looking
at
these
tracing
graphs
and
that
led
us
to
redesign
this
payments
API
systems
so
that
we
were
using
JWT
tokens
to
sort
of
distribute
permissioning
around
the
system
and
then
we
didn't
need
to
call
the
user
service.
A
A
So
we
decided
to
deploy
link
ID
in
its
most
common
deployment
model,
which
is
called
linker
to
linker,
where
each
node
in
your
system
has
a
local
link,
ID
daemon
and
you
proxy
all
your
service
to
service
communication
through
linker
d--.
We
also
proxy
outbound
traffic
to
third
parties
through
Lincoln
T
as
well,
so
that
we
get
visibility
on
calling
third
parties,
and
you
might
think
well
this.
This
kind
of
looks
like
a
bit
of
an
overkill.
Why
would
a?
Why
would
I
want
to
do
this?
A
Well,
you
get
all
of
those
client-side
benefits
which
we
talked
about
earlier,
the
retries,
and
you
also
get
the
ability
to
have
all
your
traffic
encrypted,
and
you
only
need
to
worry
about
managing
encryption
in
one
component
of
your
system
for
for
sort
of
on
the
wire
encryption.
So
you
don't
need
to
worry
about.
How
do
you
do
a
TLS
in
java
1.8?
How
do
you
do
it
in
golang?
How
do
you
net
we
just
encrypt
it
using
the
kadhi
and
the
other
services?
A
Don't
need
to
worry
about
that
concern
and
the
management
of
those
certificates
is
simplified,
because
we
know
that
they're
only
in
link
a
D
so
that
we're
obviously
don't
let
them
expire.
Another
benefit
that
we
sort
of
fell
into
really
was
that
we
then
noticed
that
the
only
communication
between
nodes
was
going
by
link
a
D.
A
Ok,
so
I've
got
a
two
little
demos
for
you
today.
The
first
demo
sort
of
represents
system
one
before
we
started
work
on
it
and
I
wanted
to
show
you
what
what
that
looks
like
to
the
end-user.
So
I
created
10
virtual
users
using
a
little
tool
called
k6,
which
I
came
across
while
preparing
this
this
demo.
A
It's
a
very
easy
way
to
do
some
load
testing
and
it
publishes
its
metrics
to
in
flux,
and
we
can
then
look
at
them
in
ingre
fauna
and
it's
just
going
to
send
those
requests
to
a
load
balancer
which
distributes
it
to
these
services
and
I've
got
four
instances
of
the
same
service
to
a
super-healthy
which
are
these.
If
I
can
do
this,
which
is
that
one
in
that
one
at
the
bottom
and
then
one
has
got
this
delay
sickness.
It's
a
mild
case
of
delay
sickness.
A
Only
two
percent
of
traffic
will
get
delayed
by
about
one
second
and
then
the
third
instance
has
got
a
terrible
bout
of
503
sickness
and
it's
been
hospitalized.
So
40%
of
that
traffic
is
going
to
return
a
503,
so
I
just
start
up
this
demo
and
then
we'll
have
a
little
look
at
the
docker
compose
file.
A
Dr.
Campos,
but
I,
don't
know
if
anyone
else
to
solve
this
problem,
but
the
sort
of
getting
all
the
timing
right
I
had
to
to
use
some
some
little
bash
scripts
that
I
put
in
the
the
bootstrapping
of
this,
but
it
seemed
a
bit
awkward,
but
it's
the
only
way
I
could
get
it
to
work
properly.
So
you
can
see
that
it's
kind
of
sleeping
waiting
for
that
load
balancer
to
come
up.
A
A
So
at
the
top
I've
got
this
traffic
simulator
using
k6
is
very
easy
to
create
ten
virtual
users
and
have
this
little
javascript
file
run
for
five
minutes,
and
this
javascript
file
is
dead,
simple.
It
just
hits
the
load.
Balancer
endpoint
and
by
default
k6
publishes
a
lot
of
interesting
metrics
to
in
flux,
but
bizarrely
didn't
publish
the
HTTP
status
code,
so
I
hacked
together
a
little
statement
there
to
add
some
buckets.
So
we've
now
got
it
hitting
the
load
balancer,
which,
as
I
mentioned,
is
a
nginx
configuration
and
is
really
straightforward.
A
It's
just
listening
on
port
80
there
and
then
it's
proxying
everything
to
this
upstream
label
system.
One
and
here
we've
got
those
four
services
defined
there.
So
it's
going
to
round-robin
each
of
those
services,
and
here
you
can
see
that
the
I
wrote
a
little
echo
program
and
if
you
it
just
echoes
out
this
text,
but
then
you
can
give
it
these
sicknesses
like
delay
or
response
code,
and
then
the
rest
of
this
file
has
just
got
the
infrastructure.
A
Now
my
Mac
might
start
to
levitate
as
well.
The
fans
spin
up
because
this
is
quite
intensive
stuff,
but
we
can
now
go
and
look
at
the
result
of
this
in
and
you
can
see
that
we've
got
quite
a
lot
of
error
codes
coming
here
roughly
about
10%
and
we've
got
also.
We've
got
these
outliers
that
have
delayed
for
about
one
second,
so
you
know
that
was
sort
of
an
exaggerated
view
of
that
first
system
before
we
started
working
on
it.
A
If
I
quickly
go
back
to
now,
just
in
case
I
hadn't
gone
to
the
demo
God's
altar.
Frequently
enough
over
the
weekend,
I
had
some
some
canned
screenshots
but
show
the
results
of
that
demo.
So
the
second,
the
second
demo
is,
is
very
similar,
but
we
will
inject
link
addi
into
the
request.
Flow
and
I
should
note
that
you
could
remove
nginx
here,
but
just
because
I
was
lazy,
I
didn't
want
to
change
this
load
testing
script,
so
I
just
kept
it
in
there
and
so
that
link.
A
A
Yeah
there
we
go
so
it's
come
up,
so
the
second
doctor
compose
file
is
very
similar.
We
got
all
the
same
components,
but
we've
added
a
few
new
ones.
So
we've
got
the
linker
D
container
here,
which
is
very
simple
setup.
All
you
need
to
do
is
Mount
a
volume.
That's
got
your
configuration
file
on
it
and
we
worth
looking
at
that
very
quickly.
This
is
sort
of
what's
required
to
get
console
integration.
You
just
add
this
name
er
here
and
then
we've
created
one
simple
Rooter
and
we've.
A
Given
it
some
sort
of
characteristics,
it's
allowed
to
retry
failed
requests,
and
this
percent
can
retry
basically
means
for
every
successful
request.
You
can
retry
twice
for
failure
and
then
we've,
given
it
some
very
aggressive
timeouts
which
are
not
really
production
kind
of
timeouts.
It
was
just
to
demonstrate
the
demo
of
how
link
addi
will
will
kick
in
on
a
on
the
timed
out
request
and
retry
it,
and
the
reason
that
link
addi
is
able
to
find
our
services.
Is.
A
A
A
So
I
can
show
you
that
here
I've
got
console
running
if
we
just
go
back
and
refresh
that
it's
going
to
pass
service
1
to
link
a
d-link
ADIZ,
going
to
hand
that
to
console
and
console
is
going
to
come
back
with
these
four
nodes
and
it's
going
to
send
it
the
IP
address
and
port
numbers,
and
we
had
to
update
nginx,
make
it
a
bit
simpler.
So,
instead
of
proxying
to
its
upstream
label,
it
just
forwards
all
the
traffic
to
link
addy
and
that's
that
path
segment.
A
A
So
yeah,
the
only
other
thing
I
needed
to
do
was
somehow
register
the
services
with
consoles
they
spun
up.
So
an
easy
technique
to
do.
This
is
to
use
console
registrate
err,
which
is
a
little
docker
container,
that
hooks
into
the
docker
socket
and
listens
for
containers.
Spinning
up
and
anything,
that's
got
exposed
port.
It
will
register
in
console
for
you.
So
it's
very
useful
for
this
sort
of
demonstration
stuff.
We
also
use
it
in
our
production
platform
as
well
and
then
I
added
the
infrastructure
components.
A
So
hopefully,
whilst
I've
been
speaking,
this
has
been
going
nice
in
the
background.
If
we
go
back
to
Agri
fauna
status
codes,
you
should
see
that
we've
had
a
lot
less
errors
during
demo
2.
In
fact,
none
and
we've
also
pulled
in
some
of
those
outliers
they're,
now
sort
of
at
hovering
and
they're
still
infrequently
but
they're
hovering
around
200.
Milliseconds
I'll
just
show
you
the
home
page
or
graph
honor,
because
the
k6
guys
have
done
a
nice
job
of
sending
some
metrics
with
their
load
balancing
that
load
testing
tool.
A
So
you
can
see
here
a
few
more
things
of
the
percentiles.
For
those
durations
so
just
to
show
that
it's
not
all
smoke
and
mirrors
I
think
it's
interesting
to
look
at
the
the
link
addy
dashboard.
This
is
what
each
node
of
link
Rd
has
its
own
dashboard
and
you
can
see
real-time
statistics,
so
you
can
see
here
at
the
top.
We've
got
the
Rueter
and
it's
got
a
hundred
percent
success.
But
you
can
see
down
here
that
this
retry
budget
is
flickering.
A
So
this
was
a
happy
path
trace,
so
the
outer
span
is
the
ingress
into
link
a
D.
So
that's
our
request.
Going
into
link
Adeel
4140
then
link
ID
is
calling
our
service
service
one.
That's
that
span
and
it's
taking
seven
milliseconds
to
process
that
get
request.
Then
the
last
span
I
just
wanted
to
demonstrate
how,
as
a
developer,
you
can
send
your
own
telemetry
to
Zipkin.
A
So
in
my
little
go
program,
I've
sent
it
a
span
that
said
that
this
this
data
access
took
12
microseconds,
but
if
we
look
at
let's
try
and
find
some
more
interesting
ones.
So
maybe
this
one?
That's
okay!
So
this
this
one's
an
example
of
a
retry.
So
here
you
can
understand
why
Lincoln
D
was
retry.
So
if
we
click
on
this
span
here
you
can
see
that
it
got
a
503
back
from
the
service.
A
We
can
also
look
up
super
slow
requests
in
Zipkin
and
take
a
look
at
those.
So
here's
one
that
took
a
took
a
second
but
what's
interesting
about
this
one
is,
although
that
span
here
took
a
second,
you
can
see
that
the
user
got
a
reply
in
200
milliseconds.
That's
because
link
addy.
If
we
click
here
it
timed
out
that
request.
That
was
taking
a
long
time
and
you
can
see
the
information
in
here.
That's
saying
it's
timed
out
and
then
it's
retried.
It.
A
So
here
the
the
sort
of
results
of
those
two
demos
side
by
side
and
I-
think
you
can
it's
obvious
to
see
the
error
rates
have
dropped
and
those
outliers
have
changed.
That
y-axis
is
is
different
instead
of
a
second,
it's
200
milliseconds,
so
I
recently
read
a
paper
that
Google
published
about
the
Taylor
scale.
So
this
is
an
approach
that
Google
have
taken
further,
reducing
those
those
outliers,
because
you
can't
really
just
use
timeouts
that
doesn't
really
work
by
lowering
the
time.
That's
super
low
in
production.
A
That's
that's
not
really
feasible
for
various
different
reasons.
So
Google's
approach
to
this
was
to
use
a
technique
that
they've
called
hedge
requests,
which
is
essentially
they're
betting
on
the
fact
that
there's
a
small
chance,
the
request
is
going
to
go
wrong
so,
rather
than
take
the
hit
of
retrying
that
timed
out
request
and
the
end
user
getting
a
bad
experience,
they
will
hedge
that
and
send
multiple
requests
all
at
the
same
time,
and
that
paper
takes
you
through
in
quite
a
lot
of
detail.
A
Their
approach
to
that
so
I'm
not
sure
if
that's
an
avenue
that
we're
going
to
go
down
in
the
future
or
if
the
the
community
is,
is
going
to
go
down
down
that
Avenue
with
with
service
measures.
But
it
would
be
interesting
to
see
if
that's
an
approach
that
we
can
use.
So
here
you
can
see
that
the
third
service
was
the
first
chat
to
respond
and
then
the
other
two
got
a
cancellation
message
to
say:
don't
bother,
processing.
A
A
So
I
just
wanted
to
say
thanks
very
much
to
to
the
linker
D
community
and
its
maintainer
z'
they've
been
super
helpful.
Whenever
we've
had
problems
that
they're
always
there
on
slack,
it's
a
very
welcoming
place.
So
thanks
very
much
those
guys,
this
demo
is
sort
of
party
bag
wrapped.
You
can
take
it
home
with
you.
A
If
you
want
to
clone
that
repo,
you
can
just
try
it
out
at
home,
and
yet,
if
I
need
to
mention
that
we're
a
young
startup,
a
small
team,
then
we're
looking
to
grow,
we're
looking
to
hire
engineers
remotely
or
working
in
our
London
office,
we're
looking
for
people
that
are
interested
in
open
source,
sort
of
cloud
native
programming
and
software,
as
well
as
people
that
think
that
payments
is
something
that
might
be
able
to
interest
them
during
their
day
job,
so
yeah.
Thank
you
very
much.
A
A
Do
you
mean,
like
a
sidecar
deployment,
I
think
I
mean
I'm,
not
a
suit
I'm,
not
a
super
expert
on
these
deployment
models,
but
as
I
understand
it.
If
you
have,
the
benefit
of
the
sidecar
approach
is
if
the
linker
Dean
know
died,
your
other
services
would
still
be
alive,
but
in
our
deployment
model
that
per
the
per
host
deployment,
if
linker
D
dies,
you
need
to
consider
that
whole
hostess
is
dead.
A
A
A
A
A
Yeah,
so
that's
right,
yeah
console
was
used
for
that
for
this
demonstration,
just
as
it
was
an
easy
route
to
get
it
up
and
running,
but
yeah
you
could
plug
into
any
service
discovery.
I've
forgotten
the
list
of
ones
that
it
supports,
but
it's
quite
a
long
list.
If
memory
serves
we
use
console
on
our
production
platform.
That's
why
I'm
sort
of
comfortable
using
it
yeah
I,
guess
that's
another
good
thing
about
link.
Is
it
so
it
plays
nicely
with
lots
of
other
tools.
A
A
Yeah,
so
your
question
at
the
moment
we're
not
running
on
kubernetes
when
we
built
the
platform
at
the
time
we
we
didn't
know
much
about
kubernetes
and
was
quite
a
small
engineering
team.
So
we
we
went
what
we're
comfortable
with
and
we
didn't
have
a
lot
of
time
to
spend
on
on
sort
of
docker
orchestration,
so
we're
running
on
ECS
at
the
moment
and
I
think
that's
definitely
something
that
we
want
to
look
into
in
the
near
future
is
going
to
a
better
Orchestrator.
A
A
Yes,
I
think
the
question
is:
how
does
a
service
mesh
replace
the
distributed
system
or
if
it
doesn't
sorry
dispute
racing
system?
Yeah
so
I
mean,
in
my
experience
they
sort
of
go
hand
in
hand,
so
we're
using
open
tracing
and
Zipkin,
and
we
use
it
integrated
into
linka
d
and
we
also
send
trace
telemetry
outside
of
linker
D,
so
they
sort
of
work
together
quite
nicely.
A
A
But
I
know
that
the
default
one
is
fairly
sensible,
but
they
tend
to
be
much
more
intelligent
than
your
traditional
Hardware
load
balancers,
because
they
have
a
lot
more
data
about
how
services
are
performing
and
which
network
routes
are
the
quickest
because
they're
in
the
flow
of
the
traffic
and
they're
analyzing
the
response
times
so
yeah.
We
don't
use
load
balancers
in
production
sort
of
at
that
part
of
the
infrastructure.
We
only
use
them
right
at
the
top
at
the
ingress
from
the
sort
of
the
public
routes
in.
A
Yes,
sir,
only
it
only
can
load
balance
as
far
as
I'm
aware
HTTP
traffic.
They
have
another
project
that
I'm
not
familiar
with
called
link,
addy
TCP,
which
I
think
is
now
superseded
by
conduit,
which
does
TCP
load
balancing.
But
you
want
to
talk
to
the
link
addy
guys
at
the
booth
about
that
they're
they're,
the
experts.
Sorry.
A
C
A
Yeah,
so
the
question
is:
do
we
plan
to
use
the
link
a
derating
to
do
canary
releases?
We
really
want
to
look
at
our
deployment
model
sort
of
holistically.
So
at
the
moment
that's
one
of
the
things
as
a
big
project
that
we
want
to
kick
off
is
how
do
we
do
deployments
canary
deployments?
I
think
that's
definitely
an
option.
Isn't
it
as
I've
seen
it's
a
feature.
A
D
A
A
A
A
A
So
we
have
a
request,
ID
and
if
the
backend
has
already
seen
that
request
ID,
it
will
throw
it
away,
but
if
it
hasn't
seen
it
it
will
then
process
it.
So
it's
quite
a
simple
technique
to
know
if
we've
processed
a
request
or
not
so
if
it's
lost
in
between
the
client
sending
in
the
service
receiving
its
it
will
be
safe
to
do
the
retry.
A
Yeah
I've
not
found
how
to
do
that.
That's
a
good
something
I've
often
wondered
is
how,
because
the
information
that
you
get
in
a
zip
code,
is
it's
just
the
status
code?
You
can't
see
the
request,
so
we've
had
to
fall
back
to
other
mechanisms
for
logging
that
at
a
different
place
in
the
sort
of
the
infrastructure,
but
we
do
have
a
way
of
like
logging,
the
dumping,
the
whole
thing,
but
it
I've
not
found
a
way
to
do
that
with
link
addy
all
right
thanks
a
lot
guys.
Yes,.