►
From YouTube: When Failure is Not an Option: Processing Real Money at Monzo with Kubernetes and Linkerd [I]
Description
When Failure is Not an Option: Processing Real Money at Monzo with Kubernetes and Linkerd [I] - Oliver Gould, Buoyant & Oliver Beattie, Monzo
In this talk, we describe how Monzo processes financial transactions involving real money and real people in way that's safe, secure, and resilient. We show how combining Kubernetes with Linkerd creates a highly adaptive system, where Kubernetes provides a baseline level of protection against hardware and software failures and Linkerd layers on request-level resilience, including including latency-aware load-balancing, intelligent retries, and service-level measures of success rates and latency. We show how the resulting system is resilient to a wide variety of failures and protects the financial transactions that flow through it from failure, yet still allows for a rapid pace of feature development and iteration.
About Oliver Beattie
Oliver Beattie is Head of Engineering at Monzo, leading the development of the distributed systems to power a new kind of bank. He previously worked on Hailo's global micro-services platform.
About Oliver Gould
Oliver is the CTO of Buoyant, where he leads open source development efforts. Prior to joining Buoyant, he was a staff infrastructure engineer at Twitter, where he was the tech lead of Observability, Traffic, and Configuration & Coordination teams. He is the creator of linkerd and a core contributor to Finagle, the high-volume RPC library used at Twitter, Pinterest, Soundcloud, and many other companies.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
I
am
Oliver
one
of
the
two
hours
of
speaking
here
today,
I'm
Oliver
of
moms
Oh,
so
I
I
work
at
mums.
Oh,
we
are.
We
have
these
things
at
the
minute.
We
have
these
fantastic,
pink
coral
debit
cards.
If
you
don't
have
one,
you
should
totally
get
one,
but
you
have
to
live
in
the
UK
for
now.
Unfortunately,
we
have
about
for
the
last
two
years
now
been
building
a
bank,
so
the
prepaid
card
was
kind
of
a
product
we
could
launch
without
having
a
banking
license.
A
We
now
have
a
banking
license
and
we
will
launch
full
current
accounts
later
this
year.
Kind
of
what
makes
mom
so
different
is
lots
of
things,
but
on
the
technology
side
we
are
building
the
best
current
account,
and
that
means
the
best
banking
app
in
the
world.
It
has
lots
of
really
cool
features.
Already,
like
you
can,
freeze
and
unfreeze
your
card
with
a
single
tap.
You
can
order
a
replacement
card
in
the
app
you
can
analyze
your
spending.
A
Whenever
you
spend
anything,
you
can
actually
see
where
it
was
which
the
most
banking
apps
is
fairly
revolutionary,
because
you're
used
to
seeing
these
sort
of
cramped
14
characters
statement
descriptors,
which,
like,
if
you've
seen
them
in
the
UK,
it's
often
very
difficult
to
figure
out
where
you've
even
spent
money.
When
you
look
at
your
transaction
history,
which
is
to
me
and
to
us
fairly
embarrassing
whenever
you
spend
with
us,
you
get
kind
of
a
very
clear
idea
of
where
you've
spent.
A
So
here
you
can
see
I've
spent
14
pounds
only
nine
in
a
shop
called
tiger.
Whenever
you
spend
anywhere
one
of
my
favorite
features.
Is
you
get
emoji
with
every
transaction?
A
surprising
amount
of
our
back-end
code
is
dedicated
to
choosing
the
right
emojis.
So
we
take
this
quite
seriously
long
longer
term.
Our
kind
of
big
vision,
OD
loss
happened
to
my
emoji.
There
is
to
basically
improve
people's
relationship
with
their
money
in
the
future
we
want.
You
know
we
have
quite
a
big
ambition
to
be
all
over
the
world.
A
We
want
to
be
basically
at
the
center
of
people's
financial
lives.
To
date
we
have
about
150,000
cards
issued.
These
are
prepaid
cards
over
the
last
year,
so
we're
growing
quite
quickly
and,
as
I
said
later
this
year
we
will
launch
the
current
account
and
one
of
the
things
that
will
kind
of
not
be
a
feature
of
the
current
account
with
the
waiting
list
at
the
minute
is
there's
a
reasonably
large
waiting
list
for
the
prepaid
card.
A
Running
lanza
and
building
munzo
involves
an
interesting
tension
in
that
we
are
both
a
fast-paced
tech
startup
and
you
know,
that's
not
something
one,
but
we
are
also
a
highly
regulated
retail
bank,
and
that
has
a
lot
of
implications
for
our
technology.
Well,
we
want
to
use
the
best
technology,
and
that
often
means
very
modern
technology
like
kubernetes.
We
need
to
be
able
to
kind
of
defend
it
and
you
know
make
sure
we're
using
the
right
thing
for
the
medium
to
long
term,
and
we
have
a
few
sort
of
set
of
requirements.
A
These
are
probably
fairly
familiar
to
people
that
we
want
from
our
technology,
and
that
is
that
it
should
be
extensible
in
the
future,
as
I
said,
we'll
be
operating
kind
of
around
the
world
at
the
minute.
It's
just
in
the
UK,
but
we
have
you
know,
plans
to
move
beyond
that
all
around
the
world.
There
will
be
a
different
set
of
regulation.
A
You
know
that
most
places
we
will
be
will
be
highly
regulated,
but
differently
regulated
will
be
connected
to
hundreds
of
third
parties,
we're
already
connected
to
a
very
large
number
of
third
parties,
things
like
payment
schemes,
things
like
MasterCard
and
faster
payments
in
the
UK
separ
in
the
rest
of
Europe.
In
the
future,
there
are
others
like
ACH
in
the
US
and
then
also
other
things
like
identity
providers.
A
So
you
know
we
need
to
be
able
to
assert
the
identity
of
our
customers,
and
that
involves
connecting
to
a
reasonably
large
number
of
third
parties
as
well
to
get
information
about
our
customers.
There
are
a
bunch
of
others.
We
want
things
to
be
efficient,
we
are
a
start-up.
We
don't
have
infinite
money
like
like
most
other
banks
and
money
is
fairly
expensive
when
you're
funded
by
the
venture
capital
community.
A
We
want
things
to
be
resilient.
One
of
the
things
that
we
kind
of
hold
most
dear,
is
our
customers
ability
to
be
able
to
use
their
cards
all
the
time
we
had
a
fairly
large
outage
recently,
which
we're
not
proud
of
that
was
you
know,
and
we're
sort
of
taking
steps
to
remedy
that,
but
kind
of
one
of
the
things
that
we
that
were
most
important
was
customers
can
use
their
cards
at
all
times.
A
Even
if
other
parts
of
the
service
are
having
problems
and
being
connected
to
hundreds
of
third
parties
means,
inevitably
there
will
be
parts
of
the
service
more
or
less
at
any
point
in
time,
and
we
want
things
to
be
secure.
I
got
a
question
about
this
earlier
and
this
isn't
the
focus
of
this
talk.
But
as
a
bank,
you
are
a
very
high-value
target,
people
can
spend
very
large
amounts
of
resources
targeting
you
and
we
need
to
kind
of
be
one
step
ahead
of
that.
A
This
kind
of
led
us
to
a
microservices
architecture.
So
we
have
about
I
think
we
have
upwards
of
200
services
in
production
at
the
moment,
that's
kind
of
unique
services.
There
are
thousands
of
instances
of
those
services,
and
each
of
them
are
kind
of
small
decoupled
units
that
do
a
single
thing,
and
you
know
hopefully
do
that
single
thing.
Well,
like
I
mentioned
emoji
is
one
of
those
very
important
things.
A
Docker
and
Cuba.
Nettie's
are
something
we
moved
to
towards
the
middle
of
last
year.
They
do
a
really
good
job
of
kind
of
packaging
and
running
these
applications.
You
know
we
have
a
fleet
of
machines.
We
want
to
schedule
our
services
replicas
of
those
services
across
those
machines.
This
has
been
a
really
kind
of
big
win
for
us,
as
well
as
packaging,
our
own
services,
which
are
generally
stateless
micro
services.
We've
also
started
to
move
increasing
the
other
bits
of
technology
in
there
as
well,
so
state
full
services.
A
We
are
putting
Casca
in
there
at
the
moment.
We
also
run
things
like
IBM
MQ
in
Cuba
Nettie's,
which
sort
of
in
the
first
instance
sounds
not
like,
not
a
very
good
fit
that
we've
actually
found
it's
a
great
way
to
run
those
things
as
well.
When
we
first
moved
to
Cuba
Nettie's,
we
kind
of
immediately
not
sure
what's
happening
there
saved
60%
of
our
infrastructure
costs
don't
ear.
A
A
Okay,
anyway,
we're
just
going
to
bear
with
this.
When
we
moved
to
Cuba
nettings,
we
immediately
saved
60%
of
our
infrastructure
cost,
and
that
was
about
six
months
ago.
Since
then,
our
user
infrastructure
spenders
about
we've
saved
about
eighty
percent
of
what
we
were
spending
before
then
per
user,
which
we
think
is
is
pretty
great.
B
A
One
of
the
things
we
we
need
to
deal
with
a
lot
in
an
architecture
like
this
is
communication
between
services,
as
I
mentioned,
Cuba
Nettie's
does
a
good
job
of
running
these
services,
but
communication
between
them
is
really
important
when
you
have
thousands
of
things
and
all
of
them
basically
need
to
talk
to
other
ones.
A
single
request.
Well
done
all
of
them.
A
single
request
that
comes
in
via
our
API
can
can
easily
sign
out
into
tens
or
even
hundreds
of
requests
internally,
so
that
kind
of
part
of
the
system
is
really
important.
A
Cuba
Nettie's
has
the
concept
of
a
service
that
really
only
solves
service
discovery,
part,
and
it
turns
out.
You
actually
want
an
awful
lot
of
things
in
the
layer
that
that
manages
communication
between
services.
These
are
a
few
of
them.
We
previously
had
an
RPC
system
built
atop
RabbitMQ,
which
relied
on
kind
of
a
large
in
process
library.
It
solves
some
of
these
problems,
it
didn't
solve
all
of
them.
A
It
was
difficult
to
maintain
and
it
sort
of
tied
all
of
our
services
to
being
to
running
in
this
one
language
I
mean
nearly
all
of
our
service,
has
written
and
go,
but
we
do
have
things
now
that
are
not
written
and
go
and
kind
of.
If
you
have
one
library
that
you
have
to
replicate
in
n
languages,
that
does
not
scale
well.
A
So
what
we
use
for
this
now
is
we
use
linker
T,
which
basically
sits
between
all
of
our
services,
it's
out
of
process,
which
means
that
kind
of
as
long
as
our
services
can
speak,
the
RPC
protocol,
which
is
HTTP
in
our
case,
which
obviously
basically
every
language
can
they
can
talk
to
other
services
and
don't
have
to
worry
about
all
of
those
concerns.
I
kind
of
listed
there
so
yeah
over
to
Oliver
to
talk
a
bit
more
about
linking
already.
B
Who,
here
before
today,
has
heard
of
link
reading
excellent,
so
I'm
not
going
to
ask
you
to
tell
me
what
it
does,
but
who
here
is
tries
Ankeny?
Maybe
that
is
another
good
question.
Okay,
by
the
end
of
this
talk,
I
hope
more
of
those
hands
want
to
be
up.
Maybe
next
time
we
talk
more,
the
times
will
be
up
so
link
or
D.
In
our
mind,
is
what
I
think
is
a
layer,
5
load,
balancer
and
we
say
layer,
5.
B
Sorry
about
that
is
the
bitter
yep
sorry,
but
they
shouldn't
have
to
know
about
pods
or
services
or,
if
they're,
not
in
communities,
they
should
have
to
know
about
zookeeper
or
any
of
the
kind
of
services
every
details
and
the
scene.
Similarly,
down
at
the
bottom
of
the
stack,
we
have
cloud
providers
and
and
data
center
providers,
and
these
people
really
shouldn't
have
to
know
too
much
about
the
workloads
that
are
running
on
them
right.
We
want
to
be
just
able
to
ride
generic
pieces
down
here.
B
You
know
up
until
we're
talking
about
layer,
4,
TCP
connections
and,
in
my
mind,
the
job
of
Cooper
Nettie's
is
to
get
you
listening
on
a
socket
somewhere
and
at
the
top.
We
have
do
the
business
and
there's
a
bunch
of
glue
stuff
in
between
and
the
last
slide.
Oliver
should
see
a
lot
of
concerns
that
aren't
solved
by
the
lower
half,
nor
should
they
really
be
solved
by
the
top
half
and
that's
where
link
D
comes
in
at
layer,
5
and
I
should
note.
B
At
the
end
of
the
talk
I'll
mention
that
we
now
have
TTP's
for
link
review
as
well.
Well,
first,
let's
step
back
and
think
about
linking
we
all
know
linkers
from
back
in
the
day,
all
right,
ldd
and
such
what
a
linker
does
in
that
model.
Is
you
know,
you're
building
a
software
artefact
go
find
all
my
dependencies
and
pull
them
in
so
I
depend
on.
Let's
see
what
where
is
Lib
C
go
buy
me
that
artefact,
the
pretty
old
idea,
and
so
we're
talking
about
memes
and
addressing
before
we
go
further.
B
That's
the
original
service
discovery
and
beyond
that
we
had
a
list
of
service
names
and
ports,
and
these
are
tricked
into
every
box
in
the
world,
and
can
you
imagine
wearing
kubernetes
if
it's
to
work
that
way?
No
brief,
aside,
I
want
to
point
out
that
up
until
Windows
95
building
a
tcp/ip
stack
was
a
business
model.
This
wasn't
commoditized
and
freely
available.
So
this
it's
changed
a
lot
and
we're
going
to
see
similar
things
in
our
space
now
moving
on
another
10
years
after
that,
so
30
years
ago
we
finally
saw
DNS
this
copying.
B
Etsy
host
files
around
between
universities
was
just
not
scaling,
as
the
number
of
computers
came
up
on
the
internet,
so
we
needed
some
infrastructure
to
support
that
something
out
of
the
net
where
I
could
say
hey.
Who,
where
is
this
thing,
perhaps
give
me
some
actual
bound
address
for
this
logical
name,
moving
forward?
We
still
have
this
right.
It's
just
a
lot
more,
automated
a
lot
easier.
B
So
in
kubernetes
we
have
services
services,
give
you
this
very
easy
abstraction
where
I
say
service
a
is
composed
of
all
of
the
pods
that
are
labeled
FA
and
none
of
the
other
pods.
It's
really
easy,
but
it's
also
pretty
fragile
in
a
micro
service
where
you're
moving
quickly
I
mean
I'm,
not
no,
not
bashing,
it
I'm,
just
saying
we
can
do
better.
B
So
what
would
a
linker
look
like
in
a
cloud
native
environment,
especially
when
we
have
multi
tiers
of
calls
right,
so
things
come
in
through
some
front
end
feature
set
and
they
get
routed
down
to
your
user
service.
Every
every
big
system
has
a
user
service
that
everything
talks
to,
and
it's
impossible
staging
versions
of
it.
So
if
I
want
to
stage
a
new
version
of
the
user
service,
how
do
I
get
all
the
things
that
talk
to
me
to
talk
that
newest
version
of
the
stage
service?
B
It's
hard
right
in
the
cabrillo
surface
model.
I
can't
really
do
that.
I
either
have
to
put
something
into
production
and
serve
real
production
traffic
or
have
a
whole
kind
replicated
other
staging
environment
of
the
full-stack.
And
if
I
am
on
a
one
team
that
own
the
user
service-
and
these
are
all
other
teams
that
takes
a
lot
of
coordination
to
do
or
a
lot
of
either
that
or
you
have
a
really
kind
of
iron-fisted
operational
model.
So
what
link
really
introduces?
B
Is
this
concept
of
logical
naming
and
every
request
that
goes
through
linker
T
is
given
is
identified
with
a
name,
and
so
in
this
case
we'd
have
service
users
is
that's.
What
link
Reese's
or
left
of
the
application
is
that
I'm
sending
something
to
the
user
service,
and
that
could
be
go.
Look
this
up
in
zookeeper
in
the
prod
namespace
or
it
could
be
go.
Look
this
up
in
kubernetes
in
the
staging
namespace
with
HIV
port.
The
details
of
this
really
shouldn't
be
bound
in
the
application
at
all.
B
In
fact,
I
may
want
to
run
these
environments
simultaneously.
Maybe
I
have
a
legacy
system
and
I
need
to
do
service
discovery
across
both
things,
encoding
that
all
in
the
application
is
really
burdensome,
and
so
what
we
have
is
a
way
to
write.
Delegations
and
I
can
say
when
you
would
go
to
route
service,
go
look
and
kubernetes
in
this
way
or
go.
Do
a
union
or
go
fallback?
It's
a
really
powerful
language
here.
B
B
It's
going
to
talk
to
you,
the
service
go
talk
to
this
other
version,
the
user
service
instead
before
starting
point,
I
worked
at
Twitter
calm,
and
this
is
how
we
solve
stage
if
I'm
back
five
levels
in
the
call
graph,
it's
very
hard,
it's
very
fragile
and
manual,
and
so
we
introduced
something
where,
as
an
engineer,
I
can
go
to
scho
tag.
My
request
in
my
browser
with
certain
behaviors
and
that
works.
B
We
can
also
use
this
for
something
like
D
trees,
where
I
want
to
intercept
calls
across
the
system
for
some
debugging
thing,
and
maybe
this
debugging
things
actually
expensive
to
run.
So
we
don't
want
to
run
it
on
all
requests,
so
I
can
either.
Just
on
my
request
to
my
browser,
have
things
write
to
this
proxy?
Then
they'll
afford
to
the
normal
service
or
I,
can
manage
essentially
be
what
we
call
name
or
D.
B
I
also
care
about
latency,
whether
the
service
on
Salerno
I,
also
care
about
tracing
and
who's,
talking
to
whom
and
being
able
to
visualize
that
descriptively,
not
just
what's
in
the
config
files,
but
what's
actually
happening
in
the
system
and
I
want
to
do
this
in
a
way
that
doesn't
bind
us
again
to
specific
libraries.
So
if
I
have
this
really
working
well
and
go,
and
then
we
acquire
a
company
that
has
a
bunch
of
node
stuff,
how
do
I
replicate
this
exact
set
of
features
and
get
this
so
that
we're
measuring
success
rate
properly?
B
It's
actually
hard
to
measure
success
rate
uniformly
across
different
implementations,
so
link
Rd
exports.
Many
many
stats
by
default.
So
you
just
by
sending
traffic
to
link
ready.
We
get
a
lot
of
runtime
information
exported.
This
is
a
project
called
linker.
Deve
is
on
github
under
the
boy
and
io
repo,
and
this
what
the
does
is
it
just
scrapes
linker
D
steps
it
can
find
all
the
service
names,
turn
them
and
generate
a
dashboard
for
free.
B
B
Similarly,
you
may
have
heard
of
Dickon
my
team
at
Twitter
open-source,
this
and
19
2012,
and
this
is
the
call
graph
at
Twitter
in
2012.
This
is
not
supposed
to
be
very
informational.
It's
supposed
to
show
you
that
it's
super
complex,
but
also
that
how
hard
it
is
to
read
throughout
interactions
in
distributed
system.
When
we
have
this
many
services
and
this
many
staging
versions
and
n.
How
do
I
know
what
services
they're,
actually
communicating?
Managing
the
only
consume
configuration
is
not
practical,
because
it's
not
real.
We
want
flowing
that.
B
The
system
can
just
tell
us,
and
so
Lincoln
II
will
omit
different
races.
If
you
forward
some
headers
to
your
application,
so
your
application
doesn't
have
to
be
instrumented
with
lots
of
tracing
specific
things.
I
can
just
say,
I'll
just
send
this
context
through
and
with
that
we
can
get
tracing
for
free
on
the
operational
side
and
I
think
this
is
what
Oliver
was
talking
about
before
we
have
things
like
timeouts
and
retries,
and
so
we
only
start
our
system
by
choose.
Choosing
some
numbers
right.
I
want
three
retries
and
400.
B
Milliseconds
sounds
good
and
then
another
team
makes
another
decision.
Well,
I
only
want
to
do
200
retries,
and
you
know
we
can
maybe
get
this
together.
If
we
all
sent
a
room
and
talked
about
it,
but
as
a
sales
assistant
grows,
it
becomes
really
hard
to
compose
these
things.
Let
me
see
any
problems
here
or
any
arithmetic
people
in
here
these
on
composable
at
all.
If
any,
if
we
fail
anything
anywhere
else,
I
have
to
do
a
retry
anywhere
lower
in
the
stack
the
top
timeout
fills.
B
So
this
is
all
for
naught
the
only
top
time.
I
is
the
only
one
that
matters
well.
How
do
we
fix
that
deadline?
Our
future
that
we
can
implement
in
linker
D
we
propagate
them,
but
don't
act
on
them
at
the
moment,
but
this
allows
your
application
to
instrument
timeouts
based
on
the
deadlines
and
what
we
try
to
do
is
set
a
timeout
on
the
feature
so
rather
than
thinking
about
timeouts
on
a
service
this
service
is
this
much
latency
I
want
to
think
about
the
feature.
B
The
look
up,
the
emoji
feature,
how
much
resources
am
I
willing
to
give
to
that
request,
and
maybe
the
transaction
feature
I'm
willing
to
give
more
resources
to
so
I
want
to
set
different
timeouts
on
these
things
or
okay.
If
you
don't
get
an
emoji,
that's
fine
time
it
out
quickly
if
the
transactions
taking
awhile,
we
really
want
that
to
succeed,
so
maybe
give
it
some
more
time
and
we
want
to
be
able
to
set
it
again
on
the
highest
feature
and
not
on
the
service
internally.
B
Reach.
Fries
are
another
really
interesting
thing:
we've
all
probably
anyone
who's
written
a
service.
The
talks
to
another
service
has
had
to
think
about
how
that
fails
and
what
to
do
when
it
fails.
Usually
what
you
do
it
fails?
You
just
try
again,
maybe
you
back
off.
We
are
in
an
incident
where
things
are
really
failing.
What
we've
done
here
is
made
an
error.
B
Cannon
I've
now
magnified
the
load
on
my
system
when
I
most
need
it
to
reduce,
and
so,
if
we
have
a
hot
database,
we're
just
slamming
more
lower
in
the
system,
and
we
need
a
solution
of
this
finagle.
What
lincolnÃs
implemented
with
introduced
retry
budget
on
a
retry
budget
lets
us
think
about
the
amount
of
the
past
we're
willing
to
absorb
or
to
provision
it
over
provision
for
incidents
like
these,
and
so,
if
I
say
20
retry,
that's
plaintiffs.
B
B
We
also
care
about
load
shedding,
so
when
timeouts
may
happen
somewhere
in
the
system.
What,
if
I'm
doing
thousands
of
database
calls
and
we're
doing
all
this
work
and
the
time
line
service
is
never
ever
going
to
get
the
response?
It
doesn't
care.
It's
timed
out.
It's
gone
to
lunch.
We
have
a
new
deploy
going,
but
this
thing's
still
working
well.
B
We
know
what
we
need
to
do
to
be
able
to
propagate
timeouts
down
the
system,
and
so
we
can
do
this
in
HTTP
by
disclosing
the
connection
and
having
this
connection
close
propagate
in
HTV
to
we
can
do
this
to
the
explicit
stream
cancellation,
which
is
a
very
nice
feature.
Senegal
also
supports
this
protocol
called
MUX,
which
has
this
built
in
as
a
first
class
citizen.
B
B
So
don't
believe
it
yeah.
This
is
Bob.
This
all
works
in
Legree
and
the
goal
here
really
is
that
we
want
to
be
able
to
signal
things
back
up
to
clients
if
we
can
so
if
I'm
failing
here,
maybe
I
want
to
fail
over
another
region
and
I
want
to
have
the
top
of
the
stack
be
able
to
make
this
decision.
We
don't
have
automatic
failover
built
in
yet
so
we
were
working
on
that
still
but
yeah.
B
We
want
to
be
able
to
force
this
back
even
to
the
clients
if
we
can
so
that
we
can
always
go
to
another
region.
These
decisions
are
best
made
higher
up
the
stack
and
my
favorite
feature
of
linker
D,
maybe
not
favorite,
I,
think
the
most
important
feature
of
linkage
E
is
request
to
where
load,
balancing
and
so
you're,
probably
familiar
with
round-robin
load
balancing.
B
You
know,
what's
in
cou
proxy
by
default,
and
in
that
case
you
just
pick
a
note
pick
a
note
pick
a
note
speaker
notes
a
canopic
note,
and
that
can
be
very
bad
when
one
node
starts
failing,
and
so
in
this
example.
Here
this
is
a
test.
You
ran
where
we
had
a
one
second
timeout'
set
talking
to
a
10
node
cluster
and
one
node
starts
timing
out
if
I'm
using
round-robin
I
am
about
95%
success.
In
that
one
situation
everywhere,
I've
been
on
called,
that's
a
wake
me
up.
B
Success
rate
and
I
really
really
don't
like
to
be
woken
up.
It's
kind
of
my
you
know.
Motivating
factor
in
life
is
to
not
be
woken
up
by
computers,
so
I
working
on
this.
So
we
can
change
that
we
can
add
a
leaf
connections
and
that
moves
up
a
little
to
about
99
percent,
which
is
pretty
good,
but
usually
we
care
about
a
little
bit
more
secure
than
that,
especially
in
a
web
scale.
Services
like
Twitter
and
so
Twitter
developed
an
Yuma
load,
balancer
exponentially
weighted
moving
average.
B
It's
also
uses
this
very
fast
power
to
choices.
Apparently
there's
this
math.
That
proves
in
a
way
that
load
bouncing
was
initially
implemented
with
a
heap.
We
keep
a
priority
heap
of
the
number.
You
know
by
weight,
an
order
thing.
We
always
takes
off
the
top
that
can
be
quitting
quite
expensive
to
manage
and
so
power
to
choices.
We
just
pick
two
nodes
at
random
and
compare
them
and
that's
equivalent
to
keeping
an
order
to
keep
amazing
random
numbers
so
with
Yuma
and
perishing
choices.
B
B
M'a
is
to
because
we're
looking
at
the
individual
request
response,
latency
we're
able
to
make
better
decisions,
and
this
is
something
you
cannot
do
in
a
pure
TCP
load,
balancer
right,
I'm
at
layer,
four
I,
don't
know
about
request
latency,
if
I'm
at
HTTP
I
know
when
I
get
a
request
and
I
know
your
response.
I
can
measure
that
time
and
use
that
to
inform
a
load
balancer.
B
Furthermore,
finagle
dismayed
aperture
load
balancer
default
we're
considering
that
and
Link
ready
we're
unsure
about
that.
At
the
moment,
the
goal
of
aperture
is
to
limit
the
number
of
nodes
I
talked
to
at
once
to
fit
my
load.
So,
let's
say
the
user
service,
because
it's
used
by
everything
at
the
company
as
a
thousand
nodes,
but
I'm
running
this
one
little
side,
project
and
I
get
about
a
request,
a
minute
or
something
like
that.
B
Well,
if
I'm,
maybe
let's
say
I'm
getting
about
one
request
consistently
to
requests
insistently,
if
I'm
load,
balancing
over
a
thousand
node
cluster
and
I'm
picking
two
at
random
every
time,
I
have
no
load
information.
There's
no
information
make
the
decision,
because
nothing
bloated
right
this
998
of
them
aren't
loaded,
so
an
aperture
does.
B
So
some
history
of
lincolni
for
those
of
you
who
haven't
heard
of
it,
we
donated
16
CFS
in
January.
That
was
a
pretty
big
milestone.
We
initially
announced
it
in
February
of
last
year.
First
talked
about
it
at
Cube
County
last
year.
Anyone
there
for
that
yay
and
it's
been
a
long
road
system.
We've
added
a
lot
more
features.
B
We've
been
helping
a
lot
of
both
small
and
large
companies,
get
up
go
to
production
with
Lync
ready
on
kubernetes
on
mezzo
on
some
combination
of
the
two
on
other
things
is
everything
under
the
Sun
out
there.
Recently
we've
been
doing
about
to
work,
we
just
shipped
ingress
support,
so
now,
link
or
D
can
be
configured
to
consume
communities,
ingress
API
to
do
routing.
I
spent
a
bunch
of
q4
working
on
a
speech
you
in
liquidy.
So
now
we
have
a
pretty
decent
I
hope
HP
to
implementation.
B
We
have
a
to
your
PC
validation
as
well,
meaning
we
can
do
Co
gen
for
link
ready,
so
link
that
you
can
talk
to
services
and
that's
what
we're
using
for
some
of
our
new
API
s.
So
we
have
a
neighbor
D
as
our
central
route
manager.
Now
the
G
or
PC
API,
we
just
shipped,
we
just
shipped
this
week,
a
new
plug-in
API,
so
link
to
be
very
pluggable.
You
can
use
things
that
we
provide,
or
you
can
write
your
own
plugins.
B
I
mean
I've,
certainly
well.
Okay,
so
is
the
question
whether
hv2
works
in
those
environments
or
what
is
what's
the
question
I
mean
hv2
generally
works
in
these
environments
because
it's
tea
left,
that's
the
HP
one,
there's
a
lot
of
nasty
middleware,
as
you
know,
which
breaks,
HTTP
and
tries
inspect
HTTP,
and
so
this
is
why
HP
two
is
TLS
by
default
is
so
that
these
things
can
even
pretend
to
look
at
them,
and
so
they
just
see
them
as
a
secure
stream.
B
How
you're
particularly
Enterprise
deals
with
that
is
a
matter
of
policy,
so
HP
is
no
silver
bullet.
It's
still
quite
new
and
I
have
to
be
honest,
mostly
installations.
I
read,
don't
implement
it
properly,
especially
your
on
flow
control.
No
offense
go
users,
but
watch
out
and
yeah.
So
what
most
of
our
users
right
now
are
in
HP,
plain
old
HP
one.
C
B
Mean
we
have
not
run
it
against
these
specific
names,
you'd
use,
but
linker
D
supports
open
protocol,
so
it
supports
HTTP
in
HP
and
if
you're
a
middleware
supports
those
things
and
it'll
work,
that's
it
sure.
Yeah
I
mean
yeah
the
Devils
in
the
details.
I
am
quite
happy
to
announce
that
we're
marching
to
1.0.
We
will
be
shipping
it
in
April,
I.
Think
we're
cutting
our
c1
today
or
tomorrow.
San
Francisco
time
and
I
encourage
any
of
you
who
are
instantly
pretty
to
go
test.
B
B
There
are
large
number
of
applications
that
have
weird
protocols,
either
we're
databases
or
weird
enterprise
things,
and
we
still
want
to
be
able
to
talk
to
service
discovery
and
abstract
service
discovery
in
the
same
way,
and
we
want
to
be
able
to
install
TLS
outside
the
application
in
the
same
way,
and
we
want
to
do
some
level
of
request
level
or
of
stream
level
load
balancing.
We
can't
do
request
level
load
balancing
here,
so
we
implemented
a
TCP
load
balancer
in
rust.
B
Any
rust
users
out
there
not
yet
talk
to
us
in
the
year,
we'll
be
a
lot
more
hands.
Rust
is
a
great
language.
It's
native,
it's
safe
by
default
means
it's
very
fast
and
very
tiny.
I
would
love
more
input
and
easier,
whoever
raised
their
hand
back
there.
Please
come
give
us
feedback
test.
It
out,
submit
pull
requests.
This
is
the
new
project
and
we're
going
to
be
spending
a
lot
of
time
on
it
coming
up.
B
B
We
would
consider
it
I
have
to
be
honest.
The
ingress
API
from
our
experience
is
under
defined
right
now.
There's
a
lot
of
unspecified
behavior
and
I.
We
way
to
figure
out
what
the
ingress
API
actually
is
is
looking
at
traffic
and
nginx
to
see
how
they've
map
it
so
I
personally
would
like
to
get
a
little
bit
clearer
on
the
support
before
we
say
that
it
fits
with
the
ingress
model
is
but
yeah
I
think
we'd
be
totally
opening
getting
this
in
the
result.
Distribution,
yeah
question
here.
B
Next
week
its
obliquity
so
totally
separate
project
communities,
it
works
well
in
kubernetes,
but
it
works
very
well
everywhere
else
as
well.
One
of
the
real
goals
of
linker
D
is
to
bridge
that
gap
so
that
you
have
a
common
set
of
communication
tools
that
aren't
bounded
the
orchestration
layer
question
back
there.
B
So
the
last
load,
the
last
load
tests
that
I
remember,
are
three
or
four
nines
with
five
milliseconds
I
think
four
9s
five
milliseconds
it
can
be
worse
than
some
workloads,
so
it's
workload
dependent.
So
please,
if
you
test
this
and
find
problems,
let
us
know-
and
we
actually
have
a
blog
post
on
blogger
point
I/o.
That
shows
that
we
actually
reduce
latency
in
many
cases.
So
when
you're
talking
to
a
large
or
when
you're
talking
to
a
slow
cluster,
we're
slow
actors,
we
come
for
latency,
substantial,
a
question
back
up
here.
C
A
B
Yeah
so
specifically,
there's
a
blog
post
on
this,
but
Oliver
did
a
lot
of
the
hard
research
here
and
gave
us
a
solution.
We've
written
many
blog
posts
about
it,
but
we
deploy
a
linker
T
as
a
daemon
sentence.
Oh
there's,
one
link
reading
instance
per
node.
The
node
has
this
hack.
It
can
use
the
downward
API
to
discover
the
local
linker
to
address
the
the
local
product.
B
A
Did
yeah
yeah
cube
dns
is
a
consumer
of
the
endpoints
API
link.
D
talk
to
be
in
point:
API
browser
is
live,
so
clients
need
to
know
where
their
local
link
D
is
the
way
it
looks.
Is
they
they
send
a
request
with
the
host
header
of
the
service
they
want
to
talk
to,
they
send
it
to
their
local
link,
D
and
the
recessive
finding
the
local
link.
Id
is
interesting
and
that's
what
I
blog
posts
about
Pat
and.
B
C
B
B
Right
now,
they're
separate
we're
going
to
I
mean,
like
I
said
we
just
launched
some
beauties
B
this
week,
so
we're
I
like
to
get
them
to
be
able
to
run
together
so
that
we
can,
for
instance,
terminate
TLS
in
the
native
side
and
yes,
this
is
evolving
still,
but
they're
separate
at
the
moment.
I
expected
a
we'll
get
closer
together
over
time.
Chris
Beck.
C
C
B
Yeah,
we
have
some
people
doing
very
similar
things,
not
with
TCP
a
but
with
linker
D,
where
you
can
just
send
everything
you
through
here
and
then
via
the
route
control
control.
What
goes
where
you
can
also
ensure
that
everything
that
leaves
TLS
and
verified
and
all
that
yeah,
that's
exactly
right.
Anything
on
this
side
of
the
room,
I've
been
standing
over
here.
First
in
the
middle.
C
B
You're
already
running
finagle:
what's
the
preferred
way
of
using
link
routing?
That's
a
good
question:
I
mean
you
can
treat
finagle
as
anything
else.
Do
you
want
right?
It's
just
an
HTP
library
at
end
of
the
day,
so
you
can
just
have
it
sends
your
link.
Ready
depends
on
what
features
you
want
at
a
finagle
and
liquidy.
B
There
is
a
bunch
of
overlap
there,
so
it
may
not
make
sense
to
run
link
ready
in
that
case,
or
maybe
only
make
sense
for
link
ready
for
the
non
pinnacle
things
or
maybe
it
makes
sense,
because
you
want
to
have
some
policy
layer,
that's
outside
the
application.
There
kind
of
depends
on
your
requirements.
I'm
happy
to
chat
more
about
this
stuff.
Anything
here
crumbs.
C
C
A
So
the
current
account
will
be
launching
later
this
year.
When
we
launch
the
current
account,
we
will
be
migrating
all
of
our
customers
off
the
prepaid
card
and
onto
the
current
account
that
will
happen
overnight.
So
that
will
happen.
The
current
account
is
a
full
bank
account.
So
currently
the
product
is
a
prepaid
card
that
you
load
with
money
from
another
bank
account
and
then
spend
it
like.
You
would
use
a
card
in
any
other
shop,
but
you
cannot
make
bank
transfers.
You
cannot
get
your
salary
paid
into
it.
A
You
can't
pay
your
rent
out
of
it.
You
can't
get
an
overdraft
either.
Those
will
all
be
features
of
the
current
account
that
are
not
features
of
the
prepaid
card.
So
basically
the
distinction
is
with
the
prepaid
card.
Technically,
another
bank
is
holding
the
money.
We
are
sort
of
it
sort
of
presented
through
the
mon-sol
app,
but
there
is
someone
else
behind
the
scenes
holding
that
money
we
will
be.
You
know
we
will
hold
the
money
directly.
We
when
we
launch
card
accounts
in.
A
For
a
lot
of
the
current
account
stuff,
yes
for
international
stuff,
yes,
and
no
so
obviously
there
is
a
large
kind
of
portion
of
our
back-end
that
will
be
shared
across
all
countries,
and
then
there
are
there
is.
There
are
a
number
of
integrations
that
will
have
to
happen
in
each
country
and
they
will
be
unique
to
those
countries,
one
just
the
UK.
The
question
back
there
yeah.
A
The
question
is
whether
we
do
our
own
processing
as
in
card
payment
processing.
Currently
we
do
not
do
our
own
card
payment
processing.
When
we
launched
the
current
account,
we
have
connected
directly
to
MasterCard,
that's
not
used
for
the
prepaid
card,
but
if
that
is
how
the
current
account
works.
So
we
are
both
an
issuer
and
the
processor
of
the
current
account.
A
Right
so
for
the
prepaid
card,
two
separate
companies
that
are
not
Mons
ou
are
the
issuer
and
the
processor.
When
we
offer
the
current
account,
we
are
both
the
issuer
and
the
processor,
the
issue
of
being
the
person
who
holds
the
funds
and
who's
basically
legally
responsible
for
them
and
the
processor
being
the
person
who's
connected
to
Master
Card,
with
both
of
those
things.
B
It
works
well,
we
don't
have
any
known
bugs
with
it.
So
please
find
something
basically
like
we
have
people
who
are
working
towards
prod
with
it.
Today
we
I
know
there
are
some
feature:
deficiencies
in
the
gr
PC
solution
right
now,
for
instance
retries
our
hard-earned
gr
PC,
because
there
at
the
end
of
the
stream,
this
is
the
status
code,
the
some
wrinkles
to
work
through
still,
but
as
far
as
the
HP
to
support
generally
known
on
bug,
please
find
them.
I
also
have
linger
D
stickers.