►
From YouTube: Banking on Kubernetes, the Hard Way, in Production - Miles Bryant & Suhail Patel, Monzo Bank
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Banking on Kubernetes, the Hard Way, in Production - Miles Bryant & Suhail Patel, Monzo Bank
Have you ever wondered what it takes to deploy and operate Kubernetes at scale, in production? Self hosted clusters often have a bad reputation as a complex and costly choice when compared to managed services. Let’s look at an alternative viewpoint and evaluate the tradeoffs. Monzo started self hosting Kubernetes out of necessity a few years ago when managed services were in their infancy. In this talk, Miles and Suhail will explain why Monzo has chosen to continue operating their own clusters, and how this can be a viable option. This talk discusses Monzo’s experiences with operating, upgrading, customising and configuring the control plane, networking and cloud integrations. We think this has been the best choice, but we also want to discuss the challenges we’ve faced along the way - like the time when network policies took down a large chunk of Monzo's cluster.
https://sched.co/Zeot
A
Hey
welcome
to
our
presentation,
banking
on
kubernetes
the
hard
way
in
production,
so
we
work
at
a
company
called
monzo,
which
is
a
bank
in
the
uk
and
we
absolutely
love
kubernetes.
We've
been
using
kubernetes
since
the
very
early
days
since
the
1.2
days.
When
we
tell
people
at
meetups
and
conferences
that
we
run
our
own
kubernetes
clusters,
we
self-manage
our
own
kubernetes
clusters.
We
generally
get
met
with
looks
of
incredulity
and
questions
of
like
what
wait
why?
Why
are
you
doing
that?
I.
B
Mean
we
realized
that
there
are
not
very
many
talks
out
there
about
people
who
actually
run
self-hosted
clusters
in
production
like
most
of
the
most
of
the
content
out.
There
is
about
using
managed
services
or
kind
of
toy
clusters
on
your
kind
of
laptop,
and
we
thought
we'd
really
like
to
kind
of
share
our
journey
of
self-hosting
talk
about
some
of
the
benefits
we've
got
and
but
also
talking
about.
Why
like,
if
we
are
starting
from
scratch
today,
we'd
probably
want
to
go
for
a
managed
service.
A
So
at
monzo
our
goal
is
to
make
money
work
for
everyone.
We
deal
with
the
complexity
of
making
money
management
easy
for
all
of
you
for
those
who
haven't
heard
about
monzo
before
we
are
a
fully
licensed
and
regulated
uk
bank
with
no
physical
branches,
you
can
manage
all
of
your
money
and
finances
by
the
app
we
hit
a
milestone
of
four
million
customers
earlier
in
the
year,
and
that
number
makes
us
bigger
than
some
major
institutions
in
the
uk
that
have
been
around
for
over
a
hundred
years.
B
A
A
How
did
we
get
started
by
self-hosting
our
own
kubernetes
cluster?
Getting
a
banking
license
in
the
uk
takes
a
really
really
long
time.
We
had
some
time
to
make
some
upfront
and
long-term
architectural
decisions.
We
wanted
to
get
these
right
because
nobody
really
wants
to
replace
the
fundamental
technology
that
underpins
a
running
bank.
We
wanted
our
systems
to
be
highly
extensible,
scalable,
resilient
and
secure,
and
naturally
we
chose
microservices
as
a
natural
fit
for
most
of
those
things.
We've
really
gone
in
on
microservices.
B
I
think
this
describes
our
like
general
architecture,
strategy
and
monsoon
right.
If
you
look,
if
you
look
at
that
graph
like
based
on
that
trend,
I
think
we're
probably
going
to
have
more
microservices
than
the
number
of
atoms
in
the
known
universe
in
about
five
years
time,
but
we'll
see
we'll
see
how
that
goes.
B
Here's
a
simplified
architecture
diagram
anyway.
Every
one
of
these
1600
dots
is
an
interpolation
microservice.
Each
one
of
the
like
the
graph
edges
is
a
connection
between
microservices
and
this.
This
number
of
microservices
allows
us
to
be
really
flexible
in
how
we
scale
our
services.
It
allows
us
to
be
flexible
in
how
we
organize
teams
and
ad
engineers.
A
B
They
start
first
started
out
with
mesos
and
marathon,
but
we
quickly
run
into
some
of
their
limitations
and
lack
of
features.
Let's
just
say
so.
We
decided
to
make
a
very
bold
and
strategic,
but
on
a
new
open
source
project
that
just
about
called
kubernetes
at
the
time,
kubernetes
had
only
just
released
like
a
1.0
version
in
mid-2015,
and
there
wasn't
really
any
kind
of
like
automated
tooling
to
spin
up
a
cluster.
There
were
some
like
was
a
nice
kind
of
like
managed,
gk
or
eks.
B
A
B
How
cp
thought
thing
works,
and
this
hundred
milliseconds
figure
is
actually
a
like
a
tunable
parameter
in
the
kernel
and
kubernetes
used
to
hard
code.
This,
and
it
didn't,
allow
us
to
kind
of
extend
this.
So
after
we
cut
this
down,
we
kind
of
opened
initially
discussed
with
the
community
proposed
to
fix,
and
we
came
up
with
our
kind
of
like
our
own
own
fix
for
this,
and
we
were
able
to
test
this
ourselves
run
it
in
our
own
cluster.
A
This
ties
in
with
one
of
the
other
benefits
is
that
we
have
the
ability
to
completely
customize
our
own
kubernetes
experience.
We
were
mocking
arch
linux
earlier,
but
arch
really
does
let
you
easily
customize
and
tune
every
aspect
of
linux
to
your
liking
and
by
self-hosting
a
kubernetes
cluster.
You
gain
the
same
flexibility.
A
We've
hit
numerous
scaling
issues
and
problems
with
our
setup
that
we
were
able
to
fix
by
tuning
api
server
flags
or
investigating
and
fixing
bottlenecks
in
components
like
lcd
we've
been
able
to
integrate
custom
components
quite
easily
like
the
vertical
pod,
auto
scaler,
and
completely
automated
the
certificate
generation
and
distribution
using
the
volt
pki
back
end,
as
well
as
our
own
custom.
Auditing.
A
Of
course,
kubernetes
is
moving
closer
and
closer
to
a
world
where
you
can
easily
extend
and
customize
without
having
to
reconfigure
the
entire
control
plane
components,
but
that
hasn't
been
our
experience
over
the
past
couple
of
years.
It's
only
now
that
the
kubernetes
project
is
moving
in
that
direction,
where
they're
splitting
off
into
more
and
more
components.
Almost
like
our
micro
services.
B
A
B
Secondly,
we
also
have
some
workloads
which
have
persistent
connections
to
payment
schemes,
and
these
would
have
also
been
quite
tricky
to
split
across
the
two
clusters
without
downtime
and
as
a
bank
planned
downtime
is
something
we
really
want
to
avoid
at
all
costs,
because,
quite
frankly,
it
shouldn't
be
acceptable.
As
a
bank
to
tell
customers,
you
won't
be
able
to
access
your
money
between
these
two
hours.
You
won't
be
able
to
go
to
an
atm
and
with
your
money.
B
So
for
us
doing,
those
kind
of
g-cluster
approach
was
pretty
tricky.
However,
we
did
find
a
way
to
run
both
overlay
networks
in
the
same
cluster.
At
the
same
time,
the
kind
of
like
the
simple
description
of
this,
I
guess,
is
that
we
deployed
two
different
node
groups,
one
with
flannel
and
one
with
calico.
B
A
So
you
you're
probably
watching
this
and
think
that
it
all
sounds
too
good
to
be
true.
So
I
want
to
give
a
little
bit
of
a
war
story
to
add
some
balance
and
perspective
about
when
things
don't
go
so
well.
So,
towards
the
end
of
last
year,
we
announced
the
completion
of
a
major
infrastructure
security
project
called
network
isolation.
A
We
wanted
services
to
explicitly
allow
specific
traffic
and
deny
all
other
traffic,
and
to
do
this,
we
leveraged
kubernetes
network
policies
and
built
a
whole
bunch
of
tooling
on
top
of
them.
So
essentially
what
the
tooling
does
under
the
hood
is.
Each
service
gets
a
network
policy
only
allowing
traffic
by
other
services,
which
have
a
particular
label
intending
egress
to
a
particular
service.
A
Under
the
hood
we
use
calico,
so
calico
enforces
the
network
policy
by
generating
a
bunch
of
ip
tables
rules
to
enforce
the
network
policy
so
prior
to
rolling
out
these
network
policies
across
all
of
our
services
in
enforcement
mode,
we
wanted
to
dry
run
them
to
make
sure
that
we
caught
all
of
the
services
talking
to
each
other.
We
wanted
to
avoid
an
incident
where
we
enforce
network
policies
and
accidentally
block
some
unconfigured,
but
legitimate
traffic
that
should
have
reached
its
destination.
A
To
do
this,
we
use
an
iptables
feature
to
log
when
a
set
were
on
a
set
of
network
policies.
Iptables
rules
hadn't
already
been
accepting
existing
traffic,
so
we
encoded
this
into
our
deployment
pipeline
for
services.
So,
as
new
services
were
being
deployed
by
engineers,
they
would
also
deploy
a
network
policy
which
would
log
packet
drops
but
not
enforce
them.
A
At
this
point,
we
hadn't
yet
completed
the
work
to
deploy
labels
to
all
services
with
correct,
correct
tagging
of
all
traffic
that
should
be
going
in
and
out
between
services.
This
naturally
meant
that
we
started
seeing
scenarios
where
packets
would
have
been
dropped
had
the
network
policies
been
in
enforcement
mode.
So
this
is
great.
The
logging
is
working
and
we
can
keep
rolling
out
more
and
more
refined
policies
with
the
correct
labels
across
all
of
our
services
and
see
this
graph
10
towards
zero.
A
A
So
what
we
found
one
day
is
once
a
few
services
had
been
deployed
with
this
network
policy
change
in
logging
mode,
we
started
seeing
a
big
increase
in
sporadic
networking
issues
within
our
kubernetes
cluster.
We
saw
effects
communicating
between
kubernetes
nodes
and
components
within
our
kubernetes.
A
Sub
sub
system
called
soft
irqs
software
irqs
are
kernel
threads
and
there
is
by
default
one
per
logical
cpu,
and
that's
where,
where
all
the
network
processing
happens,
including
the
evaluation
of
ip
tables
rules,
we
were
essentially
exhausting
these
kernel
threads
by
putting
down
a
log
raw.
This
was
compounded
because
we
were
using
ebs
as
our
root
volume.
So
essentially
we
were
logging
over
network
attached
storage.
On
these
instances,
kernel
threads
were
essentially
being
blocked
waiting
for
the
network
to
to
confirm
its
logging.
A
Whilst
debugging
this
incident,
we
were
considering
a
whole
bunch
of
theories.
We
run
on
aws,
so
we
were
thinking.
This
could
be
a
networking
issue
on
a
diverse's
side.
Aws
were
really
helpful
in
real
time
and
told
us
immediately
that
things
are
looking
healthy
on
their
end.
Once
the
packets
were
in
the
aws
networking
infrastructure,
they
were
making
their
way
reliably.
On
the
other
end,
it
was
just
a
matter
of
getting
into
the
aws
networking
stack.
A
So
we
had
to
abandon
the
logging
as
our
default
mechanism
to
catching
all
potential
packet
drops,
but
the
logging
did
give
us
some
really
useful
information.
So
we
wanted
to
tip
the
right
balance
about
when
we
would
turn
on
the
logging,
because
at
this
point
we
were
running
fully
on
calico.
We
were
able
to
do.
We
were
able
to
reevaluate
and
do
a
two-phased
approach
where
we
used
calico
metrics
and
we
wrote
a
custom
open
source
tool
which
was
able
to
take
those
calico
metrics
and
put
them
into
prometheus.
B
B
Would
we
self-host
again-
and
I
think,
there's
a
there's
a
few
things
to
consider
here.
So,
first
of
all,
running
your
own
cluster
is
now
like
easier
than
ever.
It's
no
longer
like
kind
of
arch
linux.
You
have
a
like
a
bunch
of
really
useful
tooling,
like
keep
adm,
there's
a
lot
of
work
going
into
like
things
like
cluster
api
at
the
moment,
to
kind
of
create
a
stream
where
you
can
just
like,
run
a
few
commands
and
get
a
production
ready
cluster
in
the
cloud.
B
So
it
is
easier
to
self-host
nowadays,
but
at
the
same
time,
managed
services
have
also
gotten
a
lot
better
and,
of
course,
being
managed
services.
You
lose
a
bit
of
control
and
flexibility
you're,
not
able
to
kind
of
like
fully
tweak
and
configure
your
setup
to
kind
of
every
degree.
But
you
do
get
the
benefit
of
like
a
number
of
really
really
smart
and
experienced
engineers
who,
like
have
the
like
the
wealth
of
experience,
running
tens
of
thousands
of
production
clusters
for
customers
and
I've.
B
A
So,
in
conclusion,
if
we
were
starting
today,
we
wouldn't
be
running
our
own
cluster.
The
offerings
from
cloud
providers
like
aws
and
google
cloud
bring
so
much
tooling
and
expertise
which
we
have
had
to
build
ourselves.
I
I
look,
for
example,
at
our
like
at
our
existing
cluster
configuration
and
it
has
been
iterated
on
constantly,
with
learnings
and
learnings
from
others,
and
that's
not
something
that
you
can
have
immediately
on
day
zero,
if
you're,
if
you're
starting
a
fresh
and
if
you
go
about
it
yourself.
A
Nevertheless,
we've
gained
some
incredibly
valuable
expertise
by
running
our
own
cluster.
When
we
evaluate
a
cloud
provider
offering,
we
can
quiz
them
on
how
they'd
handle
things
like
scaling,
lcd
and
handling,
not
lots
of
network
policies.
Based
on
our
experience
in
the
past,
we
can
identify
these
constraints
up
front
as
well
as
we
can
also
spin
up
our
own
clusters.
A
We
can
have
a
hybrid
setup
where
we
can
spin
up
our
own
clusters
relatively
easily,
based
on
the
existing
configuration
that
we
have
iterated
on
and
make
sure
that
kubernetes
is,
is
fit
for
purpose
and
potentially
send
more
patches
upstream.
When
we
find
scenarios
and
workloads
that
are
that
are
not
appropriate.