►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Real-time Kubernetes: How Entain Australia 10x'd throughput with Linkerd - Steve Gray, Entain Australia
Entain Australia operates a large scale Kubernetes deployment consisting of 300 microservices, 3000 pods, and multiple Kubernetes clusters across multiple regions. Due to the nature of Entain's business, this system must be as close to real-time as possible. In this talk, the Entain team describes how they moved from connection-level load balancing to request-level load balancing and increased overall application throughput by an order of magnitude by installing Linkerd. They discuss the impact of Linkerd's load balancing on gRPC calls, the decision-making process for choosing a service mesh, and lessons learned rolling out a service mesh to production.
A
We
operate
all
across
the
world
with
brands
and
teams
in
dozens
of
countries
globally
and
australia
is
one
of
them
as
the
head
of
trading
solutions
here
in
australia,
I
lead
the
team
of
engineers
that
build
and
operate
our
trading
platform
pairing
our
regional
business
we've
transitioned
over
the
last
few
years
from
a
non-agile
release
cycle
to
a
lean,
agile
management
practice
with
ownership
and
self-support
of
our
solutions.
We
use
chaos,
engineering,
automated
tests,
continuous
deployments
and
even
spot
nodes
to
optimize
our
cost
base.
A
Our
team
is
responsible
for
managing
the
torrent
of
data
coming
into
the
business
reacting
to
it
in
real
time.
Various
data
providers
from
all
over
the
world
feed
us
information
continuously,
as
sports
and
races
happen
all
across
the
world.
These
come
into
our
trading
platform
that
we've
built
here
in
australia.
A
A
So,
specifically,
what
we
do
is
this
piece
here
in
the
middle,
our
software
stack
is
based
on
very
contemporary
tools,
so
we
use
go
as
our
main
programming
language
for
messaging
and
distribution
of
data.
We
use
apache
kafka
mongodb
is
our
main
dollar
store,
but
we
do
use
other
technologies
such
as
python,
r,
rabbitmq
and
even
net
core
entire
platform
is
based
on
cloud
native
technologies.
A
A
A
It
wasn't
always
this
way
our
microservices
journey
started
off
just
with
a
single
application.
It
powered
the
entire
business
all
of
our
customer
facing
applications,
but
also
the
entire
back
of
house
trading
operation.
It
was
so
large
and
unassailable.
We
called
it
the
monolith
and
it
was
1.7
million
lines
of
php.
A
A
A
That
was
a
flex
that
was
only
possible
because
we'd
we
haven't
undertaken
this
work
in
terms
of
the
community's
practice
as
well.
It
was
really
good
for
us
because,
like
us,
the
new
customer
facing
platform
was
written
entirely
in
go
and
eventually
the
monolith
went
away,
though,
if
you
look
just
at
our
system
we're
around
347
services
just
for
our
team.
A
In
terms
of
our
service
mesh
design,
it's
actually
quite
chaotic,
here's
a
preview
from
buoyant
cloud
showing
the
topology
map
of
our
services,
but
if
we
zoom
in
on
just
a
small
section
of
it,
you
can
see
that
there
you
know
these
are
these
tiny
little
blocks?
Are
the
individual
services
that
make
up
our
fabric.
A
A
A
And
as
you
grow,
a
micro
services
stack
like
that.
There's
a
few
interesting
areas
and
questions
that
come
up
for
us.
The
three
major
ones
were:
how
do
our
services
find
each
other
in
this
topology?
How
do
we
manage
the
governance
of
what
service
does?
What
and
when
does
a
service
need
to
be
deprecated
or
removed,
and
also
how
do
we
observe
and
measure
our
services
to
make
sure
that
we
understand?
What's
working
and
what's
not.
A
So
over
the
years
we
actually
used
a
variety
of
ways
to
approach
service
discovery
and
connectivity
between
our
services,
but
as
we
transitioned
to
kubernetes,
we
transitioned
using
the
kubernetes
logical
services,
so
service
a
would
talk
to
service
b
by
referencing
its
service
name
and
route
to
the
individual
pods
of
service
b.
So
we
can
see
here.
In
this
example,
pod
a
is
balancing
across
pods
x
and
y
from
surface
b.
A
Now
we
use
grpc,
as
I
mentioned,
for
our
backbone
protocols,
and
this
means
that,
unlike
http
one,
we
have
fairly
long-lived
connections
once
grpc
has
established
a
pool
of
connections.
Typically,
these
will
keep
being
reused
for
a
fairly
long
time,
and
what
we
found
was
that
a
given
service,
the
individual
pods,
would
particularly
talk
to
other
pods
of
other
services,
because
the
connections
were
being
reused
rather
than
balancing
across
multiple
instances.
A
Now
what
would
happen?
There
is
with
a
large
enough
number
of
pods
on
both
services.
You
roughly
end
up
with
an
even
distribution,
so
it
doesn't
sound
too
bad
at
this
point
now,
where
things
got
a
bit
more
exciting
was,
as
we
rolled
out
code,
we'd
start
gradually
rotating
through
the
instances
of
service
b
and
we'd,
shut
them
down
and
start
a
new
one,
and
what
tended
to
happen
was
the
very
first
service
that
came
up
on
the
new
deployment
would
start
receiving
almost
all
of
the
connections.
A
A
A
This
went
poorly
when
we
started
bulk
processing.
We
suddenly
found
that
there
was
a
lot
of
pressure
in
our
system
and
the
the
load
was
massively
imbalanced
across
our
topology
and
the
latency
started
spiking,
and
this
is
not
a
good
thing.
We
tried
everything
we
rolled
services,
we
restarted
deployments,
we
scaled
up,
we
scaled
down,
but
no
matter
what
we
did.
A
And
as
we
started
looking
at
this
issue,
the
work
became
more
and
more
critical
hours
became
days.
We
started
working
late.
We
start
to
look
at
alternatives,
including
writing
our
own
custom
grpc
connection,
pooling
code,
something
that
would
forcibly
balance
the
traffic
between
the
pods
of
our
services,
and
this
is
something
we've
done
before,
but
we
tried
loading
in
the
data
during
quite
a
times
and
for
a
week
we
basically
lived
on
caffeinated
drinks
in
prayer,
and
then
one
of
our
team
asked
yeah.
Have
we
considered
a
service
mesh?
A
A
A
A
So
we
found
an
article
on
the
kubernetes
site
after
a
furious
5-10
minutes
of
googling,
where
we
found
william
morgan
from
buoyant
talking
about
grpc
low
balancing
on
kubernetes,
and
he
was
talking
about
our
specific
problem,
which
was
awesome,
and
so
after
a
bit
of
discussion
within
the
team,
some
prototype
installs
of
the
various
options
we
had
and
lots
of
cursing
and
configuration
that
we
had
to
do.
We
kept
coming
back
to
that
article
over
and
over
again,
we'd
found
our
front
runner,
and
that
was
linker
d.
A
So
before
we
went
into
a
large-scale
enterprise
deployment
of
it,
we
took
a
lot
of
steps.
We
essentially
joined
their
slack
channel.
We
talked
to
a
few
people.
Nobody,
there
seemed
to
think
this
wouldn't
work,
but
obviously
we
were
going
from
naught
to
60
miles
an
hour
very
quickly
on
this.
So,
after
a
bit
of
agonizing
about
the
risk
of
doing
this
in
a
production
system,
we
ultimately
decided
that,
because
we
could
uninstall
and
reinstall
very
easily,
we
gave
it
a
shot.
A
In
terms
of
the
deployment
process,
here,
we
use
spinnaker
for
all
our
deployments,
so
we
use
the
the
two
commands
to
install
linkedin
to
the
cluster.
That
was
roughly
15
minutes
of
the
process,
most
of
which
were
spent
reading
the
documentation,
because
the
installation
process
was
essentially
two
commands.
A
A
Now
that
took
us
a
couple
of
hours
because
we
had
to
go
through
one
by
one,
restarting
things
and
doing
deployments,
but
it
was
far
simpler
than
doing
code
level
changes
and
once
we
got
everything
installed
and
running
and
we
saw
our
pods
start
turning
up
in
the
mesh,
we
just
let
it
sit
for
a
few
hours,
because
it'd
been
a
long
few
days
and
also
just
to
see
if
anything
happened
under
baseline
load.
We
also
caught
up
on
some
sleep.
A
A
It's
pretty
impressive
in
terms
of
a
change.
Our
baseline
was
dramatically
improved
instead
of
saturating
the
giga,
the
20
gigabit
links
on
just
a
few
of
our
servers.
We
were
suddenly
processing
a
network
line
rate
on
all
of
our
servers,
the
cpu
and
load
evened
out
across
the
cluster,
and
we
just
kept
on
trucking.
A
A
It
turns
out
that,
because
linker
d
was
performing
the
connection,
pooling
and
rebalancing
for
us,
our
applications
were
balancing
at
the
request
level
for
each
individual
call,
whilst,
while
at
the
application
level
still
believing
they
had
a
pool
of
reused
connections,
this
evened
out
the
workload
and,
for
all
intents
and
purposes
perfectly
we
could
suddenly
restart
services
terminate
them.
We
could
go
back
to
our
chaos
engineering
practices
all
without
interrupting
the
flow
of
data
in
real
time,
and
the
way
it
works
is
by
injecting
a
proxy
into
each
of
your
pods.
A
A
When
we
installed
linker
d,
one
of
the
things
we
found
within
the
first
couple
of
days
was
our
bills
were
coming
down.
It
turns
out
that,
because
linkery
is
biasing
by
latency,
it
meant
that
there
was
a
bias
towards
traffic
staying
within
the
same
geographic
zone
versus
spreading
out
across
the
geographic
area.
A
And
aside
from
the
cost
savings
as
well,
we
had
a
few
other
small
wins.
We
had
service
metrics
out
of
the
box,
so
we
could
get
rid
of
our
own
metrics
collection
for
the
paths
from
service
a
to
b
to
c
there
was
instrumentation
and
tracing
support,
which
was
the
start
of
our
jaeger
journey
and
those
traces
could
span
across
our
network
proxies
and
track
the
flow
between
services,
essentially
for
free.
A
A
In
terms
of
our
journey
onward,
some
of
the
major
forward-looking
changes
we
have
left
to
come
for.
Our
infrastructure
include
deploying
chaos
mesh
more
fully.
We've
got
some
stuff
working
with
it
now,
but
we
want
to
take
that
to
the
next
level
and
we're
also
continuing
our
journey
with
linker
d,
we're
going
to
switch
from
having
three
separate
cross
zone
clusters
to
having
isolated
individual
clusters
per
geographic
area.
A
In
summary,
though,
we
have
a
lot
of
services,
we
need
to
do
things
very
fast
and
when
we're
using
grpc
or
other
long-lived
rpc
protocols
within
kubernetes,
it's
very
easy
to
end
up
out
of
the
box
with
hot
paths.
In
your
platform
and
inefficient
load
balancing,
we
were
able
to
deploy
a
service
mesh
to
production,
to
hundreds
of
services
with
zero
prior
experience
in
the
space
of
a
few
hours,
and
we
reaped
immediate
operational
and
cost
economy
benefits
more
information
about.