►
From YouTube: Our Journey to Service 5 Million Messaging Connections on Kubernetes - Dylan O'Mahony & Dave Doyle
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Our Journey to Service 5 Million Messaging Connections on Kubernetes - Dylan O'Mahony, Bose & Dave Doyle, Connected
We propose to present a case study on how we built a high-scale messaging service and state store on Kubernetes. The solution supports millions of persistent, concurrent connections; enables tens of thousands of messages per second; is globally addressable; stores millions of states; and responds with minimal latency (<250ms). To evaluate build approaches, the team split into Makers & Breakers. Makers developed the solution stack while Breakers focused on repurposing Locust, a high-scale load testing framework, to simulate behavior. Leveraging the flexibility of Kubernetes, we were able to scale the stack and solve blockers on the path to a viable solution. Blockers included ingress, file descriptors, service discovery and resource limits. The experience was deeply educational, generating key learnings for developers tasked with building a scaled solution on top of Kubernetes.
To learn more: https://sched.co/GrRy
A
So,
thank
you
all
for
for
coming
to
our
talk.
Reaching
5
million
messaging
connections
on
kubernetes,
so
I'm
sure
that
some
of
you
in
the
audience
might
be
wondering
why
a
50
year
old,
audio
company
and
a
product
development
company
from
Toronto
or
on
stage
talking
about
such
a
thing.
So
allow
me
to
demonstrate,
and
all
things
going
well
with
conference
Wi-Fi
in
cellular
Alexa
play
lonely
boy
by
the
Black
Keys
on
stage.
A
A
Well,
it
turns
out
it's
actually
quite
difficult
and
that's
why
Dave
and
I
are
here
today
to
share
the
lessons,
but
his
team
learned
the
hard
way
and
we
connected
Bose
connected
with
connected
to
help
us
develop
a
large-scale
prototype
of
a
100%
cloud
native
solution
that
supports
at
least
5
million
sports.
Smart
speakers
I'll
just
take
a
few
short
minutes
to
explain
our
problem
domain
and
then
Dave
will
take
you
through
the
technical
stumbling
blocks
that
his
team
demolished
one
by
one
on
the
way
to
proving
out
a
solution
so
to
solve
this
problem.
A
A
So,
let's
start
with
the
the
Amazon
echo
dot,
the
actual
voice.
Marts
live
in
amazon's
cloud,
not
in
the
device
itself
and
when
I
speak,
my
voice
recording,
is
sent
up
to
the
electric
cloud
for
processing
and
once
Amazon
actually
determined
what
it
is
that
I'm
trying
to
do,
then
it
hands
off
the
instructions
to
the
Bose
cloud
to
handle,
and
we
lovingly
refer
to
our
Bose
cloud
as
Galapagos,
and
this
is
where
the
messaging
connections
come
into
play.
A
A
So
the
best
solution
to
solve
that
problem
is
to
set
up
a
persistent
line
when
the
speaker
first
powers
up
and
we
chose
MQTT,
and
we
have
a
bi-directional
WebSocket
running
from
every
one
of
these
speakers
to
the
cloud
so
that
it
can
basically
receive
these
unsolicited
messages
at
any
point
in
time
for
even
a
few
thousand
products.
This
solution
is
obvious.
It's
very
tenable
scaling
is
the
crux
of
the
problem.
A
Managing
millions
of
connections
through
a
central
broker
is
a
tremendous
challenge,
and
so
hopefully,
at
this
point,
you've
got
a
sense
for
the
problem
that
we
set
out
to
solve
and
a
couple
of
years
ago,
when
I
looked
around
for
a
suitable
solution
and
I
actually
came
up
empty.
There
are
some
cloud
systems
out
there.
Some
managed
services
that
we
could
have
gone
with
that
scales
to
hundreds
of
thousands
of
connections,
but
we
could
not
find
any
that
would
actually
scale
to
millions.
A
So
as
we
waited
for
the
industry
to
catch
up,
we
needed
to
find
some
smart
people
to
partner
with
to
prove
out
that
this
was
even
possible
and
Dave
was
the
leader
of
these
smart
people
and
connected
work,
really
helped
us
prove
out
that
a
kubernetes
based
solution
could
be
both
cost-effective
and
reached
this
tremendous
scale
of
5
million
speakers
in
the
wild
always
connected
to
our
cloud,
so
Dave's
going
to
walk
you
through
what
it
actually
took
to
solve
this
sort
of
a
problem.
Thank.
C
You
darling
so
just
to
set
the
stage
where
I'm
coming
from
I'm,
just
a
dev
standing
in
front
of
a
room
of
cloud
engineers
asking
you
to
love
me.
So
when
you
work
as
an
app
dev
and
in
the
web
scaling
a
web
app
is
a
pretty
solved
problem
load.
Balancing
sessions,
you
see
it
every
day
with
massive
websites.
Messaging
is
another
thing
entirely,
so
this
is
just
about
just
as
much
about
the
discovery
of
what
does
and
doesn't
work
and
how
kubernetes
would
change
how
we
worked.
C
We
iterated
quickly,
there's
lots
we
could
have
done,
but
if
it
was
working
we
just
ran
with
it.
So
the
team,
four
people,
two
teams
makers
and
breakers-
the
makers
focused
on
building
the
system.
The
the
breakers
were
focused
on
building
the
test
rig.
We
could
move
people
around
teams
to
get
exposure
to
both
sides.
We
grew
together.
We
found
each
other's
weak
points.
It
was
a
really
good
dynamic,
so
we're
gonna
talk
about
the
stack,
then
the
testing
process
and
finally,
our
key
learnings.
C
C
This
is
a
kubernetes
rollout
on
AWS,
not
eks.
A
small
team
in
the
room
here
at
Bose
was
responsible
for
this
very
impressive
deployment.
It's
deployed
by
a
terraform
and
ansible
single
region,
multi
AZ
and
each
of
my
team
got
a
full
row
role
out
of
the
stack
mini
coop
is
great,
but
being
able
to
spin
up
the
full
stack
was
really
key.
If
you
are
going
to
build
something
at
scale,
you
really
have
to
test
it
at
scale.
The
reproduce
environment,
reproducible
environment
was
really
handy.
C
We
were
able
to
change
the
compute
instances
underneath
each
of
the
nodes
and
the
resources
very
easily
for
us.
So
our
solution
model,
as
mentioned
we
used
mqtt,
which
is
the
dominant
communications
protocol
in
IOT.
It's
lightweight
binary,
payload,
agnostic,
persistent
WebSocket
connections,
but
what's
in
the
cloud
have
you
ever
had
to
describe
this
to
your
parents
to
them?
It's
just
magic.
C
What
we
actually
used
was
for
an
MQ,
so
this
is
our
kind
of
end-to-end
stack.
So
on
kubernetes
we
had
h
a
proxy
as
our
ingress
and
load
balancer.
We
had
Vern
and
Q
as
our
message
broker.
We
built
our
own
custom
little
service
very
uncreated,
Lee
called
the
listening
service
that
listens
for
device.
Status
Messages,
which
is
what
we
are
simulating.
All
of
the
messages
coming
up
from
device
are
then
publishing
through
to
Cassandra,
which
is
going
to
store
a
shadow
version
of
the
device
status.
C
C
C
It
used
a
ridiculously
low
amount
of
resources
for
a
ridiculously
high
number
of
connections.
So
TCP
load
balancing
for
WebSockets
quite
a
bit
less
confusing
than
the
inter
ingress
examples
that
I
could
find
message
broker.
We
chose
Vern
MQ,
it's
a
high-performance,
open
source
distributed
MQTT
broker
built
on
Erlang
OTP
Erlang
OTP
is
used
very
heavily
in
telephony,
originally
really
solid
base
for
clustering
and
concurrency
it
scales
vertically
and
horizontally
and
commodity
hardware.
Vernon
T
really
stood
out
for
us.
We
wanted
an
open
source
solution.
Hive
MQ,
who
is
proprietary
mosquito,
has
no
clustering.
C
Emq
TTD
didn't
have
a
good
model
for
recovering
after
a
network
partition,
so
vern
mq
offered
us
clustering,
bridging,
which
is
a
way
to
move
messages
between
multiple
clusters
of
message,
brokers,
which
is
really
key
in
our
future.
Consideration
mqtt
five
shared
subscriptions,
so
this
is
a
way
you
can
load
balance
the
flow
of
messages,
since
the
listening
service
actually
has
to
process
every
single
message
that
comes
through
the
broker.
It
would
then
round-robin
through
all
the
instances
of
the
listening
service.
C
It
give
us
fault,
tolerance
and
extremely
well-defined
netsplit
behavior.
You
can
tell
it
if
you
can
still
register
new
clients
and
how
to
behave
during
a
net
split
and
time
order,
integrity,
which
is
really
important
when
you're
talking
with
a
speaker
in
your
phone.
If
you
are
telling
it
to
raise
the
volume
play
pause,
you
want
those
to
come
in
order
very
important.
C
So
the
glue
this
is
our
listening
service.
We
wrote
this
little
component
in
golang.
It's
only
about
a
hundred
lines.
It
subscribes
to
the
verne
mq,
with
a
shared
subscription
to
process
all
of
the
Status
Messages
that
we
were
simulating
and
it
writes
directly
to
Cassandra.
So
this
little
component
is
super
performant
and
so
easy
to
build.
It
was
incredible.
C
Finally,
we
have
our
datastore
Cassandra,
so
Cassandra
is
a
wide
column,
store
and
non-relational
DB.
It's
able
to
scale
and
maintain
high
availability
without
compromising
performance.
It
stores
the
most
recent
state
of
a
device
to
mirror
other
IOT
offerings
a
digital
duplicate
of
the
device.
In
the
cloud
we
were
focused
in
this
POC
on
write,
not
read,
but
what
we
needed
to
know
is:
can
it
handle
the
volume
it
did
very
easily?
C
So
how
did
we
put
this
all
on
kubernetes,
all
of
our
images,
we've
custom-built
for
everything
they
were
all
built
on
Alpine,
to
give
us
a
nice
small
image
for
Vern,
MQ
and
Cassandra,
because
they
are
clustering
pieces
of
software.
We
used
a
stateful
set,
so
it
would
bring
up
each
node
in
order
give
them
a
name
and
DNS.
That
is
consistent.
C
C
So
that's
all
of
the
build.
Let's
take
a
look
at
the
test
rank,
so
we
chose
Lucas.
This
is
what
the
other
half
of
the
team
was
building.
We
needed
to
customize
a
rig
to
simulate
different
client
behaviors,
one
to
one
publisher
to
subscriber
one-to-many
fan-out
gate
crashing.
So
we
looked
at
things
like
MZ
bench
and
jmeter.
In
fact,
run
MQ
has
its
own
tests
pre-built
in
MZ
bench,
but
they're,
not
very
flexible,
they're,
really
kind
of
built
for
request
response,
not
pub/sub,
you
can
do
it,
but
it's
not
as
flexible.
C
We
decided
not
to
go
with
it.
We
went
with
Lucas,
so
locus
is
a
Python
based
system,
it
has
a
master
node
and
it
can
instruct
workers
to
do
all
kinds
of
different
behaviors
based
on
what
you
want.
So
this
way
we
could
simulate
all
the
behaviors
we
needed.
Now.
We
deployed
this
to
bear
ec2
instances.
We
didn't
want
our
kubernetes
cluster
competing
for
resources,
so
our
high
level
architecture,
when
you
get
to
the
end
of
it,
you
had
locus
on
the
outside
and
the
deployment
of
all
the
workers.
C
They
go
into
H
a
proxy
which
was
pointed
at
Arverne
MQ
cluster.
On
the
other
side
of
that,
the
listening
service
has
a
shared
subscription
to
all
of
the
messages
coming
through
vern
mq,
and
it
then
in
turns,
writes
that
to
cassandra
so
on
to
testing
will
go
through
a
number
of
the
issues.
This
is
by
no
means
an
exhaustive
list.
C
It's
always
something:
it's
configuration.
Networking
resource
limits,
cluster
sizes.
Most
of
these
will
be
on
the
build
side,
but
there
were
certainly
challenges
on
the
test
side,
first
hurdle
in
particular.
But
this
way
we
can
validate
the
entire
throughput,
so
we're
gonna
spin
it
up
we're
gonna
make
one
locus
worker
to
see
how
many
connections
we
can
get.
So
we
can
level
set
what
kind
of
thing
we
can
do
with
it.
So
a
reminder
we
are
going
for
five
million
persistent
concurrent
connections
all
at
the
same
time.
C
C
C
Let's
take
a
look
at
the
first
result,
damn
it
that
was
unexpected
a
little
underwhelming.
So
underneath
the
hood
we're
using
Python
and
we
ran
into
a
problem
with
Python
file.
Descriptor
limits,
the
paho,
mqtt,
client
and
python
is
what
we
were
using
for
our
clients
so
underneath
that
it's
using
Python
and
the
Select
system
call.
So
this
is
what
everything
is
a
file
descriptor.
So
when
you
open
a
socket,
that's
a
file
descriptor
each
connection
requires
one
for
the
MQTT
broker
and
two
sockets
connected
to
each
other.
C
So
why
did
it
block
at
that?
So
the
workaround
we
replace
the
Select
call.
We
tried
using
a
library
called
a
sync
I/o.
This
was
an
utter
failure.
This
did
not
play
well
with
the
locus
concurrency
model.
We
had
two
different
concurrency
models
fighting
with
one
another,
so
we
rebuilt
Python,
there's
a
nice
little
constant,
called
FD
set
size,
that's
hard-coded
into
Python.
You
can
have
1024
file
descriptors
and
that's.
It
did
not
expect
to
begin
this
by
recompiling,
the
Python,
but
there
you
go
alright.
So
let's
get
down
to
the
next
test.
C
700,000
connections.
Well,
that's
a
bit
better!
So
what's
the
problem?
Configuration
defaults
and
network
address
translation,
so
we
ran
into
an
H
a
proxy
port
exhaustion
problem.
We
ran
into
just
a
basic
configuration
limit
that
was
the
default
in
verne
mq
and
the
service
abstraction.
We
had
Vern
and
Q
behind
a
service
object
in
kubernetes
and
that's
just
another
place
where
you
have
an
ad
issue.
C
So
if
you're
not
familiar
with
not
issues
for
for
anyone,
that's
new
to
this
sort
of
stuff,
you
basically
have
a
tuple
of
source,
IP
and
port
destination,
IP
and
port
and
your
protocol.
You
only
have
so
many
combinations,
that
of
those
we
did
not
properly
configure
everything,
h8
proxy
I
think
by
default,
starts
at
port
30,000.
So
we
had
to
fix
that.
So
our
workaround
reconfigure,
everything
so
fix
the
max
connection,
setting
in
vern,
mq
and
add
three
more
listeners,
it's
trivial
to
add
more
listeners
for
Vern
and
Q.
C
So
we
had
four
listeners
on
different
ports.
We
bypassed
the
kubernetes
service
instead
of
pointing
a
chief
proxy
of
the
service
we
had
it
pointed
at
each
individual
for
an
mq
instance,
and
we
would
round-robin
through
them
each
a
proxy.
So
we
had
it
Vern
round-robin.
We
had
to
increase
the
number
of
source
ports.
That
was
my
fault.
C
We
had
to
vertically
scale
the
ingress
nodes
we
ran
into
a
problem.
Aws
doesn't
necessarily
tell
you
exactly
how
much
bandwidth
and
how
many
I
ops
you
get,
but
we
ran
into
an
issue
and
as
soon
as
we
kicked
up
the
instant
size,
it
was
working
great,
so
I
even
created
a
little
Ruby
Sinatra
app.
That
would
query
the
kubernetes
api
and
it
would
return
templated,
config,
4h,
a
proxy
and
prometheus
and
Griffin
as
a
little
interlude.
Considering
all
these
problems,
discoverability
configuration
I
really
should
have
thought
of
something
like
a
service
mesh.
C
C
Our
subscriptions
fell
off
a
cliff,
so
whenever
you
create
a
connection
to
the
broker,
it
would
immediately
subscribe
to
a
unique
subscription
for
its
device.
You
know
something
like
device
slash
and
the
ID
of
the
device
burn.
Mq
nodes
were
being
terminated
by
kubernetes
kubernetes,
very
helpfully
brought
them
right
back
up
and
as
it
was,
transferring
all
the
session
information
and
blew
up
another
node,
and
so
we
had
this
beautiful
flap-flap-flap
of
nodes
going
up
and
down
this
one
was
harder
to
figure
out,
so
we
had
to
do
a
little
more
digging.
C
We
scaled
for
an
MQ,
incrementally
we
went
from
10
15,
20
40
80
nodes.
Now,
as
a
point
of
comparison,
some
of
the
biggest
Erlang
OTP
clusters
are
about
200
nodes,
so
we
were
already
at
about
40%
of
the
largest
nodes,
but
it
didn't
make
a
difference.
We
made
these
huge
clusters
and
it
did
that
made
absolutely
no
difference.
C
We
still
pitched
off
a
cliff
at
one
point:
1
million
connections
we're
running
into
some
kind
of
resize
reallocation
problem
under
the
hood
Vernon
hue
is
using
level
DB
for
all
sorts
of
metadata
and
subscriptions
and
something
was
obviously
hitting
its
limit
in
all
the
meta
data.
Replication
I
have
to
say,
do
not
wait
to
set
up
monitoring
graph
on
ax
and
prometheus
were
absolutely
key
in
figuring
this
out
using
stern
to
tail.
The
logs
of
all
of
our
running
pods
were
absolutely
key
in
finding
all
these
problems.
C
So
what
was
our
workaround
exponential
back-off?
We
modified
the
clients
to
add
a
custom
behavior,
so,
instead
of
connecting
and
then
subscribing,
we
would
delay
subscriptions,
so
they
would
happen
at
a
decaying
rate
to
make
sure
that
we
weren't
overwhelming
the
vern
mq
server.
Vern
mq
recovered.
Oh
yeah,
so
rate-limiting
may
be
a
really
key
consideration
in
this
sort
of
thing,
especially
if
you
suddenly
hit
one
of
these
allocation
problems
or
if
you
can
figure
out
how
to
pre,
allocate
all
of
these
resources.
C
Our
problem
was
the
Erlang
OTP
scheduler.
So
all
of
the
schedulers
went
to
100%
utilization
Erlang.
Does
these
lightweight
threads?
You
might
call
them
fibers,
but
Erlang
threads
are
a
thing.
It
puts
a
scheduler
on
each
CPU
that
it
has
available
to
it.
It
auto
automatically
looks
at
how
many
CPUs
it
has
available
and
it
puts
a
scheduler
on
it,
but
everything
chug
to
a
halt.
C
C
So
what
went
wrong
cgroups
so
in
the
containerized
world
c
groups
are
really
key.
This
is
the
way
docker
and
container
engines
report.
The
amount
of
resources
are
available
to
your
container
Erlang
OTP
is
not
C
group
aware
it
configured
itself
to
use
for
CPUs
it
had
12,
it
just
ignored
them,
so
we
had
to
directly
configure
the
number
of
V
CPUs
that
were
available
to
it.
Thankfully,
this
got
us
to
the
next
stage,
so
one
to
skip
a
few
99
1,004.50
m
8,
5
million
connections.
What
was
our
blocker.
C
Resources
again
starting
to
get
really
frustrated,
I
mean
we
got
to
four
point.
Eight
five
million
connections
I
mean
we
could
say
we
made
it
to
five
million,
but
you
know
it
feels
like
you're
lying
to
yourself.
A
little
I
know
what's
150,000
between
friends,
but
as
you
can
see
on
a
graph
on
a
date
dashboard
we
were
so
close,
but
not
enough
to
feel
validated
at
full
scale.
These
tests
took
five
or
six
hours
to
get
going.
We
had
to
ramp
up
all
of
the
connections.
We
had
to
run
it
for
a
while.
C
We
had
to
ramp
down
connections.
We
had
a
testing
budget,
we
blew
through
our
monthly
budget
in
a
day
sorry,
so
we
were
getting
ready
for
demos,
some
internal
demos,
and
we
were
getting
so
close.
So
we
started
this
running
again.
We
up
the
resources
we
get
ready
for
our
internal
demos.
We
walk
up
on
stage
at
our
company
in
front
of
everyone.
C
Thank
You
hot,
so
69
millisecond,
average
latency
from
publisher
to
subscriber
now.
This
is
important
because
250
milliseconds
is
a
magic
number
in
this
world
that
is
below
human
notice.
Latency.
If
you
hit
play
on
your
phone
and
it
starts
playing
on
your
speaker
in
less
than
a
quarter.
Second,
it's
a
relatively
imperceptible
note
difference.
You
don't
notice
it
if
you
wait,
half
a
second
that
half
a
second
is
enough
for
your
brain
to
go.
Why
is
that
working?
C
We
were
publishing
about
nine
thousand
seven
hundred
and
seventy
nine
messages
per.
Second,
we
had
other
tests
that
got
as
high
as
twenty
five
thousand.
It
depends
on
the
use
case.
This
was
just
one
of
twelve
scenarios
that
we
ran.
We
would
distribute
the
clients
across
different
availability
zones,
different
regions.
C
C
We
brought
each
service
up
in
order,
so
we
could
configure
each
a
proxy
and
Prometheus
later
I
know
you
can
mitigate
this
sort
of
thing
with
Hynek
containers
and
have
a
little
listener
on
a
particular
port,
but
it
was
a
little
confusing
when
you're
looking
at
a
clustered
resource.
When
do
you
say
it
is
available?
Do
you
wait
for
one
replica?
Do
you
wait
for
half
of
them?
Do
you
wait
for
all
of
them,
so
that
was
something
we
really
have
to
think
about
experiment
with
your
resource
limits.
C
It
is
hard
to
figure
out
what
your
limits
are.
You
have
to
try
different
workloads,
different
scenarios.
Until
you
push
the
pace,
you
cannot
estimate
do
component
level
load
tests,
you're
going
to
need
rate
limiting
and
there's
no
way
around
doing
it
at
scale
layers
really
troubled
complicate
troubleshooting.
We
had
layers
of
networking
resources,
ingress
routing
abstraction,
on
top
of
abstraction.
It's
Turtles
all
the
way
down
man
mo
layers
mo
problems.
So
with
all
these
layers
figuring
out
where
to
look
is
really
key.
Are
you
looking
right
down
at
your
ec2
level?
C
Are
you
looking
at
your
ingress,
your
H,
a
proxy
the
service
level?
Abstraction?
Is
it
the?
Is
it
throughput
through
the
Cassandra?
Is
it
the
broker
itself?
It's
really
hard.
This
is
why
setting
up
monitoring
early
matters
so
much
and
starting
at
scale,
is
different
than
organic
growth.
You
break
through
one
wall
and
you
hit
another
immediately,
but
you
got
to
get
back
up
and
keep
going.
C
This
doesn't
invalidate
what
you've
accomplished.
You
literally
break
through
one
of
these
walls.
Five
minutes
later
placed
on
the
floor.
Effortless
effortless
ly,
scaling
via
kubernetes
is
a
boring
feature,
but
it
is
a
killer
feature.
You
don't
realize
how
incredible
it
is
to
be
able
to
say
double
the
size
of
my
cluster,
just
like
that.
Take
pleasure
in
breaking
those
walls.
And
finally,
our
solution
was
a
lot
cheaper,
so
I
can't
share
numbers
that
nice
little
orange
line.
It
is
a
major
IOT
platform
provider.
C
A
little
blue
line
is
my
team
over
time
every
year,
you're
going
to
expect
to
add
more
devices,
so
cost
decreased
for
our
workload.
Initially
over
time,
we
were
more
expensive.
You
had
to
have
people
to
build
and
monitor
and
keep
track
of
all
the
resources
and
handle
it,
but
the
major
IOT
platform
they
build
based
on
metrics,
not
on
resources.
It
can
be
a
number
of
connections,
number
of
messages,
how
many
active
connections,
how
many
stored
messages?
C
How
big
is
your
shadow
store
further
in
the
middle
of
this
experiment,
the
cost
model
changed
on
the
one
we
were
comparing
against.
So
you
can't
predict
these
changes.
You
can't
have
cost
predictability,
so
you
even
see
it
the
two-year
mark.
There's
that
I'm
a
good
Canadian
lad.
That's
a
hockey
stick.
C
You
can't
predict
those
kind
of
changes
and
if
you
are
scaling
your
own
resources
are
what
determine
what
scales,
not
the
metrics
of
what
you're
doing
so
we
expected
cheaper.
We
did
not
expect
how
much
cheaper
so
in
conclusion,
building
a
messaging
service
for
millions
of
devices
is
a
unique
challenge.
The
blockers
evolved
as
this
solution
grows,
deploying
the
specific
components
in
pursuit
of
a
scaled
outcome
was
only
possible
through
the
flexibility
of
kubernetes.
So
thank
you
to
Cooper
Nettie's
and
thank
you
for
all
of
you.
Listening.
C
D
C
We
were
cheating
when
we
were
doing
this
all
the
connections.
All
the
clients
had
a
list
of
all
of
the
ingress
points
of
each
a
proxy
and
it
was
just
round-robin
there's.
You
know
this
is
not
an
apples
to
apples
comparison.
We
were
comparing
against
an
IOT
platform,
but
that
included
all
kinds
of
extra
things,
including
that
lovely
little
DNS
at
the
front.
That
will
distribute
the
load
through
for
you.
C
Yes,
sir,
how
many
active
connections
per
CPU
that
depends
we're
at
the
age,
a
proxy
level
of
trivial
I
think
we
probably
had
on
the
order
of
300
thousand
connections
going
through
a
single
Lecce
proxy
instance.
I
know
you
can
get
a
lot
more
than
that,
but
that
was
using
something
like
20
Meg's
of
RAM.
It
was
ridiculous
and
the
CPU
I
don't
think
it
ever
went
above
1%
at
the
runtime
queue
level.
I'd
have
to
go
back
and
check.
C
C
We
do
not
use
TLS
for
the
test,
the
idea
being
that
if
we
wanted
to
use
TLS,
we
would
terminate
terminate
it
at
the
eh
a
proxy
level.
We
were
just
trying
to
set
a
baseline
in
terms
of
number
I
mean
we
conducted
a
vast
number
of
tests,
with
varying
degrees
of
you
know,
different
numbers
of
HF
proxy
different
numbers
and
sizes
of
random
queue
clusters.
So
there
was
a
it's
not
easy
to
say
what
there
was.
C
C
C
C
That's
part
of
the
fun
we
got
to
bring
order
to
the
chaos
they
Bo's
give
us
a
very
simple
problem
that
has
a
very
complex
answer
and
they
trusted
us
to
just
cope
with
it.
We
kept
them
in
the
loop
the
whole
time
we
had
one-on-ones
with
them
every
week,
but
our
lovely
product
manager
Thomas
over
here
brought
order
to
the
chaos
for
us
and.
A
C
C
Yeah,
if
nodes
go
down,
you
don't
necessarily
lose
data.
There
are
sessions
that
are
on
each
individual,
vern
mq
node,
but
once
when
you
we
intercepted,
you
know
the
the
sick
term
and
we
it
would
vern
mq,
would
then
transfer
all
of
its
sessions
to
another
node
and
then
another
node
would
come
up
and
sessions
would
redistribute.
A
A
A
So
the
N
or
production
cluster
right
now
is
340
and
ec2
nodes,
and
we
were
tremendously
happy
with
the
with
the
uptime
the
stability
and
there's
some
people
who
do
some
incredible
work
here
in
the
room,
keeping
keeping
the
system
up
and
running.
But
it's
compared
honestly
to
a
managed
service
where
you
really
lose
control
over
your
uptime
and
your
performance.
And
it's
it's
been
a
really
really
good
choice
for
us.
I'm.
A
The
answer
is
as
a
firm
yes
and
we're
actively
in
the
process
of
trying
to
figure
that
out
it's
an
it's
yeah.
We
are
across
three
AZ's
right
now
in
one
region
and
going
multi-region
active,
active
kind
of
breaks,
things
up
and
down
the
stack.
So
this
is
it
there's
no
easy
answer,
unfortunately,
to
that
question,
but
we're
we're
committed
to
getting
there.
C
C
Locus
IO
is
an
open
platform
open
source.
We
just
deployed
that
the
client
we
wrote
ourselves
is
not
open
source,
but
it's
probably
not
all
that
complex.
We
really
all
it
needs
is
do
I
subscribe.
Do
I,
publish
how
often
so
it's
actually
really
easy
with
the
Pajo
mqtt
client
to
build
a
very
simple
client
to
use
with
locus.
C
So
the
question
was:
do
we
have
well
persistent
volumes
on
EBS?
Yes,
we
did
for
Cassandra,
we
use
persistent
volumes
as
well
as
Prometheus,
and
did
we
have
any
problems
with
Java,
interacting
with
the
C
groups?
Not
that
we
saw
Cassandra
honestly
was
one
of
the
easiest
parts
of
the
system.
It
was
never
our
blocker
when.
B
C
Conducted
a
test
that
was
a
little
more
ambitious
in
terms
of
the
distribution
of
the
cluster
it
fell
over,
but
that's
because
we
forgot
to
change
the
network
topology
settings
in
Cassandra
I'm,
not
sure.
Do
you
recall
what
version
of
Java
Scott
no
we'd
have
to
go?
Look
it
up.
Our
report
on
this
ended
up
being
about
118
pages.
So
there's
a
lot
of
stuff
to
know
about
it.
A
C
A
Oh,
the
the
costs
are
actually
per
device
per
year.
So
so,
basically
that's
the
beauty
of
cloud.
You
know
at
a
worse,
you
should
be
getting
linear
scale
and
in
fact
we
were
getting
sub
linear
scale,
because,
basically,
you
know
supporting
twice
as
many
clients
shouldn't
cost
twice
as
much
in
terms
of
infrastructure.
So
that's
why
the
cost
we're
dropping.
A
Yeah
absolutely
yeah
in
fact,
Thomas
at
the
back
of
the
room
created
a
tremendous
spreadsheet,
with
more
cells
than
I
care
to
remember
and
with
with
all
of
that
information.
So
that
was
a
big
piece
honestly.
From
my
perspective,
cost
was
one
of
the
biggest
pieces,
and
that
was
most
interesting
about
this
POC.
The
way
that
you
could
truly
get
two
sub
linear
cost
and
not
bankrupt
the
company.
By
having
you
know,
510
million
speakers
out
in
the
wild.