►
From YouTube: Building Armada – Running Batch Jobs at Massive Scale on Kubernetes - Jamie Poole, G-Research
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building Armada – Running Batch Jobs at Massive Scale on Kubernetes - Jamie Poole, G-Research
Thousands of GPUs. Hundreds of thousands of CPUs. Learn how (and why!) G-Research designed and built Armada - a system to enable massive throughput of batch jobs running on Kubernetes. In this session you’ll hear how we use large scale batch compute on Kubernetes to spot patterns in financial markets and predict the future. Armada enables us to schedule millions of batch jobs across many clusters and tens of thousands of nodes, getting optimum utilisation of our hardware to enable our researchers to run the latest machine-learning and advanced data science techniques across vast datasets. We’ll cover the architecture and approach of Armada, challenges and techniques for running Kubernetes at scale and some war stories and lessons learned along the way.
A
Okay,
yes,
so
thank
you
for
having
me
my
name
is
Jamie
pool
I'm
the
compute
platform
engineering
manager
at
G,
research
and
I'm
here
today
to
talk
to
you
about
Armada,
which
is
a
application
which
we've
created
at
gr,
to
enable
massive
scale
run
to
completion
batch
jobs
on
kubernetes.
This
is
something
which
we're
currently
using
production
running
millions
of
jobs
through
a
day
across
tens
of
thousands
of
nodes,
I'm
going
to
talk
a
little
bit
about
the
application
itself,
the
motivation
for
it,
how
we
use
it
its
architecture.
A
Some
lessons
learned
some
challenges.
We've
had
and
also
some
successes
along
the
way,
a
little
bit
about
the
road
map
and
then
explain
to
you
how
we
can
use
it,
how
you
can
use
it
first
I'm
going
to
quickly
cover
G
research
who
we
are
and
what
we
do
so
G
research
is
a
fintech
company
based
in
London
in
England.
A
It's
our
shiny
new
office
in
central
London,
which
has
just
opened.
We
employ
teams
of
quantitative
researchers
to
look
for
patterns
in
noisy
real
world
data,
sets
financial
data
ultimately
to
look
for
and
create
algorithms
that
can
be
deployed
as
trading
strategies
that
could
be
monetized
as
a
company
we've
existed
for
about
20
years
and
we've
we've
grown
quite
a
lot
in
that
time,
however,
onto
Armada,
so
what
is
Armada
Armada
is
a
multi-kubernetes
cluster
batch
scheduler
I
mentioned
that
we're
about
20
years
old
when
we
started
as
a
company.
A
Everything
back
then
was
very
windows
and
Net
Framework
based
anyone
working
in
fintech
would
have
experienced
that
over
the
last
five
to
six
years,
especially
with
migrated
heavily
towards
Linux,
because
that's
where
all
the
latest
ML
and
AI
action
is,
and
along
with
that,
we've
migrated
to
containerization
by
default
and
kubernetes
became
the
de
facto
container
orchestration
platform.
In
that
time,
we
had
a
lot
of
success,
running
our
stateless
services
and
other
applications
on
kubernetes.
A
So
we
thought
to
ourselves.
It
would
be
pretty
cool
if
we
could
also
run
all
of
our
batch
workloads
there.
So
in
terms
of
what
we
do,
the
vast
majority
of
our
compute
all
on-prem
is
actually
used
for
running
batch
jobs
because,
typically
our
researchers
want
to
run
some
experiments,
run
some
software
to
crunch
some
numbers
and
then
spit
out
an
answer.
So
historically
doing
all
this
on
Windows
we're
using
HD
Condor,
and
we
saw
the
pivot
to
Linux
and
containers
as
an
opportunity
to
work
out.
A
If
we
could
do
this
on
kubernetes,
because
we've
had
so
much
success
with
the
services
we
figured
if
we
could
have
a
common
substrate
for
all
of
our
compute
to
be
kubernetes.
That
will
be
really
advantageous
for
all
of
the
reasons
which
we've
already
heard
about
today.
Same
reason,
everyone's
sort
of
trying
to
do
this
I
suppose,
and
that's
really
where
our
model
was
born.
Armada
was
conceptually
an
application
where
we
thought.
A
We
started
having
this
conversation
back
in
2019
I
think
it
was
in
Barcelona
kubecon
and
there
was
other
people
who
were
interested,
so
we
figured
we'd
open
source,
the
project
from
the
start
and
recently,
in
fact
it's
been
accepted
into
the
cncf
sandbox.
So
it's
now
the
sandbox
application,
which
is
pretty
cool.
A
Fundamentally,
what
we
have
at
gr
is
a
large
number
of
users
and
applications
who
want
to
submit
jobs
and
get
some
answers.
The
Big
Box
in
the
middle
is
Armada
which
I'm
going
to
dive
into
and
that
contains
tens
of
thousands
of
nodes.
Many
tens
of
clusters
I've
got
some
stats
in
the
bottom
right.
You
can
read
there
and
typically
the
workflow
as
I,
described
as
a
user,
has
an
image
a
container
image
or
has
either
an
existing
one
or
just
created
a
new
one,
submit
some
jobs
to
Armada.
A
They
get
scheduled
images,
get
pulled,
containers
startup,
they
access
lots
and
lots
of
data
from
Storage
platforms
or
other
application.
Services
crunch
some
numbers
do
some
maths
and
then
write
out
a
result
somewhere.
Now.
This
picture
here
is
really
one
what
we
would
call
a
modern
environment,
so
many
environments
within
G
research,
let's
say
environment,
I,
guess
I
mean
something
like
development
staging
production
within
production.
We
have
multiple
environments
as
well
in
different
data
centers.
This
is
what
one
particular
data
center
might
look
like.
A
So
there's
some
Core
Concepts,
which
is
important
to
understand
about
the
system.
These
will
be
familiar
to
anyone.
Who's
experienced
an
HPC,
probably
all
of
you
guys,
but
I'll
go
through
them
anyway,
so
we
have
a
job
in
Armada
sense.
This
is
a
group
of
related
kubernetes
resources,
but
fundamentally
mostly
it's
a
pod
spec,
and
this
is
something
which
a
user
wants
to
be
created.
A
We
have
a
simple
grpc
API
that
users
use
to
talk
to
or
applications
and
then
uses
ultimately
subscribe
to.
Events
on
these
through
the
API
to
track
progress
of
their
jobs
or
job
sets.
So
they
will
see
that
a
job
is
go
from
queued
to
pending
to
running
and
then
ultimately,
hopefully,
succeed
or
sometimes
fail
for
whatever
reason.
A
A
Fundamentally,
we've
got
a
little
bit
of
an
odd
metadata
at
the
top
around
the
queue
the
job
set
name,
the
namespace
you
want
to
run
in
in
kubernetes
and
then
underneath
a
pod
spec
now
under
here.
This
is
just
a
raw
kubernetes
pod
spec.
You
can
put
anything
that
a
pod
supports
in
here
could
go
into
here,
which
means
that
anything
that
you
can
do
with
a
pod
in
kubernetes.
You
can
do
through
Armada.
A
A
So
this
is
meant
for
interactive
human
use
really,
and
this
allows
you
to
submit
jobs
or
sets
of
jobs
and
then
do
things
like
watch
their
progress,
and
you
get
a
simple
bit
of
output-
it's
probably
too
small
to
read,
but
it
will
it
will
move
as
you
press
after
you,
press
enter
and
you'll
be
able
to
see
transition,
state
changes
and
then
the
bottom
right
here
we've
got
a
screenshot
of
our
UI.
So
we've
built
a
UI
for
the
system
which
we
call
Lookout.
A
It's
just
a
simple
react.
Ui.
This
screenshot
is
actually
of
a
prototype
for
a
new
UI
which
we're
going
to
be
creating
in
the
coming
months.
The
one
we
have
at
the
moment
is
similar,
but
a
lot
more
basic
and
we've
want
to
put
a
lot
of
time
into
investing
in
this
UI,
making
it
much
easier
for
people
to
use
the
system
and
reason
about
what's
going
on.
But
this
allows
you
to
do
all
the
things
you'd
expect,
from
a
user
perspective
to
see
the
status
of
jobs.
A
A
So
this
is
what
we
will
call
the
server
side
of
armada
we
have
by
convention.
It
could
be
anything
but
we
we
always
put
things
in
a
Hamada
namespace,
and
here
we
have
a
couple
of
applications.
We
have
the
API
and
the
UI,
which
are
the
applications
which
we've
built
and
then
some
other
components
which
we've
chosen
to
use
to
for
the
backing
stores
for
our
system.
So
we
use
a
combination
of
Pulsar
on
redis
for
events
and
job
specific
jobs
stored
in
queues.
A
We
also
make
heavy
use
of
Prometheus
for
monitoring
as
you'd
expect
and
then
there's
the
slightly
random
elephant
on
the
outside
there
for
the
postgres
database,
which
we
use
for
the
system,
we're
actually
probably
at
a
point
of
like
Peak
complexity.
At
the
moment,
I'd
say
because
we've
been
on
a
bit
of
a
transition
through
the
architecture,
so
we're
probably
going
to
move
away
from
redis
and
just
use
postgres,
but
at
the
moment
we
have
a
few
of
these
components
mixed
around.
A
So
this
is
all
running
on
a
single
kubernetes
cluster.
In
this
this
case,
I've
put
a
note
there
as
well.
We
use
flat
car
as
our
operating
system,
but
that's
not
particularly
relevant,
but
if
we
just
have
the
cluster
on
the
left-hand
side,
this
wouldn't
really
do
anything.
This
just
presents
our
API
and
UI.
It
would
allow
users
to
submit
jobs
and
watch
stuff
that
nothing
would
actually
happen.
A
The
Clusters
on
the
right
hand,
side
where
the
actual
action
happens.
So
these
are
what
we
call
our
executive
clusters,
so
you'll
notice,
there's
multiple
and
I'll
come
to
that
in
a
second.
If
we
just
look
at
one
of
them,
what
we
basically
have
is
the
namespace
here
guermada
and
a
simple
component
deployed
into
it
called
the
executor.
A
This
component
is
the
thing
that's
responsible
for
sitting
there
looking
at
how
much
free
resource
there
is
in
the
cluster.
Talking
back
to
the
server
and
saying
hey,
I've
got
I've
got
this
much
compute,
give
me
some
jobs
and
there's
anything
queued.
It
will
lease
those
jobs
and
then
spawn
them
in
the
relevant
namespaces
I've,
actually
labeled
them
as
jobs.
Here,
however,
they
are
just
pure
kubernetes
pods,
so
it
ties
in
quite
neatly
with
all
the
other
stuff
we've
been
describing
this
morning
and
this
afternoon,
possibly
in
future
as
the
job
API
evolves.
A
A
Now
we
know
that
we
can
scale
at
given
kubernetes
cluster
to
many
thousands
of
nodes.
You
look
at
what
openai
have
done:
fantastic
work
there
to
scale
to
seven
and
a
half
thousand
nodes
and
Beyond,
but
that's
actually
quite
a
lot
of
work
to
do
that.
So
what
we
decided
to
do
was
devise
a
system
whereby
we
didn't
have
to
push
kubernetes
to
its
absolute
limits,
and
if
we
could
just
deploy
multiple
of
these
clusters,
we
could
just
scale
horizontally
that
way
effectively
indefinitely.
A
So
we
have
the
sort
of
standard
kubernetes
control
plane,
which
you
would
expect
the
head
nodes
running
the
regular
kubernetes
control,
plane,
components,
API
server,
controller
manager.
So
on
we
have
three
what
we
call
system
nodes.
These
are
to
run
cluster-wide
resources
for
things
we
decide
in
gr
that
we
want
to
use
such
as
cert
manager
dragonfly,
which
will
come
to
open
policy
agent
and
other
things,
and
then
we
just
have
n
of
what
we
call
batch
nodes
and
these
are
the
jobs.
These
are
the
nodes
that
we
actually
run
the
jobs
on.
A
We
want
to
minimize
the
amount
of
resources
that
our
cluster
acquires
are
on
these
nodes,
so
we
can
maximize
the
amount
available
for
jobs,
so
we
take
very
close
attention
to
the
demon
set.
So
we
run
on
these
things
and
the
resources
requested
and
so
forth.
On
these
nodes
we
run
the
absolute
bare
minimum
of
things
like
the
cniq
proxy
storage
integration,
pods
dragonflies
as
a
caching
layer
and
then
as
many
jobs
as
possible.
A
So
we
made
some
key
choices
along
the
way
for
scaling
kubernetes
itself
in
this
way.
So
we
have
what
we
consider
large
clusters
a
thousand
nodes,
but
I
know
it's
possible
to
get
bigger.
For
these
anyway,
we
decided
to
use
bare
metal
for
all
of
our
Estates,
including
the
master
nodes
and
system
nodes.
A
A
But
actually
one
thing
we
really
found,
which
I
think
is
remarkable
and
worth
noting
is
that
we
haven't
had
to
do
a
huge
amount
to
kubernetes
to
make
it
work
this
well.
Most
of
the
interesting
scaling
work
we
found
was
actually
in
the
dependencies.
So
silly
things
like
we
use
terraform
to
build
our
clusters
and
the
first
time
we
went
to
a
thousand
nodes.
We
actually
found
that
all
of
a
sudden,
maybe
unsurprisingly
our
planned
and
apply
times
went,
went
to
hell
really
terrible.
A
We
did
a
small
amount
of
refactoring
there
and
actually
found
we
could
10x
theme
performance
the
plan
and
applied
time
in
terraform.
Just
by
rejecting
the
way
we
represent
those
resources.
Dragonfly
has
been
massively
beneficial
for
us.
So
if
you
suddenly
scale
thousands
of
nodes,
you're
going
to
need
to
pull
lots
of
images,
that's
a
very
quick
way
to
destroy
your
container
registry,
whatever
it
is
or
get
locked
out
of
Docker
Hub
say
so.
A
Dragonfly
is
a
tool
which
you
can
deploy
an
open
source
tool
on
top
of
kubernetes,
which
is
like
a
caching
layer
for
Docker
images
with
its
own
peer-to-peer
Network
such
if
you
have
a
thousand
nodes
all
pulling
a
new
image.
It
goes
through
this
dragonfly
component
inside
the
cluster,
there's
a
single
pull
from
the
Upstream
registry,
and
then
it's
all
distributed
in
the
PSP
Network
across
the
nodes
and
then
a
lot
of
work
to
scale
off
storage
platforms.
A
On
the
security
front,
we're
a
very
security,
conscious
organization,
so
this
is
a
multi-tenanted
environment.
We
have
users
all
sharing
the
same
platform,
so
we
need
a
lot
of
focus
on
security.
We
don't
want
any
user
or
administrator
to
be
able
to
accidentally
access
anyone
else's
data
or
on
purpose.
So
we've
got
the
standard
sort
of
security
rules.
A
So
open
policy
agent
has
been
really
beneficial
for
us,
that's
a
great
way
of
just
as
ensuring
that
these
things
are
true
and
can't
be
can't
be
violated,
and
we
also
make
heavy
use
at
the
moment
of
pod
security
policy.
Although
we
know
that's
deprecated
and
we'll
go
away,
so
we're
going
to
replace
that
soon
enough.
A
So
now
I'm
going
to
talk
a
little
bit
about
some
challenges
that
we've
had
along
the
way.
There's
probably
like
four
categories
of
things
that
we've
found
really
difficult,
I
think
the
first
thing
scaling
to
this
size
just
running
kubernetes
at
this
size
is
very
difficult
operationally.
You
have
to
be
very
good
at
managing
kubernetes
and
rolling
out
changes
reliably.
A
The
biggest
thing
probably
is
performance
and
in
fact
not
of
the
system
itself,
as
I
said,
but
actually
what
you
realize
is
when
you
suddenly
are
running
back
shelves
at
the
scale
in
your
environment,
you've
effectively
created
a
giant
DDOS
machine
and
anything
you
point
at
you
can.
If
it
hasn't
been
scaled
properly,
you
can
reliably
destroy.
So
we
found
that
quite
a
lot
with
especially
our
storage
clusters
that
when
we
scale
to
this
size,
possibly
we
didn't
appropriately
scale
the
storage
and
we've
had
situations
where
users
have
suddenly
launched.
A
A
Then
the
next
two
really
are
integration
type
problems
where,
along
our
particular
Journey,
we've
been
doing
all
of
this.
At
the
same
time,
it's
moving
from
Windows
to
Linux
and
I
think
we've
all
definitely
underestimated
how
much
work
that
would
actually
be
for
everybody.
It's
not,
as
you
can
imagine,
just
a
case
of
doing
some
find
and
replace
for
backslashes
the
forward
slashes.
A
Successes
so
we're
one
of
these
organizations
that
are
always
striving
for
continuous
Improvement
and
I.
Think,
like
a
lot
of
us
do
this,
it
can
kind
of
make
us
focus
on
the
negatives
a
lot
of
the
time
and
we
need
to
I
think
take
the
time
to
stop
and
acknowledge
successes
and
celebrate
these
things.
So
for
me,
for
along
this
whole
project,
the
things
that
have
been
really
great
are
we've
proven
that
kubernetes
and
Armada
scale
really
well
and
don't
seem
to
be
at
any
kind
of
limit.
A
When
we
started
doing
this
I
think
people
were
queuing
up
at
my
desk
to
explain
why
kubernetes
wasn't
designed
to
run
batch
jobs
and
couldn't
ever
possibly
work
and
they're
right.
It
wasn't
designed
for
it,
but
we've
proven
that,
actually
with
not
a
huge
amount
of
effort,
it
can
be
made
to
do
these
things.
A
We've
had
really
good
quality
distribution.
Metrics.
That's
been
a
big
success
for
us,
so
I'd
definitely
recommend
doing
that,
because
you
need
to
be
able
to
reason
about
the
platform
and
see
what's
going
on
and
then,
after
that
point,
a
lot
of
the
kubernetes
wins
really
started
to
pay
off
so
because
we're
using
kubernetes
we
get
all
these
sort
of
ancillary
benefits
around
making
configuration
changes
really
easy.
We
could
just
if
we
need
to
suddenly
change
something
across
all
of
our
estate.
A
It's
a
couple
of
four
requests
and
running
our
automation,
pipelines
and
changes
just
go
out
which
is
fantastic.
Furthermore,
because
we're
on
kubernetes
we
get
all
the
integration
for
free
with
all
the
other
tools
that
we
might
want
to
use.
So
dragonfly
again
is
a
great
example
of
this.
We
realized
when
scaling
that
we
need
to
do
something
to
protect
our
Upstream
registry.
A
If
we
weren't
on
kubernetes
I,
don't
know
exactly
what
we
would
have
done.
We'd
probably
either
had
to
hope
something
already
existed,
that
did
it
for
our
platform
or
build
something
ourselves,
whereas
because
we're
running
on
kubernetes
we
just
Google
it
and
oh
there,
it
is
and
you
deploy
it
and
it
mostly
works
and
then
finally,
the
modular
cluster
design
of
armada
itself
has
been
a
massive
win
for
us.
So
it's
taken
the
stress
out
of
applying
configuration,
changes
things
like
kubernetes
upgrades.
We
can
stage
them.
A
Obviously
we
test
things
in
depth
and
staging,
but
even
when
he
gets
production,
as
we
all
know,
with
the
best
will
in
the
world,
that's
often
where
you
find
the
problems
for
the
first
time.
With
this
design
we
can
apply,
we
can
choose
certain
clusters
as
Canary
clusters
and
we
can
upgrade
these
first
sit
back
and
observe
and
go.
Oh
actually,
everything
is
okay
or
oh
crap.
A
So,
firstly,
it's
the
observability
piece,
so
we
want
to
put
a
lot
of
work
into
this
UI
so
that
people
can
better
understand.
What's
going
on
and
administrators
can
better
understand.
What's
going
on,
we
spend
a
lot
of
time,
answering
user
questions,
saying
hey,
why
isn't
my
job
running
and
we
can
work
it
all
out
through
grafana
and
other
things,
but
it's
much
better
if
they
can
just
have
a
UI.
That
explains
that.
A
A
You'll
probably
have
seen
from
the
design
that,
in
a
way,
we've
kind
of
been
a
bit
sort
of
keeping
kubernetes
at
arm's
length.
When
we
first
started
designing
the
system,
we
were
kind
of
pedging
our
bets
because
we
weren't
sure
if
we
actually
wanted
to
use
kubernetes
as
the
substrate
for
this
and
how
it
would
be
nice
if
we
could
be
a
bit
optional
about
this
and
maybe
use
Armada
as
a
system
on
top
of
another
platform.
A
It's
probably
the
case
to
be
honest.
Now
that
ship
has
sailed
in
kubernetes
as
well,
embedded
as
the
standard
platform
for
running
containers,
so
I
think
we'd
like
to
now
make
it
a
little
bit
more
accessible
for
people
through
kubernetes.
So
it's
just
easier
for
people
to
reason
about
so
things
such
as
having
either
using
a
kubernetes
API
directly
or
having
an
API
that
looks
exactly
like
it.
Maybe
a
couple
of
simple
crds
and
so
forth.
A
A
We
have
our,
obviously
our
GitHub
page,
because
it's
all
open
source.
The
link
is
at
the
end
of
this
presentation.
So
please
take
a
look
at
that
I
guess:
we've
got
the
of
course
Alex's
Group,
which
was
discussed
just
now
exactly
for
this
kind
of
product,
this,
amongst
other
things
and
I'm,
also
going
to
do
a
shout
out
for
the
cncf
research
User
Group,
which
me
and
Ricardo
run
every
every
other,
Wednesday
I.
Don't
remember
the
time
now
say
it's
8
A.M
as
well.
Pst!
Isn't
it
so
do
come
along
to
that?
C
A
We
want
to
be
able
to,
in
effect,
I
suppose,
Federate
jobs
to
multiple
clusters,
but
very
specifically
those
things
and
as
a
design.
Choice
we've
decided
to
keep
the
storage
of
the
state
outside
of
kubernetes
itself,
because
there
are
limits
to
how
much
you
can
put
in
xcd,
and
you
can
imagine
as
well
if
we
want,
if
you
want
to
destroy
the
system,
it's
preferable
to
overload
Armada
itself
and
they'll
have
its
own
limits,
but
in
such
a
way
that
you
don't
overwhelm
the
actual
cluster
that
you're
actually
running
on.
A
At
the
moment,
we
support
a
subset
of
things,
so
it's
pod,
specs
I,
think
we
actually
also
include
services
and
so
forth,
so
that
you
can
run
distributed
jobs
but
yeah
at
the
moment.
It's
just
really.
It's
quite
tightly
bound
to
the
resources
which
we've
found
we've
needed
within
our
within
our
company.
D
A
At
the
moment,
yes,
that's
one
of
the
main
reasons
actually
for
looking
at
moving
a
bit
closer
to
the
kubernetes
API,
because
then
it
would
make
integration
with
all
these
other
tools
much
easier
at
the
moment.
Anything
that
talks
to
a
model
needs
to
understand
this
API.
So
you
could
access
it
through
those
things,
but
you'd
need
to
write
some
kind
of
layer
to
do
that.
That
transformation.
E
A
So
what
we
do
in
the
back
end
is
we
tend
to
have
a
one-to-one
mapping
between
queues
and
namespaces.
So
we
have
our
own
automation,
which
we
use
to
apply
definitions
of
both
those
things
where
we
another
thing
which
I
think
we
should
open
source,
which
is
a
effectively
a
tool
which
says,
there's
a
bunch
of
definitions.
This
is
the
Q
Alice,
it's
accessible
by
these
users,
which
are
uses
that
kubernetes
understands
it.
Has
these
other?
You
know
priorities
and
other
other
things
and
that
gets
translated
into
namespaces
within
kubernetes.
A
F
I
can
ask
one
then,
because
we
discussed
a
lot
about
batch
and
other
topics,
but
here
you
mentioned
a
lot
about
multi-cluster
and
I.
Think
in
the
initial
definition
of
the
batch
working
group,
it
was
explicitly
stated
that
multi-cluster
wasn't
something
that
would
be
focused
or
or
scheduling
across
clusters.
F
No
okay,
because
yeah
we're
just
talking
about
like
instantiated
resources
across
clusters
accessing
the
jobs.
Do
you
access
through
to
your
mother
apis?
Or
can
you
actually
talk
directly
to
the
Clusters
things
like
this?
So
where
do
you
see
this?
This
could
fit
in
in
kubernetes
and
Cloud
native
this
discussion,
because
we
we
heard
about
the
Federation
projects
yeah.
So
how
can
we
push
this
forward?
Guys.
A
It's
it's
a
yeah,
so
it's
a
really
interesting
problem.
I
think
the
multi-cluster
thing
the
approach
we've
taken
is
probably
like
the
USP
for
our
application.
It's
the
I,
haven't
seen
anything
else
which
supports
a
multi-cluster
setup
as
well
as
this.
However,
it's
something
that
everyone
wants
to
do
and
that
all
of
these
discussions-
it's
really
interesting,
listen
to
everyone
have
their
own
sort
of
solving
this
in
their
own
way.
A
I
kind
of
feel
like
we're
in
this
Cambrian
explosion
of
people
trying
to
run
batch
on
kubernetes
are
all
going
to
develop
different
ways
of
doing
things
and
eventually
we're
going
to
have
to
see
some
things.
Some
things
win,
some
things
lose
and
I
guess
sort
of
converge
on
the
right
ways
of
doing
things.
I,
don't
know
what
the
answer
is
right
now
for
multi-cluster,
but
it
feels
like
something
which
should
be
a
bit
more
somehow
available
through
kubernetes.
A
You
know
not
not
in
the
sort
of
federation
kind
of
way,
but
ultimately
I
think
what
we'd
like
to
see
is
as
many
of
these
sorts
of
capabilities
that
we've
developed
here
being
pushed
into
the
platform
so
that
we
can
then
of
doing
ourselves
and
use
them,
and
maybe
eventually
this
all
falls
away.
I
just
I,
don't
know,
but
until
that
point
we
we
need
to
solve
our
problems.
So
this
is
what
we
do
so.
B
A
A
A
I
my
take
on
this
and
you'll
probably
talk
about
it
in
the
panel
discussion
later.
Is
it's
it's
okay
to
have
different
solutions
for
things
and
I
think
sometimes
we
try
too
hard
to
try
and
come
up
with
one
one
ring
to
rule
them
all
and
then
fail
because
it's
things
end
up
just
too
generic
and
you
suffer
from
this
generic
side
concept.
A
Ultimately,
we'll
probably
settle
on
I
think
a
smaller
number
of
patterns
for
running
software,
whether
it's
Services
batch
and
a
small-ish
number
of
other
things,
and
if
we
have
different
ways
of
solving
multi-cluster
for
that
smaller
set
of
things
that
at
least
is
common
within
those
sets.
Then
it's
probably
fine,
I
think
trying
to
be
too
Grand
about
it
and
have
Federation
for
all
or
something
might
just
be
a
bit
too
ambitious,
and
that's
why
you
just
spin
your
wheels
forever.
It's.
G
Thanks,
it
was
a
great
talk.
Can
you
talk
a
little
bit,
I
guess
about
the
general
question
of
how
you've
done
storage
with
this,
like
you've
got
users
running
all
their
ml
models?
How
are
they
getting
their
training
data
in
there?
How
are
you
making
it
so
that,
with
multi-tenancy
especially
seems
like
a
really
hard
problem,
yeah.
A
Sure
So
within
our
world,
we
tend
to
use
most
of
our
storage
on
shared
storage
platforms.
So
we
use
things
like
isilon
we're
actually
moving
quite
heavily
to
using
vast
as
a
storage
platform
I
sold
out
in
some
slides.
Previously
we
have
a
good
multi-tenancy
set
up
on
those
platforms
already
where
users
or
groups
of
users
already
have
areas
they
can
access
equivalent
or
think
of
it
like
a
bucket
I
suppose
in
S3,
which
they're
already
perm
to
use
through
sort
of
convention
I.
A
Suppose
we
allow
users
to
access
those
resources
through
Armada
through
kubernetes,
so
it's
easy
for
people
to
run
a
job
and
have
a
templated
way
of
accessing
their
personal
area
or
some
shared
area
on
on
icelon
or
vast
to
load.
Their
data
and
a
lot
of
the
performance
work
we've
been
going
and
looking
at
is
how
we
actually
make
that
interaction
as
good
as
possible,
because
you
can
really
make
people's
lives
a
lot
easier,
a
lot
better
or
worse,
by
making
that
integration
work
better
or
worse,
cool.
B
A
Do
we
need
external
storage,
yeah
sure,
I?
Guess
that's
one
of
the
motivated
for
the
architecture
we
back
of
a
napkin
calculations
when
we
first
started
doing
this
told
us
that
the
amount
of
data
we
need
to
store
because
we
have
requirements
to
store
millions
of
jobs
and
have
a
huge
amount
of
throughput
and
churn
through
the
system
is
that
putting
all
that
in
FCD
would
definitely
break
it.
A
If
we
didn't
do
anything
and
we'd
have
to
tune
it
pretty
hard
to
even
make
it
possible,
give
an
awful
lot
of
storage,
and
then
we've
got
the
other
thing.
I've
mentioned
previously
around
failure,
domains,
I,
guess
where,
if
you
then
break
xcd,
you've
actually
completely
broken
the
cluster
that
you're
even
running
the
platform
on,
and
you
then
can't
access
it.
You
know
cut
off
the
branch
that
you're
sitting
on
kind
of
thing,
so
that
was
the
motivation
for
that.
A
However,
if
that
could
be
solved
and
again
that
was
a
further
simplification,
we
can
make
I
think
a
lot
of
these
parts
of
our
design
have
been
optimizations.
We've
made
along
the
way,
but
I'm,
never
too
we're
not
too
precious
to
say
we
won't
ever
change
something
and
if
an
optimization
is
no
longer
required,
then
we
should
reevaluate
that
and
that's
what
we'll
try
and
do.
H
A
A
B
A
Yes,
what
he
uses
most
comfortable
with
I
think
frustratingly
for
us.
They
always
want
a
UI
I
find
and
that's
because
our
users
in
particular,
are
not
experts
in
kubernetes
containers,
even
Linux,
some
of
them.
These
are
their
day.
Job
is
trying
to
do
a
metal
or
you
know,
they're
basic
mathematicians.
A
So
typically,
people
want
tools
to
use
that
are
easy
to
understand
and
it's
either
uis
that
we
build
or
meta
applications
on
top
of
a
model
which
other
bits
of
our
engineering
organization
have
produced.
But
we
have
a
massive
range.
So
we
have
some
users
who
are
effectively
power,
users
who
are
perfectly
happy
running,
Cube,
CTL
or
Armada,
CTO
or
whatever,
and
then
other
people
who
just
don't
want
to
know
and
just
want
to
use
the
UI.
So
we
kind
of
have
to
support
everything
which
is
one
of
the
challenges.
B
A
I
should
have
done
a
selfie
I
would
do
that
I'm
being
told
to
stop
it's
like
if
I
can
have
a
selfie
with
things
told
to
stop
in
the
background
as
well.
I
think
that
would
be
best.
Wouldn't
it
is
it
gonna
work
right.
You
can
all
tell
me
to
go
away
Bluetooth
on
yeah.
Thank
you
all
right.
Thank
you.
Everyone.