►
From YouTube: Is There a Place For Distributed Storage For AI/ML on Kubernetes? - Diane Feddema & Kyle Bader
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Is There a Place For Distributed Storage For AI/ML on Kubernetes? - Diane Feddema & Kyle Bader, Red Hat
Containerized machine learning workloads running on Kubernetes receive benefits such as portability, declarative configuration, less administrative toil, all with marginal performance impact. The best published results for performance sensitive machine learning workloads, e.g. MLPerf v0.6, were obtained by reading the datasets from local SSDs. While the MLPerf datasets fit comfortably on a single SSD, it’s a luxury not afforded to folks training models against petabyte scale datasets. We’ll share our experience running MLPerf training jobs in Kubernetes, against datasets stored by Kubernetes stateful storage services orchestrated by Rook. Highlights include the performance and scalability tradeoffs associated with local and open source distributed storage, and how machine learning formats like RecordIO and TFRecord provide performance utility and model validation flexibility.
https://sched.co/ZerS
B
I
would
say
yes
absolutely
and
why
why
why
were
we
interested
in
in
doing
this?
Well,
there
was,
you
know,
kind
of
kind
of
stemmed
from
a
number
of
conversations
that
that
we
had
internally,
as
we
were,
you
know,
thinking
about
distributed,
storage
and
the
implications
of
machine
learning
and
kind
of
boils
down
to
you
know
many
of
you
probably
have
an
environment.
B
You
know
this
this
kubernetes
cluster
that
we're
looking
at
here
represents
a
fairly
significant
capital
investment.
So
I
want
to
make
sure
that
that
that
it's
used
as
as
efficiently
as
possible
right.
We
want
to
maximize
our
utility
from
this
from
this
investment
so
kubernetes
to
the
rescue
right.
You
know
at
a
very
simple
level
right
kubernetes
is
a
scheduler.
B
Those
nodes
have
dimensions
like
amount
of
cpu
amount
of
memory.
How
much
storage
space
do
they
have
on
their
local
storage
devices?
How
much
I
o
can
those
storage
devices
handle
and
we
want
to?
We
want
to
effectively
fill
fill
these
bins
with
pause
and
we
want
to
fill
them
as
full
as
possible
without
them
kind
of
impacting
each
other.
B
In
order
to
to
you
know
maximize
the
utility
of
that
cluster,
so
we
schedule
a
pod
and
that
pod
is
going
to
consume
some
amount
of
cpus
some
memory.
You
know
it
will
generate
some
amount
of
storage.
I
o
and
consume
some
store,
some
amount
of
storage
space,
and
you
know
if
you,
if
you,
if
you
draw
this
up
on
onto
this
kind
of
you,
know
multi-dimensional
object
here,
you
get
a
shape,
and
so
each
pod
kind
of
has
its
own
shape,
and
you
know
with
all
these
different
shapes
of
pods.
B
B
Now,
some
of
you
might
have
noticed
a
problem
here
and,
and
I'll
kind
of
point
it
out
is
that
you
know
we
we've
kind
of
exhausted
the
amount
of
cpu
resources
here,
and
what
does
that
mean?
Why
is
that?
Why
is
that
bad
right?
We
want
to
maximize
usage
of
it,
and,
and
so
we
want
to
use
the
ball
to
cpu,
don't
we
well?
Yes,
we
do,
but
we
don't
want
to
trap.
We
don't
want
to
trap
the
the
the
other
resources.
B
Basically,
we
don't
have
it
want
to
have
this
like
stranded
capacity
in
terms
of
memory
and
unused
storage,
space
and
unused
storage
io.
We
want
to
use
those.
We
want
to
use
those
efficiently
as
possible
too.
So
one
of
the
problems
that
you
know
you
want
to
solve
with
the
scheduler
is
to
avoid
trapped
capacity.
B
One
way
you
can
help
avoid
trap
capacity
is
by
having
fewer
dimensions
that
are
constraints,
and
so
this
is
this
is
where
you
know,
distributed.
Storage
gets
kind
of
interesting
because
you
can
kind
of
abstract
away,
storage
and,
and
then
there
are
because
it's
being
remotely
accessed
the
the
the
pods
have
ability
to
be
scheduled
in
more
places
and
there's
less
constraints
on
scheduling
them
such
that
it's
easier
to
to
to
fill
your
bins
maximally
and
in
turn
maximize
the
utility
of
that
that
significant
capital
investment.
B
B
So
if
you,
if
you,
if
you
need
to
have-
or
if
you
have
data,
that's
larger
than
a
node,
then
then
you're
going
to
have
to
spread
that
data
over
multiple
nodes
and
guess
what
distributed
storage
helps
helps
you
do
that
right
so
by
kind
of
virtualizing
the
storage
kind
of
collecting
a
bunch
of
disks
together
and
then
using
various.
You
know,
software-defined
storage
methodologies.
B
The
other
problem
that
might
present
itself
is
is
okay,
you,
you
have
available
compute,
right
and,
and
you
want
to
want
to
use
it
up,
but
the
data
already
exists
on
another
pod
right,
because
you're
using
local
storage
somewhere
and
your
data
is
over
here.
So
now,
you're
kind
of
you
know
sure
you
could
have
something.
That's
going
to.
You
know
make
that
that
data
over
over
on
the
the
right
host
be
remotely
accessible
right.
B
This
context
that
we
we
started
to
ask
the
questions
about
distributed,
storage
and
machine
learning,
so
very
simplistic,
right,
kind
of
truncated
version
of
the
machine
learning
life
cycle
right
now,
of
course,
there's
gonna
be
something
on
the
left.
That's
that's.
You
know
the
applications
or
sensors
or
whatever
that's
that's,
generating
the
data
and
then
there's
there's
more
on
the
right.
Like
is
you
know,
training
is
not
the
end,
all
be
all
right.
B
You're
gonna
have
to
take
take
that
model,
and
you
know
wire
it
into
an
intelligent
application
and
do
something
useful
with
it.
You
know,
shopping
cart,
suggested
products
or
you
know,
detecting
fraud
or
so
on
and
so
forth.
But
for
the
purposes
of
of
this
discussion
here,
we've
kind
of
simplified
it
to
okay.
So
you
know,
there's
there's
to
the
the
time
to
value
is,
is
something
that
we
want
to
shrink
and
some
of
the
the
things
that
need
to
be
performed
before
we
can
realize
value
from
from
that
data.
B
Is
you
know
we're
probably
gonna
have
to
prepare
it
wrangle
it
together
in
some
sort
of
way
and
then
without
distributed
storage,
we're
gonna
have
to
move
the
data.
So
if
we,
you
know,
if
it's
going
to
take
a
really
long
time
to
run
it
on
the
general
purpose,
compute
and
it
was
processed
on
the
general
purpose.
B
Compute
then
we're
gonna
have
to
move
it
to
the
special
purpose
compute
in
order
to
you
know
finish
it
in
any
sort
of
reasonable
time
frame
and
then,
finally,
once
the
data
is
there,
then
we
can
actually
start
training
the
model
against
that
data
and
then,
once
it's
once
it's
completed
or
we're
kind
of
on
to
the
next
next
phases
of
of
the
life
cycle,
so
with
distributed
storage,
can
we
can
we
compress
this?
Can
we
can
we
compress
and
have
get
to
that
value
faster?
B
And
if
you
start,
if
you
implement
kind
of
some
sort
of
distributed
storage,
you
could
potentially
kind
of
compress
the
the
data
moving
and
model
training
into
into
the
same
kind
of
into
the
same
space
and
and
and
and
perhaps
perhaps
that
does
mean
that
it
that
it
inflates
the
time
that
the
actual
model
training
takes.
But
you
you
know
you,
you
potentially
save
yourself
that
data
moving
st
stage
now,
of
course,
an
astute
reader
might
go.
Oh
okay!
B
Well,
if
the
data
is
not
remote
and
the
gpu
node
is
accessing
over
to
the
network,
isn't
it
kind
of
like
the
data
is
moving?
And
the
answer
is
yes
right.
So
it's
going
to
be
kind
of
over
the
course
of
the
model
training
it's
kind
of
incrementally
or
either
either
incrementally
over
the
course
of
the
whole
training.
B
It's
grabbing
some
of
the
data
it
needs
to
keep
the
keep
the
gpus
busy
or
it
kind
of
bulk
loads
it
in
the
beginning
and
then
and
then
and
then
processes
it
from
there
right
so
so
yeah
this
is
this
is
this
is
what
we
want
right.
We
want
to.
We
want
to
see
if
we
can
compress
that
time
to
value
by
kind
of
collapsing,
these
two
stages
by
using
remote
storage
and
accessing
the
data
in
situ
right
in
some
sort
of
distributed
file
system-
or
you
know,
object,
store
or
something
like
that.
B
So
the
other
interesting
thing
that
that
comes
about
with
distributed
storage
is
it
helps
a
lot
with
with
with
parallelism
right
and
you
know,
just
kind
of
simple
simple.
You
know
explanation
of
parallel
parallelism
much.
Many
of
you
already
know
all
this,
but
you
know
if
I'm
digging
a
ditch
and
I'm
alone,
and
I
break
it
up
into
you,
know
six
work
units.
You
know
it's
gonna.
Take
me
longer
right.
B
If
I
want
to
dig
that
ditch
faster,
I
can
recruit
two
of
my
best
friends
that
are
that
are
equally
hard
workers
as
me,
and
we'll
each
compute
com
compute
complete
two
units
of
work
and
that
ditch
will
be
dug
faster
right
and
that's
great
and
the
reduction
in
time
that
it
takes
to
dig
that
ditch
is
speed
up
right
and
that's
the
it's
kind
of
the
computer
sciencey.
What
they
call
that
is
is
speed
up.
B
So
if
we
have
this,
you
know
model,
training
and
and
collapse
model
training
and
data
moving
step.
Can
we
can
we
parallelize?
Can
we
add
additional
gpu
hosts?
Of
course,
this
would
be
after
we've.
We've
stuffed
as
many
gpus,
as
is
nvidia,
has
engine
engineered
to
be
able
to
fit
into
a
single
system,
but
you
know
once
you
once
you've
exhausted
the
ability
to
scale
up.
You
want
to
need
you're
only
left
with
the
ability
to
scale
out.
B
B
A
Okay,
so,
as
kyle
said,
we're
going
to
run
an
experiment
and
we
need
models
to
train
for
this
experiment
and
we're
going
to
train
them
with
data,
that's
sitting
on
distributed
storage
and
then
compare
that
to
data
that's
sitting
on
local
storage
and
see
how
each
of
them
do,
and
so
we
decided
to
choose
the
ml
per
bench.
Training
benchmarks
to
run
and
ml
perth
is
like
an
emerging
industry
standard
for
a
standard
benchmark
that
is
designed
to
be
used
as
a
level
playing
field
for
comparing
machine
learning
infrastructure.
A
A
A
It
allows
researchers
and
people
are
doing
benchmarks
like
we
are
to
take
real-world
problems
of
finding
these
objects
in
these
scenes,
and
you
know
outlining
them
and
labeling
them
and
seeing
how
your
model
does
at
that
accuracy.
Accuracy
level
that
you
achieve
so
the
data
set
was
created
by
cornell
in
a
joint
project
with
microsoft
originally,
and
if
you'd
like
to
check
out
the
data
set
online
next
slide,
please
you
can
download
it,
and
you
can
also
peruse
around
the
data
set
and
search
for
things
like
in
this
case.
A
A
So
kyle
talked
a
bit
about
why
we
want
gpus.
We
want
to
get
that
speed
up.
That
powell
was
talking
about,
and
in
these
benchmarks
we
use
neural
networks
that
rely
heavily
on
linear
algebra.
Things
like
matrix
multiply,
dot
product
things
like
that
and
gpus
are
extremely
efficient
at
doing
that
matrix
math.
A
A
Are
we
getting
the
data
from
this
distributed
file
system
there
fast
enough
to
keep
the
gps
busy,
so
training
times
can
take
days
or
weeks,
and
if
you
made
a
slight
error
in
your
code
and
then
you
figure
that
out
a
week
later,
then
your
turnaround
time
the
work
time
for
the
data
scientists
or
the
researcher
is
not
reasonable.
So
we
want
to
shrink
down
that
time
to
train
so
that
you
can
iterate
more
make
more
changes,
improve
your
model
quicker
next
slide.
Please.
A
In
mlperf
we
will
be
using
python.
There
are
many
languages
that
you
can
use
lists
our
java.
You
could
choose
other
languages
to
write
a
neural
network
application,
but
the
benchmarks
that
we
are
going
to
be
running,
use
python
and
there's
a
good
reason
why
many
data
scientists
prefer
python
and
it's
because
you
have
access
to
libraries
and
I'm
just
listing
four
here
that
are
very
useful
when
writing
neural
networks,
so
there's
theano,
tensorflow,
keras
and
pytorch
and
today,
in
our
example,
we'll
be
using
pi
torch.
A
Another
really
nice
thing
about
using
python
is
that
in
your
python
code
you
not
only
have
access
in
our
case
to
the
pi
torch
library,
but
you
also
have
access
to
numpy
and
you
can
move
these
multi-dimensional
arrays
back
and
forth
between
between
numpy
and
pi
torch
very
easily.
The
api
is
very
easy
to
work
with,
and
I
think
that's
why
researchers
and
programmers
like
it
so
much
next
slide.
Please.
A
A
You
can
make
your
multi-dimensional
arrays
in
numpy
run
on
gpus
when
you're
using
itorch,
so
my
church
was
created
to
so
that
it
was
easier
to
build
neural
networks.
It
uses
something
called
tensors
which
are
really
just
multi-dimensional,
arrays
imperative,
multi-dimensional
arrays
that
run
on
gpus
and
by
imperative.
I
mean
that
when
you
execute,
as
you
go
down
through
your
code
and
you
execute
a
line
that
uses
a
tensor
bit,
the
computation
is
performed
immediately
when
you
hit
that
line
of
code-
and
not
all
implementations
are
like
that.
A
A
A
So
when
we
were
preparing
our
benchmarks
to
run
on
the
cluster
that
we
created
the
kubernetes
cluster
that
we
created
in
aws,
we
first
containerized
the
ml
per
benchmark
and
we
created
a
docker
file
for
it
and
in
our
docker
file
we
used
nvidia's
version
of
pi
torch.
We
built
it
from
source,
then
we
added
the
scripts
that
read
the
mlport
benchmark
and
we
created
that
container
image
pushed
it
to
quayio.
A
A
A
But
what
I'm
showing
here
is
that
you
have
some
yaml
that
kubernetes
interprets
and
schedules
you
onto
the
correct
worker
node,
based
on
the
resources
you're
asking
for
so
here.
I'm
asking
for
four
gpus,
which
is
what
we
had
in
our
cluster
in
aws
and
kubernetes,
will
and
openshift
will
schedule
that
pod
onto
a
nose
that
has
the
gp.
B
So
what
do
we
see?
What
happened
when,
when
the
rubber
hit
the
pavement?
How
did
this
turn
out?
So
as
a
as
a
before
before
we
up
before?
I
ever
talk
about,
you
know
what
we
see
we
like
to
give
some
just
splash
up
some
details
about
the
environment,
so
that
if
people
want
to
recreate
this
sort
of
work,
they
can
they
can
do
so.
We
have
kind
of
the
the
laundry
list
of
versioning
on
on
the
left.
B
They
each
have
16
gigs
of
memory
each
we
use
ec2
for
to
run
our
our
cluster
us
west
ii
region,
and
then
we
spread
spread
both
the
compute
and
the
the
masters
storage
and
the
worker
nodes
across
different
availability
zones
for
fault,
tolerance.
And
but
of
course,
because
we
only
had
one
gpu
worker.
It
was.
It
wasn't
a
single
availability
zone.
A
So
the
first
benchmark
that
we
ran
is
an
object,
detection
benchmark
called
ssd,
and
you
can
see
here
that
you
can
see
the
software
stack
that
we
ran.
We,
you
know
just
like
we
just
described
a
minute
ago
and
so
we're
using
python
and
pytorch
and
cuda,
and
on
top
of
that,
we're
running,
openshift
and
so
and
of
course,
red
hat
core
os
is
the
operating
system
that
we're
running,
and
you
can
see
that
we
have
a
training
time
where
the
data
was
sitting
on
local
ssd.
A
So
when
we
say
local
ssd,
we
mean
on
the
same
worker
node
where
the
p100s
p100
gpus
are,
and
so
we
had
a
time
of
45
minutes
92
seconds.
Well,
it's
0.92
minutes
actually,
and
then
we
had
training
time
using
ceph
of
45.78
minutes.
So
you
can
see,
there's
essentially
no
difference
here
that
the
difference
isn't
definitely
noise.
At
this
point
we
could
have
run
it
again
and
it
could
have
been
faster
with
stuff.
A
A
B
Yeah,
it's
interesting.
I
mean
they're
they're
they're,
so
close
right
when
you're
well,
you
said
four
percent
variance
or
something
it's
like.
If
we
ran
you
know,
if
we
ran
each
of
these,
you
know
20
times.
Would
it
just
be
like
in
the
error
bars
right
it
wouldn't
even
it's
effectively,
you
know
indifferent,
and
so
we
kind
of
go
okay
well,
this
is
this
was
interesting.
Why?
B
B
B
I
was
like
you
know.
Seth
is
bored
right
like
we're
not
doing
anything,
and-
and
I
think
you
know
the
the
I
o
that
we're
mapping
here
is-
is
the
the
the
data
operations
right,
so
there
might
be,
there
might
be
some
additional
like
like
file
system
like
you
know,
stats
and
listings
and
stuff
that
are
going
on
here,
they're
that
are
increasing
the
I
o
a
little
bit,
but
for
the
most
part
these
are.
B
These
are
relatively
you
know
it's
reading
reading
like
a
sentence
right
where
like
well,
those
those
io
sizes
was
really
small
like
if
you
divide
the
number
of
ios
by
the
amount
of
data
that's
coming
out
or
the
the
particular
periods
so
and
then
we're
like
well,
you
know
it's
probably
just
reading
like
a
sentence
and
then
it's
converting
the
sentence
and
then
you
know
doing
a
comparison
type
thing.
So
it's
it's
a
relatively
you
know,
at
least
for
this
workload.
It's
a
not
a
particularly
storage,
intensive
machine
learning,
workflow.
A
Right
and
it
it
trains
on
sentence
pairs.
So
those
like
the
english
sentence
and
the
german
sentence
are
sent
together
and
then
you
train
that's
how
the
model
is
trying
to
translate
from
one
to
the
other
and
so
yeah.
It's
not
particularly
io
intensive
and
the
I
o
is
human
language
sentence
pairs
together.