►
From YouTube: Taming Data/State Challenges for ML Applications and Kubeflow - Skyler Thomas, Hewlett Packard
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Taming Data/State Challenges for ML Applications and Kubeflow - Skyler Thomas, Hewlett Packard Enterprise
The Kubeflow project brings incredibly powerful Machine Learning frameworks like TensorFlow and PyTorch to Kubernetes. The ability to parallelize training and the ability to scale workflows up and down is revolutionary. However, state and persistent storage are a much bigger challenge for machine learning workloads because of their training data, library files, and models. We will discuss what it took to create AI/ML environments running thousands of pods and that request petabytes of training data. We will explore the various state and storage challenges that crop up when you are building Kubeflow applications. We will discuss where distributed persistent storage solutions fit in the picture. We will address various storage api's including: POSIX/CSI solutions, NFS, S3, and HDFS fit into solutions. Data security and privacy issues will be discussed.
https://sched.co/Zeq3
A
A
A
A
Finally,
I'm
going
to
dive
a
bit
deeper
into
very
specific
challenges,
talk
about
different
tips
on
how
to
handle
them
when
you
run
into
them.
This
is
a
very
short
presentation,
so
we're
not
going
to
dive
super
deep,
there's
a
lot
to
discuss
in
this
area,
but
hopefully
we'll
have
time
to
answer
some
of
your
questions
after
we
present
and
at
the
end
of
the
session.
A
I
really
hope
you
leave
it
more
prepared
to
face
the
challenges
that
you're
going
to
see
with
state
and
data
in
kubeflow
and
have
some
tips
in
your
back
pocket
to
solve
them.
I'm
sorry
we're
not
together
in
person
today
that
would
have
been
amazing
to
sit
down
and
bounce
things
off
of
each
other
at
a
conference,
but
hopefully
we
we
accomplish
a
lot
and
maybe
we'll
see
each
other
at
the
next
coupe
con.
A
Let's
talk
a
little
bit
about
data
scientists
versus
it.
If
you
think
about
both
groups,
they
have
kind
of
antithetical
goals.
What
each
group
wants
is
almost
the
polar
opposite
of
what
the
other
group
wants.
Data
scientists
want
really
custom
hardware,
high
memory,
gpus
tpus
itu,
really
wants
cookie
cutter
commodity
machines
wherever
possible
data.
Scientists
want
all
the
hardware
they
can
get
and
it
really
wants
to
encourage
efficiency
and
sharing.
A
It
wants
to
standardize
applications
and
reduce
license
costs.
Data
science
really
wants
to
use
a
variety
of
applications
and
frameworks.
It
wants
software.
That's
safe
and
supported
data.
Science
wants
cutting-edge
tools
from
academia.
Finally,
it
wants
to
carefully
categorize
and
restrict
data
for
security
and
regulatory
reasons.
Data
science
really
wants
to
use
all
it
can
to
train
their
models.
There's
really
been
no
good
way
to
solve
these
problems
except
to
isolate
data
science's
environment.
A
Basically,
the
organization
backs
a
truck
up
with
a
lot
of
money
and
other
resources
and
creates
a
an
entirely
separate
environment
with
all
the
bells
and
whistles
that
data
science
wants
and
in
certain
use
cases
this
is
efficient,
other
use
cases,
it's
simply
not
sustainable.
For
data
scientists,
as
we
have
more
and
more
data
science
going
on
in
organizations,
it
becomes
less
sustainable.
A
A
Kubernetes
can
help
us
a
lot
but
before
we
think
about
kubernetes,
let's
take
a
step
back
and
look
at
how
we
were
doing
some
of
these
data
science
and
big
data
tasks
before
kubernetes
we're
using
things
like
spark,
tensorflow
and
hadoop,
and
we
we
learned
a
lot
of
things
from
those
environments
and
the
biggest
thing
we
learned
is
hand.
Rolled
environments
are
incredibly
expensive.
A
A
A
The
key
to
all
of
this
is
the
name
data
science.
The
data
in
that
name
is
the
clue.
Machine
learning
requires
a
ton
of
data
and
or
state
to
work
and
kubernetes
was
really
initially
designed
for
stateless
apps
over
time
handling,
state
and
applications
has
approved
in
kubernetes.
We
now
have
things
like
persistent
volumes
and
storage
classes,
csi
volume
based
scheduling
and
others,
but
there
are
still
major
challenges.
A
A
We
work
with
customers
doing
self-driving
cars
that
have
sensor
data.
That's
multiple
hundreds
of
petabytes.
This
presents
a
raft
of
challenges.
Some
of
the
questions
you
really
have
to
ask
about
training
data.
How
do
we
move
this
data
close
to
compute,
or
how
do
we
move
compute
rather
close
to
the
data,
because
it's
very
hard
to
move
the
data
around?
How
do
we
protect
privacy?
If
some
of
this
data
is
sensitive?
How
do
we
follow
legal
requirements?
A
Libraries
are
yet
another
area
if
you
have
a
container
and
it's
running
a
spark
job
or
tensorflow
job
they're
going
to
be
certain
compute
libraries
that
you
need.
Do
you
want
to
have
to
create
a
container
every
single
time?
You
need
a
new
library.
Do
your
data
scientists
have
to
be
experts
in
the
containers.
That's
a
challenge
models.
How
do
we
query
and
promote
model
files
into
production
logs?
All
of
these
applications
are
generating
massive
amounts
of
logs.
A
A
Okay,
so
we've
talked
a
little
bit
about
the
problem
space
and
the
challenges
we
face
and
the
types
of
state
that
we're
interested
in,
but
let's
make
sure
and
review
the
actual
data
science
workflow
that
most
people
go
through
if
you're
not
familiar
with
it.
Most
vendors
give
workflows
that
are
many
steps,
long,
I've
kind
of
simplified
things
here
into
four
steps.
A
Really
you
capture
some
data.
This
can
be
really
challenging.
I'm
not
going
to
address
it
very
much
in
today's
presentation,
but
you
need
to
deal
with
things
like
streaming.
You
need
to
do
data
cleansing.
There
are
all
kinds
of
challenges
with
capturing
the
data
and
making
sure
the
data
is
okay.
That
kind
of
bleeds
in
in
a
little
bit
of
a
grey
way
into
exploration.
A
This
is
the
work
that
a
data
scientist
does
within
a
notebook
experimenting.
Looking
at
data
trying
to
think
about
it,
try
different
regressions
on
the
data.
Try
very
simplified
toy
models
with
the
data.
Then
they
move
on
to
training
it
in
a
distributed
fashion
across
all
of
these
nodes
in
our
kubernetes
environment
and
finally,
they
serve
a
model
that
was
created
during
training.
A
A
lot
of
data
science
is
automating
these
steps,
so
you
can
do
it
over
and
over
and
over
again,
but
that's
really
kind
of
the
gist
of
things.
As
I
mentioned,
notebooks
are
really
the
heart
of
the
user.
Experience
that
a
data
scientist
has
in
the
kubernetes
environment
with
kubeflow
notebooks
are
really
what
what
these
data
scientists
work
with
they're
container
based
so
jupyter
notebooks
jupiter
is
a
technology
that
doesn't
have
to
be
container
based,
but
most
deployments
are
in
containers
and
most
notebooks
are
jupiter.
A
A
What
a
data
scientist
is
doing
here:
they're
examining
and
scrubbing
data
they're
experimenting
and
visualizing
data,
they're,
creating
toy
models
and
then
they're
launching
large
training,
jobs
and
automations
distributed.
Training
is
really
the
thing
that
gets.
Most
people
excited
about
running
these
machine
learning
jobs
within
kubernetes.
A
The
way
a
job
operator
works
is
that
a
machine
learning
job
custom
resource
of
some
type
is
submitted
via
cube
control
or
other
client
tool
to
the
kubernetes
api
server
in
kubernetes.
There's
a
machine
learning
job
controller
of
some
type
that
sees
that
custom
resource
that's
been
submitted
and
it
dynamically
creates
a
set
of
control
or
parameters,
server,
pods
and
a
set
of
worker
pods.
A
A
For
that
we
have
to
put
our
models
in
a
container
and
then
they're
going
to
be
serving
responses
to
rest
requests.
But
how
do
we
actually
put
those
model
files
in
the
container?
Do
we
generate
a
new
container
for
every
set
of
model
files,
or
do
we
actually
mount
a
universal
model
serving
container
to
a
particular
model?
A
Unlike
hadoop
machine
learning
tools
are
really
not
unified
around
a
single
data
access
api,
different
machine
learning
applications
require
different
storage
apis
and
that
creates
kind
of
a
challenge
for
us,
and
this
is
the
first
of
our
challenges
that
we'll
talk
about
and
that's
data
api
availability
since
we're
in
kubernetes.
Really,
the
traditional
way
to
think
about
talking
to
persistent
storage
is
via
posix.
A
A
It
fits
the
kubernetes
model
really
well,
but
there
can
be
issues
if
you
use
a
csi
driver
for
something:
that's
not
a
distributed
file
system,
so
csi
is
not
a
panacea.
All
the
time
when
you
want
posix,
you
really
have
to
have
an
underlying
csi
driver.
That's
that
works
well
object,
storage
or
s3
also
fits
in
really
well
with
machine
learning.
A
Kubernetes
hasn't
really
had
object,
storage
catch
up
with
posix
in
the
csi
world
yet,
but
that
that
is
something
that's
being
worked
on
with
the
storage
group,
but
from
a
machine
learning
perspective.
A
lot
of
the
tools
that
we'll
be
using
tend
to
work
with
object,
storage
and
object.
Storage
is
a
very
simple
way
to
do
things.
It
has
a
lot
of
performance
and
throughput
issues
so
pulling
your
trade
training
data
in
through
object,
storage,
probably
isn't
the
best
way
to
do
things.
A
You
could
embed
hdfs
drivers
in
your
pods.
The
hdfs
is
very
fast
performant
way
to
go
after
storage,
but
it
it
there's
nothing
built
into
the
kubernetes
infrastructure
to
deal
with
it
to
the
final
interfaces
nfs
or
database
protocols.
There
could
be
many
others
nfs.
We
we
do
see
several
machine
learning
applications
out
there
that
do
require
specifically
nfs.
I
wouldn't
recommend
it
if
you're
designing
things
from
scratch,
it
has
some
speed
issues
and
some
issues
with
stability
and
security
database
protocols.
A
The
the
key
to
really
this
part
of
the
presentation
is:
is
you
need
your
data
infrastructure
to
be
able
to
support
multiple
protocols?
Our
second
challenge
is
really
how
to
deal
with
state
that
is
used
as
libraries
or
various
resources
for
the
container.
There
are
really
two
major
schools
of
thought.
A
A
A
The
container
route
does
have
a
number
of
downsides.
The
number
of
containers
that
you
build
your
data
scientists
must
be
more
skilled
to
some
degree
to
create
the
containers
they
have
security
issues
that
you
really
need
to
think
about,
and
container
management
and
storage
volume
mounting
takes
more
time
in
planning.
You
have
to
create
smarter
containers.
A
It
probably
requires
a
csi
provider
with
a
distributed
file
system
to
do
it.
If
you
are
going
to
go
the
route
of
creating
a
container
for
for
each
job
or
notebook,
I
would
recommend
that
you
create
pipelines
or
use
a
workflow
engine
like
argo,
to
create
those
containers
rather
than
creating
by
hand
or
via
a
recipe
scripting.
It
really
breaks
down
fairly
quickly.
This
challenge
is
the
challenge
of
dealing
with
the
gravity
of
training.
Data
data
gravity
is
the
idea
that
it's
hard
to
move
data
around
and
that
data
naturally
attracts
compute
closer
to
itself.
A
There
are
a
number
of
reasons
why
data
exact
gravity
exists.
The
first
is
that
it
costs
a
lot
of
time
and
money
to
move
some
types
of
data.
When
you
have
hundreds
of
petabytes
of
sensor
data,
it's
very
costly
to
copy
that
information,
you
may
have
government
regulations.
I
can't
move
data
from
here
to
there.
It
must
reside
within
my
borders.
A
A
First,
when
data
must
live
outside
the
cloud
and
within
a
corporate
firewall,
you
need
to
choose
between
allowing
external
compute
in
the
cloud
to
ingress
into
your
data
center
or
to
create
an
on-prem
kubernetes
environment.
In
the
vast
majority
of
cases
we
see
the
on-prem
solution
tends
to
be
the
solution
that
works
the
best.
There
are
specific
instances
where
running
compute.
In
the
cloud
and
going
to
on-prem
works
a
little
bit
better,
but
performance
cost
security,
legal
reasons.
Usually
you
want
to
have
an
on-prem
environment
to
deal
with
that
data.
A
Many
cases
spinning
up
a
quick
kubernetes
solution
in
the
cloud
might
be
useful
to
handle
the
worries
that
it
has
for
data
scientists
rolling
their
own
containers
in
those
cases.
Consider
a
shared
cloud-based
data
lake
for
ephemeral
data
scientists,
kubernetes
environments,
to
all
share
addition
to
the
notion
of
data
gravity.
There's
a
closely
related
issue
of
data
locality.
A
The
hadoop
world
spent
an
extreme
amount
of
time
trying
to
ensure
data
locality
and
optimizing
hdfs
for
this
type
of
access,
as
network
speeds
have
improved.
What
we
found
at
map
r
was
that
true
data
locality
became
less
important
with
faster
networks.
In
many
cases,
fast
posix
access
to
a
quality
distributed
file
system
is
really
enough.
A
Block
storage,
scheduler
improvements
and
custom
schedulers
may
all
make
true
data
locality
in
kubernetes
and
kubeflow.
Something
that's
reasonable
for
these
machine
learning
frameworks
in
the
future,
but-
and
the
spark
project
has
worked
a
lot
on
doing
data
locality
in
kubernetes,
but
as
it's
not
something
that
I
would
suggest
you
spend
a
lot
of
your
time
and
energy
on
early
on.
A
A
I
use
the
term
data
security
broadly
and
there
are
a
lot
of
nuances.
We
can't
cover
in
this
presentation,
but
I
want
to
make
sure
you
have
just
a
few
of
the
tips
and
just
a
way
to
think
about
data
security
and
some
of
the
reasons,
there's
complexity.
There
are
two
main
causes
for
these
difficulties.
A
One
are
the
way
namespaces
work
in
kubernetes
and
the
other
is
notebook
service
accounts,
namespaces
and
data.
Don't
really
cooperate
really
well,
let's
talk
about
three
types
of
data.
There's
globally
shared
data,
and
that's
data
is
shared
between
multiple
teams,
users
and
projects.
There's
team
shared
data
and
that's
data
shared
between
team
members
and
user
data
and
that's
data
for
a
single
data
scientists.
A
The
problem
is
that,
if
you're
in
a
shared
namespace
it's
hard
to
protect
those
secrets
from
multiple
users,
so
my
tip
is
really
to
plan
your
namespaces
and
data
owners
very,
very
carefully.
You
really
need
to
sketch
out
what
pvs
live
where
who
can
access?
What
in
what
name
space
when
you're
setting
up
this
infrastructure,
the
security
implications
are
really
important.
The
second
area
of
data
security
that
you
have
to
pay
attention
to
are
team
resources.
A
Kubeflow
has
this
profile
namespace
concept
for
individual
users,
and
this
is
where
notebooks
run,
but
at
the
same
time
most
customers
that
we
see
actually
think
a
lot
about
resource
controls
and
resource
quotas.
One
of
the
values
of
kubernetes
is
that
they
can
restrict
a
teen,
a
data
science
team
to
a
certain
number
of
cpus
in
a
certain
amount
of
memory,
and
they
want
to
enforce
those
quotas
at
a
team
level.
A
So
my
recommendation
is
to
use
web
hooks
or
in
mission
controllers,
istio
and
jupiterhub
to
create
resources
like
volumes
and
storage
classes
in
your
profile,
namespaces
for
team
volumes
and
if
you're
going
to
run
your
training
jobs
in
the
team
name,
spaces
have
individual
user
volumes
exist
in
that
namespace,
but
make
sure
you
spend
a
lot
of
time.
Making
sure
the
rbac
is
is
is
secure.
A
Finally,
service
accounts
and
data
security,
so
data
scientists
live
in
notebooks.
Notebooks
live
in
containers
and
those
containers
are
launched
by
launch
processes
and
istio
routes.
The
the
browser
request
the
right
container
because
of
the
way
all
of
this
works.
These
containers
don't
run
as
the
user
themselves.
A
They
run
as
a
service
account
this,
but
inside
of
the
container,
the
user
needs
to
run
training
jobs
as
themselves.
They
need
to
talk
to
the
data
store
as
themselves,
and
this
means
they
need
credentials
passed
in.
One
of
the
big
tips
is
again
use
web
hooks
or
mission
controllers
to
create
secrets
with
these
credentials,
intercepting
the
notebook
creation
process
and
creating
creating
these
resources
be
very
careful
about
what
containers
you
allow
users
to
execute
into
because
of
the
the
types
of
secrets
that
you'll
want
to
mount
into
those
containers.