►
From YouTube: How to Use Kubernetes to Build a Data Lake for AI Workloads - Peter MacKinnon & Uday Boppana
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How to Use Kubernetes to Build a Data Lake for AI Workloads - Peter MacKinnon & Uday Boppana, Red Hat
The recent popularity of hybrid cloud architectures have left organizations with traditional data lakes behind. How do you build a data architecture that is cloud provider agnostic and is flexible enough to run in across public and private cloud data centers? How do you make this data available to data scientists and developers in a way that simplifies the creation of intelligent applications? This talk will walk through a new way of building data lakes for the hybrid cloud using Rook and Ceph community projects running on Kubernetes. With a single data architecture deployment that can run in any cloud or across multiple clouds, IT and data engineers can use open source tools on Kubernetes such as Rook, Ceph, Hive metastore, Spark, and Presto to provide unified access to massive amounts of data across multiple data centers for data scientists and developers.
https://sched.co/Zenj
A
I
am
jude
bohanna,
a
principal
product
manager
at
red
hat,
responsible
for
ai
machine
learning
and
big
data
in
the
storage
business
unit
with
me
is
peter
mccannon
and
he
works
in
the
red
hat
ai
center
of
excellence
as
a
principal
software
engineer,
working
on
ai
and
machine
learning
technologies,
and
we
have
collaborated
on
this
project
for
close
to
a
couple
of
years.
So
this
talk
is
about
our
journey
that
we
went
through
and
the
experiences
we
had.
A
While
we
were
trying
to
build
an
ai
and
machine
learning
platform
at
red
hat
and
also
about
the
challenges
we
faced
on,
how
we
solve
them.
We
also
open
source
the
platform.
Now
it's
available
for
everyone
to
download
and
get
it
up
and
running
and
try
it
out
and
pete
will
talk
to
you
about
those
details
too.
So,
let's
get
started.
A
As
you
know,
ai
and
machine
learning
are
growing
workloads
within
the
enterprise
today
and,
as
we
started
more
of
these,
our
workloads
pop
up
within
the
enterprises
and
network
had
we
saw
a
set
of
common
challenges
that
arise
from
pharaohs
from
each
of
this.
The
biggest
challenge
we
saw
was
at
the
lack
of
really
usable
data.
All
of
the
data
in
the
company
was
siloed
in
different
locations
by
departments
or
by
the
owners
of
the
data
sets
and
the
authentication
and
authorization
policies
were
different
and
preventing
access.
A
A
A
A
cloud-like
experience
and
flexibility
in
their
in
their
execution,
so
once
we
listed
a
listen
to
them
and
figured
out
the
kind
of
common
challenges
and
high
priority
issues
we
were
facing,
we
came
up
with
a
set
of
design
points
around
what
we
needed
to
do
in
our
you
know
next
generation
architecture,
so
we
set
up
three
goals.
First,
is
we
needed
a
centralized,
shared
infrastructure
that
all
the
teams
could
use?
A
This
centralized
shared
infrastructure
broke
down
the
silos
for
data
and
also
enabled
collaboration
across
the
teams,
because
they
were
all
working
off
of
the
common
platform
that
they
had.
We
also
wanted
something
that
was
more
self-service
driven,
so
the
customers
can
get
a
more
cloud-like
experience
and
also
have
a
real-time,
a
workflow
like
they
are
used
to.
A
We
also
wanted
to
build
an
infrastructure
that
was
flexible
in
in
the
sense
that
it
could
allow
bring
your
own
tools
kind
of
a
workflow,
but
also
provide
with
a
set
of
standard
libraries
that
can
be
used
on
standard
tooling.
That
can
be
used
so
data
scientists
can
get
off
the
ground
and
get
their
work
done
faster.
A
So
once
we
set
those
goals
for
on
the
challenges
we
identified,
we
also
looked
at
the
end-to-end
ai
and
machine
learning
workflow
within
our
customers
and
internally
within
our
company.
So
we
wanted
to
figure
out
the
different
personas
that
were
emerging
in
the
air
machine
learning
landscape
and
we
figured
out
it's
anywhere
from
the
business
side
of
the
world,
which
is
business
leadership
that
has
financial
goals
to
your
traditional
itu
operations.
A
So
with
that
in
mind,
we
we
we
set
out
trying
to
solve
the
problem
now
that
we
have
a
set
of
challenges,
the
personas
and
user
requirements
and
our
goals
we
set
out
to
solve
the
problem
and
find
the
right
technologies
pretty
early
on
in
the
game.
We
figured
out
that
containerizing
on
kubernetes
was
the
right
solution
for
us
going
forward
and
would
help
us
to
build
the
next
generation.
Ai
and
machine
learning
platform.
A
Kubernetes
gave
us
pretty
much
most
of
the
design
points
we
wanted
that
we
set
as
goals.
It
gave
us
the
agility
to
quickly
respond
to
end
users
and
and
also
provide
it
in
a
self-service
way.
So
customers
can
compute
manage
compute
resources
as
they
need
it.
It
also
gave
us
the
portability
of
a
common
runtime
that
can
be
used
to
move
containers
between
or
between
companies,
at
the
edge
and
just
on
the
core
and
across
different
departments
or
across
the
company.
A
So.
For
this
part
of
my
presentation,
I'll,
I
will
focus
on
how
we
build
the
data
lake
and
the
data
management
capabilities
of
our
ai
and
ml
infrastructure
that
we
have
and
and
then
hand
over
to
pete,
to
talk
about
the
higher
layers
of
the
machine,
learning,
libraries
and
the
capabilities
that
we
built
on
top
of
this
data
lake
from
a
technology
perspective
like
I
said,
we
chose
kubernetes
and
for
the
storage
layer.
We
also
chose
to
build
it
on
ceph
and
rook,
and
we'll
talk
more
about
it.
A
So
when
we
started
to
set
design
the
data
lake
and
the
data
architecture
for
our
ai
and
machine
learning
infrastructure
in
red
hat,
we
set
out
to
understand
the
workflows
and
a
couple
of
workflows
stood
out
as
the
must
support.
So
we
we
learned
early
on
that.
There
is
a
requirement
for
certain
kubernetes
clusters
to
have
storage
that
was
localized
within
the
kubernetes
cluster.
A
But,
like
I
said
in
the
initial
part,
we
also
had
a
goal
to
build
a
centralized
data
lake
that
could
be
used
across
the
organization.
A
data
lake
that
could
be
shared
across
multiple
kubernetes
clusters
and,
and
that
would
be
the
single
source
of
truth
for
all
data
sets
being
used
in
the
organization.
A
So
what
we
really
wanted
in
the
long
term
is
what
you
see
sort
of
in
the
right
most
rectangle.
In
this
picture.
We
realize
that
both
apps
and
users
will
need
a
combination
of
some
local
storage
within
that
kubernetes
cluster.
That's
running
that
it's
easy
to
manage
and
easy
to
access,
but
also
the
need
for
a
long
term,
centralized
data
lake,
where
all
the
data
can
be
funneled
into
and
stored
for
things
like
machine
learning
and
analysis.
A
So
with
that
in
mind,
we
set
out
to
choose
the
right
technology.
Like
we
said,
we
chose
ceph
for
building
the
data.
Lake
ceph,
as
you
know,
is
an
open
source
storage
project
that
provides
a
uniform
that
provides
multiple
protocol,
support
for
object,
file
and
block.
It
also
is
software
defined
and
is
highly
scalable,
so
safe
fit
our
needs
from
that
aspect.
A
A
So
rook
will
take
the
right
daemons
required
for
ceph
like
mons,
mgrs
osgs
and
managers
and
spin
them
spin
those
containers
up
inside
a
kubernetes
cluster
and
create
a
full-fledged
ceph
cluster
within
an
existing
kubernetes
cluster,
which
means
you
have
a
full
functional
storage
cluster
to
use
right
away
for
all
your
applications
running
in
that
cube,
cluster
rook.
Also
more
excitingly
does
all
the
required
configuration
and
back-end
operations
required
to
support
the
native
cube
way
of
accessing
storage.
Using
rwx
volume
claims,
rw
volume
claims
and
the
newly
emerging
pocket
claims.
A
A
And,
of
course,
like
I
said,
for
we
also
needed
a
centralized
data
lake
that
would
store
all
the
long-term
data
assets
of
a
company,
and
for
that
we
chose
to
build
it
on
ceph,
also
running
on
linux,
and
in
this
case
the
majority
of
the
data
was
stored
in
object.
Storage,
as
you
know,
object
storage
is
emerging
and
standard
that
is
becoming
pervasive
for
storing
large
data
sets,
especially
unstructured.
Data
and
ceph
has
a
rich
set
of
object,
storage
capabilities,
so
that
gave
us
a
really
good
flexibility.
Again.
A
We
also
did
something
really
exciting
to
ease
a
data
engineer's
task.
When
we
built
this
infrastructure
on
data
lake,
we
provided
a
way
to
automatically
process
incoming
data
into
the
cluster
and
place
it
in
the
right
location
within
the
data
lay.
So
this
was
built
using
a
combination
of
self-bucket
notifications,
kafka
and
k-nato
serverless.
So,
in
this
architecture,
multiple
data
streams
would
bring
in
data
from
external
sources
into
a
cell
bucket.
A
Ceph
would
then
trigger
a
bucket
notification,
alerting
about
the
presence
of
new
data
or
the
changes
in
existing
data
sets,
and
these
bucket
notifications
would
be
received
by
kafka,
which
would
process
them
and
send
them
over
to
knight
to
serverless
knight
to
serverless.
Would
then
spin
up
the
right,
serverless
function
to
process
this
data
source?
Also
things
like
removing
pii
information
tagging,
any
required
metadata
tags,
adding
the
right
security
policies
and
then
move
this
new
data
set
into
the
right
location
in
the
data
lake.
A
A
It
does
the
required
metadata
operations
and
puts
it
in
the
right
location
and,
for
example,
if
you're
in
the
medical
field,
we
prototype
this
with
an
x-ray
engine
where
there's
multiple
x-rays
or
ct
scan
vertical
images
coming
into
a
cell
bucket
and
as
soon
as
the
images
are
stored
on
the
bucket,
a
notification
goes
out
from
ceph
to
kafka
and
then
to
k
native
and
k
native
would
spin
up
the
right
function.
That
looks
at
these
images.
A
Anonymizes
them
removes
all
the
pa
information
and
then
moves
them
to
the
right
bucket
that
can
be
used
for
machine
learning
engineers
that
work
on
this
anonymized
data.
So
the
combination
of
ceph
and
root
and
this
data
pipeline
ingest
solution
that
we
created
gave
us
a
pretty
good
benefit
for
creating
the
data
lay
and
the
data
infrastructure,
and
also
a
good
basic
framework
that
we
can
build
more
workloads
on
and
at
this
point
I'll
transition
to
pete
to
go.
Some
of
our
die
and
machine
learning,
workflows
and
tools
thanks
or
to
you.
Pete.
B
Thanks
uday,
the
open
data
hub
project
started
internally
within
red
hat
as
a
data
store
for
our
own
data
engineers
and
data
scientists.
Hence
the
name
data
hub
early
on.
We
realized
that
the
data
scientists
and
the
data
engineers
requirements
for
tools
and
aiml
components
are
different
than
devops
requirements.
B
B
The
main
pain
points
are
sharing
machine
learning,
work
initially
done
in
notebooks,
moving
the
model
to
production
from
those
notebooks
and,
of
course,
managing
the
model
while
in
production.
This
includes
monitoring
it.
Making
sure
predictions
are
accurate
watching
for
data
drift,
managing
resource
usage
for
cpu
gpu
memory
and
lots
more.
B
B
Talking
about
the
personas
that
uday
laid
out
for
us
in
the
context
of
open
data
hub
for
the
data
analyst
persona
open
data
hub
provides
integration
for
data
lakes
such
as
an
s3
interface
to
ceph,
object,
storage,
there's,
sql
databases
such
as
postgresql
and
mysql,
and
data
streaming
using
kafka.
Strimzy
for
data
exploration,
open
data
hub
includes
the
superset
and
hue
projects.
B
B
Jupiter
hub
is
also
provided
for
a
notebook
development
environment
integrated
with
openshift
authentication
and
can
also
make
use
of
available
gpu
resources
if
they're
enabled
on
the
kubernetes
nodes,
multiple
tools
for
model
training,
verification
are
provided,
such
as
for
the
frameworks,
tensorflow
and
pi
torch
and,
of
course,
spark
open.
Data
hub
also
provides
pre-trained
models
as
part
of
its
ai
library
component,
so
pre-trained
models
for
things
like
fraud,
detection
and
various
other
types
of
machine
learning,
algorithms
for
creating
machine
learning
pipelines.
B
So
the
meta
project
integrates
these
open
source
projects
into
one
project
that
makes
it
easier
to
be
consumed
by
users
going
back
to
sort
of
where
all
these
projects
come
into
play
in
the
overall
ml
workflow.
It
starts
with
data
prepping
and
etling
that
data
into
a
data,
lake
or
storage
of
some
kind
and
making
it
accessible
to
the
data
scientists.
B
It
is
not.
It
is
now
integrated
to
open
data
hub
and
runs
on.
Openshift
kubeflow
brings
multiple
new
aiml
capabilities
and
features
so
for
distributed
model
training.
We
have
tensorflow
jobs
and
pi
torch
jobs,
mpi
jobs
for
model
serving
again
they're
seldon,
but
there's
also
a
new
project
called
kf
serving
which
provides
an
abstraction
layer
over
the
various
different
types
of
surveying
frameworks
for
pipelines.
We
have
kubeflow
pipelines
based
on
argo
and
as
part
of
the
goal
of
all
this,
we
upstream
any
enhancements
back
to
the
kubeflow
project.
B
B
B
we
recently
released
open
data
hub
version.
0.52
that
was
built
using
the
ansible
operator,
includes
many
of
the
aiml
tools
we
talked
about
previously,
but
we
also
support
a
a
dually,
a
new
version
of
the
operator
that
is
based
on
the
kubeflow
golang
kf
cuddle
operator,
and
that
gives
us
this
mechanism
for
better
integration
with
kubeflow.
B
So
this
dual
mode,
these
two
operators-
zero
five
two-
is
to
support
so-called
legacy
deployments.
That
going
forward
will
only
have
bug
fixes,
however,
with
the
open
data
hub
0.7
operator,
based
on
kubeflow.
It
integrates
with
kubeflow,
includes
kubeflow
components
as
well
as
some
of
the
legacy
open
data
hub
tools
that
have
been
migrated
to
the
kf
cuddle
customer
resource
definition,
so
that
version
will
be
the
target
for
all
new
feature.
Development.
B
B
A
B
And
I
will
take
number
three
and
the
question
is:
could
you
provide
links
to
the
product
or
download
page
it's
important
to
emphasize
the
open
data
hub
project
and
I'm
assuming
we're
talking
about
open
data
hub
as
opposed
to
the
the
ceph
project,
the
information
for
that
product
can
be
found
or
project.
I'm
sorry
can
be
found
at
open
opendatahub.io,
so
we
have
a
documentation
there
blog
posts
and
we
host
in
gitlab
the
the
source
for
the
various
components
there.
B
There
are
pieces
that
are
hosted
in
github
for
for
legacy
reasons,
also
with
openshift
4.
It
has
an
embedded
operator,
lifecycle
manager
and
in
fact
the
easiest
way
to
install
open
data
hub
is
just
to
make
use
of
that.
So
you
would
go
to
the
openshift4
console
and
you
would
bring
up
the
catalog
the
first
item.
B
There
will
be
ai
and
machine
learning
and
you
can
just
type
in
open
data
hub
and
it
will
come
up
as
one
of
the
community
operators
and
you
can
click
on
that
tile
and
it
will
sort
of
work
you
through,
basically
the
steps
for
installing
that
into
the
cluster
directly.
Hopefully
that
answers
the
question.
A
I
can
take
question
number
four
pete.
The
question
is
in
the
automated
injection
pipeline.
Does
the
k
native
serverless
function
read
directly
from
sephora?
Does
the
message
consume
from
kafka
contained
within
itself?
The
data
payload
as
well,
so
both
the
modes
are
are
supported
in
fact,
so
today
ceph
sent
a
notification
to
kafka
and
kafka
initiates
the
knee
to
interaction,
but
that
is
all
the
metadata
flow.
A
The
real
data
is
still
on
ceph,
and
so
cephei
is
the
one
that
starts
the
notification
process
with
bucket
notifications,
where
there
is
also
a
way
that
we
prototyped
upstream,
where
ceph
can
directly
send
a
notification
to
k
native
and
trigger
the
help
to
trigger
the
right
function.
That
is
being
tested
right
now,
it's
upstream
and
we
will
productize
it
in
the
next
coming
days.
B
A
The
way
we
backup
is
using
the
multi-site
feature
where
we
create
a
second
site
and
keep
a
copy
of
all
the
data
in
and
and
pete
can
talk
about,
some
of
the
next
generation
backup
workflows
that
we're
working
on
more
around
time-based
backup
and
restoration
and
auto
restoration
and
stuff,
and
we
are
making
some
progress
there.
It's
a
work
in
progress
for
more
real-time
backup
and
stuff.
B
B
There
are
other
solutions
that
we're
looking
to
incorporate
in
open
data
hub
with
respect
to
governance
and
there's
a
value
evaluation
going
on
of
some
other
candidate,
open
source
components
that
could
fulfill
sort
of
a
broader
governance
solution
there.
So
things
like
atlas,
ranger
algeria.
These
are
all
components
that
we're
actively
looking
at
for
providing
sort
of
a
broader
data,
governance
and
self-service.
B
B
Looking
for
comparisons
here
has
this
been
passed
passed
on
to
the
new
cncf
research
group
for
review,
as
working
with
them
for
reviews,
so
open
data
hub
is
an
open
source
project.
It
is
not
currently
affiliated
with
cncf
in
terms
of
under
any
of
the
umbrella
projects.
For
that
we'd,
be
probably
interested
in
the
community
would
be
interesting.
In
hearing
more
about
this
use
case,
the
community
meets
bi-weekly
on
mondays
and
on
our
open
data
hub
io
website.
B
A
B
Yeah,
so
one
form
of
the
question
is
that
openshift
only
the
other
form
is.
Does
this
run
on
stock?
Kubernetes
openshift
is
kubernetes,
however,
it
does
have
extensions,
and
some
of
those
extensions
are
used
natively
in
open
data
hub,
so
it
is
targeted
for
open
shift.
A
lot
of
the
constructs
will
be
obviously
familiar
in
terms
of
deployments
and
pods
and
services,
and
things
like
that,
but
we
also
make
use
of
resource
objects
and
openshift
called
routes
which
are
actually
a
higher
level
of
abstraction
of
ingress
resources.
A
So
question
number
nine
is
more
of
a
comment.
So
yes
piloting
now
and
using
many
of
the
same
tools,
we
would
love
to
have
more
feedback
and
even
have
you
collaborate
with
the
community
like
people
is
referring.
We
do
have
regular
meetings
for
the
community
that
are
on
the
website,
so
please
feel
free
to
join
in
share
feedback
and
we
can
learn
from
you
and
you
can
get
more
information
from
the
community
too,
and
question
number
10
is
the
same
the
session
in
the
slides.
A
B
B
Another
question
we're
just
closing
up
here
running
out
of
time.
We
need
disconnected
install
for
odh.
It
is
not
supported
yet.
Yes,
we're
aware
of
that.
It's
it's
an
issue
that
we're
still
looking
into
and
also
we
have
data
sets
in
nfs
large
seismic
data
sets.
How
do
you
picture?
We
can
consume
these
data?
Maybe
that's
the
last
question
for
you.
Uday.
A
Sure
yeah,
so
you
can
consume
them
using
the
native
pv
persistent
volume
construct
that
kubernetes
provides
an
openshift
provide,
or
you
can
also
do
out
of
band
access,
but
we
recommend
the
persistent
volume
a
way
of
consuming
the
data
sets
already
in
nfs,
because
that
follows
with
the
data
on
the
control
path.
That's
emerging
in
the
kubernetes
pa
to
read
and
write
persistent
data.