►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Next-Generation Data Science Workflows Using Ray - Erik Erlandson, Red Hat
Learn how to create distributed and cloud native data science workflows with Ray on Kubernetes. Ray is rapidly gaining momentum as an open source parallel computing platform that provides a scale-out cluster model inspired by tools such as Spark and Flink, yet also supports a lightweight scale-to-zero Serverless style workflow that is designed natively for modern container platforms in the Kubernetes ecosystem. Ray implements a constellation of tools that support data science and devops activities ranging from ETL, feature extraction, model training, ML pipelines, all the way through serverless inferencing. In this talk, Erik and Michael will discuss deploying Ray onto Kubernetes and integrating it with JupyterHub to create distributed and cloud native data science workflows. They will demonstrate Ray in action, running an end-to-end data science project on Kubernetes. Attendees will learn how to leverage the capabilities of Ray to do cloud native data science.
A
Okay
thanks
everybody
for
coming
I'm,
Erica,
Olson
and
today
I'm
going
to
be
talking
about
Next,
Generation
data
science
workflows
using
Ray
or
you
and
Ray
in
the
cloud.
Yes,
you
so
the
basic
structure
of
the
talk.
I'll
start
with
a
very
high
level,
description
of
Ray
and
then
I'm
going
to
try
to
put
Rey
and
the
demo
stack
you're
going
to
see
in
context,
then
I
talk
about
a
self-service,
Ray
architecture
that
we've
been
working
with
with
my
group
and
I.
A
A
Okay,
a
few
good
and
the
rest
of
you
have
Justified
all
the
contacts
I'm
about
to
deliver.
So
thank
you.
So
if
you
consider,
like
you
know
older
tools
like
open,
NPI,
one
end
of
the
spectrum,
you
have
like
extremely
fine-grained
control
of
how
you
develop
parallelism,
but
you
have
very
few
abstractions
to
help
you
along
the
way.
At
the
other
end
of
the
spectrum,
we
have
like
more
modern
tools
like
spark,
you
technically
have
less
control
over
certain
things,
but
you
also
have
extremely
powerful
abstractions
to
work
with.
A
So
how
does
Ray
work
as
a
compute
model?
It
has
two
kinds
of
compute
there's
tasks
which
are
things
like
functions
that
execute
and
then
finish
and
return
some
kind
of
results,
and
then
it
also
has
actors
which
are
like
you
know
the
analog
of
like
services
or
microservices
and
something
like
kubernetes,
and
so
you
can
take
any
python
function
and
turn
it
into
a
task
by
using
the
array
dot,
remote
decorators
you
see
above
and
similarly
the
same
decorator,
will
take
any
class
and
can
turn
it
into
a
reactor.
A
So
the
programming
user
experience
is
actually
quite
nice,
so
its
compute
model
is
a
sort
of
compute
dependency
directed
acyclic
graph
like
here
on
the
left.
You
can
see
that
I've
not
made
really
the
author
of
The
Blog
I
stole
this
from
is
creating
a
tree
of
computations.
You
could
say
it's
the
world's
most
over-engineered
edition
of
eight
integers
you
might
ever
see,
but
it's
great
for
illustrating
the
concepts
you
can
see
here
that
the
first
four
of
these
statements
are
setting
up
additions
of
some
pairs
of
integers
and
a
DOT
remote
method.
A
Error
is
a
thing
you
get
from
those
decorators.
I
just
showed
you,
so
it
basically
allows
you
to
tell
the
right
cluster
here,
I'd,
like
you
to
add
some
integers
and
then,
of
course,
if
you
look
at
these
first
statements
here,
anybody
who's
worked
with
spark
will
find
this
familiar.
These
are
declarative.
Computations.
Up
to
this
point,
Ray
has
not
done
any
work
for
you
at
all,
you're,
just
basically
telling
it.
A
Here's
some
work,
I'd
like
you
to
do,
and
here's
the
structure
of
it
and
the
dependencies
and
at
the
very
end,
if
you
issue
this
dot
get
command.
It
actually
goes
back
and
says.
Okay,
now
you
want
a
result.
I'm
going
to
unwind
all
these
computations
you've
asked
me
to
do
and
actually
compute
them
and
give
you
what
you
want
so
again
like
spark.
It
has
a
lazy
execution
model
so
raise
core
data.
Data
model
is
implemented
with
the
plasma
Object
Store
like
python.
It
is
basically
typeless
and
schema-less.
A
A
The
plasma
Object
Store
began
its
life
in
Ray,
but
has
since
been
actually
adopted
by
Apache
Arrow,
so
you
can
use
plasma
via
Arrow
if
you
ever
have
a
need
to.
Similarly,
race
scheduling
model
is
also
local.
First,
it
tries
to
use
a
local
worker
scheduler
to
do
tasking
whenever
it
can
and
only
like,
invokes
the
global
cluster
scheduler
when
necessary,
and
so
this
local
first
strategy
helps
things
to
you
know,
perform
as
well
as
possible
over
the
cluster.
A
It
comes
with
a
bunch
of
great
native
libraries.
It
comes
with
a
hyper
tuning,
hyper
parameter,
tuning
Library,
a
reinforcement,
learning
library
and
a
sort
of
generic
stochastic
gradient
descent
Library.
It
also
allows
a
scalable
and
programmable
a
serving
of
services,
so
called
Ray
serve
and
then
relatively
new
is
raid
data
sets,
which
is
actually
native
data
frames
for
Ray,
and
so
this
is
a
great
development
because
again
it
sort
of
puts
it
on
closer
to
equal
footing
with,
like
you
know,
data
frames
from
something
like
a
spark
or
pandas.
A
A
So
one
of
the
things
about
this
is,
of
course,
all
these
things
are
in
Python,
and
so
as
data
scientists
or
people
in
that
ecosystem.
We're
already
working
with
most
of
these
tools
via
Jupiter
at
least
part
of
the
time,
and
when
you
take
Rey
and
Jupiter,
there's
the
promise
of
literate
and
interactive
Ray
programming
via
Jupiter's
literate
programming,
environment
and
furthermore,
they
can
both
Jupiter
and
Reagan
operate
in
the
cloud
and,
of
course,
by
Cloud.
Today
we
mean
kubernetes
and
my
demo
will
actually
be
running
on
openshift.
A
A
So
it
gives
you
like
a
ray
head
node
and
some
worker
nodes
and
will
also
automatically
scale
them
up
and
down
for
you,
based
on
what
it's
seeing
on
the
workload
of
your
worker
nodes,
and
once
you
have
this,
you
can
connect
to
it
from
a
client
and
again
today
the
client
I
will
be
focusing
on
is
Jupiter,
but
I
want
to
stress
that
it
doesn't
have
to
be
that
it
can
be
other
python
files.
It
can
be
from
an
actual
command
line
exterior
to
the
cluster.
A
So
what
about
open
data
Hub?
Well,
it's
an
open
source
Downstream
of
kubeflow
and
as
such
it
serves
as
a
kind
of
reference
platform
for
doing
ml
workload
deployments
in
the
cloud.
It
shows
how
you
can
deploy
tools,
basically
how
to
wire
them
up
together.
I
know
from
previous
talks,
many
of
you
all
quite
familiar
with
how
to
do
this
stuff,
but
anyways.
A
If
you
would
like
to
deploy
that,
in
fact,
that's
how
really
we
got
the
integration
with
Ray
and
you
know,
along
along
the
one
axis,
the
tooling
that
comes
with
odh,
you
know,
like
kubeflow,
covers
pretty
much
the
spectrum
of
basic
tasks
in
a
data
science
project
in
the
cloud
and
along
other
axis.
It
also,
you
know,
covers
the
different
kinds
of
persona
all
the
way
from,
like
you
know,
business
stakeholders
to
data
engineering,
actual
data,
science,
ml
Engineers,
all
the
way
through
you
know
to
it,
Ops
I,
guess
we
call
it
ml
Ops.
A
So
I'll
begin
talking
about
like
the
architecture.
The
latest
architecture
we've
been
using
is
actually
even
simpler
than
some
of
the
older
ones.
We
just
created
a
sort
of
experimental
SDK
that
has
a
function.
A
python
function
called
Ray,
start
cluster
and
so
suppose
you're
in
Jupiter,
and
you
have
this
function
in
your
Jupiter
environment.
A
You
know,
connect
to
the
right
head
node
and
do
some
work
and
again
I
think
you
know
I
mentioned
getting
Jupiter
in
my
case
from
odh,
but
you
can
see
that
it
doesn't
actually
matter
as
long
as
you
have
like
a
container.
That's
like
running
Jupiter
lab
or
something
similar
for
you
and
you
have
the
right
function
and
you
have
the
array
operator
installed.
You
can
do
this
yourself.
A
So
one
thing
one
thing
with
these
things
is
of
course
we
had
to
ask
ourselves
well
if
I
spin,
up
array
cluster,
just
for
my
own
personal
use
and
then
like
the
thing
goes
down
like
my
Jupiter
pod
crashes
or
I
leave
and
I
just
close
down
the
environment.
It's
like
I'd,
like
the
cluster
to
go
away,
and
so
on
trick
we
figured
out
is
to
use
the
owner
references,
yaml
to
say
Hey,
you
know
it's
like
I
would
like.
A
I
would
actually
like
my
Jupiter
pod,
that
created
this
to
have
ownership
over
the
cluster
CR
and
so
like.
If
I
leave
and
I
forget
to
shut
down
my
Ray
cluster,
the
Jupiter
pod
goes
away
and
then
the
platform
itself
says.
Oh.
He
would
also
like
to
remove
this
CR
for
me
and
then
the
operator
notices
that's
gone
and
shuts
down
everything,
so
it's
quite
elegant
and
makes
proper
use
of
the
control
plane
and
so
now,
having
described
that
I'm
going
to
do
a
live
demo
of
this
in
action
before
I.
A
Do
I
just
want
a
quick
shout
out
to
my
teammate
Michael
Clifford.
He
took
all
my
original
prototypes
into
space
and
is
like
totally
modernized
them
and
gotten
them
working
with
Ray
2,
and
it's
just
done
a
ton
of
work
and
he
was
unable
to
be
here
today
to
help
me
present
but
raise
a
glass
it's
Michael.
So
I
will
now
bring
up
my
prefab,
Jupiter
environment.
A
And
we'll
begin
just
with
basic
stuff
doing
Imports,
you
can
see
we're
importing
a
bunch
of
Ray
tooling,
that
comes
with
and
here's
our
start.
Ray
cluster
and
I'm
going
to
give
it
a
name,
and
so
this
is
cool,
so
I
dump
a
tint.
So
if
I
ask
for
the
same
cluster
more
than
once-
and
it's
already
running,
it's
just
going
to
give
me
the
same
one,
which
is
a
nice
feature.
This
also
actually
wanted.
A
Just
because,
if
you're
trying
to
use
this
in
a
pipeline,
you
can
actually
ask
for
the
same
Ray
cluster
over
and
over
again
across
different
steps
of
the
pipeline.
If
you
like,
and
so
it
said
it
succeeded,
it
also
hopefully
printed
out
for
me
a
link
to
the
ray
dashboard
that
it's
set
up.
So
I'll
come
back
to
that
later,
but,
like
spara
gray,
has
a
dashboard
and
we
can
easily
stand
it
up
in
the
cluster
just
like
everything
else.
A
A
See
high
temperature
for
me,
there
we
go
so
Ray
will
also
provide
you,
the
in
version
and
basic
information,
we're
on
Ray
two
as
of
last
week,
so
very
exciting.
Again,
thank
you,
Michael,
and
so
now
we're
just
going
to
do
a
basic
Pi,
torch,
workflow
and
I'll
be
kind
of
blowing
through
a
lot
of
this
stuff,
but
we're
going
to
set
up
some
basic
data
sets
so
the
Oxford
pet
data
set,
which
is
a
relatively
small
thing,
there's
only
3
600
things
in
it.
A
I'm
also
sort
of
fixing
my
my
Jupiter
environment
I
just
want
to
use
CPUs
because
I'm
actually
giving
all
of
my
four
gpus
to
the
right
cluster,
which
we'll
see
in
a
second
and
then
I'm,
going
to
set
up
a
train
and
test
and
just
take
a
look
at
an
image,
some
kind
of
cow
with
long
ears.
So
this
pet
data
is
a
courtesy
of
the
community
data
license
agreement
via
the
Linux
Foundation.
Let's
do
a
brief
plug
for
this.
The
cdla
is
a
great
data
license.
A
It
makes
it
really
easy
for
people
like
me
to
give
up
give
talks
in
the
space
without
having
to
worry
about
commercial
restrictions.
So
when
you're
out
in
the
world,
favor
cdla
permissive
licensed
data
sets
just
like
you
favor,
you
know
things
like
open
source
licensing,
so
now
we're
going
to
create
an
extremely
low
budget
recognition
net,
which
can
finish
at
a
time
that
I
have
to
give
this
talk.
A
A
A
Going
kind
of
slow,
oh
there,
it
goes
so
we
can
say
here's
a
train
validation
set,
so
we're
going
to
split
this
is
actually
kind
of
fun.
If
you
see
the
shuffle
and
stuff
this
again
being
being
kind
of
an
analog
of
things
like
Hadoop
or
spark
data
frames,
you
can
see
some
of
the
things
here.
It's
doing
shuffles
and
this
thing
I
do
a
second
phase
here:
I,
either
reduce
so
again
you're
getting
a
lot
of
sort
of
familiar
functionality
with
these
data
frame.
Libraries
excellent!
A
So
now
we're
going
to
set
up
just
basic
function,
closures
to
do
training
and
validation
and
I
guess
the
one
one
thing
I
want
to
talk
about
here
is:
this
is
all
standard.
This
point
is
basically
very
standard
like
Pi
torch
kind
of
stuff,
so
it
doesn't
take
a
lot
of
any
weirdness
to
like
make
this
work
with
Ray.
You
just
give
it
to
create
a
torch
trainer
for
this,
so
we're
going
to
do
hyper
parameter
tuning,
so
I'm,
going
to
Define
myself
a
short
search
space.
A
The
search
space
has
only
four
tasks
to
go
with
my
four
gpus
and
I'm
going
to
go.
Kick
that
off.
So
at
this
point
this
thing
is
talking
now
to
my
array,
cluster
and
sending
off
a
bunch
of
hyper
tuning
parameter
tasks
to
complete
it's
extremely
chatty
in
the
logs.
But
you
can
see
things
like
you
know.
What's
pending
and
what's
running
it
starts
off,
it's
just
got
one
thing
running
and
Ray
is
waiting
to
see.
You
know.
Oh
look,
it's
asking
me
to
do
a
bunch
of
stuff.
A
A
Forget
that,
but
anyway,
it's
got
It's
actually
spun
up
four.
Now
it's
got
worker
and
three
nodes
and
I
can
also
take
a
look
at
the
dashboard
view
of
things,
and
you
can
see
that
it's
actually,
it
doesn't
just
have
nodes
here.
It
actually
has
tasks,
and
it
also
has
like
the
actions
that
are
going
on.
A
So
if
I
sort
by
CPU
I
can
bring
a
lot
of
these
to
the
top.
One
thing
you
might
notice
is
it
doesn't
just
give
me
CPU
usage.
The
green
is
over.
There
is
a
GPU
usage.
You
can
see
that
it's
not
making
fantastic
use
of
gpus,
so
I
could
maybe
make
use
of
the
Run
AI
tooling.
We
saw
earlier
to
see
what's
going
on
there,
but
it
is
using
them,
which
is
very
cool
and
we'll
look
at
how
this
is
going.
A
A
A
A
No,
it's
probably
almost
there.
It
goes
see.
It
took
a
whole
extra
minute
for
me.
So
anyway,
it's
done
this
and
we
can
now
take
the
results
back
from
our
hyperpreneur
parameter
tuning
and
see
what
we
got
so
there's
our
best
parameters
Ray
likes
to
emit
various
warning
messages
which
are
slowly
distracting,
but
we
can
also
plot
what
looks
like
on
our
training
and
eval
set,
and
we
can
see
that
it's
not
great.
A
A
So
we
did
learn
something
and
we
can
take
a
look
at
our
corresponding
confusion,
Matrix
and,
of
course,
you'd
really
like
to
see
this
all
in
the
diagonal.
If
you
stand
back
and
squint
it's
a
little
bit
of
diagonal
there,
and
here
we
can
look
at
our
cluster
name.
So
at
this
point
you
could
stop
the
right
cluster
I'm
not
going
to
stop
it,
because
I'm
actually
going
to
run
I'm
actually
going
to
run
a
race
serve.
A
B
A
Don't
have
that
problem
again
and
again
we
just
import
some
basic
torch
and
Ray
code,
and
we
can
start
our
cluster,
which
again
is
ident.
So
hopefully
it
will
give
me
the
thing
and
we
can
set
up
an
image
model
class.
The
app
serve
as
another
decorator
I
talked
about
it's
like
a
more
specialized
Ray
decorator
for
creating
like
model
serving
and
stuff
like
that,
and
I
can
now
connect.
A
Here's
a
little
bit
of
kluginess
to
find
the
actual
IP
address.
I
want
an
IP
address
of
the
head,
node
and
so
like
we
used
to
have
this
fun
procedure
where
we
just
played
roulette
until
we
got
the
right
one,
but
so
we've
started
a
server
and
it
gave
us
and
it
actually
gives
you
a
server
control
object,
which
we
don't
really
need.
At
this
point,
we
can
tell
it
okay
I'm
going
to
deploy
the
model,
it
knows
to
deploy
it
into
this
cluster
and
we
can
ask.
A
Let's
see
if
we
can
get
a
prediction:
I,
don't
even
care
if
it's
right,
I'm,
just
trying
to
show
you
that
the
workflow
works
and
sure
enough.
It
comes
back
at
the
prediction.
So
you
can
see
here.
We're
just
basically
got
a
rest
interface
now
with
not
very
much
code,
which
is
super
easy
to
deploy
so
easy.
You
can
even
do
it
inside
of
the
notebook
and
it
also
we
created
a
an.
A
A
A
The
operator
comes
with
again
I
guess,
in
contrast,
the
spark
you
know
spark
has
no
official
Community
operator
there's
a
couple
of
third-party
operators
and
with
Ray
it's
part
of
the
code.
It's
like
entry,
I
guess.
The
other
thing
you
saw
here
is
that
I
got
myself
my
own
cluster.
So,
like
you
know,
it
gives
you
an
easy
way
to
do
self-service
raise
so
your
data
scientists
or
your
you
know:
machine
learning,
Engineers
anybody
who
they
can
use
parallelism
when
they're
working
with
this
stuff
can
just
get
their
own
right
cluster.
A
You
know
on
demand
and
kind
of
turn,
you
know,
kill
them
on
demand
and
I.
Think
the
biggest
one
which
I
hope
you
saw
through
the
demo
was
that
Ray
allows
like
all
kinds
of
tooling.
Basically,
every
tool
of
consequence
in
the
python
machine
learning
ecosystem
already
talks
to
Rey,
and
it
not
just
that,
but
it
can
all
talk
to
it
at
once.
So
it's
like
you
can
mix
and
match
all
these
tools
right
in
the
same
code
base
and
I'll
use
the
same
Ray
cluster.
A
So
it's
a
unified
scaling
engine
for
data
science
and
what
that
also
means
is
especially
from
like
a
ml
Ops
perspective,
it's
a
simplified
scaling
process.
You
can
stand
these
things
up
and
you
know
your
your
people
can
get
parallelizable
scale
out
just
with
a
single
cluster,
even
with
all
the
tools
mixed
together.
A
So
there
were
some
community
communities
along
the
way
that
really
helped
us
I
did
the
original
public
deployment
of
my
prototypes
using
the
Massachusetts,
open,
Cloud
and
closely
in
partnership
with
that
I
did
all
the
specification
for
how
to
deploy
it
on
the
cluster
using
operate
first,
which
is
a
community?
That's
basically
trying
to
extend
the
idea
of
writing
software
in
the
open
to
also
actually
operating
software
and
services
fully
in
the
open,
and
what
do
I
mean
by
that?
A
A
We've
done
a
lot
of
collaboration
with
IBM
research.
They
have
a
project
called
code
flare,
which
is
also
doing
using
a
lot
of
the
same
design.
Principles
for
Ray
in
the
cloud
they're
very
much
focused
on
Pi
torch
applications
and
specifically
for
training,
large-scale,
deep
learning,
Nets
and
so,
like
they're
they're,
focused
on
that
use
case,
but
we're
doing
a
lot
of
collaboration
as
I
showed
you.
You
can
do
a
lot
of
small
scale
stuff
too.
A
A
And
I
think
you
know,
and
all
those
things
we're
looking
at
with
IBM
is
Ray
plus
a
cluster
level.
Auto
scaling
so
not
just
scaling
the
pods,
but
actually
scaling
the
size
of
the
cluster
using
Auto
scalers,
so
bringing
an
extra
whole
extra
compute
nodes
like
if
you're
on,
like
you,
know,
AWS
or
Google
Cloud.
So
in
this
in
this
mode,
you
array,
client,
submits
a
job
object
like
into
a
job
queue
and
the
job
queue
actually
talks
to
a
cluster
level.
A
Auto
scaler
says:
hey
this
job
coming
in
the
queue
they're
going
to
want
this
much
resources.
If
the
cluster
isn't
already
big
enough,
it
can
actually
invoke
the
you
know:
Auto
scaling
ability
of
kubernetes
and
start
bringing
in
actual
more
compute
nodes,
more
gpus,
and
when
the
job
is
done,
I
can
also
spin
the
cluster
back
down.
So
it's
actually
a
nice.
You
know
cluster
level,
Auto
scaling
experience
and,
of
course,
once
it
once
it
has
the
resources
it
needs.
It'll,
take
the
job
off
the
queue
and
it
can
run
the
cluster.
A
So
roadmaps,
stuff
in
green
is
stuff.
That's
actually
been
completed,
like
I
said
just
got.
Ray
2.0
working,
so
we're
excited
about
that.
We
have
our
Ray
images.
Building
using
project
Thoth,
which
is
a
AI
supported,
build
system
which
can
actually
look
at
things
like
what
kind
of
Target
Hardware.
Are
you
using
and
also
examine
a
supply
chain
and
dependencies?
We've
talked.
You
know
some
other
great
talks
earlier
about
that
issue.
So
Thoth
is
a
kind
of
cool
thing
for
like
actually
maintaining
images.
A
We
actually
just
recently
got
the
formal
integration
of
what
I
just
showed
you
Upstream,
with
the
open
data.
Hub
code
base,
things
that
are
in
progress,
we're
redeploying
Ray
on
the
Mass,
open
Cloud
and
we're
also
sort
of
transitioning
from
the
old
Mass
open
to,
what's
now
being
called
the
New
England
research,
Cloud
Newark,
so
neric
is
sort
of
like
the
descendant
of
Mass
open
and
that's
actually
being
done
by
some
student
teams
at
Boston
University.