►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
In this webinar, we’re going to demonstrate a multi-cloud ML workflow with advanced data management on Kubeflow. We will showcase how a Data Scientist can set up their own ML development environment in minutes, start working locally, and seamlessly extend their workflow to a public cloud.
Takeaways:
• Learn about Kubeflow, a complete ML platform on top of Kubernetes
• Learn why data handling is critical to an end-to-end ML workflow
• Learn how you can have all your work, along with your data, packaged, versioned and reproducible across every step of your ML workflow
• Learn how you can run ML workflows that span hybrid and multi-cloud environments
A
Alright
looks
like
the
numbers
are
stabilizing,
so
let's
go
ahead
and
get
started.
Welcome
to
today's
CMC,
f,
webinar
planet
scale,
machine
learning,
workflows
with
advanced
data
management
on
cube
flow,
I'm,
Kaitlin,
Barnard
marketing
manager
at
CNCs
and
I'd
like
to
thank
you
all
for
joining
us
today.
I'd
also
like
to
welcome
today's
presenter.
Bangle
is
cookus
CTO
and
co-founder
at
a
Rick
doe.
So
just
a
few
housekeeping
items
before
we
get
started
during
the
webinar,
you
are
not
able
to
speak
as
an
attendee.
There
is
a
Q&A
box
at
the
bottom
of
your
screen.
A
So
if
you
have
any
questions
that
come
up
throughout
the
webinar,
please
feel
free
to
drop
them
in
there
and
then
we'll
hold
Q&A
until
the
end
and
get
to
as
many
of
those
questions
as
we
can.
This
session
is
also
being
recorded
and
will
be
sent
out,
along
with
the
slides
and
after
the
presentation.
So
with
that
I'll
hand
it
over
to
Benglis
to
kick
off
today's
presentation,
Thank.
B
You
Kate,
it's
a
great
pleasure
to
be
here
today,
nice
to
meet.
You
I
hope
you
enjoy
today
sooner
so
as
Caitlin
mentioned,
my
name
is
Curtis,
founder
and
CTO
of
erecto.
What
we're
doing
here,
it
too
is
planet-scale
data
management
and
we
focus
on
kubernetes
and
ml
environments.
So
let
me
switch,
let
me
make
sure
I
can
switch
our
slides
share.
My
screen,
you
should
be
seeing
the
slide
yep.
B
So
today,
we'll
be
talking
about
climate
scale
and
mail
workflows
with
advanced
data
management
on
Cooper.
What
does
that
mean?
It
means
we'll
go
over.
What
planning
scale
means
for
us
we'll
talk
about
machine
learning
on
kubernetes
coop
flow,
which
is
BML
to
kid,
for
kubernetes
focus
on
how
managing
data
is
very
important
for
efficient
machine
learning,
talk
about
data
management
in
coop
flow,
which
is
the
area
in
which
we
have
made
a
few
contributions
to
cook
flow
and
then
move
the
platform
and
to
end
coop
flow
along
with
our
integrations.
B
So
by
now
it's
a
definite
fact
that
we
live
in
the
kubernetes
era.
Kubernetes
has
become
the
default
container
istration
engine.
No
one
can
question
that
anymore
across
platforms
across
different
kinds
of
infrastructure.
Kubernetes
is
the
single
simple
way
to
declare
your
containerized
infrastructure
and
then
the
underlying
platform
does
whatever
you
guys
could
do,
but
this
is
a
sincere
planner.
So
there's
no
reason
for
me
to
argue
for
kubernetes
parents
who
are
already
so
company's.
The
question
is
what
happens
with
machine
learning
now
that
kubernetes
is
gaining
ground
so
fast.
B
So,
although
everyone
is
doing
their
own
thing
right
now,
we
believe
that
eventually
ml
will
run
on
kubernetes
with
kubernetes
primitives
gaining
all
the
benefits
at
kubernetes
brings
with
kubernetes
primitives
means
that
machine
learning
will
be
running
in
a
declarative
way.
Users
will
be
declaring
to
kubernetes
the
kind
of
a
mail
infrastructure
they
need
and
kubernetes
will
make
it
happen.
Users
will
have
a
single
way
of
asking
what
they
want
to
have
across
infrastructure
across
locations
and
kubernetes
will
make
it.
B
It
will
make
it
happen
and
because
users
cannot
really
don't
really
have
to
speak
to
kubernetes
directly
because
their
ml
users,
the
data
scientists.
What
do
we
need
in
between?
We
need
coop
flow,
that's
the
overall
idea.
So
why
should
a
man
run
on
kubernetes,
because
this
will
bring
containerization
but
better,
more
efficient
handling
of
hardware
sources,
auto-scaling
portability
across
locations
across
infrastructure,
extensibility
and
no
vendor
locking?
B
So
we
are
just
getting
started
on
things,
but
the
very
good
first
step
is
coop
flow
is
here
and
coop
flow
is
the
ML
toolkit
for
kubernetes.
It
is
the
first
native
machine
learning
to
teach
for
kubernetes
announced
by
Google.
Just
over
a
year
ago,
end
of
2017,
it
is
already
picking
up
speed
to
become
the
de-facto
ml
platform.
B
It's
a
way
of
declaring
your
ml
platform
using
kubernetes,
primitive,
music
Burnett
as
components
so
I
will
spend
the
next
few
minutes
with
some
slides
I
got
from
a
presentation
by
J,
Smith
and
David
our
own
chick
David.
Our
own
ship
is
the
co-founder
of
the
coop
flow
project.
This
presentation
was
given
in
koukin
North
America
2018
in
Seattle
last
December
machine
dominguez
code
to
essentially
introduce
coop
flow
to
those
of
you
who
may
not
have
heard
about
it
before
or
maybe
that
week
recap
to
get
upstream.
B
B
Building
a
model
is
just
one:
stick
in
a
very
big
email:
workflow,
it's
a
multitude
of
components,
but
we
need
to
take
care
of
and
we
need
to
stitch
together
how
about
working
ml,
a
pipeline
end-to-end
from
an
initial
data
set
that
we
have
to
ingest,
we
have
to
analyze,
we
have
to
transform
and
validate
and
split,
and
then
we
have
to
train
an
initial
version
of
our
model
and
validate
that
it
works,
and
maybe
it
doesn't
so.
We
have
to
go
back
and
train
it
again
and
modify
it.
B
So
it's
an
iterative
process
and
then
we
have
to
train
at
scale
and
when
we
have
a
model
that
in
train
that
scale
and
works,
you
have
to
roll
it
out
and
serve
it
in
production.
And
then
you
have
to
monitor
it
and
then
have
a
blog
which
progress
and
then
we
have
to
go
back
and
fix
things.
So
from
the
initial
data
to
having
a
model
running
in
production.
There
is
many
steps
and
these
steps
sometimes
have
to
happen
in
different
places
and
sometimes
have
to
be
executed
by
different
people,
different
personas.
B
So
this
becomes
a
very
complex
talk
and
it
requires
expertise
in
a
multitude
of
layers,
because
it's
not
just
the
top
layers
that
the
data
scientists
or
the
data
engineer
would
care
about.
It's
not
just
the
framework
and
the
tooling
and
being
you
X
and
the
model
building.
It's
also
the
lower
level
things.
What
about
my
runtime,
what
my
drivers,
my
operating
system,
my
GPUs,
the
hardware
we
ran
on?
B
How
do
I
scale
my
infrastructure,
so
this
is
complicated
enough,
but
then,
given
that
different
parts
of
the
work
flow
will
run
in
different
places-
and
you
need
to
replicate
all
this,
so
what
we
talked
about
so
far
is
the
upper
layers.
If
you
have
to
replicate
this
for
experimentation
locally,
maybe
on
my
laptop
and
then
to
train
at
scale
in
some
bigger
infrastructure
and
then
to
have
the
model
running
in
production
on
the
cloud.
I
have
to
replicate
all
these
layers.
How
do
I
do
that?
B
So
kubernetes
is
the
platform
on
which
I
run
my
machine
learning
so
now,
I
can
move
from
location
to
location
and
from
one
kind
of
infrastructure
to
another,
from
one
cloud
provider
to
the
other,
with
Burnett
as
being
the
language
in
which
I
declare
my
infrastructure
and
then
let's
get
cook
slow
as
the
single
way
of
the
developing,
deploying
and
managing
machine
learning
on
kubernetes.
So
let's
get
could
flow
to
be
the
single
language
in
which
I
describe
kubernetes
my
machine
learning
infrastructure.
B
So
again,
kubernetes
takes
care
of
the
lower-level
stuff
and
then
coop
flow
becomes
the
language
in
which
I
declare
the
way
I
handle
my
steps
in
I'm,
island,
mail,
workflow,
so
coop
flow
becomes
this
part
and
then
have
coop
flow
everywhere.
So
my
job
is
much
simpler,
I
guess
kubernetes
in
notations
and
then
I
declare
my
machine
learning
infrastructure
using
coop
flow.
That's
the
basic
premise
behind
control,
okay,
so
coop
flow
and
kubernetes
in
our
view,
are
pivotal
in
enabling
next-generation
machine
learning.
And
what
do
we
mean
by
next-generation
machine
learning?
B
We
means
the
ability
for
distinct
roles.
The
data
engineer,
the
data
scientist,
the
DevOps,
a
person
all
interacting
across
rotations
to
make
this
workflow
function,
so
we
need
collaboration.
We
need
different
people,
doing
different
parts
of
this
workflow
and
still
being
able
to
collaborate
in
a
way.
That's
reproducible,
whatever
producer
would
mean
that
when
someone
has
a
step
completed
some
other
person
can
they
can
hand
it
over
to
some
other
person.
Some
other
person
can
take
over
their
work
and
they
can
reproduce
near
exact
state.
So
they
can
continue
the
the
workflow.
B
We
need
a
simple
streamlined,
unified
workflow
from
the
first
pre-processing
steps
to
developing
the
model
to
training
to
get
into
production.
From
my
laptop
to
the
cloud
to
the
autonomous
car
may
be
well,
the
model
relation
will
eventually
run
where
inference
will
happen,
and
ideally
in
next
generation
and
milk
would
like
everything
to
execute
fast.
So
the
question
comes:
where
is
the
data
for
the
steps
to
happen?
The
data
should
be
ideal
in
local
if
I
keep
a
data
somewhere
remote
on
a
slow
data.
B
Lake,
if
I
depend
on
a
single,
slow
object,
store,
then
no
matter
how
many
GPUs
I
have
how
fast.
My
infrastructure
is
I
will
be
bound
by
data
access
latencies.
So
this
is
where
we
as
a
Richter
coming
to
the
picture,
and
this
is
how
this
was
the
trigger
for
our
involvement
with
cook
flock.
The
question
is:
how
do
you
manage
data
for
this
ml
workflow
when
this
data
has
to
be
in
multiplication?
B
How
do
you
manage
data
in
a
pipeline
that
runs
across
locations
because
I'm,
a
milk
pipeline
is
as
good
as
its
data,
so
I
will
spend
the
next
few
minutes
describing
we
are.
We
come
from
a
systems
engineering
background.
We
work
mostly
on
data
management
for
kubernetes.
Our
involvement
with
coop
flow
is
contributing
code
to
it.
So
coop
flow
manages
data
as
kubernetes
native
resources
and
then,
if
coop
flow
Margie's
data
as
kubernetes
native
resources,
our
software
can
sit
on
the
side
and
provide
the
efficient
data
management
for
these
resources.
B
What
we
do
explicitly
is
we
allow
you
to
snapshot
version
package,
distribute
and
eventually
share
with
others,
your
full
environment,
your
resources
on
coop
flow,
along
with
their
data.
So
this
makes
the
pipeline
the
ML
workflow
work
across
teams
across
locations
and
across
different
kinds
of
infrastructure,
on
Prem
or
on
the
cloud.
B
On
the
Google
cloud,
for
example,
our
software
seeds
are
long
kubernetes
to
enable
this
kind
of
interruption.
So
our
contribution
to
tube
flow
is
a
two-fold.
We
extend
it,
so
it
works
over
PVCs
and
then
we
use
our
software
on
the
side
of
PVCs
to
enable
cross
location
collaboration.
This
is
the
overall
picture
of
what
I
will
be
coming
to
you
today.
B
What
you
get
with
this
is
a
way
to
version
packets
and
distribute
your
data,
and
then
you
can
have
a
unified
hybrid
pipeline
that
works
both
on
Prem
and
on
the
cloud
and
because
we
work
with
snapshots,
because
when
we
move
data
we
essentially
snapshot
the
data.
You
can
always
go
back
in
time
and
see
what
the
state
of
the
pipeline
was.
So
you
can
always
reduce
it
and
try
again
and
come
back
to
the
exact
state
where
you
were.
B
I've
talked
a
bit
about
these
things,
but
it's
best
if
we
can
watch
it
live.
So
this
is
the
demo
that
we'll
be
doing.
We
have
a
machine
learning
pipeline
same
as
I
showed
before
on
crude
flow.
We
have
two
users
Sarah,
which
will
be
doing
the
experimentation
part,
the
data
ingestion
and
pre-processing
part
in
location,
a
and
James
who
will
be
doing
the
training
part
location
B.
So
you
build
a
model
and
train
it.
B
So,
given
this
and
Mel
pipeline
Sarah
will
live,
will
essentially
be
doing
these
steps.
A
bit
shortened
and
James
will
be
doing
these
steps
and
for
this
we'll
use
an
example
that
a
cube
flow
community
has
created
called
the
github
issue.
Summarization
example.
This
example
is
a
machine
learning
pipeline
where
we
get
data
from
github
issue
data.
Is
she
text
and
the
model
tries
to
create
a
summary
but
describes
the
content
of
the
issues
as
good
as
possible?
So,
let's
start
with
this
I'll
use
two
locations.
B
This
virtual
desktop
is
location,
a
he
trans
on
Amazon
in
zone
uswest
1a.
It
runs
coop
flow
on
kubernetes
and
this
desktop
will
be
location
B.
It
runs
on
Amazon,
again
zone.
He
West
West
is
West,
one
be
running
with
Cooper,
so
this
is
where
changes
on
B-
and
this
is
where
Sarah
is
on
a
so.
What
does
Sarah
do
she
uses
coop
flow?
She
starts
a
Jupiter
notebook.
B
B
B
Let's
call
it
become
Delta
10
gigabytes
in
size
which
will
be
mounted
here
so
Sara
will
be
managing
her
data
as
another
locally
mounted
filesystem.
This
is
backed
by
a
persistent
volume
on
kubernetes,
so
Sara
creates
your
notebook.
So
what
happens
here
is
Sara
uses
code
flow
to
allocate
a
new
development
environment
for
her
do
a
map
and
this
development
environment
will
be
a
jupiter
notebook.
B
B
B
Summarization
example
from
cook
flow,
so
Sarah
brings
her
code
here.
This
is
your
code.
Okay,
and
this
is
where
she
bookstore
her
data.
This
data
volume
is
backed
by
a
local
ephemeral,
nvme
disk
SSD
on
an
amazon
ec2
instance,
currently
so
Sarah
stores
as
fast
as
she
can,
with
minimum
latency
to
a
local
disk.
So
she
can
work
as
fast
as
you
can.
This
could
be
her
laptop.
Even
she
is
working
on
her
laptop
and
storing
to
her
local
hard
disk.
B
Okay,
and
now
she
starts
with
a
notebook.
We
already
have
a
notebook
to
start
from,
we
won't
be
creating
from.
We
won't
be
creating
it
from
scratch.
This
is
the
notebook
we
will
be
using.
So
Sarah
can
now
import
shared
libraries,
okay
and
now,
what's
the
first
part
in
animal
pipeline,
Sarah
wants
to
experiment.
So
the
very
first
part
is:
let's
get
the
data
that
we're
going
to
be
using
into
our
development
environment.
Let's
ingest
the
data
into
the
pipeline,
so
Sarah
has
this
empty
directory
where
she
will
store
your
data
mounted
onto
her
pod.
B
She
remembers
it
in
a
variable
and
she
uses
W
gate
to
fetch
the
issued
data
as
a
big
zip
file
somewhere
from
an
external
data
lake.
So
this
is
the
first
data
ingestion
part
and
then
once
she's
brought
the
data
into
her
environment,
she
will
also
uncompress
it.
She
will
unzip
it.
So
this
is
an
initial
data
transformation
step,
so
Sarah
runs
W
gate.
She
goes
to
the
external
data
leak
register
data
into
her
environment.
B
B
B
Okay,
going
back
the
road
book,
so
sarah
has
unzipped
the
data.
She
has
done
an
initial
transformation
step.
Okay,
let's
assume
this
is
the
first
type
of
the
pipeline.
So
let's
assume
sarah
has
completed
that
to
the
ingestion
and
analysis
and
transformation
part.
Okay.
Now
at
this
point
she
may
want
to
hand
over
her
work
to
somebody
else
or
she
may
want
to
take
a
snapshot
of
her
work.
So
she
can
go
back
in
time
and
review
what
she
did.
Okay,
so
for
this,
because
of
data
is
stored
on
a
compare
method,
object,
a.
B
Persistent
volume
we
can
use
our
software
to
take
a
snapshot
of
the
whole
infrastructure,
so,
let's
take
with
two
of
our
sofa.
This
is
rock.
This
is
what
runs
alongside
coupe
flow,
and
this
integrated
would
cook
flow,
so
you
can
manage
your
data
across
locations,
so
this
is
Sarah's
deployment
on
your
local
infrastructure.
B
There
is
two
buckets
Sarah's
bike
it
and
James
bucket
buckets
is
where
we
keep
all
user
data
snapshots
of
food
environments.
Let's
have
a
look
in
a
previous
budget,
this
one,
for
example,
it
contains
a
few
snapshots
of
environments
we
took
previously
some
time
ago.
Each
environment
is
a
snapshot
of
the
workspace
volume
where
Sarah
keeps
her.
Libraries
and
the
data
volume
for
Sarah
would
keep
your
data.
So
the
Alice,
an
environment
snapshot,
is
a
group
of
two
snapshots,
a
workspace
snapshot
and
the
data
snapshot,
and
then
for
each
environment.
B
We
can
have
multiple
versions,
so
you
can
go
back
in
time
and
see
what
your
environment
looked
like
ten
minutes
ago,
20
minutes
ago
or
a
week
ago.
So
it's
essentially
a
time
machine
for
their
full
environment
based
on
snapshots,
and
this
is
how
we
enable
the
pipeline
to
span
multiple
locations.
We
move
the
pipeline
State
from
one
location
to
another
to
allow
to
move
from
location
to
location.
B
B
She'll
snapshot,
both
your
workspace
and
the
data.
This
is
essentially
get
like.
Sarah
commits
her
food
environment
into
our
software
into
Rock,
including
the
data
suggest
landlord,
so
ingress
data
unzip
into
CSV
ingress.
The
entry
date
run
initial
decompression
step.
These
are
notes
she
keeps
for
herself
in
the
complete
message:
okay,
I'm
at
this
point:
what
happens
is
our
software
creates
an
a
synchronous
task
that
gets
to
the
kubernetes
native
objects,
the
PVCs
that
Google
has
created
and
takes
a
snapshot
of
them.
B
So
we
take
a
snapshot
of
whatever
libraries
she
has
created
and
whatever
data
she
has
brought
into
her
environment,
and
it's
interesting
to
note
that
we
do
it
thinly.
So
we
know
what
parts
of
the
Degas
it
has
changed,
who
have
changed
and
we
only
read
the
changed
part.
So
if
you
look
at
the
data
volume,
she
has
brought
in
more
or
less
4
Giga
4
gigabytes
of
data.
That's
why
we're
reading
this
many
data
pieces
next
time
Sarah
wants
to
take
another
snapshot
of
your
environment.
B
B
And
click
snap
ship
is
done
so
Sarah
now
has
a
full
snapshot
of
your
environment.
At
this
point
in
time
containing
her
data
volume
and
your
library
body,
this
is
it
anak
can
continue.
Now
she
can
pre-process
the
data
okay,
so
she
will
split
the
data
into
a
training
set
and
a
test
set.
We
will
use
a
limited
sample
size
of
200
samples,
so
it
executes
is
quite
quickly.
So
this
is
the
pre-processing
part
now
running
I
gain
these
runs
over
in
local
SSD.
So
it's
as
fast
as
possible.
B
B
Instantiate
a
text
process
or
we
have
our
text
process
or
okay.
We
have
processed
our
data,
a
small
sample
of
it
anyway,
so
it
finishes
quickly
and
now
we'll
serialize
everything
to
disk.
So
at
this
point,
Sara
will
dump
the
pre-processed
state
into
the
data
volume
she'll
dump
files
into
her
mounted
filesystem
okay.
B
So
this
is
the
result
of
the
data
pre-processing
step.
So
if
server
goes
to
the
data
body,
you
see
that
we
have
the
original
dataset
at
the
zip
file
and
zipped,
and
then
we
have
the
result
of
the
pre-processing
step.
So
at
this
point,
Sara's
done
with
the
pre-processing
step.
She
has
a
trainer
already,
it's
part
of
the
notebook,
so
we're
done.
We
now
need
to
build
the
model,
so
we
need
to
move
the
pipeline
now.
So
James
can
work.
B
James
has
to
start
from
where
sara
has
left
the
pipeline
or
sara
may
need
to
come
back
to
this
step.
If
things
fail
and
try
over
or
modify
a
few
parameters
or
try
out
the
new
things,
try
out
some
new
things
and
she
needs
to
do
it
in
a
way
that
retribution
she
needs
to
know
exactly
what
she
did
to
reach
this
point.
So
what
Sara
will
do
again
after
having
completed
processing,
she
will
go
and
get
another
snapshot,
so
she
gets
our
software
snapchat,
her
full
environment
again.
B
And
she
has
done
toast
a
to
this
podium,
so
she
takes
another
snapshot.
This
time
the
snapshot
will
be
way
faster
because
she
didn't
actually
touch
all
of
your
data.
We
will
detect
that
she
has
only
inserted
new
state
in
here
volume.
The
happy
processed
data-
and
this
is
done
so
now
Sarah-
can
go
back
either
to
this
state
or
to
her
previous
state
before
starting
to
processing.
B
Okay
and
now
that
we're
here
we
can
move
the
pipeline
to
James,
so
James
will
reproduce
Sara's
State
in
another
location
and
continue
with
building
the
model
and
training
it
so
Sara's
done
huge
James
James
is
on
a
completely
different
zone.
Completely
different
location.
Ab
doesn't
actually
share
anything
with
Sara.
B
They
don't
use
a
single
s3
as
a
data
length,
exchange
data,
say
James
won't
go
to
Sara
to
find
the
data
he
needs.
The
data
has
already
been
synchronized
locally
for
him
by
our
software,
so
this
is
James's
view
of
what's
going
on.
This
is
the
sarah
bucket.
It
already
contains
data
on
his
side
because
our
software
took
care
of
synchronizing.
It
here
is
more
information
on
this.
This
packet
Sarah
actually
has
two
connected
peers.
Two
occasions
one
is
James's
location,
one
B,
the
other
is
Sarah's
location.
B
If
three
or
four
or
five
people
or
five
locations
were
involved,
you'd
see
five
items
here
underneath
we
do
peer
to
peer
synchronization
of
these
buckets,
so
you
can
have
a
pipeline
that
extends
to
multiple
locations
and
down
locations.
Synchronize
data
state
with
peer
to
peer
encrypted
connections,
so
James
already
has
the
data
he
needs.
This
is
the
day
that
he
needs.
This
is
the
latest
snapshot.
B
So
James
can
start
from
the
latest
snapshot
get
to
his
instance
of
cook
flow
log
into
his
distance
of
cook
flow
and
pre
initialize,
the
environment
with
whatever
Sarah
gave
him.
So
what
I
did
was
copy
the
mink
to
Sara's
snapshot
from
our
software
and
give
it
to
cook
flow.
So
it
can
proceed
so
here's
what
change
will
start
from
James
will
start
with
two
CPUs
and
eight
gigabytes
of
memory,
because
we
maintain
all
metadata
for
the
environment.
B
So
it's
reproducible,
James
will
have
the
exact
same
environment
pads
on
kubernetes,
manage
button
flow.
James
will
have
access
to
Sara's
workspace,
so
she'll
see
the
exact
same
libraries
unchanged.
We'll
also
see
the
exact
same
data
right
before
Sara
took
the
snapshot.
Sara
can
continue
to
work
in
her.
Local
infrastructure
can
continue
to
produce
snapshots
and
James
can
continue
with
his
branch
of
the
environment.
So
this
is
essentially
the
combination
of
coop
flow
plus
rock.
Our
software
is
get
full
data
and
your
Holdeman
environment,
so
James
spawns
the
environment.
B
B
B
B
Where
the
notebooks
lead-
and
he
can
see
the
data,
this
is
our
four
gigabytes
of
data,
and
now
James
will
use
a
trainer,
a
training
module
in
Python
to
train
his
model.
This
is
the
training
module
and
it
needs
some
information.
So
James
would
just
use
the
local
file
that
he
conceived.
This
is
the
input
data
and
he
won't
be
doing
any
pre-processing
because
we've
already
done
it
doesn't
need
to
use
in
temporary
directories.
B
He'll
just
train
the
module
into
this
file
for
sample
size
of
200
issues,
because
the
sample
size
that
Sarah
gives
during
pre-processing
so
now
James
grant
runs
training
on
a
different
location
which
could
be
the
cloud
using
GPUs.
Maybe
we
have
extended
the
ML
pipeline
from
service
location
into
James's,
rotations
to
run
training,
put
different
hardware
or
maybe
at
scale
so.
B
B
Okay-
and
this
is
the
trend
model,
so
James
has
completed
training
and
then
how
can
this
continue?
Two
options:
James
could
spawn
a
distributed
job
right
from
within
the
notebook
using
the
same
primitive,
using
a
snapshot
of
his
environment
to
power
the
distributed
our
training
job,
so
he
can
submit
a
cook
flow,
tensorflow
job
object
to
start
a
distributed,
jug
or,
and
then
this
is
what
we'll
do
now.
B
B
B
A
Awesome
thanks
so
much
for
the
presentation,
so
we
do
have
some
time
for
Q&A
right
now.
So
just
a
reminder,
there
is
a
Q&A
box
at
the
bottom
of
your
screen,
so
please
drop
your
questions
in
there
and
then
we'll
get
to
as
many
as
we
have
time
for
so
a
few
questions
have
come
in
during
the
presentation.
How
is
ROK
authenticated
to
get
access
to
Sara's
volumes.
B
This
is
a
good
question.
Okay,
so
Rach
manages
Sara's
volumes.
Rock
is
integrated
with
kubernetes
as
what
it's
called
a
CSI
plugin.
So
whenever
our
coop
flow
requests
new
volumes,
it
asks
Rock
to
create
them
to
manage
them.
So
the
question
is
the
reverse:
how
does
the
Kukla
user
interface
the
Spooner
interface
this
interface
right
here?
Let
me
destroy
certain
environment
and
show
you.
B
The
question
is:
destroy
the
in
the
interface
anyway.
I
have
to
destroy
it
either
we
have
to
stir
them
in
order
to
show
you.
The
question
is:
how
does
the
crew
flow
user
interface
authenticate
itself
to
rock,
so
it
has
access
to
its
snapshot.
We
maintain
secrets
in
kubernetes.
Sara
maintains
our
own
secrets
in
kubernetes,
giving
access
to
whatever
tokens
Rock
needs
for
coop
flow
to
access
it.
So
the
very
first
time
Sara
will
access
the
control
environment.
She'll,
see
a
warning
message
that
says
this
could
flow
deployment
is
not
authenticated
access
Rock!
B
A
B
A
B
Okay,
so
the
question
is
Sarah
James
work
in
different
locations:
okay,
seven
James
work
completely
dependent
Sara
works
on
her
laptop
and
James
may
be
working
on
the
cloud.
Would
they
be
able
to
use
the
exact
same
block
storage?
Would
it
make
sense
for
James
to
to
have
the
impact
of
latency
to
go
all
the
way
to
service
laptop?
Would
he
be
able
to
go?
Do
it
the
service
laptop
to
actually
fetch
the
same
blocks?
B
So
if
seven
James
are
working
on
the
same
location,
that
can
actually
be
actually
the
exact
same
blocks,
but
if
you're
working
across
locations,
then
the
question
becomes:
does
your
code
Traverse
regions
to
find
its
data
in
another
data
leg,
if
you're
on
an
account,
if
you're
in
an
autonomous
in
an
autonomous
car,
do
you
actually
multiply
to
find
your
data?
This
example
and
our
approach
is:
let's
bring
the
data
set
close
to
where
the
compute
part
runs?
Let's
have
the
data
set
be
the
globally
accessible
resource
refer
to
it
and
synchronize
it
locally.
B
Whenever
they
snapshot,
they
don't
create
new
blocks
locally
write.
All
the
snapshots
locally
maintain
a
shared
set
of
blocks,
but
yes,
when
talking
about
distinct
locations,
because
we
actually
want
locations
to
be
independent
in
terms
of
security
and
speed,
we
maintain
two
distinct
sets
of
blocks,
one
for
location,
a
1/4
location,
B,.
B
I
didn't
show
it
we
have.
We
have
what
we
call
the
rock
registry,
the
single
hub,
where
all
locations
essentially
discover
one
another,
because
one
question
I
didn't
really
go
into
is:
how
did
Sarah's
bucket
get
to
be
synchronized
with
James's
Park?
How
did
the
bucket
Sara
on
Sara's
infrastructure
come
to
be
synchronized
with
budget
Sara
on
James's
infrastructure?
So
what
they
did
is
they
went
through
a
publish
and
subscribe
step
on
a
single
datum
cog
that
we
called
the
rock
registry.
B
So
on
the
rock
registry,
you
can
define
organizations
similarly
to
how
you
would
define
organizations
on
github.
You
can
share
your
your
buckets,
your
datasets
with
whole
organizations
and
then,
if
a
person
leaves
the
organization,
whatever
infrastructure
is
authorized
by
them
to
access.
This
data
is
automatically
thrown
out
of
the
synchronization
storms.
So
you
can
have
members
of
the
team
leaving
the
team.
B
You
can
have
new
members
of
the
team
joining
into
the
team,
and
then
the
peer
to
peer
synchronization
is
authentic
is
updated
to
take
into
account
your
changes
to
team
members,
and
then
this
is
also
a
very
good
comment
that
I
see
on
the
Q&A
panel.
Snapshotting
is
not
just
for
sharing.
Yes,
it's
for
versioning,
even
if
you're
local
somewhere.
B
If
you
maintain
multiple
snapshots
of
your
own
environment,
even
if
you
share
with
no
one
else,
even
if
you're,
using
just
a
single
local
data
Lake,
it
makes
sense
to
be
able
to
smash
up
your
environment,
because
this
is
essentially
an
immutable
deep,
like
timeline
to
your
work.
So
when
something
goes
bad
because
not
everything
happens
perfectly
with
the
very
first
trial,
you
can
go
back
in
time
and
reproduce
your
work.
So
sharing
this
immutable
timeline
across
data
centers
is
an
extra
point.
A
B
Rock,
so
for
each
individual
location
rock
needs
some
sort
of
persistent
store
for
long-term
archival
of
its
the
duplicated
blocks.
So
all
the
live
volumes
are
stored
on
kubernetes,
persistent
problems,
so
they're
stored
on
local
SSDs,
even
super
fast
that
can't
get
any
faster
than
that.
But
when
you
snapshot
rock
the
duplicates
your
data
and
needs
up
an
archival
place
where
it
stores
your
data
in
the
long
run-
and
this
is
different
for
each
location,
so
each
location
is
completely
dependent
from
other
locations
and
that's
how
you
can
scale
to
potentially
hundreds
of
locations.
B
That's
how
this
can
work
across
a
few
thousands
of
autonomous
cars
running
on
the
roads
right.
So
you
need
some
sort
of
local
storage
storage
local
to
the
region,
to
store
your
archive
blocks.
If
you're
on
Amazon,
we
use
whatever
local
s3
instance
you
have.
So
if
you
on
us
west
1a,
we
use
the
s3
region,
us
West,
one,
if
you're
in
another
region.
We
use
us
three
for
this
region.
If
you're
on
Google
Cloud,
we
use
Google
object.
Storage.
B
If
you're
your
laptop,
we
use
part
of
your
local
disk,
a
local
directory
if
you're
on
on-prem
kubernetes,
we
just
find
with
using
Mineo,
for
example,
as
an
s3
endpoint,
to
store
the
archive
data.
So
we
just
need
some
sort
of
local
or
tribal
endpoint
will
restore
that
the
duplicated
chunks
out
of
which
we
form
the
virtual
snapshots.
A
B
I,
like
the
questions
where
they're
very
focused
exactly
the
questions
we
would
like
to
talk
about.
So
Luke
is
a
great
project
for
orchestrating
storage.
Locally
Luke
can
create
a
safe
cluster
pretty
easily
locally.
So
you
can
ask
it:
I
need
safe
to
store
my
local
golems.
Okay,
so
Luke
will
give
you
a
local,
safe
plaster.
Then
you
can
have
local
problems.
So
part
of
our
work
with
coop
flow
is
getting
coop
flow
to
use
local
persistent
volumes.
So
in
this
case,
coop
flow
would
use
safe
volumes,
managed
by
Luke
to
store
the
data.
B
But
then
how
would
you
snapshot
these
volumes?
So
you
can
go
back
in
time.
You
could
use
safe
snapshots,
maybe
or
you
could
use
our
software
alongside
safe
volumes
to
maintain
these
snapshots,
because
once
we
maintain
these
snapshots
in
our
own
format,
then
we
can
take
these
snapshots
and
synchronize
them
in
a
peer-to-peer
network
across
tens
or
hundreds
of
locations,
because
if
you
use
route
to
manage
a
local
safe
cluster,
how
do
you
then
synchronize
the
content
of
this
safe
cluster
with
other
surf
casters?
B
You
may
do,
and
most
storage
technologies
do
this
kind
of
PA
of
point-to-point,
master-slave
replication
right.
You
can
have
a
primary
site
on
the
secondary
side
and
you
can
be
synchronizing
your
volumes
from
one
side
to
the
other,
but
this
essentially
means
that
the
two
sides
trust
one
another,
and
this
also
means
that
it's
not
the
user
who's
in
control
of
the
synchronization
process.
It's
the
administrator
who
has
set
it
up
explicitly.
B
Our
approach
is,
let's
make
the
data
a
kubernetes
managed
resource,
something
that
the
users
themselves
can
actually
refer
to,
and
then
let's
allow
the
users
to
specify
what
gets
accessed.
Where
ruk
manages
primary
storage.
We
do
not
target
primary
storage.
We
do
not
store
your
data.
So
if
you
take
a
look
at
this
part
of
the
presentation,
we
assume
that
there
is
some
way
to
store
data
locally
PVCs.
This
is
our
contribution,
can
flow
making
could
flow,
work
or
PVCs.
B
So
this
could
be
route
unsafe,
provided
PVCs
also,
and
this
could
be
route
any
storage,
and
then
our
software
sits
on
the
side
and
keeps
track
of
what
has
been
changed
and
maintains
immutable
snapshots
and
then
synchronizes.
These
snapshots-
and
this
is
the
the
this-
is
the
same
answer
as
with
port
works
hardwood.
This
compare
to
port
works.
Port
hooks
is
a
very
fast,
very
nice
way
of
managing
primary
storage
on
a
kubernetes
cluster.
But
then
how
do
you
move
from
one
location
to
another?
How
do
you
synchronize
locations
that
ran
different
kinds
of
storage?
B
A
B
B
So
whatever
you
can
do
from
the
user
interface,
you
can
do
programmatically
exactly
because-
and
this
is
our
next
step-
and
this
is
what
we
will
be
demoing
next-
is
running
an
automated
pipeline
system
like
OOP
flow
pipelines
or
our
go
over
our
snapshots.
So
you
can
have
two
visibility
on
whatever
the
output
of
one
step
is
and
whatever
the
output
of
another
step
is,
so
you
can
go
to
each
individual
step
of
a
pipeline
and
say
what
is
this
step?
Do?
B
Why
did
have
a
simple
and
what
it
produces
out
can
actually
mount
its
input
and
its
output,
so
I
can
very
easily
have
first-hand
experience
with
what
happened.
This
is
great
for
debugging,
by
the
way,
it's
full
visibility
into
all
steps
of
a
pipeline
by
being
able
to
retrace
whatever
every
step
did
by
mounting
it's
immutable
input
and
it's
snapshot
at
the
middle
block.
So
let
me
show
it.
This
is
James
inside
his
environment
and
he
can
actually
access
rock
programmatically
and
say.
B
Let
me
get
my
notes
for
this,
though,
and
he
can
say,
let
me
create
a
new
snapshot
of
my
environment
inside
the
James
budget
and
at
this
point
this
is
an
API
call
that
we
do
to
rock.
So
this
is
the
same
thing
that
the
user
interface
would
show,
but
in
a
text
r2
in
CLI
tool,
so
demoing
API
driven
snapshots,
okay,
just
taking
us
using
the
API
okay.
So
we
submit
this
to
kubernetes
and
again
I
see
a
light.
B
So
totally
automated
takes
a
snapshot
of
James's
environment
and,
if
we
go
into
so
this
is
happening
right
now,
right,
I,
switch
to
the
visual
interface
and
there's
a
new
task.
That's
actually
progressing,
so
I
used
the
API
here
to
create
a
new
task
programmatically,
and
this
task
is
what
is
running
here,
and
this
task
is
almost
done.
This
task
is
almost
almost
done.
This
task
is
done
and
yes,
this
task
is
done
here.
The
emitter
boolean's
I
can
use
to
access
my
snapshot,
and
this
is
it.
B
A
Awesome,
thank
you
and
thank
you
all
for
those
great
questions
that
brings
us
to
the
end
of
our
time
today.
So
once
again,
thank
you
so
much
Vangelis
for
the
great
presentation
in
to
all
of
our
attendees.
Just
as
a
reminder,
the
webinar,
recording
and
slides
will
be
online
later
today,
I
mean
we
look
forward
to
seeing
you
all
in
the
future
ciancia
webinar
thank.