►
From YouTube: Operating Deep Learning Pipelines Anywhere Using Kubeflow - Jörg Schad & Gilbert Song, Mesosphere
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Operating Deep Learning Pipelines Anywhere Using Kubeflow - Jörg Schad & Gilbert Song, Mesosphere
Kubeflow makes it very easy for data scientist to build their own data science pipeline with Jupyter Notebooks, TensorFlow, TensorBoard and Model serving. In this talk we will walk through building a production grade data science pipeline using Kubeflow and open source data, streaming and CI/CD automation tools. Audience will learn about need for data preparation (which is frequently performed using Apache Spark or Apache Flink), data storage (using HDFS, Cassandra), automation via CI/CD (using Jenkins) and request streaming (using Apache Kafka). In this talk we look at building and operate a complete deep learning pipeline around Kubeflow for multiple tenants and topics such as: * Data Preparation/Cleansing (using Apache Spark) * Data and Model Storage * Model Serving * Distributed Training * Monitoring * Automation using CI/CD * Infrastructure Management across multiple tenants
To learn more click here: https://sched.co/FuJt
A
Nihao
and
welcome
to
our
talk
building
an
open-source
data
science
platform
based
around
cute
flow,
so
just
a
quick
raise
of
hands
who
has
to
use
tensor
flow
before
tensor
flow,
all
right,
I,
guess,
half
the
room,
so
this
talk
is
basically
about.
How
can
we
move
from
using
tensor
flow
to
words
using
Q
flow
to
what's
actually
building
a
complete
data
science
pipeline
with
all
the
features
from
data
over
to
models
serving
over
a
notebook
sharing
and
everything
else?
A
B
You're
gonna
introduce
me
no
okay,
Gilbert
I'm
relating
the
containerization
teen
atmosphere
and
Apache
Masons,
PNC
and
committee.
Basically,
today's
talks
about
as
well
how
to
even
run
Q
everyone.
How
do
we
run
keep
pro
on
kubernetes
on
mates
and
majors,
so
yeah
first
couples
like
an
interest
like
why
we
do
we
want
to
do
machine
running
and
what's
the
problem
we
are
facing
and
what's
the
potential
solution
and
potential
future
roommate,
we
can
have
so
yeah,
it's
already
like
the
second
decade
of
the
21st
century.
B
B
So
it's
too
painful
for
data
scientists,
because
they
might
probably
focus
on
algorithm
to
focus
on
like
machine
learning,
Co
and
then,
given
the
data
set,
they
should
be
supposed
to
train
the
model
and
then
come
up
with
a
solution,
and
but
the
problem
here
is
like
if
they
want
to
scale
up,
there
are
a
lot
more
problem
than
interest
off.
They
have
to
manage
the
cluster,
they
have
to
schedule
the
resources
from
the
cluster
GPU
CPU
and
memory,
and
they
have
to
do
the
monitoring
on
their
tasks
so
yeah.
This
is
just
another
slide.
B
So
as
an
example
like
we
cut
data
everywhere
from
our
life,
for
example
like
auto
driving,
we
get
GPU
data,
we
got
sensors,
we
got
photos
from
the
other
car
so
and
then
I
come
from
each
of
the
vehicle,
and
we
need
the
data
sender
to
do
with
with
all
those
data
and
then
then
assigns
could
come
up
with
resolution
like
oh,
this
is
the
user
behavior.
We
should
improve
the
way
we
ought
to
peel
out
the
card
and
we
need
to
improve
the
way
like
make
the
life
easier
for
the
drivers.
B
A
B
So,
okay.
So
that's
what
were
data
scientists
supposed
to
do
they
supposed
to
focus
on
like
imperative,
machine
learning,
Co
and
given
the
data
sent
no
matter,
it's
like
large
set
or
not,
they
should
be
able
to
change
the
data
for
the
model
and
then
they
should
be
able
to
run
the
model
and
then
they
might
want
to
improve
the
code
and
then
do
it
again
and
again
and
then
finally,
they
have
a
precise
enough
model
for
for
their
data
training.
B
So
this
is
I,
do
but
not
realistic
at
this
moment,
because
we've
got
a
lot
of
challenge
for
cluster
management,
so
yeah.
This
is
a
is
a
hidden
technical
challenge
from
machine
learning
system.
So,
as
you
guys
can
see
like
only
the
central
Bret
part,
it
is
the
machine
learning
and
for
data
scientists
yeah.
B
So
that's
what
we
are
doing
with
cube
Pro
and
this
light
you
guys
might
already
see
this
slide
earlier
this
morning
in
the
keynote
from
chakra
and
I
was
surprisingly
realized
that
oh,
they
have
the
same
slices
and
for
this
one
and
basically
it
is
a
good
thing
to
me
at
least
it
is
a
good
thing.
It
means
like
in
the
industry,
in
machine
learning
in
AI,
different
company,
different
organization
and
open
source
project.
We
realized
that
similar
problem.
B
We
want
to
resolve
this
problem
and
that's
it
and
we
get
started
from
here
and
then,
coincidentally,
we
fry
the
same
solution.
Cue
Pro
here,
yeah
I-
think,
as
you
guys
can
see
like
each
of
this
prompt.
It
might
not
be
easy
resource
management,
no
matter
kinetics
or
measures.
We
have
different
with
resource
scaling
strategy
and
people
might
need
to
deal
with
some
low-level
things
like
GPU
topology.
To
improve
the
performance
then
might
need
to
deal
with
like
the
continued
runtime,
potentially
the
prevalence,
Muslim
or
isolation
for
data
science.
B
Data
scientist
and
those
may
not
be
the
things
necessary
for
data
scientists
to
understand
we
want.
We
expect,
like
data
scientists,
just
focus
on
what
they
should
focus
on
on
just
focus
on
like
developing
the
best
algorithm
for
machine
learning
and
then
change
the
model.
That's
it
okay.
So
we
come
up
with
a
solution:
q
pro
so
Q
pro
unified,
different
training
model,
and
then
it
come
up
with
the
pipeline
with
different
parts
of
this
training
distributed
system.
B
This
is
a
quick
definition
about
q
pro
for
any
for
folks
who
never
touch
q
felt
before
so.
Basically,
a
q
proof
project
is
dedicated
to
make
deployment
of
machine
learning
workflow
on
kubernetes,
simple,
portable
and
scalable,
so
yeah
scallop
scalability
is
always
the
biggest
problem
for
assistant
engineer
for
this
building
season
is
never
easy,
so
you
might
hurt
a
lot
of
story
from
those
becoming
like
Alibaba,
like
10,
like
Tenzin
and
many
other
companies
like
Google
Twitter.
B
So
they
scale
up
to
like
hundred
of
thousands
of
machine
and
even
1
million
machine,
but
a
lot
of
work.
They
might
have
like
1,000
engineer
team
to
scout
that
up
to
improve
the
kernel
to
improve
the
core
component
of
their
scarran's
scheduling
system,
and
then
they
might
also
do
a
lot
of
limitation
on
the
schedule
to
improve
the
performance.
So
we
more
a
lot
more
discussion
about
scalability
but
yeah.
B
A
First
of
all,
couplet
was
started
actually
last
year
and
just
yesterday
they
announced
the
ODA
3
release.
So
congratulations
to
the
entire
queue
flow
community.
It's
really
great
to
see,
and
it
was
really
cool
to
see
the
progress
I've
been
making
towards
that.
This
is
also
another
talk
from
yesterday
from
Aaron
and
Fay
about
this
last
year
of
cube
flow.
Basically,
and
anyone
who's
interested
in
cube
flow
I
would
really
recommend
watching
the
recording
later
on,
because
real
focus
in
this
talk
will
focus
a
bit
like
on,
what's
happening
around
will
not
cover.
A
So
much
of
see,
details
of
cue
flow
itself
more
like
what
are
best
practices
around.
What
do
I
need
to
build
like
a
scalable
data
science
platform
and
operates
that
in
general.
So
just
a
quick
question,
how
many
of
you
have
seen
this
image
or
actually
read
that
paper?
So
this
is
tensor
flow
extended,
and
this
was
kind
of
see
it's
kind
of
the
motivation
or
kind
of
used
to
be
the
guiding
principle
for
a
cue
flow.
A
So
this
is
about
like
how
Google
this
machine
learning
and
it's
about
the
entire
pipeline
and
a
lot
of
different
pieces
in
the
tensor
flow
ecosystem
have
been
actually
developed
from
this,
and
just
cue
flow
actually
has
evolved
further
from
this
goal
or
original
motivation.
So
if
we
look
at
it
today-
and
this
is
what
I
found
really
amazing-
over
just
one
year-
we
actually-
we
are
not
just
focused
on
tensor
flow
itself
anymore.
We
also
have
pi
torch
in
there,
there's
also
some
MX
net
support.
We
have
hyper
parameter
optimization.
A
We
have
a
lot
of
other
ecosystem
projects,
stepping
in
here
like
Seldon
AI
for
serving
model
DB
being
used
to
store
metadata
for
models.
So
it's
actually
not
just
tensorflow
anymore.
It's
going
back
to
this
original
goal.
We
saw
earlier
about
becoming
the
default
kind
of
pipeline
to
orchestrate
and
deploy
machine
learning
pipelines
everywhere,
still,
even
if
we
have
this
pipeline.
A
In
my
opinion,
cube
flow
works
really
well
if
I
am
I'm
in
a
cloud
environment
if
I'm
on
Google
cloud,
if
I'm
on
Amazon,
if
I'm
on
Alibaba
cloud,
this
works
really
well
because
they
are
still
some
hidden
pieces
missing.
So
if
I'm
in
a
cloud
I
actually
have
access
to
cloud
storage,
so
I
can
store.
My
data
sets.
I
can
store
my
model
somewhere
in
a
persistent
fashion.
I
typically
have
access
to
some
kind
of
way
to
do.
A
A
batch
data
pre-processing
I
have
some
kind
of
stream
systems
where
I
can
actually
stream
my
requests
and
while
I'm
in
a
cloud
environment
I,
can
simply
use
all
of
that,
not
necessarily
for
free
but
I
as
a
developer.
I
don't
necessarily
have
to
care
about
that.
Now
many
of
our
customers,
many
of
our
users,
especially
in
Europe
or
environments,
which
are
like
constraint,
which
data
can
be
uploaded
to
the
cloud
they
actually
try
to
build
something
similar
in
their
own
data
center.
So
there
are
a
lot
of
open
source
tools
available.
A
A
What's
my
serving
layer,
so
it's
just
I
all
the
sudden
once
I
try
to
build
that
in
my
own
environment,
so
in
an
on-prem
cluster
I
start
to
have
to
consider
all
those
different
components
missing
which
I
just
get
for
free
in
a
cloud
environment
here.
So
in
the
end,
so
the
typically
like
the
advanced
picture.
So
this
is
what
we
see
people
building
over
years
and
most
likely
not
having
all
of
that,
but
let's
just
simply
go
through
what
different
steps
are
actually
needed
in
such
kind
of
pipeline.
A
What
I
really
liked
in
the
keynote
this
morning,
it
touched
up
on
some
points
which
are
very
dear
to
me,
especially
its
the
challenges,
so
you
might
actually
recognize
some
of
those
points
from
this
morning's
keynote.
So
it
all
starts
with
like
data
and
streaming
I
need
to
have
access
to
data
sets
at
best,
I
have
like
versioned
access
to
data
sets.
There
are
a
number
of
companies
actually
trying
to
get
better
support.
Sir.
A
A
Data
set,
many
systems
just
have
streaming
data,
and
so
I
could
have
like
a
Kafka
stream
of
data
which
I
might
actually
also
use
snapshots
for
for
training,
and
so
this
is
kind
of
the
first
a
basis
layer,
but
it's
really
important
and
we'll
come
back
to
some
more
details
here
in
a
second,
the
second
part-
and
this
is
where
my
data
scientist
really
wants
to
focus
on
this-
is
the
model
engineering.
This
is
the
data,
preparation
and
model
building,
so
this
is
really
typically
Jupiter.
A
How
many
of
you
know
Jupiter
as
a
project
or
have
used
it?
Some
for
those
of
you
who
haven't
I
would
really
recommend
it.
It's
a
really
nice
way
to
get
started
with
data
science
without
having
to
worry
about
too
many
of
the
details,
and
so,
as
mentioned,
this
is
like
where
my
data
scientist
he
can
go
in.
He
can
prepare
that
data
set.
He
can
extract
features
and
then
the
next
layer
is
actually
modeled
training
model
trainings.
A
This
is
really
where
most
of
the
resources
GPUs
TPU
CPUs
are
flowing
in,
where
I
have
different
hyper
parameter
runs,
and
this
is
something
where
typically
more
like
dev
ops.
People
should
come
in
because
they
actually
understand
distributed
training
and
distributed
computation
once
I'm
done
with
training.
I
have
a
of
models
which
I
need
to
store
and
I
not
only
need
to
store
the
models,
but
even
or
equally
important.
I
also
have
to
store
the
metadata
of
those
models.
How
what
was
the
Train
accuracy?
A
What
was
the
validation
accuracy
to
all
those
different
statistics,
because
here,
I
later
on
from
the
model
management
layer,
I
actually
have
to
select
the
models?
I
want
to
serve
for
production
workloads,
and
that
might
depend
on
questions
like
give
me
the
best
performing
model
for
a
GPU
environment,
which
takes
less
than
of
0.5
seconds
to
do
inference
so
I
need
a
lot
of
this
metadata
to
actually
choose
the
different
models.
I
want
to
serve
later
on
in
production.
A
So
this
is
the
simple
pipeline
walkthrough,
but
around
that
we
have
a
lot
of
other
pieces
already
this
morning,
we,
it
was
mentioned
that
we
need
2
different
kinds
of
monitoring
systems.
The
first
one
is
kind
of
fuzzy
models
for
the
data
science
or
data
scientists
themselves,
and
this
is
focus
yeah
on
the
model.
It's
on
the
model
accuracy-
and
this
is
typically
something
like
cancer
board.
Secondly,
as
this
is
becoming
a
fairly
complex
architecture,
we
also
need
some
systems
to
actually
monitor
this
architecture
itself.
A
So
if
I'm,
building
a
fraud
model,
for
example,
I
only
care
about
that
I
have
some
representation
of
a
user,
and
this
can
actually
be
shared
amongst
different
data
scientist,
and
this
is
why
a
component
such
as
a
feature
feature
catalog
or
other
systems.
They
also
call
it
features,
stores
become
really
important
because,
as
mentioned,
if
I
invest
sixty
to
seventy
percent
of
my
data
scientists
time
in
developing
those
features
and
cleaning
the
data
sets.
This
is
something
which
should
be
shared
between
different
data
scientists.
The
second
part
is
actually
all
this
work
here.
A
A
If
we
want
to
deploy
the
models
later
on
into
production,
so
having
some
kind
of
CI,
CI
c
deep
pipeline
here,
really
is
taking
off
a
lot
of
load
from
the
data
scientist
to
have
to
train
that
manually,
and
it
also
which
we'll
see
in
a
second
helps
us
with
reproducibility
the
other
piece,
which
is
we
already
mentioned
like
so
stupid
in
notebooks.
They
are
very,
very
helpful
and
very,
very
important,
so
I
also
want
to
share
those
notebooks
between
different
people
and
so
something
like
binder.
A
Those
of
you
who
haven't
seen
binder
it's
a
really
cool
open
source
project
which
basically
allows
me
to
store
and
I
PI,
some
notebooks
or
a
jupiter
notebook
to
github
and
then
simply
say,
gives
that
link
to
anyone
else,
and
they
can
simply
run
it
without
having
to
spin
up
any
infrastructure
binder
or
they
are
also
like
open
source
projects
running
that,
like
called
my
binder,
for
example,
or
google,
also
has
something
very
similar.
They
give
you
an
executable
form
of
your
jupiter
notebook.
A
I
can
just
sent
it
to
Gilbert
here,
and
he
doesn't
have
to
set
up
Jupiter
itself.
He
can
just
run
that
from
this
web
link,
which
is
really
cool.
The
other
thing
I
actually
really
want
to
share,
are
my
trained
models
so
having
something
like
tensorflow
hop
or
a
model.
Library
is
really
useful.
First
of
all,
for
sharing
sea
trained
models,
but
also
later
on,
for
transfer
learning,
so
transfer
learning
I
would
go
in
I,
take
a
fully
trained
model
like
imagenet,
or
something
like
that.
A
I
go
in
I,
take
away
the
last
few
layers
and
then
I
specialized
that
to
my
specific
data
set,
but
using
this
using
this
preacher
a
model,
it
really
reduces
the
amount
of
time.
I
and
resources
I
have
to
invest
into
training,
and
it
also
reduces
the
data
set
size.
I
actually
need
to
train
that,
because
most
of
the
base
features
already
been
covered
by
this
pre
trained
model
here
so
having
a
platform
where
I
can
actually
share
and
reuse
models,
retrain
models,
it's
also
quite
important.
A
Last
underneath
we
actually
need
something
to
manage
all
those
resources
to
manage
all
those
different
distributed
systems.
So
if
we
just
count,
we
got
like
Kafka
Cassandra
HDFS
Jupiter
is
more
like
a
container
service,
but
here
we
got
10
to
flow
SPARC
PI
torch,
so
we
actually
end
up
with
a
large
number
of
different
distributed
systems
which
we
all
have
to
manage,
and
this
is
really
hard
for,
like
a
DevOps
person
to
set
all
this
up
in
just
a
manual
fashion.
A
B
Right,
yes,
just
one
quick
one,
let
me
talk
about
like
the
challenge
of
like
having
different
like
machine
learning
framework
did
learning
framework
running
on
the
same
cluster.
We
do
have
a
customer
like
surfer
from
like,
oh
when
they
realize
they
want
to
run
spark
which
is
like
best
John
and
Grady
framework
and
then
they're
on
to
110
serpro,
while
they
also
want
to
run
the
Cassandra
and
Kappa
on
the
same
cluster
and
head
and
then
have
one
single
cluster
to
manage
all
those
distributed
system
and
machine
learning.
B
Workflow,
and
that's
really
painful-
for
at
least
for
data
scientist
to
manage
the
whole
cluster,
because,
like
some
some
of
the
tasks
they
are
quitting
and
then
they
will
occupy
other
resources
and
for
data
scientist,
it's
hard
really
hard
for
them
to
realize.
Oh,
we
need
to
introduce
more
like
scheduling,
strategy,
to
make
it
more
scalable.
Actually.
A
This
is
what
the
next
slide
is
about,
see
data
scientist
himself.
He
wants
to
be
the
superhero
writing
really
cool
code,
and
he
shouldn't
have
to
worry
about
distribution
or
distribution
strategies.
He
should
really
be
able
to
focus
on
his
models,
and
this
is
why
we
typically
see
different
personas
being
introduced,
and
so
he
actually,
he
needs
a
sidekick
helping
him
to
be
still
be.
This
really
cool
superhero,
and
this
is
typically
like
today.
A
A
Typically,
this
is
what
most
people
underestimate
if
they
start
with
that.
First,
amnesty
I,
suppose
everyone
who
is
started
with
tensorflow
has
probably
done
like
this
amnesty
example
right,
I
download
the
pre
clean
data
set
I
go
in
there
and
I
can
directly
train
my
model,
but
in
reality,
unfortunately,
this
is
not
the
case.
A
So
demo
data
sets
are
a
fortunate
exception
here,
but
typically
I
really
spend
a
large
amount
of
my
time
as
a
data
scientist
and
preparing
my
entire
data
quality
and
the
kind
of
promise
of
deep
learning,
especially,
is
that
everything
is
being
derived
from
the
data
itself.
So
just
keep
in
mind
if
your
data
is
of
bad
quality,
your
model
will
be
of
bad
quality
as
well.
The
other
important
lesson
is,
and
I
actually
seen
that
at
a
user
site
is,
if
you
update
your
procedures,
your
pipeline
to
do
the
data
preparation.
A
You
also
need
to
update
the
same
procedure
to
your
real-life
data
coming
into
the
serving
layer
here.
So
actually
what
those
people
did.
They
came
up
with
a
really
cool
way,
a
new
way,
so
they
had
an
existing
pipeline
running
in
production
and
then
they
stepped
in
and
improved
the
way
they
were
cleaning
and
preparing
their
data
sets.
Their
models
had
much
quality,
much
better
accuracy,
but
once
they
deployed
him
to
production,
it
had
really
bad
quality.
What
happened?
A
They
didn't
update
the
data
cleaning
procedures
in
the
serving
environment
for
the
live
production
data,
so
that
was
still
kind
of
the
old
data
format
which
obviously
didn't
match
those
two,
those
new
models
here,
and
so
this
is
actually.
This
was
a
sly.
I
was
looking
for
earlier
here.
This
is
actually
this
division
of
labor.
A
Maybe
he
needs
to
do
like
some
model
monitoring,
but
overall
he
can
really
focus
on
being
a
data
scientist
and
most
data
scientists
I
know
they
also
they
have
more
like
a
mathematical
background,
so
this
is
what
they
like
do
it,
and
now
we
have
those
other
people,
like
the
data,
engineer,
data,
ops,
engineer
and
they
can
actually
help
with
all
the
stuff
around
which
often
also
requires
more
knowledge
about
distributed,
computation
and
yeah.
This
is
so.
This
is
kind
of
this
rise
of
sis
data
off
Stata
engineers.
A
The
people
I
seen
it's
really
really
hard
to
find
them,
but
they
are
combining
two
key
skills,
first
of
all,
date
of
science
and
secondly,
they
actually
understand
distributed
systems
fairly.
Well,
because
this
is
the
challenge
they
have
to
match
this
world
of
data
science
with
the
distributed
systems,
engineering
quiz,
actually
controlling
all
those
different
systems.
We've
seen
earlier,
if
you
want
it's
kind
of
like
this
DevOps
person,
but
for
like
data
science
in
particular,.
A
This
is
actually
Z
of
the
original
paper.
It's
the
first
publication
on
software
engineering.
There
was
a
mentioning
before,
but
it
actually,
software
engineering
is
something
where
I
feel
we
can
learn
a
lot
for
data
science.
I
feel
data
science
right
now
is
in
a
state.
Software
engineering
was
like
around
nineteen
forty
sixty
eight,
where
everyone
was
doing
stuff
very
ad
hoc.
Everyone
was
trying
to
figure
out
his
or
her
own
ways
of
doing
things,
and
so
I
would
actually
post
that
question.
Do
we
need
something
similar
for
data
science?
A
Do
we
need
something
like
data
science,
engineering
principles,
so
a
systematic,
quantifiable
approach
to
the
development
of
models
and
also
then,
the
testing
of
models
and
everything
around
which
we
actually
have
learned
in
software
engineering
and
people
like
Ian,
good
fella,
for
example?
They
also
have
similar
opinions.
So
this
is
what
we
are
currently
working
on
in
the
Bay
Area
with
other
big
ml
users
such
as
uber
and
other
big
companies
doing
ml,
which
names
we
are
not
allowed
to
mention
on
stage.
I
always
feel
this
first
challenge.
A
If
we
look
at
software,
engineering
is
like
requirements,
engineering.
How
do
I
come
up
with
the
requirements?
I
need
before
I
start
developing
my
model
and
there
having
talked
to
a
lot
of
people.
The
most
important
question
someone
should
ask
if
he's
told
to
develop
like
a
deep
neural
network
is,
is
that
does
it
actually
make
sense?
I
currently
feel
that
especially
neural
networks?
They
are
very,
very
powerful,
but
they
often
we
apply
them
to
problems
where
they
don't
really
match.
A
So,
especially
if,
for
example,
if
I
have
a
small
data
set
which
might
not
be
sufficient
to
train
an
entire
neural
network,
if
it's
a
bad
data
quality
or
actually,
my
function
is
also
pretty
similar.
Something
like
support,
vector
machine,
so
even
like
logistic
regressions
can
be
very,
very
powerful,
so
we
shouldn't
just
blindly
use
deep
neural
networks,
because
this
is
what
our
CTO
has
just
read
about.
A
Other
requirements,
I
should
gather
is
like
what
is
actually
my
target
and
serving
environment
will
I
serve,
sat
in
a
car
where
you
need
likes
ups
second
response
times
will
I
serve
that
in
a
large
cluster
way.
I
have
a
lot
of
GPUs
available.
So
what
is
kind
of
the
resources?
I
have
in
my
serving
requirement
where
later
on,
I
will
use
that
model
and
also
what
are
the
guarantees.
So
what
timing
guarantees
do?
I
have
to
have
other
questions.
Are
there
pre-trained
models
available,
I
believe
most
of
us
shouldn't?
They
actually
develop
new
models.
A
People
like
Google
or
research
have
actually
come
up
with
a
lot
of
really
cool
model
architectures
and
there's
a
lot.
We
can
actually
reuse
here.
The
next
biggest
challenge-
and
this
is
where
I
see
most
companies
actually
fail.
Is
reproducible
builds
still
data
science
right
now,
it's
a
very
ad
hoc
process.
Data
scientist
goes
in
he's
playing
with
the
data
he's
playing
with
this
model.
Once
the
accuracy
looks
good,
he
goes
over
to
the
dev
ops
engineer
or
some
kind
of
ops
person
and
tells
them
here's
my
model.
A
They
give
me
a
way
to
get
an
artifact
of
my
data
of
my
entire
pipeline
flow,
so
I
can
actually
go
in
and
reproducibly
train
that
the
other
aspect
which
I
find
really
important
here,
is
again
the
continuous
integration
part
because
it
actually
forces
me
to
create
a
reproducible
artifact.
So
my
if
the
entire
training
process
is
being
taken
care
of
by
my
CI
CD
pipeline
I
actually
have
to
create
a
reproducible
artifact,
because
otherwise
it
cannot
be
trained.
A
Just
real
brief,
as
probably
more
people
are
familiar
with
cube
flow
pipelines.
Ml
flow
m/l
flow
is
a
project
by
data
breaks,
the
guys
behind
spark
and
the
sub
project
here
I
like
most
is
actually
the
project's
part,
because
it
gives
me
a
packaging
format.
It's
very,
very
simple,
but
it
gives
me
a
packaging
format
for
reproducible
runs
on
any
platform,
so
I
can
go
in
and
just
retrain.
My
model
q
flow
pipelines
to
something
similar,
but
I
said
you
can
use
both.
A
It
probably
depends
a
little
on
also
the
surrounding
parts,
and
so
then-
and
this
is
again
coming
back
to
this
division
between
different
personas
as
well.
It's
again
so
here
we
have
the
data
scientist
up
here,
so
he
can
deal
with
this
dataset.
He
can
use
his
Jupiter
notebook
to
explore
his
data
to
come
up
with
a
model
and
once
he's
done
he's
actually
currently,
this
is
typically
a
gift
commit,
but
once
he's
done,
he
doesn't
get
commits
from
his
Jupiter
notebook,
and
this
is
actually
triggering
the
C
ICD
pipeline.
A
A
So
this
is
for
those
of
you
who
haven't
seen
Jupiter.
This
is
kind
of
what
Jupiter
looks
like.
So
this
is
the
first
view
if
I
go
in
and
actually
I
can
simply
in
the
web.
Browser
I
can
write
on
my
Python
code.
I
can
write
my
Scala
code
for
SPARC.
I
can
write
Java
code,
so
I
don't
have
to
set
up
my
own
environment
and
I
can
then
also
execute
it
from
there.
I
can
export
those
notebooks
I
can
share
them.
A
Catalog
is
really
useful
to
share
that.
So
it's
kind
of
like
a
pre-processing
cache
I
can
even
cache
the
prepared
data
sets
and
then
also
discovery,
components
that
other
people
can
find
and
reuse
that
and
a
a
really
highly
recommended
resource
is
it's
called
Michelangelo
by
uber,
so
they
actually
in
their
blog
post.
They
talked
a
lot
about
this
feature.
Catalog
other
thing
again,
our
model
libraries.
So
in
normal
software
engineering
we
deal
a
lot
with
libraries.
A
We
should
do
something
similar
in
data
science,
so
we
can
reuse
pre-trained
data
sets,
we
can
use
pre
trained
and
embeddings.
We
don't
have
to
come
up
with
everything
from
scratch.
We
don't
have
to
train
everything
from
scratch.
So
also
this
is
something
I
seen
organizations
set
it
up
in
their
organization
and
simply
share
all
this,
and
it
really
boosted
the
productivity
and
it
also
reduced
the
amount
of
resources
they
had
to
spend
on
compute
GPUs
CPUs
I
got
one
minute
all
right,
then
let
me
jump
through
here.
A
Let
me
see
all
right,
hyper
parameter,
optimization,
also
very
important,
but
there
was
actually
this
is
being
covered.
All
of
the
previous
slides
up
covered
in
the
Q
flow
talk
itself.
One
other
thing
I
would
really
highly
highlight,
and
everyone
should
think
about
is
model
optimization,
so
once
I've
trained,
my
model
I
actually
need
to
go
in
and
optimize
it,
for
example,
for
different
serving
environments.
A
If
I
want
to
have
a
neural
network
running
on
a
mobile
phone,
it's
the
normal
output
of
tensorflow-
probably
it's
going
to
be
way
too
large
and
way
too
slow,
so
I
can
go
in.
There's
like
there
are
different
transform
steps,
but
this
is
something
you
should
really
consider,
and
you
can
also
optimize
that
for
different
serving
environments,
you
might
produce
one
model
which
is
running
on
CPUs,
one
which
is
running
on
GPUs
and
maybe
one
which
is
running
on
your
mobile
phone
challenge
challenge.
This
is
actually
what
Gilbert
mentioned
earlier.
A
A
Alright-
and
this
is
basically
leading
to
this
entire
pipeline-
if
you
have
any
more
questions,
feel
free
to
either
talk
to
us,
or
we
also
have
like
a
white
paper.
You
can
download
from
the
slides,
which
is
going
in
some
more
details.
I
would
actually
love
to
close
with,
like
my
wish
list
of
for
the
next
queue
flow
iterations.
A
As
yesterday
we
announced
like
the
odor
3
release,
and
the
roadmap
also
looks
pretty
cool.
First
of
all,
it's
really
better
see.
Icd
integration,
seeing
what's
happening
with
Argos
CD
is
cool,
but
it
should
really
be
easier
to
integrate
with
different
CD
environments.
I
should
be
able
to
share
my
notebooks
with
other
people
and
what
I
also
believe
in.
We
actually
need
a
common
metadata
layer
having
different
components,
storing
the
metadata,
for
example,
for
the
like
model
dB.
A
We
also
have
metadata,
which
we
gather
from
experiments
so
I
feel
having
a
common
metadata
layer
across
this.
Entire
infrastructure
is
going
to
be
so
helpful
in
sharing
that
data
and
being
reproducible
lasting,
but
that
was
also
announced
yesterday
is
actually
like.
A
government
ends
model
which
they
are
currently
considering
so
really
to
have
a
bigger
community
around
that,
because
I
feel
cube
flows
really
helping
us
and
having
a
reproducible
standard
way
to
set
up
machine
learning
pipelines.
Thank
you
so
much.