►
From YouTube: Managing Machine Learning in Production with Kubeflow and DevOps - David Aronchick, Microsoft
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Managing Machine Learning in Production with Kubeflow and DevOps - David Aronchick, Microsoft
Kubeflow has helped bring machine learning to Kubernetes, but there’s still a significant gap relative to how to productize these workloads. While DevOps and GitOps have made huge traction in recent years, many customers struggle to apply these practices to ML workloads.
This talk will focus on ways to effectively infuse AI into production-grade applications through establishing practices around model reproducibility, validation, versioning/tracking, and safe/compliant deployment.
We will demonstrate how to run an E2E machine learning system using nothing more than Git. This will integrate DevOps, data and ML pipelines together, and show how to use multiple workload orchestrators together.
While the examples will be run using Azure Pipelines and Kubeflow, we will also show how to extend these platforms to any orchestration tool.
https://sched.co/MPar
A
Hi
I
apologize
I
was
not
working
on
my
slides
one
minute,
I'm,
not
one
of
those
presenters,
but
I
was
very
late,
so
I
apologize.
Thank
you
so
much
I
am
David.
Ron
chick
I,
lead,
open
source
machine
learning,
strategy
and
ml
at
Microsoft
and
Azure
and
I
was
previously
the
lead
p.m.
for
kubernetes
and
I
helped
start
the
Q
flow
project
and
I'm
here
to
talk
to
you
about
how
to
bring
your
machine
learning
to
production
using
Q
flow
and
mo
ops.
A
So
at
Microsoft,
because
the
widget
is
not
working,
all
right
I
will
be
operating
from
my
laptop
and
Microsoft's.
We
do
have
a
lot
of
experience,
bringing
ml
to
production.
We
bring
together
your
data,
we
bring
together
cloud
your
models
and
and
our
job
is
really
to
help
customers,
large
customers,
small
customers,
whatever
it
is,
help
you
move
through
and-
and
this
is
something
that
we
have
a
lot
of
experience
doing.
A
We've
have
a
lot
of
internal
experience
around
mic,
Microsoft
Research
and
an
ml
generally
with
many
of
the
most
recent
benchmarks
and
achievements
from
rimmel
coming
out
of
Microsoft
Research.
We're
really
really
proud
of
that,
and,
of
course
we
do
give
all
those
back
to
the
research
community
in
the
form
of
open
papers
and
notebooks
and
data,
and
the
reality
is
that
ml
does
touch
every
aspect
of
Microsoft.
A
You
know
a
hundred
and
eighty
million
office
users
today,
every
day
use
often
a
I
features
in
office.
We
have
18
billion
queries,
asked
of
Cortana,
which
is
obviously
rich
NLP
and
other
things
and
six
point.
Five
trillion
security
events
evaluated
every
day.
There's
simply
no
way
that
we
could
operate
and
and
process
this
data
without
something
like
machine
learning.
A
But
it's
not
it's
about
the
data
processing
and
cleaning
and
all
the
various
things
that
are
involved
and
actually
bringing
something
to
production,
because
that's
the
nature
of
machine
learning.
It
is
these
many
many
micro
services,
each
of
which
have
a
very
specific
functionality,
often
very
specific
tooling,
that
do
very
specific
things
well,
but
then
need
to
be
coupled
together
in
an
intelligent
way.
And
if
you
just
focus
on
the
model,
then
you're
gonna
be
in
trouble
and
I
know
what
you're
saying
you're
saying
your
data
scientists
and
you
don't
care
and
I.
A
Believe
you
a
little
bit
but
I'm
here
to
tell
you
you
actually
do
and
the
reason
is,
is
tweets
like
this
right
models
are
relatively
easy
to
build,
but
they
are
very
hard
to
roll
out
because
more
often
than
not
data
scientists
operate
in
a
way
that
they
are
familiar
with.
They
understand
their
tools
and
they
build
using
their
tooling
and
local
laptops
and
local
clusters,
but
they
don't
know
how
to
reach
out
and
roll
it
to
production,
and
the
reason
is
is
because
you
have
this
separation
right.
A
The
data
scientists
over
here
they're
trying
to
iterate
as
quickly
as
she
can.
She
wants
to
use
frameworks
and
tooling.
She
understands
she
wants
to
mix
and
match
tools.
The
the
absolute
latest
build
of
tensorflow
or
PI
Torture,
onyx
or
whatever
it
may
be.
You
know
somebody
from
Carnegie
Mellon
just
launched
a
brand
new
tool
around
reproducibility
and
it's
respectability
and
that's
she
should
have
the
capabilities
to
do
that.
She
also
wants
to
not
worry
about
management
cuz,
it's
just
her
laptop
shave.
You
know
something
goes
wrong.
A
She
can
flatten
it
and
restart
and
on
the
other
hand,
she
wants
unlimited
scale.
She's
got
a
paper
due.
You
know
on
Thursday
at
5:00
p.m.
she
wants
all
the
GPUs
in
the
world
in
order
to
achieve
the
results
she
needs.
On
the
other
hand,
you
have
the
sre
right
and
she
needs
consistency.
She
needs
observability.
She
needs
to
reuse,
tooling,
that's
already
been
approved
by
her
organization
because
they
have
support
and
things
built
in
and
she
needs
uptime.
If
the.
A
A
Now,
before
I
begin,
I
want
to
talk
about
what
the
eee
machine
learning
lifecycle
looks
like
now.
You
saw
a
bunch
of
like
boxes
connected
together
with
arrows
there,
but
the
main
things
that
I
think
we
need
to
identify
and
solve
for
in
an
edu
lifecycle
is
what
you
see
here.
First,
you
do
have
the
development
and
deployment
of
the
model,
then
you're
gonna
have
to
package
it
in
a
way
that
can
be
used
and
migrated
to
in
production.
A
You
want
to
validate
that
model
behavior
before
you
roll
it
out,
then
you
want
to
deploy
the
model,
and
then
you
want
to
monitor
it
and
in
monitoring
it.
It
is
not
just
about
having
it
out
there
and
making
sure
it's
up,
but
it's
in
fact
taking
all
that
data
and
feeding
it
back
into
the
original
so
that
you
can
now
train
it
again
and
be
smarter
about
it.
A
But
again
it's
all
driven
off
of
get.
You
don't
have
someone
in
the
middle
there
introducing
any
new
changes,
because
those
wouldn't
have
been
tested
and
relied
or
tested
and
observed
and
made
sure
that
they
pass
all
your
bars.
And
what
that
really
gets
to
you
at
is
is
those
lines
at
the
bottom.
You
get
velocity
plus
security,
so
what
we
need
is
ml,
ops,
right
and
ml.
Ops
is
going
to
look
like
this
right.
You
start
with
the
data
scientist.
A
The
data
scientist
is
able
to
iterate
exactly
as
quickly
as
she
could
and
potentially
even
faster,
because
you're,
giving
her
the
Best
of
Breed
tools
and
you're,
giving
in
to
her
in
a
system
where
she
doesn't
have
to
think
about.
Well,
this
Python
dependency
didn't
work
with
that
other
package
that
I
had
over
there.
You
give
her
a
path
to
install
and
manage
these
things
in
a
very
regular
way,
and
then
from
that
you
start
the
second
half
right
you,
you
have
her
finish.
A
You
have
her
check
in
and
move
forward
through,
dev,
that's
integrating
it
into
the
application
itself
and
then
integrate
again
when
it
comes
to
rolling
out
to
production,
and
when
you
do
that,
you
get
these
benefits
right,
you
get
great
observability.
That
means
you
exactly
what's
being
rolled
out,
so
you're
able
to
observe
it
using
standard,
toolings
and
reproducibility.
You
get
validation
again,
because
the
stuff
is
checked
in
and
good
you
can
do
static
validation.
A
You
could
do
runtime
validation
by
rolling
it
out
to
Canaries
in
a
very
regimented
way,
and
then,
of
course,
you
get
reproducibility
and
auditability.
You
know
that
whatever
that
query
was
that
prediction
that
happened
at
the
end
of
that
very
long
cycle,
you're
able
to
trace
back
exactly
what
code
rolled
it
out
at
every
step
of
the
way,
and
in
this
byte
you
get
velocity
plus
security
for
ml.
So
some
of
the
specific
components
here
and
I'll
just
walk
through
this
really
quickly.
A
Like
I
said
you
have
the
data
scientist
she
checks
in
to
get
at
that
point.
You
do
code
set
version
or
you
do
code,
dataset
and
environment
versioning.
You
snapshot
all
of
those
things
from
there.
It
automatically
builds
both
the
app
and
trains
the
model
and
in
that
you're
able
to
do
hyper,
parameter
sweeps
and
all
those
kind
of
things
in
an
automated
way.
You
validate
the
model
at
that
testing
point
using
things
like
model,
validation
and
certification,
and
things
like
that.
A
A
So
the
question
is
this:
sounds
pretty
good
I'd
like
to
take
it
on
what
should
I
do
in
order
to
achieve
that?
Well,
I
have
a
suggestion,
for
you
go
join
one
of
these
big
companies
because
they
already
have
and
they
have
it
it's.
You
know
at
Google,
it's
tf-x
and
at
Facebook
it's
FB
Lerner
flow
at
uber.
A
It's
Michelangelo
Microsoft
has
our
own,
which
is
called
ether,
and
what
this
is
is
it's
a
very
standard
data
science
platform
where
a
data
scientist
will
come
in
and
you
know
be
able
to
interact
at
a
very
high
level,
see
other
people's
experiments,
see
her
own
experiments,
iterate
and
version
on
her
own
experiments
and
then
basically
download
and
begin
to
use
those
experiments
rather
than
having
to
start
from
scratch
every
time
she
can
start
there
and
and
move
forward,
rather
than
you
know,
resetting
everything.
So
you
have
rejected
my
suggestions.
A
That's
very
harsh
of
you.
I
don't
want
to
work
at
a
company,
so
I'm
gonna
help
you
by
building
your
own
ml
ops
platform
using
first
we'll.
Do
it
with
C
ICD.
Second,
we'll
add
in
git
and
third,
we'll
add
a
soon.
You
start
with
Q
plug
that
was
get
in
the
middle
or
get
lab
or
bitbucket
or
whatever
it
might
be,
and
then
finally
we'll
do
it
with
a
C
ICD
platform
by
the
way
I
hope
I
did
my
best
I'm
trying
to
be
very
neutral
here.
A
All
logos
are
the
same
size
and
by
the
way,
if
you
are
thinking
of
inventing
a
new
C
ICD
platform,
there
are
so
many
there
are
so
many
C
ICD
platforms.
It's
enough
just
help
another
one
help
one
that
already
exists.
I
promise
you
okay,
so
joking
aside,
let's
go
ml
wops
in
the
real
I'm
gonna.
Take
on
a
challenge
that
many
people
I
don't
know.
If
Corey's
here
Corey
are
you
here?
A
I
know
that
you're
attending
cube
con,
but
I'm,
not
sure
you
hear
Corey
doesn't
believe
that
at
multiclad
exists,
I'm
gonna
say
that
it
does
and
I'm
gonna
say
it
does
in
the
following
way:
multi-cloud
really
does
exist
in
the
real
world
and
I'm
gonna
lay
out
a
very
standard
scenario
and
then
solve
it
for
you
on
stage.
So
what?
When
I
say
multi-cloud,
this
is
what
it
looks
like
you
have
at
the
top.
A
You
have
cloud
whatever
cloud
it
might
be
and
at
the
bottom
you
have
on
Prem
and
again
that
could
be
that's
a
special
case
of
a
second
cloud.
It
could
be
a
second
cloud.
It
could
be
on.
Prem
could
be
your
local
laptop,
doesn't
matter
and
in
the
middle
you
have
get
the
reason
you've
chosen
that
top
cloud
is
because
it's
distributed
is
something
that
your
datacenter
can't
offer
you
today.
Maybe
it's
up
time.
Maybe
it's
locality
because
of
regulatory
reasons.
Maybe
it's
just
closer
to
your
IOT
deployments
right.
A
You
don't
want
to
have
that
far
big
Layton
sees
and
so
on,
but
the
reason
used
to
have
stuff
on
prem
is
because
you
actually
have
a
lot
of
data
that
you've
been
collecting
for
many
years
and
it's
just
a
lot,
a
big
pain
in
the
butt
to
move
it
to
the
cloud.
We
see
this
scenario
all
the
time
and
it's
whether
or
not
you
know,
maybe
that's
a
poor
latency
connection,
or
maybe
you
just
don't
want
to
spend.
A
You
know
many
many
millions
of
dollars
on
petabytes
of
data
being
held
in
the
cloud,
because
you
already
have
a
bunch
of
discs
and
it's
doing
its
job.
You
don't
want
to
do
that
so
and
then
we
have
our
to
cast
of
characters.
We
have
our
data
scientist.
She
wants
to
iterate
very
quickly
on
the
the
model
and
make
sure
that
it
works
and
she's
gonna
want
to
train
next
to
the
model,
but
then
the
sres
and
the
ml
engineers
are
going
to
want
to
host
in
the
cloud.
A
So
let's
do
this.
So
what
she
does
is
first
she's
gonna
check
in
her
code
and
to
get
get
will
be
watched
by
the
CI
CD
pipeline
for
a
new
check-in
and
that
will
automatically
trigger
a
cube
flow
pipeline.
It
that
cue
flow
pipeline
will
process
the
data
first,
in
whatever
way
it
needs
processing
or
if
it's
already
been
processed
and
check
pointed
then
it
can
skip
to
the
next
step
and
cue
flows
smart
enough
to
do
that.
It
then
will
run
a
training
run
automatically
on
that
process.
A
A
A
A
A
Because
I
shoulder
the
whole
time
alright,
so
this
is,
you
know
the
one
downside
of
doing
we're
moving
into
ml.
Is
you
have
to
record
all
your
videos
cuz
everything
takes
forever.
So
I
promise
you,
you
can
look
at
the
date.
I,
don't
know
where
the
data
is
on.
Anyone
see
a
date
on
there.
This
is
literally
run
last
night,
so
I
promise
you
you
can
go
to
this
repo,
which
is
public
and
download
all
the
code
and
have
a
good
time.
Okay,
where
are
you
okay?
A
A
Okay,
so
they
always
say
never
show
code
when
you're
doing
a
demo,
because
everyone's
eyes
glaze
over,
but
these
are
very
smart
room,
so
I'm
gonna
trust
you,
your
eyes
are
not
going
to
leave
over.
So
what
you
have
here
is
this
is
a
very,
very
standard,
CI
CD
pipeline.
This
is
using
Azure
dev
ops,
which
is
something
that
we
use
hosted.
Dev,
ops,
CI
CD
on
Azure,
but
the
Yambol
is
gonna,
look
super
familiar
because
it
follows
very
standard.
A
Cid
CD
practices
looks
just
like
jenkins
and
what
you
see
here
is
at
the
top,
you
see
a
trigger.
That
means
it's
triggering
off
the
master
once
you
check
in-
and
here
you
have
individual
steps.
This
in
fact
does
a
build
of
container
and
we
do
three
builds
here,
because
that's
what
cube
flow
requires
cube
flow
needs
every
one
of
your
steps
to
be
built
into
a
container,
and
then
you
roll
that
for
and
into
your
cute
flow
pipeline
and
what
you're
rolling
in
what
the
pipeline
looks
like.
A
Is
this
and
actually,
why
don't
I
get
to
this
in
just
a
second?
So
we'll
go
back
to
our
video
here
and,
like
I
said
what
you
see
is
a
standard
repo
or
is
my
thing
here,
and
that
was
just
the
code
that
you
saw
me
pointing
at
earlier
with
standard
pipelines,
and
this
is
a
very
important
one.
It
does
burritos
versus
tacos.
You
upload
and
it'll.
A
A
That's
in
the
repo
right
now
is
doing
transfer,
learning
using
very
standard
transfer,
learning
tool
called
mobile
net,
and
so
we
layer
on
on
top
of
that
so
like
I,
said
this
is
just
incredibly
pipeline
code,
and
these
are
the
pipeline
steps
that
you'll
see
here
and
I've
into
those
in
just
a
second.
But
the
idea
is
that
we
want
to
show
how,
when
you
check
in
it's,
going
to
kick
all
these
things
off.
So
we
come
back
here
and
we're
gonna
hit
merge
on
our
pull
request
and.
A
It's
gonna
kick
off
there,
we
go
so
we're
hitting
merge
and
off
we
go
and
there
you
go.
So
this
is
the
UI
for
a
DevOps.
Again,
all
we
did
was
hit
merge
and
it
all
this
kicked
off.
So
at
this
point
we're
gonna
go
forward.
You
can
see
there
right
there
build
number.
Nine
has
just
kicked
off,
and
that
is
you
know,
May
21st,
so
you
know
I'm
being
honest
here
and
you
can
see
how
the
things
are
aspiring
here.
A
So
in
this
case,
it's
going
to
go
through
those
three
build
steps
where
it
builds
each
one
of
the
containers
using
standard
as
your
build
tools
and
again
this
can
be
docker.
This
could
be
builder.
This
could
be
anything
that
makes
sense
for
you
and
you've
all
seen.
Docker
build
many
many
times
so
I'm
going
to
fast
forward
through
those
okay
and
then
this
last
step
is
where
you
take
all
those
containers
and
now
you've
uploaded
them
to,
in
this
case,
a
CR,
but
it
could
be
any
docker
registry,
okay.
A
A
You
want
to
make
sure
the
build
is
complete
that
all
the
artifacts
pass
and
already
go,
and
now
we're
going
to
do
this
release
pipeline
and
the
release
pipeline
is
where
it
reaches
out
to
cube
flow
and
executes,
and
you
can
see
this
here,
it's
reaching
out
using
a
swagger
client
that
we
wrote
to
connect
directly
to
the
cue
flow
API.
You
can
see
there,
it's
forwarding
the
connection,
and
here
you
can
see
the
run
this.
These
are
the
rough
he
had.
Previously
we
run
84.
A
And
presto,
we're
at
run
eighty-five
right
so
again,
I
want
to
call
back,
because
this
is
a
small
but
special
thing
all
I
did
was
kick
hit.
Get
all
I
did
was
check
in
and
merge
my
final
PR
and
it
kicked
off
all
of
this
goodness.
So
now
we
were
sending
through
here
and
you
can
see
a
rerunning
know.
Okay,
so
this
is
a
standard
cue
put
pipeline,
I'd
already
described
it.
A
It
does
pre
processing
here
it's
mounting
in
Azure
blob
and
you
can
see
it
watching
the
logs
right
there
and
you
can
see
the
artifacts
once
that's
done.
It
automatically
moves
forward.
The
containers
been
created,
and
here
it's
actually
doing
a
training
run
right
now
it's
using
the
latest
tensorflow
2.0.
A
Forward
there
and
now
it's
doing
registration-
and
this
is
where
you
start
to
mix
and
match
between
things
running
in
queue
flow
and
things
that
you
may
want
to
run
in
the
cloud.
So
in
this
case,
it's
going
to
do
this
last
register
step
in
Azure,
we
have
a
service
called
the
model
management
service.
A
This
is
basically
a
sophisticated
storage
location
which
understands
machine
learning
concepts
first-class
way,
and
by
that
what
I
mean
is
you're
able
to
understand
whether
or
not
something
needs
versioning,
how
to
compare
versions
within
each
other,
what
the
performance
of
something
is
and
you're
able
to
visualize
it
in
a
first-class
way,
and
so
what
it's
going
to
do
here
at
this
last
step
here
are
three
versions
that
we
ran
already
and
once
it's
done,
you
can
see
here.
This
is
the
model
registry
service.
A
Yeah,
okay,
so
you
can
see
here
it's
using
in
this
case
the
azure
SDK
natively
inside
of
cube
flow.
It's
able
to
reach
out,
and
you
can
see
they're
using
our
service
principles,
service,
passwords,
the
subscription
information,
it's
able
to
take
that
hc5
file
and
simply
push
it
up
to
the
model
registry
service,
which
then
will
kick
off
the
second
half
of
the
solution,
and
you
can
see
there.
So
that's
roughly
what
it
is.
A
You
know
rolling
things
out:
here's
where
you
can
see
some
of
the
richness
that
we're
able
to
upload
and
the
metadata
that
we
have
relative
to
these
things
and
no
I
don't
need
you.
They
were
real.
So
that's
what
model!
That's
what
mo
Ops
looks
like
in
reality
right
and
from
that
point
forward.
You
know
I
just
rolled
out
a
service
endpoint.
Everyone
seen
that
right,
nothing
special
there.
A
But
that's
what
we're
talking
about!
This
is
something
where
you
had
a
cube
flow
cluster
that
was
in
no
way
connected
to
you
know
any
of
the
more
sophisticated,
azor
or
other
components
and
with
just
running
standard
kubernetes
and
standard
cube
flow
and,
and
the
reality
is,
is
that
gives
you
the
opportunity
to
do
the
training
in
a
way
that
makes
sense
for
your
organization?
You
don't
have
to
take
your
data.
You
don't
have
to
change
the
way
your
processes
work.
You
don't
have
to
adopt
some
crazy
new
standard
just
because
that
looks
slightly
faster.
A
This
is
an
open
platform
where
you
can
pick
and
choose
and
plug
in
the
components
that
make
sense
to
you
and
that's
the
power
that
that
things
like
open
source
and
things
like
Q
flow
provide
now
I
know
what
you're
saying
it
sounds
like
a
lot
of
work.
I
would
like
to
recall
this
statement
here.
You
know:
what's
a
lot
of
work
eleven
months
sitting
around
without
your
model
shipping
out
to
production,
that's
a
lot
of
work!
This
you
know.
A
Okay,
you
have
to
spend
a
few
minutes
making
sure
that
service
principals
are
able
to
cross
clouds,
but
other
than
that
you'll
be
fine
and
and
you're,
not
just
fine
right,
you're,
actually
moving
faster
and
and
that's
what
we're
really
trying
to
provide
here.
Data
scientists
understand
their
problem
areas,
but
too
often
they
have
thrown
things
over
the
wall
and
asked
production
to.
Ultimately,
you
know
deal
with
a
lot
of
the
hardness.
What
we're
saying
is
is
we
want
to
bring
through
things
like
ml
ops?
A
We
want
to
bring
the
same
advances
that
get
ops
brought
to
app
developers.
We
want
to
help
data
scientists
be
part
of
how
to
bring
things
to
production
and
get
into
end
ownership,
and
even
better
than
that
help
them
learn
real
software
engineering.
Best
practices
like
for
too
long,
we've,
we've
said:
hey
data
scientists,
you
know,
don't
worry
about,
you
know
sweet
best
practices.
You
know
come
back
when
you're
older.
No,
that
is
not
okay.
These
are
these.
Are
software
engineers
they
want
to
help
get
things
into
reduction.
A
They
want
their
models
to
be
used
just
as
much
as
you
do.
We
need
to
help
them
bridge
the
gap,
but
not
by
forcing
them
to
you
know,
figure
out
how
to
do
git
branch
strategies
on
top
of
that
it
Abel's
continuous
delivery.
The
the
the
automated
pipeline
that
you
saw
earlier
can
be
automated
every
night,
and
that
is
something
that
we
learned.
You
know
for
years.
All
of
those
big
production
clients
you
saw
there
that
were
able
to
continuously
iterate
using
these
ml
ops
platforms.
A
They're
all
moving
be
moved
to
continuously
training
these
models,
and
the
reason
is,
is
because,
unlike
a
lot
of
software,
models
go
out
of
and
go
stale
very,
very
quickly.
You
need
to
continuously
train
them.
You
need
to
continuously
watch
your
data
and
make
sure
things
haven't
drifted.
One
way
or
another
people
change
code
changes
situation,
changes.
Continuous
training
is
the
only
way
that
you're
going
to
be
able
to
deliver
value
at
scale
over
time
and
then,
finally,
it's
impossible
to
overstate
how
much
value
you
get
by
by
instituting
these
things
around
lineage
and
auditability.
A
Unless
your
code
is
checked
in
it,
everything
is
is
moved
forward.
From
that
point,
you
will
not
be
able
to
defend
in
a
court
of
law
or
your
Terry
are
exacts
or
to
the
software
engineer,
sitting
right
next
to
you
exactly
what's
going
on
in
production
and
again,
because
we
haven't
spent
the
time
yet
to
make
a
miops
easy
we're
putting
that
that
onus
on
the
work
of
the
software
engineers
and
the
developer
and
the
data
scientists
to
build
their
own
solutions
and
and
oftentimes
they're
gonna
miss
something.
A
So
it's
up
to
us
to
give
them
a
great
platform
to
do
that.
I
always
like
to
end
this
slide
with
or
I,
always
like
to
end
my
talks
with
a
slide
like
this,
because
it's
so
important.
The
reality
is
is
that
the
people
in
this
room
and
the
people
watching
on
the
video
you're
already
there.
You
already
know
a
lot
of
this
stuff
and
you're
working
towards
this
data.
Science
will
affect
every
industry
in
the
world.
Ml
and
AI
will
affect
every
industry
in
the
world.
A
It's
up
to
us
to
figure
out
how
to
bring
our
data
science
to
these
people
and
help
them.
The
people
who
have
domain
knowledge
be
smarter
about
what
they're
doing
cube
flow
is
open
I.
We
would
love
for
you
to
come
and
contribute
to
all
of
the
work
we're
doing
around
cube
flow
with
ml
ops.
You
could
see
a
bunch
of
the
things
here.
Everything
that
I
demoed
on
screen
is
a
source
and
ready
for
you.
We
do.
A
Obviously
it
is
built
on
Azure
pipelines
but,
like
I
said,
that's
very
portable
to
other
pipelines,
and
we
would
love
to
help
you
do
that.
We
actually
have
an
entire
repo
dedicated
to
ml
ups
and
we're
hoping
to
move
that
from
it's
right
now
it's
hosted
under
the
Microsoft
one.
We
would
love
to
host
it
in
a
more
general
place
where
people
can
share
and
then,
of
course,
in
cube
flow.
A
We're
talking
about
ml
ops,
constantly
the
work
that
we're
doing
around
pipelines
is
specifically
designed
to
help
solve
this
very
critical
problem,
and
let
me
give
a
quick
shout
out
to
other
talks
here
at
the
conference
around
cubic
flow,
please
go
and
and
other
than
that.
Thank
you
very
much
for
the
time.
A
A
We
do
get
that
question
all
the
time
most.
Data
providers
today
will
have
a
way
to
do
snapshotting
or
versioning
a
lot
of
it
comes
down
to
what
the
underlying
data
set.
Oh
sorry,
the
question
was:
what's
your
best
practice
for
versioning,
datasets
checkpointing
and
things
like
that.
So
a
lot
of
large
data
providers
Sparco
do
a
sequel,
server,
my
sequel,
so
on
and
so
forth.
They'll
have
ways
to
accomplish
this
in
their
own
systems,
but
a
lot
of
it
comes
down
to
what
is
the
immutability
of
your
data?
A
Look
like
if
you
have
a
practice,
a
best
practice
around,
for
example,
audit
trails
for
your
database,
then
that
might
be
enough
right.
You
could
simply
say:
okay,
I'm
gonna
do
select
star
from
table
that
I
like
between
January,
1st,
1970
and
January
1st
1980
right.
That
could
be
enough.
You
could
simply
say:
okay,
I'm
done
and
as
long
as
that
that
dataset
hasn't
changed,
then
that's
enough
because
I
know
I
can
always
go
back
and
recreate
that
if
you
need
things
like
you
know,
deltas
and
and
more
complicated
structures
like
that.
A
Our
recommendation
is
to
work
with
whatever
data
provider
you
have
because
most
of
them
have
it.
There
are
also
a
number
of
open
source
tools
that
are
doing
some.
It
very
interesting
things
around
this
pachyderm
has
stuff.
Dvc
has
some
stuff
there's
a
bunch
of
really
interesting
stuff
around
this
right
now,
but
the
net
is,
is
I
would
start
with
wherever
you're
storing
your
day
today
and
then
go
forward
from
there.
A
D
A
A
Model
DB
is
one
and
that's
already
integrated
into
flow,
and
so
on,
more
often
than
not
what
you'll
want
to
do
is
you'll
want
to
have
your
model
registry
service,
whatever
IB
be
separate
from
your
cube
flow
deployment
because
oftentimes
those
those
exist
in
different
planes
and
one
has
a
much
longer
persistence
than
the
other
cube
flows.
Often
times
brought
up
and
torn
down
quite
regularly,
whereas
your
model
registry
you'd
want
that
going
on
forever.
A
To
be
clear,
though
I
demoed,
two
pipelines
on
stage
that
could
have
been
one
pipeline
that
could
have
been
six
pipelines.
It's
whatever
makes
sense
to
you.
Generally
speaking,
your
best
practice
is
to
have
one
pipeline
per
environment
if
you
will,
because
that
or
our
per
particular
function.
So
what
you
didn't
see
here
is,
for
example,
oftentimes
you'll,
have
a
data
pipeline
for
doing
your
overall
sharding
and
processing
of
your
data,
moving
it
to
its
final
location.
A
My
I
did
pre-processing
as
part
of
the
pipeline
that
you
saw
on
stage,
but,
like
I
said
you
can
use
the
azure
model
management
service.
That
is,
you
know,
nothing
about
cube
flow
is
required
there.
It's
just
an
open
end
point
and
you
can
use
something
like
that
or
you
can
use
anything
that
you
would
want
to
host
yourself.
A
A
Need
the
model
is
just
a
file
yeah,
so
yes,
and
no
the
model
is
just
a
file.
That
is
absolutely
correct,
but,
as
you
saw
there
on
stage,
what
you'll
frequently
want
is
more
information
about
the
model
or
more
sophistication
about
how
to
store
that
model.
So,
as
you
saw
there,
I
recorded
that
that,
even
though
this
was
there
were
in
fact,
four
different
models
there.
They
were
all
part
of
a
single
model
training.
You
know
job
if
you
will
and
what
they
would
have.
They
had
different
versions
and
the
reason
you
want
to
do.
A
That
is
because
that
will
give
you
the
opportunity
to
do
intermodal
comparison.
Oh
for
continued.
You
know,
ongoing
training
runs
and
so
on
again.
That
is
absolutely
something
you
can
achieve
yourself,
though
I
talked
about
a
sure.
This
is
something
you
can
definitely
do
it's
just
a
matter
of
you
know
if
you
want
to
host
it
yourself
or
have
someone
else
hosted
thanks.
C
B
A
The
the
important
thing
to
take
away
here
are
the
steps
in
the
pipeline.
The
not
important
thing.
The
way
is
the
exact
things
that
we
chose.
So,
for
example,
there
is
already
a
container
builder
inside
of
cube
flow.
That
could
be
perfectly
great
for
you.
We
just
happen
to
choose
this
one,
because
the
you
know
software
engineer
and
I
who
we're
working
on.
It
knew
how
to
do
that
very
quickly,
and
so
that
that's
basically
it
again.
You
do
want
to
the
the
more
important
bit.