►
From YouTube: The SAME Project: A GitOps Approach to Reproducible Machine Le... David Aronchick & Chanwit Kaewkasi
Description
The SAME Project: A GitOps Approach to Reproducible Machine Learning with Flux and Kubeflow - David Aronchick, Azure, Microsoft & Chanwit Kaewkasi, Weaveworks
We live in a time of both feast and famine in machine learning. Large orgs are publishing state-of-the-art models at an ever-increasing rate but data scientists face daunting challenges reproducing results themselves. Even in the best cases, where a newly forked code runs without syntax errors, this only solves a part of the problem as the pipelines used to run the models are often completely excluded. The Self-Assembling Machine Learning Environment (SAME) project is a new Kubernetes, Kubeflow and Flux project and community around a common goal: creating tooling that allows for quick ramp-up, seamless collaboration and efficient scaling - through the magic of GitOps. This talk will discuss our initial public release, done in collaboration with data scientists from across the spectrum, where we are going next and how people can use our learnings in their own practices.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
So
one
thing
that
you
hear
a
lot
of
in
the
news
is
how
great
ml
is
and
the
fact
that
it
changes
the
world
every
single
day.
This
is
a
study
out
of
google,
where
they
have
some
very,
very
smart
data
center
engineers
trying
to
make
their
power
usage
efficiency
as
high
as
are
exactly
as
low
as
possible.
A
A
Hundreds
of
millions
of
dollars
same
amount
of
compute,
less
energy
and
just
to
prove
it
to
themselves
when
they
turn
it
back
off
here
you
can
see
it
popping
right
back
up,
so
it
really
was
machine
learning
that
made
all
the
difference,
and
we
talk
about
these
stories
all
the
time
in
public,
and
that
is
great.
Unfortunately,
ml
is
very
hard
and
I
don't
just
mean
it's
hard,
because
I'm
bad
at
math
it's
hard,
because
most
folks
are
in
one
camp.
A
They
see
all
this
news
out
there
and
magical
ai
goodness
is
over
there
and
in
between
is
a
lot
of
pain,
and
the
pain
manifests
like
something
like
this,
where
you
see
people
developing
solutions
very,
very
quickly
and
seeing
good
results,
but
when
it
comes
to
rolling
them
out,
it
still
be
is
a
big
challenge.
Haven't
we
heard
this
story
before
we
have
and
that
this
was
a
lot
of
what
we
went
through
when
we
first
started
with
containers
and
kubernetes
and
the
way
that
they
were
able
to
break
through?
A
A
A
Well,
in
my
mind,
I
think
it
breaks
down
to
composability
portability
and
scalability
composability
looks
like
this.
You
start
with
just
a
model,
and
this
is
where
a
lot
of
people
start.
Jupyter
notebook
is
one
of
the
easiest
interfaces
I've
ever
seen
in
order
to
get
going
quickly,
but
in
order
to
do
anything
at
scale
or
across
environments
or
even
inside
a
single
environment,
you
really
need
to
think
beyond
just
that
and
start
to
think
about
all
these
other
elements
of
a
data
pipeline
data,
ingestion,
analysis,
transformation,
so
on
and
so
forth.
A
If
you
don't
have
a
way
to
reproduce
that
thing
easily
and
migrate
from
place
to
place,
you're
going
to
be
in
trouble
and
finally,
we
get
to
scalability.
So
when
you
hear
the
word
scalability
a
lot
of
times,
people
just
think
more,
more
infrastructure,
more
accelerators,
more
disks,
cpus
and
so
on.
But
it's
really
goes
beyond
that.
It
also
goes
to
more
programs.
A
A
Millions
of
papers
are
being
published
and
it's
up
to
the
data
scientists
to
just
try
and
keep
up,
but
it's
also
scaling
collaboration
and
people.
How
do
you
scale
skill
sets
where
you
have
swedes
and
sres
and
data
scientists
all
working
together,
each
of
whom
have
particular
sets
of
expertise,
scaling
your
teams,
adding
more
people
and
allowing
you
to
collaborate
and
then,
of
course,
scaling
across
organizations
could
be.
You
know
across
a
under
a
single
umbrella,
if
you're
a
very
big
company.
A
Like
me,
you
want
people
from
xbox
being
able
to
share
with
office
being
able
to
share
with
windows
or
it
could
be
sharing
across
organizations.
So
academic
institution
number
one
wants
to
share
with
academic
institution
number.
Two
that's
a
way
to
scale
as
well.
So
at
the
end
of
the
day,
more
is
correct,
but
really
people
just
want
to
do
more
programs
and
do
them
faster.
A
So
that's
composability,
scalability
and
portability,
and
those
are
the
essence
of
what
I
think
cloud
native
ml
is.
So
you
know
what's
really
good
at
composability
portability,
scalability
containers
and
kubernetes,
except
if
you
want
to
do
that
today,
you
first
have
to
become
an
expert
in
a
whole
bunch
of
things:
containers,
gpls
service,
endpoints
scaling
all
that
kind
of
good
stuff,
which
is
a
real
problem,
as
you
might
imagine,
and
certainly
not
something.
We
want
our
data
scientists
and
other
highly
skilled
individuals
having
developed
general
tools
around.
So
the
idea
would
be.
A
How
do
we
build
something
that
gives
everyone
cloud
native
and
that's
the
same
project
which
stands
for
self-assembling
machine
learning
environments?
The
l
is
silent
with
same
what
you
do.
Is
you
start
with
something
base
in
this
case
we're
going
to
start
with
a
kubernetes
and
from
there
you
develop
a
declarative
layout
of
a
what
that
pipeline
and
experiment
might
look
like
and
you're
able
to
stamp
that
out
anywhere.
A
You
have
that
kubernetes
api
now,
I
said,
start
with
kubernetes
we're
doing
a
lot
of
looking
at
working
with
other
hosted
platforms
as
well
and
they're.
You
know
the
news
is
very
good.
There's
most
of
the
stuff
that
you
see
on
a
lot
of
these
different
platforms
is
quite
boilerplate,
but
just
so
that
we
can
get
started
and
prove
out
a
lot
of
these
examples,
we're
starting
with
some.
You
know
a
narrow
scope.
We
would
love
to
work
with
other
partners
to
get
things
rolled
out.
A
A
You
have
your
data
scientist
here.
She
starts
on
a
laptop
and
she
says
you
know
I
need
a
recommender.
There
are
lots
of
recommenders
out
there
in
the
world.
These
are
the
tools
that
you
see
when
you
log
into
your
video
hosting
service
of
choice,
and
it
shows
you
one
image
and
a
set
of
other
movies
based
on
that
particular
image
that
that
they
recommend
that
you
watch
and
she
says
you
know
I'd
like
to
see
if
I
can
beat
the
recommender.
A
A
So
here
you
have
the
kubernetes
cluster
that
was
set
up
for
her
by
her
I.t
or
via
docker
locally
again,
that
very
local
thing
and
she
says
same
init
and
from
this
point
it
goes
forward
and
it's
going
to
initialize
that
environment
with
everything
necessary
in
order
to
run
a
saying.
So
this
might
include
a
pipeline
system,
a
kubeflow,
dashboard,
key
float,
metadata
store
so
on
and
so
forth.
Again,
you
can
use
ones
that
are
outside
of
the
core.
We
just
set
one
up
for
you
for
free.
A
So
then
she
goes
into
the
repository
herself
and
she
says
I
want
to
run
it
now
inside.
That
is
going
to
be
a
same
file.
This
assumes
that
nvidia
has
gone
and
created
that
thing,
but
it's
very
simple
to
create,
and
if
you
don't
have
one
you
can
create
on
your
own,
there's
nothing
terribly
complicated
about
it.
But
by
doing
this,
you
now
have
a
code
style,
declarative
statement
for
exactly
how
this
entire
pipeline
should
run.
A
The
first
thing
that
does
is
it
goes
out
and
provisions
the
necessary
hardware
using
the
cli
and
the
the
cluster
setup
that
that
is
built
into
kubernetes.
Here
she
wants
gpu
nodes.
She
wants
premium
disk.
These
are
things
that
can
be
declared
next.
She
installs
the
necessary
services
for
this
pipeline
run.
This
pipeline
requires
a
feature
store
that
she
names
2.4.
A
It
requires
tensorflow
1.8
and
it
requires
cat
tip,
1.2
she's,
able
to
declare
that
and
the
system
rolls
it
out
on
her
behalf
after
that,
it's
going
to
download
data
from
an
external
source
to
local
premium
disk
again
this
is
optional.
These
are
things
you
can
do
as
part
of
your
pipeline,
but
it
also
sets
up
a
very
clear
path
for
how
she
might
engage
with
that
data.
A
She
pushes
that
when
she
clones
it,
it
pulls
that
down
and
pushes
it
into
a
metadata
store
and
again
you
might
ask
well
why
is
this
necessary?
You
know
you
have
all
these
the
examples
and
store
there
locally.
It's
true
you
do,
but
many
people
need
to
compare
many
models
over
time
and
by
having
this
in
a
common
metadata
store,
you're
going
to
be
able
to
compare
new
runs
against
those
existing
runs.
A
So
that's
it.
It's
now
set
up
and
run
and,
as
I
said,
you
can
see
it's
same
program
run,
so
it's
now
going
to
go
forward
and
execute
it
here.
It's
executing
experiment
number
eight,
because
it
detects
that
it
is
number
eight
as
part
of
the
overall
again
making
comparison
to
previous
runs
much
easier.
A
It
pushes
the
artifact
down
to
her
premium
disk
and
pushes
the
additional
metadata
into
her
metadata
store,
and
then
she
decides
that
she
wants
to
change
a
parameter,
changing
the
number
of
epochs
to
2000
instead
of
a
batch
parameter
of
1000,
which
it
does,
and
it
runs
experiment
number
nine
and
runs
that
as
well.
So
now
she's
able
to
compare
against
these
various
things
and
see
exactly
how
it's
going
to
run
moving
forward.
And
ideally
it's
doing
this
all
from
code
and
get
ops.
A
And
this
is
where
things
like
flux
that
have
a
great
awareness
of
both
ends
of
the
spectrum,
both
the
starting
point
over
at
code,
all
the
way
through
to
the
way
that
things
need
to
roll
out
in
production.
And
that's
what
we're
going
to
demo
now.
Let
me
hand
it
over
to
chondwit
who's
going
to
demonstrate
how
to
roll
out
using
same
and
wire
that
up
using
flux
in
order
to
automate
your
ml
ops
pipeline.
B
Hello
in
this
section,
I'm
going
to
demonstrate
how
to
use
bam
and
flux,
to
implement
a
machine
learning
operation
pipeline,
of
course,
using
git
ups.
Okay,
now
we
have
the
diagram
here
and
with
this
diagram
we
have
we're
going
to
have
three
different
elements.
The
first
one
is
a
local
queen
test
cluster
and
the
second
one
is
our
shura
aks
cluster
and
the
third
one
is
going
to
be
our
prod
production,
kubernetes
cluster,
which
runs
only
the
inference
program
in
the
diagram
you're,
going
to
see
it
as
prod
ui.
B
Okay,
let's
start
with
the
local
cluster.
What
we're
gonna
do
in
the
local
cluster
is
that
we
we
going
to
clone
the
same
program
from
git
repo
after
we
cloning
it
down
to
our
laptop.
We
going
to
start
develop,
develop
our
program
and
in
this
example,
the
program
we're
going
to
use
here
as
a
case
study
is
the
tensorflow
retrain
flower
classification.
B
It's
a
deep
learning
model
training.
Okay.
Now
we
have
our
cloned
repository
here,
and
this
is
the
repository
on
kidops
and
we
also
have
a
local
cluster.
Here.
You
can
use
any
kind
of
small
cluster
implementation,
litecoin
or
k3s
or
any
then
we
run
the
same
init
command
and
this
what
same
init
command
does
is
that
it's
going
to
install
the
ku
flow
pipeline
components
into
our
cluster
after
the
installation
succeed
we're
going.
B
We
are
ready
now
to
run
our
same
program
and
the
same
program
is
inside
the
train
directory
of
this
repo.
Okay,
we're
going
to
call
this
submission
local
run,
and
you
you
see
here
the
revision
number.
This
is
a
parameter.
We
we
go
we're
going
to
send
it
to
the
pipeline
upon
running
okay,
we're
going
to
use
the
mobile
net
model
as
as
our
base
model
for
retraining.
B
So
we
retrain
the
mobilenet
model
using
the
flower
classification
data
set,
and
now
what
we
going
to
do
is
to
or
call
the
same
program
run
when
you
call
the
same
program
run
in
the
directory
containing
same
yami
file.
It's
going
to
submit
your
program
into
the
kubeflow
pipeline
cluster,
which
is
already
running
locally
on
our
on
our
laptop
okay.
B
Now
you
see
it
in
the
ui
that
the
pipeline
running
through
and
we
got
the
training
step,
which
is
the
pvc
preparation,
pre-processing
step
the
return
step,
and
finally,
we
select
the
base
model
and
the
model.
Here
we
only
have
one
model
so
yeah,
it's
the
best
one.
We
have
you
you're
gonna,
see
that
the
accuracy
is
not
that
high,
because
it's
a
model,
a
mobile
net
model
and-
and
we
run,
we
ran
it
just
only
one
epoch.
So
we
got
the
not
very
not
that
high
accuracy.
B
Now,
after
the
after
the
model
has
been
trained,
it's
gonna
store
inside
our
s3
compatible
server,
which
is
menu
at
the
moment.
Okay,
that's
the
first
part.
We
run
we
reinstall
cool4
pipeline
components
into
our
local
cluster.
Using
stamina,
then
we
ran
the
same
program,
run
command
to
submit
our
program
to
retrain
our
classification
model.
Next,
we're
going
to
do
this
in
a
more
scalable
way
on
azure
cluster
and
in
in
the
term
that
we're
going
to
do
this
in
a
more
scalable
way.
B
It's
mean
that
we're
going
to
use
github's
component,
which
is
which
is
flux,
components
we
we're
gonna,
use
same
and
flux
together
and
we're
gonna
use
the
same
controller
for
flux,
v2
inside
actual
cluster,
to
train
our
program
automatically.
Every
time
we
do
get
put.
So
this
is
a
put
to
train
technique,
and
it's
been
that
we
don't
need
to
prepare
any
kind
of
deaf
element
at
all,
because
everything
like
a
game,
controller,
same
program
also
the
google
pipeline
components.
B
We
know
we're
not
going
to
install
the
cool
flow
components
using
the
flux
bootstrap,
not
the
not
the
same
init
command
anymore,
because
the
whole
cluster
has
been
set
up
using
flux
command
and
now
we're
making
change
to
our
program
change.
The
name
of
the
program
into
actual
run
and
we're
going
to
start
with
trending
the
mobile
net.
B
Now
we
made
chain
and
commit
and
put
okay,
let's
see
what's
going
on
on
the
big
cluster,
we
got
the
more
beautiful
pipeline
graph
and
it's
compelled
and
then
at
the
final
step
of
the
of
the
work
of
the
pipeline,
you
see
that
it's
gonna
select
the
best
model
of
these
three
and
what
is
what
it
is
is
that
it
saves
the
best
one
into
the
minial
cluster
inside
our
ashura
kubernetes
cluster.
Okay,
now
I'm
going
to
explore
it
a
bit
here.
B
You
see
the
ml
pipeline
park
ml
pipeline
bucket
and
our
models
here
inside
the
cluster
inside
the
azure
cluster.
We
use
the
revision
or
number
from
our
git
repo
as
a
metadata
of
the
trend
model,
so
you
see
inside
the
menu
server
that
we
stole
the
revision
number
of
each
trained
model.
Okay,
here
the
metadata,
the
revision
number
we
took
from
git-
and
this
is
the
the
metadata
is
saved
by
the
select
base-
module
step.
Okay.
So
what
is
that
we're
going
to
use
that
revision
to
trace
our
model?
B
So
we
know
that
where
this
came
from
which
revision
is
built
from
and
this
revision
this
move,
the
revision
number
is
going
to
display
inside
our
front-end
program
too.
Now,
let's
predict
some
flowers,
and
here
it's
going
to
be
happening.
We
put
the
best
model
inside
the
model
storage.
Here
you
see
in
diagram
and
not
only
the
model.
We
also
generate
a
deployment
xiaomi
file
and
put
it
inside
the
model.
Storage
too.
Then
we
use
flag
to
sync
from
the
bucket.
I
show
you
the
bucket
definition.
B
Okay,
we
point
this
bucket
to
flux
and
we
have
a
customization
object
of
flag
to
do
the
auto
deployment
every
time
the
bucket
change.
So
that's
why
we
got
we're
going
to
get
the
ui
on
the
staging
cluster
every
time.
Now,
let's
move
to
the
production
cluster,
we
already
have
a
staging
program
running
using
the
model
in
inside
model
storage
of
the
azure
cluster.
Now,
what
we're
going
to
do
next
is
to
show
the
ability
to
tag
the
repository.
Then
we
put
this.
B
We
put
that
tag
and
the
promotion
workflow
inside
github
going
to
create
it's
going
to
generate
a
pull
request
for
us
to
do
the
production
promotion.
So
the
idea
is
that
we
tag
and
put
to
promote
the
staging
version
of
the
model
to
the
production
cluster.
Okay,
here
our
product
cluster,
then
we
initialize
everything
using
flux.
Bootstrap,
we
install
good
for
pipeline.
We
install
ui
program
we
install
either
setting
now
in
our
production
cluster.
B
We
have
another
revision
of
the
model
running
that
with
tag
number
all
through
six
v026,
that's
a
version
that
there's
a
lot
text
number
on
the
production.
Now
we
have,
let's
see
now
we
have
a
actual
cluster
on
the
azure
cluster.
We
have
another
version
of
the
module,
just
trend
store
there
and
we
like
to
promote
this
revision
of
the
model
to
the
production.
B
Now
we
tagged
it
as
the
version
0
3
0
and
push
the
text.
Okay,
let's
see
it's
going
to
create
a
new
request
for
us
using
a
github
action
and
then
you're
going
to
see
a
pull
request.
Okay,
it's
all
three!
Oh
then
we're
going
to
merge
this
using
github
just
like
what
we
do
normally
and
it's
going
to
come
in
to
the
main
branch
and
flux
on
the
cluster,
watching
the
change
and
pull
the
chains.
B
A
So
with
that,
thank
you
so
much
excellent
demo
of
showing
how
this
is
all
the
code
is
obviously
open
source.
You
can
go
ahead
and
check
it
out
right
now
and
install
it.
You
can
in
fact
use
same
we're
making
changes
to
it.
You
know
virtually
daily
right
now,
but
it
is
very,
very
not
for
production
use.
I
can't
stress
that
enough.
A
The
same
cli
experience
as
you
may
have
seen,
is
designed
to
be
really
straightforward.
One
line
to
install
it
one
line
to
run
an
experiment,
including
pointing
it
at
an
external
repo,
will
clone
that
and
do
other
things
for
you,
one
line
to
run
it
with
a
different
set
of
parameters,
which
is
something
that
folks
need
to
do
a
lot,
particularly
if
they're
doing
a
big
grid
search
across
a
number
of
different
hyper
parameters,
and
then
one
line
to
export,
and,
of
course
I
cannot
stress
enough.
This
is
very,
very
early
stuff.
A
We
haven't
implemented
all
of
these
verbs
and
features,
but
so
far
as
we
test
this
with
customers,
they
really
do
appreciate
it
and
think
that
this
is
the
right
way
to
go,
and
you
heard
me
talk
about
a
same
manifest.
Quite
often,
you
may
think
that
it's
quite
complicated.
It's
really
not
it's
designed
to
be
very
straightforward
here.
You
can
see
an
entire
manifest,
including
versions
disk
mounts
that
you
might
need
data
sets.
A
We
want
something
to
be
completely
open:
source
cover,
no
shadows
of
being
a
commercial
entity.
This
is
truly
in
open
source
contributed
to
by
lots
of
external
people.
We
want
it
to
be
a
declarative
deployment
as
long
as
you
have
a
kubernetes
or
kubernetes-like
api,
you
push
this
there
and
it
will
just
execute
and
based
on
a
universal
platform.
So
you
know
if
it
requires
you
to
install
something
that
that
really
isn't
out
there.
A
So
what
can
you
do?
We
are
just
getting
started.
The
best
thing
you
can
do
is
go
join
our
mailing
list.
Go
join
our
slack
come
ask
questions.
You
can
try
out
the
binary
we
plan
on
open
sourcing
shortly.
We
haven't
gotten
through
all
the
details
yet,
but
this
will
appear
in
open
source,
open
source
license
and
all
that
good
stuff
and
you
can
come
join
our
repo.
A
You
know
we
haven't
moved
everything
out
just
yet,
but
you're
more
than
welcome
to
come
in
file
issues
and
things
like
that,
and
we
would
love
to
hear
from
you
and
if
it's
something
else,
please
call
me
you
can
see
my
contact
information
there.
We
would
love
to
hear
from
you
and
with
that.
Thank
you
very
much.
A
Hey
everyone,
so
I'm
dave
ranchek
and
here
to
answer
any
of
your
questions.
You
know
we
got
a
question
already
about
getting
access
to
the
code,
so
the
binary
is
out
there.
You
can
just
go
to
sameproject.org
and
download
the
binary
and
try
that
out.
Unfortunately,
the
code
isn't
out
there,
but
we
do
have
a
lot
of
samples.
A
We
have
some
samples
for
mnist
and
and
reinforcement,
learning
and
and
cross
selling,
and
all
these
various
things
you
can
check
out
the
repo
and
see
all
that
code
and
download
it
and
reuse
it
it.
You
know
all
works
right
now
with
cube
flow
right
out
of
the
box,
so
you
don't
even
have
to
use
same
in
order
to
do
that,
but
you
certainly
can
do
it
with
same
and
we're
exploring.
You
know
we'll
we'll
get
the
rest
of
the
code
out
pretty
shortly.
A
A
Not
you
know
all
the
way
to
the
left-hand
side,
the
way
that
people
design
and
build
software
today
in
machine
learning
and
data
science
and
data
engineering
using
notebooks
and
help
that
move
all
the
way
through
to
your
deployment
platform,
whatever
it
might
be:
kubernetes
cubeflow
airflow,
any
of
the
hosted
providers,
azure,
machine
learning,
sagemaker
and
google
cloud
machine
learning
and
so
on
all
these
various
pipeline
systems.
So
we
we
have
a
really
big
vision
here.
A
The
goal
is,
though,
to
make
things
far
more
portable,
make
it
much
easier
to
capture
your
environment
capture,
your
intent
at
point
of
authorship
and
then
allow
it
to
flow
through
very
quickly
and
use
platforms
like
flux.
You
know
first
class
get
ops
and
and
ci
cd
platforms
that
move
all
the
way
through
and
make
it
very
easy
to
to
roll
out
these
declarative
deployments.
A
One
of
the
things
that
we
are
looking
at
right
now
is
around
different
model
types
and
different
requirements.
You
know
we
think
that
that
you
know
deep
learning
can
often
get
a
lot
of
the
attention.
This
is
certainly
not
just
a
deep
learning
only
thing
one
of
the
things
we
do
see
a
lot
of
is
people.
You
know
getting
something
running
on
their
local
machine
and
having
a
good
time
with
it.
A
But
then,
when
it
comes
to
moving
into
production,
it
requires
this
massive
gap
and
if
we
can
reduce
that
by
by
making
it
easier
to
develop
in
a
production-ready
way
locally,
using
something
like
dot
either
kubernetes
in
docker
or
kind
or
something
like
that,
and
then
when
it
comes
time
to
running
in
production,
the
difference
is
much
smaller.
So
you
know
we
can
be
really.
You
know
optimistic
about
the
amount
of
changes
that
might
be
required.