►
From YouTube: The SAME Project: A Cloud Native Approach to Reproducible Machine Learning - David Aronchick
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The SAME Project: A Cloud Native Approach to Reproducible Machine Learning - David Aronchick, Microsoft
We live in a time of both feast and famine in machine learning. Large organizations are publishing state-of-the-art models at an ever-increasing rate but the average data scientist face daunting challenges to reproduce the results themselves. Even in the best cases, where a newly forked code runs without syntax errors (often not the case), this only solves a part of the problem as the pipelines used to run the models are often completely excluded. The Self-Assembling Machine Learning Environment (SAME) project is a new Kubernetes and Kubeflow project and community around a common goal: creating tooling that allows for quick ramp-up, seamless collaboration and efficient scaling. This talk will discuss our initial public release, done in collaboration with data scientists from across the spectrum, where we are going next and how people can use our learnings in their own practices.
A
Thank
you
so
much
for
the
opportunity
to
come
and
talk
to
the
inaugural
kubernetes
ai
day.
I
have
to
say
it
is
really
astounding
to
see
how
far
this
community
has
progressed
in
the
past
few
weeks
and
months
and
years
couldn't
be
more
impressed
with
how
people
are
using
kubernetes
for
all
their
ai
needs.
A
A
So
this
leaves
users
with
effectively
two
options:
one
they
can
do
it
themselves,
which
involves
setting
it
up
from
scratch
and
integrating
with
legacy
systems
and
eventually,
probably
redoing
the
whole
thing
again,
because
they
need
to
migrate
based
on
their
business
or
two.
They
can
set
up
a
proprietary
solution
which
oftentimes
is
a
lot
more
easy
to
get
going
and
the
first
five
minutes
are
fantastic,
but
then
the
next
five
years,
you
basically
are
doing
all
the
things
you
did
with.
A
Do
it
yourself,
setting
it
up
from
scratch,
integrating
with
legacy
migrating
on
business
needs,
plus
you're,
now
locked
in
so
haven't.
We
heard
this
story
before
the
answer
is
yes:
when,
when
containers
first
came
about,
there
were
a
lot
of
proprietary
solutions
for
orchestrating
your
containerized
workloads
across
your
infrastructure
and
then
came
kubernetes,
which
was
an
open
source
very
broadly
developed
project
which
advanced
the
idea
of
cloud
native
applications.
A
Cloud
native
applications
are
the
idea
that
you
can
simply
and
declaratively
roll
out
a
deployment
of
even
something
complex
that
understands
and
the
orchestrator
takes
care
of
rolling
everything
out
for
you.
So
what
we
really
need
to
do
in
order
to
break
out
of
where
machine
learning
is
today
is
invent
cloud-native
machine
learning.
A
What
does
that
look
like?
Well,
I
think
it
basically
breaks
down
to
three
major
concepts.
The
first
is
composability.
Now
when
we
talk
about
machine
learning
today
we
often
talk
about
just
building
a
model,
but
data
scientists
and
machine
learning.
Engineers
know
that
building
a
model
is
often
just
a
small
subset
of
your
overall
machine
learning
pipeline
coming
from
data
ingestion
and
transformation.
A
A
We
I
talked
previously
about
all
the
software
components
that
are
part
of
it,
but
there
are
oftentimes
many
more
components
when
it
comes
to
rolling
out
an
individual
solution.
In
this
case,
if
I
looked
at
all
the
things
that
I
had
to
configure
as
part
of
my
experimentation
stack,
I
probably
would
have
to
do
all
of
these
and
more,
where
that
existing
pipeline
that
we
had
is
just
a
small
subset
of
them
and
and
really
when
it
comes
to
declaring
all
this
it.
A
You
have
to
go
all
the
way
down
the
stack
and
worse,
you
don't
just
have
to
do
it
in
one
environment
oftentimes.
You
then
have
to
go
and
configure
brand
new
environments,
one
for
training,
one
for
your
cloud
deployment
or
production
deployment
and
so
on,
and
then
we
get
to
scalability
scalability
a
lot
of
times
in
machine
learning,
it's
just
about
more
machines
and
cpus
and
disk
and
networking
and
so
on.
But
true
scalability
even
goes
beyond
that.
A
We
certainly
talk
about
the
number
of
programs
that
are
being
released
or
you
want
to
scale
your
teams
adding
more
skill
sets.
Maybe
you
want
a
suite.
Maybe
you
need
a
data
scientist.
Maybe
you
need
an
ml
engineer,
a
data
engineer
so
on
and
so
forth,
or
scaling
teams
where
you
have
composite
models
or
teams
across
different
organizations
or
maybe
even
going
outside
of
your
organization
entirely
and
at
the
end
of
the
day.
A
You
know
people
just
want
more
programs
faster,
but
there
are
lots
of
ways
to
get
at
that
and
what
we
need
to
do
is
have
a
system
that
supports
that
kind
of
scale.
No
matter
what
scale
is
important
to
you.
So
that's
what
composability
portability
and
scalability
are.
You
know
what's
really
good
at
these
things,
containers
and
kubernetes,
except
if
you
want
to
use
machine
learning
on
kubernetes.
A
Big
companies
solve
this
by
having
a
centralized
dashboard
that
everyone
logs
into
and
a
common
orchestrator
under
the
hood.
But
but
the
problem
is
it
really
still
hurts
with
collaboration
oftentimes.
Even
when
you
cross
internal
organizational
boundaries,
you
might
have
completely
different
version
of
these
stacks,
and
so
what
we
need
to
do
is
build
something
that
gives
it
to
everyone,
even
those
internal
folks
who
may
already
have
sophisticated
internal
pipeline.
A
A
It
must
work
on
your
local
laptop
and
be
compatible
with
your
existing
tools,
but
really
have
portability
as
a
first
class
construct,
allowing
you
to
pick
and
choose
the
components
that
are
important
to
your
pipeline
and
built
to
share
in
open
spaces.
You
shouldn't
have
to
go
and
log
into
some
central
dashboard
in
order
to
have
something
work.
Even
if
this
is
an
external
service,
it
must
work
locally
and
attached,
as
well
as
in
the
cloud,
and
I
like
to
joke
that
our
slogan
is
machine
learning.
A
Without
the
letter
k
right,
a
data
scientist
should
never
have
to
know
kubernetes.
They
should
never
have
to
know
cube
flow.
Docker
docker
has
a
k
in
it.
They
shouldn't
have
to
know
docker
or
containers.
What
we
want
to
do
is
build
an
abstraction
where
they're
able
to
operate
it
in
the
languages.
They
know
and
love,
python
and
r
and
so
and
so
forth,
but
they
are
able
to
then
migrate
that
to
production
in
a
really
elegant
way,
so
how
it
might
work.
The
reason
I
say
might
is
because
this
is
very
early.
A
We've
had
a
lot
of
great
feedback
so
far,
but
we
would
love
to
go
even
further
and
make
better
choices
through
advice
of
the
community.
We
start
with
this
data
scientist
and
she
says:
oh
you
know
here's
a
recommender
out
here
in
the
world
that
I
would
love
to
use.
The
first
thing
that
she's
going
to
do
is
install
a
very
simple
cli.
A
This
cli
is
available
publicly.
It's
actually
already
available
now,
but
it's
very
simple
to
install
and
it'll
download
that
binary
to
her
local
laptop.
We
already
have
builds
for
linux
and
mac
and
we're
going
to
be
adding
windows
soon,
after
that
she
clones
that
that
initial
repo,
again
just
using
standard
git
functionality,
she's
going
to
clone
that
and
have
a
local
copy
of
that
entire
repo
on
her
machine.
A
After
that
she's
going
to
install
a
local
kubernetes.
Now,
in
this
case
we're
going
to
use
k3s,
it's
a
stripped-down
version
of
kubernetes
that
runs
on
local
laptops,
we're
open
to
lots
of
other
options,
mini
kf
or
k3d
or
whatever
works
best.
But
we
think
what
the
most
important
thing
is
that
that
you
do
have
a
local
administrate
light
version
of
kubernetes
that
that
properly
represents
what
it
looks
like
when
you
get
to
production
so
that
you
can
start
doing
a
lot
of
debugging
locally.
A
So
it
goes
through
it
does
that
installation
and
it
also
installs
a
small
crd
that
stands
for
custom
resource
definition.
That
will
allow
us
to
do
very
simple
deployment
of
these
pipelines
to
the
kubernetes
in
the
cloud.
Okay,
so
now
she's
ready
to
go.
The
first
thing
she's
going
to
do
is
type
same
init,
and
this
initializes
that
kubernetes
cluster
that
she
has
locally
and
installs
a
bunch
of
core
components:
the
dashboard
and
the
pipeline.
It
also
installs
a
metadata
store.
Now
it
doesn't
have
to
do
that
locally.
A
A
A
A
After
that,
it's
going
to
go
through
and
download
data
from
an
external
url.
This
is
something
we
saw
very
commonly
where
you
might
use
an
external
csv
or
something
like
that
and
you'll
want
to
make
a
local
copy
of
that.
We'll
do
that
for
you
if
you'd
like
or
will
connect
to
an
external
database.
If
that
makes
sense,
it
then
deploys
the
pipeline
to
the
cluster
and
again,
this
pipeline
is
described
in
code
using
straight
python,
and
it
deploys
it
to
that
kubeflow
pipeline,
using
the
the
integrated
tools
that
kubeflow
offers.
A
A
If
she
wants
to
make
a
change,
she
can
then
rerun
it
in
this
case,
adding
some
additional
run
parameters
and
by
changing
those
run
parameters
she'll
be
able
to
rerun
it.
This
case
experiment
nine,
where
again
it
will
download
both
the
artifacts
and
the
results
into
our
metadata
store
and,
as
you
can
see
there,
it's
starting
from
number
eight
and
number
nine,
and
not
at
zero.
Making
comparison
to
previous
runs
even
ones
that
she
didn't
run
much
much
easier.
A
Okay.
So
that's
a
lot
of
architecture.
Let's
see
this
thing
actually
running
in
production,
okay,
so
to
set
the
context
here.
The
first
thing
we
have
is
a
very
simple
model
that
was
published
by
trey
research:
ai
division.
This
this
model
is
a
housing
model
designed
to
explore
different
ways
of
predicting
housing
prices.
A
The
first
is
that
we
add
some
code
down
here
at
the
bottom
to
do
metrics
recording
you
can
see
that
here,
just
very
simple
that
we'll
be
using
to
output
later,
and
then
you
have
this
single
function
here,
which
is
how
you
coordinate
everything
together
in
this
case
we're
going
to
download
two
steps
before
we
do
a
training
on
them,
pretty
straightforward
stuff,
but
just
using
very
standard
python
to
do
so.
So,
let's
get
started.
A
The
first
thing
we're
going
to
do
like
I
said,
is
a
clone
that
repo
to
our
local
machine,
which
works
fine
from
there
we're
going
to
go
into
that
directory
and
now
we're
ready
to
go
so.
The
first
thing
we
need
to
do
is
install
the
same
binary
in
this
case.
It's
just
a
standard
url
using
the
very
secure
method
of
piping,
a
shell
script
to
bash.
Now
it's
installed
and,
like
I
said
earlier,
we're
going
to
install
kubernetes
by
using
the
same
installer.
Now
I
hope
you
don't
mind.
A
I've
sped
this
up
a
little
for
our
demo
purposes.
Normally
this
takes
maybe
about
five
minutes,
but
it
is
just
a
one-time
thing
in
order
to
get
up
and
running
on
my
local
machine
after
I
do
that,
I'm
going
to
initialize
that
kubernetes
using
the
same
init
function
which
understands
that
it
had
just
installed
a
kubernetes
and
that
everything
is
configured
and
it's
going
to
go
through
and
install
that
for
me
as
well.
So
now
we
have
both
kubernetes
and
kubeflow
up
and
running
and
no
complexity.
A
I
never
had
to
type
cube
control
or
cubeconfig
or
anything
like
that.
It
just
works
again.
Forgive
me
for
speeding
this
up
just
a
little
bit
for
demo
purposes.
Normally
this
also
takes
about
five
minutes.
But
again,
this
is
something
you
only
have
to
do
the
one
time
so
now
that
we
have
that
up
and
running
we're
going
to
move
forward
here,
let's
go
and
run
that
I'm
in
my
directory.
I
have
a
same
file
in
that
directory
and
we're
going
to
execute
it.
As
you
can
see.
A
It
really
is
that
quick
once
I
hit
execute
against
that
it
goes
and
uploads
that
pipeline
to
my
local
deployment,
I'm
going
to
open
that
ui
there
to
my
local
deployment
on
port
8080.
You
can
see
here.
This
is
real,
live
running
code
and
when
I
go
and
click
into
this
ui,
I'm
able
to
see
that
housing
prices
pipeline
that
I
just
uploaded.
A
There's
the
entire
graph
laid
out,
and
it's
already
beginning
to
execute
for
me.
So
when
I
click
on
that
housing
prices
pipeline,
I
can
see
that
run
and
here's
the
graph
that
backs
the
run,
and
when
I
click
over
on
the
configuration,
I
can
see
the
parameters
that
I
used
to
run,
which
just
so
happens
to
be
the
exact
parameters
that
I
entered
on
the
same.yaml
file,
showing
exactly
how
we
can
declaratively
execute
those
runs
from
a
simple
manifest
file.
A
Now,
sometimes
you're
going
to
want
to
execute
using
nothing
more
than
command
line
flags.
This
could
be
because
you're
not
sure
about
the
results,
or
you
want
to
do
things
in
line,
for
example,
during
cicd,
and
same
supports
that
as
well.
Here,
you
can
see
we're
going
to
specify
a
run
parameter
right
there
on
the
command
line
and
re-execute
the
same
pipeline.
We
did
earlier
and,
as
you
can
see
from
the
results,
the
parameter
automatically
came
through
and
you
can
view
it.
A
So
you
have
two
runs
going
simultaneously
different
only
by
their
parameters
that
you
identified.
Finally,
we're
going
to
show
you
how
to
do
things
declaratively
as
well.
Let's
pull
up
that
same
file
again
and
from
there
we're
going
to
make
a
change
in
the
file
itself.
So
now
you
have
a
defined
file
where
you're
making
changes
and
when
I
go
back
and
I
re-execute
that
pipeline
again
with
no
different
command
line
flags.
A
The
system
automatically
picks
up
that
new
changed
same
file
and
executes
using
those
new
parameters.
So
now
I
have
a
declarative
way
via
file
changes
to
execute
new
pipeline
runs
and
when
we
go
over
to
that
dashboard
and
check
out
the
latest
runs,
we
can
see
run
number
three
executing
there
and
when
we
click
on
run
number
three,
we
can
look
at
the
configuration
of
that
and
see
that
our
new
parameter
has
been
picked
up
and
is
running
properly.
A
Okay,
so
we've
executed
from
the
defaults
we've
executed
from
the
command
line
and
now
we've
executed
from
a
changed
file.
Let's
move
this
to
production
now,
typically,
this
might
be
a
pretty
dramatic
move
for
data
scientists
oftentimes.
You
have
to
reset
up
your
entire
environment
in
order
to
get
it
working
properly,
but
in
this
case
we
have
built
into
same
the
the
ability
to
set
up
a
vanilla,
kubernetes
cluster,
whether
or
not
it's
on
your
local
laptop
or
in
the
cloud.
A
So
these
we're
going
to
change
from
pointing
at
our
local
laptop
to
the
cloud.
This
is
the
one
time
you
do
have
to
use
cube
control
and
we
are
looking
at
alternatives
here,
but
once
I
pointed
at
that
cluster
in
the
cloud
I
rerun
that
command
same
init
and
it
automatically
installs
the
necessary
cube
flow
for
this
particular
pipeline
run,
and
once
I'm
done
doing
that
installation,
it's
ready
to
go
so
we're
actually
going
to
go
and
rerun.
Those
three
things
that
you
just
saw
me
do
earlier.
A
A
We
just
execute
using
that
same
file
and
when
we
go
over
to
kubeflow
the
dashboard
to
check
it
out,
we
can
see
that
third
run,
also
executing
and
again
using
the
same
parameters.
We
handed
from
the
command
line,
so
you
can
see
again
three
different
ways
to
execute,
each
of
which
are
declarative
and
are
things
that
you
can
use
in
production.
A
Now
we're
all
set,
and
I
want
to
commit
my
workload
to
the
cloud.
The
first
thing
I'm
going
to
do
is
start
with
a
very
simple
git
command
performance.
Look
good!
Let's
check
it
in!
I
execute
it
and
push
it
to
the
cloud
and
now
very
handily.
I
am
reminded
to
go
check
out
github.
So
let's
do
that
so
we
pop
over
here
to
github,
and
we
can
see
our
tray
research,
ai
division
with
our
brand
new
same
file
that
I
just
pushed
up
now.
A
A
So
forgive
me
for
speeding
this
up.
Just
a
little
bit,
it
does
take
a
little
bit
of
time
to
run.
This
is
a
real
model
running
in
production
on
our
kubernetes
cluster
and
when
it
gets
to
the
end
here,
you
can
see
that
right
there
in
line,
we
have
attached
as
a
run,
parameter
the
commit
sha,
and
this
is
critical
for
all
the
things
that
you've
heard
about
around
lineage
and
tracking
and
so
on
and
so
forth.
A
The
thing
that
we're
really
anchored
on
is
much
less
about
the
individual
components
and
much
more
about
the
user
experience
when
we
go
out
and
talk
to
data
scientists
and
ml
engineers.
The
thing
we
hear
most
about
is
the
amount
of
new
tools
and
techniques
and
and
systems
that
they
have
to
understand
when
they
really
just
want
to
focus
on
portable
reproducible
pipelines,
and
so
from
our
perspective,
we
think
that
user
experiences,
like
you
see
here
on
this
page,
are
critical
to
driving
machine
learning
forward.
A
We
want
one
line
to
install
one
line
to
run
a
program,
one
line
to
change
the
way
that
program
runs,
maybe
it's
via
a
command
line,
flag
or
via
editing,
the
same
file
and
then
one
line
to
export,
so
that
you
can
share
that
with
other
folks
in
the
community.
Now
I
mentioned
the
same
file
several
times.
A
A
Most
of
all,
we,
we
think
that
there's
a
a
real
need
in
the
community
for
what
we
call
kind
of
the
center
of
the
venn
diagram,
where
you
have
something
that
is
completely
open:
source
governed
that
isn't
tied
to
any
single
company
commercial
or
not.
That
is
based
on
a
universal
platform
where
I
can
get
a
you
know,
the
entire
thing
spun
up
on
any
cloud
with
a
single
credit
card
and
then
having
a
declarative
deployment.
A
If
you
would
like
to
help
out,
we
are
just
getting
started.
Obviously
you
can
go,
try
it
out
yourself.
We
have
a
simple
website,
sameproject.org,
where
you
can
download
and
install,
and
you
know
obviously
come
back
and
talk
to
us
on
github
or
on
our
mailing
list,
in
our
slack
all
that
kind
of
stuff,
but
most
of
all
we're
so
early
that
we
really
want
to
hear
about
your
needs.
A
I
said.
We
are
more
than
happy
to
take
contributions.
I
will
say
it's
a
little
bit
early
and
things
are
certainly
moving
around.
But
if
you
see
a
need
for
this
in
your
environment
and
would
like
to
you
know
collaborate
as
we
try
and
build
these
things
out
to
production,
we
would
love
to
hear
about
it,
and
with
that
I
would
like
to
say
thank
you
again.
My
name
is
david
oronczek.