►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Machine Learning on Kubernetes Birds of a Feather - David Aronchick
A birds of a feather meeting discussing the best practices for getting machine learning solutions up and running, and managing them at scale. We will talk about the entire pipeline of machine learning, from data ingestion all the way to serving in production, and every step in between.
To learn more click here: https://sched.co/FuJs
A
So
hi
I'm
Dave
Ron
chick
I'm,
co-founder
of
cube
flow.
We
are
all
about
machine
learning
and
we're
here
to
talk.
This
is
just
a
birds
of
the
feather
I
hope
you
don't
mind.
I
usually
like
to
leave
these
fairly
wide
open
and
have
people
ask
questions
or
see
how
we
clap
you
know
potentially
can
collaborate
together.
A
So
with
that
I,
you
know,
you've
seen
an
amazing
amount
of
great
content
already
about
cube
flow
I
couldn't
be
more
impressed
with
China,
with
the
amount
of
ML
and
AI
happening
on
kubernetes
and
and
we're
really
excited
to
see.
People
come
together
to
start
using
coop
flow
and
in
real
production
environments,
they're
kind
of
a
couple
of
questions
that
we
get
all
the
time
the
first
is
related
to.
What
is
the
roadmap?
Look
like.
Oh
all,
right.
A
What
is
the
roadmap
for
cube
flow?
Look
like
the
big
thing
there
is
that
you
know
we
do
already
see
a
lot
of
people
out
there
using
it
in
production,
despite
my
many
pleased
to
say,
like
hey,
it's
a
little
bit
early
for
this,
and
so
what
we
want
to
do
is
we
really
want
to
get
it
to
be
production,
ready
and
stable,
and
by
that
I
mean
the
underlying
components
are
great.
Tensorflow
is
great.
Kubernetes
is
gray
TF
job
Jupiter.
These
are
all
well
understood,
very
highly
used
functions.
A
Well,
we're
looking
to
us
bring
stability
to
the
API,
and
that
means
we
don't.
We
would
like
not
to
change
it
as
much
as
possible
as
we
get
to
1.0
and
beyond,
and
so
for
us
for
that.
That's
probably
one
of
our
biggest
roadmap
items.
How
do
we
make
sure
that
it's
been
well
vetted
by
people
who
are
using
it
to
make
sure
that
we're
doing
things
properly
so
we'd
love
to
hear
feedback
around
that?
The
second
big
thing
is,
you
may
have
seen
queue
flow
pipelines
just
launched.
A
A
It
took
us
the
better
part
of
a
year,
actually
probably
even
a
little
bit
more
to
get
that
out
the
door
and
it's
now
available
and
now
in
I,
don't
know
how
many
of
you
saw
the
demo
yesterday,
but
you're
more
than
welcome
to
go
online,
see
some
of
our
documentation.
Things
in
in
Python,
straight
Python,
you're
able
to
describe
a
number
of
different
functions,
any
function
you
want
arbitrarily
and
wire
them
together
in
intelligent
ways
where
you
know
one
step
can
do
something.
A
The
second
step
can
take
the
output
of
the
first
step
and
move
on
you
can
spread
spread
steps.
You
can
move
conditionally
all
those
kind
of
things
that's
built
into
the
project
as
well,
and
that's
a
big
roadmap
item,
and
then
the
third
is
really
around
a
lot
of
enterprise
and
and
higher
level
use
cases.
I
am
are
back
upgrades
things
that
every
enterprise
needs.
A
A
A
B
C
A
Yeah
absolutely
so,
first,
let
me
I'll
repeat
the
question.
Second,
let
me
say
that
I
don't
want
to
be
the
only
person
answering
here.
I
very
much
want
to
below
to
be
a
community.
So
if
people
have
particular
thoughts
on
it
or
or
say,
Dave
you're,
an
idiot
you
know
now
is
the
time
to
weigh
in
the
question:
was
the
API
so
I'll
draw
out
an
example?
We
do
hear
this
quite
a
bit.
Are
you
gonna,
standardize
the
api's
meaning
take
serving,
for
example
right?
A
Could
you
come
up
with
a
standard
serving
model
for
your
overall
thing
and
then
every
package
that
checked
in
you
know
that
was
serving
would
have
to
follow
a
certain
API
standard.
So
that's
a
really
interesting
question:
we've
bounced
around
quite
a
bit,
there's
there's,
certainly
a
potential
for
some
things
to
get
pretty
standardized.
To
be
honest
and
it's
less
about
I
I
know,
I
threw
out
serving
the
one
that
might
be
the
most
standardized
that
we
come
up
with
might
be
the
the
operator
right.
A
Where
right
now
we
have
tensorflow,
we
have
something
you
know
cafe
being
worked
on.
Pi
torch
is
already
in
MX
net.
Has
an
operator
there's
an
extra
boost
operator.
There's
a
variety
of
things:
I
think
it's
an
operator
I'm
actually
not
sure,
but
regardless,
like
that's
something
where
there
really
is
a
lot
of
value
in
having
some
degree
of
norm,
I'm,
not
gonna,
say
a
standard.
If
someone
wants
check-in
an
operator
that
doesn't
comply,
we're
you
know
in
the
project.
A
We're
not
gonna
say
no,
but
if
you
check
in
and
if
it
follows
that
standard,
then
something
like
Kat
tube
will
just
work
or
something
like
you
know.
Packaging
your
model
for
distributed
training
will
just
work
and
that's
really
what
we
want
to
do.
So
we
want
to
pull
that
through.
It's
not
quite
you
know,
locking
people
out
if
they
don't
do
it.
What
I
would
really
love
is
to
have
people
own
those
standards,
so
that
means
like
if
someone
would
like
to
come
along
and
define
the
operator
standard
or
define
the
serving
standard.
A
That's
great
but
I,
don't
think
we
will
ever
get
to
a
point
where
we
will
take
that
standard
and
go
to
and
I'll
use
serving
again
for
an
example
we'll
never
go
back
to
you
know
ten,
so
RT
or
Selden
or
TF
serving
or
flask,
or
anyone
out
there
and
say
well,
you
know
now,
we've
all
come
up
with
this.
Like
brilliant
model,
you
better
do
it
this
way.
No
I,
don't
think
we'll
get
there.
A
A
A
One
thing
I
will
say
that
I
want
to
standard
across.
Almost
everything
is
for
common,
get
offs
and
and
monitoring
and
observability
and
other
things
you
know,
for
example,
I
really
want
to
make
it
a
standard
that
you
know
at
Google.
Every
process
in
the
entire
company
responds
to
you
know
whack
Varzi,
and
it
spits
out
all
the
variables
that,
if
your
internal,
it
spits
out
all
the
variables
for
that
particular
service.
So
that's
a
standard
I
would
love
to
establish
across
everything
again
we're
not
gonna
block
people
of
don't
do
it.
D
A
A
Absolutely
so
we
know
Chris
really
well
he's
great.
You
know,
I
think
that
that
pipeline
AI
is
broader.
They
you
know,
can
take
cube
flow
and
make
it
work.
But
you
know
I
I
think
that
they
they
have
a
little
bit
less
opinionated
Ness
to
the
underlying
platform
than
we
do
they
they're
they're,
much
broader
and
they're.
A
You
know
he
has
business
where
he's
providing
like
higher
level
functionality
and
solutions,
and
things
like
that
and
and
that's
really
what
we're
trying
to
do
pipeline
is
great,
but
it
requires
Chris
and
the
community
to
go
all
the
way
down.
The
stack,
a
single
community
that
has
to
own
everything
up
all
the
way
up
and
down.
We
and
cute
flow
have
made
in
another
choice
and
and
I
like
our
choice,
but
again
we're
open
to
feedback
which
is
hey,
we're
actually
not
gonna
care
at
all
about
the
bottom
layer.
A
Kubernetes
you
take
care
of
it.
We're
not
going
to
go
out
and
try
and
implement
a
new
storage
solution.
We're
not
gonna,
go,
try
and
implement
a
scheduling,
algorithm,
we're,
not
gonna,
go
and
try,
and
you
know
re-implemented
Erb
you
to
training
anything
like
that.
We're
just
going
to
leave
kubernetes
you
go.
Do
that
and
then
we're
in
the
middle
with
cube
flow,
a
set
of
norms
and
solutions
around
packaging,
including
workflow
orchestration
between
those
and
then
we
also
say
we're
not
going
to
go
up.
A
Were
that
we're
saying
like
we're,
not
go,
implement
a
new
machine
learning
framework
we're
not
going
to
implement
a
new
data
transformation
framework.
We're
not
gonna
implement
is
servingframework.
We're
just
gonna
take
what's
already
out
there
and
package
it
so
that
or
encourage
them
to
package
it
ideally
and
have
it
run
inside.
So
that's
our
philosophy.
We
don't
want.
We
actively
don't
want
to
do
all
those
things
we
just
want
to
focus
on
kind
of
our
narrowband,
but
you
know
everyone
has
a
different
solution,
and
so
you
know
I.
A
You
know
we
know
Chris,
really
well
and
and
I
I
believe
he's
already
if
he
hasn't
released
it
yet
I
hope
I'm,
not
giving
anything
away,
he's
already
releasing
how
to
run
cube
flow
inside
pipeline
AI.
So
just
think
of
it
as
pipeline
pipeline
is
a
larger
thing
that
that
some
people
may
need,
and
some
people
may
not,
but
cube
flow
is
just
a
platform.
B
E
A
A
We
know
about
integrating
with
external
services
things
like
taking
your
model
and
pushing
it
up
to
the
cloud
pushing
it
to
IOT
things
like
that
pipeline
a's,
pipeline
cue
pipelines
is
designed
to
not
require
everything
to
run
inside
cube
flow.
You
can
pick
and
choose
and
and
wire
things
together
through
standard
api's,
and
there
are
there
already
examples
in
the
queue
flow
repository
to
show
you
how
to
do
that.
What
else.
C
E
So,
there's
just
a
couple
more
items
that
we
can
talk
about
that
are
on
the
roadmap,
so
one
of
the
big
things
that
we're
working
on
is
notebooks.
So
one
of
the
things
that
we
hear
a
lot
is
coop
flow
and
kubernetes
are
too
hard
for
data
scientist.
So
we're
really
trying
to
create
a
great
experience
with
using
notebooks.
So
you
can
train
and
deploy
a
model
directly
from
a
notebook.
So
that's
a
lot
of
work,
that's
being
done
by
a
ripto
and
folks
at
Intel
they're
contributing
that
effort.
E
So
we
want
to
have
like
a
CR
D
to
make
it
really
easy
to
manage
notebooks
on
kubernetes.
Other
things
that
we're
doing
is
we're
focused
a
lot
on
Enterprise
scenarios.
So
a
lot
of
the
requirements
we
hear
a
lot
from
companies
is
they
want
to
run
multiple
users
in
the
same
cluster,
but
have
them
be
isolated
from
one
another?
So
you
know
one
user
can
submit
TF
jobs
or
notebooks
and
not
have
them
be
interfered
with
by
another
user.
E
That
makes
them
work
well
and
then
we're
always
focused
on
just
simplifying
the
deployment
experience,
making
it
a
lot
easier
to
deploy
on
a
variety
of
platforms
and
set
up
all
of
your
infrastructure.
So
those
are
some
of
the
key
things
that
are
working,
we're
working
on
right
now
in
our
roadmap.
Oh
I
guess
the
other
big
one
is
like
inference:
we're
trying
to
create
a
really
great
story
about
rolling
out
models
and
taking
advantage
of
sto
to
do
things
like
split
traffic
between
new
models.
B
A
I
have
a
deep
and
powerful
passion
around
this
and
I
say
it
all
the
time-
and
this
is
one
of
these
things
where
I
keep
dreaming.
If
I
keep
saying
the
words
eventually,
it
will
manifest
itself
into
reality.
What
I
would
like
to
see
is
a
simple
template
that
the
data
science
community
gets
together
and
describes
right
and
it's
it
can
be
something
really
really
basic
in
it.
You
would
have
three
directories.
You
would
have
a
directory
that
describes
your
cube
flow
deployment.
A
You'd
have
a
directory
that
describes
your
machine
learning
or
your
pipeline
and
then
you'd
have
a
directory
that
literally
contains
some
sample
data
and
that
that
that
framework
will
have.
Each
of
those
directories
will
have
some
metadata
and
some
other
files
in
it,
but
that's
something
you
can
share
literally
via
git,
and
that
would
be
incredibly
powerful
that
you
know
my
dream.
Is
you
go
to
you?
A
A
You
know
single
entity
story,
I
I,
believe
that
that
is
a
great
story,
and
we
should,
if
you
want
to
help
me
with
this.
Please
come
and
mail
me,
but
to
your
point,
there's
a
second
half
of
it
and
I
would
be
remiss
if
I
didn't
talk
about
AI
hub,
which
is
something
just
launched
from
Google,
which
is
designed
to
be
a
sharing
repository.
B
E
I
think
there's
there's
two
efforts
that
I
can
mention
that
sort
of
exist
today.
So
one
is
a
ripto
is
make:
is
a
company,
that's
building
a
storage
product
and
what
they
basically
do
is
they
have
a
container
storage
interface
plugin
that
allows
them
behind
that
they
have
a
hub
for
data
sets
and
and
data
that
they
can
basically
snapshot
using
peer-to-peer
networking,
and
then
they
can
get
mounted
by
a
PBS
and
PBC's
through
the
CSI,
so
they've
integrated
that
with
Jupiter
and
I've
indicated
that
into
the
cupola
story.
E
So
what
they
can
do
is
when
they
spawn
notebooks.
You
can
specify
you
want
various
volumes
where
your
data
will
be
snapshotted
for
both
your
notebooks
itself,
as
well
as
your
data.
They
then
have
this
hub
product
from
where
you
can
replicate
those
networks.
Those
I
forget
what
they
call
them,
I
think
they're
snapshots
or
whatever.
So
you
can
actually
take
that
data
and
make
them
available
to
somebody
else
in
your
organization
by
a
snapshot,
there's
a
PV
that
they
can
then
mount.
The
other
thing.
That's
really
cool.
E
It's
not
part
of
coop
flow,
but
it's
really
worth
checking
out.
Is
what
Netflix
has
done
with
notebooks
they're
they're
interact
efforts
so
they've?
Actually
they
use
Jupiter
the
runtime,
but
they've
replaced
the
front
end
with
a
react
based
notebook
for
a
react
based
web
app
that
serves
as
the
front
end
for
the
notebooks,
and
this
allows
them
to
do
a
lot
of
cool
things.
So
they
can
actually
store
inside
the
the
notebooks
that
the
JSON
file
itself,
data
to
power,
plots
and
visualizations
or
end
results
in
the
notebook.
E
So
they
can
then
take
that
notebook
and
effectively
just
snapshot
it
to
get
you
a
completely
sort
of
hermetic
result,
the
document
sort
of
the
code
in
the
results,
and
then
you
can
snapshot
it
to
you,
know
a
bucket
store
or
something
else
and
I
believe
they're
gonna,
eventually
release
some
I
think
called
I,
don't
know
what
it's
called,
but
some
product,
hopefully
to
browse
all
those
notebooks.
So
you
can,
you
know,
see,
see
and
share
artifacts
using
the
notebook,
so
that
would
be
really
cool.
So.
A
The
net
is
I
totally
agree
and
there's
a
lot
of
work
around
this
I
think
the
the
challenge
has
been.
Even
if
you
share
a
single
you
know
notebook
file,
even
that
is
hard
to
reproduce
right
and
that's
that's
part
of
the
the
impetus
behind
cube
flow
like
how
do
you
declaratively
describe
you
know
your
deployment
containerized
with
versions
and
all
that
stuff
and
also
declare
declaratively
describe
your
entire
transformation.
F
Not
using
in
production
yeah
we're
just
wondering
if
you
have
a
team
of
data
scientists
right,
the
problem
was:
how
do
you
share
notebooks
across
so
besides,
sharing
notebooks
like
we
want
to
like
look
through
other
people's
notebook.
So
what
I've
asked
my
peers
to
do
is
to
generally
docker
eyes
the
whole
environments.
So
then,
you
just
have
to
pull
the
whole
environment
from
the
registry,
and
you
can
recreate
this.
Is
there
a
accompanying
flow
similar
in
cubes?
Oh
yeah,.
A
That
that
is
exactly
what
cube
flow
is
designed
to
do.
The
reason
why
we're
not
a
fan
of
a
single
docker
container
is
that
doesn't
shard
very
well
and
it
really
doesn't
work
in
production.
Not
the
doctor
doesn't
work
in
production
doctors
great.
But
if
you
went
to
any
ml
engineer
and
said,
here's
this
six
gigabyte,
you
know
docker
container
that
you
have
to
do
exactly
and
oh
by
the
way,
every
time
I
make
a
single
update
to
a
single
library,
you
got
to
smash
the
whole
thing.
That's
not
so
good!
A
That's
not
really
cloud
native
you're,
just
you
might
as
well
be
building
VMs.
The
idea
in
cube
flow
is
you
you
have
a
single
yamo
descriptor
file
or
you
have
a
single,
you
know
case
Anette
file
or
whatever
it
might
be,
and
you
just
hand
that
to
the
kubernetes
master
and
the
kubernetes
master
will
pick
that
up
and
and
instantiate
the
entire
environment
for
you
with
all
the
dependencies
and
tying
everything
together.
A
Those
commands
are
just
as
easy
as
they
would
be
with
a
sing
dockerfile,
except
now
it's
production-ready,
because
it
is
broken
down
into
these
micro
services
and
and
you're
able
to
edit
and
tweak
and
use
you
know
many
nodes
and
so
on
and
so
forth
that
make
sense
yeah,
and
we
have
lots
of
examples
in
our
repo.
If
you'd
like
to
go
check
out
those
examples
and
see
what
we're
doing
and
tell
us
their,
they
don't
make
sense.
You
can
do
that
too.
A
C
A
Know
so
so
after
I
get
my
little
capsule
of
you
know
what
a
job
looks
like
done.
The
second
thing
I'd
like
to
do,
or
maybe
in
parallel,
is
come
up
with
a
standard
form
for
a
notebook,
and
by
that
I
mean
what
would
a
notebook
look
like
that
was
very
easy
to
parse
into
a
distributed
job
runner,
because,
right
now
it
really
the
you.
You
pointed
out
something
and
the
question
was:
is
there
a
standard
for
what
your
notebook
file
should
look
like
in
order
to
move
it
to
a
distributed?
A
Tf
job
and
the
reality
is
there's
no
standard.
There
are
a
lot
of
tools.
What's
that
one
and
be
something
edit,
some
n
be
convert
that
that
moves
it
into
a
file
that
it
that
can
be
consumed,
but
that's
very
coarse.
What
I
would
really
like
is
standards
for
annotations
standards
for
hypo
parameters.
Standards,
for
you
know,
hyper,
parameter
search.
A
E
No
standard
today,
but
we
are
working
on
two
to
make
it
really
easy
to
go
from
a
knocker
notebook
to
running
a
distributor
job
on
kubernetes,
right
and
so
I
think
the
credit
really
goes
here
to
lift.
They
showed
us
what
they've
done
and
they
have
a
really
great
experience
and
so
we're
basically
just
sort
of
copying
and
mimicking
them.
E
Do
a
little
bit
of
parsing
to
get
the
code
out,
then
we'll
actually
build
the
docker
image
for
you
and
then
we'll
launch
a
TF
job
for
you
on
the
cluster
it'll
provide
like
a
Python
interface
library
that
you
can
basically
use
from
the
notebook
as
well
as
a
widget.
So
you
can
do
things
like
annotate
your
model
with
hyper
parameters
and
then
from
a
widget,
say:
I
want
these
hyper
parameters.
So
that's
that's!
G
A
Thank
you
absolutely
and
again.
We
are.
We
really
do
try
and
be
open
about.
The
project
feel
free
to
ask
these
questions.
I
think
I'm
out
of
time
here,
but
feel
free
to
ask
these
questions
or
more
and
I
think
we
have
time
for
one
more
question,
but
right
directly
in
our
github
and
we'd
love
to
you
know,
get
feedback.
A
C
A
So
we
we
already
do
have
quite
a
large
community
which
we're
really
excited
about
and
and
we're
happy
to
help
you
were
slack,
there's,
email
and
so
on
and
so
forth.
I
think
my
number
one
would
be
start
with
something
like
katha,
Kota,
I,
think
they're,
amazing
resource
they
have
a
website.
You
just
go,
you
start
entering
commands
and
it
shows
you
how
to
set
it
up
and
run
a
very,
very
simple
job,
all
very
straightforward
in
your
browser,
so
you
install
literally
nothing.
You
can
also
go
to
our
documentation,
go
to
cube
flow
org.
A
We
have
getting
started
manuals
and
things
like
that.
All
you
need
is
a
mini
cube
which
is
free.
You
can
download
it
you
or
docker
which
has
kubernetes
built
in
for
your
desktop.
Once
that's
installed,
you
can
start
cube
flow,
very,
very
trivially
and
their
documentation
and
tutorials
to
walk
you
through
that.
You
can
also
start
on
on.
Almost
every
cloud
provider
has
a
you
know,
one-click
deployment
of
kubernetes
again
that
should
work
just
fine
and,
and
you
know
the
idea
is
that
cube
flow
is
designed
to
be
as
easy
and
we're
not
there.
A
A
So
it's
not
just
I'm
gonna
install
this
package,
but
now
I'm
gonna
go
and
and
download
this
entire
tutorial
and
deploy
it
to
my
local
machine
or
to
deploy
it
to
a
cluster
and
get
up
and
running,
and
the
tutorials
are
out
there
to
do
that
and
and
if
you
have
trouble
with
those
tutorials,
that's
a
p0
for
us.
So
we
want
to
fix
them
and
then
Ivan
is
our
is
graciously
contributing
as
time
to
the
cube
flow
p.m.
group.
We
do
all
our
product
management
in
public.
You
should
come
and
join
us
every
week.
A
We
do
a
p.m.
call
half
the
time.
It's
on
Asian
time,
half
the
time.
It's
on
European
time
you
can
come.
You
can
weigh
in
we're
currently
in
the
midst
of
planning
0.4
or
that's
done
somewhere
around
that,
and
you
can
literally
help
us
direct
the
product.
So
it's
open
to
you
any
I'm
around
you
can
ask
questions
or
whatever
you
like.