►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Kubeflow BoF - David Aronchick & Jeremy Lewi, Google
Kubeflow is a Kubernetes native platform for machine learning. This is a deep dive into Kubeflow aimed at individuals and organizations that are already contributing to Kubeflow or thinking about contributing. We will discuss the current state of the community and our roadmap.
To learn more: https://sched.co/Ha1X
A
A
A
I'm
Dave
Ron,
chick
I
helped
with
start
cue
flow
along
with
my
partner
in
crime
here,
Jeremy
Louie,
so
I'm
gonna
ask
a
bunch
of
question
or
ask
you
to
drive
this
and
we're
happy
to
answer
questions,
but
first
I
want
to
hand
off
to
Jeremy
who
will
give
a
high-level
of
some
of
the
items
are
around
community
that
just
a
context
set
for
everyone.
Yeah.
B
So
I'll
keep
this
very
brief,
so
just
to
just
estate
briefly,
you
know
this
is
our
mission.
We
want
to
make
it
easy
for
everyone
to
develop,
deploy
and
manage
portable
distributed
ml
and
kubernetes.
That's
the
goal
of
coop
flow.
Here's
a
quick
sense
of
how
our
community
is
doing
right.
So
this
is
basically
the
AXYZ,
probably
don't
show
up
on
the
graph
too
well,
but
this
is
basically
the
number
of
unique
PR
authors
per
month,
and
so
just
to
give
you
the
the
high-level
summary
this
has
roughly
doubled.
B
So
we
have
about
80,
80
PR
authors
in
the
most
recent
month
compared
to
about
46
months
six
months
ago,
so
we're
seeing
about
2x
growth,
the
number
of
contributors
over
the
past
six
months.
So,
thank
you
all
very
much.
Hopefully
we
can
continue
to
grow
that
and
then
just
to
hopefully
spark
some
of
the
discussion
today.
I
thought
it'd
be
flea,
be
good
to
just
briefly
go
over
our
roadmap
I'm.
So
this
is
something
we
just
recently
published
in
sort
and
the
github
cupola
repo.
B
B
So
there's
a
whole
bunch
of
items
that
are
related
to
that,
like
our
back,
I
am
etc,
as
well
as
support
for
things
like
post
hooks
file
systems
and
then
a
big
one
is
getting
to
a
one:
dot
o
and
stable
release
process
for
upgradability
I'm,
not
something
enterprises
care
a
lot
about
the
deployment
experience
we
want
to
make
it
super
easy
for
anybody
to
get
started
with
Kubb
flow.
You
know
with
one
click
or
one
command.
You
know
a
lot
of
times.
B
What
we
hear
is
you
know,
data
scientists
are,
you
know
here
about
coop
flow
to
conference
or
some
other
venue
like
a
tutorial
if
they
can't
get
started
really
quickly
with
it
and
start
seeing
the
value
of
it.
It's
there
they're
just
gonna,
go
away
and
try
something
else.
So
it's
really
really
important
to
have
a
smooth
experience.
That's
really
important
to
us
development
experience.
B
You
know
that's
sort
of
the
leadcore
or
what
we
call
coop
flow
in
terms
of
having
you
know,
different
components
for
all
the
different
problems
in
the
ml
lifecycle
for
a
coop
flow,
and
so
you
know
we
want
to
make
it
easier
to
develop
ml
solutions
and
get
those
into
productions.
Oh
I,
guess
that's
what
I
was
calling
a
advanced
internal
platform,
so
that
was
I
got
those
confused.
What
was
I
oh
so
for
development
experience?
Well,
we
here
is
data.
B
Scientists
are
very
keen
to
be
able
to
use
notebooks,
that's
all
the
rage
these
days,
so
we
want
to
have
a
really
great
integrated
notebook
experience
for
for
data
scientists.
So
I've
already
talked
about
the
advanced
ml
platform,
and
then
this
is
just
a
release
cadence.
So
we're
really
striving
hard
to
get
a
release
out
the
door
once
every
quarter.
So
we've
done
really
well
with
this
for
this
past
year.
B
A
You
so
much
quick
show
of
hands
before
we
get
to
questions.
How
many
of
you
have
installed
cube
flow
at
all,
unbelievable,
how
many
of
you
are
using
it
in
any
form
of
production,
way,
training
or
actually
doing
something
with
it
other
than
just
experimentation.
Anyone,
oh
very
nice,
all
right
good
for
you,
you're
all
bleeding-edge
much
much
braver
than
I
yeah
I
mean
I.
I
run
models
myself,
but
alright,
so
this
is
supposed
to
be
driven
by
the
community
have
at
it
questions
comments,
points
of
view,
things
that
we
should
work
on.
A
Yeah,
so
the
problem
is
a
multi-tenancy,
so
the
question
and
you
don't
have
to
walk
around
I,
couldn't
hear
it
and
I'll.
Just
repeat
the
question:
if
you
want
or
you
can
use
mic
so
the
question
was
about
multi-tenancy,
the
problem
is,
is
multi
tendency
means
different
things
to
different
people
right,
and
so
some
people
are
okay,
with
namespace
isolation
as
being
sufficient.
Some
people
want
full
hypervisor
isolations.
Some
people
want
separate
cluster
isolation,
even
if
you
have
like
hardcore
hypervisor
that
doesn't
even
supply
a
lot
of
multi-tenancy.
What
if
you
have
shared
disk?
A
What,
if
you
have
shared
file
systems?
What,
if
you
have
shared
networking,
etc,
etc?
I
think
at
a
high
level
are
our
minimum
viable
product
for
multi-tenancy
would
be
to
data
scientists
can
log
into
a
single
kubernetes
cluster,
with
cube
flow
running
on
it
and
do
stuff
and
not
screw
each
other
up
that
that
is
basically
the
minimum
right.
A
You
know
even
within
that,
it's
like
well
what,
if
they
both
mutate,
the
same
data
source
all
right,
obviously
we're
not
gonna
prevent
that
right,
like
there's
just
some
things
that
we
can't
prevent.
But
but
at
that
level
we
I
think
would
be
very
ecstatic,
and
so
you
look
at
something
like,
for
example,
auto
scaling
and
queue
flow.
A
I
I
think
that
we
actually
may
reuse
the
same
node
pool
if
it's,
if
they're
not
full,
but
you
get
the
idea
like
I,
think
that
would
be
kind
of
an
MVP
around
that.
If
you
would
like
to
you
know
start
a
short
proposal
around,
like
you
know,
priority
0
through
n,
for
what
multi-tenancy
looks
like
I,
think
that
we've
good
and
it'd
be
a
good
discussion
to
have,
but
but
at
a
minimum,
I
think
that
we
are
aiming
for
that
that
experience
to
data
scientists
not
score
each
other
up.
C
D
A
So
the
question
was:
will
q
flow
support
champion
challenger,
sophisticated
deployments?
I
assume
that's
just
in
serving
or
do
you
want
it
like
across
the
board?
Okay,
so
interesting,
question
I,
think
I
have
semi
good
news
and
then
it's
up
to
you
so
we'll
go
from
there.
The
semi
good
news
is
I.
Think
sto
probably
supports
a
lot
of
what
you're
looking
for
today
and
we
already
have
started
to
integrate
sto
into
the
overall
system.
A
My
recommendation
would
be
do
as
little
work
as
possible
in
an
automated
way
until
we
start
seeing
what
people
are
using
q
flow
to
do
is
especially
relative
to
deployments,
and
things
like
that,
but
I
would
not
expect
us
to
have
that
in
the
platform.
That's
my
personal
take
again.
This
can
be
community
driven
if
some
wants
to
dive
in
and
work
on
it.
That's
great,
but
I
wouldn't
expect
us
to
have
it
in
the
platform
until
we
reach
some
level
of
stability
and
and
so
on.
A
Yeah
is
there
any
line
of
sight
for
feature
caching
data,
caching,
things
like
that
again:
I,
don't
I,
don't
think
we
have
a
line
of
sight
to
that.
The
direct
answer
is
I.
Don't
think
we
have
that
right
now.
That
said,
you
know.
Just
yesterday
we
were
talking
to
some
folks
who
are
currently
doing
some
really
creative
things
around
serving
and
whether
or
not
they
would
eject,
for
example,
data
containers
into
the
serving.
A
So
you
could
have
that
Jason
or
maybe
extending
TF
job
to
similarly
like
pre
cache
data
and
things
that
you
know
that
the
platform
is
wide,
open
and
I.
Think
we
would
take
anything
of
that
kind.
But
the
reality
is:
is
that
for
right
now
getting
that
that
core
experience
complete
is
probably
our
number
one
priority.
A
E
A
A
Let's,
let's
form
a
proposal,
write
it
down,
so
that
people
who
are
not
in
this
room
can
see
it
as
well
and
have
an
issue
that
we
discuss
on
the
the
point
that
I
heard
and
correct
me
if
I'm
wrong
is
for
those
that
haven't
done,
ML
models
or
training,
and
things
like
that.
Obviously
moving
data
around
the
network
is,
is
very
complicated
and
and
often
very
slow.
If
you
can,
as
you
spin
up
your
particular
pod
on
a
given
node,
also
pre
cached
data
locally,
that
generally
increases
the
speed
of
that
particular
training.
F
A
Perfect
so,
and
just
so,
everyone
heard
if
they
didn't
the
problem
is,
is
that
and
let
me
see
if
I
summarized
it
properly?
The
problem
is:
is
that
data
scientists
and
production
often
come
that
the
data
is
coming
from
inconsistent
sources,
potentially
analytical
data
versus
stream
data
versus
other
things,
and
they
want
to
develop
a
system
to
build
the
features
on
top
that
makes
it
consistent,
regardless
of
where
are
the
sources
and
potentially
it
aids
in
pre,
caching
and
performance,
and
things
like
that
again
that
that's
a
wonderful
use
case.
A
A
Yeah
yeah,
so
the
question
is
about
governance.
Governance
is
in
flight
right
now
the
community
is
very,
very
wide
open.
Our
goal
is
to
be
you
know.
From
from
day,
one
we've
always
tried
to
say,
like
we
really
like
the
kubernetes
community,
we're
gonna,
try
and
look
at
that
as
a
model
for
how
we
potentially
move
forward.
This
is
something
where
again,
the
more
that
you
can
raise
your
hand
and
say:
look
I
I
would
like
a
governance
model
like
this
or
that
you
know
that
the
folks
are
there
and
ready
to
listen.
A
The
only
reason
we
haven't
you
know
we're
just
trying
to
get
it
written
down
and
out
the
door.
It
is
absolutely
our
goal
to
you
know
make
this
a
community
driven
project
with
you
know
not.
You
know,
as
Jeremy
had
up
there
already
more
than
50%
of
people
are
not
from
Google.
Those
folks
want
to
participate
and
they
want
to
have
similar
protections
for
IP
and
contributions,
and
things
like
that
and
everything's
under
the
Google
CLA
right
now,
which
is
how
kubernetes
ran
for
more
than
two
years
I
think.
D
B
Just
a
little
bit
of
the
logistics.
So
if
you
look
in
the
community
repo
for
coop
flow,
we
have
a
specific
process
that
we've
defined
for
how
to
submit
proposals.
It's
pretty
lightweight
and
you
know
you
basically
open
up
a
github
issue
and
say
I
think
we
should
do
X
and
then
people
sort
of
chime
in
and
everybody
says
we
should
do
X.
B
Then
we
can
either
sort
of
add
it
to
an
existing
repo
if
it
makes
sense
so
we
can
create
a
new
repot
repository
if
it's
necessary
and
then
in
terms
of
you
know,
sort
of
code,
reviews
and
approvers.
We
have
like
four
different
levels:
I
think
it's
a
community
member
contributor,
I'm,
forgetting
there's
a
maybe
we
only
have
three
there's
like
committer
and
approver,
and
then
anybody
can
attain
those
levels.
B
So
it's
not,
and
you
just
have
to
sort
of
demonstrate
that
you
understand
the
code
and
can
approve
other
people's
PRS,
and
then
anybody
can
review
other
people's
PRS
and
we
sort
of
encourage
that
as
a
way
of
demonstrating
your
involvement
and
your
understanding
right.
You
can
do
drive-by
reviews
and
there's
a
it's
all
documented
in
the
contributing
to
coop
flow
guide.
D
B
You're
talking
about
feast
go
check
yeah,
so
so
so
look
so
the
way
this
so
you
know
go
jack
is
basically
developing
a
future
store.
They've
been
developing
it
for
themselves.
It's
that
I
think
they
just
released
the
code
open
source
in
one
of
their
repos,
so
you
can
go
check
it
out,
but
basically
they're
going
to
go
through
the
process.
I
just
mentioned,
where
they're
going
to
submit
a
proposal
to
open
it
up
in
the
coop,
so
community
that'll
say
like
oh
we've
done
feast.
We
think
this
is
what
the
problem
is
solving.
B
Sorry
I
mean
the
integration
with
other
components
is
something
that
really
sort
of
has
to
depend
on
a
you
know,
component
by
component
basis.
Right
so
like
does
it
make
sense
to
integrate
the
feature
store
with
tensorflow
job
controller?
I,
don't
know
alright,
so
we'll
have
to
see,
but
I
think
it
would
make
almost
certainly
make
sense.
You
know
to
integrate
it
with.
A
That
does
bring
up
an
interesting
question
and
you
know
to
some
degree
a
lot
of
people.
Look
at
us
and
say
like
well.
Are
you
what's
in
cube
flow
right
and,
and
the
reality
is,
is
there
are
some
components
that
we
consider
a
cube
flow
to
be
broken?
If
it's
not
there
right,
you
know
if
you
can't
run
at
Jupiter
notebook
like
it's,
not
gonna,
work
right,
and
if
you
can't
do
some,
you
know
fairly
straightforward
training
and
so
on
and
so
forth.
A
We
consider
you
know
pipelines
to
be
part
of
that
core,
like
we've,
always
envisioned
using
Argo,
which
is
there
as
well
as
a
core
orchestration
engine
for
your
workflow
and
move
on,
but
but
beyond
that
I
think
as
a
community,
it's
up
to
us
to
define
what
we
consider
it
to
be
broken
and
for
better
or
worse.
Today,
if
you
don't
have
a
feature
story,
you
can
still
get
a
lot
of
stuff
done,
so
I
probably
would
say
that
it's
probably
optional,
but
I
want
to
be
community
driven.
A
So
if
we
say
you
know
what
this
is
not
the
platform
that
we
want,
we
want
it
to
be
complete
platform,
and
that
requires
a
future
store.
Then
great,
we
should
go
forward
and
do
it.
That
does
lend
to
a
second
thing.
I
can't
tell
you
the
number
of
components
I
have
heard
developed,
building
on
cue
flow,
using
cue
flow
thinking
about
using
cue
flow,
so
on
and
so
forth.
They're,
wonderful
and
I
want
them
all
right
like
they
really
do.
A
You
should
think
about
your
component
as
being
optional
like
have
it
or
don't
have
it
think
about
defining
api's
and
and
and
think
about
how
it
might
integrate
with
other
core
components
or
or
extension
components
to
keep
flow,
but
more
often
not
my
biggest
ask
of
all
of
you
and
and
if
you
see
something
as
you
are
like
geez
there's
this
thing
missing.
What's
going
on
here,
I
know
as
developers
and
engineers
and
so
on,
we're
like
well
I'm,
just
gonna
go,
build
it
and,
and
that's
great
I
want
you
to
go,
build
it.
A
But
let
me
suggest
to
highlight
to
the
community:
like
hey
everyone,
I
think
that
this
is
a
gap.
How
about
we
work
on
this
I'm
happy
to
work
on
it?
Do
you
mind
like
here's,
some
thoughts
I
have
about
it.
I
think
you
will
get
people
helping
faster
I,
think
you
will
get
people
thinking
about
it.
I
think
you
will
get
good
feedback
about
design
decisions
and
ensure
it's
not
as
fast
as
me.
A
Hacking
it
together
over
an
evening,
but
I
think
it
will
help
get
things
more
integrated
more
quickly
and
get
a
lot
of
help
and
you
know
ideally
reduces
duplication
of
effort,
but
but
that
is
neither
here
nor
there.
If
you
want
to
go
hack
on
it,
you
can,
but
I
would
implore
you
as
a
strong,
like
I've,
seen
many
strong
communities
form
and
it
almost
always
happens.
When
you
know
multiple
people
from
multiple
organizations
work
together,
yep.
A
So,
okay,
so
the
question
was:
do
you
have
plans
in
cube
flow
to
include
spark
operator
or
a
flink
operator
or
other
ways
to
do
data
cleansing
and
so
on?
The
direct
answer
is
I
think
we
already
have
something
but
set
that
aside.
For
the
moment,
it
has
always
been
the
intent
of
cube
flow
that
not
everything
needs
to
run
on
your
cluster.
That
has
always
been
the
intent.
A
We
fully
fully
fully
expect
that
you
will
have
massive
external
data
clusters,
or
data
lakes
here
or
storage
over
there
or
maybe
you're
serving
endpoint
is
hidden
behind
some
API.
Who
knows
what
that's
the
whole
vision
and
why
pipelines
is
so
powerful
because
it
uses
kubernetes
native
micro
service-orientation
to
pick
up
endpoints,
no
matter
where
they
are
and
are
go
under.
The
hood
has
a
very
rich
eventing
system
that
can
hook
into
things
that
are
external
to
the
cluster,
and
so
the
net
is.
A
Is
that,
while,
yes,
there
are
like
I,
said
I
think
we
already
have
some
a
spark
operator
in?
If
you
don't,
you
can
actually
run
it.
I
mean
like
obviously
it's
a
fairly
trivial
using
how
I'm
charged
or
something
like
that,
so
you
can
run
it
on
the
same
cluster,
but
I
would
expect
that
to
be
to
some
degree
a
bit
of
an
anti-pattern
I
would
not
expect
everything
to
run
in
a
single
cluster,
and
if
you
are
I'd
love
to
hear
more
about
why
you
think
that's
a
good
idea.
A
It's
neither
here
nor
there
as
why,
but
like
I.
Just
think
that
you
know
I
bet
if
I
surveyed,
you
know
this
entire
room
and
the
data
community
order
of
you
know
a
hundred
percent
of
you
have
existing
data
pipelines
and
the
thought
that
oh
here's,
this
brand-new
thing
cube
flow
that
you're
gonna
come
along
and
take
your
existing
massive
data
pipeline
and
move
it
over.
Like
that's
crazy
town,
that's
never
gonna
happen
like
I.
A
Much
much
more,
think
that
it's
gonna
be
this
external
thing
with
with
known
rules
around
data
processing
and
governance,
and
all
those
kind
of
things
will
will
export
an
endpoint
that
cube
flow
can
pick
up.
Kyouko
can
watch
cube
flow,
can
event
against
and
and
do
its
processing.
From
that
point
that
make
sense
yep
shell.
G
A
G
So
I
think
that
the
spark
operator
makes
a
lot
of
sense.
I
know
we
do
have
a
PR
that
I
think
is
still
open.
I've
been
talking
to
a
couple
of
spark
committers
just
today.
In
fact,
I
don't
know
if
Eric
is
in
here,
but
also
Sean
and
Holden,
so
Holden
created
the
the
PR.
So
if
you
have
something
to
add
to
that,
I
encourage
you
to
go
comment
on
that.
Pr
I
think
it
makes
sense.
I
think
that
it,
it
does
sometimes
make
sense
to
bring
those
things
into
your
cluster.
G
C
D
A
Absolutely
so
the
question
was
on
on
the
feature
store.
How
are
we
gonna
solve
for
the
cashing
time
to
live
and
other
things?
The
answer
is
absolutely
not
in
this
room.
We
should
go
cook
you
up
with
the
go
check,
people
and
work
on
that,
or
do
you
feel
like
that's
an
inappropriate
feature
story?
You
want
to
build
your
own,
you
can
go
to
that.
But
but
this
is
this,
is
you
know
I'm
kind
of
joking,
but
this
is
the
core
of
cute
flow
right.
A
Our
philosophy
is
we
and
cube
flow
are
not
gonna,
go
and
build.
All
of
these
things.
We're
not
going
to
go,
rebuild
tensorflow
we're
not
going
to
build
PI
torch,
we're
not
going
to
build
spark
we're
not
going
to
do
any
of
these
things.
We're
gonna
make
it
easy
for
you
to
either
run
those
things
locally
or
in
integrate
with
an
external
version
of
those
and
we're
gonna
make
it.
A
You
know
reliable
and
and
well
connected
and
give
you
pipelines
that
control
these
things,
but
it
is,
it
is
our
goal
to
integrate
those
external
projects
and
and
and
make
it
as
trivial
as
possible
to
bring
those
two
to
a
overall
cute
little
deployment.
So
we
should
connect
you
to
the
go
check.
People
and
we'll
see
from
there.
D
D
A
Yeah,
so
the
question
was
around
stability
and
what
our
goals
are
there.
It's
actually
a
really
fascinating
problem,
because
in
many
ways
this
is
you
know
those
people
who
are
raising
their
hand
about
using
cue
flow
in
production.
This
is
as
stable
as
it
gets
I'm
serious
right,
it's
native
kubernetes,
which
is
that
you
know
four
years
in
and
1.13
it's
tensorflow,
which
google
uses
across
every
single.
You
know
product
that
they
have
its
Jupiter
and
its
tensorflow.
Serving
all
these
things.
The
underlying
components
are
extremely
stable
today
and
production-ready.
A
If
I
was
to
say
there
was
one
challenge:
it's
that
when
we
see
stability,
we
actually
mean
a
little
bit
more
of
stability
of
the
API,
because
we
are
moving
very
quickly
and
we
don't
want
to
lock
on
an
API
too
soon
before
it's
gotten
real
production
usage,
and
so,
while
I
you
know,
was
joking
earlier
about
using
it
in
production.
I
would
love
love,
love
for
you
to
say,
hey,
you
know
what
I
didn't
use
it
into
production,
but
I
got
this
close
or
I'd
love
to
get.
A
This
close,
you
know
come
talk
to
it.
You
know
one
of
the
core
engineers
on
the
cue
flow
project
and
I
promise.
You
won't
like
fly
in
and
and
help
out
and
test
it
out,
because
we
would
love
nothing
more
than
to
have
20
50
100
people
actually
getting
it
that
close
to
production
and
saying
hey,
the
API
looks
great
or
we
really
need
this
or
whatever,
and
so
those
are
the
stability
that
I
think
we're
a
little
bit
closer
to
it's
it's
much
less
about.
Well,
we
ran
this
job
and
it
died.
C
H
H
A
Okay,
so
did
everyone
hear
that,
because
that
was
too
much
for
me
to
people
all
right,
I
would
say
that
cube
flow
accomplishes
two
things.
I
know
you're
supposed
to
do.
Three
comedy
comes
in
threes,
I'm,
gonna
gonna
come
up
to
one.
It
gives
you
a
standard
and
not
like
IETF
standard,
just
like
a
community
agreed
to
standard
for
how
to
run
a
component
and
machine
learning
component
on
kubernetes.
That
is
a
hard
problem.
It
is
an
annoying
problem
today
and
we
help
solve
that.
A
So,
for
example,
Jeremy
started
this
entire
project
by
going
to
tensorflow
and
looking
at
this
object
there,
this
this
necessary
deployment,
a
master
server
parameters,
servers
and
worker
notes,
and
he
created
a
tension,
float
job
object
that
used
a
cut
tensor
flow
CRD
and
deploys
them
all
unified
and
extends
the
api
and
rolls
it
out
right.
That
is
a
hard
problem
simply
to
package
and
deploy.
A
That
is
a
hard
problem
and
everyone
had
their
own
way
of
doing
it,
and
now
there's
a
standard
and
extending
that
standard
we
will
encourage
but
not
require
people
to
have
common
logging,
endpoints
common
health
checking
ways
of
doing
service
discovery.
All
these
various
things
that
is
extremely
important
and
now,
instead
of
me
having
to
go
and
deploy,
you
know,
I,
don't
know
seven
different,
kubernetes
objects
and
wire
them
together
and
have
the
correct
naming
and
all
those
kind
of
things.
A
It's
now
one
command
for
me
to
spin
up
a
case
in
that
package,
or
actually
even
now
and
deploy
tensorflow
and-
and
that
is
the
case
for
all
of
these
various
components.
Okay,
so
that's
number
one
and
extremely
important,
and
we
that's
where
we
see
a
lot
of
usage
today.
The
second
excuse
me
took
us
an
entire
year
to
get
to,
but
now
we're
here,
and
that
is
now
that
you
have
these
standard
components.
A
That
runs
on
your
laptop
that
is
portable
at
once,
anywhere
kubernetes
does,
but
it
also
gives
you
the
flexibility
and
so
for
the
activist
project.
You
know
we
would
expect
you
to
have
a
layer
on
top
of
it
that
that
gave
you
the
UI,
that
that
knew
about
all
these
components.
You
could
read
cube
flow
deployment
documents
or
read
straight
from
the
kubernetes
api
and
understand
what
had
been
deployed.
A
You
can
introspect
into
these
various
jobs
and
all
those
kind
of
things,
but
you
know
that
as
you're
building
this,
this
abstraction,
the
the
layers
under
the
hood,
are
now
stable.
Unlike
if
you
had
an
absolutely
arbitrary
cube,
tensorflow
deployment,
they
could
name
things
differently.
They
could
have
wrong
servers,
they
could
have
missing
endpoints
and
all
those
kind
of
things
go
ahead.
Yeah.
H
E
I
D
B
I
think
I
mean
I,
think
there's
a
deeper
issue
there.
It's
not
just
about
pipelines
and
I.
Think
it's
a
it's
an
overall
problem
within
the
problem
is
that
kubernetes
is
kind
of
not
enough
right.
So
when
you're,
when
you're
building,
you
know
an
ml
solution
and
you're
building
a
framework
right,
you
need
a
lot
of
different
infrastructure
to
support
that
that
goes
beyond
kubernetes,
like
you
need
an
object
store.
You
know
to
your
point
about
docker
images
right,
you
need
to
dr.
B
H
D
A
So
so
that's
thank
you.
You
said
the
magic
word
there,
which
is
and
I
actually
haven't,
checked.
I,
don't
know
how
good
our
tests
are.
This
is
our
but
I
will
say
this
as
someone
who
lived
and
breathed
the
kubernetes
community
for
years.
This
was
a
persistent
problem.
I
don't
know
it
probably
still
is
a
persistent
problem
like
we
I,
don't
even
care.
If
you
fix
it,
just
automated
nightly
tests
just
help
us
like
build
a
test
run.
A
It
nightly
run
it
regularly
once
a
week
once
a
month,
I
don't
care,
but
just
run
it
and
and
help
us
extend
the
test
framework
so
that
we
can
capture
it
and
and
go
on,
because
we
honestly
don't
know
I'm
gonna,
do
it
on
Azure
Jeremy's
gonna,
do
it
on
Google,
but
other
than
that
we
don't
know,
and
so
we
don't
have
visibility
and
that's
where
we
really
need
more
than
anything,
just
run
the
test
and
tell
us
what
went
wrong.
Are
we
out
of
time?
Okay,
oh
yeah,
go
ahead.
You
know,
go
real
quickly.