►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Kubeflow Deep Dive – David Aronchick & Jeremy Lewi, Google (Intermediate Skill Level)
The Kubeflow project is aimed at simplifying developing, deploying and using ML on Kubernetes. This session will provide an in-depth understanding of Kubeflow. The talk will focus on the different components that comprise Kubeflow and how we connect them together using ksonnet. We will also give a project update and review the roadmap for the rest of the year. After the session attendees should better understand Kubeflow's architecture and know how to get involved.
About David
David Aronchick was the Senior Product Manager for the Google Container Engine and led product management on behalf of Google for Kubernetes. David has been helping to ship software for nearly 20 years, founding and being part of the management team for three different startups, as well as squeezing in time at Microsoft, Amazon, Chef, and now Google. David is co-founder of the Kubeflow project, an effort to help developers and enterprises deploy and use ML cloud-natively everywhere.
About Jeremy
Jeremy Lewi is a co-founder and lead engineer at Google for the Kubeflow project, an effort to help developers and enterprises deploy and use ML cloud-natively everywhere. He's been building on Kubernetes since its inception starting with Dataflow and then moving onto Cloud ML Engine and now Kubeflow.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
thank
you
very
much
for
coming.
My
name
is
Jeremy
levy,
I'm,
an
engineer
at
Google
and
I'm,
leading
the
team
at
Google,
that's
contributing
to
coop
flow,
and
so
this
session
is
going
to
be
a
deep
dive
into
the
coop
flow
project.
So
it's
going
to
be
focused
a
little
bit
more
on
the
details
of
the
the
project
and
it's
sort
of
aimed
and
tailored
to
you
know,
contributors
or
people
that
are
thinking
about
contributing
to
the
project.
A
So
there
are,
though,
a
number
of
talks
at
at
coop
con
related
to
coop
flow
or
some
of
the
components
related
to
ml.
So
if
you're
looking
for
an
introduction
or
there's
more
focused
on
sort
of
the
user
experience,
you
should
check
some
of
those
out.
Some
of
those
haven't
happened
yet,
but
for
the
ones
that
have
happened
like
the
introduction
yesterday
to
coop
flow,
you
can
find
the
slides
or
the
talk
online.
A
So
this
is
this
is
this
slide
illustrates
the
problem
that
we're
trying
to
solve
with
with
coop
flow,
so
one
of
the
things
that
we've
learned
a
lot
at
Google
having
deployed
a
lot
of
ML
systems
in
in
production.
So
my
is
his
background.
My
first
job
at
Google
was
working
on
the
YouTube
video
recommendation
system,
and
so
this
slide
resonates
a
lot
with
me,
because
what
we
found
is
that,
when
you're
deploying
ml
systems
in
production,
you
spend
most
of
your
time
not
on
the
actual
ml
code,
which
is
this
bit
like.
A
So
we
tend
to
think
of
ml.
We
think
about
the
algorithms,
the
deep
neural
nets
and
all
the
complex
maps
that
sort
of
powers
them.
What
we've
actually
found
is
that
most
of
the
time
and
energy
that
goes
into
maintaining
systems
and
products
is
all
the
supporting
services
around
them.
So
you
know
you
have
to
do
all
of
your
data
ingestion,
your
data
cleaning.
You
have
to
manage
resources
for
your
serving
infrastructure,
your
data
processing
infrastructure.
You
have
all
the
actual
serving
infrastructure.
A
Containers
and
kubernetes
are
really
great
at
DevOps,
and
so
we
wanted
to
try
to
use
containers
and
kubernetes
to
solve
some
of
the
DevOps
challenges
of
ml,
and
so
the
goal
with
Kubb
flow
is
to
build
a
kubernetes
native
platform
for
ml,
so
that
we
can
make
ml
portable
and
what
we
mean
by
portable
is
making
it
easy
to
move
from
local
development.
On
your
on
your
machine
into
your
your
staging
or
test
clusters
and
then
finally
into
production.
A
So
what
exactly
is
Kubb
flow
first
and
foremost
we're
a
community.
So
we
are
a
growing
community
of
you
know:
data
scientists,
DevOps
engineers,
software
engineers,
product
managers
and
as
a
community,
we're
coming
together
to
build
an
ml
platform
and
to
build
that
ml
platform
on
top
of
kubernetes
and
the
reason
that
we're
all
coming
together
to
build
a
platform
is
because
I
think
there's
a
growing
recognition.
A
That
building
an
ml
platform
is
a
really
big
problem
and
it's
a
lot
easier
to
build
it
together
as
an
open
community
and
everybody
and
have
everybody
contributed
to
it
rather
than
trying
to
build
it
all
by
yourself
and
I.
Think
this
is
a
lesson
that
we're
taking
from
the
kubernetes
community,
where
we've
seen
that
you
know
kubernetes
was
very
successful
because
they
built
an
open
platform,
then
brought
a
lot
of
people
into
that
platform
that
contributed
contributed
to
it
to
it.
A
I
think
that's
one
of
the
main
reasons
why
you
know
kubernetes
has
been
as
successful
as
it
is,
so
we're
trying
to
recapture
that
magic,
as
I
said
before,
coop
flow
wants
to
be
a
kubernetes
native
platform
from
machine
learning,
and
so
what
that
means
a
couple
of
things.
First
and
foremost,
we
we
are
taking
advantage
of
them.
A
We
are
kuben,
we
run
wherever
kubernetes
runs
right,
we're
taking
a
hard
dependency
on
kubernetes
and
that's
because
we're
we're
relying
on
kubernetes
to
be
the
basis
of
our
portability
story
right,
so
kubernetes
runs
locally
on
your
machine
and
your
machine
using
mini,
cube
or
now
docker.
You
can
also
run
it
in
your
on-prem
clusters.
You
can
also
run
it
in
the
cloud.
So
if
you
have
a
platform
that
runs
on
kubernetes,
then
it's
easy
to
move
that
platform
and
any
products
built
on
those
platforms
across
those
different
environments.
A
We're
also
taking
advantage
of
kubernetes
functionality
and
api's
to
solve
some
of
the
problems
that
you
have
in
machine
learning.
So
a
particular
example
of
that
is
a
lot
of
your
machine
learning
frameworks
today,
like
tensorflow
support,
distributed
training,
which
is
very
important
to
scaling
out
in
being
able
to
Train
large
models
on
large
data
sets
right
and
so,
with
tens
of
with
distributed
training,
you
often
have
to
manage
multiple
processes.
So
in
a
distributed
tensorflow
job,
you
have
a
master
process
which
is
coordinating
the
training.
A
You
know
what
we're
providing
with
coop
flow
is
case
net
packages
that
make
it
easy
to
deploy
all
these
micro
services
in
a
way
that
makes
them
easy
to
manage
together.
And
finally,
we
want
to
provide
end-to-end
machine
learning,
solutions
that
are
built
on
coop
flow
and
illustrate
how
you
can
build
products
and
solutions
using
coop,
though
so,
as
I
said,
first
and
foremost
we're
a
community.
A
So
we
have
about
70
individual
members
and
we
have
over
20-plus
organizations
that
are
listed
there,
that
I've
started
to
participate
and
contribute
either
as
contributors
or
potentially
as
users
to
coop
flow
and
they
span
the
gamut.
So
it's
everybody
from
people
that
are,
you
know,
building
you
know
machine
contributing
machine
learning
code
so
like
Selden,
is
developing
code
for
handling
model
deployment.
A
We
also
have
contributions
from
you
know:
people
in
the
more
traditional
DevOps
space
that
are
trying
to
take
advantage
of
their
tooling
for
things
like
monitoring
and
ply
that
to
ml
applications
and
so
there's
a
whole
bunch
of
stats
there.
You
can
see
that
we're
very
active
on
github.
You
know
we
have
about
35
PRS
per
week
and
I
think
an
interesting.
An
interesting
stat
is
that
we
have,
but
you
know,
35
PR
creators
per
week,
so
we
have
quite
a
few
people.
A
So
that's
way
more
than
you
know
way,
orders
of
magnitude,
that's
about
an
order
and
magnitude
bigger
than
the
largest
teams
working
on
coop
flow.
So
any
given
company
has
about
most
five
contributors
to
coop
flow
right
now,
so
we're
doing
pretty
well
and
then
we're
taking
advantage
of
the
Deb
stats
project
from
the
CN
CF
for
tracking
all
of
our
stats
for
our
project.
So
you
can
get
more
stats
online.
A
A
You
then
need
to
do
actually
train
models
using
a
framework
like
tensorflow
or
py
torch.
You
then
want
to
be
able
to
scale
that
out
so
that
you
can
actually
train
at
scale.
Then
you
finally
need
to
roll
that
out
into
production
and
serve
those
model
models
and
then
once
you're
serving
those
models,
you
actually
need
to
monitor
it
and
make
it
operational
right.
So
our
goal
with
with
coop
flow
is
really
to
fill
in
all
these
existing
boxes
and
provide
a
cohesive
solution
for
using
all
these
boxes
together.
A
And
so
then,
this
is
what
the
the
user
experience
looks
like.
If
you're
actually
trying
to
develop.
You
know
ml
solutions
and
products.
You
sort
of
start
by
deploying
coop
flow
on
your
cluster
you'll,
then
typically
do
some
experimentation
and
analysis
to
develop
your
your
model.
So
typically,
you
might
do
that
in
a
data
analysis
focus
tool
like
Jupiter.
You
then
would
probably
build
dr.
images
and
then
use
those
dr.
images
to
train
at
scale
which
you
might
do
using
some
one
of
our
custom
resources.
A
For
you
know
managing
tensorflow
or
forthcoming
other
frameworks
like
py
torch.
You
then,
would
probably
want
to
build
a
model
server
to
actually
generate
predictions,
and
then
you
can
deploy
that
model,
and
then
you
have
to
integrate
that
model
into
your
app.
Maybe
it's
a
web
app
or
it's
a
mobile
app
and
then
finally,
you
have
to
operate
them.
A
So
there's
there's
two
types
of
components
within
coop
flow,
so
there's
there's.
First
and
foremost,
those
are
the
components
being
developed
within
coop
flow,
so
those
are
ones
that
are
sort
of
subject
to
kubernetes
governance,
I'm,
sorry,
coop
flow
governance
and
being
developed
by
you
know
the
core
contributors
of
coop
flow.
But
then
a
lot
of
our
components
are
actually
existing
components
that
are
developed
elsewhere,
but
are
integrated
in
into
coop
flow.
A
In
order
to
provide
that
you
know
complete
PAP
platform,
and
so
these
components
are
not
maintained
within
coop
flow,
they're
maintained
outside
and
so
they're
not
subject
to
coop
flows,
governance,
so
you
can,
if
you
so
choose,
you
can
have
maintained
header
control
of
your
component,
but
still
have
that
component
be
integrated
and
deployed
as
part
of
Gupta
and
I'll
have
some
examples
on
the
next
slide.
So
here's
a
list
of
the
projects
that
are
being
developed
within
coop
flow.
A
So
to
start,
we
have
a
bunch
of
custom
resources,
for
you
know
several
ml
frameworks,
so
our
TF
operator
is
the
most
advanced.
Our
second
up
is
probably
our
py
torch
operator,
which
will
probably
make
it
into
our
next
release.
We
also
have
a
cafe
to
operator
that's
in
early
stages
of
development
and
then
we're
integrating
other
features.
So
the
uber
folks
have
started
contributing
bod,
which
is
a
add-on
to
tensorflow
that
allows
you
to
run
distributed
tensorflow
using
MPI
and
it's
a
way
to
do
distribute
training.
A
The
Intel
folks
have
contributed
KBC,
which
is
a
kubernetes
volume
controller,
and
what
this
does
is
it
basically
moves
data
back
and
forth
from
say
your
object
store
to
your
nodes
and
cassia's
it
on
your
nodes
and
the
idea
there
is
to
move
manage
data
in
a
way.
That's
that
serves
ml
workloads
efficiently.
A
Then
we
have
cat
tib
cat
tib
is
a
hyper
parameter
tuning
system.
It's
a
clone
of
visi,
a
Google's
HP
tuning
system.
The
clone
was
not
developed
by
Google.
It
was
developed
by
some
people,
some
contributors
who
have
contributed
to
cupola,
and
then
you
know
finally
we're
sort
of
closing
some
of
the
gaps
around
using
existing
systems
so
as
an
example
we're
providing
docker
images
for
TF
serving
which
is
a
model
server
for
tensor
flow.
A
So
you
don't
have
to
go
and
build
those
yourself
we're
also
providing
curated,
Jupiter
notebook
images
so
that
you
can
get
some
notebook
environments
out
of
the
box.
Here's
a
bunch
of
lists
of
the
different
projects
that
we've
already
started,
integrating
into
coop
flow,
but
are
developed
outside
of
coop
flow.
So
argo
is
a
kubernetes
native
workflow
controller.
So
workflows
are
pretty
important
to
writing
data
processing
pipelines
and
ml
pipelines.
So
we've
started
using
our
go
for
that.
We
we
are
also
integrating
with
with
Jupiter
hub,
which
is
a
multi-user
server
for
Jupiter
notebooks.
A
We
integrate
with
pachyderm
as
well.
So
pachyderm
is
a
package
that
provides
a
pipeline
tool
for
running
data
processing
pipelines.
They
also
provide
a
data
versioning
system,
that's
equivalent
to
get
for
data.
So
that's
a
very
nice
tool
as
well.
Seldon
io
provides
a
custom
resource
and
tooling
for
building
and
serving
models
in
production.
Tenzer
to
Tenzer
is
a
library
of
tensor
flow
models
and
datasets
that
serve
a
variety
of
applications.
A
A
So
let's
take
a
look
at
our
road
map,
so
you
know,
as
I
said,
before,
we're
trying
to
fill
out
all
those
boxes
in
the
platform.
So
one
of
the
things
we're
trying
to
do
is
we're
trying
to
make
it
really
easy
for
people
to
add
components
so
that
you
know
we
can
start
filling
out
those
boxes
and
so
adding
a
components
is
pretty
straightforward.
You
basically
take
the
ammo
manifest
for
whatever
package
or
product
you
want
to
deploy.
So
if
you
had
something
like
Jupiter,
hug
or
Argo,
you
would
get
those
manifests.
A
You
then
turn
them
into
case
Annette,
so
that
basically
just
means
turning
them
from
ml
into
Jason,
and
then
you
provide
us
a
PR
and
then
there's
instructions.
They
are
linked
from
this
presentation
that
go
through
the
steps
in
more
detail
and
provide
some
instructions
on
how
to
structure
your
case
in
that
package.
A
A
The
course
UJ
that
we're
trying
to
be
able
to
fulfill
as
part
of
our
one
release
is
continuous
integration
and
deployment
for
machine
learning,
and
so
what
that?
What
this
means
in
the
context
of
machine
learning
is
you
can
think
about
every
night?
You
get
more
data,
so
you
can
imagine
if
you're
generating
recommendations
for
a
web
site
like
YouTube,
you
have
a
you.
A
Have
you
get
new
data
in
the
form
of
web
logs
from
your
most
recent
events,
you'd
like
to
retrain
your
models
using
the
most
recent
data,
evaluate
those
models,
and
if
those
models
are
doing
better
in
production,
then
your
current
models
in
production
you'd
like
to
push
them
automatically
into
put
into
production.
So
that
involves
a
lot
of
additional
components
and
functionality
beyond
what
we
have
today
and
we're
hoping
to
get
there
by
our
one
data
release
and
I.
Think
there's
there's
an
open
question
as
part
of
getting
our
one
door.
A
Release
is
whether
we
join
a
foundation
like
CN,
CF
or
possibly
some
other
foundation,
so
we're
we're
targeting
a
0.2
release
at
the
end
of
this
quarter
and
we
sort
of
hope
to
get
into
a
release
cadence
of
having
a
release
every
every
quarter.
So
in
this
release,
we're
hoping
to
have
a
versions
of
some
new
components,
so
cat
tab
for
HP
tuning
py
torch
for
training,
py
to
py
torch
models,
we'd
like
to
have
batch
inference.
A
So
with
that
I'd
like
to
switch
and
switch
gears
a
little
bit
and
try
to
talk
about
some
of
you
know
what
I
think
are
some
core
principles
for
coop
flow,
and
so,
let's
I
think
there's
about
three
of
them.
So
the
first
one
is,
you
know
we
want
to
be
open
and
I've
already
sort
of
talked
about
why
we
want
to
be
be
open.
A
It's
because
it's
such
a
big
problem
that
we're
we're
trying
to
tackle,
we
think
being
open,
is
going
to
be
critical
to
our
success,
and
we
also
think
that
this
is
an
important
lesson
to
be
learned
from
the
kubernetes
community.
What
this
means
is
from
the
beginning
of
coop
flow.
We've
spent
a
fair
bit
of
time
trying
to
ensure
that
everybody
within
the
community,
so
not
just
Google,
who
sort
of
you
know
kicks
things
off,
has
an
equal
opportunity
to
take
on
leadership
roles
and
other
roles
in
the
community.
A
So,
from
a
from
religious
logistical
perspective,
we've
tried
to
give
equal
access
to
like
our
test
infrastructure,
our
release
infrastructure.
You
know
external
external,
external
contributors,
so
not
just
Google,
so
there's
really
no
difference
between
Google
contributors
and
non
Google
contributors.
It
also
means
that
our
success
will
depend
on
everyone
carrying
water
and
chopping
wood
and
so
I
think
as
we
grow.
This
is
one
of
the
big
challenges
that
we
face
as
a
community.
A
So
as
an
example,
the
the
number
of
PRS
per
week
is
roughly
about
two
times
the
number
of
commits
right
now,
so
we're
we're
not
sort
of
keeping
up
with
with
our
velocity.
So
it's
important
for
us
to
get
more
people
to
act
as
reviewers
from
the
community
so
that
we
can.
We
don't
restrain
our
velocity
by
our
review
capacity.
A
The
second
principle
that
we
have
is
we
want
to
have
a
low
bar
and
a
high
ceiling,
so
a
low
bar
means
we
want
to
make
it
super
easy
to
get
started
with
coop
flow,
and
so
one
of
the
challenges
there
is.
We
need
to
minimize
the
number
of
kubernetes
concepts
and
api's
that
users
need
to
learn
to
get
started.
A
You
know,
there's
a
lot
of
data
scientists
and
m/l
researchers
that
would
like
to
use
coop
flow
and
start
to
get
some
of
the
advantages
of
kubernetes,
but
trying
to
get
up
to
speed
on
kubernetes
is
can
really
be
trying
to
drink
from
a
fire
hose.
So
you
know
trying
to
make
that
as
simple
as
possible.
A
A
And
finally,
we
want
to
be
you
know,
kubernetes
native,
and
so
you
know,
as
I
said
every
earlier,
we
want
to
run
anywhere,
kubernetes
runs
and
we
want
to
reuse.
Kubernetes
concepts
and
api
is
like
custom.
Resources
are
a
perfect
example
of
that.
There's
a
lot
of
functionality
and
tooling
in
kubernetes.
That
makes
it
really
easy
to
manage,
distributed
jobs
and
sort
of
handle
them
mech
mechanisms.
A
I
already
talked
a
little
bit
about
how
we're
you
know
open
so
I'm
gonna
skip
directly
to
you
know
how
we're
coop,
kubernetes
native
you
know,
as
I
mentioned
already
we're
taking
advantage
of
kubernetes
api
syn
concepts,
like
you
know
the
custom
controllers
for
TF
job
but
another
another
principle.
There
is
that
we
don't
necessarily
want
to
hide
kubernetes
api
s,
because
we
want
to
give
people
all
the
flexibility
in
in
kubernetes.
A
So
as
an
example,
the
TF
job
operator,
it
contains
the
pod
template
spec,
so
that
if
people
want
to
specify
things
like,
you
know,
requests
and
limits
to
control
their
job,
they
can
do
that
and
they
don't
have
to
learn
a
new
API.
They
can
also
take
up
kubernetes
volume
management
system
to
manage
data
and
attach
data
to
their
their
their
pods.
As
well
as
doing
do
things
like
inject
secrets.
A
The
other
thing
that
we're
following
in
terms
of
coop
kubernetes
is
we
want
to
make
sure
that
you
know
coop,
coop
flow
and
ml
is
managed
declaratively.
You
know
men
at
matching,
Kate's
best
practices
and
we'd
like
to
embrace
get
ops,
because
we
see
that
as
gaining
a
lot
of
traction
in
the
community
and
very
powerful
pattern.
A
The
other
thing
is
we're
trying
to
leverage
the
k8,
the
kubernetes
ecosystem.
So
as
an
example,
we
want
to
align
with
what
sig
Apps
is
doing
in
terms
of
the
application
CRD
and
try
to
leverage
those
ideas
in
terms
of
how
we
manage
coop
flow
deployments
and
make
that
super,
easy.
I.
Think
one
of
the
challenges
that
we
face
in
the
community
is,
you
know:
can
we
reconcile
being
kubernetes
native
and
having
a
low
bar?
A
Think
one
of
the
things
that
we'd
like
to
avoid
is
having
users
fall
off
the
cliff,
and
so
what
I
mean
by
this
is
you
know
when
you
get
started
with
with
Kubb
flow?
You
know
you
have
some
canned
examples
like
you
know.
Em
nest
is
this:
the
hello
world
of
ml,
and
so
you
can
imagine
that
we
can
package
up
the
examples
in
a
nice
way
so
that
you
don't
have
to
know
kubernetes
to
to
actually
run
the
example.
A
But
then,
when
you
actually
try
to
run
your
model,
your
own
models
on
your
own
data,
suddenly
you're
dealing
with
things
like
you
know,
connecting
to
your
external
storage
systems,
whether
that's
GCS
or
s3,
and
now
you've
got
to
deal
with
volumes
and
credentials.
Now
you
have
to
deal
with
the
logging
system
to
try
to
figure
out
why
your
jobs
are
failing.
You
run
into
our
back
and
so
suddenly
to
go
from
running
the
examples
to
running
your
actual
duck,
your
your
actual
jobs
and
doing
ml.
A
So
portability
is
our
mission,
so
we
want
to
make
it
really
easy
to
move
ml
solutions
and
ml
applications
in
between
different
environments.
So
in
between
you
know,
cloud
and
on-premise
in
case
Annette.
So
are
very
amenable
to
that,
and
so
this
slide
sort
of
illustrates
how
you
build
ml
applications
out
of
case
Annette
components
right.
So
at
the
start
you
have
in
case
Annette,
you
have
parts,
and
so
individual
parts
correspond
to
manifests
for
kubernetes
resources
right.
A
So
you
can
imagine
that
if
you're
deploying
a
model
in
production
using
TF
serving,
you
would
have
a
manifest
for
your
actual
kubernetes
deployment
object.
You
would
then
also
have
a
manifest
for
the
service,
the
kubernetes
service.
That's
going
to
direct
traffic
to
this
model.
So
when
you
combine
these
components
into
a
logical
or
functional
unit
in
kubernetes
a
net,
that's
called
a
prototype.
A
So
in
this
case
you
might
have
a
prototype
for
TF
serving,
and
this
is
going
to
be
parameterised
by
values
that
are
going
to
specify
things
like
your
model,
location
that
you
want
to
serve
so
TF
serving
might
be
one
prototype.
You
would
all
might
also
have
a
prototype,
for
you
know
doing
your
your
training
jobs,
so
you
might
have
a
prototype
for
your
tensorflow
job,
because
you
need
to
train
your
models,
and
this
will
also
have
as
a
parameter
the
location
where
you
want
this
trained
model
to
be
located.
A
So
when
you're
building
an
ml
application,
you
create
these
components
out
of
the
prototypes.
So
you
have
your
server
for
your
model
and
then
you
also
have
your
your
trainer.
And
so,
when
you
combine
these
components,
you
get
an
ml
application.
So
in
this
case
you
might
have
an
ml
application.
That's
you
know
doing
something
like
predicting
sentiment,
and
in
this
case
you
might
be
using
tensorflow
to
both
train
the
model
and
then
tens
are
flowe
serving
to
serve
the
model.
A
A
A
Yeah
yeah,
so
we
have
a
bootstrapper.
We
call
it
a
bootstrapper
and
we're
really
trying
to
do
two
things
with
the
bootstrapper
one
is.
We
would
like
to
try
to
get
an
experience
where
you
can
stress
sort
of
deployed,
coop
coop
flow
using
coop
cuddle,
create
app
from
the
manifest,
and
the
other
thing
that
we're
trying
to
do
with
the
bootstrapper
is
we're
trying
to
automatically
optimize
Kubb
flow
for
your
particular
deployment.
B
Just
the
coin
this
week,
currently
I
think
follow
the
Potter
pattern
of
creating
a
tool
box
kind
of
container
which
you
exactly
into
into
a
whole
tool
set
of
managing
cube
flow
insight
and
I
bootstrap
arama,
among
other
things.
So,
but
this
is
where,
as
Jeremy
mentioned,
this
is
very,
very
open
and
very
active
part
of
outcome
of
our
you
liked.
A
lot
of
things
happened
there.
So
if
you
have
any
good
ideas,
please
help
we
need.
A
You're
you're
relying
on
kubernetes
for
all
the
resource
isolation.
So
you
get.
You
know
so,
obviously,
to
the
extent
that
if
we
mean
you
submit
to
different
AF
jobs,
you're
relying
on
kubernetes
to
schedule
those
jobs
efficiently,
ensure
that
we
each
get
the
resources
that
we
were
requested.
We
don't
have
in
in
coop
flow.
We
haven't
really
started
taking
advantage
of
namespaces
and
other
features
like
our
back
to
isolate
users
to
prevent
different
users
from
you
know,
interfering
with
one
another.
D
Kind
of
an
extension
of
what
you've
just
described
in
terms
of
isolating
users
in
our
use
of
cube
flow,
we
had
to
isolate
certain
resources,
specifically
p3
instances
in
AWS,
because
of
of
expense,
our
primitives
likely
to
get
into
cube
flow
to
kind
of
facilitate
that,
or
do
you
consider
that
to
be
outside
the
the
kind
of
role
of
queue
flow
in
terms
of
allowing
auto
scaling
for
pricing
or
isolating
the
kind
of
huge
expense
that
can
come?
If
you
leave
GPU
instances
on
I.
A
Think
we
would
like
to
work
with
the
kubernetes
community
to
have
those
features
land
at
the
right
level
of
the
kubernetes
api
and
then
find
out
ways
that
we
can
leverage
them.
So
is
an
example.
We
haven't
done
that
particular
functionality,
but
one
of
the
functionalities
that
we
need
in
in
Koufos
batch
scheduling,
so
gang
scheduling,
for
example,
and
so
we're
taking
advantages
of
things.
E
A
Flank
as
an
example,
so
in
particular
you
know
tf-x
they
rely
heavily
on
beam
and
so
as
part
of
having
an
open
source
solution
that
can
run
anywhere.
We
need
to
use
some
of
the
open
source
runners
for
beam
in
order
to
run
distributed
on
platforms
that
aren't
GCP,
and
so
we
hope
to
take
advantage
of
you
know,
spark
and
other
support
for
running
flink
on
kubernetes
to
provide
that
as
a
solution
to
our
users.
E
C
A
I
think
so
in
in
data
management,
I
think
a
big
big
desire
from
from
enterprises
is
auditing,
so
they
want
to.
You
know,
make
sure
that
you
know
understand
like
what
code
ran
against
which
data
so
I
think
that's
an
important
feature
to
get
integrated
into
coop
flow.
I.
Think
around
that
you
have
like
data
exfiltration
protection.