►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubeflow 1.0 Update by a Kubeflow Community Product Manager - Josh Bottum, Arrikto
This session will provide a Kubeflow 1.0 Update by a Kubeflow Community Product Manager. The presentation will include a review of the Kubeflow Community and feature development process, the Kubeflow user survey results, and Kubeflow 1.0 features. The talk will highlight significant business benefits and review use cases from top deployments. It will also include a live demonstration of a workflow to build, train and deploy a versioned Kubeflow Pipeline.
https://sched.co/Zen6
A
A
A
The
ability
to
have
it
portable,
distributed
and
scalable
portable,
meaning
that
I
may
want
to
have
the
same
environment,
be
able
to
work
on
my
laptop
on-prem
in
a
larger
cluster
and
then
also
in
a
public
cloud
environment,
distributed
inside
of
machine
learning.
At
times.
You're
going
to
want
to
use
your
infrastructure
as
effectively
as
possible,
so
you
may
have
multiple
nodes.
You
may
have
different
capabilities
on
those
nodes
where
you
have
gpus
or
cpus
of
lots
of
ram.
A
A
That
is
an
interactive
tool
that
allows
you
to
teach
and
share
your
work
effectively.
Many
people
really
like
jupiter
notebooks.
One
thing:
that's
special
about
kubeflow,
jupiter
notebooks,
is
that
they
are
you.
You
do
have
isolation
for
them
and
authentication,
so
users
are
separated
from
each
other
or
they
could
share
the
same
kubernetes
name
space.
So
you
get
the
best
of
both
worlds
there
when
it
comes
to
isolation
and
authentication.
A
Those
have
to
be
mapped
into
kubernetes
master
workers
and
since
any
of
those
can
be
moving
at
any
one
time,
any
of
the
workers
could
be
moving
per
kubernetes.
You
need
an
operator,
a
kubernetes
crd.
Basically
that
will
move
those
workloads
to
the
appropriate
place
at
the
appropriate
time.
Those
are
all
delivered
as
part
of
kubeflow
and
it
really
does
allow
you
to
have
distributed
training
which
speeds
up
your
model.
A
Processing
time
workflow
building
these
are
tools
that
allow
you
to
simplify
the
way
that
you
build
a
q-flow
pipeline
and
deploy
it
and
run
it.
So
typically,
in
a
q-flo
pipeline,
you
have
to
build
a
a
containerized
step
for
every
pipeline
step,
and
then
you
have
to
write
some
qflo
dsl
domain,
specific
language
to
run
those
those
containers
in
the
appropriate
order.
A
Cubefold
pipelines
is
a
way
to
schedule,
run
and
monitor
a
workflow
that
will
run
your
machine
learning
model
so
from
time
to
time,
you're
going
to
have
your
machine
learning
model
and
it
will
be
receiving
either
new
code
or
new
data,
and
rather
than
building
it
again
by
hand,
it's
good
to
have
a
workflow
that
does
that,
for
you,
qfl
pipelines.
Does
that
for
you
and
then
allows
you
to
schedule
how
that's
run
to
to
run
it
and
then
monitor
effectively.
A
Data
management
is
below
the
container
storage
interface
on
kubeflow,
pipe
on
and
below
qflow,
but
it
does
provide
the
versioning,
the
sharing
and
the
reproducibility
that
all
have
been
very
popular
and
a
basic
field
building
block.
In
order
to
do
data
science,
you
need
to
have
a
good
data
management
layer.
A
The
tools
involved,
the
hyperparameter
tuning,
the
hp
tuning,
the
tensorboard
and
the
prometheus.
Give
you
tuning
allows
you
to
take
those
hyperparameters
and
adjust
them
appropriately
automatically
through
a
series
of
algorithms,
which
saves
you
a
bunch
of
time
and
makes
your
your
your
model
run
faster
and
more
efficiently
and
more
accurately.
A
So
that's
a
quick
look
at
the
components
inside
of
kubeflow.
Let's
continue
down
our
path
and
look
at
what
happened
with
kubeflow
1.0,
so
qflo
1.0
was
our
eighth
release.
It
was
delivered
in
march
of
2020
as
important
of
all
the
software
deliveries
that
we
did,
which
there
were
almost
200
issues
that
were
delivered
to
improve
the
fit
and
finish
was
the
process
that
the
community
agreed
upon.
One
was
to
publish
a
qlo
roadmap
which
defines
the
features
and
timing
for
releases.
A
We
also
have
a
versioning
policy
which
defines
the
maturity
of
the
components
stable,
beta
or
alpha.
We
have
delivered
a
stable
applications
requirement
guide
and
in
this
guide
it
it
talks
about
what
each
components
should
address
in
order
to
become
stable.
So
things
like
configuration
and
deployment
custom
resources,
logging
and
monitoring,
docker
image,
ci
cd,
the
documentation,
ownership
and
maintenance,
as
well
as
user
adoption.
A
So
in
order
to
become
stable,
there's
a
checklist
of
things
that
each
of
the
components
should
go
through
before
they
can
claim
to
be
stable
and
then,
finally,
you
track
everything
in
a
github
kanban
board
which
are
available
for
folks
to
get
to.
These
are
all
click-through
links
that
you
can
use
inside
of
1.0.
We've
delivered
these
stable
components
now
central
dashboard,
the
kf
cuddle,
which
is
the
command
line,
also
the
profile
controller,
notebooks
tf,
serving
training
operators
both
for
tensorflow
and
pi
torch,
as
well
as
the
documentation.
A
A
One
technology
present
the
right
product
to
the
right
customer
at
the
right
time
or
group
by
simplify,
very
complicated
deployments
and
then
in
volvo
to
be
able
to
iterate
and
and
produce
with
ease
is
definitely
a
very
good
set
of.
I
guess
comments
from
folks.
Most
of
these
folks
have
not
been
you
know
with
us,
since
2017
and
many
of
them,
probably
in
the
last
six
months,
have
deployed
and
got
benefits
out
of
cube
flow.
So
this
is.
A
This
is
great
news
for
folks
that
are
coming
in
in
in
more
recent
times,
right
and
so
cube
flow.
One
of
the
things
I
do
as
a
product
manager
is,
I
take
a
survey
about
every
90
to
180
days
of
the
users.
This
survey
for
1.0
was
the
largest
one
we
had.
We
had
about
120
people
respond
on
on.
You
know
at
maximum
this
one.
We
ask
what
best
describes
your
roles
and
you
can
see
machine
learning.
A
Engineers
are
driving
a
lot
of
the
comments
and
some
of
our
most
largest
user
base
at
about
29,
about
23,
from
data
scientists
and
about
17
from
software
engineers
and
the
smattering
of
other
people.
But
it's
good
to
know
what
the
roles
are
of
the
folks
that
are
coming
and
asking
for
either
creating
issues
or
asking
for
features.
The
other
is
how
many
people
work
in
your
organization,
and
you
can
see
you
know
about
a
large
percentage
here
with
more
than
5000
people,
but
almost
20
from
51
to
500.
A
So
you
have
kind
of
this
difference
of
maybe
an
enterprise
type
customer
and
then
our
second
biggest
set
of
users.
Being
this
group,
which
is
more,
I
would
call
a
disrupter
base
and
you
know
their
capabilities
and
their
needs
may
be
the
same,
but
at
times
they
may
be
different
and
how
many
data
scientists
in
your
organization.
A
So
you
can
see
here
that
almost
50
of
the
organizations
have
less
than
10
people.
But
again
you
have
some
organizations
that
have
over
a
hundred
data
scientists.
So
if
you
have
10
data
sciences
or
less,
your
needs
may
be
a
little
bit
different
than
if
you
have
100
data
scientists
and
then
how
do
you
use
qflo
in
your
in
your
company
24,
just
learning
21
we're
planning
to
use
a
growing
percent
here
at
30
or
using
it
in
in
development
and
16
in
production.
A
Now
we're
going
to
get
into
a
couple
slides
on
the
metrics
that
you
can
use
to
kind
of
measure
where
you
stand
with
your
machine
learning
this
one
talks
about
how
long
does
it
take
to
develop
one
model
from
idea
to
production,
and
you
can
see
there's
about
14
percent
of
the
people
it
takes
six
months,
but
30
percent
it
takes
about
a
quarter
and
25
takes
a
month
these
guys
that
are
doing
it
in
about
a
month.
That's
maybe
best
in
class.
A
Not
all
models
are
the
same,
but
it's
good
to
know
that
if
you
want
to
produce
a
model,
how
long
does
it
take
to
produce
it
from
idea
to
production?
And
if
you
can
do
it
in
a
month
that
you're
in
with
some
of
the
best
in
class
metrics
that
are
out
there
today
and
then
how
many
models
has
your
team
developed
in
the
last
six
months?
A
You
can
see
here
that
you
know
about
34
have
done
two
to
five,
which
means
you
know
less
than
one
model
a
month,
but
you've
got
eleven
percent
that
have
done
over
50
models.
So
we
do
know
that.
There's
folks
that
joined
qflo
earlier
and
they
have
become
more
efficient
with
it,
and
I
think
that's
some
of
the
value
of
being
an
early
adopter
is
perhaps
you
get
to
be
in
this
best
in
class
group.
That's
able
to
get
the
most
out
of
kubeflow
and
then
how
often
do
you
deploy
models
into
production?
A
And
you
can
see
here
again
every
quarter
about
21
every
month
about
22
or
23
and
about
18
are
able
to
do
it.
Every
week
I
talked
to
a
customer
just
the
other
day
who
said
that
they
wanted
to
deploy
models
into
production
every
day.
So
I
think
we're
driving
in
the
right
direction
and
the
model
is,
is
helping
people,
and
I
thought
this.
This
feedback
would
be
good
for
folks
to
see
and
then
just
looking
at
cube
flow
versus
non-cube
flow.
A
A
You
can
see
that
more
q
flow
users
have
developed
models
faster
when
one
month
than
non-kubeflow
users
the
same
for
those
that
did
in
three
months.
So
the
kind
of
the
idea
here
is
that
the
more
that
you
have
developing
faster,
probably
the
better
and
that's
what
it
seems
like
kubeflow-
is
doing
from
from
this
analysis
again,
not
not
that
this
is
the
end-all
be-all,
but
it's
a
data
point,
and
probably
the
best
data
point
that
we
have
in
the
market
right
now.
A
Another
study
point
here
is
what
storage
functionalities
are
required
for
ai
and
ml
workloads,
and
what
you
can
analyze
is
that
versioning
and
model
sharing
have
become
two
of
the
most
important
components,
which
makes
sense
right.
If
you
have
data
data
science,
it's
based
on
data
management,
you
want
to
be
able
to
version
your
models.
You
need
to
be
able
to
share
your
models
and
portability
still
very
respectable
at
43.
I
would
just
say:
we've
been
doing
cube
flow
surveys
for
several.
A
You
know
more
than
a
year
and
a
half
two
years.
I
think
it
is
now.
This
is
the
highest
of
any
question.
We've
ever
had
anything
over
80
percent
is
really
high
for
the
community
to
be
able
to
respond
to.
So
I
think
it's
something
that
folks
do
want
to
take
a
look
at
one
of
the
important
things
that
we've
tried
to
do
inside
of
kubeflow
is
to
give
an
end-to-end
user
experience.
A
So
the
tools
that
provide
this,
as
I
talked
about
before
one
of
them
is
called
fairing
fairing,
allows
you
to
develop,
build
train
and
deploy
your
models
more
efficiently,
using
the
jupiter
lab
notebook,
the
fairing,
as
well
as
the
jobs
crds
as
well
and
kubeflow
serving
so.
This
does
provide
you
for
an
effective
way
to
build
and
deploy,
build,
train
and
deploy
your
models
into
production.
A
Another
cj
that
is
an
ecosystem
cj,
is
the
klcuj,
also
known
as
the
notebooks.
The
pipeline
cj,
you
can
develop,
create
a
pipeline,
run
it
and
explore
the
pipeline
later,
using
jupiter,
kale
and
rock,
as
well
as
cupola
pipelines
and
rock,
to
do
some
more
exploration,
we'll
go
through
a
quick
demo
of
this
a
little
bit
later.
But
the
the
real
thing
that
I
think
folks
get
excited
about
when
they
look
at
this
is
that,
with
these
cujs
you're
able
to
reduce
the
number
of
steps
necessary
to
deliver
a
model.
A
And
so
when
you
look
at
the
workflow
steps
before
having
this
capability,
it
would
take
you
24
steps,
maybe
hundreds
of
hours-
and
you
might
say
really
it's
going
to
take
me
hundreds
of
hours.
But
in
each
time
that
you're
building
a
pipeline
you
need
to
have
the
data
scientists
write
the
code
and
then
they'll
hand
the
code
off
to
a
data
engineer
or
to
a
ml
engineer
and
probably
an
ml
engineer,
and
the
ml
engineer
will
then
take
that
code.
A
So
this
demo
we'll
talk
about
how
we
know
we
annotate
cells
in
a
in
a
model,
we'll
snapshot
the
environment,
we'll
build
a
pipeline,
we'll
train
it
and
we'll
run
the
ml
model,
then
we'll
view
the
results
and
examine
a
pipeline
step
and
iterate
further.
I
know
this
is
a
lot,
but
the
iterate
further
is
actually
very,
very
important
in
inside
of
data
science,
because
it
will
help
you
to
make
better
models
faster.
So
now
I'm
going
to
go
over
and
do
a
screen
share.
A
A
A
So
if
you
have
a
gcp
account,
you
can
see
how
many
clicks
I've
had
to
do
to
get
to
a
place
where
now
I
can
click
this
and
deploy
it
and
in
about
20
minutes
this
whole
environment
will
maybe
another
couple
clicks.
It
will
come
up
and
I
will
end
up
with
an
environment
that
looks
like
this
right.
So
this
is
the
kubeflow
central
dashboard.
A
Before
I
get
to
this,
I
will
have
to
enter
in
a
username
and
password
that
whole
system
for
username
and
password
for
authentication
can
be
integrated
in
with
your
ldap
and
ad
environments
using
istio
indexed
and
many
folks
have
have
done
that
integration
to
this
point,
you'll
also
see
that
there
is
a
a
namespace
here.
This
is
a
kubernetes
namespace
which
keeps
these
folks
isolated.
A
So
when
I
log
in,
I
can
either
choose
a
namespace
that
is
only
to
me,
or
I
can
pick
one
that
I
share
with
other
data
scientists
over
on
this
side
over
here.
You
can
see
those
components
that
we
talked
about:
the
pipelines
component,
the
notebook
server
cad
tip,
which
is
the
hyper
parameter,
the
artifact
store.
This
is
the
metadata
volumes
manager,
as
well
as
a
snapshot
store.
A
A
We
call
it
titanic
one
and
you
can
see
we
have
a
namespace
that
we
can
give
it
to.
So
we
have
isolation,
we're
able
to
pick
what
custom
image
we
want.
This
image
could
be
anything
from
cpus
to
gpus,
using
tensorflow
or
pi
torch
or
whatever
you
happen
to
have
in
the
custom
image.
Some
are
available
for
you
right
from
the
system.
A
You
have
the
ability
to
set
your
cpu
and
ram
for
this
notebook
server,
so
you
can
apply
resources
appropriately
and
then
the
workspace
volumes.
This
is
where
you'll
keep
your
work
and
you
want
to
have
the
ability
to
keep
your
work
and
protect
it.
So
in
this
you
have
a
workspace
volume
which
is
for,
as
you
would
think,
your
code
and
then
a
data
volume
which
would
is
where
you
would
put
your
data.
A
We
could
call
this
data,
you
can
give
it
a
size,
an
amount
path,
other
things
that
are
nice
about
qflo
notebooks
is
the
configuration
capability.
So
it's
easy
to
get
to
external
data
sources
using
secrets
or
environmental
variables,
so
you
don't
have
to
remember
those
and
then
setting
up
gpus
or
is
also
made
easy
from
here.
You
can
launch
this
once
I
launch
it
and
click
another
button.
I
will
get
to
a
screen
that
looks
like
this.
A
A
A
A
Yeah
and
using
this
develop
so
we'll
turn
on
kale,
which
kale
then
allows
us
to
identify
the
code
that
we
want
to
be
in
our
kubeflow
pipeline
steps,
and
we
do
that
by
annotating
each
cell
over
here.
This
first
cell
is
a
special
cell.
It's
called
imports,
so
it
won't
be
in
our
pipeline,
but
it
is
important
as
we
get
started,
then
we
go
to
load
data.
As
you
remember,
you
can
see
that
this
is
a
cell
type
pipeline
step.
A
Then
some
of
these
steps
in
your
code
as
your
data
scientists,
will
know
they're
interesting
and
you
you
want
to
look
at
this
data
like,
for
example,
you
may
want
to
know
that
more
females
survive
than
males,
but
it
may
not
be
something
that
you
need
as
a
kubeflow
pipeline
step.
So
you
may
skip
a
lot
of
those
steps
which
a
lot
this
allows
you
to
do
and
then
come
to
where
you
need
to
do
some
data
pre-processing.
A
A
So
any
code,
any
code
or
any
cell
below
data
preprocessing
that
has
the
same
color
is
going
to
be
used
in
this
step
and
built
into
the
container
that
will
automatically
build
in
a
few
minutes
and
you
can
see,
as
we
pre-process
the
data
we're
going
to
get
rid
of
some
missing
data,
we're
going
to
account
for
that
and
then
we'll
get
to
feature
engineering
feature.
Engineering
is
another
step
that
will
be
in
our
pipeline.
A
We'll
do
some
changing
some
sets
into
numeric
or
dropping
some
various
features
age
in
the
category
fares
in
the
category,
and
then
we
get
to
the
next
steps,
which
are
all
our
machine
learning
code
and
our
different
models
that
we'll
run
through,
and
those
will
be
based
upon
that
future
engineering
step
being
done
and
first
we'll
go
through
random
forest.
At
the
same
time,
we
can
be
doing
logistic
regression,
the
bays
svm
decision
tree,
and
then
the
results
will
be
based
upon
all
this.
A
Instead
of
having
to
take
that
and
break
up
all
that
code,
write
docker
containers,
upload
them
into
a
into
a
repo
and
then
write
the
pipeline
dsl
code
to
run
them
all.
We
have
to
do
is
hit,
compile
and
run
over
here,
and
what
ko
will
do
with
its
back-end.
Snapshotting
capability
will
take
a
snapshot
of
the
environment,
so
these
are
all
those
persistent
volumes
that
we
used
when
we
created
the
kubeflow
notebook
and
then
it
will
create
the
pipeline,
it
will
upload
the
pipeline
and
it
will
run
the
pipeline.
A
You
want
to
know
what
your
inputs
and
your
outputs
were,
especially
from
a
data
perspective,
and
so
this
particular
pipeline
will
take
a
few
minutes
to
run
and
why
it's
running
I
just
want
to
give
kind
of
a
a
shout
out
to
the
community.
You
know
I've
been
working
in
the
kubeflow
community
for
some
time.
A
I
know
that
the
folks
that
are
the
vendor
folks,
whether
it's
the
microsoft
or
amazon,
google
or
ibm,
become
first
name
basis
with
these
folks
and
they
do
very
interesting
contributions,
whether
it's
david
iran
check
or
jeremy
from
from
google
or
amazon
the
folks
there
ginseng
or
or
the
the
other
folks,
anna
mesh
from
ibm
debo
from
cisco,
very
much
leaders
in
in
the
organization.
But
I
think
even
more
importantly,
is
the
contributions
that
we
see
from
the
users
right.
A
A
So
if
I
go
into
the
results
tab-
and
I
look
at
the
logs
you'll-
see
here
that,
although
I
think
I
might
be
smart,
I
may
not
be
this
smart.
Each
of
my
models
came
up
with
a
hundred
percent
accuracy,
so
I
know
that
maybe
I
did
something
wrong
and
now
this
is
common
in
data
science.
Now
I
want
to
go
back
in
and
potentially
figure
out
what
I
did
wrong.
A
So
I
can
go
into
one
of
these
steps,
say
the
random
forest
step
and
look
at
the
artifacts
that
were
created
in
this
step
and
in
order
to
recreate
that
environment.
I
just
click
on
that
button,
as
I
did
there.
It
takes
me
to
my
snapshot,
store,
which
happens
to
be
integrated
with
a
system
called
rock.
I
can
copy
this
file
link
here
and
that
allows
me
to
recreate
my
environment.
A
A
This
iteration
capability
is
very
important
to
data
scientists
who
often
make
some
form
of
mistake
or
challenge
and
then
need
to
go
back
and
improve
their
model
or
there's
just
new
data
comes
in
that
they
want
to
go
through
and
and
take
another
look
at
and
see
how
this
impacts.
What
they're
doing
so,
you
can
see
here
as
we
go
through
it's
bringing
up
that
notebook
as
we
had,
and
it
will
take
me
right
to
that
random
for
a
step.
A
A
With
that,
I'm
going
to
go
back
and
finish
up
the
that's
the
end
of
the
present
of
the
demonstration
I'll
go
back
to.
Excuse
me
the
the
presentation
and
go
back,
and
just
summarize
what
we
saw.
We
annotated
a
notebook
cells
of
the
notebook
of
a
model.
We
snapshot
the
environment,
we
built
the
pipeline,
we
trained,
we
ran
it.
We
reviewed
the
results.
A
A
We
have
a
some
workflows
that
incorporate
the
ability
to
use
multi-user
pipelines
and
to
tune
a
model
efficiently
and
use
cache
steps
so
as
you're
tuning
a
model
we'll
run
through
that
model,
maybe
50
100
times,
maybe
more
in
order
to
find
the
right
hyper
parameter
settings
for
that
model.
You
don't
need
to
rerun
all
the
steps
of
your
model
if
you've
already
run
them
before
in
a
cube
flow
pipeline.
So
you,
if
you
use
the
cache
steps,
you
can
save
processing
time
and
save
resources.
A
Additionally,
we
have
stable
training
operators
for
mxnet
and
xgboost.
This
is
fantastic
for
folks,
especially
in
the
financial
industry
that
use
these
types
of
algorithms
in
their
models.
We've
improved
the
isolation
and
security
again
with
multi-user
pipelines,
very
popular
requests
from
folks,
private
gke
and
antho
support
cve
scanning
reports.
So
you
can
see
the
docker
containers
that
are
used
to
run
kubeflow.
You
can
see
what
the
cves
are
there
and
then
namespace
creation
by
administrators
rather
than
self-service
by
administrating
by
users.
A
So
what's
next
kublow
community
governance
is
big.
We're
going
to
have
plenty
of
workflow
group
working
groups
that
will
be
engaged
and
develop.
The
working
group
is
more
what
you'd
consider
a
sig
and
so
we're
in
the
process
of
defining
that
how
that
governance
works
and
then
cubeflow
1.2
will
be
planned
and
started
right
after
we
get
done
with
1.1.
A
So
if
you're
interested
in
getting
involved
with
kubeflow,
please
feel
free
to
join
in.
We
have
a
great
slack
channel.
We
have
a
discussed
mailing
list
that
you
can
join.
We
have
weekly
community
meetings
every
tuesday
and
then
you
can
use
the
slack
channel
or
submit
bugs
on
github
in
the
kubeflow
repos.
A
We
really
wanted
to
give
a
big
thank
you
to
all
the
folks
that
have
contributed
to
kubeflow
at
this
point,
as
well
as
all
the
users
who
either
have
contributed
their
time
or
effort,
as
well
as
just
giving
us
feedback
it's
a
great
community,
and
we
welcome
a
lot
of
folks.
You
can
see
almost
25
percent
of
the
folks
that
come
to
the
kubeflow
community
are
new
and
that's
been
consistent
since
we
started
the
community.
So
thanks
a
lot
for
your
time
and
we
look
forward
to
your.