►
From YouTube: Tutorial: From Notebook to Kubeflow Pipelines with HP Tuni... Stefano Fioravanzo & Ilias Katsakioris
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Tutorial: From Notebook to Kubeflow Pipelines with HP Tuning: A Data Science Journey - Stefano Fioravanzo & Ilias Katsakioris, Arrikto
An introduction to Kubeflow, the ML toolkit for K8s and the workflows you can use as a data scientist to scale up your ML code effortlessly. Ever thought how hard it is to convert your Jupyter Notebooks into deployable and composable pipelines, scale up computation and run hyperparameter tuning? With Kubeflow, this process becomes extremely easy as you make use of the many components of this ML toolkit: Pipelines, Kale, Katib, Snapshot Store. You will learn how to deploy Kubeflow in minutes, explore your ML code inside a Jupyter Notebook, convert it to a composable and scalable workflow with the click of a button, make the pipeline reproducible using immutable snapshots, go back in history and debug it, run hyperparameter tuning and distribute your computation. Did we mention you won’t need any specific SDK or CLI command to do this? Sounds like magic? Come and see for yourself!
https://sched.co/ZerG
A
A
What's
in
it
for
you
in
this
session,
what
you
learned
by
the
end
of
the
session,
you
will
have
converted
the
jupiter
notebook
to
you
flow
pipeline
with
scale.
This
makes
your
workflow
a
lot
easier
and
faster.
You
will
have
performed
by
pipeline
based
hyperparameter
tuning
with
caching,
which
makes
your
pipelines
run
faster.
A
A
A
The
machine
learning
platform
we
are
going
to
use
is
2flow
for
those
who
are
not
familiar
with
it.
Qflo
is
an
open
source
project
dedicated
to
making
the
deployment
of
machine
learning
workflows
on
kubernetes,
simple,
portable
and
scalable,
and
how
can
you
use
it?
You
can
deploy
and
manage
complex
machine
learning
systems
at
scale.
You
can
experiment
rapidly,
you
can
perform
hyper
parameter
tuning,
you
can
deploy
hybrid
and
multi-cloud
workloads,
and
you
can
have
the
show
coveted
feature
of
cicd,
continuous
integration
and
deployment.
A
A
A
A
A
Also
you
have
two
command
line:
interfaces,
jf
cattle
and
cubecad
and
of
course,
tubeflow
has
its
own
apis
and
sdks,
and
here
are
some
examples.
The
pipeline,
sdk
cadib
api
and
metadata
sdk
there's
a
perception
that
machine
learning
is
all
about
model
code
and
complex
math,
but
in
reality
machine
learning.
Applications
are
complex,
distributed
systems,
they
acquire
lots
of
devops
and
various
components
for
monitoring,
serving
managing
the
resources,
etc.
A
B
B
So
how
keyflow
and
the
tools
we're
going
to
present
will
enable
to
dramatically
simplify
and
accelerate
the
data
science
workflow,
starting
from
your
development
environment
that
is
most
often
jupiter,
jupiter
notebooks,
then
how
kale
as
a
workflow
tool
will
enable
you
to
create
pipelines,
seamlessly
seamlessly
and
run
them
in
cubesat
pipelines
and
then,
while
and
then,
while
doing
all
of
these
rock
will
enable
you
to
close
the
circle.
Go
all
the
way
back
from
running
to
developing,
essentially
with
complete
reproducibility,
enabling
you
to
explore
your
pipelines
back
in
time.
B
So
we
we're
talking
about
pipelines
as
the
as
the
core
workflow
tool
in
kubeflow.
This
is
because
by
data
science
is
inherently
a
pipeline
process.
Usually
when
you
do
data
science,
your
job,
your
work
consists
of
several
independent
steps
that
come
one
after
the
other.
This
may
be
data
ingestion
and
then
transformation
validation.
B
B
Now
this
workshop
will
focus
on
how
to
simplify
as
much
as
possible
the
building
process
of
a
pipeline
directly
from
your
local
environment
via
a
graphical
based
approach
and
how
to
completely
automate
data
versioning
to
enable
to
enable
reproducibility
and
collaboration.
Of
course,
the
tools
that
will
make
this
possible
are
kale
and
erictosrock.
B
Now
I've
been
talking
about
converting
a
notebook
to
a
pipeline,
so
this
is.
This
makes
a
lot
of
sense,
because
if
you
think
about
it,
when
you
work
in
a
jupiter
notebook,
you
are
already
defining
a
sort
of
workflow
that
is
pipeline
based
all
of
your
cells
in
the
notebook
can
be
thought
as
as
isolated
steps
that
can
be
even
combined
together
and
attached
to
one
another
with
dependencies.
B
This
process
makes
it
very
easy
to
parallelize
and
isolate
your
code
at
scale.
So
you
can,
you
could
even
easily
run
hyper
parameter
tuning
over
a
notebook
and
its
steps
and
then
also
it
becomes
very
easy
when
running
pipelines
in
the
cube
pipelines
to
apply
data
versioning
and
reproducibility
to
all
of
these
independent
steps
and
you'll
see
how
and
you'll
see
later
how
this
can
be
achieved
and
the
various
parts
of
your
notebook
that
become
independent
steps
might
might
have
different
requirements.
B
Let's
take
a
look
at
how
these
tools,
combined
together
with
kubeflow,
dramatically
improve
your
development,
workflow
and
lifecycle.
So
traditionally
you
would
write
some
machine
learning
code,
maybe
in
your
notebook,
but
then
once
you're
ready
to
build
a
pipeline
out
of
it
and
scale
it
up.
You
would
need
to
create
a
new
docker
image
where
you
package
all
of
your
dependencies
and
your
code.
B
Then
you
need
to
take
your
original
code,
your
machine
learning
code
and
actually
write
a
pipeline
with
some
specific
sdk.
For
example,
the
q
flow
pipelines
dsl
combine
this
pipeline
upload
it
and
run
it
whenever
you
need
to
go
back,
because
maybe
you
made
a
mistake
or
you
need
to
improve
your
own
code,
you
need
to
go
through
all
of
these
cycles.
This
cycle
again,
you
need
to
write
again
your
code,
repackage
it
into
a
docker
image
again
and
again,
but
now
with
cadence
rock.
B
All
of
these
processes
process
becomes
much
simpler
because
you
just
need
to
write
the
code
in
the
notebook
tag,
the
notebook
with
the
ui
driven
way,
you'll
see
later
how
and
with
the
click
of
a
button,
just
run
it
whenever
you
need
to
comment
your
code,
it's
seamless
just
write
it
in
the
notebook
and
click
the
button.
Again,
you
can
already
think
how
this
workflow
dramatically
improves
your
iteration
time
and
how
much
faster
allows
you
to
be
productive.
B
B
A
A
We
will
install
them
in
af,
then
we
will
get
some
already
cooked
machine
learning
code,
explore
it,
convert
it
through
pipeline
and
then
perform
hyper
parameter
tuning
on
a
specific
model.
The
first
step
is
to
install
minik
af,
but
let's
see
what
mini
af
is
mini.
Af
is
the
easiest
way
to
deploy
an
experiment
with
tube
flow.
In
just
a
few
minutes,
you
can
have
it
on
gcp,
on
your
laptop
or
any
on-prem
infrastructure.
A
A
Moreover,
cubeshop
provides
you
with
an
intuitive
ui
to
manage
all
your
components,
as
we
have
already
mentioned
to
your
ransomware
kubernetes,
which
takes
care
of
duplicates
of
orchestration.
But
then
you
also
need
to
manage
your
storage.
You
can
use
many
open
source
tools
for
your
cloud.
Provider's
object
store,
but
mini-af
comes
with
rok
our
data
management
platform.
A
We
have
made
a
lot
of
contributions
to
kubeflow
because
of
our
expertise
in
storage
and
data
management.
We
have
extended
tube
flow,
making
it
data
aware
you
contributed
code
so
that
your
flow
can
use
kubernetes
pvcs,
no
matter
where,
where
is
deployed
on
your
laptop
on
prem
or
a
public
cloud
rock,
takes
care
of
data
versioning
and
syncs
your
data
across
all
different
infrastructures.
A
A
A
B
B
So
I
will
head
over
to
my
mini
kf.
As
you
can
see
here,
I
have
the
queue
flow
sensor,
dashboard.
If
you
are
already
used
to
qflo,
if
you've
already
used
it,
you
might
notice
that
the
the
sidebar
here
is
a
little
bit
different.
This
is
because
we've
been
working
on
the
kubeflow
center
dashboard
to
unify
in
a
consistent
way,
all
of
the
key
flow
components
together
in
a
single
place,
and
you
will
see
how
I
will
be
using
these
throughout
the
tutorial.
B
B
B
B
B
B
B
B
B
B
B
Multiple
cells,
as
you
can
see,
can
be
merged
together
in
a
single
one,
and
then
it
is
that
simple
to
create
new
steps
and
dependencies.
Once
you
do
this,
of
course,
we've
already
tagged
this
notebook
for
simplicity.
You
can
just
select
an
eq
flip
experiment,
give
the
notebook
a
name
and
then
click
the
compile
and
run
button.
B
B
B
B
B
Okay,
so,
as
you
can
see
here,
I
have
my
two
volumes
that
are
basically
new
volumes
that
start
from
snapshot
that
were
taken
by
a
rock
when
I
clicked
the
button,
and
my
python
is
starting,
let's
give
it
a
little
bit
of
time
to
run.
While
I
go
back
to
the
slides
and
talk
to
you
and
walk
you
through
how
kale
is
actually
doing.
All
of
this
so
scale
is
basically
a
python
package
and
a
jupyter
lab
extension.
B
It
allows
you
to
convert
a
jupiter
notebook
to
cubesat
pipelines
in
a
completely
ui
driven
way
without
the
need
of
writing
any
kind
of
external
sdk.
How
does
it
do
that?
Well,
in
the
point,
you
click
the
compile,
compiler
button
running,
compile
button
ko
takes
the
notebook
and
parses
it
analyzing,
all
of
the
annotations
you've
added
via
the
key
ui.
B
This
turns
into
an
internal
graph
representation
that
is
parsed
by
ko
so
that
it
can
look
at
all
the
data
dependencies
between
the
notebook
cells,
so
cable
detects
these
dependencies,
and
then
it
takes
care
of
marshalling
the
data
between
the
pipeline
steps
for
you,
so
that
the
code
execution
is
completely
seamless.
Just
like
it
happens
in
the
notebook,
and
all
of
this
is
possible,
because
scale
also
takes
care
of
generating
a
whole
bunch
of
code
that
you
would
have
that.
You
don't
need
to
write
anymore
because
katie
is
doing
it
for
you.
B
B
So
kate
was
able
to
separate
this
code
into
multiple
steps
and
then
even
branch
out
the
training
of
the
several
different
deep
learning
algorithms,
because
the
dependencies
were
all
referring
to
a
common
step.
So
you
now
have
code
that
would
have
run
serial
in
the
notebook
running
concurrently
in
the
queue
for
pipeline.
B
Completed
as
you
can
see,
I
have
a
whole
bunch
of
logs
here
that
are
taking
that
are
basically
reproducing
the
exact
visualization
that
you
would
see
in
jupiter.
For
example,
here
the
cnn,
resonant
15
in
the
notebook
produces
a
visualization
at
the
end
of
its
computation
to
show
one
of
the
detected
images.
One
of
the
dog
breeds
detective,
so,
as
you
can
see,
kale
is
able
to
completely
reproduce
even
the
notebook
visualizations,
so
whether
you
are
plotting
with
matte
blocking
or
interactive,
with
a
visualization
with
blockly.
B
B
B
B
So
what
you
need
to
do
is
to
print
whatever
variable
your
algorithm
is
producing
and
that
you
want
to
be
used
as
an
optimization
metric
print
it
and
then
tag
the
cell
with
the
pipeline
metric
stock.
This
is
all
you
need
once
you
have
this,
you
can
go
over
the
kale
sidebar
enable
the
the
copy,
toggle
and
click
on
the
setup
cutting
job
button.
B
Now,
as
you
can
see,
okay,
it
provides
you
with
this
very
simple
to
use
ui
that
provides
the
input
parameters.
These
are
exactly
the
names
of
the
variables
I
defined
in
the
notebook
cell.
We
have
already
filled
the
values
we
want
to
use,
but
you
could
do
you
could
insert
here
whatever
values
you
want
from
ranges,
lists
integer
float
streaming
values.
What
have
you
then?
I
can
select
a
search,
optimization
algorithm
and
my
search
objective
again.
This
is
exactly
the
variable
name.
We
are
printing
in
the
pipeline,
metrics
cell.
B
That's
all
you
need
to
do.
I'm
clicking
close
and
again
the
compiler
run
button.
So
now
a
similar
thing
happens
as
before:
okay,
validates
the
notebook
rock
takes
an
immutable
snapshot
at
this
specific
point.
In
time,
then
kl
converts
the
notebook
uploads
a
new
pipeline,
and
now
here's
what's
happening.
B
B
As
you
can
see,
I
have
a
new
cutting
experiment
and
here
a
new
cubeflow
experiment.
They
have
the
same
names.
They
have
one
single
run
for
the
moment
again
with
the
same
name.
So
you
have
a
one-to-one
correspondence
here.
You
see,
which
are
the
input
parameters
that
can
be
decided
to
give
as
input
to
this
pipeline.
B
B
Ago,
okay,
so
this
was
an
experiment
created
in
the
same
way,
we
did
now
from
the
notebook
clicking
a
button
on
kl.
As
you
can
see,
I
have
several
pipelines
here
that
produced
a
specific
metric,
so
this
is
the
metric
that
we
were
bringing
from.
The
notebook
here
was
able
to
interpret
that
and
output.
This
metric
as
an
artifact.
B
B
In
the
meanwhile,
our
experiment
is
still
running,
and
you
can
see
here
you
can
have
a
live
view
directly
from
the
notebook
about
how
many
runs
are
are
running
in
this
moment
and
as
soon
as
one
at
least
one
would
finish.
You
would
see
here
a
live
view
over
what
is
the
current
best
result
as
well
all
from
your
notebook
okay,
so
that
was
it
for
the
hands-on
tutorial.
I
hope
you
will
have
time
to
go
through
your
code
lab
play
around
with
the
code.
Stop
start
your
own
pipelines
and
have
fun.
B
B
B
So
this
means
that
this
pipeline
was
cached.
Actually
these
steps
were
cached,
so
why
is
this
happening?
The
cnn,
resonant
50
step
is
the
only
one
in
the
pipeline
that
is
dependent
on
some
input
parameters.
It
is
the
only
part
of
the
node
on
the
original
notebook
code
that
reads
that
acts
upon
those
parameters.
B
This
means
that
running
load
data
and
the
tech
talks
several
times
would
be
a
waste
of
resources.
So
what
we've
done
is
we've
extended
the
current
caching
implementation
of
qflo
that
works
over
artifacts
to
work
also
over
snapshots
and
pvcs.
In
this
way,
the
running
time
and
the
performance
of
all
of
these
pipelines
is
greatly
improved.
B
B
B
Qflow
is
a
great
community
made
up
of
many
many
big
companies
and
as
a
community,
we
are
always
striving
to
find
new
contribution,
contributors
and
new
passionate
users.
So
please
get
involved.
You
can
find
here
lots
of
pointers
from
the
github
organization
to
our
slack
twitter,
handle
or
write
us
an
email
to
the
qflo
discuss.
Google
group
will
be
happy
to
support
you
and
on
board
you.