►
From YouTube: Scaling ML pipelines with KALE — the Kubeflow Automated Pipeline Engine - Salman Iqbal, Learnk8s
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling ML pipelines with KALE — the Kubeflow Automated Pipeline Engine - Salman Iqbal, Learnk8s
One of the most common hurdles with developing AI and deep learning models is to design data pipelines that can operate at scale and in real-time. Data scientists and engineers are often expected to learn, develop and maintain the infrastructure for their experiments. What's the best setup and what's involved in getting models being production-ready? Where do you start? In this talk, you will learn about KALE — the Kubeflow Automated Pipeline Engine. With KALE you can finally link the work done by data scientists in Jupyter Notebooks to a production-grade pipeline that trains the models at scale and serves them in real-time.
A
Hey,
what's
up
everyone
welcome
to
ai
days.
My
name
is
salmanik
ball
kubernetes
trainer
with
learn
k-8s.
I
also
work
as
an
emma
lops
engineer
for
appvia
at
applia.
We
work
with
a
number
of
organizations
in
helping
them
with
a
cloud-native
and
amalops
journey
with
my
friend
louis
denim
parry.
I
also
run
cloud
native
wales
meetup
here
in
wales.
We
talk
everything
cloud
native.
You
can
find
me
on
twitter
at
seoulminecpal.
A
A
Wouldn't
it
be
great
if
you
could
write
your
machine
learning
or
data
science
code
in
jupyter,
lab
notebook
and
magically
turn
it
into
a
pipeline.
I
work
with
a
number
of
machine
learning
engineers
and
data
scientists
who
think
this
will
be
a
great
idea
so
in
this
video
we're
going
to
be
talking
about
kale,
the
qfloor
automated
pipeline
engine
that
magically
turns
your
code
in
jupiter
lab
notebooks
into
pipelines,
but
before
we
dive
into
it.
A
Let's
take
a
look
at
some
of
the
problems
that
data
scientists
and
machine
learning
engineers
face
day
to
day
and
how
this
helps
them.
As
a
data,
scientist
or
enamel
engineer,
we
would
think
that
we
only
have
to
deal
with
a
few
things.
For
example,
perhaps
we
only
worry
about
gathering
training
data
and
splitting
it
up
into
training
and
test
data,
and
perhaps
we
only
care
about
writing
our
model
and
training
our
model
and
then
evaluating
what
we
got,
but
the
reality
is
far
from
it.
A
A
A
I
need
to
monitor,
what's
actually
happening
in
my
infrastructure
and
also
set
up
some
analysis
tool.
Basically,
what
we're
saying
is
a
lot
is
expected
from
data
scientists
and
ml
engineers.
You
not
only
need
to
know
about
ml.
You
need
to
be
an
expert
in
software
engineering
infrastructure,
creating
pipelines.
A
A
Imagine
you
work
in
an
organization
and
you
are
in
charge
of
three
different
models:
recommendation,
anti-spam
and
churn
rate
daily.
You
get
data
and
you
have
to
update
your
model,
and
you
know
you
keep
updating
your
model
on
a
day-to-day
basis
on,
on
a
single
day,
do
the
same
for
anti-spam,
based
on
what
the
data
is
coming
in.
You
keep
updating
it
and
also
you,
you
revise
your
model
for
the
churn
rate
use
our
monday,
tuesday,
wednesday,
thursday
friday
and
if
you're
unlucky,
which
most
of
us
sometimes
are
even
on
saturday
and
sunday
now.
A
This
is
where
containers
come
in
handy.
If
you
are
already
aware
of
what
containers
are.
This
is
just
a
quick
recap
on
what
containers
actually
are.
What
a
container
is.
I
can
take
a
runtime,
for
example,
in
this
case
we
can
take
python
and
I
can
add
my
code
files
or
any
other
configuration
files
that
might
be-
and
I
can
also
add
external
dependencies
like
tensorflow
and
package
it
all
up
in
a
linux
container.
A
Now
all
my
dependencies
all
my
code,
all
my
runtime
is
contained
within
this
container.
I
can
take
this
container
and
run
it
anywhere.
I
can
run
this
on
my
laptop.
I
can
run
it
on
the
cloud.
I
can
run
it
on
on-prem
clusters.
I
can
run
it
anywhere.
If
you
only
have
one
container
or
two
containers,
you
can
perhaps
manage
to
run
it
yourself,
but
if
you
have
more
than
one,
if
you're
more
than
two,
if
you
have
like
hundreds
or
tens
of
hundreds
of
containers,
how
do
you
manage
it?
A
This
is
the
control
plane
now
think
of
kubernetes
as
the
best
tetris
player
ever
imagine.
You
have
a
container
that
needs
to
be
run.
What
kubernetes
does
is
scans
the
nodes
and
figures
out
where
to
place
this
container.
In
this
case
it
says
I'll
put
in
the
left
one
and
then
the
same
application
comes
in
again
tries
to
spread
the
load
and
it
puts
it
in
the
second
machine
and
another
one
comes.
A
This
is
smaller
application.
Perhaps
it
doesn't
require
as
much
memory
or
as
much
cpu.
So
you
know
it
places
on
the
left
one
and
then
the
next
one
comes
and
it
starts
placing
it
basically
every
time
your
application
in
terms
of
container
that
needs
to
be
run.
Kubernetes
managed
and
kubernetes
manages
the
manages
the
placement
of
these
containers
onto
the
infrastructure,
and
if
even
one
of
them
fails,
it
can
try
and
respawn
it
again
and
what
it
will
try
and
do
is
package
everything
fully
into
the
cluster.
A
Now
it's
time
to
introduce
you
to
kubeflow
kubeflow
is
an
open
source
community
dividend
project
which
makes
it
easy
to
deploy
and
manage
an
ml
stack
on
kubernetes
qfloor
project
helps
machine
learning,
engineers
and
data
scientists
with
most
of
the
issues
that
we
discussed
earlier
on,
even
though
we're
focusing
on
q
flow
pipelines,
if
you're
not
aware
of
kubeflow,
I
think
it
might
be
a
good
idea
for
me
to
give
you
a
quick
intro
to
it.
Qflow
is
the
ml
toolkit
for
kubernetes.
A
A
A
The
notebooks
data
scientists
and
ml
engineers
can
use
to
experiment
with
model
and
data
queue
flow
pipelines,
their
reusable
end-to-end
machine
learning,
workflows
built
using
the
qfloor
pipeline's
sdk
and
the
sdk
is
in
python.
Yes,
you
heard
it
right.
It's
in
python.
You
can
write
your
pipelines
in
python.
You
don't
have
to
write
them
in
yaml
or
any
other
configuration
language.
You
can
do
it
in
python.
It
basically
is
an
end-to-end
orchestration.
You
can
write
an
end-to-end
ml
pipeline.
It's
easy
experimentation.
You
can
trial
experiments
quickly
and
iterate
over
them.
A
A
These
operators,
let
us
use
the
functionality
of
deep
learning
and
ml
frameworks
that
we
like,
for
example,
pytorch
mxnet
with
queue
flow.
A
number
of
operators
are
installed.
Hyper
parameter
tuning
is
a
big
topic
in
kubernetes
with
qflow.
You
can
use
catheter
for
automated
tuning
of
your
machine
learning
models,
hyper
parameters
and
architecture.
A
The
goal
of
the
metadata
project
is
to
help
kubeflow
users
understand
and
manage
their
machine
learning
workflows
by
tracking
and
managing
the
metadata
that
the
workflows
produce
metadata
means.
Information
about
executions,
for
example,
runs
models,
data
sets
and
other
artifacts,
for
example,
files
and
objects
that
form
the
inputs
and
outputs
of
the
components
in
your
ml
workflow.
It's
like
source
control
for
machine
learning,
resources,
kf
serving
uses,
k
native
to
provide
a
kubernetes
custom
resource
definition
for
serving
machine
learning.
A
A
The
platform
actually
consists
of
a
user
interface
for
managing
and
tracking
experiments.
As
you
will
see
in
a
second
and
an
engine
for
scheduling,
multi-step
ml
workflows,
it
comes
with
an
sdk
for
defining
and
manipulating
pipelines
and
components,
and
we
can
write
these
pipelines
in
notebooks
using
kale
and
run
them
directly
from
the
notebooks
once
the
pipeline
is
written
in
python
and
uploaded
in
qfloor
pipelines.
This
is
how
the
visualizer
of
qfloor
pipeline
presents
in
order
to
write
pipelines
in
kubeflow
there's
a
number
of
steps
we
have
to
go
through.
A
A
Once
that's
done,
you
have
to
compile
the
pipeline,
and
then
we
get
an
output
in
a
yellow
file
which
we
can
yammer
file
or
a
zip
file
which
we
have
to
upload
the
pipeline.
To
qfloop
python,
and
once
that's
done,
we
have
to
run
it
in
kubeflow
and
if
you
need
to
make
any
changes,
you
can
imagine
you
have
to
go
back
to
change
your
code,
rewrite
it
can
create
the
image
again,
compile
the
pipeline
again
re-upload
it
again.
That's
a
long
process
with
kill.
A
We
can
iterate
through
things
a
lot
quicker
when
you
have
to
write
the
code
in
the
notebook
tag.
The
notebook
cells
correctly,
which
we'll
see
in
a
second
and
then
run
the
pipeline
with
a
single
click.
You
can
imagine
if
you
like
to
iterate
through
and
test
out
the
changes
in
our
pipeline
really
quickly.
We
can
do
that
using
kale.
A
If
we
were
to
use
qfloor
pipelines
directly,
it
will
take
us
a
lot
longer
to
be
able
to
build
the
image
upload
it
and
then
come
back
and
make
the
changes
again.
So
kale,
the
kubeflow,
automated
python
engine
allows
us
to
take
the
code
that
we've
written
in
jupiter
lab
notebook,
there's
an
extension
that
we
have
to
install
for
it.
A
We
annotate
the
cells
in
the
notebook
with
the
correct
tags
and
convert
it
into
a
pipeline.
Let's
quickly
do
a
demo
of
what
it
looks
like
we'll:
compare
how
to
run
the
pipeline
using
q4
pipelines
and
then
doing
it
with
kale.
Okay,
for
this
demo,
we
will
be
using
mini
kf
mini
kf
lets.
You
run
kubeflow
locally
on
your
machine.
Now,
let's
have
a
look
at
how
we
can
do
this.
A
You
can
head
over
to
the
website
from
mini
kf
on
kubeflow.org.
That
gives
you
instructions
on
how
to
install
it.
Once
you
get
vagrant
on
your
machine
and
virtualbox
on
your
machine,
you
will
be
able
to
set
it
up
really
quickly.
You
don't
have
to
use
mini
care.
You
can
use
any
kubernetes
cluster.
You
like
now,
once
it's
set
up,
you
can
do
vagrant
in
it
a
richter,
slash,
mini
kf
and
do
vagrant
up
after
a
little
while
it
will
spin
up
a
virtual
machine
and
install
mini
kf
in
it.
A
A
You
can
use
any
of
the
other
images
or
you
can
even
provide
a
custom
image
that
you'd,
like
yourself,
you've
got
cpu
and
memory
that
you
can
increase.
Once
that
happens,
we
can
launch.
We
don't
need
to
do
anything
else.
We
can
just
launch
it
from
here
so
now
that
will
take
a
little
time
and
set
it
up,
but
what
we're
going
to
do
is
use
one
that's
already
been
set
up
and
for
it
to
load.
Now
this
is
a
brand
new
notebook.
A
A
Now
let
that
load
then.
So
what
we
want
to
do
is
see
how
we
can
create
a
pipeline
using
normal
cube
flow
and
how
it
compares
to
the
pipeline
that
we
will
be
creating
using
kale.
Now,
let's
head
over
to
the
pipelines,
we
already
have
some
examples
in
here
that
we
can
use.
So
let
me
just
show
you
one
of
them.
So
if
I
go
to
samples
core
and
we're
going
to
look
for
execution
order,.
A
So
this
is
what
a
kubeflow
pipeline
looks
like
if
you've
not
seen
it
I'll
quickly,
give
you
a
rundown
on
how
it
works.
So
what
happens?
Is
we
use
this
library
kfp
queue
for
a
pipeline,
and
we
use
this
domain
specific
language
dsl
part
of
that
library?
Now
every
step
in
here
is
a
container.
So
we
say
this
is
a
container
off
this
one
uses
a
bash
container
and
we
run
some
commands
in
it
and
we
pass
some
arguments.
A
This
is
another
another
step
in
the
pipeline,
so
these
are
all
different
steps
and
once
we
define
the
different
steps,
what
we
can
do
is
define
the
execution
order
of
the
pipeline.
We
can
say
we
want
to
run
this
one
after
the
other,
so
we
define
the
two
tasks
step.
One
is
echo
one
up
and
echo
up
two
and
once
we're
happy
with
that,
what
we
can
do
is
basically
just
to
do.
We
want
to
run
it
one
after
the
other
and
the
last
bit
here
is
kfb
compiler.
A
So
this
is
the
bit
that's
actually
going
to
compile
this
into
a
yaml
file
and
how
we
do
that
is.
If
I
head
over
to
the
directory
of
where
this
is,
we
will
be
able
to
compile
it
and
upload
it
and
see
how
it
looks.
So,
if
I
just
quickly
do
this,
so
this
will
give
us
a
yaml
file
that
we
can
go
back
and
upload
in
the
keyflop
pipelines.
Ui,
that's
a
yama
file
we
can
see
this
is
this
is
a
yellow
file
and
we
can
download
this
file
and
upload
it
in
in
kubeflow.
A
So,
let's
head
over
to
kubeflow
pipelines
in
here,
you
can
see
that
some
pipelines
have
already
been
created.
Now,
if
I
go
on
a
pipeline,
I
can
upload
a
pipeline
give
a
name.
So
this
is
our
normal
pipeline.
This
is
our
kfp
pipeline
and
we'll
give
it
some
description.
Let's
go
to
test,
I'm
going
to
upload
the
file
that
I
just
downloaded
from.
A
Kubeflow
and
now
I
can
create
this
pipeline
and
it'll
take
a
little
while
for
it
to
create,
but
I
can
show
you
one
that
I
created
earlier.
So
this
is
the
pipeline
that
we
created
and
it
will
load
up
in
a
second
and
we
will
be
able
to
see
the
graph
now
once
that's
set
up,
that's
okay!
If
you
want
to
make
a
change,
I
have
to
go
back
into
my
notebook
and
open
the
python
file
update
it,
make
the
change,
compile
it
again
and
upload
it
again.
A
It'll
take
a
long
while
for
me
to
make
any
changes
that
if
I
want
to
experiment
with
it
now
this
is
where
kale
comes
in.
If
you
go
into
notebooks,
if
caleb
is
not
already
installed,
you
will
be
able
to
install
it
yourself.
So
if
I
just
searched
for
kubeflow
kale,
let's
go
in
here.
So
this
is
the
document
github
repo
for
kill.
If
I
go
down,
you
will
be
able
to
see
the
instructions
and
how
to
be
how
to
install
it.
A
Now
the
image
that
we
already
using
the
the
one
from
ericto
that
already
has
the
kl
deployment
panel
and
available
to
us
all
I
can
do-
is
click
on
this
and
enable
it
now.
Currently,
I
can't
run
any
experiments
because
I
have
a
name.
I
haven't
opened
a
notebook
that
has
experiments
but
I'll
show
you
where
these
experiments
are
and
how
they
actually
work.
A
This
is
there
you
go
here's
the
pipeline,
that's
been
created
for
us
and
you
know
we
can
execute
this
pipeline,
create
a
run
or
whatever
we
want
to
do,
and
we
can
get
an
output
from
this.
So
this
is
a
pipeline,
that's
created
normally,
but
what
we
want
to
do
is
we
want
to
actually
run
it
directly
from
our
notebook.
You
want
to
write
the
pipeline
in
the
notebook
and
click
the
button
in
the
notebook
that
you
click
the
button
from
our
notebook.
A
That'll
run
the
pipeline
directly,
so
let's
head
over
back
and
let's
check
it
out.
So
if
I
just
head
over
to
my
pipelines
now,
so
this
is
the
examples
folder
I'm
going
to
pick
one
of
those
titanic
examples.
This
is
already
from
queue
flow
itself.
This
is
an
example,
for
basically,
what
we're
trying
to
do
is
use
machine
learning
to
create
a
model
that
predicts
this
passenger,
which
passengers
survived
the
titanic
shipwreck.
A
The
details
of
the
notebook
are
not
too
important
because
I
don't
really
want
to
go
into
the
machine
learning
part
of
it,
but
what
I
want
to
show
you
is:
how
do
we
use
kale
to
be
able
to
annotate
your
blocks,
your
sections
within
the
notebook
and
basically
run
them
as
a
pipeline
and
how
we
get
them
to
rely
on
one
another
and
make
them
dependent?
So
basically
it
runs
as
an
actual
pipeline.
A
Now,
what
you'll
see
is
the
notebook
looks
slightly
different
in
this
case
I'll,
explain
some
some
steps
and
then
we'll
go
ahead
and
run
this
now.
Every
cell
you
see
in
a
notebook,
is
annotated
and
there's
different
types
of
annotations.
You
can
add.
So
if
I
click
on
this
edit
button
here,
you
can
see
this
one
is
an
import
type.
So
it's
basically
importing
the
libraries
I've
we've
got
four
different,
five
different
types.
We
have
functions
and
what
that
allows
you
to
do
is
prepend
to
every
step.
The
pipeline.
A
This
function
will
be
pre-appended
to
every
step.
We
can
also
set
up
pipeline
parameters.
If
you
want
to
pass
pipeline
parameters
for
every
time
runner
run
a
pipeline.
Perhaps
you
want
to
run
it
with
a
different
parameter,
so
we
can
define
the
parameters
or
this
is
the
bit
that
we
used
most,
which
is
the
pipeline
step
and
we'll
give
it
the
pipeline
step,
and
we
can
also
tell
it
which
one
it
depends
on
and
the
last
one.
A
Sometimes
we
probably
want
to
skip
some
cells
that
are
in
the
in
the
notebook
for
debugging
purposes
or
visualization
purposes,
but
we
don't
want
them
in
the
pipeline.
So
this
is
import,
and
this
is
all
good.
But
before
I
even
do
anything,
I
need
to
install
the
the
requirements
that
are
required
for
this
notebook
within
within
the
notebook.
So
I'm
just
going
to
quickly
jump
back
to
my
terminal
and
run
that
so,
let's
just
pick
three
install
mysr
requirements.txt.
A
Now,
let's
just
head
back
to
the
notebook
and
see
what
it's
doing
so,
the
first
one
we
talked
about
is
an
import,
and
you
can
see
this
is
a
pipeline
step
we've,
given
it
a
name,
there's
nothing
special
about
it,
we're
just
the
normal
pipeline
stuff
and
we're
giving
it
a
name,
and
what
we
can
also
do
is
make
that
pipeline
step
dependent
on
another
step.
You
can
see
there's
some
other
steps
in
here
that
have
been
defined
later
on.
Let
me
show
you
some
more
these
steps.
A
A
And
when
we
get
down,
you
will
see
this
one
data
processing
step
this
cell
is
now
is
a
pipeline
step
that
we
said,
that's
the
name,
but
also
this
depends
on
the
load
data
step
from
before
that
we
defined,
and
you
can
define
multiple
steps
if,
if
it
depends
on
so
there's
again,
you
know
we're
basically
just
going
through
the
data
here
doing
some
training,
I'm
just
going
to
skip
that
part.
Now.
A
Now,
our
notebook
is
all
good
to
go.
All
the
requirements
are
installed.
Now
I
can
open
my
cube
flow
here
and
pick
this,
so
I
can
say.
Oh
the
experiment
I
want
to
run
is
titanic,
because
this
is
based
on
the
notebook
that
I've
just
opened,
given
it
a
name
and
what
I
can
do
is
you
can
see.
I
don't
have
to
compile
anything
from
a
command
line
or
anything
like
that.
I
can
click
on
this
button
compile
and
run.
A
So
when
I
do
that
it
will
take
a
snapshot
because
in
this
case
we
are
taking
snapshots
of
the
pipeline.
If
you
wanted
to
run
it
beforehand,
if
you
wanted
to
run
it
afterwards
again
and
what
we'll
do
is,
it
will
take
the
pipeline
and
upload
it
into
kubeflow
pipeline
itself
directly.
We
don't
have
to
download
the
file
and
do
that
and
everything
is
defined
within
the
notebook.
So
snapshot
is
done
now,
it's
kind
of
trying
to
upload,
so
the
upload
has
been
successful.
A
A
You
can
see
we
create
volume
in
the
beginning
and
loading
the
data
data
processing.
These
are
all
the
steps
that
we
saw
right
feature
engineering.
These
steps
depend
on
one
another
and
you
know
we're
doing
some
of
the
other
steps
in
the
end,
and
then
we
say
the
result
step
is
dependent
on
all
these
other
steps.
So
that's
how
we
create
this
dependency
graph
and
all
the
pipeline.
Now,
once
that
happens,
it
will
actually
kick
off
the
experiment
itself.
Now
that
is
not
something
very
exciting.
A
As
you
can
see,
the
experiment
has
been
kicked
off
is
actually
running
right
now,
but
let's
have
a
look
at
one
that
was
completed
before.
Let
me
click
on
this,
so
this
is
how
it
ran
through.
But
if
you've
not
seen
a
pipeline
before
you
can
see,
all
these
steps
are
running
through
running
the
code
that
you
wrote
in
your
notebook.
We
didn't
write
a
single
container,
there's
no
container
inside.
You
can
check
the
logs.
If
you
want
to
check
the
logs
and
then
you
know.
In
the
end,
there
is
an
output
as
a
results.
A
Output.
That's
that's
happened
here.
So
all
we
did
was
wrote
our
code
in
the
notebook
and
from
the
notebook
directly
we
compiled
and
ran
it.
We'd
never
produced
anything
of
a
container
image
or
produced
a
pipeline
file
and
uploaded
it.
This
is
the
beauty
of
kale.
We
ve,
we
very
quickly.
We
managed
to
run
this
pipeline.
If
you
like
more
details
on
this
topic,
definitely
check
out.
The
kubeflow
kale
also
hit
me
up
on
twitter
at
soulmaniqbar,
and
I'm
here
to
answer
any
questions.
Thank
you
for
attending.