►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
ML Ops design patterns with Kubeflow Pipelines - Amy Unruh, Google
When moving your ML workflows from notebook exploration to production, many new problems can arise. We'll talk about some of the reasons this transition can be difficult; discuss patterns that can address these problems; then show how Kubeflow Pipelines (KFP) can be used to support and implement these patterns, and demo KFP in action.
A
A
A
Here's
an
incomplete
list
of
some
of
the
reasons.
These
are
things
we
hear
from
customers.
First,
data
cleaning
and
processing
becomes
very
hard
at
scale
dealing
with
data
cleaning
it
getting
it
to
where
it
needs
to
be
feature.
Engineering,
et
cetera,
is
a
large
part
of
the
overall
effort
required
to
put
an
ml
system
into
production.
A
Another
set
of
problems
can
arise
from
issues
like
model
or
data
drift
or
training
serving
sku.
Your
production
infrastructure
needs
to
be
able
to
detect
and
handle
situations
where
a
model
is
no
longer
sufficiently
accurate
or
new
data
might
indicate
model.
Retraining
is
necessary
or
online
prediction.
Data
is
not
consistent
with
the
feature
engineering
done
for
model
training
in
a
production,
environment,
access
control
and
security
become
more
important
as
well
and,
of
course,
lots
more.
A
So,
let's
look
at
some
patterns
and
practices
that
can
help
address
these
problems.
Here's
a
high
level
and
incomplete
list.
First
formalize,
your
machine
learning,
workflows
for
production,
move
away
from
stitched
together,
notebooks
or
monolithic
scripts.
The
suggestion
will
probably
come
as
no
surprise,
given
that
I
love
this
talk
equally
important.
Your
machine
learning
workflows
should
behave
in
the
same
way
across
environments.
A
Workflow
execution
should
also
support
burstability,
that
is
scale
out
resources
when
needed
and
scale
down
when
they're,
not
designing
for
composability,
modularity
and
reuse
of
workflow
building
blocks
will
ensure
that
you
can
reliably
reproduce
and
re-run
your
workflows
and
similarly,
your
infrastructure
should
support
workflow
monitoring,
versioning
and
caching.
This
typically
requires
making
ml
workflow
metadata
explicit.
A
A
Informal
methods
of
providing
team
members
access
to
your
notebooks,
etc
won't
scale
anymore,
so
that
was
a
useful
list,
but
how
to
operationalize
these
patterns
and
actually
use
them.
This
is
where
kubeflow
pipelines
comes
in
I'll
briefly,
introduce
it
and
then
talk
about
how
its
features
and
capabilities
support
these
patterns
and
practices
then
I'll
show
a
couple
of
examples:
keep
flow
pipelines
as
part
of
the
keepflow
project,
which
features
in
many
of
today's
talks.
A
It
has
the
goal
to
make
it
easy
for
everyone
to
develop,
deploy
and
manage
portable
scalable
ml
everywhere,
kubeflow
pipelines,
which
you'll
often
hear
abbreviated
as
kfp,
is
a
platform
for
building
and
deploying
machine
learning
workflows.
It
runs
on
a
kubernetes
cluster
and
its
pipeline
steps
are
container
based.
It
can
be
installed
with
the
rest
of
kubeflow
or
in
so-called
standalone
mode
without
the
other
parts
of
queue
flow.
A
First,
of
course,
q
flow
pipelines
lets
you
formalize
and
automate
your
machine
learning
workflows.
You
can
think
of
a
pipeline's
platform
as
the
backbone
of
production
ml
systems.
A
typical
workflow
might
look
like
the
one
on
this
slide,
with
experimentation
and
prototyping
stages,
as
well
as
automation
of
model
eval
and
model
deployment,
monitoring
and
retraining.
A
A
More
is
needed,
though,
than
just
workflow
orchestration
infrastructure,
explicit
use
of
pipeline
metadata
is
another
key
to
effective
ml
ops.
Once
we
can
treat
workflow
artifacts
things
like
models
and
data
sets
as
first-class
citizens
with
defined
schemas.
We
can
automatically
track
and
reason
about
their
use.
A
Here
are
some
of
the
ways
that
kfp
helps
support
that
qflo
pipeline
steps
are
container-based,
so
they'll
run
the
same
anywhere.
Similarly,
kubeflow
pipelines
can
be
installed
anywhere
that
a
kubernetes
cluster
can
be
set
up.
So
this
lets
keep
flow
pipelines.
Leverage
kubernetes,
auto
scaling,
allows
pipeline
tasks
to
create
and
manage
kubernetes
resources
and
I'll
show
an
example
pipeline
that
that
takes
advantage
of
that
in
just
a
few
moments.
A
Kfp
also
allows
pipelines
to
be
versioned,
so
it's
clear
which
version
of
a
given
pipeline
was
run
when
and
it
automatically
logs
metadata
and
tracks
pipeline
artifacts
during
execution
with
kfp.
It's
also
straightforward
to
clone
a
pipeline
and
run
it
again
or
to
retry
a
pipeline.
Kfp
supports
step
level.
Caching
as
well,
which
makes
it
really
straightforward
to
experiment
with
different
pipeline
variants
or
to
debug.
A
Support
for
modular
design
and
composable
pipeline
building
blocks
is
also
important
so
that
the
pipeline
steps
can
be
plug
and
play,
support,
easier,
debugging
and
can
be
reused
and
shared.
Qflo
pipelines
allows
users
to
share
components
by
components.
I
mean
specification
of
pipeline
steps
as
well
as
pipeline
definitions.
A
The
kfp
sdk
also
makes
it
easy
to
create
pipeline
components
without
needing
to
directly
use
docker,
which
can
be
helpful
often
for
data
science
teams,
particularly
in
the
prototyping
phase.
I'll
show
that,
in
an
example
as
well,
and
while
I
won't
have
time
to
show
it
in
this
talk,
there's
now
multi-user
support
for
kfp
when
you
install
it
as
part
of
kubeflow,
given
that
pipeline
components
are
composable
and
reusable.
The
next
step
is
to
provide
pre-built
building
blocks,
to
guide
construction
of
canonical
ml
workflows
and
reduce
the
need
to
build
your
workflows
from
scratch.
A
The
tensorflow,
extended
project
provides
a
number
of
such
components
in
libraries
and
I'll
show
an
example
that
uses
one.
In
addition,
pipeline
steps
can
call
out
to
any
service,
so
we
can
provide
pre-built
components
that
wrap
ml
services.
There
are
many
sets
in
the
kfp
repo
and
more
being
added
all
the
time.
A
A
Another
ml
ops
pattern
relates
to
supporting
tooling
around
managing
pipeline
runs.
One
important
category
is
making
sense
of
execution
results.
This
includes
the
ability
to
organize
pipeline
runs
into
semantic
groupings.
Kfp
calls
them
experiments
and
to
easily
compare
and
visualize
results
across
runs.
A
Kfp
components
can
be
designed
to
output
metadata
that
renders
as
visualizations
in
the
dashboard,
and
you
can
spin
up
a
tensorboard
server
on
right
on
the
cluster
by
including
a
pre-built
component
pre-built
pipeline
step
when
you
build
your
pipeline
I'll
show
that
in
one
of
my
examples
another
important
tooling
category
is
cicd.
For
example,
when
pipelining
and
components
are
changed,
new
versions
are
automatically
built
and
tests
are
run.
A
A
A
The
machine
learning
task
in
both
cases
will
be
to
train
a
keras
model
to
predict
rental
duration.
The
model
details
aren't
important
for
this.
Talk,
though,
for
both
example,
pipelines
I'll,
be
using
a
so-called
standalone
installation
of
kubeflow
pipelines
running
on
a
gke
cluster,
where
gke
is
google's
hosted.
Kubernetes
I'll
first
show
an
example
that
highlights
some
of
the
ways
that
a
kubeflow
pipeline
can
leverage
its
underlying
kubernetes
cluster.
A
So
this
is
what
the
graph
for
the
this
pipeline
looks
like
pipeline's,
using
a
pre-built
component
to
spin
up
a
tensorboard
server
on
the
same
cluster
and
then
the
k-tune
step
launches
the
tuner
workers
and
controller
as
kubernetes
jobs
to
do
the
hyper-parameter
search
and
when
they're
all
finished
returns.
The
n-best
parameter
sets
once
the
tuning
search
is
completed.
A
The
pipeline
trains
end
full
models
using
the
best
parameter
sets
change
them
in
parallel,
using
the
keep
flow,
pipeline's
loop
structure
and
while
they're
training,
we
can
use
the
tensorboard
server
to
monitor
the
training,
runs
after
training.
Each
full
model
is
evaluated
and
if
it
is
of
sufficient
accuracy-
and
it's
a
condition
check
here,
it's
deployed
the
models
are
deployed
using
tensorflow,
serving
where
the
tf
serving
services
are
also
spun
up
on
the
same
cluster.
A
So
we're
using
the
kubernetes
cluster
not
only
to
run
the
pipeline,
but
to
launch
jobs,
to
perform
a
distributed,
hyperparameter
search
to
use
tensorflow,
serving
to
serve
models
and
to
run
a
tensorboard
server.
And,
of
course
the
cluster
will
auto
scale
when
we
need
it
to.
If
we
want
to
use
more
keras
tuner
workers
or
train
more
models
concurrently,
so
now,
let's
see
what
it
looks
like
to
upload
and
run
a
new
version
of
the
pipeline,
I'll
click
on
upload
version
to
upload
a
compiled
pipeline
archive
file.
A
In
the
next
example,
we'll
see
what
the
pipeline
specification
looks
like
by
uploading
new
versions
of
a
given
pipeline,
rather
than
just
creating
a
new
pipeline.
It
lets
me
track
and
group
related
pipelines
in
their
runs.
Then
I'll
click
create
experiment
to
run
this
pipeline
under
the
grouping
of
a
given
experiment.
So
this
lets
me
semantically,
organize
the
pipeline
runs
then,
in
the
start
or
run
page
I'll
enter
the
pipeline
run
parameters
some
have
default,
values,
I'll,
keep
and
others.
I
will
change
while
I'm
not
showing
it
here.
A
I
can
also
clone
or
retry
an
existing
pipeline
run
thing
I'll.
Do
is
specify
to
train
full
models
using
the
best
three
parameter
sets
returned
from
the
keras
tuner.
The
default
is
to
train
the
best
two
which
gives
the
pipeline
graph
shown
in
the
top
image
and
when
I
change
it
to
three
I'll
get
the
pipeline
graph
shown
in
the
bottom
image.
A
This
is
made
possible
by
using
the
pipeline
so-called
parallel
4
construct
to
launch
n
training
jobs
in
parallel,
because
I've
configured
my
cluster
to
support,
auto
scaling.
I
can
change
both
the
number
of
keras
tuner
workers
or
the
number
of
concurrent
training
jobs
and
let
the
cluster
scale
out
when
it
needs
to
next
I'll
start.
The
run
note
that
one
of
the
steps
is
configuring,
a
tensorboard
server
that
will
run
on
the
cluster.
A
The
k-tune
step
manages
and
coordinates
the
keras
tuner
workers,
which
are
kubernetes
jobs.
While
the
tuner
workers
are
running,
we
can
see
them
in
the
gke
dashboard.
The
k-tune
step
waits
for
the
workers
to
complete
all
their
trials
and
outputs
the
top
and
best
results,
and
in
this
case
we
requested
the
top
three.
A
A
A
A
For
my
second
example,
I'll
walk
through
parts
of
this
notebook.
This
example
builds
on
the
training
and
serving
components
used
in
the
first
example
and
add
some
additional
ones.
It
shows
how
to
use
the
tfx,
tensorflow
validation,
libraries,
often
abbreviated
as
tftv,
to
detect
data
drift
between
different
versions
of
a
data
set.
If
drift
is
over
a
given
threshold,
the
model
is
retrained.
A
A
While
I
won't
have
time
to
cover
it,
the
notebook
also
shows
how
to
set
up
event
triggered
pipeline
runs
I'll
share
a
link
to
the
notebook.
At
the
end
of
this
talk,
I've
done
some
setup
already.
So,
let's
skip
to
building
the
new
components,
I'm
going
to
create
python
function,
based
components,
I'll
define
a
function
that
implements
each
pipeline
step,
then
convert
it
to
a
component.
A
The
first
function
generates
the
stats
for
a
data
set.
The
function
returns
a
string
that
holds
the
path
to
the
generated
stats
info.
When
the
function
is
compiled
to
a
component,
it
will
have
an
output
called
statspath
that
other
components
can
consume.
As
input
we'll
see
this
in
a
minute,
the
given
dataset
is
large.
It
can
be
hard
to
do
this
analysis
in
memory.
The
function
allows
the
option
to
launch
the
analysis
as
a
cloud
dataflow
job
dataflow
is
google's,
hosted
apache
beam
if
it
needs
to
scale
out.
A
This
is
a
good
example
of
using
pipelines
to
orchestrate
calls
to
other
cloud
services
now
I'll
convert
this
function
to
a
pipeline
component.
When
I
do
this,
I
need
to
specify
the
base
container
image
to
use
default
is
a
python
3.7
image,
but
I'll
use
one
that
already
installs
the
tftv
libraries
to
save
time
as
part
of
the
process,
I'm
generating
a
component
yaml
file.
This
can
be
put
under
version
control
and
shared
with
others.
A
A
A
A
A
Next,
we'll
upload
the
pipeline
get
its
id
and
then
launch
a
pipeline
run.
You
can
also
combine
the
upload
and
run
as
a
single
call.
If
you
like,
when
we
run
the
pipeline,
we
can
pass
it
the
input
arguments
to
use
once
the
run
is
launched.
We
can
view
it
in
the
kubeflow
pipelines
dashboard
then
several
hours
later,
the
result
looks
like
this.
In
the
case
where
we
do
detect
drift
between
the
old
new
data
sets
and
decide
to
retrain
the
model.
A
I
showed
some
mlaps
patterns
that
can
help
and
talked
about
how
kubeflow
pipelines
helps
operationalize.
These
patterns
and
practices
you
can
think
of
the
capabilities
provided
by
kubeflow
pipelines
is
largely
falling
into
three
buckets
ml
workflow
orchestration,
share,
reuse
and
compose
components
and
pipelines,
and
then
run
them
across
multiple
clouds,
on-prem
anywhere
that
kubernetes
itself
runs
and
rapid,
reliable
experimentation,
including
metadata
tracking
and
parallel
experiments.