►
Description
A Component Registry for Kubeflow Pipelines - Christian Kadner, IBM
Kubeflow Pipelines are widely used to orchestrate machine learning (ML) workflows on Kubernetes. Pipelines and individual pipeline stages are often worked on collaboratively. To facilitate that process Kubeflow Pipelines support re-usable components, self-contained sets of code that performs one step in the ML workflow, like data preprocessing, data transformation, model training, and model serving. There is a rich set of components from community and vendors. What has been missing from the ecosystem however, is a registry for sharing reusable components with the public or among teams of data scientists. Thus many of the common tasks required to run ML workflows on Kubernetes like creating secrets, persistent volume claims, config maps have to be implemented again and again. A component registry can provide a rich catalog of components to solve those common tasks and ease the burden of creating ML workflows on Kubernetes.
A
A
I
will
quickly
talk
about
the
ai
life
cycle,
introduce
q4
pipelines
and
point
out
some
reusability
challenges,
I'll
talk
about
pipeline
components
and
the
new
proposed
component
registry
that'll
lead
me
to
the
machine
learning
exchange
and
I
will
talk
about
some
of
the
technologies
we
have
integrated
and
also
quickly
show
watson,
studio
pipelines
in
the
demo.
I
will
show
what
the
machine
learning
exchange
is,
what
it
can
do
and
also
show
the
new
template
registry
api.
A
So
let's
dive
right
in
into
it.
What
are
the
stages
of
the
air
life
cycle?
Well,
it's
mostly
about
models,
data
models
and
data
sets.
So
in
a
nutshell,
we
use
data
to
build
models
in
order
to
automate
decisions
and
each
of
those
big
steps.
They
have.
You
know
smaller
steps,
you
know
data
we
have
together.
We
have
to
analyze
it
the
models
they
have
to
be
trained,
it
can
be
traditional
machine
learning,
models,
deep
learning
models
and
once
they're
trained,
we
want
to
deploy
them
and
we
have
to
maintain
them
now.
A
It's,
of
course
not
that
simple.
If
we
zoom
in
just
a
bit,
you
see
there's
many
more
steps
involved
and
I'm
a
software
developer,
not
a
data,
scientist
or
machine
learning
expert.
So
to
me
this
is,
is
already
way
too
complicated
so
alone.
The
data
preparation
has
multiple
multiple
steps
and
stages
right.
A
You
have
to
optimize
the
hyperparameters,
you
have
to
validate
the
model
and
then
once
your
model
is
finally
trained,
you
want
to
deploy
it
on
the
cloud
or
even
on
edge
devices,
and
then
you
want
to
inference
you
feed
it
data
and
you
get
prediction
out
of
predictions
out
of
it.
All
of
that
has
to
be
monitored
and
logged,
and
you
know
if,
if
the
predictions
over
time,
you
know
show
anomalies
or
you
have
some
drift,
these
models
have
to
be
fine-tuned,
and
you
know
the
whole
process.
A
A
That
process
remains,
you
know,
bifurcated
amongst
various
teams
and
the
artifacts
created
or
consumed.
Oftentimes
are
not
shared.
It's
that
they're
going
to
be
recreated
multiple
times
over
and
then,
of
course,
there
are
challenges
with
traceability
and
governance,
risk
management
and
lineage
tracking
and
metadata
collection.
So
what's
needed
is
a
central
catalog
for
all
of
those
ai
and
ml
assets.
That
can
be
shared
and
reused
across
organizational
boundaries,
and
these
data
sets
and
models.
You
know
they
need
to
have
to
have
quality
checks,
proper
licenses.
A
You
know
linux
tracking
and
all
of
that
is
required
to
speed
up
their
lifecycle.
Now,
let's
start
with
q4
pipelines,
what's
the
purpose
of
pipelines
so
pipelines
they
offer.
You
know
end-to-end
orchestration
of
the
machine
learning
workflow
right.
They
provide
easy
experimentation,
you
know
giving
trials
and
experiments
and
you
know
you
can
manage
your
runs
and
group
them
into
experiments
and
it's
built
pipelines
are
built
with
components
and
those
components.
A
A
We
here
at
ibm
we
use
jackton
as
an
execution,
execution
engine
and,
of
course,
there's
a
user
interface
that
helps
with
you
know:
tracking
experiments,
starting
running
jobs.
You
know
checking-
and
you
know,
following
logs
as
I'll
show
later
and
an
sdk
for
data
scientists
who
actually
do
who
are
doing
the
machine
learning
training
and
the
sdk
helps
with
with
those
steps
and
the
sdk
is
especially
useful
when
data
scientists
want
to
work
with
their
notebooks,
which
are
typically
based
on
python
and
the
q4
pipelines.
A
Now
the
individual
steps
of
the
pipeline
they're,
the
pipelines,
are
built
out
of
components,
so
the
building
blocks
of
pipelines
are
components
and
a
component
is
really
a
self-contained
set
of
code
usually
performs
one
step
in
a
pipeline,
such
as
data
pre-processing
or
data
transformation,
model,
training,
et
cetera,
and
it
can
be
thought
of
like
a
like
a
function.
A
function
has
a
name
parameters,
return
values
and,
of
course,
you
know
function.
A
Now,
when
you
write
a
component,
you
have
various
options.
You
can
either
write
the
yaml
spec
directly
or
you
can
compile
the
component
from
python
dsl
in
that
component
spec.
There
will
be
metadata
name
and
description,
etc,
and
the
interface
will
be
described,
interface,
being
the
input
and
output
parameters.
A
A
A
A
Now,
there's
a
problem
with
the
existing
q4
pipeline
ecosystem
and
the
problem
is,
is
a
result
of
the
richness
of
the
exis.
If
the
number
of
components
that
exist
already
and
you
can
think
of
it.
As
you
know,
a
bunch
of
lego
blocks
like
in
the
in
the
picture
that
you
see
here
right,
it's
it's
an
unorganized
heap
of
blocks
and
what
we
really
want
is
we
want
an
organized
set
of
you
know,
building
blocks
that
can
be
used
to
build
pipelines.
A
A
As
I
mentioned
before,
there's
a
python
dsl
that
you
can
use
to
compile
your
components
or
you
can
write
the
components
back
in
yaml
directly
and
in
q4
pipelines.
V2
there's
also
the
intermediate
representation,
that
is
a
platform
independent
representation
of
a
pipeline
and
all
these
different
ways
of
authoring.
They
have
no
feature
parity
and
there's
not
a
good
way
to
document
and
publish
these
components,
similar
problems
for
hosting
right
now.
A
A
As
you
know,
things
might
be
important
for
for
a
certain
project
right
and
then
the
developers
they
move
on
project
evolves.
However,
often
these
components,
they
don't
get
maintained
very
well
now.
What
can
help
with
that
would
be
a
component
registry.
So
this
is
a
proposal
by
the
q4
pipelines
team
and
it's
currently
work
in
progress.
You
can
find
some
details
in
the
queue
for
pipelines
repository
in
issue
7382.
A
Now
that
new
protocol
is
meant
to
be
implemented
by
third
body
template
registry
servers
and
on
the
kfp
side
there
will
be
first
class
integration
into
the
sdk,
meaning
that
data
scientists
they
can
use
the
sdk
and
they
can
download
components
directly
from
a
registry,
discover
different
versions
and
you
know
have
a
simplified
way
to
search
and
find
components
they
they
need.
The
benefits
also
include
that
the
format
will
be
unified.
The
component
format
will
be
a
yama
format.
A
A
Here
on
the
left
side,
you
can
see
two
examples
of
what
the
rest
api
would
feel
like
how
you
would
upload
a
component,
given
that
you
have
the
proper
authorization
and
how
you
would
download
it,
and
the
download
really
really
is
just
the
the
host
and
then
the
name
of
the
package
and
the
version
or
the
tag
of
the
component
version
and
in
the
sdk.
It
would
look
as
simple,
as
you
know,
loading
a
component
from
a
registry.
A
So
what
is
the
machine
learning
exchange?
So
in
the
machine
learning
exchange
you
have
various
types
of
machine
learning,
related
assets,
the
pipelines
and
components,
of
course,
but
also
models.
Curated
data
sets
and
notebooks
that
can
be
used
to
train
models
using
those
data
sets
for
each
of
those
asset
types
or
assets.
A
A
A
Now,
here's
what
the
user
interface
would
look
like
here
are
eight
sample
pipelines.
You
can
filter
them
and
find
a
pipeline
that
that
you
want
to
look
at
in
detail.
You
can
see
the
pipeline
graph
details
with
description
in
the
yaml.
This
is
the
cave
protector.
Yammer
you
see
here
and
then
the
pipeline
can
be
launched
directly
from
the
mlx
ui.
You
can
provide
your
parameters
and
click
submit,
and
then
you
will
be
presented
with
the
q4
pipelines,
the
run
graph.
A
A
A
It
will
kick
off
the
run
of
the
notebook
using
papermill
in
the
kubernetes
part,
and
then,
of
course,
you
can
follow
the
logs
and
at
the
end
of
the
run
there
will
be
a
new
notebook
generated.
All
of
the
output
cells
will
be
populated
and
you
can
download
that
directly
and
of
course
you
know
the
notebook
does
whatever
it's
supposed
to.
In
this
case.
You
know
it's
training,
a
model,
here's
a
listing
of
the
the
pipelines,
components,
models,
data
sets
and
notebooks.
We
currently
have
in
the
machine
learning
exchange.
A
Now
some
of
the
technologies
we
integrated
in
mlx
is
datasham,
which
we
use
to
manage.
Our
data
sets
now
with
data
data
stream.
We
can
easily
create
persistent
volumes
right
and,
and
we
can
get
data
from
s3,
h3
or
nfs,
and
you
know
basically
reduce
the
amount
of
work
that
end
users
would
have
have
to
do
in
order
to
get
data
sets
and
work
with
them
in
communities
for
models.
A
We
have
an
integration
with
kf
serving
or
k-serve,
so
we
can
also
serve
our
models.
Natively
with
kubernetes
I
mentioned
that
mlx
is
built
on
pipelines,
specifically
google
pipelines
with
tecton
the
difference
being
the
execution
engine.
Yes,
q4
pipelines
are
by
default,
powered
by
argo,
and
we
know
we
have
techdon
as
the
engine
and,
of
course
it
runs
on
openshift
and
kubernetes.
A
A
I
did
a
port
forwarding
to
3000,
so
it
looks
like
it's
local,
but
it's
not
and
you
can
see
our
landing
page
and
you
can
browse
through
the
individual
assets.
Data
sets
models,
pipelines,
components
and
notebooks
and,
let's
start
with
data
sets,
and
one
of
the
the
most
recent
data
sets
we
integrated
is
the
project
coordinate.
A
A
Since
that
data
set
is
really
large,
we
cut
it
down
into
a
few
smaller
ones,
and
we
selected
one
for
a
language
classifier
that
I'm
going
to
show
in
more
detail.
So
once
you
go
into
an
asset
or
click
onto
an
asset,
you
get
a
more
detailed
overview
you
can
for
data
sets.
You
can
download
the
original
source
files
you
can
browse
in
the
license.
A
A
We
have
yammer
specs
for
data,
sets
models
and
notebooks,
and
you
know
they
have
information
like
the
the
name
and
the
description
version
where
it's
from
you
know
what
the
license
is,
and
you
know
other
related
information
like
here.
You
can
see
that
there
is
a
notebook.
We
have
integrated
that
is
related
to
this
data
set.
A
Project
coordinate,
we
have
a
language
classification,
notebook
and
again
you
can
see
some
details
here.
The
yama
file
that
describes
where
you
can
see
find
the
source
of
the
notebook
and
what
are
the
python
requirements
here
and
what
image
does
this
run
on
and
you
can
also
preview
the
notebook
code
and
we
use
and
the
nbv
or
component
for
that-
and
this
comes
handy,
especially
if
notebooks
change
over
time.
You
get
newer
versions
and
you
want
to
you
know
just
reassure
you
what
version
you're
looking
at
you
can
see
that.
A
So
we
can
launch
this
we're
gonna
put
in
the
id
of
the
pvc
transition
volume
we
created
and
the
mouth
path
and
click
submit,
and
then
the
alarm
notebook
component
should
start.
It
should
download
the
notebook
from
sv
storage
and
use
papermill,
download
the
the
image
and
use
papermill
to
start
the
run.
A
A
A
I
downloaded
that
earlier
and
just
show
looks
very
similar
to
the
notebook
we
just
previewed
go
through
all
the
cells
and
at
the
end,
in
this
particular
notebook.
We
take
our
trained
model
and
use
a
small
test
set
to
do
some
predictions,
and
since
this
is
a
a
language
classification,
we
give
it
10
samples
in
10
languages,
c
c,
sharp
c,
plus,
plus
the
ask
of
javascript
and
so
on,
and
here
we
can
see
that
for
most
of
them
the
predictions
are
correct,
except
for
c
plus.
A
We
only
have
nine
out
of
ten
correct
predictions,
and
if
you
want
to
compare
that
to
the
initial
run,
when
the
notebook
was
first
uploaded,
we
can
go
to
the
original
notebook
in
the
notebook
source
code,
viewer
and
scroll
to
the
bottom
and
see
in
the
original
upload.
Two
of
the
languages
you
know
were
not
quite
correct.
In
our
latest
run.
We
classified
all
of
our
c
code
terms
correctly,
so
we
improved
that
was
the
quick
demo
of
the
machine
learning
exchange.
A
Here
you
can,
you
know,
see
all
of
the
all
of
the
assets
I
showed
showed
you
just
during
the
demo,
except
in
that
read-only
deployment.
We
don't
allow
users
all
over
the
world
to
to
kick
off
training
jobs
right,
so
q4
pipelines
is
not
integrated.
In
this
we
donate
the
product
on
our
github
repository.
You
will
also
find
information
to
just
general
information
how
to
use
machine
learning,
exchange,
find
more
information
about
all
the
integrated
technologies
and
you
will
find
deployment
options
and
we
have
a
quick
start.
A
A
You
know
it
can
list
your
packages.
You
can
upload
new
packages,
delete
packages
versions,
and
here
you
can
see
one
of
those
endpoints
and
that
is
to
list
all
packages.
You
can
see
all
the
components
in
this
case
that
we
have
added
and
for
each
of
the
components
you
can
also
see
the
individual
versions
and
the
tags.
So
we
go
here
and,
for
example,
we
have
that
echo
sample
component
and
you
can
see
all
of
the
individual
versions
that
we
have
for
this.
A
A
A
You
have
the
links
to
the
machine
learning
exchange,
they
get
a
repository
in
the
website
and
you
can
reach
out
to
us,
via
slack
and
for
the
center
for
open
source
data
and
our
technologies.
You
can
visit
our
website
or
find
our
articles
on
medium.com
today
that,
if
you're
interested
there
are,
we
have
four
sessions
from
credit
at
the
kubernetes
kucon
europe
co-located
events
at
the
ai
day
edge
day
and
canadicon
and
yeah
feel
free
to
check
them.
Out
with
that,
thank
you
and
see
you
next
time.