►
From YouTube: Sponsored Keynote: Kubeflow Pipelines v2: The Next Generation of MLOps on Ku... Karthik Ramachandran
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sponsored Keynote: Kubeflow Pipelines v2: The Next Generation of MLOps on Kubernetes - Karthik Ramachandran, Product Manager, Google Cloud Vertex AI, Google
We would like to introduce Kubeflow Pipelines v2, which makes it significantly easier to author and manage model training Pipelines. In this talk, we’ll review the new capabilities of the project, talk a little bit about our plans for the future, and discuss where and how the community can help.
A
before
we
get
into
what
qfloor
pipelines
is.
Let
me
start
with
what
I
mean
by
a
pipeline,
because
the
term
pipeline
can
be
a
bit
overloaded
here,
we're
talking
about
machine
learning
pipelines,
because
machine
learning
models
require
complex,
multi-step
workflows
when
building
a
model.
For
example,
you
may
have
to
clean
and
transform
data,
create
features,
train
multiple
models
and
evaluate
those
models.
A
A
Q4
pipeline
is
an
open
source,
kubernetes
native
framework
for
building
and
deploying
scalable
repeatable
machine
learning
workflows.
It
allows
you
to
model
ml,
workflows
and
series
of
container
executions
where
each
container
can
take
input
from
a
previous
execution
and
produce
outputs
that
can
be
used
in
subsequent
steps.
A
A
A
For
example,
google
cloud
provides
components
that
allow
you
to
take
advantage
of
gcp's
data
flow
or
vertex
ai
ai's
distributed
training
service
q4
pipeline
takes
care
of
the
work
associated
with
scheduling
each
step.
The
passing
information
between
steps
and
captures
metadata
about
the
artifacts
produced
and
consumed
for
each
step
in
the
pipeline,
just
to
name
a
few
of
the
capabilities
of
the
system.
A
A
The
first
major
area
of
improvement
is
metadata.
Metrics
right
now,
qfl
pipeline
automatically
tracks
the
artifacts
that
are
producing
complaints
by
each
step
in
the
pipeline
in
v2.
We're
going
to
add
the
ability
for
specify
their
custom
metadata
for
artifacts,
enabling
users
to
provide
rich
queryable
descriptions
for
the
artifacts.
A
A
This
includes
enhancements
to
the
sdk,
to
make
it
easier
to
attach
custom
metadata
to
artifacts
the
ability
to
query
and
visualize
artifact
lineage
and
a
set
of
well-defined
integration
points
for
those
building
tooling.
On
top
of
kfp,
for
example,
you
may
want
to
build
tooling
that
enables
you
to
track
which
data
sets
you
use
to
produce
which
models
or
you
may
want
to
be
able
to
track
a
single
record
all
the
way
from
training
through
to
which
model
to
where
it
was
deployed.
A
A
A
We
have
had
a
number
of
requests
over
the
years
from
our
community
to
produce
an
intermediate
representation
of
machine
learning
pipelines
which
is
both
dsl
and
execution
engine
agnostic.
The
goal
here
is
to
make
it
easy
for
users
to
build
new
tools
for
authoring
pipelines
and
enable
kfb
pipeline
to
run
on
alternative
orchestrators
and
back-ends.
A
For
example,
one
of
the
back
ends
that
will
support
the
new
intermediate
representation
is
gcp's
vertex
pipeline
a
fully
managed
ml
orchestration
system.
That's
capable
of
running
asp
apartments
today
with
vertex
and
the
intermediate
representation
you'll
be
able
to
write
a
pipeline
or
component
in
the
open
source,
gives
the
implementation
and
then
run
it
on
gcp's
managed
service
with
little
or
no
change.
A
So
my
final
minute,
let
me
just
just
a
few
next
steps.
First,
you
can
learn
more
about
q4
pipelines
by
visiting
the
q4
pipeline
site.
We
also
have
an
initial
version
of
q4
pipelines,
v2
out
with
the
goal
of
moving
towards
the
final
version
in
late
october
or
early
november.
You
can
try
that
out
and
look
at
the
documentation
at
the
bitly
link
on
this
site,
and
you
can
of
course
go
to
our
github
repository
where
you'll
find
tons
of
opportunities
to
contribute
and
join
the
community.