►
From YouTube: Towards Continuous Computer Vision Model Improvement with Kubeflow - Derek Hao Hu & Yanjia Li
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Towards Continuous Computer Vision Model Improvement with Kubeflow - Derek Hao Hu & Yanjia Li, Snap Inc.
With deep learning gaining popularity in industry, there is a lot of material focusing on model training and serving. However, in production machine learning typically isn't complete after a single round of training. Model owners need to find ways to improve trained models regularly, and good machine learning pipelines achieve this by leveraging continuous feedback. In this talk, we will demonstrate how Kubeflow and Kubeflow Pipelines are being used to continuously improve computer vision models at Snapchat. We will walkthrough how we orchestrate multiple components with Kubeflow Pipelines to extract data, label images, and (re)train machine learning models. We will also discuss best practices for authoring Kubeflow Pipeline components based on our experiences from developing and deploying these components for production use.
https://sched.co/Uae4
A
Good
afternoon,
welcome
to
my
talk,
so
the
title
of
our
talk
today
is
towards
continuous
computer
vision
model
improvement
with
good
flow.
My
name
is
Derek
and
I'm.
A
software
engineer
at
snapchat.
A
lot
of
the
material
presented
in
English
talk
today
is
actually
done
by
my
coworker
Eason,
and
he
unfortunate
can't
be
here
due
to
visa
issues,
so
I'll
be
the
only
one
giving
the
talk
so
first,
a
very
quick
overview
about
the
agenda
of
this
talk
and
what
this
talk
is
about.
A
So
first
I'll
give
you
a
very
brief
introduction
about
what
snapchat
is,
but
it
seems
a
lot
of
you
are
aware
of
us
already
and
that's
good,
so
I'll
keep
that
part
concise.
Next
I'll
talk
a
bit
more
about
this
new
feature.
Our
team
has
been
working
on
called
scan.
Scan
basically
leads
us
to
the
problem
and
need
of
like
say
continuously
updating
and
improving
computer
vision
models
and,
more
particularly,
we
are
interested
in
improving
the
quality
of
visual
understanding.
So
this
is
an
important
part.
A
You
like
to
understand
why
we
even
need
to
have
this
talk.
Next
I'll
give
a
high-level
overview
of
what
we
mean
by
model
in
provement
and
what
kind
of
machine
learning
problem
we're
trying
to
tackle
here
and
what
kind
of
high
level
machine
learning
approach
we
are
trying
to
have
as
well.
So
after
that
we'll
do
a
case
study
and
we'll
discuss
why
we
choose
to
use
coop
flow
pipelines
and
what
kind
of
alternatives
we've
considered
and
we'll
also
describes
the
pipelines
we
have
built.
A
A
So
basically,
we
consider
ourselves
to
be
the
most
fun
way
to
share
the
moment
whenever
you
wanted
to
to
do
so
and
at
the
end
of
q3,
we
have
over
210
million
daily
active
users,
so
we
have
more
than
3.5
billion
snaps
created
every
day
and
shared
with
your
friends,
and
we
also
have
over
600,000
lenses
created
by
our
community.
I.
Do
want
to
explain
these
two
words
a
bit
more.
A
So
the
first
here
is
what
is
a
lens,
so,
as
you
can
see
here,
on
the
right
hand,
side
a
lens
is
basically
an
overlay
that
you
can
add
to
your
snap
that
creates
an
augmented
reality
experience
on
top
of
your
content.
So
basically
a
lens.
This
is
meant
to
be
fun.
Create
event
makes
your
snap
more
engaging,
and
the
lens
we
are
showing
here
is
one
of
the
most
popular
lens
with
built
like
four
years
ago,
called
puking,
rainbows
and
then
on
the
turn
community.
So
basically
we
have
this
app
called
lens
studio.
A
A
So
it's
an
overlay
on
top
of
her
face
the
second
one
at
a
stop
sign
based
on
the
movement
of
the
hand
in
the
third
snap,
we
can
see
this
cute
cat
now,
shooting
laser
beams
from
her
eyes
and
the
last
one
is
an
animated
3d
object.
That
I
believe
is
a
butter,
yeah
I
guess
so
so
like
if
you're
interested
to
create
more
lens
like
this,
you
can
always
learn
more
at
lens
studio,
dodge
snapchat
comm.
A
But,
as
I
just
said,
we
have
over
600,000
tons
of
lenses
like
this,
which
is
both
good
and
bad.
Good
thing
is
our
users
now
have
a
lot
to
choose
from,
and
the
bad
thing
is
they
have
a
lot
to
choose
from,
because,
when
you're
taking
a
snap,
it's
very
hard
for
you
to
know
like
what's
the
most
perfect
lens
or
like,
what's
the
most
engaging
lens,
you
can
possibly
use
for
the
content
that
you
want
to
share
like
if
you
have
a
cat
or
you're.
A
If
you
don't
take
a
snap
for
your
favorite
cat
and
you
don't
even
know
we
have
all
these
cute
pet
lenses
at
all.
What
can
what
can
you
do
so,
our
solution
to
this
problem
or
okay,
more
precisely
our
attempt
to
solve
this
problem
leads
us
to
this
new
feature
called
scan
so
here
for
scan.
We
aim
to
find
the
perfect
lens
for
the
content
of
your
snap
overall.
A
A
We
was
into
personalized
sorting
for
all
the
lens
you
have
and
then
give
you
a
list
of
suggested
then
tailor
to
the
content
of
your
snap.
So
as
a
result,
if
you
take
your
phone
out
open,
snapchat,
app
and
press
and
hold
on
this
cat,
which
we've
just
seen
from
the
previous,
you
should
be
able
to
see
a
list
of
dance
like
that.
Like
this.
A
Yeah,
so
now,
if
you
don't
know
what
you
can
do,
when
you're
taking
a
snap
or
what
kind
of
AR
effect
you
can
add,
now
we
have
a
way,
and
then
you
mean
you
may
want
to
stop
me
right
now,
because
you
feel
it's
already
2019
I
can't
be
talking
here
and
acting
like
this
is
still
a
hard
problem
right.
I
I
mean
you
might
say,
I
shouldn't
be
thinking.
A
That
seems
to
be
what
most
of
the
tutorials
of
computer
vision
models
seem
to
be
telling
us
I
mean
this
will
give
you
a
good
baseline
to
start.
So
what
do
we
actually
mean
by
continuously
improving-
and
this
leads
us
to
the
second
portion
of
this
talk,
which
is
a
high
level
overview
of
what
model
improvement
things
and
what
we
are
trying
to
do?
A
What
we've
learned
from
our
basic
machine
learning
courses
or
machine
learning
101
is
your
model
would
be
as
good
as
your
training
set,
and
your
model
is
only
as
good
as
your
training
set.
What
this
means
is,
if
your
training
set
sucks,
no
matter
what
kind
of
model
you
try,
your
model
would
likely
suck.
So
this
is
an
image
we
get
from
our
internal
user
feedback.
A
Basically,
what
happens
is
after
we
finish
the
first
release.
Candidate
of
this
scan
object,
classification
model,
our
internal
user,
hey
I'm
in
the
furniture
store
and
I
see
a
bunch
of
Catalans
related
to
this,
and
I
can't
apply
my
Catalans
to
any
of
them.
So
can
you
fix
that?
So
we
took
a
look,
and
apparently
we
found
this
images
now
incorrectly
classified
as
cat,
and
then
we
are
suggesting
pet
lenses
that
users
can't
apply.
So
this
is
the
bad
experience
that
we
want.
We
want
to
fix
for
sure.
A
You
know.
You
might
argue
that
this
is
incorrect,
because
our
training
process
probably
has
a
bug.
Our
network
structure
might
be
wrong
or
like
or
like
these
sorts
of
things
I'm
concerned
about
that
too.
So
months
ago,
when
I
first
saw
this
I
did
a
few
tests.
I
actually
did
this
test
last
night
and
if
you
download
this
image
or
crop
it
upload
it
to
some
kind
of
cloud
vision,
API
or
some
visual
understanding,
API
that
you
can
find
online
from
very
major
companies.
A
You'll
find
that
at
least
two
or
three
of
them
now
still
classify
it
as
cat
or
animal.
So
this
means
maybe
it's
not
because
our
training
process
has
a
bug
and
I
would
argue,
it's
basically
the
bit
it's
basically
because
in
our
training
set,
we've
never
seen
these
sorts
of
images,
which
I
believe
is
a
fluffy
pillow
of
some
sort
to
illustrate
what
what
this
means.
Let's
take
one
of
the
largest
image
data
set
that
we
now
have
open
images
as
an
example.
A
So
what
we
have
here
is
interesting,
because
right
now,
I
can't
even
see
that
on
my
screen,
nice.
So
what
we
have
here
is
this
is
the
open
images
data
sets
released
by
Google.
It
has
over
9
million
images,
and
here
are
the
images
that
are
tagged
as
positive
like
contain
pillows.
So
here,
if
we
scroll
down,
if
we
take
a
quick
glance,
we
can
see
there
are
various.
A
If
anyone
guys
can
tell
me
where
my
cursor
is
that'll
be
great
okay.
So
if,
if
we
scroll
down,
we
can
see
if
there
are
various
kinds
and
shapes
and
textures
of
pillows
from
this
image
set.
But
it's
not
easy
for
us
to
see
the
same
fluffy
or
like
furry
kind,
which
basically
means
we
can't.
We
won't
be
able
to
tell
this,
because
we've
never
seen
that
okay.
So
after
this,
the
very
natural
question
is:
how
do
we
fix
that
or
how
do
we
actually
like?
A
How
do
we
effectively
leverage
limited
user
feedback
to
effectively
improve
our
classification
model
and
almost
always
when
it
comes
to
model
in
provement?
One
of
the
most
effective
approach
is
to
get
more
training
data
or
get
more
labeled
instances,
but
here's
the
main
issue
is
what
kind
of
training
data
do
we
actually
need
that
can
correct
predictions
like
this,
because
we
only
have
like
one
or
two
or
like
not
more
than
a
handful
of
user
feedbacks.
We
can't
train
on
those.
A
The
overall
idea
we
have
is
conceptually
very
simple
and
straightforward
and
also
a
very
standard
machine
learning
approach.
Basically,
what
we
want
to
do
is
to
rely
on
similar
images
that
we
can
find
on
snapchat
and
more
more
precisely.
What
we
want
to
do
is
to
find
images
that
come
from
a
similar
distribution
as
our
scan
requests
that
look
similar
to
the
user
feedback
images
where
our
predictions
are
incorrect,
so
in
snapchat,
one
of
the
potential
data
source
we
can
use
is
called
our
story.
A
So
this
is
feedback
and
I
haven't
considered
I
can't
even
I,
don't
know
where
my
mouse
is
okay,
so
here
I
want
to
fetch
the
top
100
most
similar,
our
our
story,
posts,
I,
click
this
and
then,
after
an
embarrassing,
embarrassingly
long
wait.
So
we've
found
a
bunch
of
images
that
look
like
furniture
stores,
because
it's
very
likely
this
image
is
taken
in
a
furniture
store.
A
So
if
we
scroll
down,
we
can
see
that
most
of
the
post
we
have
here
do
come
from
the
same
type
of
places,
but
it
may
not
be
exactly
helpful
for
us
to
correct
this
case.
Luckily,
for
machine
learning
based
on
the
activation
maps,
we
can
kind
of
understand
which
parts
contribute
to
the
highest
activation
based
on
the
prediction
values
we
have.
A
So
if
we
like
crop
the
highest
night
like
the
best
region
of
interest
we
have
we
do
we
do
a
we
query
again,
based
on
this
portion,
we
now
should
be
able
to
find
more
fluffy
stuff.
That
looks
might
be
helpful
because,
like
you
can
see
here,
some
of
these
are
actually
animals.
Some
of
these
are
not,
but
so
this
kind
of
positive
and
negative
combinations
would
help
our
neural
network
learn,
which
are
the
most
important
features
to
try
to
tell
these
part
of
things
like
from
the
cat
noir
cat
type.
A
You
may
argue
this
is
not
that
important
in
terms
of
fixing
user
feedback,
but
people
do
love
cats
so
and
after
this
the
next
slide
is
a
backup
slide
in
case
demo
failed
and
luckily
I.
Don't
need
to
use
that
so,
and
next
slide
is
another
backup
start.
Ok,
so
here
is
a
high-level
overview
of
what
our
end-to-end
workflow
is.
So
basically,
we
have
two
sorts
of
input.
One
is
a
search
index
that
that
is
that
should
support
nearest
neighbor
queries
and
this
search
index
is
basically
built
from
our
story
posts.
A
We
would
compute
image
features
on
top
of
that
and
then
it's
used
to
support
querying
visually
similar
images.
The
other
sort
of
input
are
a
collection
of
seed
images,
so
these
seed
images
they
may
come
from
user
feedback.
Let
me
come
from
scan
request
image
logs,
where
the
model
is
uncertain
about
or
other
types
of
active
learning
strategies.
A
So
these
seed
images
are
then
used
as
queries
against
the
image
index
we
have
here
and
then
it
will
give
you
a
list
of
candidates,
so
these
candidates
are
then
fed
into
some
third
party
annotation
services,
and
here
we
use
stage
maker
ground
choose.
So
we
then
splits
this
image
into
training
set
and
evaluation
set.
We
will
combine
this
new
training
set
with
existing
training
images.
We
have,
we
train
the
whole
classifier
and
see
if
it
meets
some
performance
criteria
for
like
updating
and
then
we
will
deploy
again.
A
So
this
is
the
high-level
overview,
and
although
this
is
still
a
very
standard
machine
learning
workflow
it's
way
beyond
what
a.
What
a
machine
learning
engineer
can
comfortably
experiment
in
his
Jupiter
notebook.
So
we
need
some
other
better
tools
and
then
in
the
next
portion,
I
will
chat
a
bit
more
about
why
we
choose
to
use
coop
flow
pipelines.
What
kind
of
comparisons
we've
made
to
other
pipeline
orchestrators
and
an
overview
of
the
coop
flow
pipelines
we've
implemented
for
the
machine
learning
workflow
that
we
just
see
so,
as
we
all
know
so,
basically
cupola
pipelines.
A
It
essentially
is
a
pipeline
Orchestrator,
and
there
are
already
a
lot
of
pipeline
orchestrators
one
could
possibly
use.
So
why
could
flow?
We
listed
several
requirements
that
we
would
hope
our
pipeline
Orchestrator
would
have
before
we
dive
into
and
start
implementing
everything
on
this
pipeline
Orchestrator.
A
So
the
first
thing
is:
we
would
want
it
to
be
able
to
run
arbitrary
or
containerized
code,
because
this
would
give
us
the
maximum
flexibility
in
terms
of
component
language
choices,
packages,
etc.
It
should
be
able
to
share
components
easily
across
different
workflows,
because
then
multiple
engineers
or
teams
can
build
these
shared
components
where
people
can
then
provide
common
functionalities
that
shared
across
multiple
experiments.
A
It
should
be
able
to
reuse
and
modify
previous
workflows,
because
then
we
can
enable
quick
ml
experiment
iteration,
because
we
are
trying
all
sorts
of
new
ideas
every
day
that
may
or
may
not
work.
So
it's
it
doesn't
make
sense
to
simply
assume
that
most
of
our
pipeline
specifications
would
remain
fixed
most
of
the
time.
Ideally,
we
would
want
it
to
be
current,
that
is
native
as
well,
because
in
that
case
we
can
minimize
our
gap
between
the
experimentation,
computing
environment
and
our
production
environment.
A
A
Ideally,
we
would
want
to
avoid
vendor,
like
it
shouldn't,
be
completely
dependent
on
a
specific
vendor
as
well,
and
we
also
wanted
to
be
able
to
specify
workflow
as
code,
so
these
pipeline
changes
are
meaningful
and
other
engineers
can
review
these
changes
as
well.
And
finally,
it
would
be
nice
if
we
can
have
some
nice
metric
visualization
output,
either
natively
or
by
integration
with
other
tools
like
10
support.
That's
a
long
list
so
late
last
year,
while
we
were
trying
to
decide
which
pipeline
Orchestrator
to
use
eventually
we've
narrowed
down
to
these
three
choices.
A
Some
of
these
comments,
I'm
making
here,
may
no
longer
be
valid
because
it's
being
a
year
and
a
lot
of
things
has
happened
in
the
past
year,
so
I'm
afraid.
Maybe
airflow
now
supports
this
now,
but
to
be
honest
and
not
entirely
aware,
so
the
first
choice
we've
considered
is
AWS
step
function,
so
it's
widely
used
in
a
in
AWS
land.
It
has
very
tight
in
integration
with
almost
all
the
AWS
native
support
and
other
cool
it
obvious
stuff
it
has,
but
we
feel
it's
a
good
workflow
Orchestrator,
but
it's
not
a
very
good
work.
A
The
orchestrator
for
machine
learning,
experimentation,
use,
it's
not
kubernetes
native
for
sure,
and
the
way
we
we
specify
workflows
in
AWS
step
function
is
basically
these
state
machine
json
strings.
So
it's
not
entirely.
It's
not
easily
understandable
and
there
is
no
way
for
us
to
quickly
see
what
kind
of
metric
output
each
one
has
and
airflow
is
widely
used
by
a
lot
of
companies
for
sure
for
their
data
pipelines,
and
there
are
a
lot
of
nice
things.
I
love
about
airflow,
but
airflow
at
that
time.
A
One
of
the
main
thing
is
it
wasn't
able
to
integrate
with
kubernetes
easily
and
as
such,
we
won't
be
able
to
use
GPUs,
which
means
if
we
were
to
use
airflow.
We
need
to
rewrite
a
lot
of
our
code
into
cloud
platform,
machine
learning,
training
engines
like
sage
maker
or
Google
Cloud
ml
engine,
which
is
like
one
thing
we
are
trying
to
avoid.
A
It
doesn't
have
visualization
support
as
well,
and
one
of
the
main
thing
I
think
is:
airflow
is
good
for
recurring
tasks,
but
it's
not
super
easy
for
you
to
change
existing
tags
or
create
new
experimental
tags.
So
it's
good
for
recurring
tasks,
but
may
not
be
that
good
for
a
quick
experimentation
and
obviously,
at
that
time,
when
we
were
looking
to
cupola
pipelines,
it
supports
all
our
needs
and
we
went
all
in
to
implement
our
machine
learning
workflows
in
coop
flow
pipelines.
A
So
here's
an
overview
I
hope
the
font
is
still
recognizable.
So
this
is
an
overview
of
the
pipelines
we've
built
from
the
previous
model
improvement
machine
learning
workflow.
So
basically
we
split
the
overall,
like
giant
workflow
into
form
mini
pipelines.
The
reason
behind
this
is,
we
still
don't
have
artifact
caching
include
flow
pipelines
yet
so
once
we
make
any
tweaks
or
updates
in
the
downstream
components,
we
don't
want
to
rerun
everything,
especially
considering
some
of
these
pipelines
actually
contain,
like
involve
human
annotation,
which
might
be
pricey,
so
the
pipeline
implementation
part
is
actually
relatively
straightforward.
A
Given
everything
is
already
talked
arised
and
basically,
we
now
have
four
parts.
The
first
part
is
a
candidate
image
collection
pipeline.
Where
we
said
it,
we
can
have
multiple
sources.
We
can
fetch
visually
similar
images
from
our
story,
or
we
have
fetched
from
our
scan
request
logs,
as
a
second
pipeline
is
where
we
feed
data
to
save
stage
maker
ground
rules
for
annotation.
A
One
thing
to
note
here
is
we
do
cross
cloud
data
transfer
due
to
pricing
or
other
feature
considerations,
so
we
inject
AWS
ints
secrets
in
our
clusters,
so
data
transfer
can
be
done
easily,
so
the
second
pipeline
basically
does
some
customized
pre-processing
like
we
need
to
blur
the
face
of
people
from
the
image
post
before
sending
them
for
annotation,
and
then
we
would
just
pass
these
processed
images
down
to
stage
maker.
So
so
the
third
pipeline
basically
creates
the
charted
data.
A
We
need
for
training
and
evaluation,
so
here
we
will
pass
the
annotation
data
back
to
BQ
and
create
Charlotte
TF
record
files
using
data
flow.
So
here
what
we've
done
is
we've
created
a
an
additional
wrapper
around
the
official
kfp
component
for
starting
dataflow
jobs,
to
use
Python
3
and
also
to
fetch
the
output
of
dataflow
drops
and
then
pass
them
downstream
easily,
and
the
final
portion
of
the
pipeline
is
simply
a
wrapper.
That
starts
a
stage
maker
training
job
and
handles
customized
evaluation
and
then
tries
to
see
if
this
new
classifier
can
be
updated.
A
So,
for
the
final
part
of
the
talk,
I'll
share
some
of
the
pain
points
we've
seen
while
using
loop
flow
pipelines.
They
are
also
some
of
the
practice
or
hacks.
We've
come
up
with
and
I
personally
understand.
A
lot
of
these
ideas
are
far
from
ideal,
so
I
also
want
to
hear
feedback
or
what
other
engineers
do
for
to
deal
with
with
issues
like
this.
A
So
in
our
original
talk
proposal,
the
words
I've
used
were,
we
will
also
discuss
best
practices
for
include
flow
pipeline
components
based
on
our
experiences
from
developing
and
deploying
these
components
for
production
use.
I
would
say:
that's
not
really
the
case,
because
when
we
first
adopted
coop
flow
in
last
year,
like
there
are
a
bunch
of
issues,
we
are
seeing
almost
every
day
and
we've
implemented
a
bunch
of
hacks
around
this
and
at
that
time,
I
personally,
or
we
personally
consider
them
as
best
practices
after
six
months
or
after,
like
12
months.
A
A
Yet
so
we
have
proposed
some
hacks
and
I
want
to
chat
a
bit
more
about
some
and
to
share
why
we're
doing
things
in
that
way,
the
first
thing
is
artifact
caching,
and
as
I've
mentioned
before,
we
are
basically
splitting
our
entire
machine
learning
workflow
into
several
mini
pipelines,
because
we
don't
have
artifact
caching,
the
motivation
behind
artifact
caching
is
very
straightforward,
because
if
the
upstream
input
hasn't
changed,
if
the,
if
the
upstream
container
image
hasn't
changed,
arguments
hasn't
changed.
We
shouldn't
need
to
rerun
all
the
things
and
hear
a
very
natural
assumption
is.
A
Most
containers
should
give
you
deterministic
output
and
even
if
we're
operating
under
that
assumption,
but
your
container
gives
you
non
deterministic
output.
We
can
still
always
add
a
placeholder
arguments
that
changes
in
each
one.
So,
ideally,
we
think
we
should
need
support
from
our
goal
or
kfe
team
to
handle
this.
But
before
we
have
that
the
hack
where
we
are
proposing
right
now
is
very
simple:
it's
basically
a
Python
decorators
that
computes
a
UUID
output
for
G,
yes
pass,
given
the
input
container
arcs,
as
well
as
your
container
image
tax.
A
So,
basically,
at
one
time,
what's
this
decorating
function
does
is
it
checks
whether
the
cache
file
already
exists
and
the
writing
operation
has
already
done
and
if
yes,
it'll
it'll
skip
all
the
work
and
it'll
simply
write
KP
component
outputs
based
on
what
is
already
stored
in
in
the
cache,
as
I
said,
what
we
need
as
input
here
would
be
the
image
tag
and
container
arcs.
So
we
have
container
arcs
easily.
We
don't
have
image
tags,
but
we
do
need
them.
A
Otherwise,
if
we
update
the
image
itself,
but
we
don't
update
input,
we
do
want
to
rerun
the
whole
thing.
Luckily,
image
tag
is
actually
stored
as
part
annotation
thing
are
those,
so
one
thing
we
do
is
we
simply
use
field
reference
and
pass
this
metadata,
annotation
down
as
environment
variables
to
the
kfp
component
pods,
and
that's
basically
how
we
get
image
ID
and
then
do
all
kind
of
hash
to
see
if
we
should
escape
the
computation.
A
A
So,
overall,
we
feel
it
makes
more
sense
for
these
kind
of
artifact
caching
change
to
happen
in
KP
backend
for
sure,
but
if
you're
eager
to
implement
a
hack
as
a
workaround
I
feel
this
is
the
approach
that
could
possibly
work
for
you
if
you're
dealing
with
standard
tasks,
the
other
thing
I
want
to
share
a
bit
more
about
is
how
do
we
add
type
constraints
between
different
kfp
components?
Basically,
the
issue
here
is,
after
engineers
have
have
contributed
to
like
tens
or
hundreds
of
KP
components
into
your
component
Spore.
A
A
Basically,
they
have
no
information
about
what
kind
of
types
they
are
expecting,
Wow
it
while
implementing
their
code-
and
this
is
actually
okay,
if,
if
you're
dealing
with
small-scale
pipelines,
but
once
your
pipeline
grows
large,
it
becomes
unmanageable,
and
our
current
workaround
to
this
problem
is
also
simple
as
well,
so
we
basically
define
a
shared
kfp
schema
package
that
centers
around
each
experiment
and
then
all
components
used
in
that
experiment
should
import
the
same.
Kfp
schema.
A
A
In
the
past
few
months,
I
think
one
of
the
new
features
introduced
by
coop
flow
team
is
a
new
could
flow
metadata
feature,
and
we
strongly
believe
that
the
information
about
these
meta
data
should
be
tracked
in
other
places
as
well,
which
beings
which
we
shouldn't
be
tracking.
These
schema
information
in
kfp,
so
in
the
future,
what
will?
What
will
also
try
to
do
is
to
explore
this
route
and
see
if
this
will
become
our
holy
grail
to
type
management
and
artifact
tracking?