►
From YouTube: MetaOps: Metadata Operations For End-To-End Data & Machine Learning Platforms - Alejandro Saucedo
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
MetaOps: Metadata Operations For End-To-End Data & Machine Learning Platforms - Alejandro Saucedo, The Institute for Ethical AI & Machine Learning
Speakers: Alejandro Saucedo
Organisations have been growingly adopting and integrating a non-trivial number of different frameworks at each stage of their machine learning lifecycle. Although this has helped reduce time-to-value for real-world AI use-cases, it has come at a cost of complexity and interoperability bottlenecks. Each stage in the end-to-end lifecycle involves different stakeholders that make decisions and perform actions that can modify data and/or ML components with use-case-specific but ever compoinding risks, resulting in a growing need to ensure a minimum-level of metadata is collected, tracked and managed. This becomes growingly important due to the need to ensure relevant overarching compliance requirements, as well as architectural requirements on lineage, auditability, accountability and reproducibility. In this session we will dive into the challenges present in the metadata layer of large-scale systems, as well as tooling, best practices and solutions that can be adopted to tackle these challenges. We will discuss the rise of the metadata management systems, the challenges they have been able solve, as well as critical shortcomings where ecosystem-wide collaboration will be key from tooling-level alignemnt to ensure long-term robustness of these heterogeneous end-to-end platform.
B
That
is
going
to
have
a
lot
of
call
to
actions
for
the
community
that
you
know
we'll
have
to
get
together
collaborate
in
order
to
make
this
a
success.
So
today's
talk
is
metadata
operations
for
end-to-end
data
and
machine
learning
platforms.
The
objective
of
this
talk
is
to
give
an
intuition
primarily
on
the
challenges
that
are
being
faced
when
it
comes
to
managing
machine
learning
systems
at
scale
and
specifically
how
the
metadata
within
the
systems
differs
in
in
regards
to
when
it's
com
compared
to
the
more
traditional
Data
Systems.
B
So,
let's
dive
straight
into
it
a
little
bit
about
myself.
My
name
is
Alejandro
I'm
engineering
director
at
Selden,
Technologies
we're
a
company
that
focuses
on
deployments
and
monitoring
of
machine
learning
models
at
scale
and
we're
the
authors
of
one
of
the
most
popular
machine
learning
deployment
Frameworks
in
kubernetes
called
seldom
core
I'm,
also,
a
chief
scientist
at
The
Institute
for
ethical
AI
research,
centered
based
in
the
UK
that
focuses
on
developing
Frameworks
for
the
responsible
design,
operation
of
machine
learning
systems
and
I'm,
also
a
governing
council
member
at
large
at
the
ACM.
B
So
today,
what
are
we
going
to
be
covering
we're
going
to
be
giving
some
intuition
on
the
motivations?
Why
should
we
care
and
challenges
of?
How
is
it?
How
is
it
different
to
traditional
metadata
we're
going
to
talk
about
some
of
those
differences
being
the
relationships
that
we're
dealing
with
the
entities
in
the
mlops
world?
We're
going
to
talk
about
some
ways
in
which
we're
looking
to
tackle
these
challenges
through
what
we
can
refer
to
as
the
open
inference
protocol,
as
well
as
the
open
inference
schema?
B
Finally,
we're
going
to
have
a
call
to
action
for
basically
things
that
we
can
do
as
a
community
to
continue
driving
this
forward,
so,
let's
dive
straight
into
it
so
just
to
set
the
scene,
you
know
just
a
picture
early.
You
know
2010s
how
it
all
started
a
handful
of
Frameworks
in
the
mlops
ecosystem
in
the
ml
ecosystem
that
you
could
pick
and
choose
to
be
able
to
even
productionize
your
machine
learning
models
how
it's
going
today.
B
You
know
there
is
a
ever
growing
number
of
envelopes
tools
that
are
appearing
every
single
day
that
are
tackling
pretty
much
very
similar
challenges
across
very
analogous
areas
in
the
mlops
life
cycle
and
envelope
stack.
So
one
of
the
things
that
we're
seeing-
and
you
know
one
of
the
the
topics
of
another
session
that
we
had
earlier
this
week
in
the
kubernetes
AI
day-
was
that
we're
seeing
a
convergence
into
what
is
not
like
the
concept
of
the
canonical
stack
but
a
set
of
canonical
Stacks.
B
So
an
abstraction
of
you
know
what
are
the
end-to-end
life
cycle,
blueprint
components
in
your
machine
learning,
Frameworks
and
based
on
the
particular
organizational
use
cases?
There
would
be
a
pick
and
choose
the
best
of
breed
tools
for
those
different
areas.
Now,
even
though
we
are
now,
you
know
getting
a
little
bit
more
of
an
understanding
of
the
shape
of
production
machine
learning.
B
There
is
starting
to
be
a
convergence
around
being
able
to
accept
we're
moving
into
a
best
of
breed
set
of
tools,
and
we
are
having
a
sort
of
standardized
set
of
components
that
we
are
still
expecting
to
be
filled
out
right.
Like
we
still
expect
to
have
data
engineering,
we
still
expect
to
have
perhaps
even
feature
store,
a
a
deployment
and
serving
framework.
B
So
you
know
when
looking
at
this
from
the
end
to
end-to-end
envelopes
life
cycle-
you
know
this
is
the
ml
workflow
ecosystem
that
is
put
by
the
Linux
Foundation.
You
can
see
a
pretty
standard
overview
of
the
you
know:
life
cycle
of
a
machine
learning
model
from
the
data
cleansing
data
engineering
data
splitting
then
the
training
of
the
models,
the
evaluation
of
the
models
and
then
the
deployment
of
those
models
with
monitoring
logging
fine-tuning
and
then
going
back
to
the
to
the
to
the
beginning
of
the
life
cycle.
B
The
key
thing
here
the
intuition
is
that,
with
that
in
mind,
we
already
decided
that
we're
going
to
have
this.
You
know
best
of
breed
pick
and
choose
the
best
tool
for
each
of
these
stages.
Now,
what
we
want
to
make
sure
is
that,
as
the
resources
are
handed
over
across
every
single
part
of
the
stack,
you
have
full
lineage
full
reproducibility
auditability.
In
order
for
you
to
know,
where
did
this
come
from?
What
do
I
have
out
there?
How
do
I
go
back
and
to
find
who
did
what?
B
What
is
the
responsibility
and
accountability
structure
across
my
digital
resources
as
they
had
as
they
get
handed
over
across
the
stack?
So
those
are
key
questions
that
have
to
be.
You
know
asked
and
considered
now
that
we're
living
in
this
situation,
where
we're
going
to
be
bringing
this
best
in
breed
pick
and
choose
tooling
right
yeah.
B
You
know,
from
from
that
perspective,
it's
important
to
to
make
sure
that
you
choose
the
best
tool
for
your
application,
but
at
the
same
time
you
do
need
to
abide
by
the
Baseline
standards
of
best
practice
at
each
of
these
stages
and
ensure
that
each
of
these
vendors
suppliers,
open
source
tools
have
the
required
mechanisms
to
ensure
those
principles
are
enforced
around
the
you
know,
metadata
management
that
we're
going
to
be
covering,
but
then
also
other
principles
around.
You
know
that
that
we
have
other
talks
that
you
can.
B
You
know
dive
in
this
in
this
conference
around
reproducibility,
you
know
operational
management,
etc,
etc.
Now
this
is
equally
important
because
we're
moving
from
what
is
a
centralized
homogeneous
world
into
a
heterogeneous
decentralized
world.
So
you
know
I,
don't
know
if
you
remember,
perhaps
a
couple
of
years
back.
B
Well,
now
that
we
have
this
centralized,
you
know
investment.
This
is
becoming
the
bottleneck.
This
is
becoming
a
bottleneck
both
from
a
technological
standpoint,
but
also
from
the
domain
standpoint.
We
have
some
teams
that
have
very
different
requirements:
the
marketing
team,
the
operational
team,
the
data
analytics
team,
the
advanced
research
team.
They
all
have
data
requirements.
They
all
have
machine
learning
requirements
that
are
going
to
necessarily
have
some
consistent,
reusable.
B
You
know
Trends
within
their
unit,
so
this
is
where
we
are
starting
to
move
into
the
concept
of
the
data
mesh
right
data
mesh.
The
first
thing
that
you
think
about
is
an
infrastructural
data
mesh
right
like
a
set
of
tooling,
that
is
decentralized,
but
the
interesting
thing
is
that
when
you
look
at
a
lot
of
the
proposals
for
the
data
mesh
architectures,
this
is
more
an
organizational
structure
in
how
you
can
think
of
your
your
data
products
right.
B
This
is
how
you
can
actually
link
the
domains
in
a
vertical
perspective,
so
that
you
can
actually
have
the
concept
of
a
data
product,
Right
Moving
from
the
concept
of
just
machine
learning
projects
delivering
answers
into
a
reusable
set
of
infrastructure
tables
databases,
golden
data
that
you
can
revisit
now.
The
interesting
thing
is
that
this
is
actually
being
explored
in
the
world
of
data
Ops
right
and
the
world
of
data.
Ops
has
done
a
lot
of
really
interesting
research
around
how
to
tackle
this.
Now.
B
What
we
want
to
do
in
this
talk
is
to
start
extrapolating
it
into
the
world
of
envelopes
and
push
towards
bridging
these
two
worlds.
Where
you
have
the
data
analytics
in
one
side,
you
have
that
sort
of,
like
you
know,
spark
world
where
you're
trying
to
kind
of
like
Get,
interactive,
analytics
and
interactive
insights
into
the
you
know,
perhaps
mlops
world,
where
you
have
operational
deployments
of
models
real
time,
semi-real-time
serving
data,
Centric
view
of
your
ml
systems,
etc,
etc.
B
Challenges
of
metadata
management
at
web
scale
from
the
data
Ops
perspective.
There
was
a
really
interesting
talk
from
the
data
Hub
Founders,
so
this
is
basically
the
LinkedIn
internal
metadata
management
framework
that
then
was
open
sourced.
They
have
actually
some
really
interesting
insights
around
what
are
the
challenges
of
metadata
at
real
web
scale,
so
there
are
challenges
that
they
raised
around.
You
know,
complexity
and
extensibility,
so
these
are
challenges
around
facing
this
best
of
breed
tooling.
B
The
fact
that
now
you
have
to
deal
with
heterogeneity
of
interfaces,
heterogeneity
of
systems
this
you
know
push
towards
democratized
data
mesh
where
the
power
is
now
being
pushed
towards
the
domains,
as
opposed
to
a
central
data.
Lake
team,
this
integration
with
multiple
SAS
platforms,
to
be
able
to
ingest
all
of
that
metadata
and
have
the
centralized
understanding
of
what
I
have
out
there:
the
integration
with
open
source
tools
that
are
already
out
there
that
you're
trying
to
bring
into
your
organization
the
questions
about
you
know
Loosely
typed
versus
strongly
typed.
B
You
know
you
want
to
have
flexibility
for
your
users
to
be
able
to
Define
Rich
metadata,
but
you
also
want
to
enforce
structure
so
that
you
can
ask
meaningful
questions
on
top
of
that
schema
Evolution.
How
do
you
make
sure
that
you
have
backwards?
Compatibility
that
doesn't
have
breaking
changes
and
then
use
case
extensibility
right
to
make
sure
that
there
is
domain
knowledge
within
that
metadata?
There's
challenges
at
scale
and
not
only
scale
but
also
heterogeneity
at
scale.
So
these
are
the
data
relations
right.
B
Whenever
you
have
like
graph
relationships
when
you're
extending
to
like
you
know
millions
of
vertex,
vertices
and
millions
of
nodes,
you
can't
just
have
like
a
single.
You
know
graph
database
node,
a
machine
node
where
you're
running
all
of
this
compute.
You
have
to
think
about
web
scale
when
you're
actually
performing
some
of
those
queries
and
also
the
liveness
of
your
data
right.
It's
like
thinking.
Where
should
I
have
my
data?
Should
it
be
in
in
somebody
not
relational
that
is
then
transferred
or
then
should
I
pull
it
for
some
analytics
into
my
graph?
B
Is
that
going
to
be
like
a
centralized?
You
know
golden
data
set
that
is
going
to
be
consistently
available,
or
is
that
something
that
is
like,
you
know,
run
you
know
and
made
available
once
so
that
the
analytics
is
carried
out
and
then
removed
right,
there's
also
bottlenecks
when
it
comes
to
database
database
types
right.
B
So
if
you
just
adopt
transactional
databases,
and
you
have
to
have
different
types
of
queries,
this
would
be
another
challenge,
and
we
also
see
that
there's
not
only
you
know,
challenges
of
wanting
to
answer
questions
in
relational
or
data
structures,
but
also
full
text
search
right.
So
when
it
when
it
comes
to
having
not
just
you
know,
string
based
regex
and
you
want
to
have
like
you
know,
even
embedding,
based
search.
That
is
the
question
of
well.
B
You
know
how
do
I
then
achieve
that
at
scale
in
an
operational
manner,
in
my
in
my
data
mesh
perspective
and
then,
of
course,
with
reliability.
Of
course,
you
know
in
the
kubernetes
conference
we're
all
aware
about
system
reliability,
but
when
it
comes
to
data
reliability,
it's
a
completely
different
Beast.
It's
a
completely
different
problem
right
thinking
about
real-time
sync,
between
six
systems,
but
with
potential
disparities,
as
this
is
carried
out
at
massive
scale:
zero
downtime
requirements
when
actually
carrying
out
migrations
and
also
ensuring
that
you
have
the
auditability
in
this.
B
In
this
perspective,
now
thinking
about
the
scale
of
this
challenge,
you
can
you
can
realize
now
that
that
this
is
challenging,
because
the
metadata
itself
can
be
as
complex
and
as
large
as
not
just
your
big
data,
but
also
as
your
operational
ml
systems
right
so
you're
now,
just
not
only
dealing
with
big
data
and
the
complexities
of
that,
but
you're
dealing
with
big
metadata.
So
let's
not
coin
that
phrase.
B
So
now.
Actually
you
know
talking
about
the
the
author
of
this
data
Hub
project
that
was
talking
about
the
challenges
at
scale.
So
this
is
actually
something
really
interesting,
because
this
shows
you
what
is
the
architecture
of
their
metadata
management
platform
right
when
you
think
of
a
metadata
management
platform
you're
like
well,
why
don't,
we
just
like,
add
a
postgres
database
or
a
MySQL
single
server
database
and
just
store
some
of
our
basic
metadata,
but
here
you
you
can
see
that
within
their
metadata
ecosystem.
B
You
know
they
would
have
ways
of
automatically
crawling
metadata
across
different
servers
across
different
data
they're
able
to
actually
stream
it
into
the
different
applications.
They
would
have.
You
know
horizontally
scalable
jobs
that
are
able
to
process
that
data.
They
would
have
that
relational
interface
towards
our
metadata
store.
They
would
have
those
requirements
for
that.
You
know
graph
type
queries
they
would
have
the
requirements
for
full
text
search
and
then
also
that
interactive
ways
in
which
people
can
can
consume
the
the
the
I
guess
the
metadata
of
the
metadata
platform.
B
So
so,
from
that
perspective,
you
can
even
think
to
yourself.
Why
is
this
so
complex?
Why
would
anyone
go
into
that
crazy
architecture
for
just
a
simple
metadata
service?
The
reason
is
because
now
we're
we're
looking
at
the
challenge
at
web
scale.
Now
you
know
the
the
point
that
I
had
mentioned
about
bridging
those
worlds.
There
is
a
lot
of
you
know
really
interesting
research
that
has
been
done
in
the
data
Ops
World.
This
is
because
Big
Data,
big
data
has
been.
B
You
know
alive
for
a
long
enough
time
that
people
have
been
asking
well.
How
do
I
keep
track
of
all
of
the
tables?
The
views,
the
visualizations
that
are
available?
You
know
how
do
I
perform
data
Discovery?
How
do
I
store
my
my
data
schemas?
How
do
I
have
like
a
central
object,
blob
store
with
unstructured
data
and
actually
have
an
understanding
automatically
even
of
what
are
the
the?
What
is
the
shape
of
my
data
across
each
of
these
locations
right
and
the
data
ops
world
has
had
some
intersection
with
the
ammo
world
right.
B
We
have
seen
actually
people
that
are
using
spark
or
people
that
are
using
like
Central
data
lakes,
that
still
run
ml
jobs
in
like
batch
or
or
mapreduce
type
functionality,
and
we
have
seen
actually
the
the
initial
attempt
to
starting
to
trace
metadata
of
machine
learning
models.
You
know
some
really
great
initiatives
like
the
model
card
that
was
published
a
while
back
that
defines
you
know
what
are
the
the
expected
attributes
of
a
machine
learning
model
they
quit.
B
You
know
they
push
towards
model
artifacts
that
has
been
a
primary
way
of
dealing
with
metadata
management
is
through
artifact
stores
and
we're
going
to
talk
a
little
bit
about
that.
They
also
have
been
starting
to
explore
how
to
deal
with
model
versioning
right,
versioning
of
artifacts
versioning
of
experiments,
the
dependencies
of
the
models.
What
does
it
need
to
run
the
risks
of
the
use
case
of
the
model?
You
know
what
are
the
explainability
constraints
Etc,
as
well
as
the
ownership
it's
like
who
built
this
model?
B
How
does
it
link
to
the
relevant
data
set
now
we're
going
to
start
diving
into
how
mlops
adds
further
complexities
right?
How
mlops
would
be
slightly
different
to
the
data
Ops
and
ml
world?
So
the
reason?
Why
is
because
you
would
have
a
live
systems
right
like
systems
that
you
can
query
in
an
operational
manner?
You
would
be
deploying
machine
Learning
Systems,
as
you
would
with
microservices,
and
you
would
actually
have
similar
challenges
that
you
have
in
your
microservice
infrastructure
for
service.
B
Discovery
questions
your
you
know,
you
know
API,
you
know
schema
questions
to
answer
what
do
I
have
out
there,
so
things
like
ml
service,
Discovery,
inference
data,
schemas
model
data,
schemas
model,
artifacts,
schemas
pipelines
that
are
deployed
as
well
as
system
ownership
around.
You
know
who
gets
called
at
night
if
if
the
system
goes
down,
etc,
etc,
so
let's
actually
now
dive
into
the
first
point,
which
is
how
our
entity
relationships
in
the
mlops
world
different
right.
So
if
we
take
the
anatomy
of
production
ml,
you
know
we
actually
saw
this
earlier.
B
So,
let's
think
about
this.
As
you
know,
your
training
data,
your
artifact,
store
your
inference
data
and
then
your
metadata
that
we're
going
to
be.
You
know
delving
into
later
on.
So
you
have
the
experimentation
stage
where
you
have.
You
know:
machine
learning,
Engineers
data
scientists
performing
hyper
parameter.
You
know
tuning
training,
evaluation
to
ultimately
convert
and
train
artifacts
Deployable
artifacts
from
training
data
right.
You
would
then
have
you
know:
continuous
integration,
continuous
delivery
pipelines
for
you
know
whether
it's
ETL
jobs
that
are
consistently
converting.
This.
B
You
know,
artifacts
into
deployed
Services
right,
you're,
deploying
machine
learning
models
as
either
real-time
or
batch
inference.
Services
that
you
can.
You
can
consume
from
those
machine
learning
Services.
You
may
also
add
Advanced
monitoring
components,
things
like
drip,
detection,
explainability
that
detectors,
etc,
etc,
and
you
would
want
every
single
input
and
output
of
your
data
to
be
also
stored
in
you
know
what
would
be
your
inference
store
right
for
audit
Trails
for
reproducibility,
etc,
etc.
B
Now
what
you
want
to
ask
the
question
is:
if
I
was
to
have
thousands
of
machine
learning
models
deployed?
Well,
the
question
is
well.
After
I
run
my
experimentation
after
I
run
my
CI
CD
after
I.
Have
dozens
of
you
know,
data
analytics
teams,
productionizing
machine
learning
models.
What
do
I
have
in
my
production
environment?
What
do
I
have
in
this
in
in
this
you
know
highly
scalable
kubernetes
Cloud
native
ecosystem.
That
I
can
of
course,
reuse
right.
What
are
the
services
that
I
have
replicated?
B
What
are
the
instances
where
I
have
like
this
pre-processing
NLP
pipeline
deployed
40
times
across
12
different
teams
right?
These
are
the
questions
that
you
would
want
to
ask.
So
then
you
can
ask
well
what
value
can
I
extract
from
what
I
already
have
out
there
and
then
even
going
Beyond
of
like
well?
How
can
then
I
start
mapping
problems
within
my
business
with
things
that
I
already
have
as
capabilities
in
the
production
environment?
So,
let's
actually
dive
of
what
are
the
limits?
You
know,
specifically
the
limits
of
model,
artifact
stores.
B
Why
can't
we
just
use
an
artifact
store
as
a
production,
metadata
management
tool?
So,
let's
think
about
first,
our
data
set
right.
So,
let's
take
you
know
instances
for
data
set
a
so
here
we
would
have
data
set
instance,
A1
a2a3,
A4.
Until
a
n,
then
we
would
have
a
set
of
experiments
right.
This
is
still
in
our
experimentation
side,
where
you
wouldn't
want
to
train
data,
sets
it
into
trained
artifacts,
so
we
would
first
train
an
initial
experiment
that
gives
us
a
model
artifact
A1
right.
B
So
in
this
case
we
have
model
A1
that
was
trained
with
the
initial
half
of
our
data.
We
then
have
model
artifact
a
2
A3
until
am
right,
so
each
of
them
trained
with
different
subset
of
data
with
different
hyper
parameters,
etc,
etc.
Since
you
know,
similarly,
we
have
another
model
trained
with
a
different
data
set,
so
this
is
what
our
artifact
store
stores
right,
the
artifacts
that
are
available
for
deployment.
So
now,
what
is
the
relationship
that
we
have
in
terms
of
the
challenges
in
our
production
environment?
B
Well,
the
challenge
that
we
have
is
that
when
we
deploy
models,
we
are
creating
instantiations
of
existing
artifacts
right.
We
may
have
our
artifact
am
deployed
in
environment
c
x,
but
also
in
Cy
right.
We
have
our
the
same
artifact
deployed
in
two
different
environments,
but
also
we
have
another
pipeline
that
is
using
model.
M
am
but
also
Model
B1
right.
So
we
have
now
this
very
complex.
B
One-To-Many
relationship
and
many
to
many
when
it
comes
to
pipelines
that
we
know
how
to
think
about
not
just
from
the
perspective
of
the
deployments,
but
also
the
versioning
right,
because
what,
if
I
change
the
pipeline
to
have
another
model?
That's
a
new
version
of
the
pipeline,
but
still
perhaps
the
same
models
that
I
have
deployed.
So
you
know
I,
don't
expect
you
to
kind
of
like
you
know,
internalize
and
grasp
all
of
this
by
memory.
B
But
just
you
know,
the
key
thing
of
this
slide
is
that
it's
hard
right,
it's
not,
and
it's
not
something
that
we
can
just
use
with
with
the
existing
one-to-one
relationship
that
would
be
enforced
with
existing
model
artifact
stores.
So
we
have
to
consider
many
to
many
relationships
in
production
environments
and
because
of
that,
we
need
to
have
some
service
discoverability.
If
you
deploy
a
machine
learning
model
that
will
have
metadata
that
is
extended
to
what
the
artifact
would
have,
because
it
would
have
further
descriptions.
B
Perhaps
the
schema
that
is
exposed
for
the
apis
is
different.
Perhaps
the
actual
parameters
that
were
provided
in
the
environment
variables
have
a
make
the
make
the
artifact
behave
differently.
Perhaps
you
have
different
versions
that
have
to
be
considered
with
simple
changes
in
your
service,
and
these
actual
Services
would
have
to
be
able
to
in
some
way
provide
that
data
so
that
it's
actually
discovered
right.
So
with
that
now
we
can
think
about.
Well,
how
do
we
then
tackle
this
Challenge,
and
why
would
we
tackle
this
challenge?
B
What
we're
doing
is
just
trying
to
ask
the
question:
what
are
we
responsible
for
in
our
system?
What
are
the
services
that
are
running
within
our
system?
What
is
the
metadata
that
is
running
within
our
system
and
what
are
the
ways
in
which
external
systems
can
consume
that
data
for
service
Discovery
and
for
higher
level
enrichment?
B
So
so
that
is
basically
the
the
call
to
action
in
regards
to
the
to
the
relationships.
Now,
let's
dive
into
another
problem,
so
now
we
have
nailed
what
are
our
relationships
in
in
terms
of
our
envelopes
deployment
and
our
machine
learning
Services.
Now,
let's
ask
the
question
about
data
right.
We
have
now
a
problem
right.
We
have
the
data
ops
world
that
is
operating
at
the
data
analytics
side
at
the
experimentation
side
at
the
at
the
training
data.
Let's
call
it
side,
of
course.
That's
not!
B
You
know
you
know,
but
just
just
for
the
sake
of
simplicity
at
are
the
training
data
of
the
of
this
of
this
slide
that
I
had
shown
and
then
the
inference
production
machine
learning
services
are
basically
creating
data
in
your
inference
world
right.
So
this
is
on
scene
data
points
that
may
actually
have
Divergence
distribute
that
Divergent
distributions
it
might
have
like
you
know
further
considerations.
It
may
have
like
impacts
into
potential
use
cases,
so
you
have
data
that
is
being
also
created
in
this
world.
B
Now,
there's
a
question
of
how
do
we
bridge
this
to
Worlds
right?
How
do
we
make
sure
that,
for
our
our
you
know,
schema
our
data
in
our
data
Lake
of
schema
S1,
and
then
our
data
in
our
inference,
store
of
schema
S2
has
some
like
transformation
function
that
would
allow
us
to
convert
to
and
from
right,
so
we
are
able
to
actually
consume
that
data
in
a
meaningful
way,
but
not
only.
This
is
not
only
important
to
transfer
data
from
from
from
training
from
from
production
to
to
to
our
Central
data
Lake.
B
This
is
also
important
now
that
we're
moving
from
the
concept
of
a
model-centric
view
from
deploying
and
productionizing
machine
learning
models
into
this
data
Centric
view
they're
productionizing
of
machine
Learning
Systems.
So
this
is
a
question
that
we
need
to
ask,
because
now
the
machine
learning
systems
that
we
deploy
are
not
just
a
three-step
NLP
pipeline
that
has
a
pre-processing
step,
an
inference
step
and
a
post-processing
step.
Now
we're
deploying
complex
machine
Learning
Systems.
So
this
is
actually
an
interesting
paper
from
from
Facebook
research.
B
B
So
for
that,
we
have
to
ask
the
question
of
well
now
that
we
have
this
data
Centric
machine,
Learning,
System
Paradigm.
What
are
the
problems
that
we're
facing?
Okay?
Well,
let's
dive
into
some
of
those
problems.
The
first
problem
is
inference
data,
heterogeneity
right
back.
You
know
a
couple
years
back,
each
of
the
machine
learning
deployment
and
serving
Frameworks
had
a
different
interface
had
a
different
signature,
so
us
at
7
core
had
the
seldom
protocol
right.
It
looked
like
an
array
within
the
array.
B
K-Serv
also
had
a
different
protocol
had
something
that
looked
kind
kind
of
like
TF
serving,
but
also
a
little
bit
different
TF
serving
had
a
different
protocol
and
also
had
a
different
protocol
pie
torch
just
allowed
you
to
send
binaries.
So
you
know
it
had.
You
know
another
protocol,
so
you
had
a
situation
where
you
had
different
ml
services
that
are
running,
but
you
have
stuff
flying
around
in
ways
that
are
completely
unstandardized,
so
the
first
step
is
to
tackle
that
problem.
B
Unfortunately,
you
know
we
already
have
a
solution
to
that
problem,
which
is
already
in
production
and
adopted
by
several
players
in
the
market,
so
this
is
actually
called
the
V2
protocol.
So
it's
a
collaboration
between
Nvidia
Selden,
the
Triton,
the
case
serving
team
to
start
to
come,
come
into
an
alignment
of
what
is
a
standardized
protocol
that
we
can
adopt
so
that
everything
that
is
flying
around
in
your
machine,
Learning
Systems
is
actually
standardized.
B
So
you
know
for
for
this.
You
know
we
can
see
it
as
the
open
inference
protocol.
This
is
a
standardized
schema
that
allows
you
to
have
a
rest
and
grpc
interaction
between
your
machine
learning
services,
and
this
is
great
great
right,
because
not
only
we've
been
able
to
come
up
with
a
standardized
signature,
but
we
have
been
able
to
also
provide
a
minimum
level
of
features
and
functionalities
for
anyone
that
claims
to
have
a
serving
capability.
B
We
we
have
kind
of
like
the
standard
interfaces
for
multi-model,
serving
by
Design
multiversion
support
for
machine
learning,
models
that
are
deployed,
Model,
Management,
apis
model,
metadata,
apis
and
server.
Slash
model
Health
apis
right,
so
this
is
actually
basically
not
only
aligning
on
standards,
but
coming
into
an
alignment
of
what
do
we
feel
are
the
Baseline
best
practices
or
how
to
avoid
bad
practice
at
the
very
minimum
in
order
for
anyone
that
is
deploying
machine
learning
servers
into
production.
So
that
is
the
first
step
right
like
so
now
that
we
have
standardized
apis
standardized
interfaces.
B
So
now
we
can
look
at
the
holistic,
end-to-end
lifecycle
of
the
machine,
learning
pipelines
and
start
asking
the
questions
of
well.
Now
that
you
have
deployed
machine
learning
Services
now
that
you
have
experimentations
that
are
running
evaluation
at
scale
now
that
you
have
like
data
labeling
services
that
also
require
the
data
schema
to
be
able
to
function
properly.
How
do
we
start
thinking
about
mapping
all
of
these
different
stages
to
start
getting
and
empowering
the
business
units
to
collaborate
without
having
to
without
without
the
need
of
a
centralized,
a
centralized
thinking?
B
So
the
question
now
is:
how
do
we
then
map
this
this
deployment
side
with
your
data,
Lake
data,
labeling
earlier
stages
of
the
equation
and
the
way
to
think
about
it
is
well
now
that
we
have
a
standardized
protocol.
We
can
start
asking
meaningful
questions
of
what
does
this
ml
service
actually
do
right
and
we
can
think
about
an
ml
Service
as
a
component
that
has
a
set
of
meaningful
inputs
and
meaningful
outputs.
B
It
is
in
a
way
you
can
think
about
as
a
database
query
right,
you
can
have
a
machine
learning
model
that
is
deployed
and
if
you
have
a
specific
data,
a
specific
data
set,
you
can
actually
run
a
query
by
transforming
that
data
set
by
running
it
through
that
specific
model.
So
you
can
ask
the
question
of
okay.
Well,
I
have
the
this
machine
learning
service
with
this
shape?
With
this
expected
signature,
and
now
we
have
to
extrapolate
into
what
we
refer
to
as
pipelines
or
machine
Learning
Systems.
B
We
have
the
concept
of
a
simple
pipeline
right.
You
have
an
input
that
then
is
passed
into
another
model,
and
then
you
have
the
actual
output.
So
the
the
the
the
actual
interface
of
this
pipeline
is
the
input
of
the
model
1
and
the
output
of
the
model
2..
Similarly,
you
can
have
combiners
where
you
have
the
input
into
two
models,
and
then
you
have
to
the
output
from
a
from
a
combined
aggregation
of
these
two
models.
B
But
then
you
can
also
have
machine
Learning,
Systems
right
where
you
have
real
world
scenarios,
where
you
want
to
ask
the
question
of
what
is
my
inputs?
What
are
my
outputs
and,
more
importantly,
asking
the
questions
of
how
is
my
inference
being
affected,
Downstream
and
up
the
stream?
So
those
are
the
type
of
questions
that
we
would
want
to
ask.
So
again,
we
have
good
news
for
this
that
we
have
now
standardized
a
protocol
for
the
extension
of
this
op
open
inference.
Protocol
into
an
open
inference
schema.
B
B
B
You
know:
I
guess
achieve
this,
this
this
capability
for
end-to-end
interoperability
at
scale-
and
you
know
collaborate
further
into
you-
know,
defining
this
and
solidifying
this
so
that
it
can
become
a
proper.
You
know,
even
to
a
certain
extent,
a
proper
standard.
You
know
in
in
the
sense
of
like
the
isos
and
the
I3
police
and
the
reason
why
is
because
in
several
cases-
and
this
is
a
quote
to
remember-
bad
standards
can
be
better,
can
be
worse
than
no
standards.
So
yeah.
B
We
don't
want
bad
standards,
we
want
yeah
bad
standards
better
than
no
standards
right
and
it's
something
that
like
at
scale
you
have
to
like
you
know
consider,
because
when
you
have
no
standards,
you
just
have
like
stuff
flying
around
right.
Like
you,
don't
know
what
is
the
potential
value
that
you
could
consume?
So
you
know
I'm
not
saying
like.
We
need
bad
standards,
so
we
shouldn't
aim
for
bad
standards.
B
We
should
aim
for
good
standards,
but
at
the
end,
it's
like
at
least
having
some
that
is
standardized
and,
of
course,
you
know
making
sure
that
that
we're
not
just
creating
a
standard
to
rule
all
of
the
other
standards
right,
because
otherwise
we're
going
to
end
up
in
like
a
recursive
Loop
that
is
never
gonna
end,
so
yeah.
So
with
that
I'll
leave
that
in
the
in
that
call
to
action.
B
A
B
Yeah,
that's
a
great
question,
so
the
answer
is
is
no
so
so
you
know
I
can't
I
can't
stand
here
and
you
know
claim
that
I
would
understand
and
become
sort
of
like
fully
knowledgeable
in
the
other
stages.
So
one
of
the
main
reasons
for
this
talk
is
more
for
that
call
to
action
to
kind
of
encourage
the
other
areas
of
the
ml
life
cycle
to
push
towards
coming
up
with
those
tough
questions.
B
What
is
the
metadata
that
I
am
accountable
for
providing
and
that
is
able
to
be
consumed
by
those
higher
level
systems?
What
I
can
say,
however,
is
that
there
is
a
lot
of
really
interesting
and
applicable
content
in
the
earlier
stages
of
the
inner
life
cycle
in
the
experimentation
side
in
the
data
analytics
side.
B
Okay,
so
yeah,
so
the
question
is:
there
is
a
broad
range
of
machine
learning,
algorithms
that
are
continuously
growing
in
number.
How
do
we
deal
with
that
complexity
in
the
context
of
of
standardization
and
metadata?
No,
that's!
That's
a
great
question
so
the
way
that
we,
the
hypothesis
that
we
have
right,
is
basically
that,
in
the
data
science,
data
analytics,
I,
guess
part
of
the
machine
learning
life
cycle.
B
The
convergence
is
towards
heterogeneity,
right,
relevant
tools
for
Relevant
use
cases.
Some
may
be
using
Sparks.
Some
may
be
using
all
their
tools
right
and
different
algorithms
right.
But
what
we're
now
looking
into
into
the
convergence
and
on
the
later
stages
of
the
machine
learning
life
cycle
is
then
standardization
of
interfaces
metrics
and
operational
considerations,
so
you
can
have
whatever
like
in
the
microservice
world.
You
you
as
an
Ops
person.
You
don't
ask
tell
me
exactly
what
is
the
lines
of
codes
on
in
your
Django
app
right?
B
You
just
ask
the
question
of
what
are
your?
What
are
your?
What
are
your
you
know,
Service
interfaces,
what
are
the
metrics
that
you
Expose
and
what
are
the
slos,
slis
and
slas-
that
you
want
to
establish
in
order
for
us
Ops
people
to
be
able
to
like
look
after
your
services
and
make
sure
you
avoid
leakage
of
abstraction.
B
So
one
like
some
of
the
things
that
we're
focusing
on
is
creating
tools,
like
you
know:
ml
server,
for
machine
learning
serving
seldom
core
for
orchestration
and
kubernetes
to
to
to
come
up
with
standardized
layers
that
allow
you
to
just
abstract
the
data
science
science
separate
to
the
operations.
So
I
would
say
that
we
are
going
to
most
likely
be
looking
at
heterogeneous
algorithms
heterogeneous
tools,
but
with
the
with
the
push
towards
abstracting
them
from
the
Ops.