►
From YouTube: GitLab Experiment Tracking - January 2023 Overview
Description
Epic: https://gitlab.com/groups/gitlab-org/-/epics/9341
Feedback Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/381660
All Updates: https://gitlab.com/gitlab-org/incubation-engineering/mlops/meta/-/issues/16
A
Hello:
everyone,
my
name
is
Eduardo
I'm,
an
incubation
engineer
at
kitlab
working
on
mlops
and
for
the
past
few
months,
I've
been
working
on
incubating
my
machine
learning,
experiment,
tracking
agit,
lab
and
today
I
want
to
give
a
bit
of
an
overview
of
the
current
status
and
a
little
bit
go
over.
What
is
it
that
I
am
doing?
What
is
the
vision
and
where
we
are
at
what
are
the
next
steps
and
so
on?
So
for
starters,
what
is
experiment
tracking
when
we
work
with
machine
learning?
A
A
machine
learning
model
is
usually
seen
as
a
composite
as
composed
by
three
parts.
For
you
to
replicate
for
you
to
to
recreate
the
same
model,
you
need
the
same
artifact.
You
need
three
pieces
of
information.
First,
the
code
that
was
used
to
create
the
model.
Second,
the
data
that
was
used
that
was
passed
to
the
code
that
the
code
learned,
the
the
relationships
from
and
third,
the
hyper
parameters,
the
configurations
for
the
code
for
the
data
for
the
environment
itself.
A
A
We
do
a
search,
we
create
a
set
of
hyper
possible
hyper
parameters
and
we
do
a
search
over
this
for
the
same
code
for
the
same
data
and
we
set
a
lot
of
different
hyper
parameters
and
each
of
those
sets
will
create
a
different
candidate,
so
a
different
candidate
for
our
model
that
will
be
compared
to
the
other
candidates.
This
is
called
hyper
parameter
tool.
A
Now,
Beyond
hyper
parameter
tuning
experiment
tracking.
We
can
have
not
only
the
hyper
parameter
being
changed,
but
the
code,
the
data
in
itself,
the
hyper
parameter
each
one
of
these
combinations
will
generate
a
candidate
and
an
experiment
is
a
set
of
comparable
candidates.
So
we
will
have
pointers
to
the
code
to
the
data
to
the
hyper
parameter
and
each
one
will
generate
a
candidate
that
I
want
to
compare
against
other
candidates
and
experiment
tracking.
A
Is
our
metadata
registry
that
tracks
the
results
of
each
of
these
candidates,
so
I
can
have
multiple
experiments
with
many
many
hundreds
of
candidates
and
I
will
be
able
to
search,
compare
them
as
long
as
they
follow
some
of
the
same
comparison
measures,
so
some
Metric
that
they
all
have
in
common.
A
A
Some
of
the
positive
sites
of
mlflow
is
that
it
has
a
great
client,
so
the
client
code
that
go
the
data
scientist
uses
to
save
on
their
code
base
is
really
great.
It
has
a
very
large
user
base,
open
source
and
a
large
number
of
other
tools
integrate
with
mlflow
already
so
deployment,
and
already
some
deployment
tools
are
integrated
with
ML
flow.
A
The
negative
side,
however,
is
that
the
corporate
part,
the
corporate
features
that
are
often
desired
from
from
from
a
tooling,
are
not
there
so
for
there
is
no
user
management,
there's
no
authentication
with
ML
flow.
This
is
one
of
the
most
requested
features
under
on
their
issue
board.
It
needs
set
up.
It
needs
to
perform
a
platform
engineer
to
go
there
and
and
deploy
the
server
is.
It
is
an
additional
service,
so
they
have
to
go,
deploy
no
kubernetes
or
whatever
and
maintain
the
deployment.
A
It's
siled
information,
solid
knowledge,
since
it's
its
own
tool,
you
can
access
through
API,
but
you
have
to
put
effort
for
to
create
Integrations
with
your
other
operations.
So
how
do
we
come
in
at
this?
What
are
our
North
Stars
on
how
to
approach
this
one?
We
want
to
create
an
experiment
tracking
that
for
a
gitlab
for
a
user
that
already
has
gitlab
it's
there's
zero
setup.
If
you
have
gitlab,
you
have
experiment
tracking,
that's
the
first
step
and
it
for
the
data
scientists
Beyond
just
having
the
gitlab
external
tracking.
A
It
will
work
with
their
ml
flow
with
their
code
that
uses
ml
flow
with
minimal
changes,
preferably
zero
changes.
So
the
only
thing
that
we
want
them
to
do
is
change
the
endpoint
of
instead
of
calling
ammo
flow
call
gitlab
and
third,
we
want
experiment
tracking
to
be
more
than
a
vertical
feature
feature
by
its
own.
We
want
to.
We
want
to
reuse
the
information
from
the
experiment
tracking
to
improve
the
experiments.
A
Experience
on
I,
don't
know,
merge,
requests
on
issue
creation
or
or
issue
listing
or
I,
don't
know
anywhere
on
that
that
makes
sense
on
the
platform.
So
we
want
to
Leverage
The
Power
of
the
devops
platform
of
gitlab,
both
to
add
into
it
and
to
use
from
it,
and
how
does
that
manifest?
Technically
one
so
ever
flow
experiment
tracking
has
two
components:
one:
the
client
code,
which
is
the
code,
the
library
that
data
scientists
use
to
push
information.
A
A
The
positive
sides
over
here
is
that
if
the
user
has
gitlab,
then
it
works
because
they
already
have
gitlab,
they
will
have
external
tracking
by
default
it
once
we
start
working
on
it.
It
will
be
a
lot
easier
to
integrate
across
the
platform,
so
integrate
display
your
information
where
it
makes
sense.
A
There's
zero
work
necessary
by
neither
to
set
up
mlflow
itself
or
to
set
up
artifact
storage
or
anything.
We
are
reusing
gitlab
for
this.
For
the
data
scientists,
nothing
changes.
The
only
thing
that
changes
actually
is
that
a
set
of
pointing
to
another
flow,
the
URI
of
whether
they're
calling
they
now
point
to
gitlab
and
the
code
base
will
work
exactly
the
same.
A
The
downsides
here
is
that
we
need
to
keep
API
parity
over
time.
Currently
we
are
supporting
them
of
low
client
at
1.8,
but
ml
flow
is
already
2.0,
so
we
need
to
keep
the
parity
going
and
the
other
downside
is
that
we
need
to
implement
the
whole
UI
so
which
makes
the
experience
better
for
your
user,
because
it's
more
the
same
as
it
will
be
part
of
the
of
gitlab.
So
there's
no
different
Cy,
but
it's
more
work
on
our
side.
A
And
how
far
are
we
I?
Have
this
demo
that
I'm
linking
here
but
I'm
not
gonna
go
over
it?
The
point
is
gitlab
already
serves.
We
already
re-implemented
ammo
flow
API
on
the
gitlab
code
base,
so
you
can
already
Point
your
mflo
code
to
gitlab.
A
We
already
have
implemented
the
minimum
UI
for
this,
so
you
can
already
browse
your
experiments.
Browser
candidates,
compare
them,
it's
not
as
great
as
mlflow,
of
course,
and
we
are
now
dog
fooding
this
MVP
with
internally,
so
our
data
scientists
are
using
it
and
giving
feedback
on
what
they
want
from
it
what's
necessary
to
add
and
yeah.
So
we're
at
the
point
where
we
already
have
the
MVP
or
is
ready
and
is
already
being
used.
A
What
are
the
next
steps?
Where
do
we
want
to
go
over
here?
Next
step
is
consolidate
this
MVP
and
now
move
from
internal
testing
to
external
testing.
This
will
happen
in
the
next
few
weeks
and,
second,
now
that
we
have
the
feature
done
the
vertical
part
of
the
feature
we
want
to
go
horizontal.
We
want
to
see
how
that
information
can
be
leveraged
on
other
places
of
gitlab,
and
how
does
that?
Would
that
mean
for
the
user?
How
can
they
help?
A
How
can
we
they
leverage
this,
for
example,
on
the
on
the
Ci
or
on
the
mar,
are
anywhere
really
and
further
links
over
here?
If
you
want
to
continue
looking
at
it,
so
you
there's
an
epic
there's
the
feedback
issue,
all
updates,
I
release
updates
by
weekly
and
the
handbook
page
as
a
bonus.
A
Some
of
the
questions
that
are,
and
often
a
question
that
I
get
is
what
is
the
difference
between
experience,
tracking
and
model
registry
moderator
is
being
worked
on
as
git
at
gitlab
as
well,
not
by
me,
but
it
is
being
worked
on
and
it
does
have
a
lot
of
similarities.
So,
going
back,
an
experiment
has
all
the
different
code
bases
part
the
code
base,
the
data
set
and
the
hyper
parameters.
A
Each
one
will
be
generating
a
model
candidate,
but
and
some
of
these
candidates
will
be
promoted
to
models
so
on
the
experiment,
tracking
side
I'll.
Some
of
this
candidates
are
creating
even
at
the
local
machine
of
the
user
right
and
the
models
are
some
of
these
candidates.
The
best
candidates
that
I
keep
finding
over
time
will
be
promoted
to
specific
model
versions,
and
these
model
versions
will
then
be
sent,
will
then
be
exposed
to
the
real
world
to
the
application.
A
So
the
main
difference
between
experiment
track
and
model
registry
they
are
built.
They
work
together
on
experiment,
tracking
your
models.
The
model
candidates
have
not
yet
seen
the
outside
world;
they
have
not
been
exposed
to
user,
so
the
metrics
that
you
use
to
compare
them
are
mostly
model
metrics,
so
accuracy
or
area
under
the
curve.
So
more
of
the
academic
machine
learning
metrics,
while
the
model
registry,
you
actually
put
them
in
production,
and
you
can
measure
them
against
usage
metrics,
like
I,
said
on
external
tracking.
A
None
of
the
candidates
are
user-facing
yet,
while
on
the
model
registry
they
already
are,
the
artifact
creation
for
Onex
print
tracking
can
have
can
both
happen
at
the
Ci
or
they
can
happen
at
locally.
So
a
data
scientist
often
creates
those
candidates
not
from
the
the
CIA
itself,
but
when
they
are
coding
before
even
the
first
commit
is
created
when
the
model
registry,
ideally
a
model,
has
always
been
created
through
a
pipeline
there's.
Another
difference
is
that,
where
is
the
stage?
Where
does
each
one
belong
in
the
envelope
develop
stage
for
external
tracking?
A
They
are
more
Associated
to
the
create
stage
so
when
the
user
is
still
creating
their
code
or
still
creating
their
machine,
their
their
ideas,
while
for
the
model
registry,
it's
more
on
the
packaging
side,
so
you
already
have
a
model
that
you
want
to
push
to
production,
so
you
create
a
model
over
there.
You
promote
a
candidate
into
into
a
model
in
version
yeah,
and
those
are
the
difference
that
I
had.
So
this
is
what
I
had
all
I
had
for
today.
Thank
you
for
watching
and
goodbye.