►
Description
Speakers: David Desanto (Sr. Director, Dev, Sec, ModelOps), Taylor McCaslin (Product Manager, ModelOps), Mon Ray (Engineering Manager, ModleOps)
In this session, we will discuss the three pillars of ModelOps, including how to integrate Data Science into DevOps. This will include a brief history of how we got here, as well as where we are going. We're cover the personas of Modelops customers and explore how we'll integrate ModelOps into GitLab.
Chapters:
0:00 Talk introduction
1:01 David Desanto: What the ML is up with DevOps and AI?
2:02 So... What the ML is ModelOps?
7:04 ModelOps of Tomorrow
14:23 Taylor McCaslin: The People behind Data Science
19:38 Data Science Jobs to Be Done
23:23 The Tools of Data Scientists
27:02 The challenges of handoffs
28:38 Mon Ray: Applied ML - How does UnReview work?
31:41 The Full ML Stack
35:45 Challenges building ML models
37:10 The Applied ML Team
38:48 David Desanto: The ModelOps Stage
39:45 We're Hiring!
A
A
B
B
Welcome
to
our
session
again,
it
is
what
the
ml
is
up
with
devops
and
ai,
we're
going
to
give
you
an
introduction
to
model
ops
and
tell
you
a
little
bit
more
about
what
we're
doing
so
standard
public
service
announcement,
disclaimer
I'm
going
to
possibly
let
you
read
it
for
yourself,
but
short
version,
we're
here
to
inform
you
of
some
things,
we're
doing
and
there's
no
contractual
agreement
as
part
of
this
okay,
great
I'm
going
to
assume
you
spent
read
through
that
really
quickly.
Are
you
paused
it?
B
So,
let's
hop
in
so
first
I'm
going
to
talk
about
the
journey
to
model
ops
and
really
give
you
a
sense
as
to
why
we're
doing
what
we're
doing
so.
First,
you
may
not
be
familiar
with
the
term
model.
Ops
model.
Ops
is
actually
a
newer
term
in
the
data
science
community
and
it's
broken
into
multiple
components
and
I'm
going
to
walk
through
those
components
with
you.
As
part
of
this
presentation,.
B
So
model
ops
is
made
up
of
two
key
components:
data,
ops
and
mlops
dataops
is
a
very
common
term.
It
references
the
processing
of
data
workloads.
This
includes
elt
or
etl,
depending
on
which
terminology
you're
most
familiar
with.
It
really
refers
to
focusing
on
extracting
loading
and
transforming
very
large
data
sets.
B
The
goal
with
this
is
to
then
collect
it
from
multiple
sources
be
able
to
aggregate
it
together,
bring
it
to
a
unified
format
and
enable
you
to
begin
to
manipulate
clean
it
and
verify
it
for
use.
There's
a
lot
of
really
common
data.
Ops
platforms
today,
one
that
we
talk
a
lot
about
here
at
good
lab
is
meltano,
which
is
really
focused
on
that
bringing
in
the
data
and
transforming
and
getting
it
ready
for
use.
B
The
next
section
is
emmalops
and
when
you
do
this,
you're
moving
into
productionizing
your
data
workloads-
and
this
includes
everything
from
experimentation,
training
of
your
models,
testing
those
models
and
then
deploying
those
models
into
production.
This
deployment
can
be
both
a
combination
of
partial
rollout,
as
well
as
partial
roll
back
in
case
something
occurring.
You
need
to
bring
it
back
out
of
production.
B
A
component-
that's
not
talked
a
lot
related
to
model.
Ops
is
actually
applied.
Machine
learning,
so
applied.
Machine
learning
is
where
you're
now,
taking
that
data
and
you're
bringing
in
not
just
the
mloffs
and
data
ops
parts
but
bringing
in
the
traditional
software
development
together
again
to
solve
that
use
case.
B
Some
great
examples
of
this:
you
can
use
it
for
things
like
improving
user
experience,
you're
tracking
clicks
through
your
ui
you're,
taking
that
through
a
model
you're
making
automatic
changes
based
off
of
it.
You
hear
a
lot
about
it
in
financial
forecasting
and
in
healthcare
reporting,
where
you're
taking
large
sets
of
data,
bringing
it
together
and
being
able
to
make
informed
data-driven
decisions
in
those
areas.
Of
course,
there's
lots
of
other
trends,
analysis
people
bring
machine
learning
and
software
development
together.
B
One
thing
that
happens
a
lot
of
times
with
data
science,
especially
when
you're
talking
about
these
handovers
you
get
into
teams
using
specialized
tools
and
those
tools
do
not
integrate
together
very
easily.
So
what
you
end
up
with
are
teams
working
in
silos.
You've
got
your
data
science
team
using
specific
niche
tools
that
they
want
to
use.
You
have
your
software
development
team
over
working
in
traditional
devops,
and
you
have
your
security
team
over
in
another
area
and
what
ends
up
happening
here.
B
B
You
know
where
it's
going
to
be
when
it's
going
to
be
ready
to
be
deployed,
and
this
ultimately
leads
into
a
problem
of
where
applications
are
not
leveraging
the
data
well,
and
you
can
think
about
this
as
two
different
trains
all
trying
to
get
to
the
same
station,
but
because
they're
not
unified
together,
one
could
be
getting
ahead
of
the
other,
and
now
you
have
a
model
ready
to
use
data,
that's
not
being
input,
or
maybe
you
have
an
application.
That's
implemented
a
feature,
that's
not
yet
supported
for
the
back
end.
B
The
final
concern
in
pitfall
is
actually
something
we're
very
passionate
about
here
at
gitlab
and
that
is
making
sure
security
is
not
an
afterthought.
We
talked
a
lot
about
this
when
we
were
talking
about
the
launch
of
our
secure
stage
and
how
we
needed
to
shift
security
left.
The
same
thing
can
happen
here.
If
you're
working
in
silos
security
is
going
to
come
in
bolted
on
the
end
and
you're
not
making
sure
things
are
safe
and
secure.
B
Ultimately,
this
kind
of
comes
down
to
the
age-old
question.
I
remember
learning
about
this.
First
software
development
job
straight
out
of
college
someone
talking
about.
I
mean
it
worked
great
on
my
computer.
So
clearly
it
must
not
be
my
problem
and
that's
a
problem
that
can
happen
with
data
science
as
well,
and
only
when
everyone
comes
together.
Do
you
really
get
away
from
finger
pointing
into
a
unified
solution,
and
that
kind
of
brings
us
to
what
we
want
to
talk
to
you
about
today,
which
is
the
model
ops
of
tomorrow?
B
Part
of
this
is
being
able
to
address
those
pain
points
that
I
just
touched
on.
The
goal
is
to
be
able
to
get
visibility,
efficiency
and
collaboration
across
the
entire
organization.
When
you
do
this,
you
end
up
with
teams
working
much
closer
together
and
actually
proactively
collaborating
seamlessly.
You
see
them
beginning
to
point
out
like
hey.
B
One
of
the
big
challenges
with
silent
environments
is
there's
just
a
lack
of
visibility
and
by
bringing
the
team
together
into
the
model
ops
of
tomorrow,
you
now
have
teams
empowered
to
make
proactive
decisions,
be
able
to
better
partner
together,
and
that
brings
us
to
that
big
thing.
That's
important
to
get
a
lot
of,
which
is
embedded
security
when
you're,
starting
together
from
the
beginning,
you
can
begin
to
identify
security
issues
that
need
to
be
addressed
before
your
application
and
your
machine
learning
models
go
to
production.
B
B
You
can
also
get
into
the
ability
to
have
repeatability
one
of
the
biggest
things
with
software
development
and
with
data
science
is
being
able
to
take
it
and
be
able
to
repeat
it
and
have
it
to
be
defensible
and
by
bringing
that
all
together,
you
meet
the
final
goal,
which
is
getting
your
applications
to
leverage
the
data.
Well,
you
no
longer
are
in
this
situation,
where
applications
may
or
may
not
be
ready
to
accept
the
data.
The
data
team
might
be
ahead
together,
everyone's
able
to
do
that.
B
So
if
we
go
back
to
that
definition
of
model
ops,
what
we
really
need
to
do
is
overlay
devops.
On
top
of
it,
so
if
you're
looking
at
model
ops
of
tomorrow,
it's
leveraging
devops
to
connect
all
these
components
together
and
if
you
think
about
it,
devops
has
been
around
for
a
while
and
we
keep
on
making
it
better.
Every
time
gitlab
releases
a
new
release,
we're
making
it
much
easier
to
do
devops.
B
If
you
look
at
this
and
you
relay
it,
you
get
a
ton
of
benefits
and
I'm
just
going
to
list
a
couple
of
them
for
you
just
so
you
get
a
sense
as
to
the
value
you
get
first,
you
can
now
introduce
version
control
using
industry
leading
technologies
like
git.
So
now
you
can
begin
to
actually
version
all
the
components
they're
using
to
do
your
data,
ops
and
your
mlaps.
B
You
get
continuous
integration
of
your
data
science
workloads,
allowing
you
to
train
your
models
at
scale
instead
of
thinking
about
running
that
on
my
local
laptop,
doing
a
local
training
and
then
handing
it
over.
I
can
actually
leverage
my
organization's
private
cloud
data
center,
wherever
they're
running
their
ci
pipelines
to
be
able
to
get
scalability
in
my
training
you
get
baked
in
testing,
allowing
you
to
validate
your
ml
models
prior
to
rollout
and
as
well
as
get
the
benefits
of
partial,
rollout
and
partial
rollback
if
needed.
B
Application
development
is
now
being
done.
Alongside
of
your
data
science
workloads,
giving
you
validation
both
of
the
application
but
of
the
data
with
the
application
and
then
finally,
you
get
collaboration
closely
across
all
your
organizations.
This
includes
your
development
operations,
your
data
and
even
your
security
teams,
and
with
that
together,
you
get
a
much
better
data
science,
experiment
and
experience
before
we
transition
this
over
to
taylor
and
he's
going
to
talk
to
you
about
personas.
I
wanted
to
highlight
a
couple
of
things
for
you.
B
B
Our
goal
here
is
to
enable
data
science
workloads
within
git
lab.
Today,
we're
focused
on
really
two
main
areas.
First
is
mlobs.
We
actually
released
our
first
ml
ops
feature
earlier
this
year.
Back
in
february,
we
announced
nvidia
nvidia
gpu
support
for
the
gitlab
ci
runner,
I'm
here
to
tell
you
today
that
we've
seen
amazing
adoption
of
this
new
functionality.
B
B
This
may
come
in
the
form
of
us
partnering
with
meltano
autonomous,
was
a
former
component
of
gitlab
a
company
and
got
spun
out
as
its
own
organization
earlier
this
year,
or
it
could
include
partnerships
and
other
things
that
we
may
be
able
to
do
to
make
this
happen.
However,
for
the
purpose
of
today,
it
is
just
something
we're
looking
at
to
do
in
the
future.
B
Our
model
up
stage
is
broken
into
four
teams.
Those
teams
include
ml
ops,
which
today
is
staffed
via
our
incubation
engineering
team,
with
a
single
engineer
group
looking
at
how
to
provide
mlabs
functionality
within
gitlab.
We
also
have
our
applied
machine
learning.
Team
who's
focused
again
on
enriching
gitlab
features
with
machine
learning
and
the
final
team
is
anti-abuse,
and
this
is
a
team
focused
on
being
able
to
provide
better
protections
for
threats
inside
of
gitlab,
and
we've
talked
about
this
at
different
conferences,
as
well
as
blogs.
B
The
final
team
is
another
team
we're
focused
on
building
out
today,
but
we
do
plan
on
building
out
a
data,
ops
team
as
well
again
current
plans
that
could
change
as
we
move
forward
and
lastly,
I
did
want
to
highlight
one
last
thing
for
you:
in
june,
we
acquired
a
company
name.
Unreview
unreview's
goal
is
to
improve
the
developers
experience
by
automating
code
review,
leveraging
machine
learning.
B
The
goal
of
this
is
to
be
able
to
identify
who
is
the
right
code
reviewer
to
do
the
code
review
based
off
the
line
of
the
lines
of
code
changed
the
section
of
the
code.
That's
changed
current
load
and
lots
of
other
variables,
we're
very
excited
to
bring
unreview
into
git
lab,
and
these
features
will
ship
later
this
year.
The
team
is
actually
doing
their
integration
today
and
we're
seeing
some
really
great
dog
fooding
internally
here
at
gitlab.
C
Awesome,
thank
you
david.
So
now,
you've
heard
a
little
bit
of
an
overview
of
what
we're
focusing
on
and
some
of
the
challenges
that
exist
with
model
ops.
Now
I
want
to
take
you
through
a
little
quick
tour
of
some
of
the
people
who
are
actually
doing
this
work
day
to
day
and
how
they
differ.
So,
let's
take
a
look
at
some
of
the
personas.
C
This
is
everything
related
to
building
testing,
deploying
monitoring
software,
and
our
whole
thought
here
at
gitlab
is
that
we
want
to
take
the
experience
that
we've
had
building
a
devops
platform
and
really
layer
a
lot
of
those
technologies
on
top
of
model
ops,
to
make
it
easier
to
version
your
machine
learning
models
to
connect
them
with
data
sources
to
do
that
in
a
repeatable
way,
david
talked
about
the
it
works
on
my
machine
problem.
That's
really
a
large
piece
of
what
we're
trying
to
solve
here.
C
We've
already
done
this
with
git
lab
as
a
whole,
making
it
repeatable
and
easy
for
people
to
run
ci
pipelines
and
to
deploy
to
the
cloud.
That's
now
what
we're
applying
to
the
machine
learning
capabilities
here.
So,
let's
take
a
look
at
some
of
the
people
who
are
involved
with
all
of
these.
So
we've
got
the
data
ops
group,
which
is
really
about
getting
the
data
cleaning
it
and
processing
it,
and
when
we
look
at
some
of
the
folks
that
are
involved
with
this,
we've
got
people
who
are
called
data
engineers.
C
We've
got
data
analysts,
the
problems
that
these
people
are
really
focused
on
solving
are
things
like
organizations
have
data
all
over
the
place.
They've
got
messy,
slow,
big
and
useless
data,
and
what
I
mean
by
that
is
that,
because
data's
all
over
the
place,
it's
in
a
lot
of
different
formats,
it
needs
to
be
cleaned.
You
need
to
ingest
it
that
can
be
very
slow
because
you
may
be
pulling
data
from
the
apis
of
various
platforms.
That
data
can
be
really
big.
C
So
you
need
to
be
able
to
store
lots
and
lots
of
data
and
because
it's
messy
there
might
also
just
be
data
points
that
are
really
useless
there
for
you
to
manage
and
may
just
have
no
business.
So
you've
got
to
be
able
to
clean
that
data
to
get
really
useful
data
points
for
the
business
and
then
finally,
you've
got
to
be
able
to
extract
load
and
transform
that
data.
C
So
what
we
mean
by
that
you'll
hear
this
commonly
referred
to
as
elt,
where
you're
extracting
data
from
various
sources,
you're
loading
it
into
a
centralized
database
and
then
transforming
it
into
something,
that's
useful.
You
might
hear
this
called
shaping
data
so
that
you
can
then
run
useful
queries
against
it.
Now,
let's
look
at
ml
ops.
This
is
really
about
doing
something
with
all
that
data
once
you've
got
it
collected.
C
Some
of
the
personas
that
we
see
here
are
data
scientists
and
machine
learning
engineers.
The
problems
that
these
people
are
really
focused
on
is
what
david
was
talking
about
earlier:
the
it
works
on
my
machine,
mlaps
experts
and
data
scientists
are
not
devops
engineers,
they're,
not
super
focused
on
creating
production
level,
engineering
pipelines.
C
Instead,
it's
more
about
getting
and
processing
that
data
and
doing
something
useful
for
it.
So,
in
terms
of
what
you're
focused
on
with
business
problems
here,
you
really
just
need
enough
development
skills
to
be
dangerous.
That's
what
we
see
with
a
lot
of
data
scientists.
They
know
enough
to
get
connected
to
their
data
sources
and
then
do
all
of
the
advanced
machine
learning
techniques
they
use
on
it.
However,
they
don't
really
push
that
model
to
production,
so
they're
not
doing
all
of
the
devops
associated
with
deploying
those
models
to
a
production,
application,
they're.
C
Also
looking
for
the
simplest
solution,
machine
learning
is
not
a
trivial
industry.
It's
a
very
advanced
industry,
and
so
one
of
the
common
problems
with
machine
learning
and
data
science
is
that
you
want
to
use
the
simplest
solution
possible
so
rather
than
over
engineering,
something
you
want
to
try
to
keep
it
really
simple,
so
that
you're
not
introducing
more
complexity
than
necessary.
C
C
These
people
are
focused
on
velocity
of
pushing
software
out
we're
now
in
a
world
where
engineering
teams
deploy
multiple
times
a
day
and
push
out
code
multiple
times
a
day,
they're
focused
on
repeatability,
because
you're
pushing
code
so
much
and
so
frequently
you
need
to
be
able
to
do
that,
repeatably,
whether
it's
on
my
machine,
whether
it's
in
a
cloud
runner
or
on
someone
else's
team,
and
then
you
also
need
to
be
able
to
do
that
securely.
So
they're
focused
on
working
in
securing
their
software
earth
application
security
testing
tools.
C
So
now
that
we've
taken
a
look
at
these
different
personas
and
some
of
the
business
problems
that
they
experience,
let's
talk
a
little
bit
about
the
jobs
that
they
need
to
do
within
organizations.
So
let's
go
back
to
the
data
ops
side
of
things.
Dataops
is
really
about
aggregating
disastrous
data
sources.
Your
data
can
be
all
over
the
place
across
many
different
applications.
Different
databases,
different
clouds.
You
need
to
aggregate
all
of
that
together,
so
that
you
can
merge
it
and
enrich
data
as
needed.
C
C
D
C
Data
looking
for
answers
to
business
questions,
it's
great
that
we
have
lots
of
data,
but
we
need
to
be
able
to
do
something
with
it
and
that's
where
a
lot
of
the
data
analysts
come
through.
They
come
and
interpret
the
data.
They
query
the
data
to
answer
really
important
data
questions.
Now,
let's
focus
on
ml
ops,
so
some
of
the
jobs
that
these
folks
need
to
do
is
they
need
to
explore
and
understand.
The
data
sets
just
like
the
data.
Ops
folks
need
to
be
able
to
analyze
and
answer
business
questions.
Ml.
C
Ops
folks
need
to
understand
the
business
problems
they're
trying
to
solve
so
that
they
can
then
teach
a
machine
to
go
and
find
those
answers
automatically.
This
leverages
a
lot
of
advanced
mathematics
and
other
data
analyst
type
of
functionality
where
these
people
are
really
trying
to
wrap
their
head
around
the
problem
that
they're
trying
to
have
a
machine
answer
for
them.
They
need
to
understand
those
business
problems
deeply,
because
a
small
tweak
in
a
model
can
result
in
a
really
large
change
in
the
end
answer
or
prediction
that
that
model
might
have.
C
They
then
need
to
build
and
train
those
ai
models
and
ml
models.
These
are
very
complicated
and
very
mathematical.
Models.
Mon
will
talk
about
some
of
the
folks
that
were
hiring
to
do
this,
but
these
are
very
specialized
techniques.
You
can
very
quickly
get
into
higher
order.
Matrix,
math
and
other
crazy
mathematical
needs
to
understand
these
ml
models.
C
That's
what
machine
learning
experts
are
here
to
do
and
then
finally
really
understand
what
those
models
are
doing
and
the
output
that
they're
generating
so
that
you
can
iterate
on
them
and
make
them
better
over
time,
and
then,
when
we
take
a
look
at
our
devops
folks,
these
are
people
are
focused
purely
on
writing
code,
they're,
building
and
testing
that
code
and
then
trying
to
get
to
repeatable
ci
cd.
So
it
doesn't
matter
who's
deploying
it.
You
have
that
repeatable
build
every
single
time
and
then
you
actually
deploy
and
run
that
software.
C
So
now
that
we've
looked
at
some
of
the
jobs
to
be
done,
I
want
to
briefly
talk
about
some
of
the
tools
these
people
use.
So
some
of
the
tools
that
data
ops,
folks
use
are
elt
platforms.
There's
a
variety
of
different
elt
platforms.
Now
that
make
it
easy
to
extract
load
and
transform
data
wherever
your
data
may
be.
These
are
platforms
like
snowflake,
which
is
a
popular
data,
warehouse
big
data,
warehouses
from
google
and
amazon
like
bigquery.
They
are.
C
C
Now,
when
we
look
at
the
mlops
personas,
these
people
are
using
what
are
called
python
notebooks
if
you're
not
familiar
with
them.
Basically,
they're
a
software,
that's
written
in
python
that
allows
you
to
visualize
data,
to
interact
with
the
data
and
then
to
run
models
on
it.
You'll
hear
the
term
like
a
data
frame
where
you're
collecting
data
you're,
exposing
that
to
the
notebook
and
then
you're,
exploring
that
data
frame
and
data
set
within
that
python.
Notebook
you've
got
open
source
ml
frameworks.
C
There
are
all
sorts
of
things
like
pi,
torch
and
other
open
source
ml
frameworks.
The
list
of
frameworks
is
kind
of
endless
we're
seeing
a
lot
of
trends
in
the
ml,
ops
and
model
ops
space
that
are
similar
to
the
early
days
of
devops,
where
there
were
just
a
ton
of
tools
that
did
lots
of
different
things,
but
they
didn't
talk
to
each
other.
That's
kind
of
the
stage
that
we're
in
with
ml
ops
right
now,
too,
there's
just
tons
of
frameworks,
open
source
libraries
platforms,
data
tools
that
you've
got
to
smurge
together.
C
There
is
a
new
rise
of
data
science
platforms
that
try
to
solve
some
of
these
problems,
but
they
don't
use
a
lot
of
the
standard
technologies
that
we
expect
from
a
devops
platform
like
git
and
source
code
management
in
ci
cd.
Instead,
they're
built
with
very
basic
version
control
many
times
just
visual
user
interface
versioning.
C
C
Now,
looking
at
the
devops
folks,
it's
very
common
for
developers
to
use
things
like
interactive
developer
environments
or
ides
popular
tools
like
jetbrains
and
vs
code.
You've
also
got
ci
cd
platforms
like
gitlab,
where
you
can
do
those
repeatable
builds.
You've
got
source
code
management
platforms,
also
like
git
lab,
where
you
can
version
your
source
code
and
keep
track
of
it
as
you're,
trying
to
build
and
iterate
on
software,
and
then
you've
got
security,
testing
tools,
things
like
gitlab
and
other
security
vendors
who
allow
you
to
test
and
secure
your
code.
C
So
when
you
look
at
all
of
these
different
personas,
there's
a
lot
of
similarities
between
them,
but
there's
still
a
lot
of
differences
as
well,
and
so
that's
part
of
the
problem
that
we're
trying
to
solve
with
our
model.
Ops
stage
is:
how
do
we
make
it
easier
for
all
of
these
people
to
interact
and
collaborate
more
seamlessly
when
you
think
about
the
handoff
between
data,
ops
and
mlabs?
It's
really
about
the
data
and
the
code
that
processes
all
of
that
data.
C
The
data
pipelines
mlops
engineers
have
to
be
able
to
connect
to
those
data
sources
to
be
able
to
do
anything
interesting
with
that
data
to
run
models
on
it
and
then,
as
the
machine
learning,
experts
are
building
these
models
and
get
to
something
that
does
produce
business
value.
They
then
need
to
be
able
to
hand
it
off
to
the
devops
engineers
to
be
able
to
version
and
deploy
those
models
into
production
applications.
C
These
handoff
points
historically
have
been
really
difficult
for
these
teams
to
manage,
as
you've
already
seen,
they
use
different
tools.
They
use
different
source
code
management
platforms.
They
use
different
platforms
and
open
source
libraries.
There's
a
lot
of
disjointedness
between
how
all
of
these
people
interact
today,
but
that's
kind
of
what
we're
trying
to
solve
here
with
our
model.
Ops
stage
is
make
that
collaboration
really
centralized
in
a
platform
like
gitlab,
so
that
you
can
collect
data
store
it
expose
it
to
machine
learning
models,
iterate
on
those
models
and
then
deploy
them
to
production.
D
D
So,
as
david
had
mentioned,
unreview
is
really
about
making
the
developers
life
cycle
seamless,
with
adding
intelligence
to
the
code
review
process.
The
way
it
all
works
is
looking
into
a
lot
of
data.
A
lot
of
the
comment.
History,
data
merge
request,
data
collected
analyzed,
investigated
and
building
a
patent
lab
library
to
really
really
understand
every
sort
of
biases
or
unbiases
in
the
review
process.
D
D
D
That
was
here
number
of
approvers
that
the
person
has
done
per
month
on
an
unclosed,
merge,
request,
unapproved
merge,
request
the
reviewer's
ideal
time
range
other
kind
of
code.
The
reviewer
is
also
doing
so.
We
don't
want
to
overload
the
reviewer
and
so
on
and
so
forth.
A
lot
of
those
different
attributes
taken
out
of
this
json
extract
to
train
the
model
test
it
and
then
predict
accordingly.
D
D
Git
based
writing
of
the
ml
code.
To
then
the
ci
cd
pipeline,
which
is
just
taking
with
every
trigger
being
able
to
actually
going
into
extracting
to
model
retraining
to
deploying
in
one
seamless
way,
and
then
the
model
registry
versioning
to
prediction
services
and
then
the
final,
the
full
metadata
store.
So
all
in
all,
very
complex
for
a
very
very
may
seem
a
very
simple
prediction,
but
has
a
lot
a
lot
of
details
to
actually
make
it
go
seamless,
orchestrate
it
and
dial
up
the
accuracy.
D
As
far
as
looking
at
that,
full
ml
stack,
our
unreview
architecture,
looks
something
like
this.
Where,
with
every
trigger
of
a
gitlab
ci
template,
we
have
a
extractor
that
extracts.
Those
json
extracts
that
we
saw
going
into
google
pops
up
into
a
loading
transformation
process
to
the
postgres
where
it
stored
the
data.
D
From
that
data
stored,
we
do
model
training
as
well
as
model
serving
again.
This
whole
ci
pipeline
is
a
gitlab
ci
pipeline,
which
is,
I
would
say,
the
first
emmelop
ci
pipeline
that
we've
used
internally
for
a
gitlab
product
to
then
go
into
that
serving
to
to
the
final
part
of
that
ui
of
recommending
the
reviewers
or
what
we
have
for
now
is
also
a
bot
integration,
so
we
can
actually
collect
the
feedback
of
every
time
our
beta
customers,
which
is
our
internal
gitlab
customers.
D
If
there
is
a
wrong
recommendation,
we
can
actually
collect
that
feedback
and
dial
up
the
accuracy
of
the
model.
The
way
I
like
to
think
about,
it
is
almost
like
a
the
amazon
recommendation
engine.
When
someone
is
trying
to
look
for
shoes
and
and
five
years
ago,
probably
it
was
doing
one
out
of
20
shoes
right
and
maybe
now
it's
doing
six
out
of
20
shoes
right
so
really
taking
that
feedback
data
which
is
really
really
important
and
then
dialing
up
those
accuracy
of
understanding.
D
As
far
as
that,
gitlab
ci,
that
we
are
really
excited
to
actually
use
for
the
emelops
part
of
it.
This
is
how
it's
sort
of
gone,
where,
with
every
sort
of
trigger
it's
integrated
to
the
source
code,
the
container
registry,
the
model
training,
the
serving,
as
well
as
the
object,
storing
and
the
training
models
as
part
of
the
full
orchestration
and
automation
to
just
be
able
to
then
take
the
data
and
then
retrain
and
deploy
accordingly.
D
Some
of
the
things
that
we
are
working
on
and
we
want
to
optimize
and
to
better,
are
where
is
that
data
basically
coming
from
and
how
it
will
end
up
whether
it's
building
those
features
and
how
do
we
store
those
features?
Accordingly,
the
kind
of
versioning
of
the
data
and
the
model
that
we
are
using?
How
do
we
actually
help
with
that
reproducibility
and
that
practice
for
it,
making
that
continuous
integration?
D
Currently
we
have
certain
manual
triggers
as
well,
so
how
do
we
actually
take
that
manual
part
all
the
way
out
of
it
and
then,
at
the
end,
when
we
are
building
this
product,
we
are
building
for
customers
all
over.
That
includes
the
self-hosters
as
well
as
gitlab.com,
which
later
we
would
want
to
ponder
and
how
that
what
are
the
complexities
for,
let's
say
for
self-hosting:
where
do
we
train
the
model?
Accordingly,
where
do
we
keep
the
data?
How
do
we
actually
track
all
the
configurations
and
the
authentications?
D
D
As
far
as
the
team,
the
team
is
really
really
growing.
Currently
we
have
the
unreviewed
team
that
join,
who
is,
I
would
consider
the
full
stack
scientist
and
we
have
taylor
as
well
as
me
as
part
of
the
product
management
and
engineering
management
and
then,
as
far
as
taylor
speaks
about
the
personas
that
is
actually
needed
to
build
machine
learning
models
within
our
team.
D
We
are
hiring
a
classical
email
back,
an
engineer
and
then
also
someone
the
whole
team
writes
in
python
to
someone
who's
really
fluent
with
just
packaging
code
in
python
and
being
able
to
deploy
it
accordingly,
and
then
we
also
have
our
full
stack
scientist
from
the
review
process.
Who
is
looking
into
questions
as
to
what
kind
of
training
of
models
that
we
are
going
to
be
using
whether
it
would
be
pie,
touch
versus
tensorflow,
already,
building
the
ci
templates
and
looking
into
better
ways
of
attributing
the
features,
as
well
as
retraining?.
B
B
B
So
if
you're
interested
and
you're
a
gitlab
employee
feel
free
to
reach
out
to
mom
and
wayne
or
taylor
and
myself,
and
we
can
tell
you
about
the
open
roles
and
make
sure
it's
a
good
fit
and,
of
course,
if
you're
watching
this
as
a
non-gitlab
employee,
whether
you're
a
community
contributor
or
someone
who's,
just
decided
to
watch
the
video
on
youtube
because
you
thought
we
all
look,
really
cool
and
I
think
we're
all
really
cool
so
that
works
out.
Well,
please
feel
free
to
reach
out
to
us.
B
Also,
there's
another
contributor
talk,
demystifying
machine
learning
for
engineers,
great
presentation
by
again
alexander
who's,
part
of
the
the
applied
mission
learning
team,
as
well
as
sean
carroll,
who's,
our
backend
engineering
manager
for
our
source
code
team
or
for
those
who
are
not
familiar
with
that
team.
It's
the
scm
component
of
the
product
source
code
management,
part
of
the
product,
so
two
really
talented
and
distinguished
members
of
our
team,
giving
another
great
talk
on
machine
learning,
but
with
that
we
want
to
thank
you
for
watching.