►
From YouTube: Create Deep Dive #4: Git LFS
Description
In this Deep Dive session, Francisco Javier López, Senior Backend Engineer on the Create team at GitLab, shares his knowledge of GitLab's Git LFS implementation.
Download the slides: https://gitlab.com/gitlab-org/create-team/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf
Learn more about Deep Dive knowledge sharing sessions: https://about.gitlab.com/handbook/communication/knowledge-sharing/#deep-dive-sessions
Find out when the next Create Deep Dive is taking place: https://gitlab.com/gitlab-org/create-team/issues/1
---
Read more about our product vision: http://bit.ly/2IyXDOX
Learn about FOSS & GitLab: http://bit.ly/2KegFjx
Get in touch with Sales: http://bit.ly/2IygR7z
A
A
Today
we
are
going
to
talk
about
ETL
FS,
sorry,
we're
going
to
talk
about
giggle
fest,
but
there
are
several
topics
related
to
detail.
Fs.
We
could
talk
about,
so
we
are
focusing
on
what
is
detail
Affairs,
how
you
can
use
Intel
FS
the
specification
behind
needle
FS
and
how
we
build
our
own
l,
FS
server
here
at
isla
okay.
A
This
is
the
table
of
contents,
especially
what
I
have
just
said.
First
of
all,
I
want
to
say
that
I
think
you
all
know
this,
but
it
doesn't
track
binary
files
like
audio
video
or
image
files.
The
same
way
that,
with
with
text
files,
any
change
in
a
binary
in
a
binary
file
requires
to
upload
to
push
a
new
copy
of
the
whole
file
in
the
repository.
Okay,
we
get
can
generate
these
files
for
binary
files.
A
So
that
means
that,
with
a
big
binary
file,
if
we
made
some
modifications
or
update
to
it,
the
repository
can
grow
really
big
really
quickly,
so
the
history
will
also
grow
bigger.
That
means
that
usual
operations,
like
clone,
virtual
or
pool,
will
decrease
the
speed
change
this
page
okay.
So
what
is
digital
FS
keep
LFS
is
an
open
source
project.
A
B
A
A
B
A
A
look
at
the
size
of
this
repository,
the
object
directory
is
almost
four
gigabytes
and
the
file
size
of
the
image
is
almost
device.
So
that
means
that
the
two
that
the
two
changes
pushed
to
new
binary
files
to
the
repository
okay,
now,
let's
take
a
look
at
a
repository
with
digital
FS
I,
have
the
same
image.
A
A
A
A
A
A
But
how
de
LFS
works?
Okay?
First
of
all,
we
need
a
new
LF
s:
entry
in
our
git
repository
complete
file,
this
entry
stores,
the
URL
of
the
LFS
server.
So
now
we
have
two
different
concepts:
we
are
going
to
have
the
usual
remotes
for
git
repositories,
and
we
will
also
have
this
new
LF
a
century
for
the
LFS
server
okay.
But
if
all
when
you
install
the
digital
effects
extension,
the
LFS
URL
is
going
to
be
the
git
repository
URL.
A
As
you
can
see,
here
is
the
entry
of
my
my
LFS
server.
Okay,
then,
once
we
have
installed
that
dealer
get
the
digital
first
extension,
we
have
to
tell
it
which
files
to
track.
It
is
really
easy.
It
use
a
pattern
matching
style,
so
we
if
we
want
to
track
only
PNG
files,
we
have
to
say
great
LFS
track
and
this
pattern.
Okay,
it
only
works
for
new
files.
A
So
if
we
already
have
existing
PNG
files
in
the
repository,
we
have
to
run
git
LFS
migrate
before
and
one
we
execute
details,
SS
track
a
new
file
called
Jesus
tributes
will
be
created
in
the
repository
and
that
file
has
to
be
pushed
your
repository
as
well.
Okay,
in
the
file,
we
will
store
the
filters
used
or
the
filters
that
DDL
FS
will
use
to
track,
which
files
replace
four
exposures.
Okay
I
mean
it
doesn't
track.
If
the
file
is
indeed
a
binary
file,
I
mean
you
can
track
text
files.
A
If
you
want
it's
just
a
pattern
matching,
it
also
provides
file,
locking
capabilities
little
fast.
So
that
means
that
if
you
set
a
lock
on
a
file-
and
you
push
that
block
to
the
to
the
repository,
nobody
will
be
able
to
update
that
file
in
the
repository
until
you
remove
that
block
okay.
But
how
detail
first
act?
Is
this
process
through
big
hooks?
Okay,
it's
it's
really!
It's
really
simple!
If
we
go
to
this
repository.
A
A
Okay,
how
does
this
work
conceptually?
Okay?
First,
we
perform
the
gate,
push
that
is
tools
get
to
the
LFS
hook,
and
those
LFS
hooks
will
be
that
if
you
have
any
file
to
track
okay,
if
it
files
that
we
see
it
find
any
of
those
files,
it
will
generate
a
pond
with
that
information.
The
pond
where
it
will
also
replace
the
file
with
this
text
pointer.
So
these
pointers
will
go
to
the
repository
and
it
will
send
the
other
the
real
binary
files
to
to
the
Elector
server
and
in
a
hit
pool.
A
The
process
is
quite
similar.
We
perform
the
gate
pool,
then
the
repository
will
provide
the
pool
data
and
the
LFS
pointers.
This
information
will
go
to
the
elephants
hooks.
The
elephant
cooks
will
access
the
LFS
server
with
all
the
LFS
pointers
that
it
had
found,
and
then
the
leopard
server
will
return
the
by
ipod.
With
this
information,
we
will
return
the
pool
data
and
the
binary
files
to
the
user
by.
A
Re-Authentication
I
have
to
say
that
git
LFS,
although
I,
think
it's
almost
three
three
or
four
years
old
I,
don't
remember,
it
still
uses
HTTP
basic
authentication.
That
means
that,
for
security
reasons
we
should
use
HTTPS
or
use
SSH
okay.
But
where
do
these
residential
come
from?
It
can
come
from
the
git
remote
of
the
LFS
URL
I
mean
you
can
edit
the
config
file
of
the
repository
and
put
your
credentials
or
they
can
come
also
from
the
usual
get
reversal
keychain
and
if
the
remote
is
through
FSH,
the
process
is
a
little
bit
different.
A
Okay.
When
we
perform
the
push,
for
example,
git
LFS
will
connect
to
the
repository
through
SSH,
then
it
will
run
the
command
git
LFS
authenticate.
This
command
is
handled
by
github
cell.
In
our
case
then
get
that
cell
will
connect
to
the
internal
API
to
the
LFS
authenticate
end
point
and
that
end
point
will
return
a
talking.
That
token
has
an
expiration
time
inside
it,
and
with
that
token
we
will
return
a
response
with
that
response.
We'll
have
a
hitter
section
with
panoply
patience
either.
A
Basically,
what
it
does
is
checks
if
the
response,
the
request,
has
any
authorization
here.
It
will
decode
the
content
in
that
official
authorization,
giving
us
the
login
and
the
password
or
the
login
and
the
token
and
with
us
with
lo
with
that
information,
it
will
call
the
method
find
for
the
client
in
the
earth
class
that
will
iterate
over
each
of
the
authentication
methods
we
already
have
in
order
to
get
a
successful
response
for
those
credentials,
and
then
it
will
return.
This
very
short
class.
A
A
Get
information
about
electrics
files?
Okay,
it's
very
important:
the
concept
of
requests,
the
ability,
if
the
user
is
not
a
prize
of
course,
no
information
about
the
LFS
files
will
be
will
be
sent
if
everything
is
okay
in
the
response,
we
will
have
some
information
about
where
we
can
find
the
LFS
file.
Okay,
usually
that
that
that
URL,
the
action
to
access
that
the
RFS
file
is
done
in
a
different
endpoint,
not
through
this
one.
This
endpoint
is
used
for
both
uploading
under
loading
and.
A
A
Mandatory
with
this,
this
mimetype
okay,
this
is
how
the
request
to
the
to
the
former
api
look
like.
We
have
an
operation
field
and
it
can
only
be
upload
or
download.
We
have
transfer
field,
which
is
a
list
of
the
client
transfer,
adapters
and
only
basic,
is,
is
use
at
the
moment.
We
also
have
the
rest,
which
is
a
reference
in
the
repository
and
the
main,
the
main,
the
main
param
object,
which
is
an
array
of
NFS
objects
to
interact
with
okay.
So
in
this
example,
what
we
want
to
do
is
upload
sorry.
A
A
Okay,
then,
in
the
response
we
have
the
transfer
again,
which
is
the
same
turtle.
Adapter
I
need
to
the
same.
We
using
the
request
and
the
main
the
main
field,
the
the
parent
object,
easily
list
of
objects
with
information
they
all
ID
the
size
authenticated.
That
indicates
if
the
request
is
authenticated
or
not,
and
finally,
actions
in
actions.
We
see
which
operation
can
be
performed,
H
red,
which
is
the
URL
where
you
can
access
the
LFS
file.
A
We
have
the
shearers
which
it
will
usually
be
used
to
store
the
authorization
headers
and
finally,
the
expiring
expires
mean
or
expires
at
that
indicates
when
the
purpose
we
are
fire.
So
in
this
case,
what
we
are
going
to
do
or
what
this
response
says
is.
If
you
want
to
download
this,
the
very
first
file
with
the
OID
1
1,
1
1,
you
can
access
it
in
the
H
ref
HTTP
GET
lab
calm,
ok,.
B
A
A
A
Then
the
user
will
perform
a
get
request
to
the
URL
found
in
the
in
there
in
the
response,
and
that
will
go
to
the
LFS
storage
controller
again.
That
controller
will
perform
the
authentication
of
the
user,
and
if
it
is
authenticated,
it
will
send
a
response
with
the
heater
accent
file
and
the
content
of
that
file
will
be
the
pass
in
disk
of
the
LFS
file.
This
is
important
because
we
are
not
sending
any
binary
data
from
rails
today
to
get
LFS
okay,
we
only
send
a
request
with
that
fever.
A
A
Okay,
on
how
a
blood
work,
the
same
way
will
perform
the
washing
place,
the
the
LFS
API
controller.
We
authenticate
this
request.
It
will
return
response
with
an
H,
ref
URL
and
then
the
cookie
and
the
good
time
will
perform
a
put
request
to
that.
H
read
that
that
request
will
be
intercepted
by
workhorse
workhorse,
then
we'll
access
Rails
to
get
an
authorization
to
perform
that
upload.
If
everything
is
okay,
very
safe,
it's
okay
to
upload
that
file
workers
will
then
upload
the
file
and
save
it
to
disk
okay,
and
when
the
upload
has
finished.
A
Workers
will
send
a
request
again
to
rain.
Saying:
hey
I
have
finished
the
upload.
That
request
doesn't
have
any
binary
data
inside
the
file
has
already
been
saved
to
disk.
It'll
only
has
the
ID
and
the
size
and
basic
information
so
in
the
method,
verified,
verified,
finalized
of
the
LFS
storage
controller.
A
A
And
now
the
file
locking
API,
ok,
this
is
this
is
very
simple.
This
API
has
four
endpoints
the
first
one.
The
first
one
with
the
post
method
is
to
create
a
log,
and
this
is
important,
because
this
is
a
new
project.
Only
single
branch,
locking
issue
port.
That
means
that,
if
I
log
a
file
in
master,
it
doesn't
mean
that,
if
somebody
check
out
that
branch,
it
doesn't
mean
that
the
other
users
can
update
that
file.
Okay
in
that
branch.
A
A
Okay,
another
and
I
think
the
final
end
point,
and
it's
very
interesting,
is
the
verifying.
This
end
point
is
used
to
check
if
any
any
file
in
your
gate
pose
much
any
of
the
Lots
set
in
the
repository.
Okay,
usually
when
you
use
a
little
FS,
a
warning
is
raised,
saying
hey.
If
you
want
to
enable
this
functionality,
please
execute
this
command.
Okay,
so
you
have
to
enable
it.
A
The
response
returned
by
this
endpoint
is
pleaded
basically
into
two
fields.
Hours
and
begs
the
fill
hours
will
store
the
Lots
created
by
the
user.
That
makes
the
repel
and
there's
obviously
will
be
the
logs
owned
by
other
users.
So
when
you
perform
a
git
push,
if
any
of
the
file
matches
any
of
the
locks
of
the
user
in
the
field
hours,
the
hit
bush
will
succeed
and
after
the
git
pull
git
LFS
will
display
the
logs
involved
in
that
git
push.
In
order
to
tell
you
hey
in
this
gate,
push
these
logs
are
present.
B
A
So,
as
you
can
see
in
the
first
one
in
the
LFS
api,
we
have
the
batch.
We
have
this
one
are
deprecated
and
in
the
locking
we
have
the
LFA
sorry,
the
logs.
We
have
their
love
to
verify,
and
here
in
the
LFS
object
storage.
We
have
the
root
path
to
download
LFS
files
and
we
also
have
the
upload,
authorize
and
upload
finalized
used
in
get
pushes.
Okay
and
with
so
I
mean
this
method
will
be
called
by
workhorse.
A
And
here
is
where
we
handle
the
authentication
of
LFX
files,
specifically
here
here,
as
we
saw
in
the
in
the
in
the
slides,
we
call
Giga
off
and
the
method.
Fine
forget
client
and
we
pass
the
login
and
the
password,
and
we
want
to
use
for
that
authorization
and
the
result
would
be
a
result,
object
which
will
allow
or
not
this
this
operation.
A
A
A
A
A
The
same
goes
for
the
abduction
we
will
take.
We
will
iterate
over
each
of
the
objects,
then
call
the
upload
action,
and
in
this
upload
action
we
will
generate
the
proper
response
of
that
action.
Okay
again
with
the
with
the
same
URL
and
with
the
same
LFS
object.
Storage
notice
that
the
URL
in
this
case
is
the
URL
when
we
download
and
the
URL
when
we
want
to
upload,
is
different.
Okay
in
uploads,
we
usually
provide.
We
have
to
provide
the
size.
Why?
A
A
A
Okay,
first
of
all,
what
do
we
do?
What
do
we
have
to
do
when
somebody
reaches
a
controller,
authentication?
Okay,
the
same
way,
the
LFS
api
controller.
Did
this
class
is
a
true
class
of
the
gate,
httpclient
controllers,
so
it's
the
same
method.
Again,
it's
called
the
authenticated
user.
Who
can
then
it
call
the
good
luck
of
clear
with
that
of
class?
A
A
Okay,
this
is
a
concern
and
it
will
reach
this
file.
This
dislike
sorry
this
file
in
this
file.
We
will,
we
will
tell
rails,
to
call
the
friend
file
method
with
this
path.
Okay,
this
method
is
the
one
that
will
well
noted
man,
this
method,
okay,
but
they
fade
business
or
does
that
this
method
will
set
the
hitter
X
sent
file
in
the
response.
Okay,
so
here
we
are
not
sending
any
data,
any
data
to
the
get
client.
We
are
sending
the
path
of
the
binary
file
to
workers.
A
Okay,
now
we
have
two
methods
we
have
upload
authorize
and
we
have
uploads
finalized
in
upload.
Authorized
work
horse
is
going
east,
Elena's,
hey
somebody
want
to
upload
this
file.
What
do
we
do?
Can
I
do
it?
Can
I
authorize
it
and
then,
based
on
the
response,
we
will
allow
it
or
not,
and
once
the
upload
has
finished,
this
method
will
be
called
again.
It
will
be
called
by
our
request
sent
by
workhorse.
A
A
A
A
A
B
A
Finally,
let's
go
to
the
LFS
locks,
API
controller.
This
is
a
very
simple
controller.
We
have
the
basic
operations
we
have
created,
we
have
index
and
then
we
have
also
verified
and
a
lock.
They
created
used
to
create
the
lock
a
lot
to
remove
the
log
in
there
to
loose
them
I'm
verify
to
check
if
the
LFS,
if
they
log
exist
here,
you
can
see
that
the
result
of
this
finder
is
splitted
into
the
response
hours
and
days.
Okay,
and
that
response
here
is
what
we
saw
in
the
slides.
I
mean
this.
A
A
B
A
B
A
B
A
A
B
A
Yeah
it's
a
simplification.
Please
yeah
I
mean
it's,
it's
not
really
how
it
work
internally,
but
it's
a
simplification
concept,
a
concept
or
simplification.
How
it
works
I
mean
it
will
be
interesting.
I,
don't
know
if
you'll
be
interested
in
knowing
how
the
heat
LFS
client
specification
is
because
we
don't
have
to
deal
with
that.
Yet
at
grid
lab
I
mean
if
you
are
interested
I'm
happy
to
help
you
with
us.