►
From YouTube: 2020-11-17: Experiments with Git object offloading to object storage and partial clone
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
A
The
idea
is
to
use
partial
clone
to
basically
scrub
repositories
on
the
server
side
from
some
data
and
we're
mainly
thinking
about
large
blobs
right
now.
So
let's
say
someone
pushes
a
blob
of
100
megabytes.
We
store
it
on
this
right
now,
but
that
is
expensive.
If
we
could
upload
that
that
would
be
great
cool.
Let's
get
started.
B
A
A
Those
config
options
are
set
by
default
and,
funnily
enough,
I
just
noticed
that
for
http
they're
set
in
our
code
base
for
ssh
they're
set
in
rails,
not
great,
but
it
works.
I
guess
all
right
and
now
we're
gonna
clone
from
the
just
cloned
repository
and
we're
using
the
non-local.
So
it
doesn't
hard
link
on
disk
and
it
actually
closes
the
data
and
pretends
to.
A
A
A
A
So
it
is
a
ref
list
which
is
super
nice
as
a
tool.
It
has
25
flags
which
are
all
pretty
cool,
especially
if
you
combine
them
now.
We
just
list
all
the
objects
and
if
they're
missing,
we
print
them
and
then
we
pipe
it
to
pearl
just
for
the
regex
match.
If
it
starts
with
the
question
mark,
we
know
it's
missing,
then
we
pipe
it
to
work
out
dash
line,
so
23
lines
were
outputted,
so
we're
missing,
23
objects,
which
then
implies.
A
If
we
need
that
object,
we
need
to
go
to
the
server
as
server
to
get
those
objects.
If
we
do
a
checkout
where
we
are
missing
those
objects,
those
blobs
now
we're
going
to
validate
that
the
server
has
them
all.
The
server
now
is
a
remote
repo,
a
local
repository
on
disk,
which
we
cloned
from
with
the
no
local,
but
it
was
a
full
clone
from
the
gitlab.com
repositories.
A
A
A
C
A
Great
question:
it's
not
so
here
we
were
wait,
where's,
my
mouse
now
all
right
yeah
here,
so
we
clone
into
client
one
right.
So
what
happens?
First
is
when
you're
reading
objects,
we're
receiving
yada
yada
and
then
git
does
exactly
the
same
loop
again
with
two
objects,
and
these
two
objects
are
in
the
current
checkout.
A
So
first
it
tells
the
server
give
me
everything,
except
all
the
blobs,
larger
than
800
kilobytes
and
then
git
actually
needs
two
of
those
objects
and
goes
to
the
server
again
and
say:
hey.
You
know
what
those
two
shots
I
want
those
and
so
on
demand.
It's
actually
doing
a
git
fetch
with
two
shares.
This
is
the
second
fetch
right
here,
so
it
it
went
exactly
as
you
thought
it
went,
but
then
it
was
clever
enough
to
just
fetch
those
objects.
It
was
missed.
C
A
Yeah
the
second
one
is
done
because
git
wants
to
do
the
checkout
notices
that
it's
missing
blobs
and
then
doesn't
do
it
so
and
then
mrs
blobs,
and
then
does
do
it.
So
what
it
had
to
do
is
enumerate
the
objects.
It
wants
two
additional
objects
and
then
it's
going
to
fetch
those.
But
what
we
could
do
is.
A
A
A
A
A
A
And
then
we're
gonna
set
this
one
remote
as
a
promissor
remote,
and
the
promissory
in
this
context
means
if
I'm
missing
objects.
I
can
iterate
through
all
the
promissory
modes
and
one
of
those
I
can
just
ask
hey:
do
you
have
this
shot
and
this
is
used
to
differentiate
remotes
from
one
another?
So
let's
say
you
have
the
same
repository
on
multiple
servers
and
some
of
them
do
act
as
a
promising,
remote
and
the
other.
Don't
then,
this
is
still
compatible
throughout
all
those
remotes.
A
All
right
so
yeah
a
cat
file,
which
is
a
git
plumbing
feature.
It
just
shows
you
whatever
is
in
the
hash
and
the
dash
b
means
print,
and
this
is
one
of
those
big
files.
The
hash,
it's
the
big.
The
hash
is
actually
this
value
because
we
already
uploaded
this.
We
must
be
missing
it
and
we
pipe
it
through
less
because
it's
a
big
file
and
then
it's
gonna
take
a
while.
A
A
A
Okay,
but
the
point
of
this
cat
file
was
just
to
require
it
like
the
checkout
we
just
discussed
like
if
git
needs
something
and
it
doesn't
have
it.
It
goes
to
one
of
the
promisers
and
tries
to
obtain
this
object
is
missing,
so
it
just
did
that
it
went
to
one
of
the
promisers
and
the
promiser
returned
the
object,
so
the
missing
object
count
has
decreased
by
one.
B
A
A
A
For
now,
we'll
continue
with
the
update
hook,
so
this
command
I'm
just
going
to
execute
copies
a
shell
script
as
the
update
hook,
the
update
rook
runs
for
every
reference
update.
So
if
you
update
a
branch
from
one
shot
to
another,
this
happens
on
a
push,
but
also
with
no.
This
happens
on
a
push
period,
and
then
it
runs
the
update
hook
per
one
of
those
reps.
So
if
you
push
multiple
branches,
then
it
runs
the
update
hook
for
each
of
those
branches.
A
This
hook
well,
first
create
a
big
blob
and
then
I'll
explain
later
so
what
this
does?
It
creates
one
mega
one
megabytes
I
think
yeah
so
had
one
meg
with
count
of
one
megabyte
from
defu
random
and
redirect
standards
out
to
git
client,
one
new
random
data,
which
just
means
create
a
new
blob
in
git
client,
one
new,
random
data.
A
And
we
paste
that,
and
now
we
can
push
that
to
the
origin.
Remote
there
we
go
and
what's
interesting
now
is
that
there's
the
hook
is
actually
being
run,
so
large
push
attempted
with
a
file
named
and
the
size
is
over
the
one.
No,
this
is
like
the
one
megabyte
yeah,
never
mind,
and
this
is
the
upload
blob
output.
A
A
A
A
B
A
It
doesn't
make
sense
why
it
triggers
so
it's
fine.
I
think
I
could
reason
about
it
within
a
few
seconds.
So
if
I
could
do
that
in
a
few
seconds,
everyone
can
get
client
free-
and
I
guess
the
point
here
is
the
new
random
data
is
no
longer
in
the
checkout,
so
we
won't
fetch
it
again.
So
the
additional
fetch
will
fetch
two
objects,
not
three.
A
A
A
A
So
then
we
remove
a
whole
section
remote
at
origin.
This
is
a
quick
little
trick
that
you
do
it
from
through
config
and
not
through
remote,
and
then
some
zip
command
fun.
Little
story,
remote
is
actually
a
very
thin
wrapper.
The
remote
subcommand
is
a
very
thin
wrap
around
the
config
and
then
the
second
command
just
checks.
If
there's
anything
in
the
config,
which
is
has
the
text
origin
and
the
master
branch
still
points
to
the
origin
as
a
remote.
B
A
B
B
A
A
C
A
A
A
B
A
B
A
B
B
A
A
A
A
B
A
B
When,
when
you,
when
you
request
something
from
from
a
server-
and
here
we
we,
we
use
it
to
to
repack
not
to
send
stuff
anywhere.
So
so,
usually
when
you
repack,
you
don't
use
a
filter.
So
that's.
Why
that
I
guess
that's
why
they
disabled
the
filter
option
when
you
don't
choose
std
out,
but
it's
an
artificial.
A
B
B
Yeah
by
default,
when
d3
pack
uses
some
options
that
we
should
remove,
because
otherwise
it
it
doesn't
work
with
the
dash
filter.
So
that's
why
I
I
we
have
to
do
a
kind
of
manual
repack
by
using
pack
objects
or
self
and
then
by
moving
the
the
pack,
they
generated
the
pack
files
so
that
they
replaced
the
the
old
ones.
A
A
A
A
I
just
wanted
to
call
out
for
a
second
that
we
touched
a
promissor
pack
as
well.
So
if,
like
the
package,
didn't
change,
so
all
the
objects
are
in
this
pack
there's
an
index
on
this
pack
and
if
it's
missing
then
there's
an
empty
pack,
but
it
has
the
promissor
extension,
which
then
means
that
it's
the
problem,
it's
a
promised
object
and
yeah.
That's
why
you
might
see
that
file.
A
A
B
B
A
Oh,
that's
not
short,
this
is
short
and
we
just
how
many
are
with
a
six
three
one,
two,
three
okay.
This
is
rudimentary
check
summing,
but
I'm
just
gonna
claim
that
they're
equal,
so
it
might
be
25
or
26,
but
the
main
point
is
we
got
it
on
the
server
if
we're
missing
it
on
the
local
repository
and
just
to
show
that
we
can
fetch
one
of
those.
A
C
A
B
A
A
Yeah
and
the
advantage
of
that
would
be
that
we
have
them
local
if
there
is
a
load
coming
so
only
the
first
time,
it's
missing
and
then
the
rest
of
the
day.
We
have
it.
Let's
say
a
new
repository
is
very
popular
and
then
at
night
we
scrub
it,
and
then
we
try
to
go
without
having
that
foul
locally
again.
A
Because
wait
like
what
I'm
trying
to
prove
is
that
this
gitlab
git
is
it's
missing
objects.
We
just
shown
that
the
26
we
fetched
one.
So
it's
25
now,
but
this
repository
should
be
gitlab.com
like
in
a
few
iterations
of
backend
server
work,
but
for
the
customer
we
want
to
do
a
git
clone
and
then
dash.
No
local,
so
don't
reuse
any
objects
from
gitlab
git.
Oh
I
yeah.
A
Then
we
also
see
d,
and
we
just
run
this
one
liner
to
to
counter
misses
missing
objects,
because
then
it
would
be
fully
transparent
to
the
end
user
and
like
in
in
air
quotes.
The
only
thing
we
still
got
to
do
is
make
sure
that
hit
biff
and
all
those
server
actions
do
not
eagerly
fetch
all
the
big
blobs.
B
A
A
A
C
B
B
B
B
B
It's
because
this
script
is
named
git
remote,
test,
http,
git
and
and
git
when
it's
passed,
a
url
that
starts
with
stuff.
It
tries
to
see
if
there
is
a
git,
remove,
dash
stuff
split
in
the
path
and
and
it
uses
as
a
helper
to
access
the
remote
and
that's
what's
going
on
when
when
he
tries
to
fetch
from
the
from
optimizer
remote.
B
C
Great
cool
nice
yeah,
I'm
not
sure
if
this
is
something
we
want
to
answer
now
or
we
have
thought
about
at
this
point.
Is
it
because
now
we
have
like
one
one
web
server
that
lists
a
bunch
of
blobs?
Is
it
the
intention
to
like
group
those
blobs
and
subdirectories
for
each
repository,
or
is
it
like
becoming
like
considered
like
one
big
object,
storage,
which
has
blob
for
everyone
who
likes
to
store
blobs
on
my
http
server.
B
B
B
A
B
A
There's
one
giant
bucket
on
some
object,
storage
and
the
advantage
there
is
that
if
you
fork
a
repository-
and
we
already
have
this
one
giant
blob
on
the
bucket-
then
we
don't
upload
it
twice.
The
disadvantage
is
that
if
I
know
a
shower
of
a
blob
I'm
interested
in
then
I
can
pretend
that
I
have
this
shy
in
my
repository
push
it
and
pull
it
down,
because
I
now
have
access
to
it
and
given
sha1
is
shattered.
C
Yeah,
I
think
that
that
answer
that
it's
something
we
we
need
to
decide
on
how
we
want
to
do
that,
depending
on,
what's
what's
the
most
beneficial
according
to
yeah,
the
features
like
like
we
have
that
elevates
like
the
ups
and
down
sides
there.
There
are
like
arguments
for
both
both
ways
to
implement.