►
From YouTube: Using Strace to Understand GitLab - Part 4: Gitaly
Description
A walkthrough of straces of two Gitaly requests.
0:00 FindCommit - a typical Gitaly RPC
8:04 UserCommitFiles - complex Gitaly-Ruby RPC
12:08 Gitaly InfoRefs cache invalidation
15:26 GitalyRuby requests to main Gitaly process
20:10 Pre-receive hooks
28:55 Update hooks
34:00 Post-receive hooks
37:39 Regenerate InfoRefs cache
Part 1: https://www.youtube.com/watch?v=fcY5pPUq35U
Part 2: https://www.youtube.com/watch?v=tThs8QeP2qY
Part 3: https://www.youtube.com/watch?v=J-GkU7Mmqy4
A
Hi
everyone:
this
should
be
the
last
s
trace
video,
so
this
one
will
go
into
s,
tracing
giddily
and
understanding.
What's
going
on
there
all
right,
so
let's
go
ahead
and
get
started.
So
I
will
first
do
a
real,
quick
request
and
then
we'll
do
a
more
difficult
one
more
complex.
A
A
A
Gitly
instance
is
remote,
so
it's
not
on
the
app
server
it's
on
a
separate
host
which
is
connecting
or
communicating
over
tcp
with
the
the
actual
rails
node.
So
if
you
look
here,
that's
just
what
I
want
to
do
is
find
a
fine
commit,
because
that's
a
pretty
quick
and
easy
request.
Okay,
here
we
are
so
find
commit,
so
here
we're
reading
it
on
port
8075,
here's
our
remote
ip
address
and
so
like
we
saw
in
the
last
video
here-
are
the
basic
fields.
A
A
Here's
our
path
to
the
storage
product,
number
project-
name:
branch-
okay,
all
right!
So
we
read
that
over
the
wire.
Then
we
check
if
there's
anything
else,
we
have
nothing
comes
back
so
then
what
kindly
will
do
is
it
will
start
validating
that
these
directories
exist?
So
let's
say:
does
the
base
directory
exist?
What
about
the
objects
directory
and
then
we
check
if
the
refs
directory
does
the
head
exist
and
packed
refs
which
may
or
may
not
exist?
Typically?
Well,
okay!
A
So
that's
the
first
thing
we
always
do
whenever
an
rpc
comes
in.
Is
you
just
kind
of
validate
these
directories
that
exist
in
those
files,
but
because
skeletally
is
written
in
go
and
it's
go?
We'll
typically
have
an
eight
threads
per
process.
You're
going
to
see
what
happens
is
that
what
process
or
pid
in
s
trace
is
doing
the
work
jumps
around
a
lot,
and
so
it
can
be
hard
to
follow.
A
Okay,
all
right!
So
8623
is
our
pid,
so
you
can
see
that
it
does
that
it
creates
a
pipe
which
I
think
is
going
to
use
to
communicate
with
the
get
process
and
it
creates
an
e-pole
to
wait
on
something
I
guess
the
pipe
yeah
and
then
catch.
It
creates
a
few
more
pipes,
presumably
for
a
standard
or
another
one.
Next
to
me
for
standard
error,
I
guess
actually
creates
three
centered
in
out
an
error,
yeah
all
right.
That
makes
sense.
Okay,
then
down
here
it
gets
its
pin.
A
A
So
let's
actually
go
to
the
end
of
that
first
and
you
can
see
that
it
comes
back
with,
in
this
case
it's
creating
a
new
process
because
we
don't
have
the
clown
underscore
thread
flag
in
here.
Also,
this
clone
v4,
which
is
a
good
sign,
that's
a
new
process,
and
so
anyway,
so
here's
the
process
number
we
get
back.
You
can
actually
see
that
26
or
27634.
A
A
A
Like
I
mentioned
in
the
earlier
videos,
that's
a
good
way
to
tell
that
you've
got
a
new
process
coming
up,
and
here
it's
setting
up
its
standard
in
and
then
standard
out
and
then
standard
errors.
So
it's
taking
these
pipes
that
I
got
from
its
parent
process
and
using
them
to
create
its
input
and
output
pipes
and
then
at
that
point
it
then
execs
get
here's
the
git
directory
that
we're
using
and
we're
doing
a
cat
file
batch.
A
So
catfile
batch
will.
Let
you
pass
in
refs
and
you'll
get
back
the
actual
contents
of
that
of
that
file
or
blob.
A
Okay
or
commit
which
I
guess
find
commit
is
probably
going
to
return
a
commit.
Isn't
it
okay
and
then,
at
this
point
we're
doing
what
we
saw
in
like
get
video?
We
we
check
whether,
where
our
heap
ends
when
we
start
loading,
our
shared
libraries,
I'm
just
going
to
skip
past
all
that,
since
we
haven't
seen
all
this
in
detail
previously
literary
git
configs,
and
here
we
start
loading
in
the
get
directories.
A
Let's
get
past
all
that,
and
this
should
be
pretty
quick
just
because
basically,
it's
going
gonna
do
almost
no
work
here.
We're
just
gonna,
get
a
single
commit
back,
yeah,
okay,
so
here
we
write
out
that
we
gotta
commit,
and
then
here
are
the
details
of
the
commit
and
then
we
get
in
here's
another
commit
to
print,
and
so
we've
printed
out
details
of
that
as
well.
A
Yeah,
no,
okay,
actually
rewrite
that
out.
Let
me
write
out
this,
which
is
okay,
yeah,
all
right.
So
this
what
happens
here
is
so
you
write
out
this
output
that
goes
to
gitly.
Let's
actually
look
at
that
more
detail
here,
all
right,
so
we're
right
up
to
our
standard
out
here.
Here
is
the
sha,
and
here
are
the
details
and
then
8623,
which
somewhat
surprisingly,
was
the
process
that
forked
all
this
in
the
first
place,
is
the
one
that
picks
that
up.
So
it
reads
it
off.
A
You
know
you
can
see
it's
a
big
chunk
of
data
here
and
then
here
it
writes
out
okay
to
find
commit,
so
we've
actually
finished
at
that
point.
That's
a
little
interesting.
Actually
that
makes
sense
yeah.
Okay,
all
right!
So
then
what
are
we
actually
putting
in
the
body
of
the
response
here
yeah
here?
It
is
so
here's
the
body
of
the
response,
and
you
can
see
it's
just
encoded,
the
data
that
we
got
back
from
get.
So
that's
fine
comment,
that's
quick
and
easy!
A
Basically,
we
just
have
this
grpc
level
over
git,
so
it
comes
in
gateway,
fork's,
new
git
process.
Git
then,
does
exactly
what
you
saw
with
the
local
get
pipes
that
back
out
to
gita
leak.
Italy
then
does
some
minor
transforms
on
it
and
sends
it
back
over
the
wire
out
to
gitlab
so
not
too
complex
once
you
know
what
git
is
doing
and
pretty
easy
to
follow
all
right.
So
that
was
the
easy
one.
Let
me.
A
A
Okay,
great
all
right,
so
let's
s
trace
skittle
again.
A
A
Okay,
that's
done
now.
Let's
actually
take
a
look
now,
let's,
what
we'll
do
first
is.
A
We'll
okay,
here
we
are
okay,
so
just
to
give
you
a
quick
idea
of
the
structure
of
a
request
before
we
dig
into
the
details
in
s
trace.
A
So
this
is
a
gilded
request,
so
the
gitly,
normal
gitly
worker,
is
going
to
take
the
request
and
it's
going
to
write
over
a
socket
to
getalyruby,
which
is
a
ruby
process,
as
you
might
have
guessed,
which
is
multi-threaded
and
so
it'll
have
one
listener
thread
which
will
then
pass
it
on
to
a
worker
thread
and
then
we'll
kind
of
do
three
steps
we'll
run
our
pre-receive
hooks
go
to
the
actual
commit
and
then
we'll
do
the
post
receive
hooks.
A
So
you
can
kind
of
see
that
this
is
the
the
structure
of
the
of
how
it
works.
So
these
pin,
ids
or
pins
in
brackets
or
threads
ones
without
brackets,
are
full
processes.
So
this
is
the
one
four
four
four
five
is
the
cable
worker
that
gets
the
request.
It
then
launches
the
pre-receive
hooks,
which
does
some
work
and
then
roll
right
out
to.
A
A
Well,
we'll
see
that
we'll
go
find
that
when
it
gets
to
it
anyways,
that's
the
basic
the
basic
over
the
over
the
view
of
what
we're
going
to
do,
we're
going
to
run
pre-receive
hooks
we're
going
to
update
the
ref,
we're
going
to
do
post
receive
hooks
and
we're
going
to
see
the
same
basic
pattern,
particularly
as
previously
imposed.
You
see
folks
of
what
happened.
So,
let's
actually
dig
in
into
the
s
trace
itself
and
see
what
happens
here
all
right,
so
this
is
user.
Commit
file
was
the
rpc
recalled
all
right.
A
So
here
is
the
the
main
gitly
process.
Reading
that
off
the
wire
port
8075
user
commit
files,
we
can
probably
see
blah
blah
in
here
somewhere.
Maybe
no,
I
guess
not.
A
Here's
the
actual
details
of
the
push
okay,
so
first
thing
to
go
into
is
as
of
12.10,
italy
will
cache
the
refs
for
a
given
project
to
make
it
more
efficient
to
respond
to
http
requests,
and
so,
whenever
a
request
comes
in
that
modifies
the
the
refs
for
the
project
so
like
like
making
it
push,
you
recommend
we
need
to
update
that
cache.
A
The
first
thing
we
do
is
we
delete
the
latest
file
here,
so
we
use
unlink
at
to
delete
that,
and
you
can
see
it's
in
this-
the
basic
repository
directory
and
then
plus
gitly,
the
state
then
hashed
that
and
so
on
from
there.
So,
basically
we're
saying
the
the
the
cache
state
that
we
had
there
is
no
longer
valid
or
deleting
all
these
files
and
then
from
there
we
can
go
into
the
standard
new
f-stat
ad
that
we
saw
earlier.
Actually,
no,
this
is
still
for
the
cache
okay.
A
A
Okay,
yeah
all
right
now
here,
so
we
kind
of
created
this.
This
is
going
to
be
the
pending
refs
change,
so
we're
not
going
to
commit
that
until
after
everything's
done,
though,
so
it
just
kind
of
hangs
out
there
until
we're
done
with
everything
else.
Okay,
let
me
close
out
the
file
handle,
and
now
we
can
start
doing
the
new
fs
to
add
that
we
stop
at
the
start
of
the
find
commit
request.
So
we
validate
all
the
trees.
Are
there
and
then
here
we're
actually
writing
out
this
the
same
pit
yeah.
A
I
guess
we
do
that
after
we've
cleared
the
cache-
and
now
here
are
writing
of
our
unix
socket,
which
is
going
to
be
going
to
gitly
ruby,
and
you
can
see
down
here
for
whatever
reason
that
asterisk
doesn't
capture
what
the
path
of
that
file
is
from
the
sender.
But
you
can
see
down
here
that
this
is
in
fact
ruby1.socket.
So
this
is
gili.
Gilly
repeatworker1
is
getting
the
message
back
or
getting
the
message
over
that
socket
from
the
main
giveaway
go
process,
and
so
you
can
see
it's.
A
Add
new
file,
here's
the
server
details
and
so
on
same
basic
content
that
we
saw
above
all
right,
so
one
four
four
two
six
reset
in,
but
it
doesn't
actually
do
the
work.
It
just
just
reads
that
socket
and
then
we
pass
it
over
like
we
saw
in
the
parser
output
over
to
one
four
four,
four
five
and
there's
no
explicit
syscall
here,
because
it's
all
within
the
same
process.
A
So
you
just
kind
of
have
to
watch
for
what
process
or
what
what
pit
here
starts
getting
active
so
based
on
between
the
parser
and
just
seeing
1445
start
working.
We
know
that's
the
next
one
that
actually
can
work.
Okay,
so
1445
is
going
to
do
most
the
work,
I
believe
yeah.
So,
let's
just.
A
Narrow
that
down,
okay,
and
so
what's
the
first
thing
it
does.
It's
writing
this
out.
Oh
that's
for
any
poll.
Okay,
we
can
ignore
that.
A
Okay,
we
create
a
new
socket
and
then
we
are
connecting
the
port
80
75
to
gilly
itself.
So
basically,
what
happens
here?
This
is
a
really
common
point
of
failure.
Something
to
be
aware
of
is
that
this
gilly
ruby
request
in
particular
needs
to
call
into
the
main
ghillie
process
before
as
part
or
as
part
of
their
collecting
the
data
it
needs
to
actually
go
out
and
do
the
request.
So
the
first
thing
it
does
is
connect
to
the
main
guilty
process
using
the
address
set
in
the
rails,
server
config
right.
A
A
A
Okay,
so
we
we
are
trying
to
connect
over
udp
for
some
reason.
First,
that
fails
because
we're
not
listening
on
udp
and
we
try
again
with
ipv6
tcp
and
that
does
connect
eventually.
A
We
saw
most
of
this
pattern
in
the
the
puma
video,
so
I
won't
go
over
in
too
much
detail.
Let
me
send
a
message,
and
can
we
actually
see
the
rpc
here?
It's
not
obvious
where
the
rpc
is
okay.
Here
we
are
yeah,
so
it
has.
Local
branches
is
one,
and
if
you
look
at
the
search
code,
you
can
see
all
this,
but
frankly
the
gitli
resource
code
is
kind
of
convoluted.
A
Okay,
so
we've
done
that,
then
we
check
our
references.
We
just
we
create
a
new
connection
again
and
now
at
this
point,
we've
gotten
the
data
we
need
from
the
main
gitly
process
and
gitly
ruby
starts
opening
up
the
rebooted
itself,
so
it
starts
studying
the
the
get
get
depository
or
repository
path.
So,
instead
of
using
get
here,
we're
using
rugged,
which
is
ruby
bindings
to
libgit2,
which
is
a
c
library
that
basically
is
but
is
works
like
git,
but
it's
not
actually
get
proper.
A
We
use
both
gitlab,
but
you
can
see
it's
the
same
basic
process.
We
make
sure
the
directory
exists
and
read
the
configs.
It's
you
know.
Look
at
two
is
supposed
to
be
entirely
feature
impeachment
parody
with
regular
get
okay,
so
reader
configs
reading
our
pact
reps
here.
A
Okay,
well,
actually
we're
still
going
out
to
main
gilly
process
again.
Okay,
so
I'll
skip
over
most
of
this,
because
this
is
what
we've
seen
previously:
okay,
all
right.
So
we've
gotten
to
this
point
where
we
are
writing
out
a
temp
object.
So
here
we
have
our
temp
file
name
like
we
saw
earlier.
We
make
a
directory.
A
Fsync
here
to
make
sure
the
change
goes
through,
I
don't
think
we
saw
that
was
the
local
get.
That's
something
you
can
set
in
your
config
and
by
default.
Gitlab
does
turn
that
on
which
is
good,
helps
print
data
loss.
Okay,
then
we
rename
the
temp
object.
The
real
name,
we've
seen
all
this
before
all
right.
Let's
just
skip
ahead
a
bit,
because
this
is
a
pretty
long
request.
A
A
And
then
immediately
we
start
cloning.
Here
you
see
the
clone
thread,
field
or
flag
being
passed,
so
we
know
this
is
not
actually
a
new
process.
It's
a
thread
for
that
process,
all
right
and
then
we're
writing
out.
This
is
kelly
review
process.
Writing
to
the
the
prereceive
hooks,
here's,
the
old
ref,
a
new
ref
and
here's
the
ref
that
we're
changing.
A
A
A
A
A
Okay,
another
thread
here
we
are
opening
up
the
gilly
hooks
log.
A
A
A
A
Okay,
here
we
are
so
now
we
are
injecting
the
pre-receive
hooks
in
gitlab
shell,
just
as
a
reminder
so
yeah.
So
here's
one
four,
four,
four
five
here
we
launched
the
kiddily
hooks
which
in
turn
launches
the
gitlab
shell.
Actually
it's
pre-received,
let's
get
that
shell.
A
A
Yeah,
this
is
a
review
process,
so
we
can
see
it
loading
in
all
these.
All
these
shared
libraries.
Actually
we
haven't
gone
to
the
gems.
Yet,
okay.
A
A
That
is
normal
for
that
to
happen
when
a
library
is
being
loaded,
that's
just
the
way
ruby
works,
so
it
looks
bad
and
it's
it's
therapy
way,
but
it's
not
unexpected
with
ruby.
A
A
Okay,
here
we're
loading
in
the
cert
bundle
that
we
bundled
with
gitlab.
So
all
the
search
that
we
show
us
by
default,
we're
loading
openssl,
http
helpers,
all
right-
and
here
we
go
all
right.
So
at
this
point
the
pre-receive
hooks.
This
is
another
common
point
of
failure.
Is
they
make
a
call
out
to
the
internal
api,
the
internal
allowed
endpoint
to
say
this
is
what
I've
got?
A
Am
I
allowed
to
do
this,
and
then
we
have
to
wait
for
rails
to
get
that
request
and
respond
back
to
confirm
that
the
user
and
this
change
is
allowed.
So
in
this
case
we
get
the
response
back
pretty
quickly,
but
it's
not
unusual.
I
mean
I
mean
that
unusual
is
the
wrong
word,
but
it's
not
unheard
of
for
that
to
take
a
long
time
or
to
error
out.
So
that's
another
common
place
that
this
request
can
fail.
A
Okay,
but
in
this
case
it
works
and
let's
go
down
yep.
So
let
me
log
out
to
the
gitlab
shell
log
that
everything's.
A
A
Okay
yeah,
so
this
is
actually,
I
think,
brand
new
with
13.2.
We
do
this
in
and
go
this
used
to
be
on
ruby
things.
Keep
changing
all
right
so
here
that
although
pre-receive
hook
clusters
are
exiting-
and
you
can
see
that
the
the
ruby
worker
gets
the
sick
child,
that's
children
have
exited,
so
the
knight
wakes
up
and
starts
doing
the
next
job,
which
is
the
update
hook.
A
A
And
looks
like
28285
is
the
new.
A
A
Yeah,
so
then
we
write
out
sending
another
rpc
to
the
main
guilty
process
over
internal
sockets,
that's
kind
of
confusing
that
we're
both
using
the
external
and
internal
side
or
connections
for
kidly.
Here,
depending
what
we're
doing
that's
like,
I
said,
that's
new,
so
we're
seeing
this
for
the
first
time.
A
Okay,
so
you
read
off
the
socket
and
then
we're
gonna
see
this
time.
Let
the
hooks
and
go
do
the
task.
Okay,
so
14361,
actually
28294.
A
Great,
we
load
in
our
many
ruby
string,
error
files.
Let
me
see
the
same
things
last
time.
We
should
need
to
go
out.
The
network
though
come
on
there.
We
go
okay,
all
right,
so
we
didn't
get
like
a
shell
config.
A
A
Must
be
missing
something
here
anyway,
so
we
can
see
that
it
did
work
and
then
we're
actually
even
trying
to
kill
that
file
or
process,
even
though
it
already
exited.
A
Files,
okay,
well,
this
video
is
already
long.
So,
let's,
let's
move
on,
I
don't
want
to
take
all
day
all
right.
A
A
So
as
far
as
gets
concerned,
we've
actually
made
the
change.
So
now
we
just
need
to
run
our
poster
seat
folks,
and
so
you
can
see
once
again
one
four
four
five
picks
up
again
and
starts
doing
some
work
and
before
get
another
process,
and
here
we
have
posters
efforts
and
so
once
again
it'll
be
the
same
pattern.
A
We
get
the
base
name
for
that.
I
think
we're
going
to
exactly
that
here.
Yeah
here
we
are
all
right.
So
I'm
going
to
get
the
hooks
plus
receive.
A
A
Okay
and
there's
no
other
traffic
on
this
instance,
which
helps
keep
the
noise
down.
So
if
this
was
nest,
traced
from
a
production
instance
you'd
see
all
kinds
of
traffic
coming
through,
which
would
make
it
much
harder
to
to
follow.
What's
going
where
oh,
what
we're
going
to
call
the
internal
post
receives.
One
point:
aren't
we
okay,
let's
find
out
okay
here
we
are
yeah,
so
let
me
call
into
the
internal
api
for
the
third
time
this
in
this
call.
This
time
we
call
post
receive.
A
A
Exit
go
returns
back
to
kelly,
ruby
that
the
post
receive
hook
was
successful.
We
exit
our
many
processes
and
threads.
We
wake
up
the
ruby
worker
yet
again,
and
now
we
should
be
basically
done:
okay,
yeah,
let's,
let's
go
down
to
that.
A
Yeah
all
right,
so
we
wake
up,
and
then
we
write
out
to
the
review
one
socket
again
back
to
the
main
gitly
process
that
we're
done,
and
I
guess
this
is
our
new
rough
here.
Okay.
A
A
A
Now
we
need
to
update
our
cache.
So
that's
what's
going
to
happen
here
is
that
we
go
in
and
create
the
new
rest
file
right
to
it,
and
then
we
log
the
updated
cache
successfully
and
then
here
we
log,
the
actual
user
commits
file
itself
was
also
done
and
took
a
little
over
a
second,
that's
with
us
trace,
running
and
sell
everything
down.
A
And
then
we
log
something
more
about
the
disk
cache,
but
okay
yeah,
so
that
that
has
user
commit
files.
That's
one
of
the
most
complex
rpgs.
We
have
just
to
recap
so
to
get
italy,
then
the
gitly
ruby
gives
the
ruby
then
goes
to
gitly
again
several
times.
Then
it
allows
the
repo
itself
internally.
Then
it
launches
previously
folks
it's
going
to
go
back
to
gitaly,
which
then
go
out
to
the
rails
application.
A
Let's
come
back
to
get
the
ruby,
then
we'll
do
the
update,
and
then
we
do
the
post
receive
hooks,
and
then
we
go
out
to
rails
again,
yeah,
so
we're
just
all
over
the
place
and
any
any
at
any
point.
If
any
of
those
connections
break,
the
entire
thing
falls
over.
So
that
is
something
to
be
aware
of
okay,
so
that
is
gettingly
in
asteroids
all
right.
Hopefully,
this
was
useful
and
interesting.
Thanks
for
watching
take
it
easy.