►
Description
An introduction to the GitLab (file-based) Import/Export tool by George Koltsov.
A
A
So
I
guess
the
first
thing
that
we're
gonna
do
is
show
a
quick
demo
of
how
it
works.
So
I
have
a
project
page
here.
If
you
go
to
the
general
settings
under
advanced,
there's
going
to
be
an
option
to
export
the
project,
which
is
a
sidekick
background,
job
which
creates
a
tarball
and
set
and
makes
it
available
on
the
under
a
certain
url,
so
you
can
generate
a
new
export
or
download
it.
A
They
look
alike
so,
for
instance,
here's
eight
issues
here
and
eight
issues
here.
So
that's
a
quick
demo
and
the
same
applies
to
groups.
You
can
take
the
group.
For
example,
I
have
a
group
right
here.
You
can
go
to
settings
general
and
do
exactly
the
same
procedure,
but
you
get
an
idea,
not
gonna
show
it
in
full.
A
Underneath
this
feature,
import
export
is
used
essentially
what's
happening.
If
you
have
a
project
and
you
set
it
as
a
template,
then
when
you
create
a
new
project
that
is
based
off
that
template
underneath
it
exports
that
project
and
then
it
imports
it
under
a
new
destination.
So
you
have
like
a
a
template
project
to
work
with.
A
Projects
with
memory
optimization
in
mind,
you
know
so
that
our
memory
usage
doesn't
grow
because
before
we
we
had
a
project,
a
json
file.
That
was
one
massive
object
and,
if
you
imagine
that
contains
like
an
array
of
issues,
an
array
of
merge
requests,
an
array
of
milestones
whatever,
and
let's
say
you
have
a
project
that
has
ten
thousand
labels,
ten
thousand
issues,
ten
thousand
merge
requests.
A
All
of
that
is
one
massive
object
that
we
need
to
load
in
memory,
which
is
not
very
memory
efficient,
so
instead,
so
we
serialize
it
to
json
right
so,
and
you
can
imagine
it
being
something
like
this
project
that
issues
to
json
as
simple
as
that,
and
we
pass
a
list
of
options
which
sub
relations
to
include-
and
this
is
an
example.
This
is
not
an
accurate
representation
of
what's
included,
the
actual
list
of
included
relations
is
pretty
big
and
I'll.
Show
you
where
to
find
it
later
and.
A
On
the
import
side,
we
again
on
the
high
level
and
I'm
going
to
go
in
a
bit
more
into
detail
later
on
the
import
side.
We
read
those
files,
we
consume
them
line
by
line,
and
then
we
do
some
processing
on
the
line
that
we
read.
For
instance,
we
convert,
we
convert
the
json
object
into
the
activerecord
object,
including
the
subrelations
recursively.
So,
as
you
can
imagine,
you
have
a
an
issue
and
I
include
the
screenshot
here
to
kind
of
better
illustrate
what
I'm
trying
to
say.
A
A
So
from
the
example
above
right,
you
have
an
issue
and
that
issue
includes
notes.
You
know
it's
kind
of
hard
to
read,
but
you
you
cannot
find
where,
where
it
says
notes,
but
one
issue
can
have
notes.
One
note
can
have
award
emojis
associated
with
it.
You
know
so
every
object
when
it's
serialized,
it
can
be
quite
big
and
it's
it
has
nested
relations
in
it.
A
So
on
import
what's
happening
is
that
we
convert
that
string
recursively
and
what
we
identified
what's
included,
what
kind
of
relation
it
is
and
we
convert
them
into
objects.
So
if
you,
if,
let's
say
you
have
an
issue
with
10
nodes
in
it,
we
convert
an
issue
into
an
issue
object
and
every
node
is
converted
into
a
node
object
as
well.
A
A
And,
as
you
can
see
here,
every
node
has
an
author,
and
this
author
is
is
root.
The
importer
user,
in
this
case
I'll
touch
on
the
user
mapping
a
little
bit
later
as
well,
and
once
we
convert
all
of
the
subrelations
into
objects,
it
gets
persisted
and
you
know
we
move
on
to
the
next
to
the
next
object.
A
A
A
So
all
of
these
exporters
can
write
on
disk.
Then
all
of
that
gets
packaged
into
the
tarball
compressed
and
then
you
know
uploaded
for
you
to
download
so
where,
where
do
these
things
come
from?
How
do
we
know
what
to
export
and
what
to
import?
Well,
there
is
a
main
file
for
both
import
for
project
import,
export
and
for
group.
Import
export
is
the
import
export
yaml
file
and
the
this
file
defines
what
we
are
going
to
include
what
kind
of
relations
we
are
interested
in.
A
What
we
are
going
to
exclude
what
kind
of
attributes
we
want
to
see
in
the
in
the
exported
file
so,
for
example,
yeah
in
order
to
show
a
little
bit
more
detail,
we
have
an
import
export
project,
import,
export,
yama
file.
As
you
can
see,
it's
quite
big
for
groups,
it's
a
little
bit
smaller,
but
for
x
for
projects.
It's
it's
quite
it's
quite
big.
We
also
have
a
section
for
ee.
A
A
One
thing
on
debugging
like
there
are
two
ways
to
kind
of
observe
and
if
something
goes
wrong,
let's
say
something
is
missing
right.
You
import
a
project
and
one
issue
is
missing.
There
are
two
things
that
can
help
you
identify.
What
was
wrong.
One
thing
is
the
importer.log
file
under
the
development
repositor
development
under
the
local
developer
environments.
A
This
file
is
located
under
log,
slash
importer.log.
We
have
on
production
in
kibana.
I
think
it's
a
tag
that
you
need
to
put
like
importer
through
or
something
I
don't
remember
exactly,
but
it
is
possible
to
view
importer
only
records
on
the
on
the
I
think
it's
it
should
be
in
the
sidekick
index.
I
remember
there
was
a
catch.
A
That
some
of
the
logs
are
not
where
you
expect
them
to
be,
but
anyway
on
the
developer
environment,
it
should
be
under
importer.log
and
there's
also
employers
failures
database
table
that
tracks
all
of
the
failures
that
happened.
Let's
say
that
issue
that
failed
to
be
created
it
should
it
should
have
a
record
in
this
table
as
well
regarding
what
what
what
failed.
A
So,
for
example,
if
I
show
three
room
relation
three
restore
just
to
to
show
you
where
it's
happening
like
here's,
the
method
called
process
relation
item,
here's
where
we
call
save
on
the
relation
object
that
I
showed
you
previously
that
issue
that
doesn't
have
an
id.
So
we
just
rescue
an
exception
and
we
create
an
import
failure
record
in
the
database.
For
that.
A
A
A
Using
carrier
wave
gem,
you
know
that's
all
handled
for
us
and
then
we
remove
the
temp
directory.
One
thing
to
note
is
that,
before
you
know,
we
upload
it
to
object
storage,
we
need
to
write
somewhere
all
of
these
exporters.
They
need
to
write
their
data
somewhere
and
we
write
that
on
on
the
temp
directory
in
on
disk,
but
it's
not
a
temp
directory.
A
A
Especially
on
on.com,
it
can
indicate
that
this
is
an
nfs
mount,
but
import
export
does
not
have
nfs
dependency.
It
just
happens
to
be
writing
under
this
path.
I'm
not
too
sure
why,
but
this
is
more
of
a
legacy.
Historical
reason.
The
import
export
itself
is
is
a
single
process.
It
does
not
require
files
written
to
be
on
nfs
and
share
any
files
with
any
other
processes.
A
A
A
A
You
know
something
that
is
a
autodevops,
something
that
is
not
a
collection.
Sometimes
it's
just
a
single
single
object
of
data
and
there's
a
distinction
between
those-
and
you
know
that's
all
handled
here
and
then,
depending
on
on
type
of
data.
Here,
for
example,
we
have
a
serialized
menu
relations.
A
We
do
that
in
batches
we
preload
stuff
that
needs
to
be
preload
and
when
it
comes
to
individual
records.
You
know
right
here
is
where
it's
happening
record,
that
to
json
and
we
pass
in
the
options,
and
then
we
take
that
and
json
writer
does
the
writing
and
then
the
end
result
is
the
entity
sometime.
A
For
import,
it
gets
a
little
bit
more
difficult
because
there
are
a
lot
of
things
that
need
to
happen.
You
know
when
they
export
you
just
serialize
everything
to
to
json
and
that's
it
for
import.
There
are
a
few
more
things
that
need
to
happen,
so
I
guess
the
very
first
thing
that's
happening.
Is
this
importer.
A
This
is
similar
to
the
to
the
saver.
We
have
sorry
to
the
export
service
with
the
list
of
exporters.
The
importer
has
a
list
of
you
know
restorers
as
we
call
them,
so
you
import
the
file
and
you
call
each
of
those
importers.
You
know
you.
The
very
first
step
is
import
repository
then
v
key
then
project
three,
I
guess
the
the
project
three
is
going
to
be
my
main
focus
here.
A
So
here's
the
three
restorer.
This
is
the
one
that
is
getting
called,
and
the
three
restorer
is
the
entry
point.
Essentially.
Might
I
mean
nothing
not
entirely
true,
because
there's
there's
a
code
pass
leading
to
to
this
point,
but
that's
where
the
three
restorer
star
kicks
in
and
restores
data
from
from
the
ng
json
files.
A
A
I
guess
this
is
where
everything
starts
to
be
even
more
difficult
than
before,
because,
as
you
can
see,
even
the
method
signature
is
huge.
You
know
we
have
members
mapper
object,
builder
relation
factory
reader.
What
are
those
things
right
so
I'll
try
to
mention
what
they
are,
but
obviously
I
won't
be.
A
So
in
this
case
the
relation
three
restorer,
it's
actually
the
place
where
you
know
that
save
happens
on
the
transformed
object,
like
something
that
I
showed
you
previously.
You
know
like
the
relation
object.save
is
when
we
persist
it
in
the
database,
but
all
of
the
conversions
are
happening
in
this
class
too,
in
a
sense
that
we
traverse
data
and,
depending
on
the
type
of
data
that
we
identify,
whether
it's
you
know
if
it's.
A
A
A
A
It's
like
a
map,
for
example,
if
you
have
project.ci
pipelines,
you
know
we
replace
ci
pipelines
with
this
class.
We
we
map
what
what
is
the
ci
pipeline
and
some
of
the
relations
don't
require
this,
but
some
do
and
that's
the
reason
for
it
because
activerecord
doesn't
know
what
is
commit
author
right.
A
A
A
But
yeah
the
relation
factory
is
where
all
of
the
modifications
happen
to
the
hash
that
you
you
have,
for
example,
depending
on
the
relation
type
we
perform
certain
things.
You
know.
If
it's
a
note,
we
set
up
the
note
appropriately
if
it's
a
pipeline,
you
know
we
do
something
else
and
we
don't
have
a
lot
of
models,
as
you
can
see
here
that
require
such
modifications
most
of
the
time
the
exported
hash
is
enough.
A
A
We
take
the
relation
name,
you
know
we
and
we
convert
it.
We
convert
the
string
to
to
the
to
the
file.
Oh
sorry,
to
the
object.
I
guess
it's
happening
actually
here.
This
is
the
subclass
of
the
base.
Relation
factory
yeah,
there's
a
lot
of
a
lot
of
the
things
going
on,
but
we
replace
user
references
and
that's
responsible
for
a
responsibility
of
members,
mapper
and
I'll
touch
on
that
in
a
second.
A
A
A
So
there
is
a
process
for
that
as
well,
and
if
you're
curious
it's
attribute,
cleaner
is
the
one
responsible
for
making
sure
that
you
know
that
the
in
hash
is
clean.
All
the
references
of
ids
are
removed,
for
example,
here
this
a
list
of
prohibited
references,
everything
that
ends
on
underscore
id
or
ids
or
html
gets
removed,
except
for
a
few
things
that
are
allowed,
but
I
won't
go
through
them
yeah
now.
A
This
is
where
the
transformation
happens,
and
you
know
there
are
a
few
cases
where
we,
if
it's
a
unique
relation,
we
want
to
do
something
else.
If
it's
an
existing
object,
we
want
to
do
something
else.
So
there
are
a
few
things
that
we
we
do
here
to
accommodate
for
different
types
of
relations.
Right.
A
A
A
So
we
have
an
object
builder
and
it's
very
so
it's
somewhat
simple,
because
what
we
do
is
we
create
a
we
check.
We
take
a
list
of
attributes
and
we
take
the
class
and
you
know
on
a
high
level.
Is
this
right?
We
either
find
an
object
using
the
list
of
attributes
that
we
are
we
passed
or
you
know
we
create
a
new
one
and
for
project
object
builder.
A
I
think
there's
title
things
like
title
here:
title
description
and
created
that
and
project
id
or
group
id
so-
and
this
has
been
proven
enough
to
be
able
to
identify
existing
objects
instead
of
creating
duplicates.
So
if
there
are
two
issues
issue,
one
creates
the
label
issue.
Two:
oh
there's
a
label
with
title
a
description
b
and
created
at
this.
We
check.
Does
this
label
exist
and
already
yes
use
that
instead
don't
create
a
new
one
and
something
that
I
mentioned
before
in
the.
A
A
Obviously,
we
cannot
preserve
these.
We
need
to
replace
them
with
something
else.
A
user
with
id1
on
my
instance
is
root
on
gitlab.com,
it's
someone
else.
You
know,
I
cannot
just
upload
the
project
and
suddenly
said
authored
everything
I
have
right
so
on
any
occurrence
of
these
references
we
use,
we
replace
them
with
them.
A
A
Yeah,
every
exported
relation
has
the
some
sort
of
a
user
reference
and
we
use
the
public
eye
public
email
that
that
was
exported
from
user
or
the
email
for
older
exports.
It
has.
This
has
been
a
recent
change
to
use
public
email
only
and
not
use
the
email,
but
in
order
to
provide
backwards
compatibility
with
user
with
older
exports,
we
use
in
email
as
well.
A
A
We
use
a
verified,
you
know
primary
email,
so
it's
it's
not
a
secondary
or
unverified
or
whatever
it's.
It
should
match
the
source
instance
primary
email
address,
and
one
thing
to
note
this
whole
members
mapping
functionality
works
if
you're
an
admin.
If
importer
user
initiates
an
import
and
your
an
admin,
then
a
user
mapping
will
work.
A
A
So
yeah
there's
a
lot
going
on
in
import
export,
especially
in
import,
because
you
know
export
is
quite
quite
simple
procedure,
but
import.
You
have
a
lot
of
things
that
you
need
to
cover
and
relation
factory
is
quite
extensive,
there's
a
lot
to
read
and
to
understand,
and
it
can
be
quite
overwhelming
to
grasp
everything
right
and
but
yeah
and
that's.
I
hope
I
hope
this
was
helpful,
a
helpful
overview
of
what
import
export
does
and
what
are
these
classes
and
what
they
are
responsible
for.
A
Other
things
to
mention
yeah,
like
I
said
before,
all
of
the
above
exists
for
group.
Import
exports,
separate
files,
separate
importer
classes,
mostly
subclasses,
because
they
all
share
a
lot
of
functionality.
There's
an
import
export
api
that
you
can
use
to
initiate
it's
not
only
in
the
ui.
It's
also,
you
know
via
the
api,
and
another
kind
of
last
note
is
that
for
exports,
it's
possible,
especially
when
done
via
the
api.
A
You
can
pass
and
import
your
upload,
url
and
upload
params
in
order
to,
in
order
to
upload
the
the
turbo
to
the
provided
url.
For
example,
if
you
have
a
long
run
in
export
and
you
want
to
upload
it
directly
to
a
storage,
something
like
an
aws
s3
bucket,
you
know
you
can
provide
that
url
and
the
file
will
be
uploaded
there
in
a
way.
It's
like
an
hook
yeah
but
yeah.