►
From YouTube: Create Deep Dive #1: Repository Pull Mirroring
Description
In this Deep Dive session, Tiago Botelho, Backend Engineer on the Create team at GitLab, shares his knowledge of GitLab's Pull Repository Mirroring functionality.
Download the slides: https://gitlab.com/gitlab-org/create-team/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf
Learn more about Deep Dive knowledge sharing sessions: https://about.gitlab.com/handbook/communication/knowledge-sharing/#deep-dive-sessions
Find out when the next Create Deep Dive is taking place: https://gitlab.com/gitlab-org/create-team/issues/1
---
Read more about our product vision: http://bit.ly/2IyXDOX
Learn about FOSS & GitLab: http://bit.ly/2KegFjx
Get in touch with Sales: http://bit.ly/2IygR7z
A
Me
start
the
presentation,
so
hey
everyone
I'm,
my
name
is
Thiago
Thiago
I'm,
a
back
engineer
for
the
create
team
and
I
want
to
welcome
you
to
the
to
the
first
deep
dive
of
the
create
team
where
we
are
going
to
be
talking
about
the
pool
mirroring
system.
So
let's
jump
right
into
it.
So
the
purpose
of
this
talk
is
for
me
to
share
my
knowledge
of
the
pool
mirroring
feature
with
the
entire
it
lab
team
and
to
make
this
deep
dive
session.
A
reference
for
everyone.
A
A
We
will
not
talk
about,
though
we
will
not
talk
about
the
mirroring
through
SSH
push
mirroring
and
bi-directional,
mirroring
due
to
time
constraints
so
yeah.
So
what
is
full
mirror?
It's
a
feature
that
is
available
on
get
lab
starter
in
bronze
tiers
and,
basically,
what
it
does
is
it
automatically
pulls
the
changes
from
an
external
repository
into
a
project
in
bit
lab.
We
make
an
effort
to
keep
healthy,
mirror
synchronized
with
external
repository
at
least
every
30
minutes,
and
a
user
is
also
able
to
update
more
often
by
using
a
functionality.
A
You
know
we
call
update
now
which-
and
we
also
make
an
attempt
to
handle
from
unfriendly
scenarios,
gracefully
it's
very
useful
for
teams
that
have
a
canonical
version
of
their
coded
and
external
photo
service
and
want
to
have
a
secondary
version
hosted
on
either
get
lab
perform
or
their
own
instance.
So,
for
example,
users
can
have
their
code
hosted
on
an
external
code
hosting
service,
and
let's
say
that
they
want
to
leverage
our
CI
service.
They
can
set
up
a
full
mirror.
A
That
is
captain
Singh
and
it
runs
all
the
pipelines
that
were
configured
for
that
project,
which
is
a
really
common
use
case
for
this
feature.
So
let's
quickly
do
a
demo.
So
right
from
the
start,
we
have
a
really
basic
project
that
I
created
on
get
calm
and
I
have
a
bare-bones
project
that
I
created
on
my
own
instance,
and
so
we're
just
gonna
quickly
copy
the
HTTP
URL
from
this
project
and
go
to
settings
repository.
A
A
In
this
case,
we
want
to
synchronize
into
my
own
my
own,
both
of
insulins
towards
the
external
one,
so
he
extracted
who
the
password
the
authentication
method
don't
not
be
necessary
because
we
are
dealing
with
public
repositories,
people
privately
from
higher
education,
and
then
we
have
a
couple
of
other
objects,
mainly
the
mirror
user,
which
is
going
to
be
the
user.
That
won't
be
the
author
of
every
events
that
is
created
internally
and
there
also
options
such
as
over
lines
diverge
branches,
this
slate.
A
So
that's
when
you're
trying
to
mirror
and
both
versions-
don't
have
this
to
the
both
versions,
have
like
an
diverge
French.
It
will
overwrite
that
one
instead
of
just
painting
and
not
at
all
the
trigger
pipelines
we
already
mentioned,
but
it
what
it
does
and
the
only
more
protected
branches.
So
based
on
the
specific
set
of
rules
that
you
for
detective
branches,
you
can
only
meter
those
those
those
branches.
It's
really
useful,
for
instance,
for
something
that
we
call
a
bidirectional
theory.
Then
I
won't
get
to
their
influence
on
this
dog.
So
let's
go
ahead.
A
A
A
A
So
there
are
a
couple
of
factors
that
make
that
kind
of
make
is
all
work,
though
these
are
the
ones
like
the
capacity
that
musician
of
states,
state
management
and
determining
whether
your
update
will
be
attempted
again
and
I
want
to
share
it
as
well,
like
some
imagines
to
manage
even
scale
of
how
this
operates
on
the
Autobahn.
So
we
have
over
50,000
50,000
nerds
being
updated
within
the
last
30
minutes.
So
it's
it's
a
pretty
large
scale.
A
A
The
total
capacity
is
the
biggest
number
that
can
be
configured
by
by
any
bit
one
instance
administrator,
and
it
is
basically
limiting
the
size
of
the
cycle
so
that
we
don't
schedule
a
lot
of
projects
at
once.
It's
mostly
like
a
bottleneck
that
we
can
we
first
that
that
we,
if
you
want
to
this
world,
but
if
you
want
to
have
your.
B
A
A
That
will
that
will
ensure
that
psychic
always
has
jumped
talk
to
perform
and
go
in
essence
translate
into
more
detailed
updates.
It's
if
you
want.
On
the
other
hand,
if
you
want
to
have
less
frequent
updates-
and
you
don't
want-
you
don't
want
to
have
like
before
results,
all
the
time,
keep
it
lower
the
capacity
and
instead
of
having
that
the
bottleneck
on
the
actual
psyche,
concurrency
being
the
bottle
I
can
stand,
have
the
task
as
part
of
it,
so
the
next
one
will
be
spec
transition.
So
why
would.
A
Basically,
as
I
as
I
mentioned,
so
we
have
a
big
figure.
Computing
see
the
yellow
concurrency
for
psyche,
and
basically
there
is
this
psychic
queue
that
we
have
and
if
and
it's
basically
managed
by
the
capacity
that
we
have
if
we
make
it
lower
and
the
concurrency
is
higher.
It
means
that
if
they
will
pick
up
every
job
and
the
we
will
run
out
of
things
keeps
run
in
the
capacity
instead
of
just
having
always
having
like
project
IDs
ready
to
be
scheduled
and
anymore
moment's
notice.
So
that
makes
sense
they.
A
B
B
If
you
lower
the
capacity
to
the
exact
recurrence
you
sidekick
and
handle
at
a
time,
then,
once
a
new
spot
opened
up
and
sidekick,
we
first
have
to
wait
for
the
update
O'meara's
worker,
which
we'll
see
in
a
bit
to
come
alive
again
the
fact
that
there's
another
gap
in
capacity
schedule
another
job,
and
only
then
can
it
be
picked
up
by
sidekick,
which
means
we
lose
crash,
a
seconds
during
which
we
could
be
updating
another
mirror.
So
the
capacity
we
use
to
always
keep
that
queue
filled
up
enough.
B
A
So
now
now
made
acknowledge
it's
on
forget
about
state
officials
and
by
the
way,
I
didn't,
send
you
something
but
feel
free
to
interrupt
me
at
any
moment.
If
you
have
any
questions,
I'm
not
currently
checking
the
shop,
but
we
will
have
to
take
so
state
transitions,
so
the
way
nearing
works
is
divided
by
five
states.
We
have
the
gun
scheduled
in
started,
finish
and
fire,
and
each
of
them
are
responsible
for
performing
a
specific
task.
A
For
instance,
the
scheduled
state
will
be
responsible
which
actually
scheduling
the
worker
that
will
perform
the
updates
it
started
will
mark
in
the
database
the
point
where
it
actually
started
me
for
me.
Yes,
a
job
and
the
finish
the
finished
work
at
the
future
state
will
mark
the
time
where
the
mirror
Society
successfully
finished,
thank
20
and
sets
the
time
and
when
you
will
get
updated
again
and
the
same
for
the
file,
but
instead
it
will
mark
the
time
where
mirror
finish,
unsuccessful
disease.
A
This
is
also
really
useful
information
to
use
evidence
later
on
and
to
find
mirrors
that
are
inconsistent
sites
with
the
overall
system.
For
example,
a
mirror
dies
in
a
started
state,
but
that
doesn't
have
psychic
talk
running
back
so
there's
also
state
management
and
basically
the
mirroring
system
folk.
A
If
the
specific
job
has
finished
or
not
and
with
the
events
capacity
said
as
I
mentioned
previously,
we
are
able
to
know
if
a
project
is
more
toxic
but
waiting
for
be
updated
or
it's
already
being
mandated.
Of
course,
the
the
objective
is
to
make
the
system
as
self
healing
as
as
as
possible,
but
there
could
be
better.
There
are
some
scenarios
where
the
system
doesn't
recover
against
and
we
have
to
do
it
perform
an
operation
by
hand,
and
we
are
planning
on
doing
a
party
test
about
it
soon.
A
Another
thing
is
to
know
when
City
mirror
get
scheduled
and
from
this
we
develop
a
formula
that
basically
takes
into
account
the
the
time
that
includes
one
plate,
a
mirror,
the
last
time
and
the
number
of
retries
that
were
necessary
to
actually
get
access
successful
new
out
of
it.
So
we
can
even
move
little
more
detail
into
the
formula
we
currently
have
this
back
off,
Vivian
and
Jeter,
because
previously
this
is
a
bit
of
legacy
that
that's
currently
in
our
in
our
in
our
code
base.
A
But
the
objective
was
to
spread
the
load
a
bit
more,
but,
as
you
see
further
down
the
line,
you
have
the
minimum
delay
at
the
maximum
light.
It's
kind
of
found
the
timestamps
a
bit
the
the
main
thing
that
I
want
to
focus
if
the
retry
count,
which
thinks
it's
a
huge
factor
in
when
we
are
actually
going
to
update
the
mirror.
The
next.
A
Usually
the
mirrors
finish
the
finished
work
in
two
seconds,
so
it's
fast
in
this
case
we
are
going
to
have
30
minutes
30
minutes
Eli,
which
is
just
immediately
like
that
you
can
have,
and
as
I
mean,
we
want
to
penalize
mirrors
that
fail,
often
from
running
as
frequently
as
the
health
we
do
so
anymore.
We
always
want
to
prioritize
healthy
mirrors
again,
if
the
the
mirrors,
for
any
reason,
each
a
maximum
number
of
retries.
A
A
So
this
is,
this
is
a
basic
rundown
of
the
workflow,
so
the
scheduler
will
wake
up
every
every
minute.
Easter
is
a
cron
job
and
you'll
pick
up
all
the
mirrors
that
have
the
next
execution
do
for
an
update
so
previously
to
the
current
time
that
we
are
on
and
it
will
schedule
all
the
mirrors
until
there's
no
more
capacity
available
at
the
moment
and
after
each
mirroring
starts.
It
will
basically
fetch
all
the
changes
from
the
provided
from
all
tomorrow
and
updating
respective
branches
and
tags
new
information
after
updating
the
mirror.
A
Will
you
always
move
that
project
from
the
capacity
list
and
set
the
next
execution
time?
But
then
we
who
have
like
two
of
them,
you
can
either
finished
or
file
and
if
it
finishes
it
gives
a
bit
like
Downton,
basically
sets
it
to
zero
again
and
if
it
fails,
it
gets
implemented.
And
if
you
surpasses
a
certain
limit,
as
I
said
previously,
it
will
get
hard
fail.
A
D
A
Or
a
finished
error
that
will
circle
back
scheduled
state
gap
and
in
the
lifecycle
update,
all
new
exporters
will
get
reschedule
every
minutes
to
wake
up.
It
only
communicates
with
the
project's
database.
It
requests
the
mirrors
that
are
ready
to
be
updated
and
basically
schedules
the
project,
import
schedule
portal
for
every
every
single
new
earth
and
then
that
that's
that
schedule,
worker
well,
as
you
ever
even
see
in
the
total
bytes
equals
cancel
a
repository
apply
to
the
world.
It's
the
only
purpose
of
that
of
that
job,
and
then
you
have
a
new
report.
D
A
A
The
first
one
that
I
want
to
touch
on
is
the
project
import
States.
So
if
you
can
see,
firstly,
that
you
can
see
is
that
the
table
name
is
actually
called
project
earth,
and
this
is
also
a
bit
of
legacy
code
that
we
have
basically
this
feature
with
previously
only
world
for
years,
but
we
have
since
migrated
important
ports
as
well.
A
This
politic
and
every
project
will
have
a
project
reports
type,
and
we
see
why
we
decided
to
separate
the
newly
imported
parts
politic
into
their
own
model
was
because
the
the
table
the
projects
table
was
going
to
be
so
in
Wrightsville,
okay,
a
bit
too
expensive,
and
you
can
make
it
a
bit
more
like
like
this,
but
the
second
thing.
The
third
thing
that
I
want
to
show
you
in
this
PCB
file,
is
this
finishing.
Some
betweens
issued
from
the
Phoenix
on
file
like
schedule,
will
actually
perform
this.
This
method
all
add
import
on
that.
A
We
can
see
here
so
an
important
over
here
and
that
you
can
see
that
if
it's
an
import
and
the
repository
doesn't
exist,
they
will
actually
cancel
an
import
or
export,
but
otherwise
it
will
be
people
scans
of
the
repository
update
the
actual
job
that
performs
the
update
operation,
and
we
can
also
see
that
to
see
dr.
part,
don't
tell
you
that
the
super
basically
calls.
This
is,
fortunately
I,
don't
know
any
part
or
a
fourth
another
yeah.
So
this
is.
This
is
basically
what
it
does.
A
If
you
try
to
Tom
ID
and
we
update
the
current
project
adapter
by
detectable
ICP
track.
Make
sure
that
when
you
don't
finish,
if
you
get
to
clear
them,
otherwise
it
is
in
a
start.
It's
like
we
gotta
like
make
sure
that
the
Thomas
took
the
particle
so
going
on
to
the
doctor.
Part
of
the
project
import.
C
A
For
finished
state
and
implement
so
capacity
when
we
go
through
the
schedule,
touch
target
stage
and
again
from
the
started
to
file
or
finished
site
only
two
differences.
It's
the
increment,
which
I
count
as
opposed
to
the
reset
which
I
come
from.
We
finish
and
we
will
set
the
next
execution
times
thank
and
on
all
the
failed.
A
We
only
set
the
last
update
that
when
he
actually
finished
in
set
the
last
successful
not
play
that
as
well
mark
successful,
so
your
I,
oh
I,
would
also
like
to
show
you
the
actual
kinetic
that
sets
the
necks
of
you.
Thanks
Dan,
it's
also
a
nice
light,
but
you
can
see
here
we
actually
dismantle
all
thanks
to
Mike,
like
that,
as
the
formula
I
show
to
you
from
that's
thanks
them.
The
last
day
that
started
at
the
device
and
optimized
me
on
the
project
model.
A
That
is
really
important,
and
that,
basically,
is
the
core
of
the
of
the
entire
feature
needs
to
keep
like
mirroring,
also
enhance
the
logic
of
scheduling,
cron
jobs,
managing
the
capacity
and
checking
the
available
capacity
and
so
on.
So
you
can
see
here
that
we
are
configuring.
The
front
off
of
the
scheduled
employer
called
update
on
your
record,
and
it
runs
every
minute.
A
A
A
The
work
right
now,
okay,
so
the
repository
am
taking
your
worker,
its
it
basically
transitions,
the
project
from
the
scheduled
in
started
state
and
then
calls
the
update
music
service,
which
is
the
one
that
you
don't
fetch
to
the
mount
and
update
branches
and
attacks
from
the
external
depository.
At
the
end,
it
will
interface
or
file
and
we
handle
bones
in
those
in
these
methods,
so
we
either
finished
transition
to
the
village
state
or
markets
file.
This
method
actually.
A
A
Its
place,
it's
a
real
basic
service,
he
basically
just
fetches
the
mirrors
and
then
updates
it
back
to
them
and
there's
a
bit
of
before
logic
because
of
the
as
we
mentioned
back,
the
protected
branch
is
only
very
protective
functions
or
override
diverse
countries.
So
you
can
see
the
logic
there.
So
only
mirrors
protected
branches,
you
get
it.
It
defines
if
you
should
speak
the
branch
or
not
the
way
we
have
of
the
library.
A
The
other
really
important
order
that
we
have
is
it's
also
a
cron
job
that
is
formed
starting
ports
on
Porter
it
once
every
15
minutes,
don't
get
misled
by
this
beauty,
knowledge
50
hours.
This
is
just
to
as
a
mark
for
the
time
out
of
all
the
work,
events
are
dependent
of
its
instruction,
so
it
will
be
sure
that
every
here
job
will
time
out
as
15-hour
to
actually
attempting
to
update.
We
are
planning
on
on
on
decreasing
this
number
by
the
way
yeah
right.
A
D
A
D
C
A
I'm
looking
at,
for
instance,
like
the
project,
mirror
updates
overview
and
the
viewer
updates
at
least
10
seconds.
You
can
see
this
huge
mountain
of
projects
like
15,000
projects
that
are
not
getting
updated
at
the
right
time
and
you
can
also
see
that
they
go.
Incidentally,
if
the
project
report
schedule
worker
and
a
repository
affected,
I'm
not
performing
any
work
during
those
times,
or
at
least
very
meaning
employed.
So
in
this
specific
case,
but
if
what
happened
was
that
it's
not
important
work,
there
wasn't
time
laughs
and
that's
why
it
was
signing
out.
A
And
scheduled
storage
capacity
stays.
The
same
is
because
we
are
not
being
able
to
mark
or
spelled
projects
as
fire,
because
it's
talking
about
something
that's
mighty
back
so
there's
a
week
as
well.
If
you
want
to
check
in
every
point
detailed,
let's
go
find
graphs.
They
are
really
interesting
and
yeah,
so
for
troubleshooting.
A
A
You
can
take
a
lot
of
inspiration
from
that
model
or
practices.
You
can
combine
this
method
the
project
near
the
same
time
doc.
Now
thanks,
we
see
if
there
is
there's
actually
projects
to
build
the
capacity
and
for
some
reason
we
are
not
putting
that
capacity
or
something
like
that.
That
might
indicate
that
the
scheduler
is
not
working
correctly,
which
is
something
that
happens.
A
So
you
can
also
you
can
also
check
the
status
of
the
workers
for
each
mirror
in
the
schedule
of
solid
state,
as
I
mentioned
behold,
which
are
my
eating.
The
database
for
each
started
state
and
those
are
my
TV-
can
actually
Lucas
command
called
hit
five
psychics,
that
is
job
studies,
I
like
them,
and
possibly
Tom
IDs,
and
it
will
basically
tell
you
if
they
are
completed
or
not.
You
can
also
retrieve
the
project
ID
baggies,
currently
in
the
red
sent
by.
A
It's
useful
projects
that
are
technically
stuck
or
be
inconsistent
information,
for
instance
a
village
or
a
file
projects.
We
never
see
never
be
in
the
capacity
they
Somali.
So
if,
for
some
reason,
a
project
transition
profile
not
finished
state,
it
might
indicate
that
the
service
is
not
communicating
well
into
by
this.
A
So
for
clearing
beta
inconsistencies,
so
when
the
database
is
inconsistent,
something
the
whole
point
of
us
having
like
executed,
model
or
acknowledged
but
say
is
because
we
want
to
keep
it.
We
want
to
make
Excel
feel
right
and
we
wanted
to
keep
this
data
as
civilized
as
possible.
So
inferences
from
the
database
is
inconsistent
in
psyche
like
a
project
database
starting
to
divide
a
database,
but
psyche
already
faced
the
talk.
A
We
spoke
before
self
worker
or
you
can
schedule
some
people
talk,
forehead,
look
at
the
tall
I
eat
maintained
by
if
our
psychic
status,
friends-
and
usually
this
happens
when,
when
the
the
actual
uptake
worker
finds
out
or
for
any
reason,
you
can
also
have
the
capacity
set
being
inconsistent
in
the
database
and
Psyche.
The
princess
in
the
project
is
finished,
but
the
project
is
still
present
into
the
past,
except
it's
but
the
scenario
that
I
mentioned
previously
right
now
there
is
no
build
self
healing
by
zombies.
This
has
to
be
removed
by
hand.
A
While
you
moving
positive
ID
from
the
capacity
said
you
can
do
it.
You
can
refine
by
either
running
the
decrement
capacity
and
being
in
the
project,
ID
be
altered
if
it's
something
like
if
it's
something
that
you
see
and
it's
occupying
like
the
entire
capacity
might
be
worth
just
believing
the
whole
thing
and
starting
from
scratch.
A
A
So
now
let
me
show
you
what
a
healthy
living
system
looks
like
and
you
can
see.
That
is
a
huge
difference,
so
you
actually
see
that
project
in
points
cancel
worker
and
depository
at
like
they
are
insisting
people
for
the
jobs.
You
see
that
schedule
more
sorry
capacity.
Does
it
it's
not.
It's
not
sale.
So
it's
not
stuck
with
we
project.
A
Important,
perhaps
that
we
have,
which
are
the
project
new
updates
on
with
you
and
the
Newark
thanks
at
least
10
seconds.
We
are
really
healthy.
You
can
see
that
the
amount
of
projects
there
are
they
get
10
seconds
of
the
open
updates
from
the
actual
schedule
site
and
that's
what
schedule
times.
It's
me
so
yeah
these
are.
These
are
also
some
some
colleagues
that
I
valued
on
time,
so
that
the
communication
system,
safety
seemed
actually
perfect.
Architecture
doesn't
mean
you
smarty
pants.
The
whole
system
relies
on
state.
A
A
Coholic
effects
to
the
guidelines
and
there's
also
release
you
open
dynamically,
determine
mirror
update
interval
based
on
the
total
number.
So
we
want
to
get
feed
of
the
lower
bound
and
yep
about
1020
times,
and
instead
we
want
to
make
it
dynamically.
Based
on
information.
That's
system
has
there's
again
the
front
of
community.
That
has
a
lot
of
information
and
please,
if
you
have
any
questions
about
giving
good
free
to
ask
in
the
vit
channel
at
any
time,
are.
D
A
B
The
time
in
between
two
updates
premiere
go
down
to
not
overload
the
system
too
much,
but
we
didn't
have
time
to
implement
that
stuff
at
some
point
when
we
did
need
to
fix
the
dire
situation,
because
we
had
far
more
mirrors
and
you're
expecting
and
we're
not
hitting
any
kind
of
reasonable
update
frequency
loss.
So
we
kind
of
invented
that
hard-coded
30
minute
limit
and
we
still
have
been
kind
of
made
your
file
society
through
just
fiction,
dynamic
approach
we
had
originally
wanted
to
deal
so
right
now
is
hardcore
up
to
30
minutes.
B
But
the
idea
is
that
on
get
up
of
comma
30
minutes
is
pretty
much
the
idea
of
elements.
So,
even
if
we
install
the
dynamic
stuff,
it
would
land
on
the
30
minute
frequency,
but,
unlike
a
smaller
instance,
update
rate
would
be
much
higher.
You
would
still
want
some
kind
of
lower
bound
there.
So
she
knows
you
know,
try
to
update
the
repo
every
five
seconds
because
it
happens
to
be
the
only
mirror
in
the
system.
So
it's
like
you
can
handle
it.
Of
course,
that's
not
somewhere.
B
You
want
to
go
either,
but
the
idea
is
not
for
this
to
ever
be
configurable,
as
in
just
pick
a
number
and
get
Apple
to
try,
but
more
for
it
to
be.
You
know,
get
up
eluate
best
within
the
resource
units
because
ultimately
get
that
knows
and
has
the
information
available
to
it
to
figure
out
what
number
it
needs
to
be
updating
it,
but
it
doesn't
overload
and
but
still
abuses,
the
capacity
and
the
resources
to
fool.
B
Babe
I
think
you
have
I
mentioned
earlier
that
we
have
the
ability
of
you
know
someone
can
choose
to
host
their
project
somewhere
else,
but
then
use
get
that
just
for
CSV
and
we
can
find
the
Perl.
You
know
effectively
do
this.
Nobody
knows
so
you
can
call
it
line
upon.
That
will
immediately
trigger
an
update
which,
like
Java
mentioned
earlier
effectively,
is
setting
yet
next
execution
times
event
now,
which
means
that
you're
gonna
next
pick
of
the
optimum
years.
We
pick
it
up
immediately
and
wait.
B
Your
scepter
did
that
project
up
to
be
a
CI
CD
project
for
a
project
on
github,
for
example,
we
use
the
github
webhook,
which
will
be
called
every
time.
Someone
pushes
to
get
up
automatically
trigger
the
update
now
and
point
things
so
that
the
mirror
will
sink
immediately
and
the
pipeline
will
run
immediately
when
someone
pushes
to
get
help,
and
we
are
actually
smart
enough
to
also
report
that
I
kind
of
set
us
back
to
get
up.
B
D
A
A
So
people
stay
in
started
States,
but
if
you
will
still
hold
that
your
mighty,
but
the
project
already
it's
already
finished
and
what
we
do
to
kind
of
catch
these
scenarios
here,
it's
we
have
the
stock
reports
off
worker
inside
so
down
here,
but
we
have
this
this
job
for
air
that
basically
catches
all
the
projects
there
are
in
the
started
or
scheduled
states
and
grabs
they're
exhibiting
their
job
IDs
buying
from
week.
So
we
grab
some
IDs
for
those
projects
and
all.
A
So
we
find
the
job
IDs
that
are
already
complete
and
if
it's
complete,
that
means
that
we
have
a
project
that
should
be
marked
as
file
because
it
shouldn't
be
started
anymore.
It's
all
finished,
though,
at
some
point
there
wasn't
there
wasn't
the
communication
to
the
database
in
transition
deposit
back
state.
That
makes
sense.