►
From YouTube: Infradev: ProcessCommitWorker memory optimisation
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
B
Yes,
I've
been
working
on
this
issue,
which
is
process,
commit
worker
memory,
allocation
problems
and
I
think
this
has
been
partially
fixed
by
some
changes.
The
eagle
put
in
a
few
weeks
back
tied
to
a
regex
as
in
it
just
doesn't
seem
as
bad
anymore
anyway,
but
it's
still
bad
and
one
of
the
so
I
started
looking
into
exactly
what
it
does
and
it's
tied
to
what
this
worker
gets
run.
Every
time
a
new
commit
is
added
so
on
Branch
push
or
anything
like
that.
B
A
new
job
for
this
is
is
pushed
up
and
it
runs
through
a
load
of
things.
So
I
did
a
bit
of
an
investigation
on
it
and
so
yeah.
You
can
see
the
numbers
now
up
at
the
top.
It's
like
45
million
45
billion
45
billion,
some
things
bytes
it's
in
bytes,
okay,
yes,
I
can
read
it's
very
small
on
my
screen,
so
it's
45
billion
bytes,
which
is
something
I'm
not
going
to
do
the
conversion
and
it's
now
only
14
billion.
So
that's
like
brilliant.
A
Miss
massive
but
sorry
Robert,
would
you
mind
just
just
stepping
back
and
just
so
it's
every
time
we
do
a
commit
or
we
push
a
commute
yeah
this
is
executed
and
what
does
what
does
it
what's
it
for?
What
does
it
do?
It
does.
B
Way
too
much
it
does
unless
I'll
go
so
far
quickly.
B
One
right
everything
on
Zoom
is
in
the
way
it's
kind
of
irritating
right,
so
process
commit
worker,
takes
a
project,
the
user,
that's
doing
the
thing,
the
commit
hash,
which
is
curious
as
well.
That's
like
a
hash
of
some
commit
data.
It's
not
like
a
commit
ID
that
sort
of
plays
into
a
bit
later.
It
then
scans
the
commit
message
using
band's
eye.
This
isn't
immediately
obvious.
It
doesn't
have
any
like
big
Banzai
references
here
and
then
you
can
ignore
my
change
here.
Basically,
what
it
does
is
it's.
B
It
goes
through
any
mentioned
issues
that
are
in
that
commit
message
and
closes
them.
It
also
does
some
stuff
tied
to
metrics.
It
creates
cross
references.
B
So
like
a
note,
if
that,
if
something
is
mentioned
in
the
commit
so
like,
if
you
mentioned
an
issue
or
you
mentioned
another,
merge
request,
I
believe
it
adds
a
little
bit
of
text
to
it,
so
it
does
sort
of
like
three
things.
B
It
does,
however,
have
a
few
issues,
one
of
them
being
that
as
far
as
I
can
tell
there's
no
limit
on
how
many
issues
you
can
tag
and
a
commit
message,
but
it
should
be
limited.
Can.
A
B
Yeah,
it's
looking
for
closes
that
uses
a
regular
expression.
That's
in
part
of
the
band's
eye
pipeline
the
problem
with
the
the
Banzai
power
plants.
My
first
thought
was
well.
This
looks
like
it's
just
loading
every
single
issue
into
memory
at
once,
and
you
could
potentially
mention
infinite
issues
for
closure
could
be
a
problem.
It
probably
is
a
problem,
but
I
think
that
the
band's
eye
pipeline
time
outs
will
probably
stop
that
from
happening.
If
there
was
too
many
mentioned,
and
also
this
isn't
a
rail
scope.
B
This
is
this
is
an
array
of
issues.
They
are
all
loaded
at
once.
So
there's
no
and
I
tried
to
put
my
first
attempt
to
fix
this
was
to
go
in
and
change
this
to
like
a
find
each
it's
an
absolute
nightmare.
It's
not
physically
possible
you'd
have
to
change
most
of
the
Banzai
pipeline,
so
I
gave
up
on
that.
So
my
new
solution
to
maybe
help
improve
this
worker
is
to
do
this,
which
is
to
close
the
issues
in
a
separate
worker
so
for
each
issue
wants
to
close
it
loads
into
a
separate
worker.
B
B
So
potentially
it
could
be
quite
quite
heavy
whilst
it's
a
little
bit
weird
is
that
there
is
already
a
dedicated
worker
for
issues
that
are
closed
by
a
merge
request,
but
not
one
for
closing
by
commit.
B
So
ultimately,
it
probably
makes
sense
for
them
to
be
kind
of
generic,
because
it
is
basically
the
same
job,
but
it
just
figures
out
which
ones
are
closed
slightly
differently
or
has
a
slightly
different
bit.
So
that
doesn't
change
at
some
point.
Does
the
other
worker
Belong
To
Source
Code,
also
no,
which
belongs
to
cloverview?
So
it
might
explain
why
we,
we
didn't
have
this
in
our
part
of
the
part
of
the
application,
so
yeah.
B
My
now
solution
after
a
reasonable
amount
of
looking
into
this
was
is
to
just
do
a
separate
issue
as
a
separate
worker.
So
that's
why
I've
been
working
on
so
I've
shifted
this
here
so
for
each
issue,
it
now
Q's
a
new
job.
To
close
the
issue
of
the
worker,
where
it
gets
bizarre,
is
that
this
collection
of
issues
isn't
a
collection
of
issues.
B
It
actually
passes
the
class
name
through
because
there
is
no
way
of
saying
like
instantiate
this
issue
or
this
external
issue,
the
ID
can
be
numeric
or
it
could
be
a
string.
It
could
be
for
any
external
service
like
jira,
it's
a
bit
weird.
So
if
there
is
a
better
way
of
doing
that,
I'd
like
to
know
because
I
haven't
figured
it
out
but
yeah
for
the
moment,
it's
got
a
really
rubbish
conditional
in
it
so
yeah.
It
basically
takes
the
issue
for
the
external
ones.
B
It
can
also
still
do
the
closing
side
of
it.
I
stole
this
logging
from
the
merge
request,
Closing
one
but
yeah
aside
from
that,
doesn't
really
do
anything
very
interesting.
It
takes
the
commit
hash
through
and
it
just
calls
the
service
in
the
same
way
that
it
was
previously
called.
Well,
not
quite
the
same
so
here
when
this
was
done.
B
So
that's
the
new
worker
there's
some
other
Oddities
I've
had
to
move
around
to
sort
of
get
it
kind
of
working,
but
the
actual
that
side
of
it
hasn't
been
particularly
awful
where
it
gets
kind
of
bad.
Is
that
the
Pipelines
it's
got
less
red
than
it
did
before,
but
where
it's
starting
to
get
quite
frustrating
is
that
I
have
14
failures
in
Branch,
Push
Service
spec,
because
it
has
14
separate
tests
that
expect
closing
of
an
issue
under
different
conditions.
B
So
my
so,
and
some
of
them
are
external,
as
you
can
see
so
like
it's
expecting
calls
to
external
Services,
all
this
sort
of
stuff
and
so
I'm
a
little
bit
conflicted,
because
I
could
just
delete
all
those
lines
of
code
and
say
yeah.
It's
expected
to
queue
a
job
which
I
mean,
would
kind
of
work
or
I
could
run
the
jobs
in
line
and
then
still
keep
these
same
conditionals
so
that
you
know
it
still
actually
checks
that
the
issues
get
closed.
B
B
A
B
It
just
feels
a
bit
weird,
but
I,
don't
think
it's
I
I
think
that's
technically
the
better
option,
because
at
least
you've
got
those
expectations.
They're
actually
like
checking
to
see
if
it
did
the
thing
that
it
was
expecting
it
to
do
so.
I
think
that
is
probably
what
I'm
just
going
to
do,
but
some
of
these
tests
are
a
little
bit
weird
as
well.
In
the
way
the
expectations
are
written,
so
I
just
need
to
go
through
them
and
paste
an
additional
lighting
I.
B
Also,
it's
been
so
long
since
I've
run
a
sidekick
queue
in
line,
but
I
I
can't
remember
how
to
do
it
so
I'm
looking
for
references
yesterday.
So
if
anyone's
done
that
recently,
you
know
this
has
to
yeah.
Send
me
a
link
because
I
haven't
found
it
again
yet.
B
But
I
think
he's
there
almost
I
think
that's
the
only
under
coverage,
my
new
most
hated
pipeline.
It's
part
of
our
spec
runs
now
alongside
RoboCop
I.
Still,
hate
rework
off
as
well,
but
I
think
that
it's
mostly
about
it
missing
logging.
A
A
This
one
yeah
I
mean
like
project.find
I,
think
if
the
project
ID
is
not
existing,
that
it
will
return
an
exception,
so,
okay
yeah,
you
might
be
right,
yeah,
you
can
just
change
it
find.
B
By
ID
yeah,
that
makes
sense
I'll
swap
that
over
I
think
I
think
because
I
copied
this
from
the
merger
quest,
one
I
was
like
I.
Don't
understand
why
they're
using
find
by
ID
I'm
just
going
to
use
find
instead.
So
that's
probably
where
the
change
happened,
got
it
yeah,
so
I
think
that's
kind
of
it
really.
If
that
has
anyone
got
any
questions
on
it
or
any
other
suggestions
on
how
to
lower
memory
usage
on
this
worker.
A
One,
oh
quick
note,
or
maybe
just
yeah,
just
a
couple
of
thoughts
regarding
the
issue,
type
issue
class
and
issue
ID
yeah.
It's
actually
reminded
me
the
problem
with
the
I
guess
it's
called
I'm,
not
the
problem,
but
the
solution.
It's
called
Global
ID
in
rails,
yeah
and
but
I'm
not
sure
whether
we
encourage
it
to
encourage
using
it
or
not.
A
I
send
send
a
link
to
the
chat
chat
and
it's
actually
have
an
example
that,
for
example,
before
we
had
to,
we
had
to
send
the
class
and
the
ID,
but
now
with
application
job
we
can
just
send
an
object
and
everything
works
great
yeah,
but
I
believe
we
don't
inherit
from
application
job
or.
B
Yeah
that'd
be
interesting
and
come
across
that
before
the
active
job
is
one
of
those
things
that
I
I
basically
haven't
used
anything
new.
That
rails
has
added
since,
like
version
three
I
think
because
they
went
through
a
period
of
just
adding
loads
of
stuff.
That
I
was
already
using
elsewhere,
like
replacing
sidekick,
replacing
carrier
wave
all
those
sorts
of
things
so
yeah
I've
not
come
across
that
but
yeah.
B
If
we
do
have
a
way
of
using
that,
that
would
be
quite
nice
to
make
it
a
bit
tidier
because
it
is
a
bit
and
I.
Actually
I
I,
don't
know
if
there's
another
option
like
I,
don't
maybe
there's
other
types
of
these
too
I'm,
not
really
sure
at
all,
and
it's
going
to
be
a
bit
of
a
pain
in
the
ass
to
test
this
as
well.
So.
A
Yeah
so
maybe
maybe
external
issues
not
to
prepare?
Oh
yeah
yeah.
B
External
issue
is
really
really
weird:
it's
it's
a
really
basic
class
that
just
says
like
oh
yeah,
it's
an
external
issue
and
it
has
this
ID
and
then
it
sort
of
gets
hooked
into
loads
of
things
to
like
say
where
it's
from
it's.
It's
a
little
bit
bizarre.
B
Yeah
I'll
have
a
look
at
that.
That'd
be
interesting
because
I
mean
this
and
I'm
fairly
certain
that
rubric
I
was
going
to
complain
about
my
case
statement
or
something,
as
it
usually
does
as
well,
so
I'll
yeah
I'll
probably
have
to
change
it
anyway.
A
About
the
Global
ID,
are
we
using
active
job
I
thought
we
were
using?
Just
just
like
you
know,
psychic
workers
without
using
the
rails,
active
job
functionality
or
application
job,
whatever
it's
called.
A
A
Oh
yeah,
yeah
I
sent
a
sent
a
link
to
the
code
to
the
best
practices
which
mentions
that
we
don't
we
don't
use
application,
application,
job
or
active
job,
but
at
the
same
time
there
are
some
entries
in
the
code
and
documentation
about
like
two
Global
ID
methods.
So
maybe
maybe
there's
a
way
to
use
it
yeah,
like
separately
without
active
job,
actually.
B
Yeah
that
makes
sense,
I
think
this
code,
like
this
changed
to
add
this
new
worker
was
of
interest
to
Heinrich
as
well,
by
the
way,
because
I
was
asking
some
questions
in
slack
and
I
think
someone
else
somewhere
else.
This
was
going
to
be
applicable
to
which
could
be
kind
of
curious,
basically
with
the
it's
kind
of
hard
to
tell
with
whether.
B
What
okay
yeah!
So
this
closed
service
does
a
load
of
work
well
issue,
which
is
why
it
kind
of
isn't
great
to
have
it
running
in
a
loop
I.
Don't
think
it's
going
to
make
a
huge
difference,
but
it
could
make
a
huge
difference.
It's
like
I
highly
doubt
anyone's
closing
50
issues
at
once,
but
if
they
do
close
50
issues
at
once,
then
this
could
cause
quite
a
problem
because
it
does
quite
a
lot
so
I
can
close
to
do's
and
stuff
as
well
like.
B
So
there's
quite
a
bit
and
I
wasn't
expecting
that
when
I
first
looked
at
either
but
yeah
in
theory,
it
should
be
better
running
it
in
its
own
worker.
Although
then
that
work
will
probably
have
its
own
memory
issues,
so
you
know.
Well,
that's
that's
a
problem
for
future
me.
You
know,
that's
that
guy's,
probably
he
can
start
off.
B
Yeah,
so
this
is
the
original
graph
these
are
taken
from.
B
Is
it
that
it's
Thanos?
Isn't
it
or
no,
the
kibana,
so
yeah
it's
from
Cabana
these
ones.
You
can
get
like
a
Thanos
link
to
it,
so
this
should
in
theory,
I
can't
even
see
the
butter.
Now
this
should
generate
the
current
state
of
it.
It
is
very
slow.
B
Which
which
either
could
just
be
that
there's
less
of
an
issue
with
it
yeah
I
mean
there
was
part
of
me
that
Sora
is
currently
14.
It's
like
right.
Cool
issue.
Closed!
Oh,
that's!
The
only
work,
so
it's
been
magically
solved,
but
I've
kind
of
felt,
like
the
issue.
Closing,
is
probably
a
bit
of
an
issue
anyway.
So
it
would
make
sense.
But
yeah
going
from
45
to
like
14
is
a
huge
change
and
it
could
well
be
from
this
thing.
B
That's
a
eagle
mentioned
before,
which
was
that
he
changed
one
of
the
regular
expressions.
A
Yeah,
but
it
was
only
merged
today,
I
guess
yeah
because
three
weeks
ago,
but
it
was
awaiting
the
security
release.
Okay,
it
was
yeah,
it
was
merged
today.
So,
let's
see,
maybe
this
one
helps.
Okay,
this
one
is
one
as
well.
Oh.
B
There
we
go
this,
the
Thanos
query,
doesn't
even
work.
Yeah
see
yes,
I
know,
I
have
never
had
any
success
with
that.
Usually
so
yeah
I
mean
potentially
that
I
have
a
an
effect
on
it.
B
I
think
it's
probably
just
down
to
if
it
is
tied
to
the
amount
of
issues
that
people
are
closing,
like
somebody
just
closing
50
issues,
then
the
it
will
just
fluctuate
week
by
week,
because
someone
just
might
not
be
doing
it
that
week
so
yeah
in
theory,
it
might
be
a
bit
better,
but
I
do
think
it
makes
sense
to
try
and
finish
the.
A
This
stuff
anyway
yeah,
it
might
even
be
I,
mean
you've
shown
us,
there's
so
much
happening
in
in
the
worker,
as
well
as
what
the
work
is
calling,
but
yeah
yeah,
like
maybe
a
first
step
to
to
breaking
it
into
multiple
sub
workers
and.
A
Big
effort,
obviously,
let's
find
that
quickly.
B
So
yeah
they
have
essentially
the
same
worker
in
fact,
is
nearly
identical.
The
only
thing
it's
got
that's
different
is
that
it
passes
the
merge
Quest
instead
of
a
commit,
but
it
is
marked
as
code
review,
so
ultimately
I
will
and
it's
also
got
a
low
urgency
as
compared
to
the
original
one,
which
was
high
urgency.
So
that's
a
little
bit
of
a
weird
difference,
but
ultimately
I
will
probably
just
delete
this
and
just
merge
into
the
other
one.
It
makes
sense
just
have
it
pass
in
a
merger
Quest
instead
of
commit.
B
You
know
that
sort
of
thing,
I,
think
that
makes
sense,
but
now
that
I
know
that
this
probably
has
code
coverage
for
these
logging
statements,
I'm
going
to
go
and
steal
that
next
as
well,
so
that
I
don't
have
to
write
that
feel
that
I'm
not
hardly
makes
sense.
A
A
Yes,
yeah
I
kind
of
like
one
question
suggestion
I,
don't
know
if
it
was,
if
you
tried
that,
but
sometimes
it
helps
to
find
the
exact,
let's
say
in
kibana,
so
we
have
a
record
that
consumes
like
14
gigabytes
of
data
now
so
in
some
cases
we
can
find
the
exact
file,
and
if
it's
like,
let's
say
if
it's
like
a
public
repository,
then
we
can
take
a
look
like
I
have
like
a
visual
representation
of
this
Comet
and
we
might
spot
something
inside
of
the
comment
message,
but
I
think
it
works
only
for
public
one.
B
Yeah
I
think,
if
I'd
seen
the
numbers
still
being
the
same,
I
would
probably
go
and
have
gone
and
done
that
if
I
can
get
the
same,
if
they've,
given
us
a
link
to
that
yeah,
there
is
a
link
to
that
see.
All
these
are.
B
The
current
numbers
are,
but
yeah
I
would
have
been
tempted
to
to
do
that
and
to
to
look
up
exactly
what's
going
on,
but
it
can
be
I
know
getting
access
if
it's
not
a
public
commits
if
it's
a
private
commit
yeah,
it's
gone
back
up
again
now
see
interesting
how
much
it
fluctuates.
B
Yeah,
the
only
problem
I've
found
lately
in
terms
of
tracking
down
stuff
on
production
is
the
getting
production.
Console
access
is
an
absolute
pain
in
the
ass
right.
Now
you
have
to
put
in
Access
requests
in
slack
I,
think,
and
you
have
to
hope
that
someone
from
Ops
sees
it
or
at
times
out
within
like
three
hours,
and
then
you
have
to
do
the
whole
thing
again.
B
So
yeah,
I
kind
of
just
preemptively
couldn't
be
bothered
with
that,
but
it
would
have
been
nice
to
actually
find
the
real
feel.
A
B
Much
I
was
talking
to
Veggie
bass;
it
might
even
be
less
than
that
he's
been
doing
it
for
what
does
he
work
on
the
like
the
customer
billing
sort
of
side
of
stuff
and
he's
had
he
had
like
an
entire
day
last
week,
where
he
just
kept
submitting
these
requests,
and
no
one
picked
them
up.
So
that
sounded
like
a
bit
of
effect
to
me,
but.
B
A
Sorry
I
do
have
one
final
question:
we've
got,
we've
got
like
three
minutes
or
something,
but
you
mentioned
that
it
takes
the
hash
rather
than
the
mid
ID
yeah.
B
It's
kind
of
interesting
this:
it's
because
it
commits
don't
exist
in
the
database
and
equally
because
they
get
read
off
of
off
of
disk
is
kind
of
slow
to
look
them
up,
so,
instead
of
them
having
just
an
ID
that
gets
passed
around,
there's
a
hash
and
it
it's
kind
of
yeah,
it's
a
ruby
hash
with
like
so
like
a
you
know,
like
a
dictionary
key
sort
of
thing,
it's
a
key
value
sort
of
thing
with
details
about
the
commit,
so
you
can
build
the
commit
from
those
details
rather
than
fetching
them
off
the
disk
again.
B
And
I
mean
you
can
look
at,
you
can
look
up
or
commit
from
the
repository,
but
it's
a
disk
read
as
far
as
I
know
pretty
much
every
time.
So
just
passing
around
the
hash
details
is
actually
quite
good
in
that
regard,
and
so
you
can
do
I
think
I
did
in
the
specs.
B
You
can
do
commit
to
hash,
and
then
you
can
restore
it
doing
build
from
sidekick
hash,
which
is
something
I've
half
moved
I
stole
this
out
of
the
this
worker,
because
it
has
this
thing
to
do
with
date.
Suffixing
so
and
then
commit
has
a
from
hash.
So
then
I
just
turned
this
into
it
from
I'd,
get
cache
and
then
shoved
it
into
the
commit
model.
B
So
it
basically
allows
you
to
build
the
objects
from
those
hashes
that
are
getting
passed
around
I'm,
not
entirely
certain
how
necessary
that
part
of
it
is,
but
it
it
was
there
before
so
yeah
I.
Imagine
it's
there
for
a
reason.
A
Okay,
thanks
Robert
I'll,
stop
the
recording,
now
sort
of
recording.