►
From YouTube: 2020 11 18 Memory Team 2GB Sync
Description
1:00 Overview
- Agenda overview - https://docs.google.com/document/d/12-NOJCobK8wLShCv1Spx69l1VPc-UyvCl8QSBaGqbyM/edit#heading=h.j74stu9fgzjy
1:50 Run Puma in Single Mode recap
- https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/2#note_449139363
- Each worker takes about 150MB
10:50 Started to apply Puma 5.0.x
- https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/6#note_449815086
- Further research needed to understand the numbers
16:30 Discussed results of using running Omnibus with Spring
- https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/3
18:20 Discussed zswap
- https://gitlab.com/groups/gitlab-org/memory-team/-/epics/2#note_449804623
19:20 Tried running without unneeded services
- https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/8
21:00 Discussed sharing code across the whole codebase
- Our monolith is not structured to have a clear split of usage patterns
- Eager loading
24:55 Discuss switching to Omnibus with Ruby 2.7
- Check malloc/preallocation sizes
- Investigate ordering of .rb load for optimizations
32:30 Wrapped up looking into compaction
37:20 Discussed tooling (GDB) and visualization of memory allocation
38:50 Investigate Memory Content
- https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/4
45:20 String deduplication
49:30 Systemtap example
52:00 Heap analysis with different tools
1:04:20 Thoughts on the format of this discovery week
B
D
First
on
the
list,
let
me
quickly
go
through
my
points,
so
yesterday
I
was
continuing
with
boom
in
single
mod,
so
I
really
wanted
to
check
pss,
because
previously
I
was
only
checking
like
just
resilience
at
size,
so
I
found
that
I
found
a
single
mod.
At
least
a
needle
upgrade
is
better
and
for
me
the
difference
was
250
megabytes
quite
consistently
for
a
single.
C
D
A
D
D
A
D
A
D
A
D
A
So,
just
to
really
make
sure
we
understand
what
this
means.
This
is
a
completely
virtual
value.
This
has
like
no
basis
in
any
like
right,
real
like
so
so,
the
the
more
so
this
will
go
down
if
you,
if
you
spin
up,
if
you
have
like,
if
you
these,
are
two
workers
right.
So
if
you
double
the
number
of
workers
here,
this
will
have.
D
A
D
B
B
A
C
B
So
like
we
run
the
cluster
mode
because
of
the
cpu
of
like
much
better
cpu
utilization.
But
if
you
run
or
maybe
like
to
vcpu
instance,
maybe
it
doesn't
make
sense
to
run
in
the
clustered
mode.
Maybe
it
just
makes
sense
to
run
in
a
single
mode
and
kind
of
save
this
150
megabytes
out
of
each
worker,
basically,
that
you
have
to
run.
D
B
It's
it's
my
logic
as
well,
because,
like
I
think
like
that,
this
difference
between
pss
in
our
assassin's
single
mode
is
as
matthias
is
saying
like.
We
are
likely
serving
ruby
binary.
We
are
like
reserving
some
cert
libraries
and
like
they
were
used
the
same
memory
pages
basically
in
this
context,
so
I
think
this
is
why
we
see
this
difference
of
like
16
megabytes
in
the
second
one,
but
I
kind
of
agree
if
you
like,
we
should
sum
up
pss
and
compare
with
the
pss
from
the
second
one.
A
D
Yeah,
google,
chrome,
google
chrome
each
one,
this
one
yeah,
okay,
so
and
yeah.
I
wasn't
able
to
fix
metrics
for
a
single
mod.
I'm
not
sure
I
need
to
spend
some
time
to
to
fix
it,
but
it
would
allow
me
to
measure
everything
under
pressure
not
to
measure
to
chart
it.
But
I
I
bumped
a
couple
of
issues
yesterday.
D
D
B
D
Okay,
okay,
so
I
started
to
look
in
puma
50x
also,
so
I
I
built
the
latest
omnibus
image
from
the
latest
master
and
supplied
puma
bump
and
also
applied
rb.
How
can
you
ask
so
feel
free
to
use
it
for
me,
it
was
like
super
fast
to
install
and
super
stable.
Everything
worked
as
a
charm,
so
but
the
results
about
updated
boom
are
quite
confusing
also,
so
I
see
that
in
clustered
mod
there
is
a
slight
difference
in
which
is
updated.
Boom
acts
better,
but
in
a
single
mode
it
acts
kind
of
worse.
D
D
B
So
I
guess,
like
nikola,
work
on
the
snapshots
could
be
useful
because
maybe
we
could
like
get
some
sort
of
the
master,
get
a
snapshot
of
the
worker
after
okay
get
a
snapshot
of
the
after
processing.
Some
requests
get
a
snapshot
after
processing,
maybe
gpt,
to
understand
if
memory
is
retained
or
like
if
it's
not
retained
what
it's
being
changed
significantly
because,
like
I
think
that
puma
experimental
they
do
this
dancing
with
the
forking
from
worker
to
actually
reduce
this
idol.
B
Memorial
I'll
sorry
like
make
this
process
more
like
the
hot,
before
processing
any
requests
after
the
fork.
So
yeah
I'm
actually
curious
from
my
perspective
like
before,
do
we
actually
have
to
load
another
code?
We
have
to
change
the
state
of
the
objects,
and-
and
actually
this
is
also
something
that
I'm
kind
of
like
it's
interest
for
me,
because
maybe
this
would
really
answer
like
why
we
need
this
150
max
more
on
the
on
the
each
worker
after
processing.
B
D
Okay,
so
to
conclude
my
report,
I
think,
for
today
I
I
really
need
to
make
better
sense
of
like
these
numbers,
because
I'm
still
a
bit
confused
about
this
and
as
camille
suggested,
try
experimental
settings
and
see
what
we
what
we
could
do
with
them
and
yeah.
Maybe
I
will
connect
with
nikola
and
we'll
try
to
look
in
the
snapshots.
B
Can
you
can
you
see
my
screen
yeah?
So
I
think
I
kind
of
finished
with
the
spring
and
I
don't
really
want
to
pursue
that
further
like
it
seems
tempting
if
we
can
like
reload,
but
I
kind
of
expected
significantly
more
benefit,
and
actually
maybe
it's
like
connected
with
the
like
rxe
your
work
and
snapshot
investigation.
Maybe
we're
gonna
achieve
much
higher
efficiency,
because,
right
now
these
efficiencies
are
about
only
250
megabytes,
which
seems
like
pretty
low.
D
B
B
B
B
B
We
github
exporter,
because
this
ruby
process
is
critical
like
uses
about
50
megabytes
psychic
cluster,
because
it's
ruby,
it's
also
50
megabytes
grafana
is
low,
so
it's
smaller.
It's
probably
it
was
something
about.
I
think
10
or
20
megabytes,
but
like
for
these
two
processes
around
like
we
use
100
megabytes,
because
because
of
the
fact
that
they
are
written
in
the
ruby,
I
I'm
not
sure
like
exactly
why
they
use
so
much
memory,
but
we
also
run
them
in
the
context
of
the
bundle
and
github.
B
So
maybe
it's
the
problem
that
we
are
loading
to
manage
dependencies
as
well
and
like,
and
I
noticed
that,
like
we
actually
have
this
very
bad
pattern
in
our
code
base
that
we
share
the
code
base
across
all
the
tools.
Even
if
these
tools
don't
need
another
part
of
the
code
base.
So
like
I
noticed
that,
for
example,
in
the
race
context,
we
are
loading
deep
side,
key
cluster
tooling,
that
is
used
by
the
inside
cluster
like
exclusively
and
it's
never
used
by
the
by
the
rice.
B
So
I
guess
our
memories,
my
my
like
understanding.
Right
now.
Our
monolith
is
not
structured
at
all
to
be
like
to
have
that
like
clear,
split
of
the
like
usage
patterns
that
you
could
like
define
and
our
monolith
has
like
different
tools
that
are
always
like
being
served
from
the
single
like
bucket,
and
this
bracket
is
loaded
in
like
race.
Basically,.
B
B
I
did
not
measure
exact
memory
being
used,
but
I
saw
like
a
number
of
very
big
objects
allocated
for
handling
roots
and
happy,
for
example,
and
I'm
not
sure
if
this
is
something
that
we
could
somehow
optimize
or
not
to
kind
of
reduce
that
memory,
maybe
like
in
the
side
key.
Maybe
there
is
some
way
to
basically
not
load
these
routes
and
happy
content.
B
I
mean
roots
totally
with
the
controllers
and
the
rp
and
have
these
kind
of
paths
like
expansion
to
be
much
easier
in
much
easier
form
implemented
because,
like
I
did
not
measure,
but
I
would
kind
of
expect
that
these
roots
are
p,
maybe
inflating
side
key
roots
up
here
and
I
should
add,
also
graphql
in
the
psychobabics
inflating
these
processes
by
50
or
maybe
even
100
megabytes,
because
we
are
loading
that
and
like
like.
We
have
these
rules
for
the
expansion
pretty
extensive.
B
So
I
did
not
go
further
into
that,
but
I
will
probably
at
least
try
to
remove
github
matrix
instrumentation
and
see
like
if
it
makes
any
dent
in
the
memory
usage
on
the
pss
after
like
starting
this
puma
in
the
single
mod.
Without
any
request,
so
kind
of
like
idle
memory
and
kind
of
like
stop
that,
so
this
is
like
my
kind
of
aspect
that
I
looked
yesterday
what
I
kind
of
plan
to
do
today.
B
I
want
to
use
switch
to
omnibus
with
ruby
2.7,
because
I
want
to
actually
check
the
malloc
pr
location
size.
You,
your
materials,
like
look
so
yesterday
that
our
malloc
bytes
limit,
I
think
it's
configured
to
32
kilobytes.
Sorry,
32
megabytes,
if
I
remember,
but
there
is
also
this
interesting
about
the
free
pages
like
I
think
that
I
saw
the
the
pc
in
the
ruby
codebase.
Where
says
that
we
are
freeing
pages
as
soon
as
they
are
free.
B
But
I
from
what
I
saw,
it's
actually
seems
to
be
mostly
be
recycled,
but
the
the
memory
is
not
like
recycled
to
the
system.
So,
like
memory
pages
are
not
free
so
like
we
have
this
inflated
process
size
and
parts
of
that
is
like
because
of
the
ruby
heap,
but
parts
of
that,
it's
like,
maybe
it's
due
to
like
malloc
happening
elsewhere,
and
I
was
thinking
maybe
some
of
these
native
implementations
that
we
use
like
nokia
giri
grpc.
B
Maybe
they
have
some
like
memory
buffers
that
they
use.
Maybe
there
is
like
some
dc
finalizer
that
is
not
happening
and
like
maybe
it's
not
that
we
are
leaking
but
like
we
are
releasing
this
memorial,
maybe
never,
because
we
are
only
using
like
the
buffer
for
a
configured,
and
this
is
maybe
why
it
inflates
our
memory
over
time,
instead
of
like
keeping
this
kind
of
like
constant
at
some
level.
B
B
After
forking
sorry,
if
we
wrote
parts
of
the
application,
when
ruby
I
mean
between
a
moment
where
ruby
is
initialized,
master
is
deciding
to
fork
and
we
are
forking
to
serve
the
single
request,
because
I
I
want
to
check
if
I
can
somehow
force
the
ruby
to.
Basically
this
allow
a
loading
code
and
assume
that,
as
soon
as
everything
is
right
there,
we
are
no
longer
loading
the
application
part
of
the
application
so
like.
When
you
start
the
process.
B
This
process
has
like
all
parts
of
the
all
pieces
already
loaded,
so
we
are
not
further
loading
and
any
more
codes
after
forking.
So
I
want
to
actually
check
if,
if
it's
actually
happening,
because
I
don't
know,
I
want
to
check
if
the
ordering
of
the
rb
load,
I
want
to
check
if
the,
if
this
happens
after
the
forking.
B
E
B
A
D
Well,
camille
one
more
question
but
sir
mateusz,
but
you
mostly
restart
the
application,
the
warm-up
request
and
just
get
the
date
right
most
most
of
during
most
of
your
experiments.
You
are
checking
the
memory
on
idle
right
on
after
the
load,
loading
happens.
So
do
you
run
some
gpt
and
then
measure.
B
So,
okay,
like
so
I'm
actually
like
testing
that,
in
different
conditions
like
to
check
like
how
it's
basically
changing
over
time.
I'm
just
like
I'm
trying
to
find
like
the
anything
that
would,
let's
say
dynamically,
allocate
memory
and
retain
that
to
the
some
objects.
I
didn't
yet
find
it
and
like
after
my
investigation,
I
only
saw
the
guitar
matrix
instrumentation
to
be
one
but
like.
B
B
Showing
some
some
side
effects:
okay,
thanks!
It's
a
little
like
all
over
the
place
like
I'm
just
trying,
I'm
checking
like
the
next
ideas
that
pop
up
on
my
head.
So
I'm
saying
that
I
want
to
look
at
the
rb,
but
maybe
in
one
hour
I
figure
out
something
different.
I'm
gonna
be
testing
something
different,
because
that
appear
to
be
like
more
promising.
B
A
All
right,
let
me
do
my
update
quickly.
It's
gonna
be
super
quick,
so
I
wrapped
up
looking
to
compaction
yesterday
because
I
didn't
feel
like
this
was
going
anywhere.
I
don't
think
it
has
any
meaningful
impact
on
memory
use,
so
I
kind
of
ended
up
kind
of
drifting
a
little
bit
into
what
nikolaj
was
doing
as
well.
A
So
I
spent
some
time
trying
to
understand
like
where
memory
goes
that
we
allocate
so
like
one
thing
I
realized
was
that
I
looked
at
different
numbers
based
on
what
the
ruby
vm
reports
and
there's
different
ways
to
look
at
it
and
I
think
it
kind
of
adds
up
as
well
with
what
nikola
was
doing
with
the
tools
he
used.
It
looks
like
when
we
sum
up
the
memory.
A
C
A
B
C
A
I
don't
know
what
elsewhere
means
so
so,
where
I
got
stuck
yesterday
was
mostly
how-
and
I
think
I
need
help
with
this-
I
try
to
research
it
a
bit,
but
I
didn't
find
a
good
answer
like
how
can
we
look
into
like
a
raw
keep
down
like
what
is
the?
What
is
the
memory
used
by
the
process?
I
don't
care
about
what
the
ruby
vn
thing,
so
the
garbage
collector
sees
because
because
clearly,
that's
only
just
a
small
portion,
I
think,
like
we
should
figure
out.
A
A
A
Think
I
think
those
are
all
ruby
internal
well,
these
are
not
brilliant,
but
there's
like
eye
memos
and
stuff
which
are
all
these
ruby
internal
things
and
which
I
think
I
have
this
stuff
here
like
30,
I
memo
so
so.
This
is
like
the
ruby,
vm
right
and
then
there's
the
open
question
of
the
compliant
instruction
sequences.
I
think
it
could
make
sense
to
look
at
like
how
how
much
memory
goes
towards
that,
but,
like
my
main
question
would
be
to
you
as
well.
C
A
Go
back
and
ask
the
ruby
vm:
what
do
you
think
we're
consuming
in
terms
of
memory
but
like
what
is?
Are
we
really
consuming
in
terms
of
memory
from
the
at
the
operating
system
level
and
I'm
not
sure
what
the
best
way
is
to
go
about
this?
So
I
I
think
it
would
be
good
to
look
into
this
and
yeah
I'd
be
happy
to
pair
up
with
someone
on
this,
which
is
all
I
think
I've
said
down
here.
C
C
To
derail,
I
tried
different
kind
of
tools
and
I'll
explain
more
when
it's
my
turn,
because
it
really
depends
how
you
like
get
that
snapshot,
because
it
really
depends.
Also,
when
do
you
turn
trace
object
allocation?
So
if
you
do
it
on
the
boot
start,
it
will
give
you
more
information
about
allocated
space.
A
B
C
A
We
will
have
to
figure
out
how
to
use
these
tools
and
then
also
how
to
visualize
the
data
right,
because
the
binary
heap
dump
sure
like
you,
you
could
look
that
look
at
it
with
a
hacks
editor
or
something,
and
it
will
tell
you,
maybe
you
will
see.
Oh
there's
a
bunch
of
like
these
strings
or
whatever
that
we
like
allocate.
So
maybe
something
stands
out,
but
it
will
mostly
be
like
not
human
readable.
So
I
don't
know,
I
don't
know
if
it's
time
well
spent,
but
so
so
yeah
camilla
as.
A
All
right
here
it
is
like,
if
you
search
for
like
how
can
you
visualize
a
hypdum
like
all
the
content
you
find
on?
Google
is
for
how
to
do
java
so
because
there's
really
good
tools
for
java
to
look
at
like
java
heap
dumps
and
stuff,
but
there's
just
no
good
tools
for
ruby,
unfortunately,
that
the
good
tools
for
ruby
are
all
the
stuff
we're
already
using.
That
only
looks
at
the
ruby
heap
right
so
that
I
don't
think
I
mean
I'm
not
saying
this
will
not
get
us
anywhere.
I
think
there's
so.
A
A
Yeah,
I
think
this
is
only
what
the
the
ruby
heap
with
a
footnote,
because,
if
you
so
so,
this
is
what
the
object
space
module
tells
us
it's
allocated
and
I
think
it
depends
on
the
type
of
I
think
it
depends
on
the
type
of
object,
whether
that
includes
individually
mellowed
space
or
not
like.
If
you
look
at
how
mems
sizes
are
reported
for
strings
it
yeah.
A
So
again,
this
is
like
all
with
like
a
big
footnote,
because
I'm
not
an
expert
on
the
on
the
ruby
vm
but
like
it
looks
like
this
is
actually
the
full
string
length
in
memory.
So
it's
not
just
the
object.
Slot
of
you
know
40,
bytes
or
whatever,
and
I
think
the
same
is
true
for
arrays,
but
but
not
for
for
every
everything
so
anyway,
so
so
yeah
I
looked
at
what.
So,
how
can
we?
How
is
this
broken
down
currently
in
in
terms
of
why?
A
B
A
Ordering
by
size-
and
it
looks
like
50
megabytes
on
the
ruby
heap-
I
just
ruby
heat
plus
x-
I
guess
I
just
consumed
by
strings
so
50
megabytes
and
then
36
megabytes
by
classes-
and
this
is
what
I
was
just
trying
to
update-
is
I'm
trying
to
drill
down
a
bit
more
into
like
what?
What
is
what
is
class?
You
know,
then,.
B
So
I
guess
you
would
have
to
basically
try
and
play
with
the
class
and
make
like
a
different
amendments
to
this
class,
maybe
like
adding
some
ruby
methods
and
checking
like
if
the
mems
size
of
time
just
may
be
setting
distance
variables.
I
mean
the
class
variables
on
this
class
and
check
if
it's
also
trending.
A
C
A
A
Yeah,
it
doesn't
look
like
it.
It's
not
recursive
right.
So
if
you
ask
an
object
for
so
memsize
off
does
not
just
by
looking
at
this.
It's
not
recursive
in
the
sense
that
it
will
try
to
like
traverse
the
entire
object
tree,
and
then
you
know
like
count
them
all
towards
your
nested.
You
know
object
that
you're
asking
for
so
so
it's
flat
in
that
sense,
but
based
on
the
type
it
includes
different
things,
yeah.
B
B
B
C
C
C
A
C
A
B
So
I
guess,
like
it's
more
tricky
like
to
deduplicate,
because
he
would
have
to
find
I
mean
you
would
have
to
write
it
in
the
native,
but
I
guess
like
I'm.
This
is
this
is
why
I'm
asking
you
like,
because
if
this
is
50
megabytes
of
the
strings
that
we
have
now
and
if
it
says
that,
like
these
50
megabytes
are
frozen
strings,
maybe
can
we
really
kind
of
reduce
it?
To?
B
I
don't
know
five
megabytes
in
the
total
if
we
did
duplicate
the
these
frozen
strings
internally
I
mean
there
is
also
like
the
question
about
like
these
puffs
being
used:
ruby,
rpm
jams,
ruby,
gems,
italy.
This
is
like
also
being
stored
in
the
ruby
somewhere
yeah.
We
can
look
at
that.
We
can
look
at
that.
I'm
just
also
wondering
like
how
much
these
paths
really
also
consume
this
place.
D
B
Yeah
yeah,
I
mean
for
the
batteries
purposes,
so
I'm
just
kind
of
thinking
like
how
much
like
these
paths
also
consume,
because
we
have
a
lot
of
files,
basically
that
we
are
loading
as
well.
So
I
guess
like
like
optimizing
the
strings
you
search
probably
could
be
something
that
can
bring
us
pretty
nice,
like
memory
improvement.
A
A
It's
not
massive,
but
it's
per
well
outside
of
memory
sharing
considerations
it's
per
process
so
could
help
at
least
with
sidekick
as
well.
I
guess
so.
This
is
by
the
way
from
just
to
be
clear.
This
is
from,
I
took
all
my
measurements
in
the
same
spot,
which
was
just
before
just
after
master,
has
pre-loaded
the
application
and
just
before
we
fork
into
a
worker.
That's
where
I've
been
pulling
all
this
data
so
far.
B
A
Yeah
so
before
before
I
want
to
get
into,
I
want
to
make
sure
we
focus
on
the
right
thing.
I
think
22
megabytes
doesn't
sound
all
that
juicy
to
me.
I
think
what
I
want
to
focus
on
right
now
is
this.
This
is
like
crazy
right,
so
we
just
looked
at
the
rss
worker
rss.
Just
when
we
before
we
did
a
single
page
load
is
800
megabytes,
but
the
the
heat
size
is
only
134
megabytes,
so
I
think
we
should
understand,
and
you
know
deep,
duplicating
strings.
A
You
know
that
might
take
us
days
and
it
will
shape
of
20
megabytes.
It's
really
not
that
meaningful
like,
but
where
do
the
remaining
600
or
700
megabytes
go
here?
I
think
that's
what
we
should
really
be
focusing
on
like
at
least
understand
what
is
that
memory
even
be
be
used,
for
that
would
be
my
next
proposal,
which,
by
the
way,
is
a
nikola.
Maybe
we
can
talk
about
how
to
best
go
about
this
or
what
you
have
done
already,
but
so
one
tool
I
wanted
to
look
at.
Is
this
system
tab?
A
A
But
it
allows
you
to
it.
Has
like
this
little
this,
like
dsl,
that
you
can
write
probes
and
these
probes,
you
you
give
it
an
executable
and
you
can.
You
can
like
tag
methods
like
native
native
calls
that
the
ruby
vm
might
be
performing
to
hook
into
it,
and
then
you
can
inject
code
and
observe
what's
going
on,
so
you
can
then
do
so
in
this
tutorial.
It
does
stuff
like
counting
so
so.
A
This
is
like
kind
of
a
top
of
like,
like
you
know
how
to
like
top
like
the
process
top
tool
but
for
function
calls
in
ruby.
So
he
kind
of
wrote
this
just
with
this
step
tool,
so
you
can
see,
like
you
know,
for
for
this
process.
So
many
calls
happen
to
I
o
right
right.
So
so
maybe
we
can
hook
into
this
by
if
we
look
at
the
how
the
gc
works
and
it
has
a
allocation
function
right
for
every
object
that
is
being
requested
to
be
created.
A
So
maybe
we
can
write
a
system,
tab
probe
that
intercepts
these
and
we
see
exactly
from
where
these
come,
because
because
this
might
be
complementary
to
the
work
nicola
did
already,
because
so
far
we've
been
mostly
looking
at
breaking
down
or
what
memory
profiler
does
breakdown
by
file?
How
many
bytes
allocated,
but
not
really,
also
how?
How
often
we
do
that
and
also
not
yeah,
and
I'm
not
convinced
it
com
includes
everything
right,
because
clearly
the
total
sum
it
returns
to
us
is
much
more
smaller
than
the
actual
memory
we
consume.
A
A
C
Go
okay,
so
yesterday
I
also
struggled
with
analyzing
this
hip.
I
tried
different
gems.
I
tried
this
happy
jam
and
I
also
tried
this
hip
analyzer
tool
that
is
giving
us
a
good
visualization
like
the
percentage
of
objects
that
are
stored,
but
it
doesn't
give
us
like
it's
fancy.
It
looks
nice,
but
I
didn't
get
good
info
like
I,
I'm
still
struggling
with
the
same
question.
The
matthias
is
asking:
how
can
we
analyze
this
raw
heap
dumb?
C
C
I
couldn't
find
any
info,
and
I
was
also
struggling
with
taking
the
the
arbitrate
snapshot
from
the
production
virtual
machine
instance,
because
I
often
get
the
time
out
waiting
for
every
response.
I
tried
like
extending
the
timeout
it
works,
but
it
happens
that
the
actual
file
is
corrupted.
For
example,
last
few
lines
are
missing,
so
I'm
never
sure
if
the
snapshot
is
complete
or
is
not
so
sorry,
what
do
you
do
with
rb
trace?
What
what
is
it?
What
do
you
call.
C
C
C
A
C
A
C
C
To
run
those
derailed
benchmarks,
like
the
rail,
had
a
lot
of
tools
and
one
of
them
which
is
cool,
is
this
static
benchmark?
It's
analyzing.
The
memory
used
and
objects
created
on
required
time,
so
it
doesn't
boot
your
application.
It
just
requires
the
dependencies,
so
it
can
give
you
really
nice
information
about
like
each
of
the
gems
and
it
does
whole
memory
profiling
thing
around
dependencies.
So
it
gives
you
the
full
picture
before
the
application
is
actually
started.
C
C
C
This
one
will
give
you
actually
the
profiler
info
like
when
you
run
the
memory
profiler,
where
you
will
get
the
summary
of
all
allocated
and
retained
memory,
pair
file
per
class
per
location
and
also
summary
of
all
the
strings.
Interesting
thing
is
that
summary
of
the
retained
and
allocated
string
is
totally
different
than
when
we
run
the
the
static
one.
So
those
strings
contains,
as
camille
said,
some
parts
that
we
like
absolute
pads,
that
we
use
and
some
strings
that
are
pointing
to
some
google
classes
for
some
gems
and
etc,
etc.
C
And
it
has
this
cool
thing.
That's
called
mem
over
time,
which
is
like
hitting
your
application
and
showing
you
how
the
memory
increases
over
time.
So
you
can
see
like
how
much
memory
has
increased
over
time
and
maybe
to
detect
that
you
have
some
memory
leak
or
like
how
it's
behaving.
So
this
is
all
cool
to
run.
But
there
is
a
problem
I
couldn't.
C
C
C
So
I
tried
playing
around
and
like
setting
the
password
for
the
gitlab
user
on
postgre
and
updating
and
reconfiguring
configuration
and
some
different
things
that
I
found
on
internet.
But
I
failed
to
run
on
virto
instance.
I'm
not
sure,
if
is
it
worth
like
spending
more
time,
trying
to
run
it
on
like
production,
similar
environments,
since
it
will
give
us
a
lot
of
useful
information
or.
A
D
D
C
C
C
A
C
E
D
A
A
A
But
I
totally
agree
that
some,
like
I
also
felt
like
yesterday.
I
got
stuck
at
some
point
where
I
wasn't
sure
if
I'm
going
the
right
direction,
but
also
I
I
started
noticing
that
was
overlap
right.
I
started
noticing
that
I
was
looking
to
things
that
nicola
was
also
looking
into
and
it
only
occurred
to
me
like
an
hour
into
it.
That
oh
wait.
A
C
Sen
stands
out
and
we
can
like
feel
that
oh,
this
jam
is
huge.
Let's
remove
it
and
try
it
or
like
this
string
is
repeating
like
a
zillion
times
and
takes
like
100
megabytes.
Let's
try
to
see
what's
going
on
there,
but
I'm
not
sure
if
we
have
some
more
important
work
that
we
can
help
like
investigating.
C
Why
I
don't
know
instrumentation
is
there?
Should
we
remove
it?
If
we
remove
it?
Is
it
going
to
help
like
because,
like
that,
instrumentation
is
loaded
or
like
it's
overriding?
Every
object
that
we
have
in
maybe
that's
more
important
than
trying
derailed
and
again
reading
all
those
reports,
I'm
not
sure,
just
just
saying,
because
we.
A
I
think
I
see
what
you
mean
yeah.
I
also
had
a
feeling
that
so
yeah,
I
don't
know
it's
sometimes
difficult
to
to
say
to
decide
how
how
far?
How
deep
do
you
want
to
go
into
something
yeah,
because
some
of
the
stuff
is
super
time
intensive
and
then
you
then
you're
not
even
sure
if
it's
worth
it
like
this
whole
string,
stuff
right
finds
it's
like
there's.
A
That
maybe
I
would
be
okay
with
by
the
end
of
the
week,
even
if
we
didn't
have,
I
don't
think
we
should
focus
necessarily
on
like
having
all
the
solutions
by
the
end
of
the
week.
I
still
think
that
some
of
the
insights
we
get
out
of
it
are
going
to
be
useful
right.
I
think
we
really
need
to
understand
like
where
we
are
spending
memory,
and
then
we
can
make
a
decision
of
how.
How
much
is
that?
A
And
then
we
can
say
well,
this
is
really
chunky,
so
we
should
focus
on
that
right
and
then
in
like
come
next
week.
I
guess
we
can
then
go
back
and
look
at
these
results
and
that
are
hopefully
documented
by
then
and
then
then
pick
some
of
these
things
and
actually
try
to
try
to
improve
them.
So
I
I
don't
think
I'm
gonna
spend
time
this
week
to
actually
like
build
anything,
but
I
just
want
to
like
really
understand
like
where
all
where
our
memory
budget
is
kind
of
going.
B
I'm
kind
of
thinking
that,
like
there
is
like
limited
amount
of
the
big
things
that,
like
you,
can
optimize
and
have
like
very
big
amount
of
the
memory
saving
like
there
is
probably
like
one
of
the
outliners
and
like
we
saw
that
with
the
puma
single.
Maybe
there
is
like
another
one,
but
I'm
kind
of
thinking
that
really
like
the
the
actual
reduction
from
the
memory
usage.
We
come
from
many
small
improvements,
so
like
github
memory,
github
metric
instrumentation
is
like
it's.
B
It's
maybe
like
giving
us,
let's
say
like
half
a
mac
or
like
100
kilobytes,
but
if
we
find
many
of
these
small
improvements,
it
kind
of
adds
up.
Basically,
so
I'm
kind
of
like
thinking
that
any
improve
that
we
find
it's
good.
I
mean
like
if
it
takes
like
very
little
time
like
to
see
that
it's
even
better
because,
like
we
may
save
like
this,
I
don't
know
a
few
hundred
kilobytes.
It's
actually
a
few
hundred
kilobytes
left
that
we
that
we
have
so
it's
like
a
meaningful
improvement.
B
If
we
find
10
of
such
it's
kind
of
then
becoming
a
few
megabytes
already.
So
I'm
kind
of
like
looking
at
thinking
that,
like
any
meaningful
improvement
that
we
can
find
that
reduces
it's
actually
beneficial,
even
if
this
is
super
small,
if
it
takes
us
very
little
time
to
ship
that,
like
it's
even
better
because
it's
gonna,
I'm
it's
gonna,
basically
like
add
up
over
time,
so
I'm
kind
of
like
in
between
because,
like
I'm
with
matias
on
understanding
where
this
memory
is
like
going
to.
B
But
I
think
the
direction
is
really
like
on
understanding
what
aspects
we
can
fix
quickly
easily,
basically
as
well.
So
maybe
this
strings
like
it's,
it's
a
it's
something
maybe
like
20
megabytes
to
save
it's
pretty
big
number
like,
if
you
add
to
the
two
processes,
maybe
matrix
instruments.
Instrumentation
is
like
something
that
gonna
save,
maybe
like
from
that
investigation
right.
Yes,
you're
gonna
find
another
aspect
that
we
can
save.
B
So
nikola,
I
guess
like
there
is
no
easy
answer
to
your
question
like
it's
really
up
to
you
like.
What
do
you
feel?
Is
it
like
the
wharf
investigate
folder,
or
is
it
not
worth
if
you,
if
you
consider
that
this
is
like
it
takes
a
lot
of
time
and
like
you,
you
don't
really
have
like
a
good
idea
what
to
do
next?
B
Basically
and
like
do
it
like
in
your
way
like
pick
the
topic
that
is
like
interesting
to
you
and
and
shows
like
what
what
is
like
the
most
promising
or
like
what
is
the
something
that
you
would
not
investigate.
Otherwise,
I
don't
know
it's
it's
fully
flexible.
That's
that's!
The
whole
point
like
we
meet
every
day
to
like
to
share
our
like
experiences
working
with
different
topics,
basically
like
harming
maybe
even
the
same
topic,
but
from
the
different
perspective
and
different
angle
as
well.