►
From YouTube: Are Your V8 Garbage Collection Logs Speaking To You? by Joyee Cheung, Alinode(Alibaba)
Description
Are Your V8 Garbage Collection Logs Speaking To You? - Joyee Cheung, Alinode(Alibaba)
In this talk, Joyee will talk about alinode's experiences in analyzing the V8 garbage collection logs and diagnosing performance problems caused by V8 GC pauses and memory leaks.
About Joyee Cheung
I'm a developer at Alibaba Cloud working on alinode, an application performance management solution for Node.js. I also provide consultations for our clients, both inside and outside the Aliababa Group, to optimize their Node.js applications.
A
Good
afternoon,
everyone,
my
name
is
Joey-
are
in
a
developer
from
Ali
Papa
and
today
I'm
going
to
talk
about
VA,
garbage
collection
locks.
So
a
little
bit
of
background.
Why
are
we
looking
into
these
locks?
Well,
for
starters,
Alibaba
is
using
nodejs,
it's
probably
the
company,
with
most
no
J's
developers
in
China.
We
use
node.js,
mostly
cube.
A
U
application
servers
on
top
of
our
infrastructures
at
the
moment,
node.js
is
running
on
thousands
of
machines
in
our
clusters
and
you
can
find
it
in
almost
every
business
unit
of
Alibaba,
but
in
our
company,
Java
is
still
the
mainstream
error
for
the
application
servers.
We
chose
not
just
for
productivity,
but
we
can't
be
confident
about
this
choice
unless
we
can
be
sure
about
the
availability
of
our
node.js
applications.
A
So
one
thing
different
about
optimizing
javascript
in
the
cert
and
in
the
browser
is
how
much
more
crucial
garbage
garbage
collection
can
be
because
server-side
applications
tend
to
run
longer
and
handle
more
data
than
average
client-side
applications
and
garbage
collections
can
eat
up
the
CPU,
making
your
application
less
responsive
or
they
can
hold
on
to
your
memory
affecting
other
processes
on
the
same
machine
when
misled
by
developers.
They
can
lead
to
memory
leaks
and
even
crash
your
processes.
A
That
is
why
most
Java
developers
in
Iran
new
LEDs,
a
few
things
or
two
about
jvn
and
its
garbage
collection,
but
our
knowledge,
as
developers
usually
come
from
a
phone
and
background
where
a
garbage
collection
is
less
of
a
big
deal
ders.
So
we
still
need
to
catch
up
on
that,
and
that
is
why
we
started
to
look
into
the
garbage
attraction
of
v8.
If
you
are
interested
in
the
story
of
no
GS
at
Alibaba,
you
can
check
out
this
interview.
A
So
what
choice
do
we
have
if
we
want
to
take
on
performance
problems
caused
by
VA
garbage
collections
while
most
tears?
We
use
come
from
the
chromium,
but
they
are
more
about
optimizing
for
many
applications.
At
the
moment
we
have
heap
snapshots
heap
timelines
and
allocation
timelines,
but
they
are
more
about
allocations
less
about
garbage
affections
and
you
could
get
lost
in
all
the
details
they
captured.
A
We
have
CPU
profiles
to
know
when
the
garbage
collections
are
triggered,
but
it
doesn't
tell
you
how
much
it
doesn't
tell
you
what
this
garbage
can
collections
do,
and
this
could
have
a
significant
impact
on
the
performance,
so
it
might
be
hard
to
get
them
running
in
staging
or
production
environment,
even
if
you
can
only
reproduce
your
problems
there,
but
luckily
VA
provides
garbage
collection
works
for
us.
It's
not
documented.
A
You
have
to
use
command
line
flags
to
get
them,
but
they
can
give
you
more
knowledge
about
heap.
Then
a
number
of
memory
usage.
Without
all
the
details
of
heap
profiles,
they
can
help.
You
verify
your
fixes
by
displaying
the
patterns
of
garbage
collections
in
your
applications.
It's
no
silver
bullet,
but
it's
definitely
a
nice
to
have
in
your
box
of
tricks.
So
this
talk
is
based
on
versions
of
VA
used
by
node
v4
and
v6
LTS.
A
There
are
few
subtle
differences
in
other
versions
of
the
aid,
but
most
of
these
talk
should
still
apply
so
to
make
sure
we
are
on
the
same
page.
What
is
garbage
attention
exactly?
Well,
it's
the
process
of
cleaning,
up
useless
objects
and
freeing
up
memory
for
allocations
in
v8,
most
objects
life
on
the
heap
and
when
I
am
when
an
object
can
not
be
reached
by
traversing
pointers
from
the
root
objects
it
become
even
it
becomes
garbage
and
needs
to
be
collected
in
later
rabbit
election
cycles.
A
A
In
v8,
most
garbage
collections
are
triggered
by
allocations
when
VA
can
can't
allocate
an
object,
because
we
don't
have
enough
memory
for
it.
It
would
collect
the
garbage,
hopefully
freeing
up
some
memory
and
try
again.
If
we
a
still
fail
after
two
collections,
it
will
go
into
panic
mode
and
do
a
more
thorough
garbage
collection.
Invoking
we
call
backs
trying
to
squeeze
out
all
the
memory
it
can
if
it's
still
don't
have
enough
memory.
After
this,
the
MVA
would
just
give
up
and
crash
the
process
saying
we
run
out
of
memory.
A
So
how
can
garbage
collections
have
an
impact
on
the
performance?
That's
because
we're
doing
garbage
collections,
the
jobs
were
engine
must
might
need
to
pass
the
program
execution,
so
it
can
look
into
the
heap
or
move
memory
around.
This
pass
can
come
in
two
different
flavors
for
stop
the
world
garbage
collections.
The
whole
program
is
gonna,
be
freezed
throughout
the
garbage
collection
cycle.
But
if
we
can
somehow
divide
tasks
into
small
pieces,
we
can
make
the
garbage
collection
incremental.
A
We
we
can
have
the
program
run
a
little
while
then
do
a
little
bit
of
garbage
equation
in
march.
Small
pass
and
execution
and
so
forth.
So
in
this
way
the
program
will
be
a
lot
more
responsive
and
if
we
can
take
a
step
further
and
organize
the
heap
in
such
a
way
that
we
can
run
the
program
on
main
thread
while
doing
garbage
collection
on
not
threat,
then
we
will
have
a
concurrent
garbage
collector.
We
can
even
make
a
parallel
by
using
multiple
garbage
collector
stress
in
VA.
A
There
is
more
than
one
garbage
collector,
so
we
have
all
these
different
kinds
of
pulses
during
different
stages
of
the
garbage
collections,
people
who
build
garbage
attack.
Our
collectors
have
noticed
that
most
objects
died,
young
and
those
who
don't
tend
to
live
forever.
This
is
called
weak
generational
hypothesis.
Based
on
this
observation,
we
can
kick
the
those
young
objects
more
frequently
in
order
to
reclaim
memory
in
time
and
check
the
Oh
objects
less
frequently,
so
we
can
save
the
CPU.
A
There
are
a
lot
of
language
implementations
flowing
following
this
strategy,
and
BIA
is
one
of
them.
The
garbage
collector
responsible
of
cleaning
the
young
generation
will
stop
the
world,
but
the
one
cleaning,
the
old
generation
has
a
mixture
of
incremental
concurrent
and
parallel
garbage
collection.
A
So
the
memory
manage
by
VA
looks
like
this.
On
the
heap
we
have
two
generations
of
objects
and
five
spaces.
The
young
generation
only
has
one
space
the
new
space
we
consist
of
contiguous
memories,
pages
and
most
objects
start
and
end
their
lives
in
this
space.
In
this
space,
the
new
space
is
divided
in
half
each
one
is
called
a
semi
space.
The
old
generation
have
four
spaces
dedicated
to
different
kinds
of
objects.
The
O
space
contains
objects.
A
Thus,
death
arrives
more
than
two
garbage
attractions
in
the
new
space,
and
if
we
ate
things
there's
some
objects
are
going
to
be
promoted
anyway,
then
they
will
be
allocated
in
the
old
days.
The
post
phase
contains
as
a
cutable
call
compiled
by
v8.
The
map
space
contains
the
maps
also
known
as
hidden
classes,
which
contain
metadata
about
of
July
outs.
The
large
object
space
contains
I'm
just
too
large
to
fit
into
a
page
of
in
any
other
spaces.
A
Va
has
a
memory
allocator
to
manage
the
space
too
many
pages,
IRRI
sieves
from
the
operating
system
and
for
not
yes,
we
have
some
external
memory
managed
by
the
C++
code
like
the
buffers.
So
when
allocating
this
memory,
we
will
report
to
VA
about
the
amount
of
memory
allocated
and
register
callbacks
to
clean
them
up
when
the
corresponding
JavaScript
objects
become
unreachable.
A
So
VA
puts
a
limit
on
the
memory
you
can
use
by
default.
The
two
semi
spaces
in
the
new
space
can
each
have
a
most
16
megabytes
of
memory
and
the
whole
old
generation
can
have
no
more
than
1.4
gigabytes
of
memory.
Since
we
can
move
objects
from
the
new
generation
to
the
old
generation,
we
usually
run
out
of
memory
when
we
hit
the
limit
of
the
old
generation.
A
A
They
uses
the
scavenge
garbage
collection
to
clean
up
the
new
generation,
as
we
mentioned
before,
new
space
is
divided
into
two
equally
sized
semi
spaces.
Only
one
of
them
is
active
at
a
time
when
we
need
to
allocate
a
new
object
and
the
active
semi
space
is
full,
we'll
start
to
scan
references
from
the
root
objects
and
copy
the
reachable
objects
to
the
other
semi
space.
A
The
old
generation
is
managed
by
a
different
garbage
collector.
So
when
a
certain
amount,
a
certain
memory
threshold
is
hit,
VA
will
start
to
identify
the
live
objects
on
the
heat
and
leave
marks
on
them.
When
marking
is
done,
VA
will
start
to
reclaim
memory
either
by
sweeping
or
compacting.
When
sweeping
sweeping
a
page.
Va
will
add
memory
box
taken
up
by
detect
objects
into
the
free
list.
So
the
next
time
in
needs
memory
from
this
page.
It
will
pick
a
block
from
the
free
list.
A
Overwriting
the
dead
object
with
another
object,
a
page
could
be
heavily
heavily
fragmented
after
a
few
sweepings.
The
VA
can
instead
move
all
the
live
objects
in
this
page
into
a
new
page
or
another,
less
fragmented,
suspect
page.
This
is
called
compacting
and
after
compacting,
the
page
left
behind
him.
We
returned
to
the
operating
system,
so
in
summary,
GC
pauses
in
VA
looks
like
this
most
of
the
times
we
allocate
near
objects
in
the
new
space
and
when
it's
full
VA
will
start
scavenger
protection.
A
A
This
process
is
incremental
and
again
each
step
of
marking.
She'll
has
just
a
few
milliseconds
if
the
liveliness
of
an
object
is
changed
after
it's
been
marked,
something
called
the
right
barrier
will
be
triggered
to
remind
VA
of
checking
it
later.
So
when
the
marking
is
complete,
VA
might
not
jump
into
sweeping
immediate
immediately.
A
It
wait
for
the
right
time
right
time
to
start
sweeping,
since
we
are
sweeping
that
objects.
This
should
this
should
not
interfere
with
the
program
execution
and
should
be
safe
to
paralyze,
so
sweeping
is
both
concurrent
and
parallel.
After
sweeping
VA
will
do
a
scavenge
garbage
collection
in
the
new
space
things.
We
have
also
marked
the
young
generation
in
the
marking
phase,
and
this
will
need
to
pass
the
program
for
a
few
milliseconds.
A
A
There
are
more
than
one
option,
each
with
a
different
output.
We
are
only
going
to
talk
about
ones
that
we
found
to
be
the
most
useful,
so
there
is
trace
GC,
which
point
which
we
point
line
for
each
job
detection
describing
the
when
the.
How
and
what
and
why
of
this
garbage
occasion-
and
there
is
a
both
version
of
it,
which
gives
a
summary
about
a
space
in
the
heat
after
each
complication.
There
is
also
chase
PC
MVP,
which
prints
dozens
of
name
value
pairs
for
each
step
of
GC.
A
A
Okay,
so
next
I
am
going
to
talk
about
how
to
reduce
docks.
The
help
of
trace
GC
looks
something
like
this
will
have
one
line
of
this
for
each
copy
faction
on
each
line.
First,
we
have
where
this
garbage
collection
is
coming
form.
There
is
a
process
ID
and
in
I,
told
ID
and
isolate
is
like
an
instance
of
v8.
It
has
its
own
garbage
collector,
the
green
one.
There
is
when
this
garbage
collection
takes
place.
That
is
how
many
milliseconds
has
passed
since
the
process
has
started.
A
A
The
purple
one
is
how
much
memory
has
been
reclaimed
during
these
garbage
collection
on
the
left
order
of
the
arrow.
We
have
the
memory
usage
after
the
garbage
collection.
The
numbers
outside
the
parentheses
are
the
total
sizes
of
objects.
The
number
inside
the
parentheses
are
the
number
da
VA
actually
allocated
from
the
operating
system.
A
The
number
on
the
right
hand,
side
of
the
arrow,
is
memory
usage
after
these
garbage
fraction,
so
the
numbers
are
inside
and
outside
the
parentheses
are
different
because,
as
we
talked
out
before,
the
spaces
are
page
and
can
be
fragmented.
After
that,
we
can
see
how
long
these
garbage
collection
takes.
So
before
the
slash
we
have
the
time
that
time
the
program
has
been
passed
on
the
right
of
the
slash
stash.
A
We
have
the
time
we've
spent
on
external
memory
and,
following
that,
we
have
the
reason
of
this
garbage
collection,
as
we
talked
about
before
most
of
the
times
is
allocation
failure
and
the
next
one
is
the
both
flag.
This
has
to
be
used
with
chase
GC
together
to
get
the
output
so
for
each
garbage
exception.
This
will
print
similar
lines.
The
memory
allocator,
as
mentioned
before,
is
an
abstraction
that
VA
uses
to
manage
memory
allocated
for
on
the
OS.
A
The
use
memory
is
the
memory
VA
has
already
allocated
from
the
operating
system,
and
the
available
memory
is
how
much
more
memory
the
VA
can
get
from
the
OS.
Then
we
have
the
statistics
about
each
space,
for
most
of
them
used
memory
is
how
much
memory
the
objects
has
occupied
in
this
space
available
memory
is
how
much
memory
left
in
the
pages
and
committing
memory
is
the
sum
of
these
two
notice
for
the
new
space.
Since
we
have
to
have
two
decimal
spaces.
The
Committee
memory
is
the
twice
of
the
Sun.
A
The
the
available
memory
of
the
large
object
space
is
much
bigger
because
that's
the
available
memory
from
the
memory
allocator
minus
the
memory
can
use
to
put
objects
like
the
page
headers.
Then
we
have
the
total
of
these
spaces.
Like
amount
of
external
memory
reported
by
the
user
code,
the
time
we've
spent
on
each
garbage
collection,
some
Lockwood
report,
a
heap
growing
factor
that
VA
uses
to
calculate
threshold
for
next
marking.
A
A
So
here
are
some
performance
problems
we
have
found.
There
are
related
to
garbage
collections,
we've
seen
tons
of
scavenge
garbage
collections,
eating
up
the
CPU.
These
could
be
cache
misses
caused
by
in
appropriate
caching
strategies
or
it
could
be
excessive
tip
cons.
Some
applications
do
that
you
avoid
mutations.
A
Some
of
our
applications
handle
huge
chunk
of
RPC
payload
and
the
serialization,
and
this
year
TC
realization
could
generate
a
lot
of
large
objects
blocking
up
the
large
object
space
and
there
are
some
peculiar
cases
like
a
bug
in
our
templating
engine,
keeps
compiling
unnecessary
JavaScript
functions
from
the
templates,
and
these
functions
live
in
the
Coast
base.
So
we
can
see
the
code
space
keep
growing
in
the
garbage
collection
blocks,
some
of
our
external
clients
use
compacts
and
closures
to
build
applications,
and
this
can
be
problematic
in
long
connections.
A
Connections
because
VA
doesn't
minimize
the
closures
at
the
moment.
So
these
cultures
can
keep
a
lot
of
irrelevant
objects.
References
referenced
then
VA
will
keep
promoting
these
objects
to
the
out
space
and
when
you
hit
the
limit
of
your
own
space,
this
can
crash
your
process
and
they
are
some
tools
to
help.
You
read
the
garbage
collection,
lock,
we've
open
sourced
our
parser
is
on
the
APM
and
it's
ready
to
use
for
people
who
already
have
applications
running
in
production.