►
From YouTube: What's Up with the Community Post-Mortem Diagnostics Working Group by Richard Chamberlain, IBM
Description
What's Up with the Community Post-Mortem Diagnostics Working Group - Richard Chamberlain, IBM
It's important to be able to figure out what's going on when things go wrong in your Node.js production application. Tools are needed to investigate memory leaks, crashes and other "interesting" events in production. The post-mortem community working group (https://github.com/nodejs/post-mortem) is working on these problems. Come and learn about the key issues being worked, and the progress of the working group so far as illustrated through examples and code.
About Richard Chamberlain
Node.js diagnostics
About Yunong Xiao
I’m a software engineer at Netflix. I also maintain the open source Node.js framework restify. I have spent stints of my career at AWS and Joyent, respectively, where I worked on distributed systems and helped launch several cloud computing products. I’m especially proud of AWS IAM and Manta, two projects that I helped shape and build. These days I’m leading the Node.js platform at Netflix.
B
B
It's
great
one,
second
and
and
so
working
work,
your
own
man.
Now
all
right
we'll
have
to
click,
but
that's
okay,
so
yeah
I
mean
I'm
I
work
at
Netflix,
hopefully
something
you
have
checked
out.
My
talk
from
yesterday,
but
we
use
know
quite
a
bit
and
the
tooling
that
we're
about
to
talk
about
is
really
quite
essential
to
everything
that
we're
doing
at
Netflix.
A
High,
so
I'm
richard
chamberlain,
I
work
at
ibm's
location
in
her
sleep
park,
hampton
england.
This
is
my
office
and
nice
stately
home
there.
That's
not
actually
true.
My
office
is
round
behind
the
trees
on
the
right.
That
office
is
the
boss's
office.
I
get
to
go
there
occasionally
when
things
are
bad,
so
this
talk
is
about
what
the
something
called
the
nodejs
post-mortem
diagnostic
work
group
bit
of
a
mouthful.
A
A
B
B
Oh,
no,
it
works
this
week,
so
here,
debugs
nodejs
ends
up
going
started
on
danger
bugging
to
use
any
of
the
tools
that
we've
got
on
these
slides
right,
that's
great,
so
these
are
sort
of
the
typical
tools
of
one
would
use
when
something
goes
wrong
with
their
no
application
right.
It's
pretty
standard.
You
attach
a
debugger,
what's
sort
of.
What's
all
the
same
about
all
this
stuff
like
what
is
there
something
there's?
Is
there
a
common
theme
that
ties
all
these
debugging
tools
together?
I
won't
leave
you
hanging
it's.
B
You
can
only
really
use
anything,
a
test
environment
right
and
the
reason
is
that
you
know
you
can't
you,
wouldn't
you
can't
really
go
and
attach
a
debugger
to
production
right,
because
if
you
did
that,
then
the
process
would
halt
and
whatever
customers
you
have
that's
currently
connected
would
not
receive
a
response
back.
So
what
do
we
do
about
production
right?
What
how
do
you?
How
do
we
handle
failures
and
bugs
in
production,
and
we
need
to
debug
them
in
production?
Well
solve
that
problem.
B
I
thought
I
use
a
little
nice
little
analogy
here
saying
you
know
what
this
is
come
on,
got
some
nerves
out
there
right.
That's
right!
This
is
Pluto.
I
was
actually
really
excited
last
year
when
the
New
Horizons
probe,
flew
by
Pluto
and
took
some
really
amazing
pictures,
and
if
you
really
think
about
it,
you
know.
B
Nasa
kinda
has
the
same
problem
right
there,
sending
a
probe
out
into
space
to
take
some
pictures
and
collect
a
lot
of
information
about
it
about
Pluto,
but
you
only
fly
by
Pluto
once
and
so,
let's
say:
you're
fine
by
Pluto
and
there's
a
bug
in
the
application
that
the
probe
is
running
on
right.
Well,
what
would
they
do
with
a
attach
a
debugger
and
like
debug
the
problem,
and
then
it's
okay?
We
fight.
B
We
figured
out
the
problem
three
hours
later
oops
through
phone
fast,
blue,
all
right
and
we
don't
get
this
one
shot
to
fly
past
Pluto.
So
how
does
Nast
I
really
deal
with
this
problem?
Well,
it
turns
out
they
use
something
called
core
dumps
and
post
more
in
debugging,
so
even
back
in.
So
what
is
the
71?
They
released?
B
So
they
can
go
back
and
we
begin
data
acquisition
again
because
again,
the
only
five
hats
like
clue
or
Jupiter
or
Saturn
once
and
you
want
to
make
sure
that
you're
always
you're
up
time
is
always
there
to
capture
all
the
data
and-
and
this
is
kind
of
very
similar
to
sort
of
what
most
of
you
folks
probably
have
have
problems
with
in
your
production
environment.
So
you
have
customers,
you
have
to
serve
and
you
want
to
do
the
same
thing,
but
a
little
bit
more
about
core
Tom's.
B
The
core
post
war
and
debugging
was
also
born.
I
love,
who
were
saying
earlier.
We
have
some
of
these
pretty
critical
production
constraints,
just
like
NASA
for
at
least
enough
Netflix
off
time
is
critical
right.
You
know.
If
we
have
a
bugging
our
systems,
we
want
to
make
sure
you
go
back
and
watch
all
the
great
content
that
we
have.
So
you
know
we
want
to
make
sure
that
we're
always
up
a
lot
of
these
bugs
are
not
easily
reproducible,
because
we
have
you
know
tens
or
hundreds
of
thousands
of
nodes
in
the
system.
B
You
know
villains
of
rps
per
day,
so
we
can't
really
simulate
late
that
environment
and
again,
our
primary
goal
is
to
resume
normal
operation
as
soon
as
possible,
and
that
generally
means,
then
you
know,
have
you
restart
with
a
the
application
of
the
service
and
so
post
mortem
debugging
really
likes
you
take
this
more.
I
want
to
say
novel,
but
this
techniques
been
around
for
a
while
for
very
long
time.
Actually,
this
really
awesome
approach,
which
is
like
you,
take
a
cordon
which
snapshots
the
entire
process
state
of
the
application.
B
And
then
you
take
the
top
path,
which
is
you
immediately?
Then
we
start
application
right
and
and
then
you
continue
to
serve
traffic.
So
whatever
bucks
you
had,
you
know
you
restarted
it
now
and
you
should
be
hopefully
back
back
to
serve
in
traffic
and
then
you
can
go
ahead
and
take
this
court
open
loaded
somewhere
else,
you
are,
your
stack
is
getting
to
run.
Run
and
it's
healthy,
but
you
take
the
cord
on
loan
in
somewhere
else
and
then
debug
it
sort
of
at
your
own
leisure.
B
Take
your
time
and
really
engineer
a
fix
and
figure
out
what
the
problem
is,
and
we
think
this
is
really
important
because
it
lets
you
route
class
a
lot
of
these
problems
and
prevent
them
from
happening
in
the
future
right.
So
how
do
we
do
this
with
node?
And
actually
this
is
one
of
the
reasons
we
use.
Nona
Netflix
is
because
it
is
there
there's
this
rich
set
of
tooling
that's
available
and
which
is
going
to
talk
a
lot
more
about
that
later
on,
but
you
can
actually
configure
no
to
abort
on
cod
exception.
B
Here
was
just
that
flag.
So
now,
if
there's
an
uncaught
exception
within
note,
instead
of
just
that
stack
trace,
you
see
there
see
on.
The
bottom
will
actually
also
dumps
a
core,
so
the
offering
system
will
actually
save
a
cordon
of
the
process
and
what
the
courtroom
would
really
give
you
in
addition
to
layer,
which
is
the
stack
trace,
which
you
normally
already
get
anyway,
is
why
I
like
what?
B
What
is
it
on
the
heap
and
the
stack
very
will
state
that
that
actually
caused
the
pot
process
to
crash,
and
so,
in
addition,
that
happens
just
know,
dump
core
when
is
on
call
exception,
that's
been
thrown.
You
can
also
generate
a
cordon
with
utility
co
ji
core,
which
is
available
on
most
unix-like
systems,
and
that
will
take
a
core
dump
of
any
process.
Well,
it's
running
no
positive
process
very
briefly,
and,
and
we
use
that
all
the
time
of
production
as
well.
B
B
So
yesterday,
I
able
to
talk
about
our
new
sort
of
a
netflix
hei
stack,
and
so
you
notice,
on
the
second
column,
from
the
left,
where
we
have
a
node
API
stack
and
that
connects
to
our
tour
edge
services
API
and
the
way
that
the
RPC
really
works
is
that
there
is
a
connection
pool
so
the
connection
pool
within
node,
and
it
maintains
the
set
of
connections
to
a
bunch
of
API
instances,
and
so,
as
we
were
sort
of
building
others
named
stack.
We
kept
seeing
this
in
the
logs,
which
is
which
said
hey.
B
There
are
no
more
free
connections
in
the
connection
pool
to
connect
to
the
downstream
services,
which
is
really
weird,
because
we
would
look
at
the
downstream
servers
and
they
will
be
all
healthy
and
they
would
all
have
you
no
avail
availability.
So
it
was
really
weird
how.
But
then,
how
do
we
debunk
this
problem
right,
because
there's
only
happening
production
in
some
really
really
rare
cases.
We
were
unable
to
reproduce
this
thing
in
tests
and
we
in
fact
had
no
idea
what
the
state
was
well.
B
B
So
now
at
least
we
knew
like
we
had
free
connections.
There
were
no
problems,
there,
there's
probably
some
problems
in
the
logic
in
which
we
were
freeing
or
to
freeing
the
connection
list
or
something
like
that,
and
so
this
really
gave
us
a
really
critical
view
into
like
hey
how
things
were
working
and
it
made
it
made.
It
really
easy
for
us
to
at
least
pinpoint
without
problems
coming
from.
B
So
one
thing
that
I
want
to
leave
you
leave
you
with
is
that
for
the
post
war
in
debugging
is
really
critical
for
us
to
our
large-scale
production,
no
deployments
and
I.
Think
for
any
of
you
who
run
you
know
no,
not
a
large
scale
with
a
lot
of
production
traffic
with
customers
that
are
directly
impacted,
it'll
be
really
helpful
for
you
too.
B
A
Yeah
so
I
glad
to
hear
that
they
call
dumpster
so
useful
with
some
legacy.
Skills
might
come
in
handy
here,
so
the
things
we
focused
on
in
2016
and
in
the
in
a
working
group
were
a
Linux
Mac
solution
for
analyzing
core
dumps.
I'll
go
over
that
in
a
minute,
a
proposal
for
putting
a
JavaScript
API
on
top
of
that
capability
to
give
us
more
ability
to
develop.
You
know
more
usable
tools
on
top
of
that,
a
bit.
A
Dump
and
a
lightweight
dump
called
which
we're
calling
no
report,
so
here's
some
of
the
main
pieces
in
the
picture
so
now
down
the
bottom
there.
Autumn's
now
key
thing
that
the
day
for
check
on
the
other
guys
enjoyed
it
a
few
years
ago,
was
to
add
something
called
post-mortem
metadata
into
the
v8
runtime.
So
what
this
gives
us
is
just
a
nice
set
of
offsets
and
constants
and
types
that
we
can
use
to
pull
apart.
The
JavaScript
object,
headers
and
the
JavaScript
stack
frames,
and
this
is
this-
is
important.
A
A
A
And
you
can
more
nice,
look
at
that
and
straightaway
see
what
your
problem
is
if
you've
got
a
memory
leak
or
you're
running
out
of
memory,
so
that's
kind
of
pointer
to
the
sort
of
nice,
tooling,
end-to-end
story.
We
want
with
core
dumps,
but
we
don't
have
that
yet,
on
the
left
hand,
side
something
I'm
not
going
to
talk
about
so
much
is
what
you
can
do
with
a
live
application
terms
of
tracing
logging
and
profiling.
A
There's
another
working
group
which
ones
cause
that
the
tracing
working
group
now
call
the
diagnostic
working
group
and
they're
doing
some
good
stuff
ali
shaikh,
in
particular
on
adding
the
low
overhead
tracing
capability
into
v8,
and
hopefully
we
have
to
use
some
of
that
in
a
production
deployment.
So
going
on
about
core
dumps.
A
Of
these
plugins
using
the
metadata,
we
can
now
look
at
general
scripts
tax
heap
objects
and
source
codes
in
a
core
dump
after
the
fact,
so
to
catch
up
with
what
they
had
on
MDB
guy
worked
with
called
Howard
heli
has
done
a
lot
of
work,
actually
landing
changes
into
lltv
itself
and
LOL.
Bb
3.9
has
got
enhancement
which
lets
us
get
to
all
the
memory
regions
at
the
he
/
in,
and
that's
a
big
step
forward
in
terms
of
getting
java.
Rob
JavaScript
objects
out
of
the
core
dump.
A
A
Just
to
show
you
the
commands
you
got
available
with
the
LDV
debugger
and
the
others
first
column
is
the
joint
mdv
debugger
commands
second
column,
the
ldb
commands,
which
is
no
plugin
and
the
third
column
and
earlier
tool
the
IBM
have
for
their
platforms,
call
ID
de
so
the
key
ones
there.
You
got
a
stack
to
print
a
stack
trace,
that's
usually
a
lease
on
the
ll
node
one.
It's
a
mixed
stack
trace
shows
you
the
native
frames
and
the
JavaScript
frames.
A
The
further
on
down
these,
the
find
references
were
important
when
you're
looking
for
a
memory
leaks
to
find
out
what's
keeping
an
object,
survive,
so
you've
got
ways
in
which
you
can
navigate
around
the
around
the
cord
I'm
using
those
so
good
set
of
commands.
Now
on
both
those
two
debuggers
which
covers
the
platforms,
we
need
just
a
quick
example,
so
this
is
jess
screaming
print
command.
On
early,
be
showing
the
native
module
object.
A
You
can
see,
it's
pulled
out
the
properties,
file,
name,
ID,
exports
load
and
loading,
so
we've
got
a
couple
of
strings
there.
A
couple
of
billions
and
one
of
the
properties
is
the
exports
property
is
actually
a
function,
so
we
can
drill
down
into
that
we're
on
jer's
print-
s
on
that
address
and
you've
got
the
source
code
for
that
function
and
other
information
about
that
function.
So
we've
got
lots
of
capability
now
for
looking
those
core
dumps,
but
this
still
isn't
particularly
usable.
A
This
isn't
still
nasty
line
mode
using
that
those
tools
I'm
just
going
to
switch
now
quickly
to
to
heap
dump
because
they
say
that
shows
us
the
way
in
which
we
have
a
nice,
tooling
story,
and
there
are
there's
a
very
popular
NPN
written
by
been
gnawed
house,
and
it's
a
very,
very
commonly
used
tool
gives
you
a
snapshot
of
the
heat
in
JSON
format.
The
problems
with
it
are
that,
because
the
snapshot,
API
is
doing
a
lot
of
work,
its
indexing
objects
and
it's
separating
objects
out
from
the
references
between
objects.
A
It
actually
is
a
big
hitch
when
you
run
this
thing,
and
it
also
consumes
a
lot
of
memory
as
it's
producing
the
heap
dump.
So
in
a
in
a
production
system,
especially
when
you're
running
our
memory,
that's
not
an
option,
so
I,
don't
think
you
you
know
would
use
it,
but
like
I
say
it
does
show
how
you've
you've
got
a
nice
for
max,
which
then
lead
you
into
a
nice
GUI
tool
which
will
take
you
straight
to
the
to
the
problem.
A
So
what's
coming
up,
hopefully,
next
year
with,
on
top
of
those,
the
ll
load
library,
we
can
build
a
JavaScript
API,
which
will
give
us
a
nice
way
of
producing
some
more
usable
tooling
on
top
of
that
solution.
So
the
API
would
have
things
like
get
thread.
Get
frame,
get
object
up,
calls
in
it
and
a
contribution
Dave
Clements
Inn
near
forum
was
to
say:
can
we
output,
Jason
representation
of
objects
in
a
in
a
core
dump,
so
kind
of.
A
Objects,
if
you
like,
they
can
then
go
into
other
tools
or
other
lot
of
viewers,
so
look
out
for
that.
So
the
way
that
would
that
would
build.
On
top
of
that
we
currently
got
now.
The
green
box
is
what
we
have
the
solution
we
have
now.
The
yellow
DB
come
on
line
debugger
with
that
plug-in
being
able
to
read
JavaScript
information
from
a
core
dump
on
those
platforms,
so
on
top
of
that
a
JavaScript
API
and
then
we're
bored
givi
tools
using
the
API,
so
got
a
quick
example
here.
A
What
that
might
look
like
a
simple
example
running
on
node
using
that
API,
so
we've
loaded
the
core
dump,
because
the
into
this
into
this
application
and
we've
got
a
couple
of
options
to
display
the
threads
in
the
than
the
the
heap.
So
here
we've
just
done
a
listing
of
the
objects
of
heat
by
by
total
size
taken
up,
and
you
can
see
this
is
obviously
rogue
application
with
the
and
a
thousand
of
those
my
record
objects.
A
A
Just
to
cover
one
or
two
other
discussion
topics
like
they've,
been
going
on
in
the
in
the
community
improving
that
metadata
I
talked
about
some
code
sharing
between
the
different
tools.
We
had
quite
a
lot
of
discussions
about
heap
done
before
Matt
there's
some
other
interesting
topics
about
catching
or
not
catching
exceptions,
whether
to
continue
or
or
a
bought.
Some
good
discussions
on
github
on
those
topics.
A
We've
been
moving
these
projects
to
the
no
foundation
which
is
a
better
place
to
have
them
than
scattered
around.
And
lastly,
this
lightweight
report
will
be
calling
no
report
which
we've
been
working
on,
how
to
show
you
quick
couple
of
slides
on
that.
So
this
is
now
available
as
an
npn.
So
it's
a
the
idea
is
that
core
dumps,
whatever
you
do,
are
still
going
to
be
big
ugly
things
that'd
be
nice
to
have
a
simpler
solution,
for
least
some
basic
problems
or
for
early
triage
of
problems.
A
So
this
is
an
MPI
written
in
its
native
MPN,
so
that
you
can
get
this
thing
out
when
javascript
to
stop
running
so
in
out
of
memory
situations
and
so
on.
We've
got
it
triggered
by
exception,
fatal
error
in
the
node,
triggered
by
a
external
signal
sent
in
to
kill
signal
and
a
JavaScript
API
for
it,
and
just
to
show
you
what
sort
of
things
it
contains.
I've
got
one
here
which
has
been
I've,
trimmed
it
out,
but
the
top
there's
a
section
that
says
what
the
event
was
that
generated.
A
This
little
dump,
in
this
case
signal
sent
in
sig
user
to
blow
that
the
JavaScript
stacktrace
blow
that
the
native
stack
trace
and
then
some
stats
about
the
current
heap
heap
sections
helpful.
They
are
how
big
they
are.
Some
resource
uses
information
and
we've
got
more
information
about
the
parameters
used
in
that
program,
the
environment
variables.
Even
the
settings
that
kind
of
stuff
we've
still
need
to
add
some
more
event:
loop
type
information
into
there
for
that
to
be
useful
and
outstanding
callbacks.
A
A
We
could
do
with
a
few
more
participants,
so
I
was
a
bit
of
call
dramas
here
if
anyone's
interested
in
helping
out
have
a
look
at
what's
on
there,
no
jazz
/
post-mortem
and
it's
all
in
the
open-
and
there
are
two
projects
on
there
that
you
could
have
a
look
at
contribution
to
and
yeah
do
have
a
play
with
some
of
this
stuff,
and
let
us
know
what
things
work
and
what
things
don't
work
very
interested
here.
How
people
getting
on
with
that
tooling,
especially
the
newer
core
dump
tooling
using
ll
DB.