►
From YouTube: GMT 2018-05-16 Performance WG
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
A
A
A
A
A
A
C
C
A
A
C
C
C
C
Cpu
consumption,
so
in
this
one
it's
the
master
and
it's
pretty
heavy
heavily
loaded.
So
it's
not
as
interesting,
but
if
I
look
at
say
like
a
agent
I
can
see
that
there's
like
Oh
interesting,
there's
like
these
periodic
CPU
spikes
and
I
can
drill
into
one
of
these
and
should
generate
yeah.
So
it's
just
gonna
give
me
a
flame
graph
for
that
specific
time.
Window
and
I
can
see
that
it's
mostly
like
okay,
real
path
like
OS
real
path
is
13%
and
then
the
rest
is
like.
C
So
it's
like
reading
doing,
L,
stat
for
real
path
and
malloc,
and
free
and
F,
open
and
I
think
when
I
had
grabbed
some
other
stocks
of
this.
C
C
Os
children
I
think
it's
not
here
either.
Always
children
is
like
super
sense
of
because
it
goes
and
touches
like
all
the
processes
in
proc
and
I
guess
the
only
way
to
improve
that
was
kind
of
a
well.
The
only
major
way
to
improve
all
the
system
call
overhead
is
that
that
new
system
call
you
were
showing
James,
which
didn't
end
up
landing.
C
So
you
run
perfect
court
and
then
this
one
makes
it
kind
of
I
think
it
flattens
the
all
the
stacks
like
it
kind
of
just
makes.
The
file
include
like
a
massive
collection
of
stacks,
usually
this
files
a
lot
bigger.
So
you
run
this
like
father
massages
it
into
like
a
format
that
I
guess
scripts
can
understand
more
easily,
not
quite
sure
why
it's
a
separate
command
that
maybe
it's
just
a
lot
bigger
in
the
file
and
all
you
can
do-
is
take
this
file
and
drop
it
in
flames
that
flame
scopes.
C
C
So
in
here
there
was
a
comment
about
perfect,
supporting
a
workaround
for
missing
frame
pointers,
Libin
wand,
which
uses
dwarf.
This
can
be
enabled
with
call
graph
dwarf
so
in
the
master
stacks
actually
used
that
I
passed
that
in
and
I
think
they
were
better,
but
I
haven't
had
time
to
more
rigorously
compare
okay,
yeah
there's
this
bug
in
flame
scope,
where
sometimes
this
happens,
throws
an
exception
and
I
filed
an
issue
with
them,
but
I'm
not
sure
what
triggers
it
exactly.
C
Okay,
here's
all
right
here,
it
looks
like
you
know,
maybe
we're
getting
a
little
bit
more
like
process
manager.
Resume
is
showing
up.
This
looks
like
metrics,
but
I
I,
don't
know
why
it's
showing
up
in
the
await
process.
I
would
expect
that
it
is
spending
its
time
and
like
the
master
and
allocator
and
so
on,
competing
gauges,
but
instead
of
this
shows
up
so
I'm
wondering
if
that's
because
we
don't
have
friend
pointers,
earth
or
if
it's
just
been
optimized
and
inlined
or
something
I'm,
not
sure.
A
Our
internal
builds
not
very
complete,
I
guess
so
trying
to
get
dwarf,
support
in
the
right
places
with
tool,
chains
and
symbols
and
the
right
kernel
package
in
them.
There's
a
bunch
of
packaging
and
distribution
issues
which
made
which
made
the
dwarf
sort
of
unreliable
or
hard
to
get
so
friend.
Pointers
are
easy
to
enable
and
seemed
to
basic
seems
to
work
in
the
cases
that
I
could
see.
A
I,
don't
know
how
mature
the
dwarf
stuffy's,
especially
in
the
house,
because
there's
a
ton
of
dwarf
like
the
dwarf,
the
dwarf
is
huge
and,
as
I
think
you
might
need
I,
don't
know
if
you
want
to
do
the
extended
dwarf
that
set
this
like
I,
think
there's
gee
gdb
three,
so
there's
a
lot
of
different
knobs
to
twiddle
around
this
area
that
it's
pretty
time-consuming
to
explore
all
the
possibilities.
Okay,.
C
Yeah
I,
also
just
don't
understand,
what's
happening
under
the
covers
to
make
this
I
think
toon
was
telling
me
that
it
looks
like
it
might
be
that
the
dwarf
information
includes
the
the
stack
frame
size
or
something,
and
that
lets
that
lets
perf
know
where
the
how
to
jump
to
like
the
previous
frames
and
so
on.
Yeah.
A
Ooh
I
should
we
have.
The
dwarf
should
have
enough
information
to
let
like
a
stack
trade,
a
step
walker.
Do
things
like
figure
out
that
something
got
figure
out,
that
a
function
got
in
lined
and
that
you're
really
in
an
inline
function
and
present
them
as
a
stack
frame
and
and
it
should
be
able
to
fill.
C
A
C
A
C
So
I
and
the
other
thing
is
that
this
flag
is
like
one
or
the
other.
You
you
it
either
uses
FP
or
Dorf
or
lb.
Are
that
doesn't
seem
like
it
can
use
both
for
some
reason.
B
A
A
That's
good
No
G
just
generates
debug
info.
Okay,
it's
like
orthogonal
to
the
optimization
level
you
can
be.
You
can
do
Oh
G,
which
is
supposed
to
optimize,
but
still
make
your
code
debuggable,
but
at
least
my
experience
of
trying
to
get
debug
stuff
in
resources
that
it's
not
debuggable
enough.
So
if
I
needed
a
debugger,
I've
usually
had
to
go
2-0
I.
A
A
C
Yeah,
this
is
a
master
example
where
it's
like
this
master
is
using
almost
300%
CPU,
and
you
can
see
it's
spending
time
and
like
allocating
and
I
noticed
that
it
was
actually
like.
There's
a
big
chunk
of
that
which
is
just
computing.
The
available
resource
on
the
agent,
the
subtraction
operator
is
actually
you're
expensive.
So
we
have
already
seen
a
lot
of
interesting
stuff
come
out
of
this
I
actually
wanted
to
yeah.
A
C
It
looks
like
that
we've
managed
to
make
the
addition.
Here's
here's
that
bug
again,
it
looks
like
we've
managed
to
make
the
addition
very
efficient
because
I
don't
see
it
showing
up.
But
this
looks
like
JSON
responses,
because
in
this
particular
cluster
I
mean
the
Jason
endpoints
are
getting
hammered
and
then
the
allocator-
and
this
looks
like
metrics
to
me-
it's
showing
up
a
little
bit
weird.
You
know
that's
coming
in
and
waited
but
I.
C
C
C
So
that's
this
is
a
broken
flame
graph.
I
filed
a
ticket
with
them.
They
haven't
responded
yet,
but
yeah
I
seem
to
hit
this
whenever
there's
I
don't
know
a
lot
of
data
I'm
not
quite
sure
exactly.
What's
triggering
it
so
in
this
graph,
I
can
only
really
look
at
like
small
little
windows,
if,
like
in
be
thirty,
forty
milliseconds.
C
And
yeah-
and
this
is
always
what
seems
to
show
up
here-
waited
anyway,
it's
pretty
easy
to
use
it
I'm,
hoping
other
folks
can.
C
I'm
also
hoping
we
can
update
the
profiler
end
point
to
maybe
make
it
very
easy
to
just
get
one
of
these
files
back
without
having
to
get
on
to
the
box
and
run
these
commands
and
SCP
this
stuff
back.
Hopefully,
you
can
just
hit
a
end
point
and
get
the
file,
and
then
we
can
take
the
file
and
put
it
in
flame
scope.
Would.
A
C
But
yeah
I
know
one
thing:
I
ran
into
when
I
was
trying
to
do
this
I
think
with
chin
chin
was
helping
us
mom
there's
this
flag,
but
by
default
it
will
actually
grab
all
this
sub
subprocesses
stacks.
So
I
was
running
it
on
an
agent
and
I
was
getting
all
this
JVM
stuff
and
I
was
really
confused
and
then
I
realized
you
had
the
past
I
or
something
no
inherit
to
get
just
the
process
that
you're
passing
with
the
pig
parameter.
C
A
No
I
think
that
looks
pretty
cool
thanks
for
digging
into
that.
Then
it's
really.
It's
really
really
promising
yeah.
C
C
Delayed
as
well
as
it
can
induce
a
lot
of
CPU
consumption
on
their
master
and
allocator
actors,
I
assume
that
you
guys
are
maybe
just
pulling
it
from
one
thing
like
a
metrics
monitoring
system
or
something
but
I
think
in
some
other
cases,
people
are
pulling
it
from
different
sources
and
it
might
be
getting
pulled
pretty
frequently.
So
what
I'm
hoping
to
do
is
migrate.
C
C
F
F
Okay,
so
I
don't
really
have
much
to
show
as
such,
but
I
will
just
actually
talk
a
bit
about
what
we
have
been
doing
in
terms
of
the
benchmark.
And
what
do
you
expect
so,
hopefully
by
the
next
next
meeting
that
we
have
here?
I
will
be
able
to
present
some
numbers
to
actually
get
more
more
feedback
so
anyway,
just
to
actually
give
an
overview
of
what
is
the
problem
that
we
are
dealing
with?
F
I
just
start
about
start
with
the
scenario
where
we
have
a
bunch
of
frameworks
that
are
registered
with
the
master
and
of
those
frameworks.
There
might
be
a
class
of
frameworks
which
are
actually
resource-hungry
as
in
whatever
resource
is
thrown
at
them.
They'll
actually
accept
it
and
will
have
always
some
jobs
to
launch
and
so
on,
and
then
there
is
a
different
class
of
frameworks
which
actually
doesn't
care
about
many
resources.
F
So
if
you
are
sent
an
offer,
they
will,
if
you
send
an
offer,
they'll
actually
accept,
maybe
like
half
of
them
or
maybe
like
only
a
fixed
number
of
offers,
and
they
keep
declining
everything
else.
The
way
DRF
works
is
when
it
is
time
for
the
next
allocation
rounds.
It
puts
the
the
frameworks
in
a
priority
queue.
F
Well,
the
highest
priority
is
given
to
those
frameworks
which
have
not
used
their
fair
share
yet,
and
so
what
DRF
will
try
to
do
is
will
say:
okay,
these
are
the
five
agents
that
I
have
example
and
out
of
these
20
frameworks,
I
have
five
frameworks
which
have
used
less
than
their
fair
share.
So
let
me
send
the
offers
to
these
guys
now,
when
you
do
that
these
five
guys
will
not
accept
these.
F
Often,
if
they
don't
accept
this
offer,
then
they'll
decline
the
offer
and
implicitly,
if
you
do,
if
they
don't
do
anything
else,
they
are
actually
setting
a
declined
time
out
or
the
refused
time
out
to
five
seconds,
which
means
they
won't
be
receiving
any
offers
for
the
next
five
seconds
and
that
works
well.
The
allocator
then
goes
on
to
the
next
set
of
frameworks
to
offer
these
resources
and
the
next
set
and
so
on.
F
C
F
G
F
Okay,
so
on
the
on
the
Left,
what
you
are
seeing
is
this
matrix.
So
let
me
explain
what's
going
on
in
here,
so
the
first
column
is
the
framework
ID
in
this
particular
test.
We
launched
25
frameworks
and
there
were
I,
think
five
agents,
yeah,
so
25
frameworks,
five
agents,
and
we
actually
did
this
decline
timeout.
We
actually
do
since
to
make
a
synthetic.
F
We
set
the
decline
timeout
to
one
second,
so
every
time
you
decline
an
offer,
you
will
be
actually
also
telling
the
master
that
you
don't
want
to
receive
any
offers
for
the
next.
Second,
the
default
value
is
five
seconds,
but
this
makes
the
test
run
quicker
and
then
what
we
have
is
we
have
the
some
trimmers
which
are
actually
accepting
a
fraction
of
the
offers
that
is
sent
to
them
and
others
are
not
accepting
any
offers.
So
in
this
table
that
we
are
seeing
the
first
column
is
the
framework
ID
zero
through
24.
F
The
second
column
is
the
total
number
of
offers
that
this
framework
received.
The
third
column
is
the
total
number
of
offers
that
it
accepted,
and
the
fourth
is
the
total
declined.
So,
interestingly,
what
you
see
is
the
first
framework
because
it
accepted
an
offer
like
it
declined
the
first
offer,
but
as
soon
as
it
accepted
the
like
one
of
these
offers
and
started
using
those
resources,
it
never
received
any
more
offers
back.
The
same
thing
happened
with
framework
number
one
and
two.
F
The
reason
why
framework
one
and
to
see
more
offers
is
because
they
accepted
the
initial
offer
after
quite
a
while,
so
they
reject
they
kept
rejecting
several
offers
and
then
at
some
point
they
accepted
his
offer
as
soon
as
they
accepted
the
offer,
then
never
got
any
new
offer
back
and
the
frameworks
from
number
three
all
the
way
down
to
number
24.
They
were
the
ones
which
were
repeatedly
offered
the
resources
and
they
kept
repeatedly
declining
the
resources.
F
So
that's
pretty
much
the
the
offer
starvation
bug
that
we
are
looking
at
and
mang
actually
is
working
at
the
patches
for
addressing
them
using
the
the
exhaustion
oeufs
the
exhaustion
testing.
So,
for
the
benchmarking,
what
we
are
planning
to
do
here
is
we'll
actually
define
some
sort
of
goal
state
and
the
idea
will
be:
let's
run
this
benchmark
and
see.
C
It's
actually
a
little
bit
more
complicated
than
we
thought
and
it's
a
little
it's
a
little
bit
not
clear
how
it's
going
to
fit
in
if
we
make
rolls
hierarchical.
So
it's
a
little
concerning
for
that
reason,
but
I'll
kind
of
talk
about
the
different
options
we
have
and
folks
and
give
some
feedback
there
on
the
email
thread.
G
C
C
There's
like
there's
like
a
lot
of
slave
ID,
copying
being
done,
and
it
actually
shows
up
quite
heavily
in
the
in
the
profiles
that
he
got
so
as
part
of
just
running.
This
benchmark
also
just
found
some
room
for
improvement.
It
probably
shows
up
in
some
of
the
other
benchmarks
too,
but
I
think
now
we'll
probably
take
a
look
at
those
and
see
if
we
can
fix
some
of
the
performance
problems
in
there,
but
the
performance
is
kind
of
orthogonal
to
this.
This
problem,
this
is
going
to
happen,
even
if
allocator
is
really
fast.
C
A
C
A
I
think
it'd
be
pretty
cool
for
me
to
it
like
I
haven't.
Looked
it
looked
quite
briefly
at
the
dark.
I
might
study
that
significantly,
but
maybe
I
could
write
something
which
will
sample
those
metrics
over
a
few
hours
anonymizer
to
produce
some
kind
of
table
or
some
kind
of
parameters
that
we
could
then
feed
in
to
this
benchmark.
C
C
So
that's
kind
of
what
I
ended
up
doing,
but
without
using
the
benchmarks
or
the
metrics.
We
just
kind
of
we
we've
been
running
a
particular
set
of
scale
tests
where
frameworks
behave,
particular
ways,
and
so
we've
kind
of
encoded
one
of
the
worst
cases
here
in
this
benchmark,
where
everything
is
just
declining
without
any
large
timeout,
and
they
don't
suppress,
there's
still
issues
even
if
they're
trying
to
be
good
citizens,
but
this
particular
benchmark.
C
So
yeah
we're
kind
of
looking
at
the
worst
case
here.
Hopefully,
that's
sufficient
and
in
the
future
we
could
start
modeling.
Other
setups
metrics
would
probably
help
us
figure
out
what
what
those
setups
look
like
like
based
on
how
how
they
decline,
how
they
suppress
and
so
on,
but
I
kind
of
hope
that
we
don't
have
to
I.
C
C
Yeah
I
think
us
us
making
allocation
decisions
is
I,
don't
know
how
sustainable
it
is.
We
might
have
to
move
to
optimism
at
some
point
before
this
gets
too
out
of
hand,
but
for
now
we're
just
trying
to
address
the
the
current
behavior
and
try
to
make
sure
that
it's
a
little
bit
better
for
these
cases
where
things
are
starving.
C
C
A
E
Yeah
so
I
think
DRF
yeah
for
us
it's
basically
the
same.
It
is
not
add
a
lot
of
value
in
most
cases
and
it
actively
prevents
frameworks
from
getting
offers
in
other
cases,
so
I
think
round-robin
would
definitely
be
an
improvement
in
I
hope
most
cases
for
us,
but
I
also
think
that
the
request
base
model
would
probably
be
helpful
because
in
most
cases
we
know
exactly
how
many
resources
we
need
yeah,
so
that
would
that
would
definitely
have
us
for
other
cases.
E
C
It
might
be
a
little
bit
too
I
more
higher
than
it
should
be,
but
it's
like
we
can
get
kind
of
close
to
what
their
demand
should
be,
hopefully,
but
I'm
hoping
that
I'm,
hoping
that
before
we
build
too
much
complexity
into
the
current
model,
we
can
consider
migrating
towards
a
more
optimistic
model
where,
in
a
non
revocable
world
at
least
we
let
we
just
let
it
be
a
free-for-all
where
we
impose
constraints
like
you
have
a
limit,
or
you
know,
there's
like
a
opt
in
attribute
here
that
you
can't
get
unless
you're
off
to
dinners
and
things
like
that.
C
E
C
A
C
E
And
so
the
exhaust,
if
nacelle
it's
a
bit
like
I
mean
it
is
somewhat
similar
to
setting
filters
right
yeah.
So
we
have
had
issues
in
the
past
with
setting
too
many
filters
where
the
allocation
times
just
skyrocketed
and
okay,
but
it's
definitely
something
that
we
have
to
be
aware
of
like
we,
unless
you
find
a
magic
solution
for
that,
then
I
would
be
perfectly
fine
with
it,
but
we've
certainly
seen
problems
with
filters
in
the
past,
and
so
we
can't
use
anything
like
that.
If
it's
not
like
super
performant,
okay.
C
C
We
have
to
have
a
per
agent
set
of
the
rows
and
frameworks
that
have
been
allocated
to
for
that
agent
and
that
has
to
fill
up
essentially
and
then
get
cleared.
I
have
seen
some
performance
problems
as
well
with
filters
where,
because
they're
resource
subset
based,
you
can
actually
build
up
a
lot
of
different
filters
on
a
single
agent,
and
that
starts
to
get
really
expensive,
because
we
have
to
check
all
those
filters
and
they're
all
like
slightly
different
subsets.
E
I
would
assume
only
one
okay,
because
what
can
happen
we
did
not
fit
your
face
on
any
resource
type
or
anything
else.
We
just
checked
just
as
much
offer
match.
If
it
doesn't,
then
we
just
gonna
figure
it
for
the
next
round
or
fall.
Okay,
however,
and
is
there
things
like
reservations
involved
or
no
that
that
was
way
back?
That
was
okay,
no,
maybe
to
two
and
a
half
years
back.
Okay,.
C
Okay,
so
it
shouldn't
be
a
lot,
but
it
could.
It
could
still
be
several
given
that
we
could
start
sending
you
some
amount
and
then
madam
could
grow
a
little
bit,
but
it's
still
not
enough
and
so
on,
and
each
time
would
be
different
like
there
are.
Some
of
them
are
subsets
of
each
other,
but
we
don't
actually
check
for
that.
We
actually
store
all
like
if
you,
if
you
filter
for
a
superset
of
what
we
previously
have
the
filter,
for,
we
won't
get
rid
of
that
earlier,
subset
filter.
C
We
would
leave
it,
so
we
would
have
to
at
that
point
so
that
could
be
happening
but
yeah,
maybe
maybe
for
you
guys.
It's
a
smallish
number
I
think
hopefully,
now
that
we
have,
you
know
things
like
flame
scope.
It
would
be
great
if
you
guys
ever
run
into
any
high
utilization
or
performance
problems,
just
grabbing
a
flame
scope
and
then
sharing
it
with
folks,
and
we
can
all
drop
it
into
the
tool
and
take
a
look
at
what's
happening.