►
Description
https://wiki.ceph.com/Planning/CDS/CDS_Giant_and_Hammer_(Jun_2014)
24 June 2014
Ceph Developer Summit G/H
Day 1
MON: dispatch messages while waiting for IO to complete
B
Okay,
so
this
is
basically
what
I
believe
we
discussed
much
of
this
back
on
the
last
CD
s.
B
The
thing
is,
we
barely
hit
any
work,
I
I,
don't
think
we
actually
did
any
work
for
about
them
the
whole
having
the
monitors,
dispatching
message
as
well,
independently
and
whatnot,
and
likely
we've
seen
huge
issues
with
monitors
under
heavy
workload
being
getting
just
stuck
while
performing
operations
on
my
believing
and
those
those
waiting
times
tend
to
basically
have
the
monitors,
wiping
all
the
way,
though
we
really
really
need
some
way
to
to
have
the
monitors
continuing
their
work,
especially.
B
Replying
to
who
please
messages-
and
you
know,
while
waiting
for
operation
is
to
finish
prom
leveldb-
that
I
spoke
to
with
such
last
week
and
the
current
the
current
idea
is,
to
just
add
a
new
pixels
tight,
such
that
the
once
we
get
that
state.
We
really
increase
control
to
another
threat
that
will
continue
doing
the
housekeeping
staff.
B
C
Yeah
so
I'm
just
to
give
it
a
tiny
bit
more
background,
so
that
the
specific
issue
that
that
is,
that
we're
seeing
we
have
it.
We
need
a
big,
especially
big,
stuff
cluster
lots,
lots
of
pg's
and
that's
under
load.
Then
you
have
a
proportionately
higher
number,
just
more
ug
map
updates
that
are
flowing
through
the
monitor,
and
so
those
update
transactions
can
be
just
big,
so
I've
seen
50
megabyte
like
Paxos
updates.
C
That
would
take
a
second
or
two
to
commit,
and
then
another
couple
seconds
to
like
reread
out
of
the
local
DB
cache
or
whatever,
to
update
the
memory
state.
The
first
obvious
thing
is
just
we
can
not
block
message
processing
while
we're
waiting
for
that
I
owe
to
complete
and
I
think
additional
steps
would
be
to
you
know,
deal
with
the
move
to
like
reader
writer
locks
or
something
like
that,
so
that
we
don't
so
we
can
update
that
other
information,
but
I
think
just
starting
with
the
I/o
part
is
positive,
simplices
right
now.
C
B
This
doesn't
work
well
when
I,
oh
sorry,
I
could
to
the
concerning
part.
Is
that
it?
It
doesn't
happen
just
for
50
megabytes
transactions.
It
can
also
happen
for
mobile
transactions.
As
long
as
there
is
a
compaction
running
on
the
background,
because
rights
will
block
until
the
compaction
finishes
and
that
can
take
depending
on
the
sore
and
on
the
the
work
on
the
bride
wore
clothes.
It
can
take
one
minutes
licensed
in
during
the
day.
B
I,
almost
honestly,
don't
know,
usually
you'll
have
rights
going
through
I
believe
it's
mostly
when
the
right
buffer
gets
clogged
or
something
I.
C
B
C
B
Loverly
be
compacts
level
zero.
It
will
block
I'm
honestly,
not
not
sure
whether
it
happens
when
other
levels
are
being
blocked
as
well.
C
Don't
even
think
we
need
to
worry
about
threads
at
this
point.
It's
just
a
matter
of
making
it,
so
we
submit
the
right
and
it
off
to
the
key-value
store
whatever
and
then
just
wait
for
the
call
back
teaching
processing
messages.
We
just
need
to
have
a
state
so
that
we
can
continue
doing
work
while
we're
in
that
for
that
period,
while
we're
waiting
for
the
callback
for
the
right
to
complete.
B
C
C
C
B
Right,
we
can
simply
have
a
threat
that
will
run
faxes
transactions
and
that
thread
can
walk
at
will.
Instead,
it
just
working
around
the
whole
the
whole
sink
and
a
sink
their
hands
actions.
So
we
just
have
delegate
that,
but
would
push
that
to
the
queue,
because
we
already
have
that
cue
and
have
this
one
thread
grab
the
head
of
the
queue
propose
it.
B
B
May
not
even
need
the
river
to
read
fight
to
talk
really
because
we
just
have
to
well
yeah
sure
we'll
have
to
have
that
feedback
lot,
but
just
protecting
that
queue,
because
all
the
services
will
have
to
push
to
have
that
nurse
tied
to
that
cue
to
the
tail.
So
we
just
have
to
protect
access
to
the
tube.
B
Right
we
yeah,
we
actually
will
have
to
change
the
way
we
lock
and
unlock
on
dispatch,
because
there's
a
certain
amount
of
messages.
Well,
the
Praxis
messages
that
will
have
to
go
to
that
thread
instead
of
being
handled
by
the
monitor.
So
we
will
have
to
grab
the
monarch
dispatch
those
messages
to
to
that
thread
and.
D
D
B
Yet
so
give
and
we
grab
that
block
and
we
dispatch
something
to
the
the
threat
and
only
unlock
when
dispatch
finishes.
We
end
up
with
the
same
pretty
much
the
same:
the
same
behavior
we
have
right
now,
which
is
single
threaded,
regardless
of
their
threat,
being
run
concurrently
to
the
remaining
workflow.
So
we
really
need
to
dispatch
those
messages
and
relinquish
the
lock,
but
that
the
new
message
can
be
handled.
C
Okay,
yeah
I
guess
the
thing
worries
me.
I
want
to
make
sure
that
what
we
do
here
is
is
on
the
path
towards
what
are
the
larger
changes
that
we're
eventually
going
to
want
to
make
where
I'm.
At
the
end
of
the
day,
we
want
to
be
able
to
satisfy
any
sort
of
reads
for
the
packs
of
state
without
blocking
the
rights
would
be
able
to
be
prepared
and
sequenced
and
obviously,
but
and
I'll
bet
all
the
message.
Processing
shouldn't
block
on
the
underlying
I/o
subsystem.
D
C
Yeah
I
mean
the
thing
that
the
thing
that
worries
me
basically,
is
that
if
you
look
at
there's
are
all
rights
goes
funnel
through
apply
transaction
and
if
you
grep,
/,
apply
transaction
in
the
tree.
They're
like
15
instances
in
monitor,
which
are
mostly
okay,
because
they're,
mostly
around
all
the
initial
sync
stuff
on
startup,
and
we
don't
care.
If
we
block
make
a
vest
and
stuff,
I
don't
care
free
blog.
There,
they're,
like
eight
of
them
in
packs
of
CC
on
those
ones,
I
think
do
matter.
B
So
yeah,
the
no
one
thing
to
take
to
be
careful
about
is
that
even
the
smallest
of
Rights
can
trigger
a
huge
Aug
waiting
time
because
level
we
may
decide
to
compact
or
may
already
be
contacting,
and
we
will
wait
on
decks
on
that
compaction
to
finish
so,
even
even
just
updating
a
small
version
or
as
the
last
chemical
or
something
or
even
a
PN
proposal,
number
of
it.
Just
a
couple
of
bites
may
very
well
end
up
waiting
a
lot
of
time
and.
B
Although
it's
most
common,
its
most
common
to
be
figured
when
there's
a
heavy
right
workload
with
bigger
transactions
better
than
one
megabyte
because
the
well
you
fill
up
the
buffer
quicker
and
the
I/o
is
bigger
overall,
so
it's
not
as
common
for
lower
values.
So
if
we
fund
that
that
stressed
escalate
their
values
up
to
a
couple
hundred
blue
bites,
it
will
not
be
as
obvious
oh
yeah,
yeah.
C
Guess
want
to
make
sure
that,
while
that
is
actually
happening,
none
of
the
other
events
can
happen
right
and
when
it,
when
that
background
thread,
does
drop
the
lock
effectively
when
it's,
when
it's
doing
that
big
update,
it's
the
idea
that
any
other
Paxos
message
would
just
sort
of
get
queued
up
the
queue
that
you're
talking
about
before,
and
so
they
would
all
sort
of
yeah.
It's
real
eyes
on
that.
But
meanwhile
everything
else
could
continue.
B
As
a
new,
the
whole
idea
is
that
in
this,
this
could
be
the
cornerstone
for
a
better
message
handling
on
on
the
monitor.
As
long
as
we
keep
it
such
that
right
now,
we
dispatch
all
pixels
related
messages
to
one
given
thread
that
will
handle
those,
and
basically
all
this
box
happen
independently
from
the
other
messages
and
as
long
as
the
other
messages
do
not
trigger.
B
C
B
B
B
C
B
C
B
C
B
C
C
B
B
Ok,
so
for
ideally,
we
would
have
a
different
thread
dealing
with
all
the
messages
that
would
not
clash
with
or
hinder,
or
that
could
eventually
hinder
progress
of
other
messages
being
handled.
That
would
be
the
ideal
case
for
a
first
step.
It
seems
that
we
can
just
create
a
thread
that
will
handle
the
transactions
and
build
from
there,
because
that
should
be
easier
than
doing
the
whole
a
new
thread
dispatching
all
the
messages
and
either
in
at
least
we
will
have
a
sense
of
whether
that
is
worth
pursuing
or
not.
C
D
B
C
It
basically
does
apply
transaction
and
then
at
the
very
it
does
like
four
things.
Afterwards,
it
rereads
first
committed
after
the
transaction
is
applied.
It
calls
sanity
check,
which
I
think
is
really
matter
and
then
removes
legacy
versions
which
by
doesn't
matter,
but
the
problem
is
the
problem.
Is
that
I'm
a
mean
the
main
store
state
collar
is
and
handle
commit,
which
just
happens
on
the
on
the
peons,
where
you're,
just
like
handing
it
stuff
and
so
I.
C
Think
in
that
case
it's
pretty
simple
you're
just
you
just
have
a
completion
that
would
do
that
refresh,
but
they're
a
bunch
of
other
store
states
elsewhere.
That
I
think
wouldn't
take
so
kindly
to
being
a
synchronous
like
in
handle
last
like
during
the
initial
Paxos
exchange,
those
ones.
Maybe
we
can't?
Oh
that's
the
only
to
grip.
C
B
May
have
to
have
a
lock
to
protect
the
in-memory
site
when
we
finish
applying
the
transaction.
If
we
want
to
keep
this
sort
of
stuff
installed
in
there
in
source
type,
because
that
there
might
be,
you
may
be
a
way
to
get
well
to
split
salt
door
side
into
the
portion
that
is
committed
by
the
yellow
transaction
right.