►
From YouTube: 2017-SEP-06 :: Ceph Developer Monthly
Description
Monthly developer meeting for the coordination of Ceph project development.
http://tracker.ceph.com/projects/ceph/wiki/Planning
A
B
C
Okay,
on
in
the
last
months
is
CDM.
We
we
talked
about
the
Raiders
level
replication
and
a
sage
that
just
had
that
and
a
main
main
problem
of
implementing
this
replication
is.
We
have
to
maintain
straight
across
a
watering
and
so
I.
We
did
some
details.
We
did
some
considerations
on
the
details
of
implements
of
maintaining
cross
orderings,
and
this
is
the
first
issue
on
I'm
gonna
talk
about
I,
don't
know.
Can
you
hear
me
clearly.
A
C
Okay,
okay
thanks.
Secondly,
we
talked
about
how,
after
maintaining
the
course
altering,
we
have
to
a
guarantee,
the
replication
of
the
correctness
when
there
are
OSD
failures,
and
we
did
some
more
of
other
considerations
and
that's
the
third
issue,
and
that
even
started
with
the
first
SH
well
suggested
last
last
month
that
the
for
implementing
for
maintaining
coring,
we
can
split
the
whole
system
runtime
into
a
series
of
translate
it
and
all
client
operations
during
the
sense
unsliced
are
in
the
sent
in
the
same
transaction.
C
C
You
violated
across
class
boundaries
if
given
understanding
him
called
Bradley.
This
should
be
suggestion
that
he
did
the
last
month,
though,
we
did
some
more
considerations
on
the
details.
We
think
that
implemented
such
mechanism
we
had
to,
though
we're
faced
with
two
major
problems:
the
first
days,
how
to
make
all
clients
pause
and
exactly
every
times
last
boundaries,
and
second,
and
the
second-
is
how
to
make
all
client
operations
in
the
st.
I'm
slice
are
they're
all
replicating
or
all
not
hurt
that
they
do
that's
their
fault
for
the
first.
C
For
the
first
problem,
I,
we
are,
we
think
much
we
can.
We
can
do
this
in
this
way,
our
first
on
every
times,
the
last
boundaries,
the
monitor
can
send
a
tunt
attempt
at
understand
to
to
all
clients,
to
say
it's
t,
TS,
pound
and
and
then
of
all
clients
that
their
post
timers
according
to
the
following
rules.
From
the
first
days
before
they
they
said
the
post
timers.
C
They
have
to
make
sure
that
their
local
system
class
had
to
be
synchronized
with
the
same
ntp
server
as
monitors
and
the
town
school
is
small
and
in
the
post,
timer
expiration
time
should
be
p.
Ts
bell,
+
Tom's,
lies
and
subtract
T
local.
This
means
that
this
this
should
be
the
time
that
they
gonna
course
in
the
future.
C
Then,
when
the
post
timer
expires
and
if
all
clients,
local
distant
clock
bill
satisfies
this,
miss
considered
miss
condition,
and
then
the
clients
should
pause
for,
for
a
period
of
time
to
say
it's
deport,
but
we
think
that
people's
has
to
be
larger
than
the
time
centralization
air
about
on
its
it's.
Just
like
this
picture
shows.
C
If
we
can
make
the
P
pores
larger
than
the
time
slice
at
the
time,
synchronization
Paragon
then
the
given,
if
they
are,
they
are
not
course
pushing
for
posting
at
the
everything
exactly
the
same
time
they
can
port
across
the
the
tons.
Sync
error
bound,
though
that
should
be
enough
to
maintain
watering.
C
C
On
the
other
hand,
though,
I
stays
in
the
master
cluster
are
should
periodically
reported
as
monitoring
that
of
their
abilities
replicated
client
Opie
on
each
each
client
operation
is
is
associated
with
tans
down
along.
If
all
the
clients
have
has
pores
have
finished
nests
have
finished,
the
port
is
that
they
said,
according
to
our
TDS
panel,
asked
and
all
the
Westies
have
started
to
replicate
client
operations
that
which
time
stem
is
later
than
PD
has
found
the
last
then
the
teeth.
C
In
a
meal,
in
the
meantime,
osp's
in
the
masterclass
keep
replicating
client
operations
to
the
back
to
the
backup
cluster
I.
Despite
the
tons
last
constraints,
they
just
keep
replicating
and
when
they
receive
this
whenever
they
receive
the
tongue,
the
confirmed
time
/
boundaries
from
the
monitors
in
the
master
cluster
and
then
in
before
they
before
they
received
before
they
receive
this
in
the
stands.
Last
memories:
they
just
cashed.
C
Yeah
yeah,
it's
just
like
this
signed
him
and
so
I
put
it
together.
Would
we
we
think
we,
maybe
maybe
we
can
implement
the
whole
for
a
replication
mechanism
just
as
this
picture
shows.
First,
the
monitor
send
a
TS
bound
to
all
the
clients.
Then,
after
all,
the
clients
has
been
has
post,
they
send.
The
TT
has
found
the
last
to
the
monitor.
In
the
meantime,
all
OSDs
report
to
the
monitor
off
their
latest.
C
Right,
the
general
branding
operations
back
to
their
back
and
store,
and
at
this
time
the
the
replication
of
that
often
the
operations
in
this,
in
particular
on
its
lies,
should
be
finished,
but
we
think
if,
if
this
is,
if
this
is
right,.
C
C
The
say
our
clients
said
their
timer
to
expire
at
10
seconds
later,
but
if
this
time,
if
this
ten-second
time
isn't
can
can
be
influenced
by
the
time
by
the
time
synchronization
service,
then
they
may,
they
may
be
only
waited
for
nine
seconds
and
then
the
Tamarack
expired
and
they
thought
they
have.
They
have
waited
for
10
seconds
and
that
that
will
make
make
the
replication
go
wrong
and
the
second
is:
if
one
client
failed
to
pause
correctly,
then,
and
then
this
time
slice
should
be
merged
into
the
next
time.
E
B
E
C
C
In
the
last
issue,
we
would
consider
this.
Oh,
we
think
that
on
making
all
clients
to
posterior
periodically
since
not
necessary
system,
because
we
may
be
only
only
a
few
of
them
need
to
be
a
neat,
they
need
be
beneath
the
point
in
time,
consistency
or
that
maybe
we
can,
we
can
give
declines
an
option
to
identify
themselves
as
whether
they
need
the
point
in
time,
consistency
and
then
only
those
who
need
this
consistency
need,
of
course,
I.
Don't
know
whether
I'm
saying
the
same.
The
problem
that
you
are
talking
about.
B
E
I
think
I
think
there.
So
this
is
this
is,
if
I
understand,
I've
missed
the
very
beginning,
but
if
I
understood
correctly,
this
sounds
almost
exactly
like
what
the
clinic
team
looked
at
a
couple
years
ago,
and
they
they
were
the
proposal.
There
was
to
do
the
same
delay.
Where
you
have
you.
Basically,
everybody
has
that
tea
pause,
I,
guess
that
you're
calling
it
so
that
so
that
you
have
that
that
bound
whatever
I,
don't
know
what
words
you're
using,
but
the
difference
is
they
were
looking
at
just
doing
it
on
the
OS
DS.
E
So,
basically,
all
the
OSD
s
would
have
a
delay
of
based
on
what
the
NTP
error
bounds
are,
so
that
there
was
prevent
any
causal
ordering
between
operations,
and
then
they
did
some.
They
did
some
simulations
with
different
network
latencies
and
NTP
like
protocols
to
see
what
how
big
that
delay
would
have
to
be.
They
wrote
a
report,
I'm
happy
to
email
it
to
you.
E
It'll
probably
explain
it
in
more
detail
more
clearly
than
I
can
on
the
spot,
but
I
think
the
biggest
difference
is
just
that
they're
doing
it
on
the
OS,
DS
and
I.
Think
one
of
the
other
kind
of
nice
things
about
that
approach.
Is
that
do
C's,
don't
actually
have
to
stop
doing
work.
They
can
still
process
the
requests,
but
if
they
just
delay
the
replies
back
to
the
client,
then
they
prevent
any
causal
dependency
of
some
other
operation.
C
E
C
E
C
C
C
So
if
the
original
okie
journal
gets
removed
after
is
cresting,
the
corresponding
original
hope
is
replicated,
then
the
the
second
condition
is
this:
the
second
is
in
the
recovery
and
back
or
backfield
phase
recovery,
sauce
a
replicate,
all
journals
related
to
the
recovering
object
before
they
pushing
the
object.
If
these
two
conditions
are
guaranteed,
then
we
think
that
their
application
correctness
of
a
single
object
should
be
guaranteed
and
I
don't
know
if
this
is
right.
F
C
F
In
the
it's
possible
for
us
to
overwrite
data
in
the
OS
in
the
OS,
these
data
store
right.
So
it
might
actually
be
gone
by
the
time
you're
trying
to
read
it
back
out
unless
you
have
it
persistent
somewhere,
but
I
think
you
basically
need
to
either
do
a
full
op
data
copy,
which
there's
no
mechanism
for
in
the
OS
team.
That
would
be
really
expensive,
or
else
we
need
to
have
some
kind
of
snapshotting
system.
That's
orthogonal
to
what
we
used
right
now
that
likes
store
all
the
times
light
as
you
care
about
yeah.
G
E
B
F
E
E
Mean
in
principle
all
the
at
all
does
all
the
data
path
stuff
can
do
copy
and
write
the
key
value
stuff
doesn't
so
that
would
probably
just
be
yeah.
We'd
have
to
do
something
different
way
of
doing
to
keep
the
idea,
but
for
data
it'll
kind
of
mess
with
like
the
block
allocation
stuff,
because
it
you
can't
like
overwrite
any
blocks,
you
keep
allocating
new
stuff
and
make
it
maybe
more
fragmented,
but.
E
C
E
G
E
I
mean
I
copy
and
write
I.
Guess
it's
not
it
you'd
want
to
you
basically
write
you'd
make
it
so
that
all
of
the
blocks
that
are
referenced
at
that
point
in
time
are
effectively
frozen.
There's
this
yeah,
you
could
have
a
lot
of
reco
listing
of
rice
right.
If
you
do
this
snapshot
approach,
as
opposed
to
yeah.
E
E
Read
it
it
should
only
it
only
really
affect
so
the
way
blues
store.
Does
it
the
anything
it
only
affects
stuff
below
them,
italic
size,
which
is
like
c4k
to
16
K
somewhere
in
there.
So
it
doesn't
make
that
big
of
an
impact.
I
guess
unless
you
have
like
a
random
small
write
workload,
but
even
that
it
something
I'll
just
get
hidden
in
the
noise
anyway,.
I
I
E
J
Okay,
okay,
so
actually,
since
last
CDN
I
mean
over
stadium.
We
have
some
updates
on
the
general
architecture
and
also
we
have
some
new
design
for
the
cache
constants
you
see
so
here's
the
general
architecture.
Basically,
we
have
three
components
here.
First,
one
is
on
a
leaf
cache
store,
which
would
be
a
common
library
that
can
do
read,
write
on
SSD.
A
second
one
is
a
standalone
cache
demo,
which
owns
private
policy
that
controls
the
promotion,
emotion,
sings
country.
J
We
have
a
very
simple
area
based
Parsi,
but
we
can
actually
design
more
fancy
things
like
sar
you
or
to
kill
things
like
that,
and
the
last
one
is
the
hooks.
Actually,
we
have
a
very
light
hook
inside
the
body
that
can
call
api
of
libcast
file.
With
that
we
can
redirect
those
we
right,
the
memcache
store.
J
So
we
do
cash
promotion
on
weeds
and
own
rights.
We
just
to
cache
invalidation
and
also
all
of
the
rights
right
forecasts
were
going
to
read
us
to
actually
and
there's
some
missing
missing
features
in
that
PR
and
country.
We
only
have
a
very
happy
data
pass
and
all
of
the
share
cash
I
mean
for
the
parents,
image
cash
are
promoted
on
the
opening
of
first
child,
so
missing
things
we
have
to
design
and
promote
don't
amount
mechanism
to
make
sure
the
cash
is
is
only
stored
with
what
we
needed
and
also
we.
J
Okay,
so
this
is
the
initial
results,
and
actually
we
have
the
data
pass
implemented.
So
you
can
see
from
the
table
here
with
actually
with
a
very
small
cache
size.
You
can
get
almost
of
the
same.
J
J
J
So
here's
our
new
custom
own-
we
have
been
thinking
about
this
for
a
few
days.
There
are
actually
several
key
components
here.
The
the
general
idea
is
all
of
the
read
my
blocks
from
the
protected
snapshot
cached
in
our
shared
area,
only
compute
nodes,
for
example.
We
have
a
VM
e
SSD
mounted
at
to
some
folder,
and
then
we
store
those
shared
contents
to
to
that
folder
and.
J
On
each
computer
knows
there
will
be
a
cache
demon
to
control
the
shared
cache
state,
for
example,
if
one
vini
vidi
instance
want
to
read
from
some
specific
offset
it
just
score.
Is
that
customer
with
an
IPC
mechanism
and
and
customer
tell
if
it
is
there
if
it
is?
If
the
cache
is
already
there,
then
the
IBD
instance
were
trying
to
acquire
some
reader
lock
and
then
just
open
that
file
and
read
the
content.
J
J
H
Like
independent
mailboxes
between
each
already
instance
and
the
cache
team
and
you'd
still
have
to
have
some
way
that
basically
ping
and
prog
to
each
other
to
say:
hey
I,
put
something
there
in
the
mailbox
or
whatever,
and
that
would
involve
a
lock
realistically
which
crashed
having
it
locked,
and
you
know,
let's
think,
but
at
least
you'd
be
I,
guess
the
private
lock
that
when
that
instance
goes
away,
it
comes
back
and
Bree
establishes
brand
new
mailboxes
and
things
like
that.
But
it's
all
them
surmountable
I
think
it's
essentially
challenging
the
messages.
E
E
E
H
You
would
I
think
the
only
messages
you
need
to
go
to
would
be
basically
saying
like
hey.
You
know,
it's
a
like
hey
like
make
sure
that
you're
marking
this
in
your
policies,
like
the
hot
object
or
whatever
and
then
also
at
some
point,
the
demon
says
no
I,
don't
have
it
or
we
got
to
promote
it.
Here's
what
it
righted
two
or
whatever
go
ahead
and
do
it.
You
know
type
of
thing,
but
even
but
that
would
just
be
I.
Imagine
a
straight
file.
J
Yeah
I
see
I
think
I
can
have
a
try
with
local
socket
to
test
to
the
latency
okay.
So
we
also
have
a
so
country
we're
trying
to
do
a
stateless
caste
mo
here,
and
so,
if
the
demon
crashes,
we
sync
all
I
need
to
restart
that
demon
and
demon
where
try
to
load
load
the
matter
file
for
for
this
year
within
the
cache
and
then,
if
the,
if
the
load
fails,
we
just
starts
from
an
empty
empty
cache.
J
J
Okay,
so
here
here's
some
details
for
the
promotion,
commotion
and
read
me
right.
So,
for
example,
if
there's
a
really
request
from
a
be
warm
and
then
just
try
to
look
up
in
the
caucus
mapping,
basically
this
will
be
a
copy
of
the
object
map
which,
which
can
tell
if
the
if
the
reader
content
is
is
in,
is
only
call
cop,
hat
or
or
we
were
safe
to
read
from
the
parent
humid
cache
yeah.
H
H
The
time
you
don't
even
have
to
read
it,
the
cache
doesn't
have
to
care
about
it,
cuz
it's
out,
because
if
you
imagine
that
the
images
you
have
a
parent
image
and
they
can
form
a
chain
of
clones,
okay,
so
the
the
read
and
write
requests
are
issued
from
the
let's
Akeem
you
against
the
clown
image.
The
clone
image
is
what
does
and
these
and
when
I
say
image
in
memory.
H
I'm
talking
about
the
Lib
RBD
image
context
yeah,
if,
if
the
clown,
the
lowest
level
in
the
chain,
determines
that
hey,
you
know
the
my
object
map
says:
I,
don't
have
this
data
you
know
associated
with
me.
It
automatically
then
riri
issues
that
read
request
to
the
the
next
parent
in
the
chain
and
so
on
and
so
forth
up
the
chain
until
some
parent
can
satisfy
that
request.
So
the
cache
is
tied
to
the
parents,
image
contact.
So
by
the
time,
the
parent
image
context,
the
I/o
path
of
the
parent
image
gets
a
read
request.
H
H
J
And
so
that's
actually
the
step
two
here,
so
the
IBD
image
will
occur
in
the
customer
and
then
custom
own
folder,
like
you'll,
see
you'll,
see
policy,
here's
a
private
policy
to
determine
if
we
should
promote
or
we
should
that's
the
ability
to
read
from
the
back
end
directory.
So
if
it
decides
to
promote
that
object,
we
actually
have
a
right
to
lock
on
the
only
cash
and
then
I
just
promote
from
the
raddest
here
and
country.
J
We,
we
have
a
very
small,
optimization
here,
so
when
you're
doing
promoting
and
we
can,
we
can
actually
configure
the
promotion
unit.
For
example,
we
can
promote
only
hk4
for
that
single
request,
but
but
for
a
GW
caching,
where
we
just
promotes
the
four
object
here
when,
when
the
promotion
is
down,
we're
just
a
simple
and
notify
the
image
to
to
read
it,
maybe
a
core
back.
J
J
Okay,
so
here's
the
IO
flow
for
weed
and
basically
the
flow
is
quite
similar
with
the
promotion
flow
and
the
image
would
look
up
in
the
table
first
and
then
try
to
curry
the
customer,
and
if
the
cash
is
already
in
the
initial
folder,
we
just
acquire
some
gridlock
on
that
cache
and
then
check
metadata
and
then
read
from
the.
If
it
is
already
there,
then
we
can
read
from
the
share
cache.
J
J
J
We
have
think
about
the
the
first
one
is
on
how
to
do
a
VM
migration
and
also
how
to
handle
VM
crashes
and
so
basically
over
vm
migration.
We
have
where
we
have
to
rebuild
rebuild
the
cache
state
when,
when
the
ability
is
reopening
bug
them,
how?
If
we
have
the
shared
cache
on
that
new
computer
note,
and
then
we
can
try
to
read
from
that
chair
cache.
If
we
don't
have
that
shared
cache,
we
are
just
promotes
own
demand.
I
mean.
H
H
Nobody
liked
the
the
cash
date.
It's
not
something
that
should
be
in
memory
of
the
the
Camilla
Barbie
D
process.
Right,
it's
the
cash
there's
the
demon
and
it's
you
know
the
files
that
exist
on
the
SSD
so
and
you're
not
I
mean
you're,
not
gonna,
promote
the
full
parent
image
on
startup
or
anything
like
that.
You're
gonna
wait
for
the
policy
to
say
yeah.
J
J
Yeah,
that's
right!
That's
right!
And
yes,
second
Casey's
Jason
actually
commented
only
one
six,
seven,
eight
eight
PR.
So
there's
a
case
that
seemed
our
API
note
were,
will
be
a
separate
node.
So
there's
a
case,
some
abhi
DS
will
be
removed
from
the
remote
node.
So
we
have
a.
We
have
a
policy
thread
that
in
the
custom
own
can
check
the
cash
if
the
cash
is
very,
very
cold
and
we
it
will
be
removed
eventually,
so
we
don't
have
any
offering
cash
there
and
sir
case
is
custom
crash.
I.
H
H
Whatever
your
tribe
attendee
uses
your
policy
and
I
mean
not
that
you're
updating
on
just
February
every
ping
you're
getting
but
like
he
was
periodically
persisting
it
so
that
you
can,
in
theory,
boot
up
it
to
back
to
a
warm
state
as
opposed
to
starting
from
scratch
and
having
to
scan
the
entire
file
system
or
whatever,
to
say
what
you
know
to
figure
out.
What's
out
there
or
I,
don't
know
how
you
keep
your
metadata.
L
L
Okay,
for
the
second
point,
I'm
thinking
things
we
already
got
a
parent
demon,
so
is
it
possible
when
we
want
to
remove
it?
We
just
need
to
contact
the
parent
and
things
no
matter
the
shared
cache
or
the
local
cache
is
always
on
the
same
note
on
the
same
folder,
so
we
can
just
remove
those
data,
so
we
don't
need
to
do
the
periodically
check.
H
H
L
Are
you
eviction
thing,
because
those
cache
is
already
there
they're?
Actually,
image
is
already
removed,
so
in
in
that
case,
I
just
need
to
contacts
apparent
and
led
the
parent
to
delete
the
child.
The
cloned
cash
I'm,
though
another
another
thing
we
need
to
do
is
to
just
to
remove
those
slow
low
heat
ratio.
Cash
and
air.
Are
your
eviction
so
there's
different
operation
in
different
face
I
mean
it.
One
is
when
the
image
is
still
exists.
Another
is
when
the
image
is
already
removed.
J
I
think
the
last
one
is
more
difficult,
because
there's
there's
a
case
that
there
may
be
some
errors
on
the
file
system
or
only
SSD.
So
contrary
we,
we
don't
have
any
mechanism
to
check
check
those
integrate
a
those
far
integrator
of
those
files.
We
simply
rely
on
the
fascism
Tallulah
check.
That
means,
if
there's
some,
we
ever
happen
when
on
weed.
We
seen
probably
ignore
that
and
we
issue
I
read
from
Redis
and.
H
H
E
And
you
can
write
to
a
temporary
file
that
sync
it
and
then
rename
that's
sort
of
a
usual
thing.
Then,
though,
you
know
it's
complete,
I
think
the
only
issue
with
the
reads
is:
you
have
to
be
a
little
bit
careful
because
you're
in
mapping
the
file
and
if
you
have
a
page
fault
that
gets
an
IO
error
and
turns
into
a
signal
I
believe
that's
how
that's
communicated
the
process
is
a
you
Danny
fault
or
a
cig
fault
or
whatever.
L
E
E
Yeah,
then,
you
need
to
write
particular
blocks.
I
think
that
and
then
update
your
database
thing
whatever
it
is
Smita
file,
if
you're
doing
a
map
MF,
it's
like
gonna,
be
them
I,
think
if
that's
gonna
be
the
most
efficient
weights
as
far
as
like
a
void,
a
memory
copies
and
so
on,
because
everyone
will
be
sharing
the
same
pages.
But
if
there's
a
media
fault,
it's
also
the
most
awkward
go,
get
a
signal
and
things
will
just
crash
and
then.
H
For
the
demotion
addiction
process,
though
so
I
mean
you'll
have
to
have
like
you
have
to
take
a
file
lock,
right,
I,
guess
the
person
that's
doing.
The
eviction
would
have
to
try
to
get
it
right
or
lock
on
it,
because
they
won't
be
able
to
release
the
reader
lock
until
after
until
the
buffer
list.
So,
finally,
like
freed.
L
Guys
had
any
suggestion
here,
since
we
already
used
a
mm
map
so
they're
in
that
better
way,
like
maybe
I
we
create
and
release
it
then
create
new
ones,
something
like
this.
So
too,
we
can
make
a
checkpoint
thing
to
make
sure
like
yes,
maybe
after
before
the
we
create
a
new
one
as
those
already
being
stalled
in
the
file,
and
there
won't
be
a
problem.
So
maybe
in
this
way
I
can
we
can
make
this
much
more
consistent
or
more
safe.
E
L
To
to
update
some
metadata,
so
if
there
is
a
fearful
so
some
some
metadata,
maybe
just
a
messed
up
and
so
I'm
thinking,
I,
originally
thinking,
maybe
I,
just
use
em
think
so
I
can
make
sure
the
file
is
already
stored
on
the
disk
but
I'm
now
thinking
if
the
page
fails,
so
maybe
the
this
whole
4k
page
may
be
just
too
messed
up
and
did
their
is
useless.
So
the
better
way
is
I
always
like
use
a
parrot
thing
to
new,
a
new
file
and
the
Tuesday.
L
E
E
E
H
E
J
I
No
I
think
having
hey
you
and
Cindy
covered
it
pretty
well
Jason,
so
the
only
thing
is
is
basically
the
you
know
on
the
right
right.
You
know
basically
a
copy
on
the
right.
Did
you
did
you
already
mention
that
we
will
use
the
object,
my
app
feature,
and
so
will
there
be
an
assumption
that
will,
you
know,
will
need
object,
map.
E
E
E
E
The
only
other
idea
I
had
was
I'm
doing
periodic
trimming
proportional.
The
idea
here
is
that
say
every
second
you're
going
to
turn
all
the
PGS
here.
This
is
all
written
down.
I
can
post
in
here.
Okay,
I
want
to
read
it.
So
every
every
second
you
look
at
all
the
PGS
say:
I
want
I
want
over
all
the
OSD
I
wish.
I
had
you
know:
20,000
PG
log
entries
I
have
22,000,
so
that
means
I
want
to
trim.
What
is
that
10%
of
the
PG
log
entries?
E
E
That
was
actually
that
was
kind
of
the
first
thing.
I
thought
about
too,
but
I
couldn't
figure
out
was
how
you
decide
what
to
trim
up
to,
because
the
request
might
not
be
distributed.
Sort
of
uniformly
over
time.
I
guess
again,
you
could
just
Parks
I
mean
so
if
the
oldest
I
mean
be
knowing
how
to
start.
H
D
E
H
Was
like
if
everything
else
is
already
started
like
we're,
trying
to
keep
this
on
this
car
and
the
Sun
is
coming
this
way
and
let's
go
you're
already,
aligning
all
your
locks.
You
know
in
theory,
along
those
extents
that
this
kind
of
the
complication
this
it
seems
kind
of
natural
that
you
would
also
have
yeah.
F
F
Updating
the
shared
data
structure
that
says,
hey
like
justify
us
by
my
oldest
entry
or
like
I,
have
ten
that
are
older
than
60
seconds
or
something,
and
it
seems
like
you,
design
a
data
structure
like
that
to
just
gets
updated
every
several
seconds,
and
then
people
can
say:
hey
like
and-
and
you
can
also
put
in
there.
I
have
this
many
so
that
you
can
like
look
and
see
when
you're
when
you're,
adding
new
entries
like
you
can
or
you
can
pass
around
flags
that
say
hey.
E
F
Ya
or
I'm,
not
sure
I
mean
I,
mean
yeah.
There
are
a
couple
different
ways
about
it.
It
might
also
be
like
these
are
my
five
oldest
entries,
or
you
know,
I
just
added
ten
PG
logs,
and
so
we
we
like
collectively
need
to
remove
that.
Many
and
I
did
this
many
on
this
time
stamp
and
then
during
that
round,
people
then
the
other
P
DS
can
be
like
it.
Well
I
trim
this
money
then,
and
I'm.
F
E
Yeah
well,
okay,
so
that
the
simplest
thing
would
be
just
to
take
the
oldest
one.
Each
of
them
says
this
is
my
oldest
record.
You
take
the
oldest
of
all
of
those,
and
then
you
say
that
was
you
know
twelve
minutes
ago
and
I
want
to
trim
20%
and
so
I'm
going
to
tell
everyone
to
trim
everything
older
than
ten
minutes
ago.
G
M
G
I
was
well
I.
Guess
I
was
thinking
like.
Why
isn't?
Why
aren't
there
two
modes,
one
mode
is
just
where,
when
you're
adding
stuff
to
the
front,
you'll
trim
stuff,
that's
beyond
the
the
time
that
we're
allowing
and
that
just
always
happens,
and
then
you
only
have
to
do
this
extra
trimming
if
there's
so
much
activity
within
our
time
window
that
we're
now
like
going
to
exceed
our
memory
limits
right
I
mean
that's
pretty
much.
What
you're,
mostly
talking
about
yeah.
E
I
guess
that
would
be
it
that
would
be
sort
of
a
different,
so
you
could
just
you
could
basically
just
estimate
what
the
right
request
rate
is
for
the
OSD
and
say
that
means
that
if
I
want
10,000
records
and
that's
you
know
90
seconds
of
ops
and
then
you
just
say
that's
the
time
horizon
that
might
be
even
easier.
Okay,
that
make
sense.
B
G
Yeah,
no,
that
makes
sense
I
was
sort
of
thinking
that
you
had
a
sent
parameter.
That
says
we
all
we
try
to
keep
ten
minutes
or
I,
don't
know
what
you
were
thinking,
but
some
amount
and
that
that's
sort
of
like
your
first
thing
that
you
maintain
now
they
only
need
to
worry
beyond
that.
If
now
you
aren't
gonna
have
enough
room
or
if
you
start
to
now
not
start
using
too
much
memory
right,
I.
E
G
M
E
M
F
E
F
B
E
Motivating
idea
here
was
that
we
would
have
some
amount
of
memory
that
we
decide
that
we're
gonna
devote
to
this.
We
have
a
budget
so
that
we
don't,
you
know,
blow
up
memory
and
then,
when,
if
we're
degraded,
then
we
would
basically
say
we
would
steal
memory
from
the
blue
store
cash.
So
I
would
say:
blue
store,
used
less
memory
for
your
cache,
so
that
I
can
use
it
for
Geechee
logs
so
that
when
whatever
roasty
comes
back
up
I
don't
to
do
a
backfill.
E
E
In
reality,
most
of
the
PGS
are
not
going
to
be
degraded,
and
so
the
inserts
and
the
deletes
are
going
to
be
like
separated
by
a
second
less
than
a
second,
in
which
case
they're,
probably
not
even
going
to
drop
out
of
the
rocks
TV
wall
right
buffer
at
all.
Right.
That's
probably
gonna.
Just
like
cut
out
a
ton
of
work
that
we
were
throwing
at
the
rocks
to
be.
It's
really
only
the
degraded
P
G's
that
need
to
be
sort
of
landing
in
level,
0
and
level.
1.
E
M
E
But
if
they're
but
they're
gonna
be
end
of
them,
that's
the
thing
so
assume
there
are
10
to
create
a
few
G's.
They
have
different
write
rates,
and
so
we
want
to
trim.
We
want
the
time
horizon
to
be
whatever
it
is
like
13
and
a
half
minutes,
but
it's
going
to
be
different.
Lengths
of
vlog
for
those
different
few
G's
right.
G
F
G
E
E
E
All
right,
the
next
topic
is
the
dee
doop
I
just
want
to
get
everyone
an
update
about
this,
so
jung-hwan
@sk
has
been
working
on
this
pretty
extensively
off
and
on
with
mixing
with
a
couple
other
things
at
the
pass
like
I,
don't
know
how
long
it's
been
nine
months
or
a
year
and
I
just
wanted
to
make
sure
everyone
sort
of
knew
what
he
was
working
on
and
what
the
architecture
was.
So
the
I
tried
to
summarize
the
current
status
and
the
doo
doo
pad,
so
the
short
version
is
stuff.
E
Raitis
has
a
redirect
concept
now
we're
in
the
base
here
in
the
object
info
tea,
it'll,
there's
a
flag
that
says
there's
a
manifest
on
the
object
info
and
there
are
a
couple
different
types
of
manifests.
The
simplest
one
that's
implemented
right
now
is
that
it's
a
redirect,
so
the
object
controls
you
just
has
the
object
name
of
some
other
object
in
some
other
pool,
which
basically
just
means
that
this
object
is
stored
over
there.
E
Test
reduce
greatest
model
thing
is
setting
these
and
thrashing
these
and
mixed
in
with
all
these
stuff.
It's
been
in
there
for
a
while.
So
that's
all
there,
so
the
basic
idea
for
applying
this
to
D
dupe
is
that,
instead
of
having
that
manifests,
just
be
a
simple
pointer
for
the
entire
object
being
somewhere
else.
E
It
would
instead,
the
chunked
manifest
that
says
this
beginning
part
of
the
object
is
stored
over
in
this
object,
and
the
second
chunk
is
over
in
that
object
for
all
the
different
bits
that
the
object
is
d
duped
and
chunked
into,
and
then
those
chunks
would
be
stored
in
a
content,
addressable
pool
elsewhere
I'm.
So
that's
the
basic
idea
there's
a
full
request
with
some
of
the
chunks
manifest
infrastructure,
that's
in
flight,
so
it's
all
sort
of
being
proposed
and
then
and
that
pull
request
as
they
hit
the
read
handling.
E
So
if
you
get
a
read
on
a
chunked
object,
it'll
sort
of
break
it
into
lots
of
little
reads
that
gets
sent
to
the
back-end
objects
and
then
reassemble.
The
results
and
writes
would
simply
trigger
promote.
So
just
pull
the
object
back
into
that
based
here
and
it
turned
back
into
a
regular
object
and
then
things
would
life
would
proceed.
That's
before
that's
sort
of
that,
the
current
bit
that
he's
working
on
sort
of
next
steps
beyond
that
would
be
eventually.
E
So
the
there
are
a
couple
sort
of
basic
ideas
so
far.
The
first
one
is
that
since
we
don't
have
multi
object,
transactions
and
radius,
the
ordering
of
operations
would
be
so
that
if
there's
some
intermediate
failure
at
any
of
these
operations,
the
failure
mode
would
be
a
leak.
You'd
leak,
a
reference
to
an
object,
as
opposed
to
like
point
to
something
that
doesn't
exist.
E
That's
just
a
number
on
the
objects.
You
could
have
back
pointers
and
because
you
have
some
objects
that
are
some
chunks
are
gonna
have
like
a
bazillion
objects
that
have
that
same
content
do
YouTube
to
it,
and
so
you
can
actually
point
all
the
subjects.
The
idea
was
to.
One
of
the
ideas
is
to
have
this
sort
of
fixed
size
back
pointer
map.
So
if
you
have
a
small
number
of
back
pointers,
then
it
would
actually
point
to
the
objects
that
actually
constitute
those
references.
E
E
So
if
you
were
to
ever
do
a
scrub,
you
could
conceivably
go
enumerate
all
those
objects
and
try
to
match
them
up,
but
the
more
popular
and
object
is
sort
of
the
less
precise
those
back
pointers
are
and
correspondingly
the
less
likely
it
is
that
you're
ever
going
to
delete
it
anyway
and
so
the
less
that
matters.
If
you
leak
a
reference
so
yeah,
that's
the
idea.
E
Yeah
I,
like
it
I
like
it
more
as
time,
goes
on
so
the
deal
I
think
the
biggest
concern
I
have
with
the
overall
approach.
I,
don't
really
care
about
the
delete
week
to
offer
object,
references
I
think
it's
like
never
gonna
matter
like
if
you
went
and
deleted
every
single
object,
then
you'd
have
some
a
bit
slept
over,
but
like
nobody's
ever
gonna.
Do
that.
E
So,
of
course,
the
only
thing
that
worries
me
is
that
we
need
to
make
the
the
chunks
that
get
stored
in
the
backend
pool
big
enough
that
the
like
metadata
overhead
is
manageable,
so
that
the
overhead
of
it
being
a
distinct
object
with
you
know
the
blue
store
metadata
associated
with
it
and
so
on,
and
also
so
that,
if
you
have
a
format,
object,
that's
d
duped
into
like
you
know,
100
pieces
when
you
read
that
form
X,
it
turns
into
a
hundred
different
areas.
E
So
the
bigger
those
chunks
are
that
the
less
gross
here
like
I
understand
you
get
none
reads,
so
you
want
chunks
as
big
as
possible
and
but
the
bigger
they
are
the
harder
it
is
to
actually
get
a
good
deed,
oof
ratio.
So
it
all
really
kind
of
sound
alike.
How
are
you
doing
the
chunking
and
what
is
the
worklet?
Is
it
actually
to
do
Google?
E
The
good
news
is
that
they
have
a
bunch
of
code
that
does
like
reeving,
fingerprinting
chunking,
so
that
looks,
looks
at
the
content
and
tries
to
pick
chunking
boundaries
and
so
on,
and
there
there
are
other
things
you
can
do
to
you.
Can
you
can
do
content
based
chunking?
You
can
also
have
like
bits
of
code
that
actually
look
at
the
content
at
the
object
and
recognize
patterns
like
it
might
recognize
that
it's
a
tar
file
and
chunk
at
the
file
boundaries,
for
example,
which
is
kind
of
cool.
These
are
all
additional
gravy
abyss.
E
Guess
you
can
like
add
that
in
to
make
it
smarter
over
time.
Obviously,
we'd
start
out
with
the
simplest
things,
but
when
I
talked
to
so
I
talked
to
Jared
Floyd
their
CTO
about
this
and
went
over
it
and-
and
he
said
that
all
sounds
fine
as
long
as
you're
again.
Basically,
his
concern
was
the
object
size,
but
he
per
was
an
alternative
architecture.
That's
sort
of
less
ambitious
but
possibly
safer.
If
it
actually
works.
So
the
idea
there
is,
you
could
take
their
existing
content.
E
E
E
That
they'll
D
do
well
against
and
that's
sort
of
an
open
question,
and
it's
not
clear
to
me
how
that's
different
than
just
doing
the
chunking
in
the
first
place
and
then
doing
reference
counting
with
that
content.
Addressable
pool,
but
I
need
to
follow
up
with
him
and
see
if
he
said
any
bright
ideas.
After
is
after
his
week
at
Burning,
Man
expanding
his
mind
can
occur.
E
E
Well
so
he
talked
to
Sean
Cohen
about
this.
About
OpenStack
and
Sean
got
super
excited
cuz.
This
is
like
a
checkbox
that
they
want,
but
Sean's
question
was
like:
why
not
just
do
to
do
on
the
RB
client
side,
but
the
problem
there
said
it's.
You
know
it's
just
it's
single
user,
like
UD
dupes
within
that
one
image,
but
not
between
images,
which
is
sort
of
the
whole
point.
E
H
E
That's
true,
that's
true,
and
actually
that
might
make
sense
for
rgw,
where
the
the
objects
are
like
right
once
or
immutable
or
whatever
they're
not
modified
in
place,
because
you
could
just
chunk
as
the
data
is
coming
in
on
rgw,
and
it
could
go
right
it
directly
into
the
cast
pool
in
whatever
size
chunks.
It
wants.
H
E
E
G
E
C
E
H
E
That's
that's
that's
kind
of
the
idea
right.
So
it's.
How
do
you
like
construct
that
algorithm?
It
makes
like
all
the
content
in
the
vote
and
try
to
steer
it
to
the
right
place,
and
even
and
when
it
when
you
when
it
goes
to
a
particular
list,
you're,
not
actually
picking
the
OST
you're
picking
a
position
in
the
likes
of
YouTube
it
hash
namespace
for
that
pool
was
like
it's
super
weird
right.
E
E
B
E
E
E
Are
they
open
sourcing,
their
library
yeah?
Let's
read
at
everything
we
buy,
we
open
source.
So
that's
all.
It's
called
coming
upstream
well,
they're
mainly
focused
on
getting
the
kernel
driver.
Obscure
cool,
yes
is
sitting
around,
so
we
can
as
soon
as
we
decide
to
use
it,
we
can
just
open
source.
It.
E
Yeah,
okay!
Well,
anyway,
that's
an
overview
of
what
is
going
on
so
I'm
young
one
couldn't
make
it
but
he's
continuing
to
work
on
this,
so
the
next
I
probably
still
follow
up
with
with
Jared
and
see
if
he's
had
any
bright
ideas
and
I
definitely
want
to
hear
from
the
rest
of
you
guys
about
what
how
you
think
these
two
approaches
compare
either
now.
Are
they.
M
M
Kind
of
partial
to
the
first
one,
just
because
it
is
a
lot
simpler
in
terms
of
managing
the
space
usage
and
not
kind
of
constraining
placement,
but
this
kind
of
worried
me
in
terms
of
it
being
very
similar
to
the
cash
curing
stuff
in
terms
of
being
difficult
to
manage,
with
like
goop
detection
or
anything
else.
Let
me
need
to,
or
current
wouldn't
currently
need
to
do,
between
different
gears
that
might
not
be
supported
in
the
backing
of
all
four.
We
do
I.
E
I
think
the
saving
grace
here
maybe
is
that
when
you're
deducing
something
you're
doing
it
because
it's
cold
and
that
it
feels
like
there's
sort
of
this
implicit
understanding
that
it's
gonna
be
slow,
whereas
the
cash
hearing
was
supposed
to
like
make
something
faster
but
never
slower,
and
that's
here
we're
only
making
things
slower,
and
so
it's
less
it's
less.
That's
going
to
be
less
jarring
for
people
that
I'm
sure
people
still
complain,
but
I
feel.
H
E
E
Have
this
like
high
latency
but
like
more
efficient
right
but
I
would
just
my
guess
is
that
what
we
originally,
what
we
would
initially
implement
would
just
be
you
to
write
it
into
the
base
tier
like
any
normal
object
and
then,
sometime
after
the
fact
for
some
sheets
or
not
decide
that
this
is
a
cold
object
and
like
I,
don't
care
how
fast
it
is.
I'm,
gonna
D
do
that
and
so
it'd
be
like
sort
of
an
after-the-fact
offline
thing.
E
Jung-Hwan
has
a
proposal
to
sort
of
do
both
so
that,
and
so
he's
asked
like
a
his
ideas
to
have
stuff
that
goes
into
the
base.
Tier
have
go
into
an
LRU
and
if
it
doesn't
get
touched
for
some
period
of
time,
then
it
gets
T
duped,
but
if
it
does
and
it
stays
there,
let's
say
yeah
this
sort
of
like
logical
log
type
concept.
E
E
E
Yep
I
mean
the
cool
thing
about
their
their
DM
driver
is
awesome,
it
does
it
does.
Every
4k
is
d
duped
independently?
It's
like
it
has
this
like
fast,
very
fast
fingerprinting
that
tries
to
figure
out
quickly
whether
we
might
have
a
copy
or
not,
and-
and
it
also
does
compression
and
like
we'll,
take
all
these
4k
blocks
and
squish
them
down
and
compress
it
out.
E
It's
the
remove
snaps,
removing
the
remove
snaps
list
from
the
OST
map.
I
was
reminded
of
this
because
there
was
this
whole
series
of
patches
recently
from
somebody
whose
name
I'm
blanking
on
at
the
moment,
because
they
have
a
bazillion
deleted
snaps
and
it's
like
crazy,
slow,
they're,
just
printing
CPU
processing,
ghosting
maps
and
doing
the
appearing
stuff
and
so
they're
optimizing.
E
The
interval
set
calculations,
which
is
all
great,
but
it
sort
of
ignoring
the
fact
that,
but
there's
no
reason
to
like
have
these
huge
lists
in
memory
on
the
OS
to
use
at
all
times
like
once.
The
stamp
is
removed
like
we
should
mostly
be
able
to
forget
about
it.
So
I
went
through
and
read
through
all
of
the
code
that
looks
at
that
data
structure
and
it
sort
of
falls
into
a
couple
different
bins.
E
E
There
are
cases
when
you're
like
trimming
snaps
or
when
you
are
looking
at
our
request
is
coming
in.
That
has
a
snap
context
you
need
to
like
know
whether
those
snaps
have
since
been
deleted,
since
our
request
has
been
sent
and
that's
sort
of
the
trickiest
one,
because
you
need
to
like
do
a
look-up
on
each
of
those
snap
IDs
to
see
if
it's
it's
now
gone.
So
I'll
talk
about
that
last.
E
But
the
main
thing
is
that
when,
when
you
delete
a
snapshot,
it
gets
published
in
the
OSD
map
in
the
PG
pool
info
and
that
basically
tells
the
OSD
to
put
it
on
a
queue.
It's
the
snap
trim
queue
on
that
PG
that
then
chews
on
to
actually
delete
all
those
snapshots,
and
once
it's
done,
then
it
puts
it
that
snap
ID
onto
a
list
in
the
PG
info
of
purge
snaps
and
it
stays
there
pretty
much
forever
as
far
as
I
can
tell,
which
is
also
stupid.
E
The
first
thought
was
that,
basically,
the
monitor
would
do
this
sort
of
coalescing
and
pruning
of
things
that
are
in
the
OSD
map
that
have
been
totally
deleted
from
everywhere
on
a
on
a
very
coarse
basis
like
every
say,
100,
OST,
math,
epics,
4
or
whatever
it
is,
it
would
update
it
to
the
next
a
criminal.
Let's
say
these
ones
have
all
been
successfully
purged.
E
You
can
forget
about
them
and
it
would
record
that
somewhere
in
the
in
the
monitor
would
hit
like
a
have
an
old
deleted
or
old
purged
snaps
data
structure,
and
then
the
OST
s
could
also
have
a
local
copy
of
that
structure.
So
they
can
always
anybody
can
always
look
up
in
that
structure
instead
of
the
hosting
map
that
they
need
to.
F
I
think,
most
to
make
sense,
I
don't
have
in
my
head
all
the
rules
about
how
snapshot
requests
retreated
from
clients
if
the
snapchat
doesn't
exist
and
when
they
appear
and
don't
so
that's
the
part
I'd
be
worried
about
whether
we
can
actually
throw
it
out
on
the
OSD
here
whether
we
need
to
keep
that
giant
list
like
differently
already
in
the
OSD
map.
Only
my
Shalini,
the
PGS
to
have
all
of
them
for
filtering
I.
Don't.
E
F
E
E
E
F
G
G
E
G
E
It's
yeah,
there's,
there's
just
no
explicit
tracking
to
make
it
scale
so
they're,
both
unbounded
yeah.
So
the
fact
that
we
were
enumerated
explicitly
is
it's
kind
of
surprising
that
we've
gotten
away
with
it.
As
long
as
we
have
I
think
it's
the
way
to
put
it,
it
would
be
better
I
think
you're
right.
It
would
be
better
to
explicitly
enumerate
the
ones
that
do
exist
instead
of
don't
exist.
That
would
be
better
than
what
we
have,
but
it
doesn't
really
address
the
problem.
I
think
I'm,
actually
not
sure.
E
E
So
it
feels
like
the
best-case
scenario
would
be
that
we
make
it
so.
The
USD
map,
perch
snaps,
has
like
enough
history
that
it
goes
back.
You
know
like
a
couple
hours
or
days
or
something,
and
if
that's
long
enough,
then
we
should
never
see.
We
would
hopefully
never
see
an
IO
request
with
a
snap
context
that
has
such
an
old
list
of
snapshots
in
it
that
it
includes
stuff
that
has
been
purged
or
that
you.
E
That,
though,
right
right,
so
if
we,
if
we're
super
lucky
having
such
a
snap
context,
could
be
sort
of
you
know,
we
send
an
e
again
or
something
to
the
RVD
or
it
just
would
like
not
do
anything
bad
like
it
long
as
it
wouldn't
cause
the
cause,
a
clones,
a
bad
clone
to
be
created.
That
would
never
get
burst.
As
long
as
that
didn't
happen,
then
it
almost
I.
F
E
Bad
thing
would
be
that
it's
asking
for
a
snapshot
that
doesn't
exist
and
you
would
get
like
right,
yeah,
you
would
say
yeah
they
give
it
like
the
wrong
clone
or
no
you
just
say:
I,
don't
have
it
and.
E
Could
know
that
the
OSD
map
could
have
the
lower
bound
of
things
that
have
been
forgotten,
the
sequence
number.
So
it
basically
say
if
you
ever
get
a
sequence
number
older
than
you
know,
snap
ID,
you
know
1
million
in
12,
then
oh
I'm,
not
gonna.
Tell
you
whether
that's
ninth
was
deleted
or
not
I,
don't
I,
don't
remember,
I
forgot
because
it
wasn't
worth
it.
F
F
H
G
N
F
E
F
E
M
Yeah
so
the
another
piece
for
me,
but
it
didn't
see
his
dressed
explicitly
there,
but
maybe
you
already
thought
about
it
was.
If
we
do
have
an
older
owes
to
you
that
still
has
some
snaps
that
haven't
been
trimmed.
M
E
E
Don't
think
will
solve
the
whole
thing
there.
That's
the
almost
full
problem,
and
then
last
the
last
thing
I
have
is
really
quick.
So
this
came
up
recently
with
a
couple
different
clusters
where
both
these
were
failing,
and
so
all
the
PGS
kept
getting
like
backfill
to
a
smaller
and
smaller
number
of
OS
T's,
and
those
associates
ended
up
with
like
way
too
many
PGs
than
they
should
have.
E
E
The
problem
is
that
the
there
are
two
ways
that
PJ's
get
created.
One
is
the
monitor
will
tell
to
create
them.
That's
sort
of
these
you
want
to
deal
with,
but
more
generally,
the
primary
primary
for
the
PG
is
somewhere
else
and
you're
sending
notifies
caring
notifies
and
that's
will
instantiate
the
PG
and
you'll
just
refuse.
You
won't
do
anything,
and
so
that
peering
will
just
basically
get
stuck,
which
is
kind
of
what
we
want
that's
better
than
like
making
that
OSD
get
overloaded.
E
So
pad
isn't
loading
for
these
from
reason,
but
I
have
an
old,
maybe
hopefully
floating
for
one
else.
That
goes
so.
The
basic
idea
I
had
was
that
whenever
that
happens,
if
we
refuse
to
create
a
PG,
we
just
put
it
in
a
set.
So
we
have
a
list
of
what
those
PGs
are
and
your
unique
ID.
So
we
know
how
many
there
are
and
there
yeah
how
many
there
and
then,
after
we
remove
enough
P
G's
that
we
decide
it's
worth
it
to
like
restart
peering
on
those
are
the
ones.
E
B
E
When
I
looked
when
I
originally
looked
at
this,
like
forever
ago,
I
think
was
like
two
years
ago:
I
thought
it
was
complicated,
but
I
think
it's
less
it's
not
as
complicated
as
I
thought.
It
was
so
I
mentioned
this
tacky
foo
after
stand
up
because
he's
looking
for
something
new
and
he's
chomping
about
that.
He
must
run
with
it.
So
I
think
it
would
be
good
yeah.
E
N
E
F
E
K
E
So
all
the
situations
that
I've
seen
where
this
happened-
it's
all
it's
all
like
a
it's
a
skewed
curve
or
whatever
right,
where
you
have
a
bunch
that
have
the
normal
amount
of
P
G's
and
then
as
they
failed.
The
Fiji's
kept
piling
up
in
a
small
in
swallowing
upper
nodes,
and
so
it's
not
like
they
moved
around.
They
flipped
physicians,
they
all
just
sort
of
like
collapsed,
and
so
this
would
let
them
drain
back
to
where
they
go.
Sort
of
it
was
at
least
address
that
case
and
I.
E
Guess
it's
possible
if
you
got
into
that
situation
and
then
like
change,
your
crush
map
entirely.
So
everything
was
like
randomly
scattered
somewhere
else.
You
could
have
P
G's
that
we
need
to
go
to
those
over
full
notes.
Do
not
the
other
way
around
and
they
would
just
get
stuck
basically
until
they'd
get
drained.
E
E
N
N
N
N
N
N
C
West
must
be
able
to
find
the
Q
s,
control
from
the
unique
identifier
of
the
arriving
recast
so
done.
The
space
available
for
storing
specific
information
in
south
cluster
is
a
edge
of
our
writing.
First
it
across
the
map
and
such
as
motor
and
West
PG
crush,
and
yes,
the
second
is
a
type
object,
for
example,
is
our
BD
at
a
time
so,
based
on
this,
how
we
will
describe
the
subclass
for
Union
structure
that
we
have
developed
already
force
originally
or
richly
cast?
N
Has
the
full
information
in
this
case,
which
is
Q
s
identifier
and
the
for
property
is
stored
in
in
the
OSD
map?
Thus
the
desire
did
class
control.
Information
can
be
added
to
the
existing
property
of
h4
Oh
from
the
West.
If
standpoint
CH
OSD
has
the
radius
OS
map,
you
can
know
the
rest
control
of
the
floor,
corresponding
to
receive
recast
and
receipt
us
through
4
and
class
counter
information.
N
The
date
is
our
first
implementation
and
those
CRI,
a
common
item
facing
Humphreys
edge
or
Faro
West
serve
as
the
first
set
and
poor
ID
and
ledger
and
4,000
die
offs.
That
is
some
huge
entrance.
So
in
this
case
of
IVD
there
is
right
difference
from
/
h,
racast
can
deduce
IBD
information
through
ID
and
profit
for
IBD
is
studied
in
IBD
had
a
type
of
jack
also
the
DJ
did
cross
country.
N
Information
can
be
edited
to
other
profit
of
HIV
D,
but
in
case
of
IBD
in
cross
country,
information
is
of
same
from
IBD
had
or
object
at
all.
I
will
at
the
IBD
section
in
generalization.
Thus,
cursed
country
information
will
be
included
in
the
recast
age.
As
a
result,
the
worst
can
avoid
asking
another
West
again
to
obtain
the
cross
country
information
associated
with
all
rivalry
cast
IBD.
N
N
From
the
usual
standpoint.
A
yes
agile
follows:
first,
you
just
create
us
unique
identifier
in
short,
q,
qu,
ID
and
related
class
control,
q,
ID
and
related
cruise
control.
I
should
stored
in
in
global
cross-table
in
short
chickity,
which
will
be
included
in
cross
map
after
the
q.
Id
is
mapped
to
specific
class.
You
need,
such
as
IBD
or
for
or
specific
object
for
either
user
and
the
completed
mapping
is
also
studied
in
GT
on
the
right.
N
N
Of
course,
the
beautification
can
occur
in
the
process
of
finding
q
ID
for,
for
example,
recast
parent
to
post.
U
ID
0a
v
w1
+
q,
ID
1,
+,
4,
0
age
on
exam,
as
an
in
the
example
table
in,
in
this
case
on
internal
a
pre-k
internal
application
for
where
we
needed
or
injustice
small
range
tanks,
tanks,
precedence
over
or
large
range
that
is
Teddy's.
Our
VD
has
higher
on
Friday,
then
/
o.
If
we
qu
ID
is
found,
the
request
will
be
sent
to
OSD
and
was
receiving
the
recast
o.
N
E
That
the
first
you
look
right
to
me
and
the
pool
the
pool
limits
and
yet
already
setting
on
setting
them
as
a
property
on
the
ER
VD,
the
big
q2id.
One
worries
me,
though,
the
way
that
it's
described
here
it
looks
like
the
OSD
will
get
a
request.
The
queue
ID
will
be
twelve,
it
doesn't
know
what
that
means,
and
so
it
has
to
go.
Look
that
up
like
go
query
the
monitor
or
something
in
order
to
find
out
what
what
the
parameters
are
for.
E
B
E
N
E
Last
met
yeah,
so
the
idea,
then,
that
the
disc
us
table
was
I
mean
in
order
for
it
to
be
stored
in
the
OST
map.
It
has
to
be
pretty
small
or
recently
small,
so
it
has.
It
would
be
like
the
that
the
generic
set
of
policies
like
fast,
medium
and
slow,
or
something
like
that
and
each
of
those
each
of
those
policies
would
be
applied
to
lots
of
images.
Is
that
the
idea.
D
E
Because
yeah
I
mean
that's
sort
of
what
I
would
guess,
given
the
diagram,
except
that
your
example
says
that
zero
listed
like
a
specific
RVD
image,
name
yeah.
E
E
H
N
Become
bigger
size,
so
I
think
I
think
those
pests
will
spread
across
the
control
space
information
or
such
as
chickity
is
I.
Think
first
is
Christmas
and
seconded
any
and
you
had
a
type
object.
So
we
can,
we
can
say,
store
security
in
the
other.
Header
type
objects
like
IBD,
yeah,
West
bed,
West
bed
is
not
properly
proper
way.
It
is
not
proper
way.
If
we,
you
think
we
can
observe
I,
don't
so.
E
E
The
lebar
buddy
just
tells
liber8
owes
tagged
my
requests
with
like
us
parameters,
500
I,
ops
and
that
gets
past
the
OSD
and
everything
everything
simple
the
place
where
I
think
that's
going
to
be
a
problem
is
again,
you
have
this
huge
OpenStack
cluster
you've
got
10,000
images
and
they
all
have
500
apps,
and
you
say
you
know
what
I
think
these
easier
should
get.
600
iOS
and
you
don't
want
to
like,
go
and
touch
every
single
image
to
change
it.
You
just
want
to
have
one
switch
that
says
my
gold
policy
is
now
600.
H
E
H
H
If
it's
already
attached
it's
already
attached
and
you
have
to
like
detach
it
and
bring
it
back
right
and
then
it
would
still
be,
it
would
still
it
wouldn't
be
RBD
controlling
those
QoS,
but
actually
be
cinder,
telling
RVD
what
the
key
aspects
are.
So
from
an
for
your
example:
OpenStack
there
actually
isn't
a
management
that
we
need
to
do
we're
just
gonna,
be
told
on
every
single
image.
Let
me
a
bandwidth
to
this
or
limit
your
I
up
to
that,
or
also
we
off
test.
H
E
H
It's
still
I
think,
even
if
you
had
something
like
that,
it
still
has
to
be
because
the
volume
you
they,
the
disconnect-
is
already
there
that
these
are
volume,
types
that
have
Q
s
add-ons
to
them
and
those
are
signore
manage
types
and
they
don't
correspond
to.
You
know
low
level,
storage,
back-end,
less
things
seem
to
get
mapped,
but
man.
E
H
You
would
you
would
probably
the
pro
verse
writing
that
send
a
driver.
We
have
to
do
it
so
that
now
you're
trying
to
keep
in
sync
the
cinder
driver
and
cute
best
volume
types
with
this
RBD
QoS
day
and
you
know,
sender's
not
gonna
Besame
cinder
is
the
owner
of
the
data
right.
You
know
in
terms
of
the
data
model,
yeah
yeah,
so
you
need.
E
H
To
have
some
process
that
I
always
trying
to
keep
our
BD
in
sync,
with
whatever
the
volume
type
QoS
facts
are,
or
you
just
or
whoever.
Whenever
that
our
BD
hook
back-end,
look
it's
implemented,
if
you,
if
they
ever
do,
support
dynamic
updating,
it
would
just
have
to
iterate
through
all
the
images
in
like
yeah
call
on
its
own,
like
update
key
lesson
that
would
ping
the
header
to
reread
something
or
whatever.
E
H
E
H
H
E
Well,
you,
if
Sue
me,
you
could
parameterize
it
like.
It
could
be
that
you
specify
this
is
a
policy,
and
this
is
the
size
that
you
divide
in
or
whatever
you
do
something
like
that.
But
but
in
any
case
that
would
that
would
mean
that
the
you'd
have
a
slightly
different
tagging
on
the
request
and
then
when
it
hits
the
OSD
that
it
can
look
up
in
the
OSI
map
and
apply
those
parameters
but
sort
of
ends
up
being
the
same
thing
as
far
as
one
say,
it's
the
deem
plug
scheduler
I.
E
H
H
E
Yeah,
it
could
even
be
a
separate,
monitor,
structure
that
they
subscribe
to
you
through
the
mod
client
or
whatever
so
ticket
they
find
out
when
those
show
us
policies
change,
if
you
really
wanted
to
have
the
life
update
thing.
Yes,
yes,
the
only
the
only
thing
that
this
this
would
provide
as
propose
where
it's
done
on
the
OSD
would
be
get
that
sort
of
like
protection
against
cheating
or
a
modified
client.
It's
just
feeding
in
bogus
parameters
in
order
to
steal
IUP's,
except.
H
G
E
N
So,
first,
what
is
the
incorrect
data?
Low
phase
parameter?
Also,
I,
don't
know
it
just
required
it.
Our
M
clock
argument
for
us
in
this
frigid
environments,
routing
system
and
these
Indians
must
be
recorded
to
provide
cures
between
specific
range.
The
current
implementation
situation
is
edge
of
our
host.
First,
the
operation
types
with
Dettol
of
our
meter
is
West.
Oh
OS
do
P
and
M
o
OS
gop-led
price
crest,
which
is
reproducible
for
main
current
IO.
N
In
other
words,
don't
they
are
handy
handily
advisor
the
incorrect
is
produced
distributions
of
Stryker
second
instant
in
insertion.
10
kilometers
into
the
existing
existing
Socrates
is,
as
that
that
it
is
added
directly
to
filter
over
and
over
recast
and
mos,
GOP
and
mos.
The
openly
progress
here
here
is
an
example,
consider
the
M
Sheen
and
me
Eric,
pants
and
us
on
both
hands
or
DMZ
response.
N
So
I
want
to
say
if
the
issue
for
this,
this
page
is
first
issue
about.
On
number
of
shot
in
the
operation
queue
in
one
st
and
there
will
be
Kiera
coos,
depending
on
the
number
of
the
shot,
such
a
one
woman
and
therefore,
as
you
as
the
number
of
shot
increases,
the
number
of
the
increase
in
the
number
of
the
encourage
cues
in
a
single
Wesley
also
encourages
our
large
number
of
Jing
crackers
can
cause
problem,
as
shown.
The
bottom
of
the
left
graph
as
a
group
operation
ad,
is
pretty
divided.
N
The
included
number
of
the
ink
recuse,
the
tinkerer,
the
ink
roller
cutest
problem
becomes
more
worse,
in
other
words
of
stacking
operation,
ranks
of
which
the
incur
a
queue
is
dirty.
Indeed,
as
shown
on
the
left.
Two
pictures
pictures.
N
N
What
is
about
the
weight
control
with
data
and
row
for
background
I
or
anchor
crank
use
current
Angeles
current
cues
current
identifier
is
a
pair
of
OPA
type
or
client
entities,
in
other
words,
the
current
IOM
background.
The
IOUs
are
upon
preform
performed
in
the
one
queue
at
the
same
time,
I
just
shown
on
the
bottom
picture,
those
in
weight
control
to
ensure
variety
between
current
IO
and
background
io,
o
back
Rhonda,
also
requires
data
and
row.
N
So
our
first
approach,
II,
is
that
in
background
I
used
normalized
data
low
according
in
canto
FD,
for
example,
you
need
time
average
value.
So,
if
our
fourth,
there
are
some
two
issues,
so
I
saw
another
utterly.
If
solution
is
here
so
I
wanna
I
will
spread.
O
O
In
addition,
we
can
stack
in
the
shingled
en
crack
here,
whether
the
eternal
across
multiple
char
killed.
This
is
all
the
advantages
book
us
process
is
the
Keydets
growth.
The
downside
is
that
purpose
can
be
better
this
or
cry
until
kids.
Trying
cheating
want
and
there's
no
push
testing
process
passed
would
be
longer
because
quietly
kissed
must
know,
quiet.
O
O
O
B
E
Think
they
would
work
the
same
because
I
think
the
once
once
your
serializing
on
that
queue,
then
the
real.
The
only
real
advantage
to
having
the
starting
stuff
is
that
you
is
having
multiple
threads
so
that,
if
1pg
blocks
on
a
read
or
something
like
that,
it
won't
slow
everything
else
down
and
having
multiple
threads
per
shard
gets
you
that.
O
B
O
Standing
are
you
guys
order
to
mr.
Joe
mm
ascending?
Are
you
based
in
additives?
What's
wrong,
make
place
more
annoyed
and
more
boy?
They
are
soaring
weakest
in
the
o'clock,
and
none
normally
expect
so
against
coming
in
being
quite
will
be
bypassed
without
any
petition
with
no
other
requests
this.
Why
christ
me
austere
torah,
the
purpose
of
a
wire
throttle
is
to
dispatch
only
enough
to
the
gas
that
can
get
enough
performance.
O
Remaining
requests
will
stay
in
the
vehicle
at
the
cure.
Death
can
be
Roger,
no
more
scheduling
why
you
want
to
measure
everybody
stupid
and
every
girl
standing.
Are
you
the
why
you
might
also
tracks
maximum
support?
If
enough
are
you
studies,
pettiness
was
panned.
We
inquire
at
this
time
or
a
proportional
scheduling,
stuff
inquire.
O
It
is
impotent
to
figure
out
how
much
else.
Turning
on
you
is
for
free,
a
two
comments:
icon.
It
situation
odd.
If
more
recast
than
that
question,
whether
I
will
fetch
it
abolish.
The
gradation
can
be
minimized
even
with.
Hopefully,
if
it
is
important
and
find
the
collective
situation
work,
never
I
ever.
Is
you
why
you
and
the
trooper
will
be
measured?
Every
200
find
the
saturation
it's.
The
average
is
replaced
quite
than
the
previous
max
Cooper
increase
increase
mixture
for
the
end
with
situation.
G
O
Next
page
is
the
test
result
we
tested
five
prior
to
using
insert
one
who
the
bottom-left
graph
is
the
proportional
testing
result
between
that
uses
sort
of
all
polls.
We
showed
20
a
que
yo.
We
got
30
s
configuration
on
the
other
hand,
who
we
usually
stop
around.
We
can
get
the
ratio
5,
4,
3,
2,
1
and.
O
E
E
It
it
sounds
to
me
overall,
like
were
on
the
right
track.
I
would,
it
seems,
like
the
next
steps
are
probably
to
just
propose
what
the
specifics
SCO
like
commands
are
to
configure
the
the
first
two
types,
that
the
pool
cue
s
thresholds
and
already
QoS
thresholds
and
then
and
then
get
your
pull
request
that
does
the
Rho
Delta,
whatever
it
is
stuff
in
deliberate
OHS,
and
then
you
know,
go
ahead,
implement
it.
E
The
I
would
wait
on
the
universal
QoS
stuff
until
the
other
parts
are
implemented
and
we
have
sort
of
a
better
understanding
of
what
how
that
should
behave,
and
then
the
iOS
rattler
I
guess
would
be
the
last
piece.
But
overall
this
thumbs.
This
sounds
awesome,
looks
really
good
Jeff
any
concerns
or
questions
Jason
near
be
fine.
H
No
I,
just
on
the
IRC
I
did
post
I
was
looking
through
cinder
again
it.
It
does,
restrict
you
from
not
updating
front-end
base
QoS,
but
on
the
back
end,
if
the
back
end,
doesn't
it
it
iterates
through
all
the
volumes
associate
with
that
back
and
then
just
changed
on
a
per
image
basis
so
again
like
from
a
cinders
perspective,
it
wouldn't
do
any
help.
It
wouldn't
be
any
help
over
to
be
a
hindrance
really
to
have
this
okay
extra
layer
on
top.
H
E
It's
like
copied
into
swift,
but
it
was.
It
was
always
pasta,
a
lot
more
work,
but
you
were
able
to
implement
like
efficient
back-end
operations,
presumably
same
thing
with
like
multi-site
copy.
Like
you
know,
my
default
Center
will
go
and
like
literally
copy
the
thing
and
stream
it
across
and
like.