►
From YouTube: CDS G/H (Day 1) - OSD: Scrub & Repair
Description
https://wiki.ceph.com/Planning/CDS/CDS_Giant_and_Hammer_(Jun_2014)
24 June 2014
Ceph Developer Summit G/H
Day 1
OSD: Script & Repair
A
B
First
and
foremost,
I
think
is
that
if
you
do
a
scrub
and
there
are
inconsistencies,
you
have
to
be
watching
the
central
log
and
know
how
to
interpret
it
to
actually
figure
out
well
as
well
as
inconsistent.
So
that's
the
first
thing
to
want
to
fix.
We
need
an
interface
so
that
the
OSD,
when
it
scrubs
at
PG,
racks
team
consistencies,
ideally
not
in
memory,
because
there
might
be
thousands
of
them
in
some
kind
of
really
powerful
case.
B
So
they
think
the
first
step
is
to
define
all
of
the
ways
in
which
an
object
to
be
in
systems
that
will
become
the
base
data
structure.
We
pass
around
between
USD
librettos
and
not
through
the
librettist,
all
that
and
when
the
OSD
encounters
in
the
inconsistent
object.
It
describes
it
using
this
data
structure
and
then
it
sticks
it
into
leveldb
using
an
architectural
device
like
the
snap.
Ever
this
it's
boring
and
uninteresting,
but
it's
nice
that
it
we
won't
have
to
rewrite
it.
B
There's
there's
an
existing
piece
of
code
that
generically
maps
a
key
to
buffer
list,
mapping
onto
the
backing
file
store
and,
more
importantly,
keeps
in
memory
and
the
unstable
keys,
so
that
if
you
read
while
there's
an
in-progress
transaction,
it
reads
out
of
my
memory
instead
of
out
of
the
file,
sir,
not
interesting,
but
at
least
it's
a
step
that
trouble.
So
we
can
just
borrow
that
bab
structure
for
writing
up
these
diso's
the
information
or
at
the
per
object.
Inconsistency.
Information
also
important
is
that
the
inconsistency
information
is
logically
tied
to
an
active
interval.
B
So
when
we
go
through
peering
or
the
acting
set,
changes
at
all
I
think
it's
perfectly
fine
to
just
discard
this
information,
so
the
object
into
which
we
will
be
recording.
This
information
does
not
need
to
be
a
wrap-up;
it
doesn't
need
to
be
sent
to
the
replicas
and
it's
fine
if
it
gets
clearly
if
we
go
for
Gary,
maybe
that
turns
into
a
problem.
We
change
it
later,
but
that's
fine
for
now
yeah,
okay,
so
that
has
some
implications
in
the
ratos
interface.
It
means
that
everything
we
asked
of
the
pete
of
do
SD.
B
So
first
there's
one
that
queries
the
monitors
we
need
to
find
out
which
VG's
are
actually
inconsistent,
I,
don't
think,
there's
a
liber8
O's
PG
dump
then.
Nor
should
there
be,
though
at
the
moment
it
will
very
inconsistent,
p
juice
and
it'll
littell
page.
So
you'll
you'll
get
a
certain
of
RPGs.
At
times
we
provide
it
with
the
last
thing
you
got
to
get
the
next
cell.
C
B
B
Come
then
we
need
a
repair,
inconsistent
object.
This
is
where
the
activation
epic
is
pretty
important,
because
the
detail
how
we
identify
the
replica
is
pretty
important
to
which
effort
we
got
the
information
from
so
the
OST
needs
to
be
able
to
say.
Actually
you
need
to
really
very
that
often
because
we
have
pill,
information.
C
B
So
that's
down
here,
I
haven't
decided
how
to
do
that.
Guy
I
go
for
the
my
boyfriend
idiot
see
it's
in
the
blueprint
if
you
go
further
down.
This
is
this.
Is
that
discussion
I
want
to
have
I,
don't
want
to
reproduce
all
the
current
interfaces,
so
we
need
to
be
able
to
get
the
object
payload
in
all
the
ways
you
can
currently
do
that
you
need
to
build
to
get
the
exciters.
B
You
need
to
get
the
bleep
do
map,
but
we
don't
want
to
have
like
it
didn't
consist
adopted
version
of
all
of
those.
So
yeah
is
this.
Your
portable
property
you
set
on
the
IO
context.
Is
this
another
version
of
submit
or
sorry
about
what
what's
a
call
operate?
Is
this
another
version
of
a
operate
that
only
takes
liberators
read
operation
that
might
makes
fake
make
sense,
actually.
C
E
A
C
B
E
Thought,
let's
add
support
for
see
but
making
sure
we
get
these
new
interfaces
and
see
as
well
be
pretty
good.
Okay,.
B
Page,
we
previously
talked
a
bit
about
I
felt
strongly
that
we
should
prompt
you
through
the
primary
I,
no
longer
feel
that
so
I
think
we'll
just
co-op
the
existing
f.
What
I,
really
didn't
want
to
do
is
have
a
stateful
thing
that
the
optin,
where
the
object
or
just
sort
of
magically
redirects
it.
So
where
else
for
some
reason,
yeah
those
long
as
if.
C
B
C
B
B
The
point
the-the-the
in
consider
you
the
system
in
for
information
for
the
object,
will
also
give
you
back
the
PG
epic
identifying.
That
interval
so
probably
be
the
first
epic
med
interval.
You
will
then
provide
that
you
enacted
a
at
this
special
operate
hall,
which
the
rep
can
we
replicate
and
then
use
it
to
verify
that
it's
supposed
to
satisfy
the
request.
I
mean.
B
C
C
B
C
B
I
think
yes,
no
I,
don't
think
we
should
were
able.
Yet
so
no
I
mean
I
think
either
it
decodes
that
it's
repairable
or
it
deletes
the
object.
B
Okay,
yeah
I'm
fails
to
do
if
it
fails
to
repair.
Oh
and
this
repair
thing
needs
to
be
asynchronous
I
think
so
you
can
get
an
error
code
if
it
fails
to
repair
you'll
get
back
at
error
code
and
if
you
leak,
worry
that
object,
you'll
get
it
be
in
position
info
t
will
say:
not
only
does
do
these
shards
not
match
I,
actually
can't
repair
it
and
making.
A
C
Repairable
thinking
night,
oh
you,
sir,
you
might
have
a
situation
where
all
the
CRC's
look,
okay,
like
you,
have
all
the
shards.
They
say
pure
fine,
but
it
fails
to
decode,
and
in
that
case
you
will
want
to
ask
it
to
decode
by
excluding
each
shard
in
turn
to
see
if
one
of
them,
our
example
was,
was
written
with
the
bad
crc
that
happened
to
match
something
like
that.
All
right,
that's,
okay,.
B
C
C
A
C
C
D
D
B
C
B
That's
that's
why
it
try
to
be
pompous
about
putting
more
things
in
the
ad
in
the
face.
If
they
don't
fog
lights,
maybe
they
should
be
right,
but
maybe
they
shouldn't
matter
that
very
well.
C
D
B
We
definitely
can't
do
that
at
least
not
without
enormous
amount
of
pain,
because
they
don't
act.
Anything
like
the
ratos
operating
right.
Now,
it's
a
there's,
a
locking,
they're
flocking
machinery
with
some
weights
with
log
gentle,
separate
special
message
type.
Maybe
we
could
emulate
it
using
a
size
long
and.
D
C
B
D
D
D
E
B
E
B
B
That's
a
little
that's
a
little
trick
here.
First
off,
we
can't
fix
that
without
a
right.
So
that's
it.
That's
actually
a
pretty
good
reason
to
make
repair
a
right
rather
than
just
a
recovery
operation.
B
Because
we
may
need
to
actually
update
the
object
info
to
something
that
isn't
on
any
of
the
objects,
so
it
would
be
something
like
the
write
operation
would
contain
the
information
required
to
update
the
metadata
and
then
as
a
side
effect.
It
will
kick
out
certain
objects
which
it
will
then
Marcus
missing
and
initiated
with
hair,
very
a
recovery
when
both
the
recovery
and
the
right
Oh
complete
it
with
them.
Acta.
B
B
C
B
D
B
E
B
C
Yet
I
think
I
mean
in
generally
speaking,
yes,
I,
guess
it's
yeah,
that's
why
I
wouldn't
want
to
turn
it
into
a
right
that
goes
through
all
the
other
right
machine
whirring
it
that
isn't
actually
what
you
want
to
do,
but
I
just
made
a
Dateline
data
actually
doing
yeah
I
mean
in
sort
of
the
weirder
case.
You
could
imagine
you
had
three
replicas.
C
B
B
E
E
B
Also,
while
we're
here,
it
would
be
cool
if
you
could
enumerate
your
unfound
outfits
and
declare
them
if
you
could
also
have
a
liberator
interface
to
fix
that
also,
basically
I.
B
Also,
you
can't
tell
to
mark
a
specific
object
unfound
and
you
can't
get
any
information
about
it.
This
inconsistent
into
a
key
will
probably
suffice
to
give
you
exactly
the
information
about
a
nun
thought
of
it
that
you
want.
C
C
F
Yeah,
actually,
I'm
sorry
miss
for
the
conversation
but
like
to
hear
more
about
how
we
expect
that
means
to
actually
use
this
cuz
I
mean
when
they
come
in
board
and
what
they
want
to
repair.
What
they're
usually
saying
is,
I
have
three
copies,
and
one
of
them
is
wrong.
I
want
you
to
just
pick
the
one,
that's
obviously
right
from
from,
because
there's
yeah.
B
B
This
case
you
later
the
admin
says
one
is
obviously
correct.
You
mean
the
admin,
looks
at
the
apt
on
disk
structures
and
says:
oh
this
one's
probably
right,
based
on
its
four
megabytes
instead
of
half
a
megabyte,
or
do
you
mean
that
there
are
two
instead
of
one
and
a
vos?
He
should
just
think
of
all
that
that
one.
B
C
But
I
mean
maybe
they
want
to
do
that,
but
I
think
in
general
and
father
diva
behavior
would
be
that
it
would
flag
it
and
then
they
would
have
to
go
dad.
Man
would
go
inspect
it,
so
they
have
a
command
like
that's
gets
cover.
Isolde
right
will
be
like
rye,
toast,
fpp
list,
grub
errors
just
and
then
for
a
specific
object.
Get
scrub
result.
F
D
C
F
B
D
D
C
C
C
So
I
mean
the
unfound
stuff
right
now,
that's
through
the
PG
tell
interface
is,
is
Jason
and
it's
annoying
like
all.
C
It
means
it's
like
that,
much
that
many
more
functions
you
have
to
implement
in
the
radius
tool-
that's
unlike
interpreting
and
printing
and
and
parsing,
and
all
that,
whereas,
if
it's
like
that,
the
tell
thing
nice
thing
about
the
Jason
is
is
just
whatever
you
can
output
anything,
because
everything
is
a
dump
function
and
you
can
parse.
You
know
anything
that
you
add
a
parser
for
which
we've
done
with
all
the
rgw
stuff.
C
It's
not
that
hard
and
then
and
it's
all
implemented,
OST
side
like
there's
no
special
tool
needed
it's
just
a
generic
fcl
I
tool
using
in
fact
it's
not
even
PG,
but
if
I,
just
like
tell
I'll
point
you
right,
like
the
only
the
main
thing
is
that
you
have
to
you're
talking
to
a
specific
PG.
So
it's
like
one
more
step:
they're,
not
the
users
interact,
Ernest,
ok,.
D
C
It's
going
to
be
the
same.
It's
going
to
the
same
set
of
checks
that
are
in
the
they're,
similar
checks
that
are
in
the
current,
do
op
thing,
but
I'm
already
that
the
duo
up
stuff
isn't
going
to
necessarily
match
either
because
it's
it
thinks
about
reads
versus
rights,
and
this
is
sort
of
neither
it's
like
a
I
think
either
way.
You're
going
to
have.
C
E
B
C
But
it
oh,
but
yeah
I
mean
it's
it's
it's
both
less
structured
in
the
a
disaster
sure,
but
it's
I
mean
we
can
define
whatever
structure.
We
want
right
if
we
just
say
that
it's
know
it
outputs,
yeah,
Jason,
written
these
arguments
are
specified
of
this
type
in
Jason,
so
it
will
just
parse
it
apart.
State
object
team.
It's.
B
C
It's
like
hundreds
and
hundreds
of
lines
of
boilerplate
crap,
but
that's
in
the
rate
of
Steel
I
told
they
have
to
implement
and
in
the
liberators
API
and
in
like
it's
so
much
I
dabble
the
lazy
here.
But
it's
like
so
much
less
work
that
semester
that
that
it
said
that
other
stuff
is
buying
us.
That
much
because
we
have
to
fun
with
you
object,
ER
and
inland
burritos
and
then
the
race
cool
and
we
had
to
format
it.
And
then
we
and
then
we
have
to
party.
D
C
C
B
C
C
C
B
D
C
D
C
C
B
I
also
worry
that
the
chore
of
plumbing
the
stuff,
through
the
object
ER
into
the
greatest
tool,
is
causing
us
to
pour
more
stuff
at
a
tell
that
it
done
perhaps
which
we
should
be.
If
the
problem
is
that
we
actually
shouldn't
have
any
object
or
users
and
ndf
go
straight
through
the
burritos,
and
they
should
be
the
same
air
face
rather
than
having
to
bounce
through
the
app
vector
through
the
parade
of.
B
C
Okay,
so
that
much
is
true,
and
but
even
so,
just
the
fact
that
you
have
to
add
support
for
this
in
the
rate
of
CLI
tool,
and
the
nice
thing
about
tell
is
that
it's
it's
it's
an
out-of-band
administrative
interface
that
is
no
intentionally
more
flexible.
Like
sure
you
can
have
structured
things
in
there.
C
E
C
E
B
D
C
Well,
yeah,
I,
guess:
I
think
the
key
is
to
get
the
plumbing
stuff
in
the
OSD
to
sort
of
do
this
and
then
and
also
the
stuff.
That
obviously
needs
to
go
in
the
object
or
like
being
able
to
read
from
particular
replicas
yeah
and
do
is
t
tracking
that
metadata
and
then
at
the
point
which
it
comes
time
to
expose
it
without
hopefully
more
insight
and
certainly.
B
C
Guess
sorry,
the
last
thing
is:
if
it
is,
if
it
is
a
tell
thing,
then
the
input
arguments
are
also
unstructured.
So
you
can
add
additional
arguments
after
the
fact,
whereas
if
it's
part
of
a
burritos
and
then
you
literally
have
to
like
revised
liberate
us
to
add
new
arguments
to
the
function
in
stood
right.
D
A
Right
I
think.
Actually,
this
is
a
probably
good
time
to
take
a
15
minute.
Break
I
didn't
build
a
break
into
the
afternoon,
but
it
is
getting
a
little
long
in
the
tooth
here.
It
also
will
take
about
15
minutes
and
that'll,
bring
us
right
back
on
schedule
and
we'll
pick
up
with
a
little
bit
more
OSD
stuff
and
talk
about
local,
irreparable
codes
and
go
from
there.
We'll
see
everybody
back
here
in
a
lot.
15
sounds
good.