►
From YouTube: Ceph Code Walkthrough 2020-08-25: kRBD I/O Flow
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
B
B
B
And
then
we
have
the
monitor
client.
The
osd
client,
which
is
basically
the
equivalent
of
objector
in
liberators
osd
map.c,
is
where
we
decode
the
osd
map,
invoke
crush
and
post
process
crash
to
get
the
actual
object
placement,
because
we
need
to
account
for
things
like
digi
temp
settings,
dj
up
map
settings,
you
know
primary
affinity,
etc.
B
B
So
the
way
rbd
images
are
mapped
and
unmapped
is
through
the
ccfs
interface.
It's
done
by
writing
to
these
provide
only
files
which
are
called
attributes
and
when
you
type
rbd
device
on
map
or
rbd
device
on
map
on
the
command
line,
the
rbd2
constructs
the
configuration
string
and
writes
it
to
one
of
these
attributes.
B
Of
course,
there
is
a
bit
more
to
it
because,
at
least
for
mapping
the
command
line
two
has
to
parse
the
supplied
options.
It
has
to
fetch
the
sefax
key
and
add
it
to
the
kernel
keyring
and
after
the
image
is
mapped,
it
waits
until
udab
finishes
processing
the
associated
events
and
creates
various
siblings,
although
that
does
interfere
with
container
use
cases
and
we're
currently
working
on
fixing
that,
but
in
a
nutshell,
all
it
takes
is
a
single
write
system,
call
reason
that
there
are
two
add
attributes
and
two
remove
attributes.
B
This
add
and
add
single
major
and
remove
and
remove
single
major.
The
reason
is
that
we
support
two
major
minor
device
numbering
schemes
under
the
legacy
scheme.
Hrbd
mapping
consumes
a
major
number
for
itself
and
the
problem
with
that
is
on
a
typical
linux
system.
There
is
only
about
200
majors
available,
and
so
this
limits
the
number
of
rbd
mappings
you
can
have
on
the
node
and
potentially
also
cripples
the
known,
because
if
all
available
major
numbers
get
allocated
to
the
rvd
driver,
attempts
to
create
other
devices
might
fail.
B
B
C
B
C
B
B
B
And
here,
if
we
find
one,
we
bump
a
reference
count
on
it
and
return
it,
and
here
we
ensure
that
the
returned
libsar,
for
instance,
has
the
latest
osd
map,
because
it
may
have
been
sitting
there
idle
and
it
could
be
that
the
pool
we
are
mapping
an
image
from
has
just
been
created,
and
this
lipstick
instance
hasn't
learned
about
it.
Yet
because
it
serves
the
map,
is
you
know
a
few
epochs
behind?
B
A
B
B
We
need
to
be
careful
not
to
overrun
it,
and
the
first
thing
we
do
here
is
look
up
the
image
id
based
on
the
image
name,
and
there
is
a
special
rbdid
object
for
each
image,
and
what
we
do
here
is
we
format
a
name.
We
format
its
name.
This
is
a
known
prefix,
followed
by
the
image
name,
and
we
call
a
class
method.
B
B
This
is
how
we
know
that
the
image
has
been
resized
or
a
snapshot
has
been
created,
for
example,
and
then
we
need
to
refresh
our
view
of
the
image
header
and
also,
this
is
how
the
exclusive
log
is
implemented.
So
you
can
see
some
exclusive,
lock
notifications
here,
apart
from
the
header
update
one,
the
second
callback
is
called
on
errors,
and
here
what
we
do
is
just
let
everyone
know
that
there
has
been
an
error
and
a
cue
and
work
item
to
reestablish
your
watch
when
that
becomes
possible.
B
B
B
B
B
You
get
a
10
gigabyte
clone
if
you
shrink
that
clone
down
to
say
5,
gigabytes
and
then
growing
back
to
10
that
second
half
of
the
parent
image
could
no
longer
be
visible
for
the
clone,
because
it
should
be
all
zeros
just
like
if
you
had
shrunk,
a
regular
standalone
image
and
then
grown
it
and
the
parent
overlap.
Value
is
how
we
track
this,
and
whenever
we
calculate
a
mapping
of
clone
extends
onto
the
parent
extent.
B
We
take
it
into
account
and
proven
extents
that
are
outside
the
overlap
region,
and
here
we
actually
probe
the
parent
image
that
we
didn't
find
any
parent
in
the
previous
function,
which
is
bail.
Otherwise,
we
make
sure
that
our
parent
chain
is
not
too
long
and
after
creating
a
structurability
device
for
the
parent
image,
call
rbd
of
image,
probe
recursively
with
an
incremented
depth,
and
this
can
go
on
for
a
while,
because
this
parent
can
have
its
own
parent
and
so
on.
B
B
B
So
now,
if
there
is
no
object
map,
we
we
can
announce
the
disk,
but
if,
if
there
is,
if,
if
there
is
an
exclusive
log
or
an
object
map,
what
we
want
to
do
is
grab
the
exclusive
log
here
and
once
that's
done,
load
the
object
map,
because
we
want
to
attribute
the
latency
of
grabbing
the
exclusive
lock.
I'm
loading
the
object
map
to
the
parent
to
the
mapping
process
instead
of
the
first
io
request.
C
B
Can
you
see
it
now?
I
am,
at
the
rbd,
add
acquire
log
much
better
thanks,
yep
and
yeah.
So
once
the
lock
is
acquired
and
the
object
map
is
loaded
and
the
object
map
gets
loaded
from
the
post
acquire
handler
that
the
exclusive
log,
our
state
machine,
calls
when
the
exclusive
clock
is
acquired,
I'm
not
going
to
show
that
we
announced
the
disk
to
the
world
and
they
returned
to
suse
fast
and
from
there.
B
B
Block
device
id
and
an
optional
force
flag,
and
here
we'll
after
parsing
that
we'll
look
up
a
struct
rbd
device
with
the
with
the
id
that
we
need
and
check
the
open
count
if
the
image
is
still
opened
by
someone
we
refuse
to
unmap,
but
the
force
flag
overrides
the
open,
gown
check.
This
override
was
added
with
a
specific
iscsi
related
use
case
in
mind,
and
it
isn't
really
useful
for
anything
else.
So
don't
use
it.
B
B
A
D
B
B
All
right,
let's
jump
to
serving
io
the
entry
point
here
is
rbdq
iq
function,
there's
the
definition
it
is
registered
with
the
block
layer
and
the
block
layer
calls
it
for
each.
I
o
request
and
wants
us
to
handle,
and
here
we
grab
the
pre-allocated
struct
rbd
image
request:
translate
the
block
layer
up
code
into
the
rbd
code,
mostly
just
for
historic
reasons.
As
you
can
see,
the
mapping
is
now
one-to-one
and
initialize.
The
image
request.
B
Note
these
two
operations
discarded
and
zero
out
different
for
a
long
time.
We
guarantee
that
this
card
would
zero
the
discarded
region
and
there
used
to
be
a
system
attribute
called
discarded,
underscore
0's
underscore
data
which
was
set
to
true,
and
some
people
relied
on
that,
but
with
the
addition
of
the
zero
out
up,
this
is
no
longer
the
case.
B
The
semantics
have
been
relaxed
and
if
your
discard
request
is
small
enough-
and
we
suspect
that
nothing
will
actually
be
deallocated
on
the
usbs,
we
will
simply
drop
it
or
if
your
discount
request
is
big
but
not
suitably
aligned,
we
will
reshape
it
and
discard
a
smaller
region
and
then
request
it
zero
out
up.
On
the
other
hand,
this
is
guaranteed
to
zero
every
single
night.
We
never
second
guess
zero
out
requests.
B
So
if
you
want
zero
in
semantics,
that's
what
you
should
use
and
in
the
end
we
offload
the
actual
work
of
putting
together
the
image
request
to
the
work
view.
This
is
done
because
the
block
layer
has
certain
restrictions
on
what
this
qrq
handler
can
do
and
in
particular
it
cannot
sleep,
but
we
actually
need
to
take
a
couple
slipping
locks
in
the
process.
B
So
we
offload
to
this
rbd
keyword,
fm
function,
and
here
we
grab
the
offset
and
length
from
the
bulk
layer
request,
which
is
referred
to
as
rq
here
run.
Some
checks,
capture
the
snapshot,
context
or,
and
some
other
element
fields
from
the
from
the
image
header
and
move
on
to
filling.
The
image
request.
B
Discard
and
zero
out
requests,
don't
have
any
data
pages,
but
read
and
write
requests.
Of
course.
Do
the
block
layer
request
is
more
or
less
just
a
single
linked
list
of
this
so-called
bio
structs
hbiostruct
contains
a
bunch
of
data
pages
in
it
which
we're
going
to
need
to
either
write
out
or
within
to,
and
here
we
pass
the
pointer
to
the
head
of
the
bio
list.
B
B
B
B
Basically,
it
works
through
the
given
image
extent.
Does
the
mapping
one
piece
at
a
time
and
any
time
it
encounters
a
new
object.
It
calls
this
allocathem
callback
and
anytime,
it
counters
a
new
stripe
unit.
It
calls
this
action
fn
callback
and
if
you
look
at
how
we
invoke
it
for
the
a
new
object
callback,
we
pass.
B
This
alec
object
extend
function
that
just
allocates
a
new
object
request
and
adds
it
to
the
current
image
request
and
the
image
required
basically
serves
as
a
container
for
the
group
of
related
object
requests
and
for
the
stripe
unit
callback.
We
pass
this
stuff
from
the
appearing
context
and
again
this
is
how
data
pages
get
added
to
the
right
object,
requests
in
the
right
order,
no
matter
how
fancy
your
layout
is
and
on
return
from
file
to
extent.
B
B
And
the
the
actual
state
machine
resides
in
this
function
that
begins
with
two
underscores:
it
returns
a
boom
and
when
it
reaches
a
final
state,
it
would
be
true,
and
when
that
happens,
there
are
two
cases.
If
this
image
request
is
not
a
request
to
the
current
image,
then
we
just
let
the
block
layer
know
that
we're
done,
but
otherwise
we
need
to
kick
the
state
machine
in
the
image
above
us-
and
this
is
what
this
branch
does.
B
B
We
assert
that
either
we
don't
need
the
lock
or
we
are
the
lock
owner
and
we
kick
off
the
object,
request,
state
machines
and
again
an
image
request
is
really
just
a
container
for
object
requests.
So
all
the
work
happens
inside
inside
object,
request
state
machines.
Here
we
just
wait
for
them
to
complete
and
gather
the
result
once
the
pending
count,
which
is
zero.
We
return
true,
which
signifies
the
termination
on
the
state
machine.
B
B
There
we
don't
have
to
deal
with
object,
map
updates
and
a
whole
separate
copy
up
state
machine,
business,
discards
and
zero
outs
go
through
the
write
state
machine
because
they
modify
the
image
just
like
a
regular
write
does
and
so
on
a
read.
The
first
thing
we
do
is
we
consult
the
object
map
if
it's
present,
if
the
object
map
says
that
the
object
doesn't
exist,
we
move
on
to
handle
to
handling
email
and-
and
this
is
where
it
happens.
B
B
Second,
if
the
client
crashes,
after
updating
the
object
map,
but
before
creating
the
object,
the
object
map
will
have
a
record
of
an
object
that
doesn't
actually
exist
in
radius
and
that's
fine,
but
the
reverse
is
not
possible
if
they
are
because
if
it
were
possible,
it
would
lead
to
data
corruption
if
the
object
exists
in
radius.
The
object
map
will
always
have
a
record
of
it.
B
First,
the
state
is
transitioned
from
exists
to
dependent
division
and
then
the
delete
is
performed
and
then
the
state
is
transitioned
from
pending
division
r
to
non-existent,
and
this
is
how
we
ensure
that
the
reverse
and
that
inconsistency
can't
ever
happen
back
to
the
read
state
machine.
If
the
object
map
says
that
the
object
may
exist,
we
issue
a
radius
read:
it
can
play
with
annoyant
if
the
object
is
not
actually
there,
and
that
would
mean
that
our
object
map
is
inconsistent.
B
But
again,
that's
fine,
and
here
is
how
our
read
is
issued.
We
allocate
an
sd
request,
format,
read
rsd
op,
allocate
the
messages
and
submit
them
to
rip
chef
and
or
when
we
allocate
an
osd
map.
We
provide
a
callback
which
alexef
invokes
when
it's
done
with
the
request
and
in
this
callback
we
basically
just
grab
the
return
value
and
kick
the
state
machine
for
the
associated
object,
request.
B
B
The
extents
that
are
completely
beyond
the
overlap
mark
are
dropped
and
the
final
overlapping
extent
is
trimmed.
It
could
very
well
be
that
all
parent
extents
get
dropped
here
and
we
return
in
an
empty
array,
and
in
that
case
it
is
no
different
from
an
email
and
without
a
parent
image.
We
handle
it
again
by
zeroing
the
request,
but
if
pruning,
if
the
pruning
process
left
us
with
at
least
one
parent
extent,.
B
B
It
with
the
object
request
that
we're
currently
processing
capture
the
necessary
fields
from
the
header
and
fill
it
in
much
the
same
way
as
we
filled
the
original
objective
image
request
that
was
initiated
by
the
block
layer.
The
only
difference
is
that
this
image
request
is
initiated
by
the
object
request
that
we're
currently
trying
to
process
and
it's
going
to
get
one
or
more
of
its
own
object
requests
and
they
will
be
filled
again.
B
B
B
B
B
B
Yes,
that's
what
I've
showed
in
that's
what
I've
showed
earlier
in
this
rb
init
disk
function.
This
this
parameter
refers
to
the
number
of
hardware
queues,
so
the
number
of
the
number
of
software
cues
is
not
controlled
by
the
log
device
driver
at
all.
It's
totally
up
to
the
multi-queue
framework,
the
block
device
driver
controls,
the
number
of
hardware
cues,
and
we
set
it
to
the
number
of
present
or
present
cpus
just
to
in
the
hopes
of
increasing
parallelism.
B
We
haven't
actually
benchmarked
this.
This
is
a
fairly
recent
change.
We
used
to
have
a
single
hardware
rq,
but
that
was
because
we
had
some
global
logs
in
the
lipsap
kernel
module,
which
you
know
which
any
sufficiently
parallel
submissions
from
the
rbd
driver
would
bump
into.
But
those
have
been
six
have
been
fixed
a
long
time
ago,
and
so
we've
we've
flipped
this
number
of
hardware
queues
in
a
number
of
cpus.
D
D
Is
there
any
what
I
basically
understood
from
the
code
walkthrough
and
when
I
read
some
scientific
literature
regarding
rpd
as
well?
Is
there
any
way
someone
can
abstract
the
kernel
module
to
the
user
space,
because
what
I
understood
is
that
the
entire
self
client
kernel
module
is
the
the
entire
set
time
module
is
in
the
kernel
module,
so
it
has
no
user
space
involved.
B
B
Yes,
if
you
want
to,
you
know,
make
use
of
the
user
space
code
but
get
a
kernel
block
device
presented.
We
have
the
rbd
npd
driver
for
that
and
that's
basically,
you
know
the
the
kernel
has
an
mbd
client
and
we
place
an
mbd
server
on
top
of
an
instance
of
library
and
they
talk
via
the
nbd
protocol
and
that
way
the
rbd
image
ends
up
being
exported
to
the
kernel,
but
for
krbd
and
lipsaf
and
saffs.
B
These
are
totally
separate
from
scratch.
Implementations
in
kernel,
space,
there's
nothing
shared,
except
for
the
implementation
of
the
crush
algorithm,
and
it's
the
same
and
it's
written
in
c.
So
it's
it's
the
same
in
libretus
and
in
lipsef,
and
also
some
header
files
are
shared
which
define
stuff
like
latest
feature
bits
and
these
some
parts
of
the
one
wire
format
such
as
messenger
tags,
message,
header
and
message
footer.