►
From YouTube: Ceph Developer Monthly 2021-09-01
A
B
B
B
B
B
B
C
B
B
C
D
D
D
A
D
B
B
D
B
A
C
A
A
C
B
A
A
A
The
from
an
implementation
that
would
probably
look
something
like
the
config
now
has
some
kind
of
a
baseline
config
that
lives
above
the
most
basic
defaults
but
below
the
specifically
set
ones.
Something
like
that.
You
need
a
way
to
dump
all
this
stuff,
so
there
need
you
to
be
ui
elements
for
exposing
that.
C
C
Yeah,
it
just
needs
to
be,
I
mean
it
could
even
be
implemented
at
the
well.
Maybe
if
we
implement
it,
it
couldn't
implement
it
entirely
in
the
monitor
and
then
it
would
be.
The
monitor
would
feed
it
through
to
the
demon,
and
so
the
source
of
damage
is
concerned.
To
pick
options
come
up
from
the
monitor
or
you
could
plummet
so
that
that,
at
the
demon
level
you
can
actually
see
the
source.
B
C
C
C
A
C
Well,
if
we,
if
for
any
given
profile,
you
say
that
it's
driven
by
a
unnamed
config
option,
then
you
could
set
that
config
option
so
like
say
that
the
osd
scale
or
whatever
osd
profile,
is,
has
a
set
of
key
value
pairs,
and
you
have
all
the
different
like
values
for
it.
You
could
define
what
the
config
option
is
that
sets
the
profile,
and
then
you
could
set
that
using
the
normal,
like
hierarchical,
whatever
config
thing.
B
A
C
B
For
example,
it's
got
a
file
system
running
on
top
of
it's
it's
it's
local,
lock
device
and
if
the
file
system,
like
the
xp4
or
xfs,
starts
getting
io
errors
because
the
disk
backing
in
in
that
poor
one
died,
it's
not
going
to
behave!
Well,
it's
going
to
essentially
craft
this
local
power
system.
If
it
gets
some
eaos
and
some
successful
rights.
B
B
C
C
C
B
B
E
E
C
E
C
The
reason
why
I'm
partial
to
the
pool
eio
flag
is
that
it
separates
out
the
sort
of
the
implementation
mechanics
of
having
a
well-defined
point
in
time,
after
which
you
return,
I
o
and
before
which
you
don't
and
having
it
that
ordering
so
that
it
behaves
the
way
like
maybe
a
disc.
If
you
disconnect
the
cable
or
something
I
behave
well,.
C
E
E
C
The
point
is
that
it
separates
that
policy
decision
of
when
to
return
the
I
o
from
the
mechanics
of
saying,
okay
set
the
flag
in
the
osd
map,
and
now
we
return
the
I
o
for
that
pool
and
we
can
make
sure
that
the
client
implementation
and
the
osd
are
properly
coded
so
that
that
behaves.
You
know
in
a
way
that
doesn't
have
weird
ordering
issues
or
apps
that
are
resubmitted
later
or
happen
to
be
in
flight
like
we
can
make
sure
those
issues
are
don't
exist
and
then
separately.
We
can
have.
E
D
C
If
you
say
you
have
a
hung,
osd
and
you
have
a
bunch
of
ops
that
are
in
flight
and
are
hanging,
and
then
you
set
the
flag,
so
those
ops
that
were
already
in
flight-
maybe
maybe
the
ocd
actually
instant
down
and
the
up
the
op
like
made
it
to
the
osd,
is-
was
processed
or
maybe
it
didn't.
And
so
I
think
I
think,
the
the
implementation
that
made
sense
to
me
at
least
was
that
for
existing
ops
on
the
client
side.
We
don't
do
anything
and
we
wait
for
the
osd
to
respond.
A
A
I
think
there's
a
simple
way
to
do
this.
The
pool
eio
flag
is
semantically
equivalent
to
deleting
the
pool
the
osd's
immediately
start
getting
rid
of
their
state.
They
drop
any
in
flight
ops
period.
The
client
eventually
sees
the
flag
goes.
Oh,
that's
why
it
didn't
get
any
responses.
It
starts
returning
yeah.
C
D
C
Doesn't
have
to
so
I
was
assuming
that
this
could
be.
You
could
remove
this
flag
like
some
time
went
by
and
you're
like.
Oh,
I
didn't
actually
lose
the
data.
I
don't
actually
want
to
delete
it,
because
if
you
just
wanted,
we
could
just
delete
the
pool
and
we
could
get
what
you're
talking
about
right
and
we
just
fix
the
client
so
that
if
the
pool
doesn't
exist,
then
you
get
the
io
instead
of
you
know
something
like
that.
A
C
D
C
It's
discounted
for
this
long
then
like
make
it
look
like
the
disc
guide,
and
you
start
returning
io
and
then
you
know
12
hours
go
by
and
the
administrator
realizes
that
their
system's
down,
because
that
happened
to
too
many
disks
and
it
turns
out,
we
didn't
actually
lose
any
data.
They
were
just
offline
for
a
long
time
and
the
policy
that
was
triggering
this
like
didn't
realize
that
it
killed
two
volumes
that
would
break
the
upper
application,
all
the
data's
still
there,
and
so
you
would.
C
A
A
A
E
A
E
A
I
mean
I
personally
don't
think
that
the
semantic
difference
matters
very
much
because
I
don't
think
the
I.
I
think
we
will
almost
never
use
this
feature
on
purpose
on
a
replicated
pool,
I
think
99.999
percent
of
the
time
this
will
be
used
immediately
before
deleting
a
pool
and
that's
the
use
case
we
really
care
about.
So
I
don't
think
we
want
to
go
out
of
our
way
to
like
make
this
harder
just
to
make
replication
behave
in
a
slightly
less
weird
fashion.
Oh.
C
The
client
well,
I
think
I
think
there
are
two
things
first.
I
think
I
disagree
with
sam
that
this
is
99.9
when
you're
going
to
delete
the
data,
because
I
think
that
actually
I
mean
sometimes
that's
going
to
be
true,
but
I
think
they're
going
to
be
lots
of
cases
where
the
osd
is
offline
for
some
reason
and
that's
going
to
trigger
all
the
policy
that
makes
these
disks
appear
to
fail.
But
the
oc
actually
isn't
like
it's,
not
because
the
disk
failed.
B
C
Hopefully,
but
maybe
two
hosts
are
down
and
you
do
it
the
two
discs
and
can't
recover
until
is
stuck.
You
turn
the
host
back
on
they
come
up
like.
I
think
this
might
actually
need
to
be
coupled
with,
like
the
equivalent
of
the
there's,
some
other
annoying
osu
flag.
That's
like
auto
out
auto
out
so
that
if
it
comes
back
up,
it
clears
it
automatically
or
maybe
whatever.
That's
that's
a
separate
discussion
but
like
it,
we
need
to
the
having
plugging
the
discs,
conceptually
or
logically
or
whatever,
plugging
metaphorically
plugging.
C
B
B
D
A
D
D
I
still
want
to
recover
something,
so
I
will
say:
okay,
I
I
will
do
this
manual
like
let
us
say,
there's
a
cascading
failure
like
what
sage
is
describing
that
all
the
nodes
for,
even
though
their
replica
one
pulse
suddenly
became
unavailable.
At
that
point,
we
want
to
still
recover
the
data,
but
it
was
just
one
off
incident
that
happened
and
we
don't
care
about
the
data
that
should
be
like
a
manual.
A
B
A
And
paserio
I
was
imagining
wasn't
so
much
administrator.
It
was
ocs
observed
that,
for
whatever
reason,
a
pool
marked
scratch
or
something
replica,
one
became
unresponsive
and
it
automatically
took
its
own
steps
to
increase
the
replication
factor
on
or
whatever,
and
because
this
is
a
thing
that
has
to
happen
automatically.
Freeing
up
the
space
also
has
to
happen
automatically.
C
E
A
E
B
A
B
C
I
mean
I
think,
live
rvd
might
yeah.
It
probably
doesn't
already
do
that.
Xfs,
for
example,
will
do
that
if
it
gets
any
io
on
any
of
its
metadata
structures,
it
goes
into
like
this
zombie
state
or
you
can
maybe
read
stuff
or
whatever,
but
it
won't
write
anything
else
if
whatever
it
should
do
the
same
thing
like
once
it
sees
in
the
I
o.
E
C
D
C
D
A
C
B
C
A
A
C
So
I
think
that
that
take
away
from
this
is
that
it
requires
osu
changes
that
are
like
relatively
well
defined
and
understood.
There
are
some
client
changes,
also
again
a
mirroring
of
what's
up,
but
both
right.
That
means
that
burritos
needs
to
be
changed.
The
kernel
client
needs
to
be
changed,
so
if
this
is
going
to
go
in
order
for
this
actually
used
like
we
need
to
get
those
all
those
changes
upstream
into
the
upstream
kernel
and
into
the
downstream
kernel
and
all
that
stuff.
So.
D
C
A
C
C
But
the
part
is
that,
if,
if
we
are
thinking
about
these
as
analogous
to
disks,
then
I
think
there
are
two
ways
to
do
it.
We
could
we
could
set
it
up,
so
these
pools
are
like
actually
sort
of
correspond
to
disks,
and
then
you
have
like
multiple
rbds,
maybe
on
the
same
disk
disk,
slash
pool
right,
so
the
pool
becomes
the
failure
domain
and
then
we
have.
If
there
are
lots
of
applications
that
are
doing
this,
then
they
would
have
multiple
they
would
each
have
their
own
rbd.
That's
in
that
pool.
C
C
E
C
A
C
C
A
C
A
C
A
E
E
B
C
A
A
C
A
I
mean
probably
we
shouldn't
make
the
object
or
die
any
more
aggressively
than
it
needs
to,
but
rbd
is
like
no
rpd
users.
User
is
ever
going
to
continue
operating
without
actually
reinitializing
rbd,
because
no
one's
going
to
want
to
build
some
kind
of
an
online
reinitialize
setup
thing.
That's
too
much
work
for
no
reason
waiting.
Just
on
map
and
remount.
B
So
the
idea
with
this
one
is
there
are
different
ways
that
we
can
try
to
test
very
large
scale
clusters
without
necessarily
having
physical
hardware.
That's
with
they
might
we
can't
necessarily
test
the
capacity
or
the
performance
characteristics
exactly
the
same
way
without
that
same
number
of
osds,
but
just
have
it
having
say
thousand
supposed
to
use
in
a
very,
very
small
number
of
nodes.
B
B
More
efficient
to
create
use
a
little
bit
of
the
persistent
storage
on
disk
for,
like
the
metadata
pools
and
os
internal
metadata
for
like
pg
log
pg
info
that
kind
of
thing
so
that
we
can
restart
the
osds
and
test
peering
and
well
recovery
of
no
data.
Perhaps
but
it's
like
with
gabby
about
this
earlier,
and
he
was
guessing
like
having
that
the
data
pools
essentially
be
like
devnl
and
subzero.
So
rights
go
nowhere
and
reads:
return
zeros
or
some
deterministic
data
based
on
the
h
object.
B
B
C
B
B
A
Object
creation
flight,
whatever
hint
that
says,
I
assume
you're,
going
to
lie
to
me
about
the
contents
of
the
subject
and
then
create
a
pool
flag.
That
says,
don't
actually
store
any
of
the
data
for
the
objects
in
this
pool
yeah.
This
is
a
fake
pool
like
the
pool
will
store
all
of
its
metadata.
It'll
do
all
the
usual
appearing
stuff.
It's
got
a
pg
log,
but
the
actual
objects
won't
have
any
data
they'll
just
return.
Zero
is.
A
E
A
C
B
C
I
mean
you
could
do
the
same
with
menstrual.
You
could
think,
I'm
sorry
do
the
zero
detection
and
then
you
could
also
do
this
with
a
bazillion.
Although
I
the
thing
that's,
I'm
sorry
is
that
you
probably
want
to
test
things
like
you
know,
stopping
a
bunch
of
osds
and
starting
them
up
again
and
like
all
those
like
crashing
type
behaviors
and
I'm
sorry
doesn't
do
that
super
well,
I
actually
do
want
to
write
down
everything
else.
A
A
A
C
So
I
wrote
a
bunch
of
like
fake,
monitor,
osd
things
like
I
don't
know
10
years
ago
and
they've
just
languished.
I
I
periodically
have
to
like
touch
them.
If
I
ever
like
changed
an
interface
and
there's
all
this
like
annoying
test
code
that
never
gets
run.
Probably
I
have
to
go
fix.
I
might
have
finally
ripped
it
out,
but
it's
I
I
think
the
maintenance
burden
is
like
I
don't
know.
C
A
C
B
E
C
I
guess
the
the
concern
I
always
have
with
faking
listings
is
that
you're
never
sure
that
you've
faked
it
right
so
yeah.
You
might
run
this
like
crazy
scale.
Tester
like
monitors
are
great
but
like
you're
running
they're
running
getting
us
pickles
these
that
aren't
realistic
and
realistic
or
have
some
different
behavior.
That's
the
same
plus.
C
E
A
B
C
Yeah,
sorry,
I
was
distracted.
That's
what
right
now
that
the
ansible
job
is
the
thing
that
every
every
time
the
machine
is
brought
up
for
a
particular
test.
It
creates
recreates
those
like
four
lvs
and
they're
each
100
gigs
right
now,
and
so
we
could
have
an
alternate
answer
which
or
whatever
it
is
so
we
create.
Instead
of
creating
four
of
them,
we
create
40
that
are
10
gigs
each
as
long
as
bluestorkin,
fine,
so
yeah.
C
C
C
C
B
C
B
C
B
E
E
A
C
So
I
think
the
counter
argument
is
that
maybe
these
things
should
be
using
the
the
new
local
storage
operator
thing
that
we've
been
talking
about
and
they
shouldn't
be
using
ceph
at
all,
and
obviously
that
has
the
benefit
that
you
get
better
performance
you're
not
using
this
complicated
stack
to
do
it.
I
think
the
counter
argument
to
that
is
there
are
cases
where
replica,
I
think,
their
cases
where
replica
one
makes
sense,
and
you
want
the
full
feature
set
that
stuff
provides
like
rbd
mirroring,
for
example.
Well,.
E
A
C
A
A
A
A
E
C
E
C
A
E
C
Because
that's
not
necessarily
enough,
it's
it's
a
policy
thing
like
what.
Maybe
that's
what
the
environment
user
administrator
application
wants,
and
maybe
it
isn't
so
I
I
really
feel
like
we
could
implement
a
particular
policy
in
a
manager
module.
Some
external
thing
can
implement
its
own
policy
and
just
call
like
fosd
pool
set
flag.
Eio
whatever
like
this
like
opens
up
all
those
possibilities,
since
we
have
total
flexibility.
C
C
B
B
E
A
What
I,
what
I
mean
is
that
there
should
be
business
logic
and
stuff
that
takes
the
raw
information
from
the
current
stats
and
makes
a
choice
based
on
the
set
policy.
Yeah
yeah,
that
business
logic
should
live
in
the
manager
and
not
in
ocs,
see.
I
should
specify
the
policy,
not
the
implementation
yeah.
Otherwise,
we
get
into
things
like
ocs
interpreting,
really
specific
details
of
like
status,
which
is
probably
not
good,
probably
not
a
good
thing.
C
C
D
C
A
A
Oh,
I
know
what
I'm
what
I
mean
is
the
the
time
frame
is
less
about
cassandra's
expectations
of
when
it
does
fails
and
more
about
how
long
you
should
go
before
ocs
should
start
setting
up
another
replica.
It's
replication
delay,
I
think
not
not
the
application's
internal
expectations
or
a
recovery
point
objective.
Whatever
time
you're
allowed
to
be
degraded.
A
C
I
think
the
the
good
news
is
that
the
fact
that,
like
the
I
thought
it
was
it's
like
the
non,
I
thought
there
was
some
category
of
device.
I
don't.
Maybe
it's
not
stated,
but
it's
like
the
non-enterprise
whatever,
where,
like
you,
can
get
ios
that
just
take
a
long
time
before
the
firmware
decides
to
time
it
out.
E
E
A
A
Much
application
correctness
as
how
does
it
behave
when
a
hang
happens
and
how
long
are
we
supposed
to
wait
before
spinning
up
another
one
yeah,
because
with
kubernetes
you're
going
to
be
setting
up
what
a?
What
do
you
call
it
replica?
No,
I
forgot,
but
it's
like
kubernetes.
Please
keep
end
of
these
alive.
Basically,.