►
From YouTube: Ceph Developer Monthly 2021-06-02
B
B
B
B
C
It
seems
like
there
are
sort
of
two
roads
to
go
down.
If
I'm
understanding
this
correctly,
like
I
mean
clearly
master
shouldn't
crash
right,
but
you
could
imagine
continuing
with
the
old
behavior
where,
if
the
ocd
restarts
we
sort
of
forget
that
the
object
was
unreadable
and
had
an
eio
and
then,
if
we
try
to
read
it
again
or
we
scrub,
then
it
appears
it's
unfound
again.
B
B
D
Certainly
for
the
older
versions,
I
think
the
right
answer
is
to
shore
up
the
existing
behavior
don't
crash,
but
don't
try
to
fix
it
either
per
se
scroll
we'll
catch
it
later
and
that's
good
enough
well
sort
of
for
master,
though
we
now
have
the
capacity
to
write
down
things
that
are
in
the
missing
set
so
plausibly.
We
could
add
a
message
to
the
back
full
process
that
forcibly
adds
things
to
the
missing
set
on
replicas.
D
So
when
we
do
a
backfill
read
if
we
get
back,
if
we
learn
during
that
process,
further
ratio
coded
pool
that
some
replica
doesn't
have
a
copy
or
the
primary
doesn't
for
a
replicated
pool.
We
can
add
that
to
the
missing
set
and
wait
for
that
to
be
done
before
before
letting
glass
backfill
advance,
which
would
give
us
the
state
required
to
ensure
we
don't
forget
it
later.
D
That
would
also
give
us
the
right,
primitive.
I
think,
for
dealing
with
this
and
spread
as
well.
When
we
find
one
of
these
objects
isn't
where
it
should
be
or
is
unreadable,
we
can
add
it
to
the
missing
cell,
the
relevant
osd
and
continue
on
that
way.
It'll
give
us
sort
of
a
baseline
piece
of
state
to
deal
with
these
things
in
general,.
D
C
A
A
A
E
I
maybe
missed
the
initial
part
of
this
discussion,
a
few
minutes
late,
but
so
nicola
from
what
I'm
understanding
from
your
latest
test
results.
Is
that
with
the
patch
that
you
applied
on
nautilus,
the
same
test
that
is
failing
in
master
is
not
failing
in
nautilus,
so
everything
seems
just
fine,
with
nautilus,
even
with
file
store
right.
B
F
B
B
We
do
not
lost
this
state
when
a
non-primary
is
restarted
anyway.
We
still
lost
it
when,
when
the
primary
osd
is
restarted,
so
it's
kind
of
a
partial
fix,
but
yes
to
just
if
it
was
not
clear
from
the
context
we
actually.
One
of
the
issue
was
that
the
state,
this
information
about
unfound
date
was
lost
and
another
that
there
was
some
crashes
on
on
the
master
on
purpose,
not
use
branches,
and
I
think
they
are
related
to
refactoring
of
the.
E
So
yeah,
I
think
in
general,
sam
I'd
like
to
hear
your
opinion
too.
The
idea
of
maintaining
backfills
in
flight
for
anything
but
the
primary
which
which
mercola
is
essentially
manually,
backboarding,
which
you
had
in
your
refactor,
seems
right
to
me
and
that's
probably
why
that's
fixing
the
problem
and
nautilus
with
the
test
that
mikola
has
and
if,
if
that
is
valid,
then
even
the
other
patch
which
he
was
talking
about
the
crash
and
master.
E
That
is
a
follow-up
of
of
that
right,
because
you're
expecting
that
backfills
in
flight
and
the
recovering
set
are
not
the
same
except
the
only
change
is
the
primary
can
have
something
which
is
which
is
preventing
it
from
being
the
same.
So
those
two,
those
two
changes
are
currently
with
this
particular
patch
that
he's
talking
about
are
same
in
master
and
nautilus,
but
there
seems
to
be
something
extra
or
like
something
missing
in
master,
because
of
which
we
see
that
other
crash
that
he
saw
while
running
that
test.
B
D
E
Saying
yeah:
that's
what
I
was
trying
to
see
and
even
maybe
worth
looking
at,
why
master
is
crashing
versus
nautilus
is
not
crashing,
because
this
test
seems
pretty
controlled
like
this
is
just
one
pg
and
we
should
probably
look
at
why
the
same
crash
does
yeah
just
comparing
and
yeah
it's
worth
mentioning
that
this
is
only
happening
in
file
store.
I
have
no
idea
why,
because
it's
not
happening
one.
B
E
Yeah
sure
yeah,
what
I'm
just
trying
to
say
is
that
maybe
compare
that
that
same
test
that
you're
running
in
file
that
same
test,
that
is
passing
on
nautilus
and
compare
the
same
test
on
master
and
see
what
the
difference
in
code
path
is.
Why
is
master
crashing
that
that
could
give
us
a
rough
idea
about?
You
know
whether
there
was
something
missing
or
there's
some.
You
know:
invariant,
that's
being
broken
somewhere.
E
D
If
we
get
better
or
better
implementations
of
these
error
cases,
we
could
do
something
like
have.
The
error
injection
suites
deliberately
keep
a
set
of
objects
with
at
least
one
broken
copy
and
poke
at
the
relevant
osds
to
keep
that
count
where
it
should
be.
The
osds
would
then
have
a
really
high
chance
of
encountering.
One
of
these
broken
objects
being
forced
to
deal
with
it
in
self-test
radars.
C
Why
not
yeah
I
mean
it
seems
like
the
the
air
injection
that
I'm
familiar
with
right
now
is
transient
like
it's
an
asoc
command
that
has
some
state
and
if
you
restart
it'll
forget
whatever
like
it
seems
like.
If
we
have
a
persistent
thing
where
we
say
this
object
on
this
osd
should
get
eio
yeah,
and
then
we
go
to
randomly
poked
objects.
That
way,
then
we'll
be
much
more
likely
to
hit
them
when
we
start
moving
data
around.
D
G
C
D
D
C
C
A
H
H
C
I
I
I
I
I
Okay,
so
anyway,
that
was
kind
of
an
outline
of
what
I
wanted
to
talk
about.
So
there's
about
four
main
kinds
of
aws:
rados
rgw
s3,
encryption.
First
kind
of
encryption
is
a
client-side
encryption
and
that's
where
all
of
the
encryption
decryption
and
key
management
happens
on
the
client
side.
That's
that's
a
feature
that
aws
sdk
and
that
doesn't
have
any
implications
whatsoever
for
the
server
second
kind
of
encryption
is
called
sse
dash
c
and
that's
where
the
client
provides
the
key.
I
I
I
The
right
way
to
think
about
tokens
with
vault
is
that
tokens
are
basically
like
kerberos
tickets.
They
they
have
a
short
lifetime
and
you
want
to
get
a
new
one,
every
so
often
there's
at
least
20
different
ways
in
vault
to
get
tokens
and
that
that
would
mean
a
lot
of
code.
If
you
were
going
to
try
to
support
all
of
those
directly
and
so
the
right
way
to,
I
think,
to
manage
vault
tokens.
I
Is
you
run
an
instance
of
alt
agent
and
you
make
it
responsible
for
renewing
tokens
or
getting
a
new
token
as
necessary?
All
of
the
codes
in
fault
agent,
and
it's
it's
easy
to
use
it
transparently.
So
there's
there's,
there's
no
reason
inside
of
ceph
to
manage
that
vault
tokens,
except
for
in
terms
of
using
the
vault
agent
last
last
thing
that
vault
provides,
which
is
very
useful
for
seth.
I
I
Ask
it
to
decrypt
that
and
then
you
can
use
that
to
decrypt
the
actual
object
cool
thing
about
that
is.
That
makes
key
rotation
very
simple,
because
key
rotation
basically
consists
of
of
inside
a
vault.
You
asked
you
can
ask
it
to
take
one
of
these
transit
secrets
and
create
a
new
key
version,
and
at
that
point,
whenever
you're
creating
a
new
object,
you're
going
to
get
it
encrypted
under
the
new
key
and
vault
keeps
the
old
key
around
under
the
under
a
different
key
version
number.
I
I
I
I
I
I
H
I
No,
no
encrypt
encryption
is
the
the
thing
that
actually
controls
whether
the
encryption
happens
or
not,
is
the
app
attributes
that
are
set
in
the
object
at
the
time
that
you
do
a
put
object,
but
what
you
can
do
with
policy
is
you
can
require
that
those
encryption
parameters
actually
be
there?
I
think
you
can
actually
set
you.
Can
you
can
set
or
require
that
particular
values
be
provided
for
the
encryption
also,
so
that
means
you
can
create
a
bucket
that
has
to
have
has
to
have
everything
encrypted
inside
of
it.
G
I
H
G
K
I
think
I
I
think
I
I
disagree
with
that.
If
you
look
at
the
aws
s3
api,
if
put
bucket
policy,
is
enforced,
then
put
bucket
encryption
policies
enforced
then,
for
every
request
put
request
which
comes
with
a
header
which
says
that
if
you
want
server
side
encryption
only
then
it
encrypts
with
the
sc
s3
created
a
key.
K
But
if
you
want
all
the
objects
in
that
bucket
to
be
encrypted
all
the
time,
then
you
have
to
set
another
policy
on
the
bucket,
which
says
that
deny
unencrypted,
uploads
and
deny
the
I
think,
some
header
specific
header.
There
are
two
such
things
which
have
to
be
specified,
so
these
two
policies
have
to
be
then
enforced.
Only
then
all
the
objects
which
come
you
know
if
the
header
of
server
side
encryption
is
missing.
Then
the
then
this
server
throws
an
error
that
you
better
mention.
The
header.
K
K
K
I
I
H
K
No,
actually,
I
I
think
marcus
mentioned
something
about
came
up
when
to
use
gamer
versus
when
to
use
world,
so
I
was
a
little
confused.
What
I
was
thinking
from
sscs3
point
of
view
is
that
I
don't
think
ceph
will,
like
you
know,
ever,
have
its
own
kms
kind
of
thing,
so
it
will
always
talk
to
some
world
it
be.
It
came
a
barbican
or
you
know
the
hashicorp
with
which
it
is
already
integrated,
three
of
them.
K
So
at
the
back
end
we
have
any
of
these
three
kms
and
in
the
but
to
the
client.
We
say
that,
yes,
we
support
ss3
and,
depending
upon
what
type
of
kms
server
is
there
at
the
back
end.
It
would,
you
know,
have
all
those
policies
and
everything
and
generate
do
the
key
management
with
that
type
of
kms.
So
that
was
my
understanding.
I
Okay,
if
you're
specifying
the
km
kms,
if
you're
using
vault,
for
instance,
one
of
the
parameters
for
vault
is
which
one
is
it
it's
the
default
prefix
and
that
basically
names
which
part
of
the
the
vault
namespace
you're
using
for
sse
s3.
That
needs
to
be
a
different
string
than
what
you're
using
for
kms,
because
you
don't
want
users
to
be
interacting
with
the
the
secrets
that
seth
is
managing.
I
I
K
K
K
I
K
I
K
K
I
K
K
So,
in
some
sense,
if
I
understand
correctly
like
we
will
have
like
you
know,
two
sets
of
apis
or
interface,
one
would
be
which
would
do
all
the
key
management.
So
maybe
we
would
need
some
different
type
of
rados
gateway
admin.
Key
management,
sub
commands
which
would
be
required
to
you
know,
set
up
the
keys
that
if
you
want
to
create
a
pool
of
keys
or
do
you
want
to
create
a
separate
key
for
every
bucket?
I
I
I
I
My
thinking
is
that
if
you
just
want
to
use
bucket
encryption,
you
just
turn
it
on
and
you
don't
interact
with
the
radius
admin
command
at
all.
When,
when
it's,
when
you
turn
it
on
the
next
time,
staff
stores
an
object,
it
goes
out
and
sees
that
the
key
exists
in
in
vault
and
if
it
doesn't,
it
creates
a
bucket
key
automatically,
and
when
you
delete
the
bucket,
it
would
just
go
out
and
always
try
to
delete
the
key
that
should
belong
to
that
bucket.
I
I
K
Change:
okay,
the
key
name,
I'm
noting
it
down
in
the
notepad,
so
the
key
name
should
be
based
on
the
bucket
uuid.
That's
a
good
point,
however.
Like
you
know,
we
would
need
some
sort
of
a
rado
skateboard
command
for
the
key
rotation
and
wrapping
right
or
do
you
think
you
want
to
do
it
automatically?
K
I
I
At
the
the
very
least,
there
should
be
a
radius
admin
command
that
you
can
ask
it
to
force
key
rotation
that
that's.
That
seems
like
a
good
basic
starting
point.
I
think
it
would
probably
be
desirable
to
have
the
ability
to
say
rotate
keys
once
a
month
or
something
like
that.
There's
probably
a
distinction.
H
I
I
The
very
simplest
way
to
do
the
key
rotation
is
just
always,
you
know,
encrypt
new
objects
and
don't
bother
to
go
back
and
and
try
to
re
rewrap
old
keys.
But
if
you
want
to
rewrap
old
keys,
you
need
to
iterate
through,
I
think,
probably
the
simplest
way
to
do.
That
is
just
iterate
through
all
the
objects
in
a
bucket,
and
just
just
you
know,
get
the
attributes
one
by
one
and
and
re-wrap
the
attributes.
I
I
I
Rotation
like,
like
I
said,
the
key,
the
the
key
rotation
is
really
two
parts
to
just
create
a
new
key.
The
only
thing
you
have
to
go
talk
to
is
vault.
You
don't
have
to
do
anything
else
to
re-wrap
all
the
keys
in
a
bucket
you
have
to
you,
have
to
crawl
through
the
bucket
and
find
all
the
objects
encrypted
with
that
or.
K
I
K
No,
I
mean
okay,
so
I
think
if
we
have
to
do
automatic
key
rotation,
the
key
rotation
we
will
be
going
through
all
the
say,
there's
an
automatic
key
rotation
after
every
60
days,
then
we
go
to
all
the
buckets
which
have
the
policy
encryption,
enforced
and
and
then,
like
you
know,
create
the
based
on
the
uuid,
create
the
kkk
name
and
call
call
the
wall
to
rotate
those
keys
right.
That's
what
you're
saying.
I
K
I
M
M
Along
with
you
know,
a
support
for
sse
s3,
which
would
you
know
like
say,
use
the
bucket
marker,
which
is
unique
as
the
key
id
and
have
an
option
to
talk
to
any
vault
of
choice.
Whether
the
word
of
choice
would
be.
You
know
something
very
modular
and
we
could
have
multiple
implementations.
You
know
added
later,
but
we
could
start
with
one
of
the
implementations
say
something
like
you
know,
hashicorp
vault
and
which
will
have
you
know
some
of
the
safeguards
that
you
mentioned.
M
M
M
M
I
M
I
I
M
G
K
I
M
M
H
A
G
G
H
H
H
A
H
H
M
I
E
A
A
A
C
C
I'm
trying
to
figure
what's
like
if
there
are
well
look,
it
seems
like
there's
like
there's,
maybe
a
midterm
like
a
short
to
midterm,
fix
and
then
there's
like
the
long
term
strategy,
and
there
was
one
reference
at
the
end
of
this
thread.
I
think
of
there's
some
new
multi,
isolated,
sub-interpreters,
that's
experimental,
but
maybe
we'll
develop
further.
A
Term,
possibly
if
those
caveats
are
resolved,
then
it's
usable
for
everything.
I
think
that
I'm
not
sure
how
how
it's
going
to
develop.
I
guess,
but
we
could
kind
of
try
to
try
to
focus
on
a
short-term
solution
that
works
for
the
next
year
or
two
and
see
where
that
goes.
That's
kind
of
what
I'm
thinking.
C
Because,
if,
if
we
don't
need
to,
if
we
aren't
yeah,
if
we
just
need
a
short
to
medium
term
solution
that
I
wonder
if
the
simplest
would
be
just
combining
everything
back
under
a
single
interpreter
again
yep,
that's
that
is
efficient.
It's
relatively
simple.
I
think
the
only
real
challenge
is.
We
have
to
make
sure
we
only
have
one
user
of
cherry
pie,
but
that
doesn't
seem
like
it
would
be
that
hard
to
fix.
C
O
N
C
A
K
C
C
C
O
N
O
Yeah
I
was
asking
because
I
was
checking
that
it's
prediction
module
and
it
seems
like
the
well.
It
was
just
the
first
impression
right.
The
mbs
module
number
is
only
used
in
the
pre-processing
states,
so
maybe
it's
not
needed
in
in
the
running
module.
It's
only
during
some
states
or
something
so
maybe
it
can
be
imported
in
that
function
and
we
avoid
importing
that
as
a
global,
the
states
that
that
might
not
be
the
case.
A
C
C
J
C
C
C
If,
if
we
move
away,
I
mean
I'm
attracted
to
the
idea
of
just
holding
on
to
the
last
object
that
we
had
indefinitely
until
and
we're
using
it
if
the
epic
hasn't
changed,
because
there's
only
one
of
every
type
of
thing,
though,
like
the
memory
that
we
would
consume
to
be
bounded,
we
could
even
discard,
if,
like
on
the
manager
side,
we
see
that
the
epic
updated.
We
could
throw
out
the
cache
copy,
since
we
throw
it
out
again
later
anyway.
So
it
would
be
a
short
usage
of
memory.
C
I
don't
think
that
doesn't
seem
like
the
memory
usage
would
be
that
significant
and
because
then
you
don't
have
to
worry
about
correctness
on
the
side
of
the
manager
modules.
They
don't
have
to
concern
themselves
with
the
idea
that
they
might
get
something
that
isn't
out
of
date,
so
that
isn't
all
the
way
up
to
date,
which
in
like
the
case
of
like
the
balancer
module,
I
could
imagine
being
problematic
and
like
the
pg
autoscaler,
would
possibly
get
confused
if
it
had
stale
information.
C
I
had
it
seems
like
that.
The
downside,
the
main
problem
with
that
is
that
you
can't
you
can't
tell
if
one
of
the
callers
one
of
the
manager
modules
went
and
like
modified
the
object,
and
then
you
return.
The
root
pi
object
again
to
some
other
caller
and
they
see
the
modified
data
it
sticks
around.
But
I
was
thinking
that
we
could
introduce
a
have
a
debug
option
that
will
basically
regenerate
every
call
it'll
regenerate
the
object,
but
in
the
debug
mode,
it'll
compare
it
to
the
cached
version.
A
C
O
O
O
O
Basic
shed
objects,
so
you
need
to
go
through
all
the
objects
basically
and
that
more
or
less
end
up
in
the
same
thing
as
doing
a
d
copy
right.
So
then
we
explore
the
copy
and
write
things.
So
you
just
modify
the
copy.
The
thing
when
there
is
an
actual
modification
and
well
it's
not
a
necessity.
The
alternative
is
just
to
basically
create
a
new
table
by
overwriting
the
set
attribute
methods
and
then
maybe
discovered.
A
C
O
I
think
historically,
it's
been
the
aussie
map.
I
think
that
there's
been
a
lot
of
issues
there
right
and
also
when
the
network
stats
were
added.
There
were
some
issues
with
with
that.
One
too
there's
a
long
list
of
trackers.
I
just
collect
a
few
ones
yeah,
but
I
guess
those
probably
will
be
the
hardest
ones
using
map
and
pj
map
yeah
yeah
apart.
O
Yeah
time
taking
for
pi
formatter
to
generate
the
python
object,
there
might
be
to
be
some
contention
in
the
finisher
bread
when
that
happened.
So
something
like
that
that
was
that
the
last
tracker
that
you
faced
there
it's
basically
includes
a
description
of
the
queue
of
your
length
of
the
finisher
under
some
large
cluster.
O
O
C
A
A
C
Yep,
it's
it's
really
easy
to
add
targeted,
get
calls
on
in
the
c
plus
manager
code
instead
of
dumping.
The
whole
map
just
dump
just
the
thing
you
need
both
if
you
can
narrow
down
like
either
like
specific
things
that
you're
always
pulling
out
of
the
whole
steam
app
or
subsets,
and
that's
a
pretty
quick
to
implement
in
sequels
plus.
E
Yeah,
I
think
we
did
something
similar
earlier
for
at
least
the
networking
time
stuff
and
the
balancer
and
the
progress
module.
There
was
a
whole
cleanup.
We
did
for
the
the
stuff
that
we
were
dumping
only
dump
the
stuff
that
we
need,
but
maybe
there
are
other
places
as
well
ernesto.
What
are
the
recent
bottlenecks
that
you're
seeing
any
particular
modules
that
are
being
a
problem
right
now
earlier?
I
know
the
progress
and
the
network
stuff
was
the
problem.
O
F
F
A
A
F
And
I
think
the
problem
is
how
what
the
surface
of
the
the
public
interface
we
need
to
tackle.
If
it
is
very
large
we
we
should.
We
probably
should
leverage
the
existing
pi
for
matter,
because
that's
what
we
already
have
using
the
formatted
infrastructure.
But
if
the
surface
is
very
small,
we
could
just
leverage
the
the
python
object
or
sorry
simple
object
and
exposed
minimal
interface
exposed
to
the
python
python
code
and
wrap
it
around
with
a
python
python.
O
C
C
F
C
F
How
about
exposing
a
json
dump
for
the
object
which
is
interested
by
by
python
code,
the
pinpoint
if
we,
if
we,
for
example,
if
if
the
python
code
is
interesting,
a
certain
feature
or
sorry,
a
certain
property
of
a
crash
map,
we
could.
We
could
just
expose
an
interface
like
dump
and
it
will
that's
very
if
method
will
call
into
oh,
we
will
use
python
type
formatter
to
dump
that
very
structure
on
demand.
F
C
Kind
of
what
I
was
suggesting
with,
I
think,
at
least,
if
I'm
understanding
it
properly,
that's
what
I
was
suggesting
with
extending
get
to
include
whatever
more
narrow
field
instead
of
dumping.
The
entire
osd
map,
if
you
just
need
to
know
like
the
up
front
for
rio,
steve
then
have
a
command
that
just
dumps
that
those
those.
F
C
A
N
F
We,
I
think
we
we
don't
really
in
most
cases,
we
don't
really
interested
in
the
whole
python
object,
but
we
choose
to
dump
it
all,
no
matter
what
what
part
we
are
interested.
So
I
think
we
need
to
to
to
look
into
the
python
code
which
exactly
part
we
are
interested
and
we
need
to
look
at
and
to
probably
send
it
over
to
permissions.
C
O
Yeah,
maybe
it's
worthy
yeah
to
explore
this
idea
of
instead
of
using
the
pi
formatter
using
the
regular
json
formatter
and
importing
that
directly
in
the
with
the
json
decoder
in
in
python
site.
We
are
passing
up
white,
it's
counterintuitive,
but
it
seems
like
it's
more
performant
with
passing
the
python
update
itself.
So.
C
Yeah
yeah,
the
the
pi
formatter
is
a
little
bit
fraught
and
the
sequence
looks
good
because
you
have
to
be
really
careful
about
the
kill.
But
if
we
could
not
do
that,
then
that
might
also
simplify
the
measure
code.
But
I
would
be
interested
in
seeing
a
confirmation,
but
that's
really
the
case-
that
it's
faster.
O
Yeah
well
in
that
period
from
perry
you
see
that
they
took
some
measures,
so
at
least
the
coping
time
for
solo
copy,
shallow
copy,
sorry
to
call
json
and
copy,
so
the
optimal
one
is
pico,
which
is
a
binary
format.
I'm
not
sure
how
feasible
is
that
from
the
c
plus
plus
side
articles.
I
think
v
code
is
yeah,
it's
internal
to
python,
so
it's
not
standard,
but
maybe
there
is
a
code
that
we
can
use
to
realize.