►
From YouTube: Ceph RGW Refactoring Meeting 2023-03-15
Description
Join us every Wednesday for the Ceph RGW Refactoring meeting: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contribute
What is Ceph: https://ceph.io/en/discover/
A
A
A
A
C
A
A
C
Yeah
I
wonder:
I,
don't
know
why
whenever
I
hear
that
idea
about
scanning
scanning
for
Orphans
after
everything,
every
job,
I
sort
of
wonder
if
we
might
have
maybe
it'll,
be
more
attractable
to
have
a
special,
a
particular
particular
workflow
that
that
get
that
that
that
checks
for
reference
or
thank
you.
D
A
Yeah,
that's
a
good
question.
We
we
have
a
specific
job
right
now,
I
think
it's
under
rgw
Tools,
that
does
the
SK
uses
the
orphan
and
gapless
tools,
and
that
does
take
quite
a
while.
A
Possibly
yeah,
but
it
wouldn't
give
us
as
much
coverage
I,
think
I
think
it
would
be
valuable
to
to
catch
issues
from
any
of
our
testing
and
I
mean
the
idea
at
least
with
S3
tests.
Is
that
it's
cleaning
up
all
of
the
objects
and
buckets
that
it's
testing,
and
so
as
long
as
we
run
garbage
collection
before
scanning
for
Orphans,
then
I
would
assume
that
there
would
not
be
that
much
to
scan.
D
Yeah
I
mean:
could
we
do
it
stochastically?
So,
like
10
percent
of
the
RSW
verify
things
chosen
at
random,
you
know
just
choose
a
random
number
in
the
script.
You
know
run
it
so
we'll
we'll
catch
things
ultimately,
but
won't
add.
D
D
Yeah
I
believe
the
rgw
orphan
lists
test
that
that's
run
currently
exercises
things
quite
a
bit.
I
mean
it
does
S3,
it
does
Swift,
but
I.
Don't
I'd
have
to
look
and
remind
myself
what
the
scale
of
of
objects
of
how
many
objects
there
are
in
it
is,
but
it
does
try
and
exercise
every
every
type
of
of
object
possible.
A
D
Oh
I
see
what
you're
saying
maybe.
A
D
D
We
might
have
to
even
go
beyond
the
bi
list.
I
might
have
to
read
in
the
manifest
from
the
head
object
as
well.
Well,
no,
never
mind
I'm
getting
into
orphanless
territory,
never
mind.
A
A
Right
and
like
the
the
bucket
index
for
multi-part
uploads
would
have
index
entries
for
each
part
until
the
multi-part
complete
happens,
and
then
it
should
clean
everything
up
right.
So
maybe
just
scanning
for
the
multi-part
entries
could
be
enough
for
the
script
uh-huh.
D
A
D
Got
it
and
that's
definitely
a
case
where
excess
on
in
completed
multi-part
upload
components
appear
in
the
in
the
bucket
Index
right.
A
C
D
D
But
after
the
re-upload
finally
succeeds,
we
should
not
find
any
entries
in
the
bucket
index
and
we
should
not
find
any
objects
in
the
data
pool.
D
The
re-upload
achieves
that
the
re-upload
process
should
achieve
that
right.
B
C
D
Okay,
so
that's
okay,
I
I've
dabbled
in
this
python
stuff
before
in
these
python
tests
before
so
I
should
be
good
yeah.
If
you
could
just
send
me
your
script,
Matt
I'd
appreciate
it.
C
We'll
do
but
one
thing
that
what
that
I
think
you
should
mention
was
that
you
should
also
test
up
word.
Multiverse
same
thing
happened
there:
okay,.
A
B
Hi,
yes,
so
the
topic
is
about
the
inditex
request
and
this
one
is
a
long-standing
request
where
the
archive
Zone,
where
they
are
saying
that
the
archive
Zone
generates
a
new
version,
even
if
only
the
metadata
of
the
object
has
changed
and
not
really
the
just,
not
really
the
contents
of
the
object
itself.
B
So
a
very
naive
approach
would
be
you
know
to
compare
the
new
versions.
The
new
object
versions
check
some
with
the
existing
versions
that
are
already
existing
and
and
decide
to
discard
or
keep
it.
But
but
we
know
that
it's
going
to
be
a
very
costly
operation.
B
Is
this
even
something
that
we
should
consider
solving
at
all,
especially
since
we
already
have
Matt's
archive
Zone
I?
Think
it
was
life
cycle
policy
which
sort
of
helps
us
expire,
the
objects,
and
we
can
keep
the
number
of
objects
in
control
on
archive
Zone.
C
What
that's
interesting
well
I've
been
thinking
about
this
for
a
while
for
of
late,
I,
I
sort
of
think
as
a
quick
fix.
It's
not
a
great
idea
yeah.
C
It
occurs
to
me
that
and
if
I
brought
this
up
to
to
a
couple
people-
and
it's
been
thought
of
before
that
at
some
stage
we
consider
the
content,
addressable
representation
for
objects-
the
rados
either
either
perhaps
is
an
option
rather
than
a
replacement,
for
we
certainly
do.
But
there's
a
couple
of
benefits
that
might
have
one
of
them
is:
it
might
increase
the
information
density
and
the
object
names
for
rados,
which
would
be
convenient.
C
But
if
you-
but
if
you
had
some
some
some
block
based
content,
controversible
blocks
mean
for
our
largest
objects
in
the
entire
say
bucket
in
an
entire
bucket
or
a
group,
or
some
other
way
of
notation
than
you
would
our
existing
techniques
for
for
sharing
objects
or
citations.
It
seems
like
you'd
get
you
could
you
could
get
a
fairly
flexible,
d-doop
system?
C
If
you
use
large
block
sizes
like
current
stripe
size,
it
would
trivially
handle
the
the
the
three
output
cases
and
then
it
wouldn't
be
specific
or
buckets
and
there's
object
of
the
same
name.
Etc.
A
Right
so
for
this
archive
Zone
case,
it
would
keep
creating
new
versions
for
each
change,
but
each
of
the
versions
would
share
the
the
actual
object
data.
Since
it's
the
same
yeah
yeah.
My
intuition,
is
that
I
mean
it
seems
like
they
do
actually
want
to
create
new
versions
when
the
metadata
changes,
otherwise
they
they
just
won't,
have
a
record
of
the
current
metadata.
B
C
Sort
of
tangentially
our
Downstream
discussions
like
product
management
level,
discussions
and
above
that
there's
there's
a
substantial
interest
in
relying
more
rather
than
less
on
our
kinds
of
so
I.
Think
that
that
perspective
justifies
looking
at
long-term
data
reduction
rather
than
reading
at
SSN
has
some
kind
of
a
hack
or
a
point
fix.
B
Ed,
okay
yeah!
If,
if
we
want
to
further
discuss
on
this,
and
since
this
is
probably
Downstream
related,
I
could
add
this
as
a
topic
for
our
Monday
meeting
discussion.
B
A
Yeah
on
the
agenda,
I
added
a
link
to
a
Trello
card,
that's
been
on
the
Upstream
stuff
backlog
for
a
long
time.
Yehudo
is
really
keen
on
this,
but
nobody
is
kind
of
proposed
the
design
for
it
really.
C
A
C
C
But
if
we
want
to
resurrect
it,
I'm
excited
to
find
out
that
it's
something
before,
but
in
terms
of
the
you
know
something
that
or
Friedman
brought
to
me
was
there
was.
It
was
actually
researched
by
by
Gabby
bananek
that
that
he
he
proved
that
he
seemed
to
prove
that
that
if
we
had
a
contract,
addressable
radius
object
names.
That
was
such
such
particular
sense
that
we
were
relying
not
relying
to
the
same
to
the
current
extent
on
these
these
long.
C
A
B
Yeah
I
just
thought:
that's
right,
yeah
I
would
I
would
actually
make
it
like
to
make
it
more
generic
than
just
that.
Guy,
someone
specific,
because
I
think
we
have
we're
already
doing
a
lot
of
things.
That's
changing
the
semantics
romanticide
for
archives.
C
Yeah,
that's
exciting,
so
let's
talk
about
it
further
Inland.
We
raise
a
couple
interesting.
Some
interesting
points,
there's
other
dimensions
to
this
right:
Block,
Base
versus
fingerprinting,
other
techniques,
but
I
think
we
should
explore
it.
If
objects
were
getting
very
large,
we
could
perhaps
do
some
kind
of
scaling
factor
for
the
trunk
sizes
or
things
like
that,
or
maybe
that
would
maybe
that
would
be
useful
or
maybe
but
reduce,
but
it
would
reduce
due
to
accuracy.
C
But
from
the
point
when
I
opened
up
when
I
was
thinking
along
the
line,
so
it
was
was
something
which
were
large
chunks,
like
maybe
maybe
striped
chunks
and,
if
I
think
I
think
my
presumption
there
is.
That
is
that
in
such
a
such
a
scheme,
we're
not
we're
not
right.
What
we're
doing
is
eliminating
identical
we're
not
seeking
aggressive.
You
know
Common
blocks,
but
different
strategies
probably
coexist.
A
Yeah
I
think
if,
if
we're
interested
in
pursuing
this,
maybe
starting
a
discussion
on
the
Upstream
mailing
list
about
kind
of
design,
ideas
to
start
with.