►
From YouTube: Ceph RGW Refactoring Meeting 2023-05-03
Description
Join us every Wednesday for the Ceph RGW Refactoring meeting: https://ceph.io/en/community/meetups
Ceph website: https://ceph.io
Ceph blog: https://ceph.io/en/news/blog/
Contribute to Ceph: https://ceph.io/en/developers/contribute
What is Ceph: https://ceph.io/en/discover/
B
Hey
so
I,
this
is
something
that
the
red
hat
documentation
team
wanted,
and
there
was
a
downstream
bug
open
for
it.
B
The
approach
I
took
was
I,
have
a
list
of
all
of
the
config
vars
and
spreadsheet,
be
happy
to
send
it
out
and
I
just
gone,
I
think
20
at
a
time
and
if
I
thought
that
the
config
wire
was
cached
in
any
way
or
if
it
ended
up
getting
returned
as
a
part
of
a
function,
call
I
marked
it
as
Deeds
restart
and
if
not,
if
it
was
just
in
line
I
left
it
as
runtime
in
the
spreadsheet
and
I'm
about
halfway
done.
B
B
I'd
want
some
more
eyes
on
the
people
that
actually
wrote
some
of
these
config
variables,
but
the
the
blanket
recommendation
I
had
was
to
just
say
that
all
of
the
anytime
you
change
config
variable.
You
need
to
restart
rgw,
but
I.
Don't
think
that's
the
direction
we
want
to
move
in
as
a
as
a
product
and
I
was
wondering
what
other
people's
thoughts
were.
I
know.
Matt
had
mentioned
that,
at
least
from
the
IBM
side
that
there's
a
desire
to
have
less
restarts
and
to
be
able
to
configure
things
at
runtime
more.
D
C
In
controlling
the
semantics
of
of
of
config
variables,
changing
and
consistently
during
rolling
upgrades
things
like
that,
we
need
to.
We
need
to
understand
why
the
semantics
we're
trying
for
are
and
how
to
chart
a
path
to
getting
to
ones
that
are
acceptable
at
scale.
A
Yeah,
so
if,
if
we
do
have
a
list
of
these
variables
somewhere,
maybe
it
would
be
a
good
idea
to
try
to
prioritize
some
that
require
restart
that
that
it
would
be
especially
useful
to
have
runtime
support
for
and
we
can
discuss
kind
of
what
the
constraints
are
or
the
consistency
issues
around
it.
B
Yeah
I
was
going
to
add,
like
I,
was
going
to
say,
go
on
it
from
a
sub
component
basis
like
all
the
GC
ones
or
the
LC
ones,
or
the
Keystone
ones.
Are
there
there's
usually
pretty
big
families
of
config
variables
in
there,
but
I
think
Casey
Casey's
approach
is
even
more
granular
and
better
for
that
I'll
share
the
list
out.
D
B
That's
that's
all
the
discussion.
I
had.
Does
anybody
from
Bloomberg
on
the
call
and
have
a
comment
or
know
anybody
else
there
that
we
could
reach
out
to
and
to
keep
this
discussion
Upstream?
That
is.
E
Yeah
I
mean
I
would
be
nice
to
have
that
with
the
the
document,
the
variables
which
would
be
impacting
in
terms
of
restarting,
as
we
said
right
like
we
don't
have
scale
so
it
it
is
kind
of
not
trying
to
typical
for
us
to
just
kind
of
restart
the
gateways.
D
B
Yeah
and
so
I
haven't
gone
through
every
single
config
variable.
Yet
once
I
saw
a
pattern
that
the
majority
of
the
config
variables,
at
least
in
my
in
my
opinion,
required
restarter
I
would
have
felt
better.
Restarting
the
Gateway
I
I
kind
of
left
it
at
that,
but
I'll
send
you
the
list
of
all
the
variables
and
what
I've
done
so
far.
You
know
I'll
send
it
out
to
the
team
too.
If
people
want
to
look
and
take
a
look
at
the
work,
I've
done
or
change
anything.
D
D
E
B
B
Okay,
I
think
we
can
move
on
and
or
if
anybody
wants
to
take
a
look
at
that
right
now
they
can
go
ahead.
I
just
pasted
the
link
or
in
the
in
the
meeting,
chat
and
I
want
to
emphasize.
This
is
not
the
source
of
Truth.
This
is
just
me
taking
a
stab
at
it.
All
of
the
variables
are
in
alphabetical
order.
B
F
D
B
Do
you
want
to
shoot
me?
Your
email
address
sure.
B
I
have
this
under
my
personal
Gmail
now
so
I
should
be
able
to
yep
well
I.
Think
he's
maybe
due
to
Bloomberg
constraints
on
that
I
can
access
it.
With
my
personal.
F
B
D
B
Maybe
I'll,
maybe
I'll,
add
a
new
column
or
tab
to
this
column.
Sorry
column
to
this
sheet
about
something
that
would
for
a
high
priority
investigation
into
other
restarts
required
or
not.
B
We
merge
my
PR
so
that
the
flag
can
be
added
to
the
docs
upstream
or
at
least
for
the
rgw
config
variable
Docs,
and
then,
as
we
figure
out
the
list
of
config
variables,
that
we
would
like
to
be
able
to
enable
or
move
from
requiring
restart
to
able
to
change
at
runtime.
We
tag
those
as
restart
and
then
as
they
get
converted
over.
We
change
it
to
runtime.
B
D
D
B
Great!
That's
it
for
me
here.
A
A
My
my
suggestion
there
was
to
use
tracker
issues,
foreign.
C
F
C
C
A
Okay,
well,
I
do
use
a
multi-site
tag
on
the
tracker.
A
We
could
create
a
custom
query
that
searches
just
for
that.
Another
option
would
be
to
create
a
separate
sub
project
under
rgw
Just,
For,
multi-site
Stuff,
but
I
think
my
preference
would
be
to
keep
it
all
the
same
so
that
we
see
them
during
bug.
Scrubs
can.
C
A
A
A
So
we
currently
have
the
rgw
bug
scrub
call.
Every
Thursday
would
would
there
be
a
benefit
to
having
a
separate
scrub
for
the
backlog
or
maybe
a
a
monthly
meeting
or
something.
C
C
A
Okay
so
I
think
the
other
action
item
would
be
to
go
through
these
backlog
documents
and
create
break
them
up
into
tracker
issues,
either
as
bugs
or
tasks
and
subtasks.
F
Yeah
yeah
we'll
take
them
up
as
freckles
I
think
we
I
just
have
listed
the
the
backlog,
but
yeah
I
haven't
really
broken
them
down
into
tasks
so
yeah.
We
can
do
that.
E
Okay,
so
just
to
understand
this,
math
correctly
is
like
oh.
So
what
are
you
talking
about?
You
and
Shilpa
had
a
document
right
now,
in
terms
of
which
could
potentially
be
a
multi-side
project,
but
currently
in
the
docker
in
the
document,
and
then
what
Casey
suggesting
we
could
add
them
to
the
tracker
and
tag
them
as
multi-site
backlog
is
that
the.
E
And
then
yep
as
and
then
probably
invite
or
it's
up
for
anyone
to
grab
that
project
and
then
can
attend
this
call
and
then
just
discuss
another
progress.
That's
that's
the
idea.
Well,.
C
C
Whether
we
were
you
know
whether
we're
going
to
keep
working
on
on
the
on
the
on
the
sync
fairness
data,
sync,
fairness,
part
of
the
of
the
of
that
of
that
multi-part
fix,
and
the
answer
is
yes,
I
mean
deeply
involved
in
it.
We're
about
to
merge
the
first
part.
C
What
are
the
ideal
things
to
work
on
and
maybe
what's
the
context,
and
so
the
people
are
here?
Okay
can
all
help
feedback.
You
know
what
are
good
ideas,
dialogue
with
you
about
what
what's
on
our
list,
so
we
don't
so
maybe
we
can
put.
Maybe
we
can
share
that
list
in
the
docs
and
and
the
in
the
pad
for
for
for
discussion
here.
E
E
I
just
I
was
telling
Matt
yesterday
in
the
email
like
since
the
beginning
of
this
quarter
in
April,
Bloomberg
has
got
a
new
team
which
has
set
up
five
double
six
developers
which
are
going
to
be
working
only
for
like
for
this
website,
and
so
we
basically
at
this
point
in
time,
are
looking
for
projects
where
we
could
help
him
kind
of
chime
in
it
could
be
a
collaboration
project.
E
It
could
be
an
independent
project
as
well,
so
so
the
first
I
mean
perhaps
the
thing
fairness
was
more
of
importance
because
we
have
multi-site
in
our
production
earlier.
So
that
was
one
thing
we
were
looking
to
kind
of
get
traction
on.
So
that's
why
we
kind
of
shoot
email
to
Matt,
saying
that
is
there
any
way
we
could
contribute
there.
Basically,
it
was
not
like
working
independently,
but
more
about
contribution
or
kind
of
get
this
over
the
line
with
the
folks.
We
have.
C
Yeah,
well,
you
could
help
there
too
I
mean
greatly
as
we
as
we
put
together
the
branch.
You
know
the
data
log.
You
know,
data
basic
fairness,
have
you
to
help
us
test
it
and
find
things,
but
but
we
should
also
give
you.
You
know
the
opportunity
to
take
on
new
fixes,
pick
on
or
take
on
new
tasks.
Yep
our.
E
Project
yep
talking
about
that
I
just
forgot
to
mention
that
right,
like
we
did
test
the
metadata,
seeing
fairness,
PR
and
it
seems
to
be
working
great.
So
we
given
like
we're
kind
of
leveraging
the
the
the
the
the
machines
or
the
test
framework
or
infrastructure
we
have
at
Bloomberg.
It
might
like
the
way
we
tested
the
replication
work.
We
can
always
kind
of
chime
in
also
on
testing
any
any
kind
of
bugs
or
any
any
PRS.
If
that
that
also
a
form
of
help
that
we
could
provide
as
well.
F
Yeah
sure
friends,
so
I
I
have
something
going
on
for
data.
Sync
fairness
and
I
should
have
a
PR
soon,
so
once
I
have
that
I'll
add
you
and
the
folks
you
want
to
add
them
on
the
pr
and
you
could
just
take
it
up
for
testing
perfect.
E
Yeah
we
could
start
the
piano
like
yeah,
run,
run
the
test
and
just
update
the
update,
the
the
the
VR
itself
with
the
test
results.
If
not
the
case,
we
could
already
be
looking
forward
to
even
showing
the
calls
weekly
and
then
kind
of
provide
inputs
there
as
well
yep.
F
Yeah
so
I
I
have
shared
the
document
on
The
Ether
pad.
No
me
and
Casey
will
work
on
breaking
them
down
into
trackers
and
and
tasks
within,
and
then
you
are
free
to
pick
up
whatever
you
want
awesome.
E
F
Yeah
I
think
yeah
I
think
putting
them
in
tracker
will
will
give
some
more
clarity
for
you.
Yeah
are.
C
F
One
of
them
was
the
thing
that
you
mentioned:
Matt
about
cleaning
up
of
deleted
bucket
index
objects,
which
already
has
a
Tracker
I,
can
put
that
here.
Give
me
one
second.
F
Yeah
I
have
put
that
in
the
in-call
messages
and
then
we
definitely
need
help
with
the
having
a
working
decide,
functional
tests
and
currently
I
think
it's
it's
kind
of
broken
and
we
haven't
really
been
able
to
dedicate
time
to
fixing
that.
F
So
we
would
need
some
help
with
improving
our
functional
tests,
Upstream
and
then
there
there
are
other
things
like.
Oh
you
know,
we,
we
don't
have
a
good
retry
mechanism
for
rather's
Gateway
admin
commands
when
you
know
one
of
the
rgw's
go
down,
and
then
there
is.
There
is
one
that
you've
already
picked
up,
which
is
the
race
with
object
deletion,
and
there
is
also
one
of
the
other
issues
is
versioning.
That's
with
the
null
ID
isn't
working
with
multi-site
I.
F
Think
that's
I'm,
not
sure.
If
someone
in
our
team
is
already
working
on
it,
I
I
don't
have
info
on
that,
but
I
will
still
create
a
Tracker
or
if
there
is
already
a
track
arrival
put
that
there
I
yeah
I
think
these
are
some
of
the
things
that
are
that
I
can
get
from
the
top
of
my
head.
But
is
there
anything
else
you
want
to
add
Casey
anything
from
sync
info
provider
or
from
anything
else,
I,
don't
know.
A
I
think
I
would
just
like
to
stress
prioritizing
fixing
the
functional
tests,
because
I
mean
we
could
take
on
all
these
other
projects,
but
without
good
test
coverage.
It's
hard
to
that
said
to
verify
that
they
work
so
yeah.
C
A
Well,
I
think
it's
just
the
distributed.
Nature
of
multi-site
makes
it
hard
to
test
kind
of
in
general,
the
a
lot
of
the
tests
are
kind
of
timing
based
or
at
least
they'll
time
out
after
a
while,
and
we
see
a
lot
of
issues
in
the
toothalogy
environment
just
because
some
of
the
parts
are
slow
but
investigating
exactly
where
those
happen
is
difficult
and
hasn't
really
happened.
C
I
mean
I,
think
I
mean
I,
think
it's
worth
investors
in
everything,
but
I
I
think
it's
I
think
it's
worth
taking
me.
You
know,
especially
with
folks
here
like
from
from
like
Cornell
and
and
so
on
and
others
and
have
a
that
have
that
can
bring.
Maybe
a
fresh
perspective
on
how
you
test.
Are
you
there?
How
you
validate
replication
at
scale
on
a
production
workload
might
be
worth
saying?
Okay?
Well,
how
would
we
solve
that
problem?
C
What's
what's
and
what's
in
test
multi,
and
then
you
know,
look
at
the
cross
product
of
that
I
mean
I.
I
mean
tell
me
the
problems
of
tooth
algae
is
sort
of
sort
of
orthogonal
and
of
our
time
on
the
first
instance.
What's
what
would
we
want
to
do
and
what
are
the
constraints
required
to
do
it,
and
then
we
can
look
at
the
problem
of
well.
What
is
what
is
technology
serving
us
or
our
strenology
broker?
In
some
respect
we
can
fix-
or
rather
you
know,
sort
of
that's.
C
A
A
Well,
regarding
the
python
tests,
as
they
are,
I
have
had
better
luck,
running
them
locally
against
vstart
I,
don't
see
nearly
as
many
of
the
failures
that
we
get
regularly
in
toothalogy,
but
running
locally
doesn't
really
scale
to
PR
testing
of
features
Upstream,
so
I
I
do
think
that
the
200g
piece
is
important
to
get
in
and
make
consistent.
C
Maybe
it's
as
much
a
problem
with
some
aspect
of
how
technology
is
running
then,
as
as
with
our
tests
or
or
maybe
there
are
pieces
that
you
know
our
Concepts
get
compatible
barriers
or
I,
don't
know
what
they
got
into
the
box
but
making
things
up,
but
but
that
we
try
to
figure
out
as
part
of
this
process.
A
Yeah
and
I
mean
I
think
a
lot
of
the
things
are
timing
related,
especially
the
one
of
the
major
causes
of
failures,
is
metadata,
sync,
just
getting
stuck
in
all
of
the
tests.
Failing
on
that,
I'm
not
sure
how
timing
and
tooth
allergy
would
cause
that
to
happen.
Unless
there's
a
discrete
bug.
C
F
Yeah
I
think
Mark
has
been
working
on
that
and
we
did
notice
invalidation
attempt
failures
with
regards
to
the
notify,
rwsi
notify
robust
notify
and
that's
where
we
are
timing
out.
A
A
Another
thing
during
the
multi-site
research
project
bucket
sync
changed
a
lot
and
somewhere
along
the
line.
The
multi-site
tests,
which
used
to
take
about
five
minutes
to
complete,
are
taking
more
like
30
to
45
minutes.
A
So
there
there
may
be
just
multi-site
sync
performance
issues
in
there
that
we
haven't
identified
that
could
be
part
of
the
cause
for
timeouts
in
toothachology
tests.
A
But
Matt
I
agree
that
we
should
look
at
other
things
outside
of
this
functional
test
framework.
You
mentioned
more
scaled
workload.
Stuff
I
also
think
that
in
the
opposite
direction,
some
smaller
unit
testing
would
help
a
lot
so
that
we're
just
testing
small
pieces
in
isolation.
Instead
of
all
the
complexity
of
the
whole
distributed
system.
C
A
C
C
A
Okay,
well,
we've
got
some
action
items
mostly
on
the
tracker
from
from
Bloomberg's
side
I.
Would
you
know
you
guys
should
always
feel
free
to
create
tracker
issues
for
issues
that
you
find
or
projects
that
you
want
to
work
on
and.
E
Okay,
do
we
have
any
tracker
right
now
for
the
affair,
data
sync
issue
or
the
fairness
thing
fairness
issue
like
I,
did
see
a
Tracker,
but
then
there's
not
a
lot
of
update
on
that.
Is
there
any
other
tracker?
Apart
from
that
that
you
guys
are
tracking
or
working
with.
F
Yeah
I
think
it's
good
to
be
merged.
F
Will
do
so
as
for
data
sync
I'm,
just
yeah
I
think
I'm
still
working
on
it
once
I
have
something
passing
I'm
just
going
to
let
you
know
I'm.
E
Sure,
thank
you.
We
can
always
yeah
leverage
testing
and
we
can
provide
you
the
test
results
or
something
if
the
skill
testing
for
sure
yeah.
Thank
you
looking
forward
and
looking
forward
for
the
all
the
trackers
and
the
documents
that
you
have,
probably
where
we
can
help
it
from
the
development
perspective.
Yeah,
definitely
forward
yep.
So
you
said
that
you've
not
yet,
given
the
permission
for
the
document
you
you
plan
to
add
it
to
the
trackman,
then.