►
Description
RGW S3: Feature Progress, Limitations & Testing - Robin H Johnson, Spaces DigitalOcean & Ali Maredia, Red Hat
What’s new in the world of RGW S3 features & their parity in relation to other S3 providers?
What are the performance costs of S3 features (garbage collection & Bucket Lifecycle)?
This session will cover: development and roadmap in testing S3 compatibility; specification coverage; feature performance (and costs to reach that performance); operational behavior; and war stories testing S3.
As future work, what does large-scale compatibility in the global S3 ecosystem look like? (and how to test it: s3-tests and beyond)
How does deliberate divergence from the S3 specification provide new functionality? (Such as RGW PubSub instead of S3 Bucket Notifications)
About Robin Johnson
DigitalOcean
Senior Engineer, Spaces
Robin presently improves Ceph to fit operational needs for the DigitalOcean public cloud environment, as part of the Spaces product. After many years of focus on Gentoo Linux, Robin explored Ceph after a non-profit deployment revealed deficiencies & problems. While developing solutions, Robin improved other aspects of RGW. This included implementing the S3 Website API & constantly chasing bugs in S3 client implementations.
About Ali Maredia
Red Hat
Software Engineer
Ali Maredia works on Ceph for Red Hat with a focus on object storage. Ali maintains Ceph's S3-tests repository, which is responsible for testing Ceph’s S3 interface. Ali has done work all over Ceph, including in RGW, ceph-ansible, and the testing infrastructure. This spring Ali is mentoring a group of graduate students from Boston University on a project to implement object caching in the RGW. Ali also coordinates Ceph's Google Summer of Code and Outreachy program. Ali got involved with Ceph while working for CohortFS.
A
B
A
So
Amazon
publishes
their
spec,
they
don't
version
it,
they
change
it
underneath
things,
sometimes
for
the
better,
sometimes
for
the
worse.
If
you
ever
run
into
a
much
more
detailed
change
list
of
it
than
the
abbreviated
one
they
publish,
it
would
be
fantastic
if
somebody
could
send
that
out.
I've
been
asking
them
for
a
detail,
change
this
for
a
very
long
time.
A
So
we
break
down
aw
what
s3
is
into
basically
three
pillars.
It
has,
first
of
all
how
things
are
stored,
primarily
the
objects
and
then
how
you
access
them,
and,
lastly,
things
you
can
do
to
groups
of
them
for
examples
here
for
the
objects
yeah,
for
example,
the
objects
themselves
have
a
cl's
as
a
property.
They
have
expirations
on
the
objects
themselves,
they
may
have
encryption
for
access
to
earth,
they
have
a
AWA.
A
A
A
So,
for
example,
here
object,
expiration
and
encryption
were
of
newly
available
in
luminous,
as
was
the
very
first
release
of
life
cycle
in
mimic,
which
was
now
added
after
luminous,
the
only
real
things
we
got
here
with
tagging
and
object
policies
and
then
AWS
as
they
say,
as
they
release
more
changes
as
well,
and
so
I
dropped
the
other
things
we
can.
You
obviously
see
here
they
add
a
new
object,
locking
functionality
for
legal
retention
and
they
changed
how
some
things
work.
A
They
changed
a
tiny
fragment
of
a
CL
such
that
you
could
no
longer
specify
a
CLS
in
one
way.
They
had
more
region,
specific
behavior
and
more
recent
things
you.
It
was
in
the
news
that
they
were
turning
off,
wanted
to
turn
off
path,
style,
access
to
buckets,
and
then
they
rolled
some
of
that
back
because
there
was
too
much
of
an
outcry
about
it.
They
do
add
more
features
as
well.
They
added
select
in
batch
queries
to
do
more
things
across
many
objects
and
Notley.
A
A
C
B
Tests
is
a
test
suite
that
the
rgw
developers
maintain
that
tests
they
can
be.
They
can
test
any
s3
interface
to
see
if
it
complies
with
what
the
spec
is.
Some
history
about
s3
tests
is
s3
test
was
written
in
2011
and
I.
Imagine
back
when
they
wrote
it.
They
went
down
whatever
Amazon
publicized
as
their
spec
and
wrote
test
cases
against
that.
But,
as
s3
test
has
evolved,
the
tests
that
have
been
added
have
matched
the
RG
w's
feature
or
gel
B's
feature
set
and
now
AWS
is
feature
set.
B
There
are
also
go
and
Java
test
Suites
that
cover
a
subset
of
what
the
Python
s3
test.
Suites
cover
those
have
been
written
by
SEF
outreach
interns,
Joanna
from
Nairobi
wrote
the
started
on
the
work
and
Antoinetta
from
Switzerland
last
year.
Got
it
working
in
2002
lot
of
the
cleanup
for
the
work
as
well.
B
Yes,
that's
right,
Cameron
refused,
yes,
so
as
three
tests
does
not
cover
any
of
the
rgw
back-end
specifics
like
rattles
class
operations
and
adding
and
the
rather
CW
admin
command,
nor
is
it
used
for
performance
testing.
It's
purely
unit
tests,
testing
the
s3
interface
and
not
for
longer
running
garbage
collection
or
life
cycle
testing.
B
B
S3
tests
also
is
not
run
against
AWS
regularly,
as
I
mentioned
before,
tests
aren't
written
and
then
run
on
AWS,
first
and
the
rgw
second
they're
written
and
they're
run
on
the
rgw
first.
They
also,
they
also
return,
not
implemented
for
tests
that
or
four
4s
three
operations
that
are
not
implemented
on
the
routers
gateway
at
the
time.
B
C
B
Aws
spec
over
the
last
few
years
or
actually
no.
This
is
this
is
what
it
says
today.
If
you
go
on
the
website,
you'll
see
that
the
specification
says
the
name
of
the
source
bucket
should
be
separated
by
a
slash.
The
example
has
a
slash
in
the
very
beginning
and
all
that
it
says,
underneath
that
is
that
the
string
must
be
URL
encoded,
but
it
doesn't
specify
whether
or
not
to
use
pat
style
or
query
style
and.
A
A
B
A
A
So
there
are,
there
are
some
other
more
complex
examples
of
missus
that
will
end
up
biting
people
very
recently.
It
turns
out
that
if
you
run
create
bucket
with
the
location
constraint
as
the
payload,
the
headers,
the
header
for
the
signature
is
not
always
computed
correctly
and
then
the
rtw
side
says
nope.
You
have
you've,
got
your
signature
wrong
denied
and
it's
gonna
fail.
A
Browser
post
was
originally
intended
as
a
way
for
the
browser's
to
do
a
CGI
forum
style
post
to
upload
an
object,
but
they
said
you
needs
to
be
a
way
to
control
the
policy
so
that
the
user
doesn't
try
to
upload
a
giant
object,
but
there's
a
bug
and
it
doesn't
validate
the
length
quite
the
way
that
it
should
and
over
the
span
of
a
couple
of
bites,
it
will
flip
from
supposed
to
be
allowing
you
object
to
saying
no,
it's
too
big.
No,
it's
too
small
because
we
just
crossed
over
an
integer
boundary.
A
So
now
before
we
carry
on
to
other
parts
of
how
rgw
does
some
of
the
things
and
the
problems
with
that
I
want
to
briefly
recap:
how
lgw
lays
out
objects
into
Redis
objects.
If
the
object
is
very
small,
it
will
simply
have
a
single,
a
single
object.
Single
Redis
object
with
the
metadata
attached
to
that
that
object
is
called
the
head.
If
you
did
a
classical
large
upload,
this
look
definitely
larger
than
4
Meg's.
You
will
have
the
head
and
you
will
have
simple
striped
parts
there
after
one
or
more.
A
A
So
now
what
happens?
If
you
want
to
go
and
delete
these
various
layouts,
if
most
of
it
is
most
of
it
at
the
moment,
unfortunately,
as
synchronous,
the
parts
that
matter
to
the
speed
of
the
customer
deleting
the
head
is
entirely
synchronous.
It
is
not
garbage
collected
in
any
way.
The
only
pot
gets
garbage
collected
are
the
tails
now
so
the
deletion
speed
of
your
head
matters,
and
it
turns
out
that
if
your
head
is
entirely
empty
like
this
was
originally
a
multi-part
upload,
it
deletes
much
faster.
A
A
So
how
do
we
get
this
to
go
faster,
I
think,
ultimately,
all
of
the
objects
need
to
having
the
manifest
layout,
even
if
they
are
a
small
upload
and
at
the
point
that
they
are,
the
manifests
can
ultimately
move
to
a
faster
pool.
These
will
all
be
definitely
small
and
you
can
have
them
on
a
pool
of
SSDs,
while
keeping
your
data
is
still
on
this
low
spinning,
rust.
A
There
will
be
other
advantages
to
doing
this
as
well.
Life
cycle
operations
that
depend
heavily
on
metadata
will
get
much
faster,
CDN
use
cases
that
say,
hey
has
the
object
changed
underneath
me
with
a
kick
conditional,
get
will
also
be
fast,
so
what
happens
worth
on
garbage
collection
now,
if
some
of
the
customer
workloads
end
up
deleting
a
large
quantity
of
data.
This
is
an
example
from
one
of
our
clusters.
Every
morning,
customers
upload
lots
of
data
and
then
overnight
deletion.
A
C
A
A
You
could
turn
it
off,
but
the
only
way
of
turning
an
offer
at
the
moment
regrettably
requires
restarting
you
rgw
daemon
or
using
some
of
the
configuration
knobs
that
were
meant
to
control
the
speed
of
it,
but
they
don't
really
do
a
good
job,
and
so
this
needs
to
end
up
being
fixed
fixed
for
making
garbage
collection
go
better,
but
garbage
collection
is
also
in
his
prison
design
really
hard
on
the
OS
DS.
It
hits.
A
Some
focus
short
flip
card
showed
me
that
garbage
collection
hurt
their
workload
so
much
they
put
it
on
its
own
OS
DS,
just
because
it
ends
up
slamming
the
end.
The
SSDs
that
you
have
the
Purple
Line
here
is
just
how
much
garbage
collection
reads
its
own
OMAP
data
back
incredible
scans
across
all
of
it,
and
then
it
falls
down
if
your
SSDs
get
slow.
A
So,
lastly,
lost
part
of
garbage
collection
problems,
there's
no
insight.
How
much
stuff
is
left
in
garbage
collection.
You
can't
really
tell,
and
the
only
way
to
figure
it
out
at
the
moment
is
to
do
a
list,
a
same
listing
scan
across
the
entire
garbage
collection.
No
map
pool,
there's
no
and
there's
no
set.
How
fast
is
it
going?
A
I
can't
tell
you:
rgw,
doesn't
export
this
data
so
now
onwards,
tube
life
cycle
in
Nautilus
and
October,
the
upcoming
octopus
stuff
big,
thank
you
to
you
who,
de
for
rewriting
some
of
how
life
cycle
is
done.
The
old
code
was
much
worse.
The
new
stuff
is
definitely
better.
Unfortunately,
a
lot
of
people
are
not
running
it.
Yet
so
life
cycle
in
general,
for
those
that
don't
know,
allows
you
to
cover
from
the
state
or
age
of
an
object,
possibly
in
a
path
prefix
or
with
object.
A
Tagging
do
something
to
it
at
some
point
in
the
future,
and
how
is
that?
How
is
this
stored?
There
is
the
pot
the
policy
document
is
attached
to
the
bucket
and
and
when
it's
regularly
processed
it
has
a
series
of
state
transitions
that,
ideally,
it
will
transfer
through
every
day.
If
there's
not
too
much,
life
cycle
work
to
do
when
it
goes
to
do
this
every
time
it
scans
the
entire
for
every
rule
in
life
cycle,
it
scans
the
entire
bucket
index
list,
but
this
is
for
every
rule.
A
Now
your
life
cycle
policy
could
have
I
think
limits,
256
rules
at
the
moment.
It's
a
configurable
knob
but
I
think
the
defaults
256.
So
it
might
scan
your
entire
bucket
256
times.
If
your
bucket
has
a
lot
of
objects,
this
can
be
painfully
slow.
We
just
yes.
Last
night,
we
discussed.
How
can
we
make
this
at
least
one
scan
instead
of
256
counts.
A
A
Ideally,
it
would
have
lifecycle,
be
able
to
process
by
hitting
only
the
index
and
not
having
to
even
touch
the
metadata
objects.
I
think
that
might
be
possible,
but
I'm,
not
hundred
percent
Sur,
large
buckets
or
arrows.
It's
still
very
expensive
scanning
them
is
expensive,
depending
on
the
size
of
your
bucket.
It
might
be
cheaper
to
instead
maintain
a
secondary
index
per
rule
ordered
by
time,
because
you'd
only
have
to
consume
things
from
the
head
of
it
when
you
were
doing
expire
ease
and
only
up
to
a
certain
point.
A
New
objects
would
just
be
appended
at
the
end
and
much
like
garbage
garbage
collection,
because
lifecycle
borrowed
its
code
from
garbage
collection.
To
start
with,
there
is
no
insight
to
how
much
lifecycle
work
is
outstanding
or
even
going
on.
You
can
hack
around
this
and
we
have
we
a
digital
ocean.
We
did
put
in
a
very
messy
patch
that
says:
I
want
to
do
a
lifecycle
on
a
single
bucket
at
a
time,
because
we
had
some
customer
buckets
that
had
a
lot
of
lifecycle
work
to
do
so.
A
B
Yeah
and
going
back
to
this
to
areas
that
are
GWT
testing
does
need
to
improve
on
is
number
one
mismatching
rgw
and
OSD
versions,
and
this
is
something
that
happens
quite
often
in
during
upgrades,
both
at
the
rgw,
an
OSD
version
in
this,
in
the
single,
in
a
single
upgrade,
for
example,
in
a
very
large
cluster
or
if
you're.
If
you
have
a
multi-site
configuration,
and
you
are
running
an
older
version
of
stuff
in
one
cluster
and
a
newer
version
of
stuff
in
another,
some
of
the
longer
running
effects
that
aren't
very
thoroughly
tested.
B
It
in
terms
of
future
testing
for
the
s3
for
s3
amazon
has
quite
a
few
SDKs,
and
we
would
like
to
make
sure
that
they
are
all
tested
along
with
that
there's
also
community
SDKs
that
are
not
officially
Amazon
supported.
That
should
also
be
tested
and
there's
also
old
versions
of
current
SDKs,
that
more
people
are
using
than
the
new
versions
at
this
moment
that
also
need
to
be
tested.
Basically,
this
comes
to
a
very
large
test
matrix,
but
making
sure
that
comprehensive
testing
is
around
this
test.
Matrix
is
important
and.
A
Will
specifically
point
out
that,
while
it
Amazon
themselves,
publish
I,
think
it's
the
13
plus
SDKs
that
can
all
touch
s3,
they
don't
all
function
the
same
way.
They
have
some
unique
bugs.
For
example,
it
turns
out
that
for
copy
object,
that
was
that
specification
failure
was
encoded
into
the
go
client.
The
go
client
doesn't
encode
the
header
itself.
You
have
to
pass
an
encoded
header
yourself,
boto
does
which
ones
right.
Well,
the
specification
is
not
clear.
B
And
in
terms
of
future
s3
testing
on
the
roadmap
for
the
current
repository
is
a
porting
the
tests
over
to
python
3,
adding
java
and
go
to
adding
the
Java
and
go
testing
frameworks
to
be
run
a
little
more
continuously
and
also
being
able
to
toggle
between
v2
and
v4
signatures.
But
for
the
next
four
for
the
fall,
there's
three
main
things
we'd
want
to
do.
One
of
them
is
move
the
testing
framework
over
to
PI
test
from
knows
test,
since
nose
is
not
being
maintained
anymore
and
PI.
B
A
B
By
the
octopus
releases,
we
would
like
to
have
we'd
like
to
be
able
to
support
not
just
the
Python
SDK
but
SDKs,
but
other
SDKs,
and
that's
really
need
a
lot
of
community
help
is
because
we
can't
just.
We
can't
write
the
same
tests
for
every
single
SDK
and
we
don't
know
all
the
gotchas
that
all
the
SDKs
have.
A
So
move
moving
back
to
the
futures
of
things
here
there
there
are
many
people
that
implement
s3,
there's
a
number
of
private
clouds
all
with
their
own
implementations,
Google's,
10
cents
Oracle's.
Then
there's
all
the
open-source
implementations.
Besides
SEF,
we
have
Mineo,
which
I
think
le
le
cloud
by
le
Barbara,
using
as
well
as
open
io,
that's
in
scale
in
scale
ways
deployments,
and
is
there
any
consistent
test
framework
across
all
of
them?
When
somebody
brings
a
client,
is
it
going
to
always
work?
A
A
Sorry
I,
ultimately,
I,
don't
think.
There's
anybody
from
Google
of
Amazon
in
this
room,
unfortunately,
but
I
was
talking
to
somebody
from
Amazon
recently
and
the
as
the
future
of
the
s3
test
side
should
hopefully
allow
it
to
be.
Can
all
the
providers?
Let
us
run
it
again:
let
themselves
run
it
against
their
clouds
and
collaboratively
improve
this
test
suite
and
if
they
know
there
is
their
own
SDKs
Amazon
perhaps
can
contribute
more
of
those
tests
to
say,
hey
this
works
and
they
could
I'm
em.
A
If
Amazon
wants
to
tell
me
hey
rgw
is
not
responding
to
this
one
request
correctly.
Can
we
make
it
better
for
everybody
else
with
the
s3
clients
I'd
probably
be
happy
to
fix
that
bug,
because
it
means
I,
don't
have
to
worry
about
an
end
user.
Saying
hey,
my
s3
client
is
doing
something
wrong.
It
works
against
the
Amazon.
Why
doesn't
it
work
against
you.
A
And
if
you
do
want
to
contribute
back
to
the
s3
tests,
you
can
find
it
in
the
sepal
organization
on
github
and
there's
three
tests,
and
they
will
probably
be
an
additional
future
repo
for
more
of
the
rgw
specific
side
which
will
probably
move
out
of
the
s3
tests,
particularly
for
more
performance
related
tests
in
our
GW,
and
thank
you,
I'm
Robin.
This
is
Ali
happy
to
take
questions.
D
So
I
have
a
question
about
stock
user
stats,
bucket
stats
user
quotas
because
none
of
those
things
are
exposed
by
the
s3
API.
As
far
as
I
know,
which
is
probably
intentional
by
Amazon
I
guess
they
don't
want
to
get
easy
for
you
to
know
how
much
you're
using
using
I
guess
I,
don't
know,
that's
just
speculation.
You
speaks
it.
Can
you
speak.
D
D
Come
come
it's
because
you're
on
that
side
of
the
speakers,
so
the
question
is
about:
like
user
stats,
bucket
stats
user
quotas
bucket
quotas,
these
these
things
that
we
track
internally
in
rattles
gateway
but
are
not
exposed
by
the
s3
API.
So
I
want
to
know
like
what
is
the
best
way
like.
How
are
people
actually
like
for
users
to
find
out
their
usage
right
now,
like
what's
the
best
way,
I
guess
to
use
a
different,
API
and
I
assume
that
companies
that
stuff
have
made
dashboards
for
this,
but
for
just
s3
users.
D
What
we
do
is
we
we
enable
the
switch.
We
create
a
sub
switch.
You
sorry,
Swift
user
for
them,
because
Swift
API
supports
these
stats,
but
s3
doesn't
so
I
want
to
know
is
how
the
question
is
like.
What's
the
most
convenient
way
for
end
users
to
get
this
information?
And
the
second
question
is:
would
you
guys
ever
consider
extending
making
like
SEP
specific
extensions
to
the
s2,
the
API,
to
quickly
get
this
information
I.
A
I'd
be
happy
to
add
something:
I'm,
not
certain.
It
belongs
inside
the
SBS
3s3
part
itself
that,
but
there
should
be
some
reliable
way
for
the
users
to
get
it
out.
I
think
most
of
the
commercial
deployments
would
want
to
export
that
stats
and
expose
it
to
the
cost,
the
customer,
through
their
own
mechanisms.
That's
why
I
don't
think
it
belongs
in
the
s3
side
itself:
API,
yes,
s3
ap
I
am
Not
sure
it's.
D
E
F
You
could
use
through
the
you.
We
could
certainly
add
new
api's,
that's
not
a
big
issue.
We
have
a
lot
of
private
RW,
specific
api's.
Maybe
not
very
well
documented,
not
sure
about
this
one
because
actually
I
don't
remember
anyone
asking
for
it
for
the
every
API
they
may
be
because
of
people
didn't
know
it.
The
option
even
existed.
A
The
closest
part,
it's
something
that
exists
already.
If
your
user
has
some
of
the
system
caps
and
you
do
the
head
of
a
buckets,
there's
a
header
return
that
says
the
size
of
the
bucket
without
having
to
mute
to
compute
it
yourself.
So
there
are
leak.
There
are
leaks
already
for
it,
it
leaks
badly,
but
you
have
to
have
system
caps,
I,
don't
what
you
shouldn't
have
system
caps.
A
D
G
A
Learn
it
has
been
raised
that
and
that
I
should
start
writing
something
and
so
and
submit
it
to
the
IETF.
I
was
trying
to
get
buy-in
from
somebody
at
Amazon
before
a
lot.
Actually,
in
fact,
last
year,
at
cephalic
on
and
I
didn't
so
much
as
get
the
time
of
day.
I
know
has
a
better
contact
at
Amazon
that,
hopefully
this
can
be
done.
A
That
said,
that
I'm,
not
certain
it's
a
formal
specification
in
the
long
sense,
is
going
to
be
be
helpful.
I
map
within
the
IETF
specifications
has
a
much
more
flexible
process.
They
call
I
map
keyword
registry,
where
they've
broken
down
each
of
the
I
map
keywords
into
a
so
it
sounds
much
smaller
specification,
which
is
a
much
better
way
of
tackling
the
problem.
G
A
E
They're
they're
far
along
there's
a
business
there's
a
functionally
complete,
complete
prototype,
ovale
of
an
implementation
using
key
cloak
as
the
IDP
for
Federation
and
and
it
supports
OAuth,
2
and
open
ID,
Connect
and
I
think
there's
also
documentation
for
almost
almost
all
of
it.
But
if
you
provide
something,
if
you
communicate,
whether
I
can
get
some
email
or
or
if
you
post
a
SEF
users,
I'm
pre
the
rest
of
our
others
who
are
working
on
it
could
post
there.
C
Thanks
you've
said
quite
explicitly
that
the
s3
test
is
not
about
the
internal
operation,
that
the
rgw
are
you
planning
on
doing
some
more
testing
for
the
internal
operations
on
things
like
orphans
and
that
sort
of
things
we
have
all
sorts
of
problems
with
orphans
in
the
past.
As
you
know
wondering
if
there
was
any
testing
in
that?
Yes,.
A
So
I
think
there
needs
to
be
two
separate
test.
Suites.
The
s3
side
is
about
s3
compatibility
and
then
the
separate
Audrey
Audrey
W
tests
are
more
around
the
internal
side
of
it
life
cycle
and
garbage
collection.
Really
they
don't
really.
They
briefly
touch
s/3
s/3
that
you
might
set
the
s3
lifecycle
policy,
but
everything
past.
That
is
some
internal
rgw
behavior.
That
needs
a
test
suite
of
its
own,
as
ultimately
I
think
there
needs
to
be
two
separate
tests
test
realms.