►
From YouTube: Optimizing Ceph Object Storage for Production in Multisite Clouds - Michael Hackett & Vikhyat Umrao
Description
Optimizing Ceph Object Storage for Production in Multisite Clouds - Michael Hackett & Vikhyat Umrao, Red Hat
Today, more than 60% of data storage involves unstructured data—including images, video, audio, and other types of data. The Ceph object gateway provides an object storage solution with S3 and Swift APIs that is ideal for storing unstructured data in multisite and hybrid cloud storage scenarios, scaling up to petabytes and beyond.
In this presentation we will look at sizing a Ceph Object Gateway cluster, identifying performance requirements, identifying suitable hardware and configuring storage policies with varying performance characteristics. We will also look at Ceph features such as Erasure Coding, Compression bucket index sharding and integration with OpenStack.
Failover and recovery options with also be touched upon with discussion on RGW multisite deployments.
About Vikhyat Umrao
Principal Software Engineer, Red Hat
Vikhyat Umrao works for Red Hat a a principal Software maintenance engieer on the Ceph Support team. He has co-authored the Ceph Cook Book and has presented at Red Hat Summit, Cephalocon and OpenStack conferences.
About Michael Hackett
Red Hat
Principal Software Engineer
Westford, MA
Michael Hackett is a storage and SAN expert in customer support. He has been working on Ceph and storage-related products for over 13 years. He co-authored the Ceph Cook Book and holds several storage and SAN based certifications. Michael has presented at several Red Hat Summits and last years Cephalocon
Michael is currently working at Red Hat, based in Massachusetts, where he is a principal software maintenance engineer for Red Hat Ceph Storage and the technical product lead for the global Ceph team.
A
So,
from
an
objective
standpoint,
what
we're
basically
looking
to
cover
here
is
a
generic
overview
of
hardware
selection
based
upon
performance
profiles,
use
cases
I'm
just
making
smart
choices
to
do
their
do's
and
don'ts
of
using
safe
object,
storage
and
then
kind
of
basically
just
a
high-level
overview
of
running
through
configuration
of
a
multi-site
environment.
What
we're
not
going
to
do
is
deep,
in-depth
talks
in
regards
to
crush
placement
groups,
things
like
that
or
any
type
of
deep
in-depth
into
into
multi-site,
because
we'll
need
a
lot
longer
than
3540
minutes
to
do
that.
A
So,
just
about
every
presentation,
you're
going
to
see
today
is
going
to
have
this
one
slide
that
probably
talks
about
what
it's
F
I'll,
keep
this
quick
open
source
software
solution,
multiple
storage
level,
access
for
file
block
and
object
all
from
a
single
platform
we
excel
at
high
availability,
seamless,
scaling,
seamless
upgrades
things
along
those
lines.
Saif
is
is
currently
right
now
than
that
one
backing
storage
solution
being
used
by
OpenStack.
A
A
We
have
multi-site
fill
over
capabilities
integration
with
OpenStack
via
Swift
LDAP
apt-get
directory
and
we're
able
to
separate
different
performance
profiles
via
pools
for
throughput
optimize
capacity,
optimize
just
basically
assigning
different
media
to
media
back
in
those
pools
and
VMI's
for
your
applications
that
require
higher
throughput.
Then
then
save
archival
use
cases
and
you
can
separate
those
by
pool.
A
So
why
choose
SEF
object?
Storage
over
60%
of
all
data
stored
is
unstructured.
It's
growing
rapidly,
so
sustainable
storage
solutions
required
to
obviously
services
growth,
and
that's
basically,
why
cephus
built
it's
ideal
for
streaming
audio
video
4k,
any
type
of
workloads,
imaging
analytics,
archival
things
along
those
nature.
A
highlight
here
is
basically
the
same:
skills
required
for
use
of
the
the
RVD
client,
the
ffs
client
or
the
rgw
client.
A
So
basically
we
pull
the
cluster
map
or
read
the
config,
and
this
is
done
in
order
to
determine
which
OS
DS
we
need
to
talk
to
in
order
to
get
the
data
rewrite.
The
data
from
the
cluster,
the
monitor
is
sits
nowhere
in
between
the
actual
data
path.
So
once
we
grab
that
map
information,
the
monitor
that
doesn't
sit
in
between
any
of
the
read
write
operations
coming
from
the
client.
A
So
this
right
here
is
a
basic
summary
of
what
occurs
during
an
I/o
operation.
Once
the
client
pulls
the
the
copy
of
the
cluster
map,
and
we
know
which
pool
we're
going
to
be
reading
or
writing
to
the
OSD
will
will
act
any
of
the
rights
back
to
the
client
until
we
actually
write
to
the
third
replica
of
the
pool.
A
So,
jumping
into
further
details
into
into
the
rgw
rgw
has
ten
pools
basically
set
just
for
the
service
layer.
We
need
all
these
pools
in
order
to
function
properly.
Each
pool
can
be
backed
by
a
a
separate
crush
hierarchy
depending
on
how
you
choose
a
rule
set
for
it
and
it
can
be
backed
by
different
speeds
of
media
for
those
rule
sets.
A
So
when
our
GW
is
deployed
or
a
client
is
configured
several
of
these
pools
get
created
by
default,
others
are
gonna,
have
to
go
through
manually,
create
and
and
the
documentation
watch
all
through
that.
So
that's
that's
not
something
that
we're
going
to
cover
here
with
multi-site.
You
can.
Actually,
when
you
create
your
pools,
you
can
create
them
with
a
particular
like
zone
naming
creation.
So
if
you're
replicating
from
say
Boston
to
California,
for
example,
you
can
create
US,
east
and
us
West
from
a
from
a
specific
zoning
exam
I.
A
Won't
go
into
details
about
each
one
of
the
pools.
The
documentation
does
a
great
job
on
that.
Just
kind
of
highlight
that,
with
specific
pools,
for
example,
like
your
your
your
log
pool
or
your
your
index
pool
you'd
want
to
back
that
with
a
with
a
performant
media
to
an
order.
The
service,
metadata
requests
and
and
multi-site
log
updates,
and
things
like
that.
So
performant
media's
preference
for
your
alog
pools
in
your
index,
pools.
A
If
the
index
is
the
object
metadata
for
the
objects
in
the
bucket
pool,
this
metadata
can
include
size
of
object
information
as
well
as
where
the
object
is
actually
located.
So
what
bucket
and
things
of
that
nature,
the
bucket
pool
is
the
actual
data
location
of
the
object.
Extras
is
used
when
doing
multi-part,
uploads
or
on
larger
sized
objects
or
like
gig
size
objects.
For
example,
we
will
write
the
the
different
parts,
so
we
keep
a
record
of
where
that
is
and
then
multisite
the
log
pools
are
used
for
any
of
the
DR
features.
A
So
hardware
defines
performance,
so
your
scepter
is
only
gonna
perform
as
well
as
any
of
the
hardware.
That's
chose
in
the
back
the
cluster
so
everywhere
that
you'll
read
in
any
of
the
documentation
any
of
your
SEM
propaganda,
its
advertisers
running
on
commodity
hardware
by
commodity
Hardware.
It's
not
meant
to
go
to
your
local
electronic
store.
Best
Buy's
office
grab
a
couple
of
laptops
and
then
deploy
a
production
cluster
expected
to
provide
on
a
price
service
on
something
along
those
lines.
A
So
I
mean
that's
it
that's
a
cheap
example,
but
I'm
assuming
you
get
the
picture
you
you
only
get
out
of
it.
What
you
put
into
it
from
a
safe
Monitor
standpoint,
they're
there
they're
extremely
important
to
the
Ceph
cluster.
Is
there
going
to
hold
all
the
maps
required
for
the
cluster
they're
mission-critical
services,
they're
lightweight,
but
a
loss
of
quorum
between
monitors,
you're,
basically
you're
ending
up
in
a
read-only
situation?
No
new
clients
can
connect
to
the
cluster
and
things
of
that
nature.
A
The
why
we
why
we
highlight
is
hardware
defining
performance
is
basically
going
cheap
on
your
initial
design.
If
you're
SEF
cluster
can
lead
to
higher
costs
for
support
down
the
road,
paying
somebody
to
come
in
and
recover
a
cluster
or
or
get
new
hardware
in
order
to
to
be
able
to
support
a
higher
performance
profile
that
you
didn't
expect
you
needed,
and
it
also
it'll
save
headaches
for
all
the
storage
admins
out
there
in
the
future.
They're
gonna
be
actually
supporting
your
platform.
A
There's
several
hardware
architecture
guides
that
that'll
that'll
help
you,
when
choosing
hardware
or
or
this
companies
that
provide
solutions,
architect,
services
that
can
design
this
for
you,
but
I've
listed
some
of
the
documentation
that
you
can
go
to
it's
available
upstream.
It's
available
the
red
hats,
who's.
A
intel
has
some
in
Super
Micro
has
a
nice
reference
architecture
out
there,
but
they're
in
the
valuable
source
of
information.
A
So
from
the
hardware
planning
side
of
things,
I
kind
of
break
this
down
into
four
separate
steps
when
you're,
when
you're
planning
out
your
cluster
step,
one
is,
is
the
most
important
I
think
is
which
basically,
you
want
to
understand
your
use
case.
So
why
I
say
this
is
this
is
this
is
in
regards
to
picking
the
type
of
media
that
you're
going
to
be
actually
using?
You
should
look
at
what
type
of
hardware
do
you
need
to
back?
Are
you
going
to
be
doing
streaming?
Media
any
image
store?
A
A
Step
two
is
after
you
understand
your
use
case
identify
the
type
of
workload
you're
going
to
be
using,
so
this
can
be
related
to
choosing
type
of
service
that
you're
going
to
be
deploying
RAM
CPU
things
of
that
nature.
Are
you
looking
more
for
eye
ops,
optimized
capacity,
optimized
or
throughput,
optimized
and
step
three
is,
is
look
through
expected
growth
on
the
cluster?
A
Usually
what
what
we
talked
about
from
the
support
standpoint
is
we
recommend,
looking
into
a
1-2-3
a
growth
plan
of
the
cluster,
and
you
plan
out
based
upon
that
number
of
disks
expansions
that
we're
going
to
need
to
be
making
in
that
one
to
three
year
period,
percentage
of
growth
for
servers,
number
of
racks.
You
might
need
over
a
three-year
period
based
upon
what
you
use
for
your
crush
replication
and
step.
A
So
you
want
to
factor
in
when
you're
looking
at
that
the
OS
disk
we're
talking
Rados
gateway,
you
want
to
factor
in
a
slot
for
the
the
metadata
disks
and
with
blue
store.
You
want
to
factor
in
possibly
another
another
media
slot
for
your
block,
DB
or
even
or
even
your
wall,
and
obviously
Envia
nvme
and
SSD,
for
the
the
for
looking
at
that
type
of
media,
the
back
to
your
index,
pool
with
gateway
and
your
and
your
your
wall
and
your
your
block.
Db.
A
A
So
this
kind
of
kind
of
kind
of
highlights
on
the
last
slide
out
performance
problems,
often
beginning
with
the
network,
not
with
SEF
again,
not
because
I'm
a
support
guy
I'm
not
throwing
out
a
whole
back
the
other
way.
But
that's
just
that's.
That's!
Basically
what
we
see:
we've
I've
seen,
customers
stack
and
I'm
users
stack
nodes
with
SSDs
and
VMI's
high-speed
media
and
then
just
not
half
the
network,
the
back
in
network
to
support
that
that
that
type
of
media
and
then
neat
and
then
the
network
becomes
the
bottleneck.
So
it's
it's.
A
A
A
From
from
the
performance
standpoint,
obviously
SSD
and
nvme
drives
are
going
to
be
important,
obviously
for
your
your
iap
to
optimize,
but
then,
but
then
also
for
backing
the
the
index
pool
your
metadata
pools
and
then
also,
when
choosing
for.
What's
going
to
be
your
you're
backing
media
for
blue
store,
your
block
DB
when
you're
designing
your
your
blue
store,
OS
these,
it's
it's
it's
recommended
with
with
with
Ray
dos
gateway.
A
Do
the
sides
of
the
metadata
that
we're
going
to
be
writing
that
you
size
out
to
about
four
percent
of
what
you're
going
to
be
using
for
data
block
in
order
to
set
that
DB
size?
When
we
talk
wall
configuration
of
a
wall
device
wouldn't
wouldn't
be
required
unless
we're
using
say
three
different
speeds
of
media.
A
A
A
So,
just
a
quick
running
through
some
some
cluster
tuning
options
and
and
things
that
we
recommend
with
jewel
and
prior
larger
clusters
can
benefit
when
we,
when
we
can
set
the
cluster
map
size,
as
you
could
lead
to
some
slowness
and
performance
issues
when
we're
initially
loading
cluster
maps.
So
so
we
basically,
we
can
set
these
into
the
OSD
cough
and
the
global
cough
to
in
order
to
alleviate
those
types
of
issues.
A
A
So
from
the
scrubbing
standpoint
for
anyone
that
doesn't
understand
what
scrubbing
is
it's
basically
a
it's,
a
consistency
check
between
the
the
the
objects
in
the
OSD
replicas
to
validate
that
there's
no
corruption
between
each
one
of
the
replicas.
You
have
a
light
scrub
in
the
deep
scrub
light.
Scrub
is
basically
just
checking
object,
object,
name
and
verifying
its
across
the
same
across
all
replicas
with
a
deep
scrub.
A
A
So
this
this
is
just
an
example:
server
inside
Iraq,
where
we
give
we're
showing
10h
of
the
OS
DS.
This
is
one
SSD
for
block
DB,
so
that
one
SSD
will
actually
have
a
block
DB
for
each
one
of
those
10
HT
DS,
lucilla
block
data
and
then
there's
one
SSD
that
we
set
aside
for
the
bucket
index
and
extras
and
log
pool.
A
We're
saying
use,
there's
a
note
in
there
in
regards
to
setting
PGS
per
a
pool,
as
sage
talked
and
his
his
opening
this
morning,
with
with
with
nod
lists,
we
introduced
the
the
the
auto
PG,
basically
manager,
where
you
don't
have
to
worry
about
what
you're
setting
for
pools
now
PG
counts
for
pools,
now
and/or
shooting
yourself
in
the
foot
later,
because
we
can't
reduce
those
sizes.
That's
always
been
a
problem,
but
so
now
we
can.
A
We
can
basically
set
it
in
a
an
option
in
there
to
say
we're
going
to
be
presenting
this
amount
of
data
on
a
pool
it'll
take
into
account
the
yet
what
the
pool
is
going
to
be
used
for
and
it'll
set
a
PG
count
based
upon
that.
So
you
won't
have
to
worry
about
that
in
in
dauntless
in
them
in
the
future.
B
Look
fluent
so
I
would
be.
Thank
my
I
would
be
talking
about
how
the
multi
side
works
in
object,
gateway
and
in
safe
object
storage.
So
let's
talk
about
the
basic
building
blocks.
What
we
have
in
the
multi
side,
so
first
thing
is
zone
I
mean
that
there's
a
logical
layer
I
mean
we
can.
It
has
multiple
reduce
gateway,
daemons
configured
and
it
those
diamonds
can
be
in
the
same
clusters
or
they
can
back
different
different
cluster
in
different
sites
and
moving
forward.
B
We
have
zone
group
which
is
nothing
but
container
to
different
zones,
and
these
zones
can
be
in
the
same
site
or
they
can
be
backed
by
different
cluster
in
different
sites.
Then
we
have
realm,
which
is
nothing,
but
it
can
it's
kind
of
a
sandbox,
so
they
contain
multiple
zone
groups
and
these
zone
groups
and
zones.
If
you
don't
want
to
have
a
configuration
which
is
shared
between
different
zone
groups
and
zones,
you
can
have
different
rhythms
and
they
won't
be
sharing
any
details
on
configuration
between
each
other
and
then.
B
Finally,
we
have
a
period
which
is
nothing
but
a
configuration
which,
which
holds
a
unique
ID
and
a
epoch.
So
whenever
you
do
any
changes
to
these
configuration,
we
commit
a
period
and
that
period
contains
the
latest
information
about
the
whole
configuration.
So
these
are
the
four
building
blocks
which
will
be,
you
know,
be
used
throughout
the
multi-site
configuration
and
management
part
of
it.
B
So
this
is
how
the
it
looks
like
so,
for
example,
you
have
two
sites
and
one
is
in
Massachusetts
Boston
and
one
is
in
California
San
Francisco
and
you
want
to
replicate
the
data
between
both
sides
and
you
want
to
be
SLA
based
model
and
the
client
which
is
near
to
the
side
that
should
get
I
Oh
or
something
like
that.
So
these
are
global
clusters
and
you
can
have
a
both
the
cluster,
accepting
the
writes
and
reads,
and
you
can
replicate
data
around.
B
This
is
how
it
should
its
design,
and
so
let's,
let's
talk
about
how
to
configure
it.
So
the
first
thing
what
you
have
to
do
is
that
you
have
to
create
your
global
realm.
So
this
this
is
the
command
to
whatever
commands.
We
will
be
using.
They
are
nothing
but
shadows,
get
to
admin
commands.
So
the
moment
you
run
this
command,
it
will
create
it
in
and
it
would
be
your
default
Leon.
B
You
can
have
multiple
name,
but
this
would
be
your
default
if
it
is
the
first
year
in
your
cluster
and
then
we
have
to
do
Master
zone
group
creation,
because
each
each
realm
should
have
a
master,
Joan
group
and
moving
forward.
So
these,
what
they
do
accept
is
that
they
they
need
an
endpoint,
the
reduce
gateway
dam,
an
endpoint,
and
it
should
be
a
master
and
default
the
one
group
for
that
site.
B
Then
you
have
to
do
a
master,
Joan
group,
Joan
creation
in
that
zone
group,
which
will
hold
your
all
pool
level
configuration
like
the
pools
which
service
pool
and
data
pool
and
index
pool,
and
this
will
also
accept
the
endpoints
and
the
configuration
details
for
the
zone.
And
if
you
are
not
using
your
default
configuration
which
came
as
a
default
installation,
you
can
delete
the
default
zone
group
zone
and
the
backing
pool.
B
If
you
are
not
using
them
and
moving
forward
from
here,
you
have
to
create
user,
which
will
be
used
as
a
replication
user,
or
you
can
say,
a
system
user
to
configure
your
multi-site
to
different
zones.
And
while
you
run
this
command,
you
have
to
note
the
access
key
and
the
secret
key,
because
we
would
be
using
this
configuration
at
the
later
point.
While
we
will
be,
you
know,
conferring
configuring
the
secondary
sites,
and
what
we
do
is
that
we
have
to
inject
this
excess
key
and
the
secret
key
to
the
zone
configuration.
B
So
there
is
simple
command
zone
modify
and
once
you
do
that,
you
update
the
period
it
updates
all
the
configuration
as
I
was
talking
earlier.
The
moment
you
update
the
period,
it
updates
the
configuration
for
the
whole
zone
group
realm
and
everywhere
after
that,
you
have
to
update
your
self
configuration
in
your
safe
con
to
take
this
configuration
in
effect
that
okay,
now
these
are
those
gateway.
Daemons
are
talking
to.
You
know
this
zone
group
and
zone
and.
B
Till
this
point,
your
master
site
is
configured
and
now
it's
time
to
configure
your
secondary
sites.
So,
for
example,
now
let's
move
to
San
Francisco
site
and
you
just
start
pulling
the
because
you
have
already
read
them
and
what
information
you
need
from
the
master
site
is
nothing
but
the
endpoint
and
the
access
key
and
the
secret
keys,
which
you
created
for
the
replication
user.
The
moment
you
run
this
command
your
secondary
site
cluster
will
have
then
some
information
from
the
rhythm
side.
B
Then
you
pull
the
latest
period
from
the
master
and
after
that
you
create
the
secondary
zone
here.
So
the
moment
you
create
the
secondary
zone,
you
will
you
can
again
delete
this.
The
site
default
configuration
if
you
want,
if
you
are
not
using
them
and
by
default
these
this
zone
would
not
be
a
read-only
zone.
This
is
a
global
clusters.
B
If
you
want
to
only
accept
the
reads,
you
can
configure
it
as
a
read-only
zone
and
after
that
the
moment
you
update
the
period
here,
it
will
send
the
information
back
to
the
masters
own
zone
and
zone
group
that
ok
I.
Now
this
side
has
a
secondary
zone.
Also
same
thing:
you
have
to
update
the
configuration
on
the
secondary
Radha's
gateway,
daemons
and
restart
them.
So
this
is
what
I
talked
about
the
manual
way
to
do
it,
but
latest
sensible
does
support
the
multi-site
configuration.
B
So
you
can
check
these
pull,
requests
and
I
think
there
is
a
documentation
also
available.
If
you
look
into
that
this
talk,
they
are
playbooks
which
you
have
to
configure
with
the
your
master
zone
zone
group
information,
and
once
you
run
the
play
books
on
the
both
the
side,
they
will
take
care
of
configuring,
the
you
know,
multi-site
environment.
B
They
they
advanced,
rgw
multi-site
configuration
a
for
example.
Some
of
the
users
and
customers
require
having
a
one
master
site
and
having
multiple
dr
sites,
and
during
that
configuration
most
of
the
time
we
have
seen
that
people
want
some
like
backup,
archival
and
maybe
more
global
custer,
like
configuration
like
most
of
the
like
merchandising
companies,
or
something
like
we
supposed
the
website
and
anything,
they
require
a
local
client
access,
so
they
configure
multiple
clusters.
B
So
what
you
can
do
here
is
that
there
are
two
flat
configuration
which
is
nothing
but
syncs
from
all
and
sing
from
you
can
make
use
of
them
and
you
can
have
a
multiple
secondary
site
having
a
one
master
site
and
the
moment
you
will
use
these
flying
flags,
I
mean
one.
Each
side
can
be,
you
know,
replicating
with
each
other
and
you
it
can
be
a
bi
direction
on
our
unidirectional.
B
So
how
to
manage
the
multi-site
clouds
with
the
reduce
gateway
come
on
I
mean.
So
this
is
the
basic
command
to
look
into
the
sync
status.
There
are.
A
lot
of
features
are
coming
in
the
dashboards
where
you
can
look
the
the
safe
board,
which
we
are
talking
which,
where
you
can
check
the
status
of
your?
How
much
is
the
replication
and
going
on
and
which
buckets
are
being
replicated
and
everything.
But
this
is
the
secondary
side
sync
status
when
you
will
run
a
rod
or
gate
to
admin.
C
B
Two
default
pools:
one
is
the
master
two
zones,
one
is
the
master
June
and
the
second
one
is
a
secondary
zone.
You
will
see
that
the
meta
data
is
getting
synced
from
the
master
side
and
data
is
also
getting
things
from
the
master
zone.
But
if
you
run
this
command
on
the
master
side,
you
will
not
see
the
metadata
think,
because
the
all
the
metadata
being
synced
from
master
to
the
secondary
and
it's
based
on
the
pull
mechanism.
So
always
the
secondary
side
would
be
pulling
and
any
operation.
B
If
you
are
doing
from
the
bucket
and
user
creation
point
of
view
from
the
either
from
the
reduce,
get
wet
min
or
from
the
any
external
application,
the
bucket
creation
will
be
transferred
to
the
master
site
and
the
creation
will
happen
from
the
master,
but
the
user
request
is
not
implemented.
If
you
create
any
user
on
the
secondary,
it
will
fail.
So
it
is
advised
to
do
metadata
operations,
always
in
the
master.
As
for
now
moving
forward,
I
think
there
are
some
taste
some
work
going
on
to
support
metadata
operations
on
the
secondary
sites.
Also.
B
This
is
very
much
latest.
We
have
our
GW
Data
Sync
buff
counters,
and
this
is
getting
back
ported
and
Nautilus,
and
the
these
two
this
feature
is
mostly.
This
is
a
very
good
feature,
because
till
now
we
will
not
having
we
were
not
having
any
kind
of
performance
numbers,
how
much
the
bandwidth
we
are
getting
and
how
much,
how
many
number
of
bytes
or
number
of
objects
that
are
being
compete
over
from
each
other
side,
so
pool
latency
and
fetch
bytes.
B
These
are
pretty
much
new
configuration
and
the
black
sea
which
you
can
use
with
the
help
of
the
admin
sockets.
So
when
you-
and
it
has
to
be
opposed
it.
If
you
want
to
check
the
replication
bandwidth
from
the
master
to
secondary
and
secondary
master,
you
have
to
run
the
admin
demand
side,
for
that
admit,
demand
for
that
rgw
on
that
side.
B
So,
for
example,
if
you
want
to
check
on
us
west
how,
on
the
primary
side,
how
much
bandwidth
it
is
getting
from
the
secondary,
you
have
to
do
a
uswest
admin
socket
list,
so
it
will
have
the
fetch
bytes
and
the
pool
latency.
And
then
you
can
figure
out
like
how
much
how
much
bandwidth
you
are
getting
and
how
many
number
of
bytes
are
being
replicated.
B
So
we
do
have
some
rgw
multi
side,
important
commands
like
bucket
level.
The
good
thing
is
that,
for
example,
the
use
case
is
like
you
have
a
site
configured
and
we
have
seen
a
requirements
like
some
users,
don't
want
to
replicate
all
the
buckets.
So
let's
say
you
create
thousands
of
bucket
and
you
don't
want
500
of
them
to
be
replicated,
and
you
don't
want
to
go
to
the
complication
of
having
a
local
zone
group
and
a
local
zone,
and
you
know
how
very
maybe
creating
a
different
reason.
B
So
you
can
have
some
buckets
which
are
disabled
for
sinking
and
some
bucket,
which
are
only
doing
the
same
thing
that
this
can
be
done
and
then
to
check
the
bucket
level
single
status.
You
also
have
the
bucket
single
status
command,
and
this
is
one
of
the
important
command.
If
you
want
to
see
a
global
view,
let's
say:
if
you
have
four
or
five
sites
which
I'll
configure
in
a
multi-site
the
moment,
you'll
run
the
ADO's
gateway
admin
period,
gate
command.
You
can
see
the
whole
picture
that
how
many
number
of
zones
you
have.
B
B
So
this
is
like
if,
if,
if
you
feel
that
there
is
some
problem
in
problem
in
your
multi-site
application
and
some
of
the
shards,
when
you
run
a
reduce,
get
wet
min
sync
status,
command
are
kind
of
stuck
from
a
day
or
you
know
they
are
recovering
or
something
like.
So
there
is
a
command
called
lordos
gateway
admin
data
single
status
with
a
flag
called
shard
ID.
B
Once
you
put
that
you
can
see
that
this
is
how
it
looks
like
like
how
many
number
of
buckets
are
being
recovered
recovered
and
what
is
the
bucket
ID,
and
then
you
can.
What
is
the
bucket
name?
And
then
you
can
use
the
radio's
gateway.
The
command
which
I
talked
about
the
bucket
layer
come
on
the
bucket
sync
status,
come
on
put
the
bucket
name
and
you
will
come
to
know,
okay,
where
it
is
behind
and
what's
going
on,
and
if
you
see
any
errors
or
anything,
you
can
run
a
or
Oto's
gateway
admin.
B
Sync
error
list
keep
in
mind
that
this
only
lists
the
latest
thousand
entries
thousand
shard.
So
what
you
have
to
do
is
that
if
the
error
is
old
and
you
have
to
trim
them,
they
they
sing
carrot
trim
come
on,
you
can
trim
them
and
then
you
can
see
the
latest
one.
So,
for
example,
error
happened
last
year
and
this
this
buffer
is
full
from
thousand
entries
from
last
year.
B
You
will
not
see
the
latest
error
so
to
see
the
latest
error
you
have
to
trim
first,
and
then
you
have
to
you
will
be
able
to
see
that
so
this
is
recent
I
mean
doing
some
like
last
five
or
six
ones.
We
have
seen
that
the
OMA,
oh,
my
growth-
was
happening
because
the
replication
log
trimming
was
not
going
on
properly
it
specifically
the
data
log
and
bi
logs.
B
D
A
B
B
So
I
mean
you
won't
it's
it's
a
continuous
process.
You
write
its
get
sing
and
it's
based
on
pull
request.
So
it's
like
a
so
secondary
side.
Gateway
always
keep
sending
get
request
to
the
multi
side,
to
the
primary
side,
gateway
or,
and
the
primary
side
gateway
will
be
always
sending
that
request
to
the
secondary
side,
and
they
will
keep
talking
to
each
other
and
there
is
a
it's
kind
of
polling.
So.
B
Yep
yeah
yeah,
that's
a
good
question
so
during
the
time
of
disaster
recovery
or
failover
and
recovery.
So
what
we
do
in
the
let's
say,
example:
the
Boston
site
happened
to
be
a
lost
site.
I
mean
there
is
always
a
delta.
For
example,
it's
a
iesson
replication,
it's
not
synchronous
replication.
So,
for
example,
it
depends
on
your
the
tunnel.
What
networking
tunnel
you
have,
how
fast
it
is
and
how
much
it
is
replicating.
B
For
example,
there
is
a
gap
of
half
an
hour
or
one
hour
that
much
of
data
would
be
missing,
obviously
because
it
has
not
replicated
those
outside,
but
everything
else
would
be
available
as
it
it
the
what
you
have
and
it's
a
global
cluster
as
I
talked
because
you
made
this
cluster,
not
read-only.
If
you
have
made
this
cluster
read-only,
what
was
going
to
happen
is
that
the
move
you
have
to
enable
later,
for
you
have
to
remove
the
read-only
flag
and
there
is
a
command
to
do
that.
B
Radha's
great
weight
means
don't
modify
the
moment.
You
I
mean
for
that
time.
If
you
transfer
the
your
client
I/o
request
to
this
side,
they
will
fail
because
this
site
is
a
read-only
slide.
So
you
have
to
make
a
right
arrow
side
to
this
side,
and
then
you
have
to
transfer
all
the
client
ID.
You
have
to
give
your
each
a
proxy
or
load
balancers
endpoint
to
your
clients,
to
this
sides
and
cluster,
for
example.
B
So,
but
if
it
is
possibility
that
you
were
able
to
rep
recover
that
side
and
that
half
an
hour
of
one
day,
data
which
you
were
not
which
was
not
saying
you
can
recover
that
but
yeah,
it's
a
a
Seneca,
I
think
replication.
There
is
a
chance
of
still
a
data
loss
if,
in
the
disastrous
situation,
when
the
replication
was
not
end-to-end
immediate,
it
was
not
exactly
the
same.