►
From YouTube: New workload balancer in Ceph
Description
Presented by: Josh Salomon & Laura Flores
New workload balancer in Ceph
One of the new features in the Quincy release is the introduction of a new workload balancer (aka primary balancer). While capacity balancing exists and works well since the introduction of the upmap balancer, the issue of primary balancing in order to even the load on all the OSDs was never handled. This proves to be a performance problem, especially in small clusters and in pools with less PGs. In this presentation we will discuss the difference (and sometimes the contradiction) between capacity balancing and workload balancing, explain what we did for Quincy, and outline future plans to further improve the Ceph balancing process.
A
B
Hey
and
my
name
is
Laura
Flores
I'm
part
of
the
rados
core
team
at
red
hat
and
the
things
I've
worked
on
in
the
past
have
been
or
and
still
am,
working
on,
our
Telemetry,
the
manager,
module
and
I've
done.
Some
work
for
blue
store
and
I
do
a
lot
of
work
for
Upstream
testing
to
validate
Patches
at
other
core
greatest
related
things,
and
for
the
past
few
months,
Josh
and
I
have
been
working
on
this.
The
new
rebalancer,
which
will
be
giving
a
presentation
on
today.
A
A
A
So
what
is
motivation
and
why
it
comes
only
now
so
the
the
distributed
storage
system,
the
before
under
high
load.
We
see
that
the
performance
is
following
the
weakest
link
in
the
chain
effect.
So
if
we
put
more
load
on
a
single
OSD,
we
actually
block
the
entire
system,
because
the
other
ocds
could
not
perform
in
their
full
capacity.
A
A
This
means
the
the
capacity
balancer
also
means
that
the
right
operation
are
a
pretty
well
balanced,
at
least
when
the
device
is
on
the
same
size,
but
nothing
was
done
for
read.
Balancing
and
the
reason
is
that
actually
read
balancing
their
Crush
creates
PD
balance
reads
on
the
larger
on
larger
clusters,
but
to
explain
more
why
we
need
the
read
balancer
now,
so
we
started
by
tackling
the
the
balancing
process
even
before
we
started
to
do
the
read
balancing.
We
also
tackled
some
problems
with
the
existing
capacity
balancing.
A
So,
first
of
all,
we
we
collected
a
lot
of
OCD
Mark
files,
so
we
could
do
a
testing
on
various
system.
Actually,
some
of
them
are
production
systems.
Some
development
system
is
smaller
cluster,
larger
clusters
and
the
second
thing
we
refacting
refactored
the
existing
balancer
code.
The
function
called
calc
PT
upmobs
because
it
was
really
difficult
to
understand.
It
was
like
more
than
1500
lines,
not
easy
lines
to
read
with
go-to's
in
the
middle.
A
Now
it's
we
refactor
it
to
smaller
pieces,
so
it
would
be
easier
to
to
touch
and
change,
and
this
is
also
a
call
out
to
people
who
hear
this
and
are
interested
in
the
capacity
balancer
and
so
the
code
from
previous
versions
before
Quincy.
This
was
added
to
Quincy,
stole
the
code
and
decided
don't
want
to
touch
the
code
because
it's
too
complex
try
it
again.
A
But
we
decided
to
start
small
on
risk
version
and
to
do
only
the
simplest
thing:
it
works
on
a
pull-by-pool
basis,
and
the
reason
for
this
is
that
each
pool
represent
different
type
of
workload
reads
versus
rights
and
therefore,
let's
assume
that
we
have,
we
have
read,
only
pull
and
write
only
pool
if
we
take
the
pgs
together
and
decide
to
put
on
each
of
USD
same
number
of
pgs,
but
we
don't
care
from
which
pool
we
will
create
fluctuations
over
there.
We
want
to
have
the
same
number
of
pgs
from
the
pull.
A
A
We
must
do
it,
unlike
the
read
balancer,
which
is
a
performance
requirement,
so
the
system
performs
well,
it
works
well,
but
with
less
performance
capacity
balancer
without
it
the
system
getting
full
too
early
and
we
lose
capacity
of
the
entire
cluster
balancing
these
expenses,
because
we
need
to
move
data
and
during
this
process
also
we
lose
performance.
We
don't
so
we
don't
want
to
do
something
which
it
changes
dynamically.
You
want
to
set
the
pgs
and
how
the
pages
are
mapped
to
a
steamed
work
like
this
and
not
otherwise.
A
So
I
the
read
balance
there,
unlike
this
first
of
all
its
performance
requirement
and
the
balancing
is
cheap,
we
are
playing
with
metadata.
We
are
just
changing
where
what
PG
is
primary
out
of
the
three
pgs
that
or
assuming
replica
three
out
of
the
three
pgs
three
osds
which
compose
of
each
PG
we
just
decide
which
OCD
is
the
primary.
So
it's
it's
cheap.
We
could
do
it
all
the
time
you
could
change
it
because
it
just
metadata
operation.
A
We
are
not
moving
data,
it's
balancing
the
read
performance
again
if
all
the
devices
are
homogeneous,
but
it
also
since
it's
something
that
is
cheap
and
it's
easy
to
change.
First
of
all,
we
could
use
it
for
systems
where
we,
the
devices,
are
not
in
the
same
size.
This
is
something
that
we
plan
for
for
the
next
version.
A
If
we
have
smaller
devices,
these
smaller
devices
get
less
workload,
because
we
have
less
pgs
mapped
to
the
smaller
devices,
because
we
want
all
the
devices
to
be
full
with
the
same
level,
but
since
these
devices,
assuming
they
have
they
have
the
same
performance,
get
less
workloads.
We
could
give
them
more
read
workload.
Assuming
we
know
per
pool.
We
have
assumption
on
the
ratio
between
read
and
write
performance.
We
could
actually
let
the
devices
the
smaller
devices
with
spgs
could
do
more
read
performance.
They
could
have
more
primaries.
A
By
doing
so,
we
reduce
load
from
the
the
larger
devices
which
gets,
which
are
the
bottleneck
because
of
the
the
weakest
thing
in
the
chain
effect,
and
we
could
give
them
and
they
could
do
more
operations,
because
we
took
some
read
operations
more
than
so.
We
we
freed
some
more
performance
for
them.
So
basically
we
could
use
this.
This
mechanism,
in
the
future
version
to
have
to
squeeze
more
performance
out
out
of
the
the
same
system
by
taking
the
devices
which
are
not
fully
loaded
and
fully
load
them.
A
So
every
all
the
devices
will
be
fully
loaded,
and
this
is
something
even
more
advanced.
We
could
do
something
to
dynamically
and
respond
to
changes
due
to
no
performance
fluctuations,
so
we
might
have
Network
problem.
We
might
have
noisy
neighbors,
we
might
have
all
kinds
of
things
that
create.
One
node
suddenly
performs
a
a
start
to
under
performance
since
it's
not
performing
well,
it
becomes
a
weakest
link
in
the
chain.
Since
the
the
balancing,
the
rebalancing
is
really
cheap.
A
We
could
immediately
or
when
we
identify
this
phenomena,
decide
that
we
take
the
reads
out
of
this
node,
and
this
node
would
be
for
write
only
for
some
a
specific
for
some
a
period
until
it
is
fixed
and
still
diminish
the
the
impact
of
the
weakest
link
in
the
chain.
It
still
could
become
the
weakest
in
the
engine
only
for
the
rights,
but
not
for
the
weeds.
By
the
end
of
my
stats.
Also,
there
is
an
we
are
talking
about
changing
primaries.
A
We
we
have
the
feature
of
read
from
non-primary.
This
is
also
not
necessarily
good
solution
for
performance.
It
improves
the
performance
because
it
reads
from
the
closest
OSD.
So
it's
it's
less
work
to
get
to
this
OSB.
But
if
you
have
high
workload,
let's
assume
that
we
have
Steph
on
on
two
or
three
data
centers,
and
we
assume
that
we
read
from
from
the
closest
one.
A
So
we
also
read
from
the
same
data
center
so
still
under
high
read
load,
the
bottleneck
would
become
the
data
centers
that
comes
from
this
workload.
So
what
we
suggest,
but
not
the
mechanism
that
we
implemented,
but
the
ideas
that
are
that
we
we
put
here,
could
be
also
used
to
do
some
kind
of
dynamic
balancing,
saying,
okay
under
above
some
workloads,
I'm
skipping,
the
or
I'm
changing
the
read
from
non-primary
and
doing
only
80
to
read
from
non-primary
interest
going
to
other
to
other
data
centers
to
save
the
workload.
A
It's
it's
using
the
same
idea,
but
it's
not
a
mechanism
that
we
implemented.
This
will
require
a
bit
more
work,
but
still
the
this.
What
I'm
trying
to
say
that
they
apply
to
do
that?
Sorry,
the
need
to
apply
Advanced
read,
balancing
is
applicable.
Also
when
you
use
a
read
from
non-primary
we
we
are
working
on
changing
primary
on
the
metadata,
but
if
someone
has
the
requirement
to
do
this
for
read
from
non-primary,
this
is
something
that
could
be
done
by
changing
the
way
that
you
select
the
read
from
non-primary
in
the
clients.
A
If
anyone
wants
to
play
with
this
idea,
it's
beyond
the
scope
of
this
presentation,
but
please
drop
me
a
message
and
I'll
be
happy
to
explain
what
can
be
done
next
slide.
Please
so
currently
we
have
on
homogeneous
systems.
We
don't
have
any
active
balancing
code,
but
the
crush
on
the
distribution
works
pretty
well
on
larger
clusters,
but
it's
not
necessarily
smaller
systems
tend
to
be
less
balanced
and
we
saw
some
extreme
examples
of
these
like
OCD.
A
That
gets
one
primary
and
another
gets
like
15
primaries
in
in
the
same
cluster
and
the
the
reason
that
we
we
start
coming
now
is
that
we
started
applying
sales
to
small,
kubernetes
or
open
shifts
clusters,
and
we
see
this
more
and
more
so.
The
first
thing
that
we're
going
to
show
now
solve
the
the
beat
balance
in
the
show.
Now
we're
going
to
solve
the
problem
of
the
smaller
clusters
that
are
not
balanced,
because
the
crush
statistics
doesn't
work
well
for
them.
A
It's
not
a
case
for
larger
cluster
for
larger
clusters.
They
tend
to
be
heterogeneous
over
time,
it's
very
difficult
to
keep
large
clusters
and
and
keep
it
devices
with
exactly
the
same
performance
as
exactly
the
same
size.
When
you
start
expanding
the
cluster,
you
add
new
devices,
probably
new
technologies
or
the
software
technologies
will
be
the
same
technology
hdsd
dnvme.
A
But
from
from
economic
perspective,
from
the
the
return
to
the
dollar,
you
will
buy
larger
devices
and
start
getting
devices
with
different
sizes,
and
this
is
something
we're
going
to
solve
in
the
next
version,
using
the
same
infrastructure,
but
doing
some
more
sophisticated,
a
policy,
but
something
really
really
simple
relatives
to
to
the
size
of
this.
It's
just
a
new
calculation.
So
the
conclusion
is
that
in
Reef
the
rebalancer
is
applicable
mainly
mostly
to
to
smaller
clusters,
but
we
want
to
change,
read
balancer
and
that's
the
terminology.
A
That's
why
we
say
this
to
workload
balancer,
which
will
take
into
account
the
more
than
just
rid
of
the
pool,
but
the
entire
workload
on
on
each
OSD,
and
this
is
planned
for
the
next
version
and
it
will
be
based
on
the
infrastructure.
It
will
be
pretty
simple
to
implement,
based
on
what
we
currently
have.
If
we
want
to
do
things
that
relates
to
the
read
from
non-primary,
it
would
be.
A
We
require
some
additional
work
and
if
you
want
to
do
something,
that
is
something
that
dynamically
change
things
based
on
the
behavior
of
the
cluster.
We
need
to
have
a
bit
more
work
on
monitoring
the
cluster
understanding.
What's
going
on
that,
we
have
some
places
in
the
cluster
that
that
underperform
and
try
to
take
clothes
for
them.
It's
all.
A
So
next
slide
so
I'm
going
to
present
two
PRS
that
I
added
to
with
and
there
are
going
to
actually
be
used
by
the
actually
the
actual
balancer
code,
which
Laura
is
going
to
present
in
a
minute
so
and
sorry,
I
didn't
mention
these
read:
balancing
is
only
for
replicated
pool
and
for
Erasure
coding,
it's
way
more
complex,
and
if
we
do
the
fast
motor
regression
coding,
it
tries
to
read
from
all
the
all
the
all
things
at
once.
So
we
can't
even
save
on
this.
A
It's
only
very
quickly
cool.
So
the
first
thing
that
we
added
we
added
read
balance
score.
We
see
here
it's
in
the
short
term
in
the
command,
the
OSD
pool,
LS
detail
the
the
shorter
form
we
see
that
we
have
pool
number
six
I
marked
it
because
we
will
see
the
number
later
it's
replicated
and
we
had
read
balance
score.
Read
balance
score,
it's
actually
what
it
means.
It's
the
the
most
loaded
OSD,
how
much
it's
it's
more
loaded
than
the
optimum.
So
here
we
say
that
the
most
loadable
thing
is
1.5.
A
The
the
the
optimal
score
is
one.
If
we
get
scores
one.
The
system
is
fully
balanced
for
read
so
here.
It
says
that,
roughly
that
the
the
read
buttons
that
the
most
loaded
OSD,
which
will
become
under
low
the
weakest
link,
is
the
chain
gets
150
percent
more
than
the
optimal,
which
means
it
will
become
the
the
the
weakest
link
in
the
chain
when
other
osds
would
be
roughly
66
to
Third
full.
So
it's
it's
that's
the
meaning
of
of
this
score.
Roughly.
A
It
means
that
under
low
we
are
going
to
to
get
to
send
to
this
OSD
1.5
of
the
load
that
it
should
get
in
optimal
conditions
and
therefore
it's
going
to
become
the
the
weakest
link
in
the
chain
and
start
and
block
the
system
for
ratio
coding.
It
just
doesn't
raise
your
code,
pulls
it
just
doesn't
exist.
We
don't
see
anything
till
next
slide.
Please.
A
So
I
explained
this
before
the
score:
1.0
is
optimal.
Okay,
all
scores
are
in
the
range
from
1.0
to
replica
count.
Replica
count
is
the
worst
possible
case
and
roughly
it's
not
exact,
because
we
need
to
do
a
bit
more
than
this,
but
roughly
a
score
of
1.2
is
20.
Aggregation
1.5
is
50
aggregation.
It's
it's
not
it's
just
easier
to
explain.
It
is,
but
but
that's
a
good
product
good
approximation,
even
though
it's
not
100
accurate,
but
it's
a
good
one.
A
So
we
we,
when
we
see
a
1.5
and
we
reduce
it
to
1.2,
we
actually
saved
about
30.
Instead
of
50
degradation,
we
get
the
20
degradation
we
saved.
So
next
slide,
please.
So
that's
that's
the
first
first
PR
and
actually
for
this.
We
also
if
we
get,
we
want
more
information.
So
we
we
issue
the
same
command
and
OSD
pools
details,
but
with
a
Json
format
or
XML
format,
which
they
produce
exactly
the
same
output.
A
Do
we
get
a
more
than
this,
so
the
number
that
we
saw
before
is
what
I
call
score.
Acting
is
the
read
score
based
on
the
acting
primaries?
It's
not
on
the
stable
configurations
acting
because
if
you
see
some
performance
degradation
we
want
to
to
understand.
What's
going
on,
what's
going
on
now.
What
is
important
is
the
acting
acting
primaries.
That's
what
so
this,
but
but
in
the
in
the
full
version
we
get
a
score
of
the
acting
and
the
stable
so
that
stable
configuration
optimal
score.
A
It's
not
always
a
one,
because
when
we
have
fractions
one,
you
need
all
the
the
the
Stars
to
be
in
the
same
line.
So
the
number
is
one,
but
sometimes
because
of
fractions
and
because
of
the
number
pitches
number
of
SD.
Then
it's
it's
a
little
more
than
one
and
we
have
row
score.
This
is
where
this
is
a
situation
where
all
the
devices
are
homogeneous.
The
numbers
are
well
a
bit
well
and
we,
oh,
we
don't
have
any
acting
in
any
any.
A
All
of
this
out
acting
and
the
stable
same
configuration
and
all
the
devices
have
the
same
primary
affinity
and
the
same
weight.
We
take
into
account
the
primary
Affinity
of
OSD,
which
is
the
user,
tells
us
to
pick
to
put
less
read
load
on
specific
osds
up
to
zero.
Obviously,
so
here
we
don't,
we
see
that
the
the
the
weighted
and
all
the
scores
are
are
one
and
in
a
minute
we'll
see
what
happens
when
we
change
a
bit
of
primary
Affinity.
A
I
have
the
same
primary
Affinity,
so
we
again
score
acting,
is
so
score
stable
because
acting
is
similar
the
same,
but
the
optimal
score
now
is
different
and
the
reason
it's
different
is
because
there
are
ocds
that
we
can't
give
them
all
the
primers
that
we
want
to
give
the
row
score
is
also
different,
and
if
you
see
the
score
score,
acting
is
actually
row
score,
acting
divided
by
the
optimal
score,
so
it
still
has
the
same
meaning
it.
It's
correct
that
optimal
support
does
not
represent
the
Optimal
Performance
of
the
system.
A
If
we
didn't
have
the
limitation
of
the
primary
affinity,
but
the
score
acting
says
of
how
much
I
lose,
how
much
performance
I
lose
relative
to
the
limitation
that
the
user
gave
me
with
primary
affinity.
So
it's
the
row
score
divided
by
Optimal
score,
gives
you
the
the
real
score.
So
that's
that's
important.
A
We
also
see
that
the
primary
Affinity
by
weight
is,
and
the
average
primary
we
see
that
it's
not
one,
because
the
user
asked
us
to
put
less
load
on
some
of
the
devices
and
again
all
the
numbers
here
are
the
same,
because
we
have
the
same
weight
it's
if
we
had
the
different
OSD
with
different
weights
in
different
capacity.
We
would
see
different
numbers
here
and
it
wouldn't
be
exactly
the
the
same
as
it
is
now.
A
But
that's
that's
Advanced
details
if
you
I
really
want
to
to
go
into
the
algorithm
of
calculating
this
score
and
try
to
improve
it
next
slide,
please.
So
the
next
thing
that
we
did
is
we
I.
We
added
a
new
command
actually
pair
of
commands,
PG
Atma
primary,
which
takes
the
PG
and
change
the
primary
with
one
of
the
osds,
which
is
part
of
this
PG.
You
it's
not
moving
data.
It's
changing
the
order
of
the
pgs
in.
A
In
the
PG,
so
we're
not
moving
data
and
we,
if
we
give
OSD,
which
is
not
part
of
the
PG,
you
will
get
an
error
message
and
it's
a
sister
command
that
rnpg
of
my
primary,
which
returns
to
the
defaults.
So
we
talked
about
a
pool
6,
so
we
that's
the
pool
that
I
showed
before
it
was
in
green
I,
said
we'll
use
it.
So
we
use
it
here
now.
So
we
see
that
a
TG
6.1
the
situation
is
that
the
primary
is
one
and
the
order
is
one
three
two.
A
Then
we
issue
the
command
the
PG
Atma
primary
of
6.123,
so
we
want
OCD
3
to
be
the
primary
and
we
get
the
message
marked
in
yellow
change
primary
to
sd3
and
when
we
issue
the
same
command
of
pages
brief
again
we
see
that
now
the
PG
looks
like
three
one
two
and
it's
changed,
and
this
changes
obviously
is
kept.
It's
kept
in
OCD
map
and
it's
transferred
to
all
the
pgs
and
everything.
So
this
command
is
actually
used
to
change
a
primary
per
PG
and
next
slide.
Please.
B
B
The
first
is
a
function
called
calc
desired
primary
distribution,
and
this
can
be
thought
of
as
a
policy
function
that
can
be
changed
in
the
future,
based
on
enhancements
that
we
make
and
the
second
function
is
balanced
primaries
and
this
function
is
the
overall
balancing
algorithm.
This
is
the
I
would
say
like
the
tree
trunk
of
the
framework.
This
is
the
core
of
the
algorithm
which
shouldn't
change.
B
What,
overall?
It
calculates
the
optimal
amount
of
primary
pgs
that
should
be
allotted
to
each
OSD
per
pool,
and
we
do
this
by
first
checking
that
the
pool
we're
balancing
is
replicated,
because
this
does
not
get
this.
The
rebalancer
is
not
meant
for
Erasure
coded
pools.
Next,
we
find
the
replica
count
balancing.
B
B
Next,
we
calculate
by
how
much
each
OSD
deviates
from
the
optimal
distribution,
for
instance
in
OSD,
could
have
two
more
primaries
than
it
should
or
it
could
be
under
by
one
primary.
So
this
is
where
we
find
out
how,
by
how
much
it
deviates
and
then
based
on
the
above
deviations,
we
swap
pgs
or
primary
pgs,
so
that
each
OSD
gets
as
close
to
its
optimal
distribution
as
possible
and
the
step
in
the
step.
B
How
might
these
functions
change
in
the
future,
so
the
first
one
calculate
calc
desired
primary
distribution.
This
implementation
can
be
thought
of
as
phase
one,
since
we
currently
assume
all
devices
are
identical.
Future
improvements
might
involve
enhancing
the
policy
so
that
we
put
more
reload
on
smaller
disks,
which
are
known
to
have
less
write
load.
B
As
for
the
primer
balance,
primaries
function
aside
from
any
enhancements
or
optimizations.
The
basic
structure
of
this
function
will
not
change
and
it
can
be.
We
can
think
of
it
as
the
base
tree
trunk
that
can
grow
future
optimizations
or
enhancements,
but
the
overall
algorithm
will
be
as
it
is
so
now.
I'll
move
to
some
live
demos
of
the
primary
and
a
couple
of
notes.
So
Josh
was
explaining
the
new
commands
that
we've
implemented
each
the
primary
or
it's
PG
about
yeah,
pgf,
math
primary.
B
So
in
this
demo
we
use
a
different
command
called
primary
temp,
and
this
is
because
the
pgf
math
primary
commands
haven't
been
merged
yet,
but
the
only
difference
between
these
two
commands
is
that
primary
temp
is
more
of
a
temporary
solution
where
it
temporarily
adjusts
the
acting
set.
But
for
the
purposes
of
this
demo,
this
will
work
out,
fine
and,
and
we,
the
the
future
command
will
be
available
for
Reef
and
lastly,
screenshots
of
this
demo
are
available
at
the
end
of
the
presentation.
B
So,
first
of
all
so
check
the
cluster
since
I
set
this
up
on
a
v
start
cluster.
What
I
did
was
I
set
it
up
and
I.
Let
the
cluster
settle
and
I
let
the
capacity
balancer
move
the
PG,
so
it
should
be
capacity
balanced
with,
but
we'll
check
that
in
a
moment
next
we'll
check
the
pool
details
for
the
current
read
balancer
score
and,
as
you
can
see,
pool
six
which
is
default.
B
B
And
always
we
want
to
make
sure
that
the
capacity
is
as
balanced
first.
So
what
I
did
was
I
ran
the
up
map
capacity
balancer
from
the
OSD
map
tool
on
the
OSD
map
that
we
just
grabbed
to
make
sure
that
the
capacity
is
balanced,
and
in
this
case
it
is
the
it
is
already
balanced.
So
we
don't
need
to
do
anything
here.
B
B
B
B
B
B
It's
only
three
and
zero
that
have
been
set
to
primaries
so
now
we'll
check
the
pool
details
for
the
current
rebalancer
score
and
we'll
improve
the
the
same
pool
as
we
did
before
rule
six.
This
default
rgw
control
and
for
this
one,
since
we've
adjusted
primary
Affinity,
the
read
balance
score
is
now
1.41.
B
B
B
The
previous
score
was
1.4
and
then,
after
with
the
rebalancer
suggestions,
we
can
see,
we've
lowered
it
down
to
1.03
with
a
total
of
six
changes
suggested,
and
we
can
see
again
what
the
read
balancer
suggests
from
the
out
file,
and
these
are
all
regular
stuff
commands
that
you
can
run
yourself
or
we
can
apply.
It
live
to
our
system,
go.
B
So
that's
how
you
would
run
the
rebalancer
in
this
offline
version
on
your
on
your
life
system
and
I'll.
Stop
there
I
had
one
one
more
slide
to
go
through,
but
I'll
just
save,
verbally
that
the
future
improvements
would
be.
To
add
this.
Add
this
as
a
non-offline
version,
but
as
a
a
live
version,
just
as
the
capacity
balancer
is
on
by
default
with
the
balancer
module
and
we
plan
to
do
some
more
Upstream
testing
and
performance
testing
around
this
feature.
B
A
I
answered
some
of
the
questions
that
I
saw.
These
questions
that
arrive
now
about
the
primary
Affinity
per
pool,
primary
affinity's
existing
feature,
and
it's
not
the
pool
and
it
doesn't
I
I-
think
it
does
make
sense
to
do
it
per
pool
as
a
featured
idea
as
I
see
it
from
perfect.
The
only
place
that
I've
seen
someone
actually
using
primary
Affinity
in
production
is
I.
A
We
had
a
use
case
when
user
wanted
to
use
the
capacity
the
spare
capacity
that
they
have
on
the
system
devices
for
the
safe
cluster.
But
since
it's
system
devices
they
didn't
want
to
put
too
much
load
on
this.
So
they
said,
let's
put
it
with
primary
Affinity
zero.
So
we
only
keep
the
information
for
right,
but
we
don't
read
from
them.
So
this
is
something
it's
it's
something
of
the
device.
A
A
It's
now
about
the
read
balancer.
Theoretically,
you
could
Implement
primary
Affinity
per
pool
because,
as
Laura
explained,
what
we
actually
have,
we
have
a
policy
function
which
calculates
the
target
configuration
that
you
want
and
that
you
want,
and
then
we
have
something
that
takes
the
card
configuration
and
bring
it
to
the
Target
configuration.
So
if
you
just
and
this
one
the
policy
Works
people.
A
So
if
you
decide
that
for
some
reason
you
don't
want
in
your
pool
that
all
the
osds
with
a
primary
primary
number
index
like
three
five,
seven
eleven,
they
would
not
be
primaries.
You
could
implement
this,
you
would.
What
you
need
to
do
is
make
sure
that
in
your
desired
configuration
they
would
get
no
primaries
and
then
the
the
other
function
would
give
it
to
you.
So
no
problem
for
prime
numbers,
if
you
want
that
yours
is
these
prime
numbers
would
not
be
primaries.
A
If
that's
what
you
get
so
you
could
implement
this.
It's
not
that
if
you
have
the
use
case,
yes,
you
could
do
it.
The
thing
about
this
policy
that
these
policy
is
something
that
is
pretty
easy
to
do
so
and
I
see
now.
India
said
about
the
read
from
non-prime
is
client,
and
this
is
client
side
service.
A
If
you
want
to
do
something
more
sophisticated,
as
I
said
before,
a
do
partial
read
from
non-primary
that
do
it
up
to
a
certain
level
and
then
move
to
to
also
to
others,
you
could
do
it
on
the
client
side
again.
This
is
more
more
complex,
it's
beyond
the
scope
of
here,
but
if
someone
wants
to
talk
about
this,
you
could
do
it.
We
actually
we
thought
about
implementing
this
feature
through
client
changes
and
use.
A
We
could
find
a
simple
solution
how
to
implement
this,
but
if
you
want
something
really
sophisticated,
you
could
do
this
by
changing
the
way
the
client
decides
where
to
read
from
when
the
client
supports
read
from
non-primary,
so
you
could
do
something
a
way
more
sophisticated
by
changing
the
client.
We
decided
that
for
90
at
least
from
what
we
want,
it's
not
needed
and
we
went
to
to
simpler
and
safer
sufficient
than
this.
But
it's
not
it's
not
something
that
can't
be
done.
It's
it's.
C
Things
yeah
and
it's
actually
all
like
to
some
extent.
It
actually
already
exists
because
for
the
read
from
non-primary
feature-
or
it's
also
sometimes
referred
to
as
replica
reads,
apart
from
the
mode
which
you
know
kind
of
makes
sense
to
everyone,
which
is
you
set
up
a
crush
location,
Rule
and
then
direct
your
reads
to
to
the
to
the
most
local
availability
Zone.
If
that
makes
sense,
apart
from
that
mode,
there
is
also
a
mode
which
is
has
always
been
there.
C
It's
also
confusingly
named
balance,
so
it
set
it
instead
of
localized
to
balance,
and
it
would
go
just
to
random
OSD
in
the
PG,
and
you
will
get
most
of
the
of
that.
You
know
dynamic
Behavior
that
way,
and
it
would
be
quote
unquote
balanced
for
free
just
because
of
the
hopefully
pseudo
random
nature
of
the
workload.
A
Yeah
the
thing
that
I
thought
is
that
I
I
think
that
there
might
be
some
workloads
when
you
want
to
do
what
you
say.
You
know
with
a
bit
of
minor
change
that
you
will
give
the
distribution
for
the
random,
so
it
won't
be
full
Roundup,
but
you
say:
okay
for
the
for
the
the
one
for
the
local
OSD.
This
is
the
activities
on
cents
60
and
the
other
40
distribute
randomly
that's
what
I
said
that
could
be.
A
You
know
the
ultimate
control
over
what
you
do
that
you
want
more
more
requests
to
go
to
the
local,
but
not
all
the
requests.
So
that's
and-
and
this
is
I
guess
a
minor
change
because
of
the
current
existing
situation.
You
have
the
random,
but
you
need
a
bit
more
control,
but
again
I'm,
not
sure
it's
needed.
Let's
say
it's
it's!
C
A
A
A
A
A
A
There
is
limit
to
what
you
could
do
you
can't
you
can't
do
if
you
have
I
heard
about
a
user
that
has
a
devices
of
one
terabyte
and
then
decide
decided
to
a
device
it
of
six
terabytes
to
the
same
cluster.
Okay.
This
is
something
that
we,
you
can't
compensate
for
everything.
Even
if
you
you
know
you
say
all
the
reads
are
for
the
smaller
devices.
You
can't
compensate
one
to
six.
A
When
you
have
replica
three
or
replica
two,
you
can't
accomplish
it
on
one
to
six
the
size
differences,
but
on
some
level
we
we
can
compensate,
and
you
know
to
do
what
we
could
do.
It's
not
it's
not
magic
but
reasonable
configuration.
We
could
improve
performance
on
non
on
heterogeneous
clusters
again.