►
Description
CRUSH-ing the OSD Variance Problem - Tom Byrne, Storage Sysadmin
Tom will be talking about the challenges of keeping OSD utilization variance under control in a large, rapidly growing cluster.
In the past year he has been managing a large and heavily utilized Ceph cluster that has grown from 1500 OSDs to 5000 OSDs (40PB), while maintaining an average OSD utilization of over 50% throughout the year. This has presented some unique challenges, and Tom will discuss these, along with the positive impact the upmap balancer has had on this process, and general advice for growing near full clusters.
About Tom Byrne
Science and Technology Facilities Council
Storage Sysadmin
Oxford, United Kingdom
LinkedIn Connect
The Science and Technology Facilities Council is a world-leading multi-disciplinary science organisation. We provide access to large-scale facilities across a range of physical and life sciences, enabling research and innovation in these areas. We do world-leading research, and need world-leading computing to facilitate it.
As a storage systems administrator for STFC, I've been working with Ceph for a number of years, and have been part of a team that manages several Ceph clusters for various use cases. The largest cluster currently has five thousand disks and 40PB of raw capacity, and is used to store data for the LHC experiment at CERN.
Running a cluster at this scale has presented a unique set of challenges, and has allowed me to develop an understanding of how to make best use of Ceph’s features to maximise efficiency, data security and performance. Since the last Cephalocon we have tripled the size of our large and very full cluster, going from 1.5k to 4.7k OSDs while continuing to fill the cluster to capacity, which has led to a number of insights into managing the growth of a large, full Ceph cluster.
I've talked previously about our large Ceph cluster at the Cephalocon APAC 2018 (Talk title: Erasure Code at Scale), and have talked about Ceph at various conferences in the field of computing for high energy physics.
A
Hello,
everyone,
I'm,
probably
gonna,
get
started.
I've
got
quite
a
lot
to
get
through
so
yep
cool,
so
I'm,
Tom
and
I'm
gonna
be
talking
about
crushing
the
OSD
variants
problem.
So
a
little
bit
about
me,
I'm
a
sysadmin
I
work
at
the
Rutherford
afternoon,
laboratory
which
is
part
of
the
science
and
technology
facilities
Council.
My
main
job
is
working
with
researchers
and
helping
them
store.
Data
yeah
and
I
started.
Looking
at
Ceph
in
around
2015
I
think
the
release
of
Firefly
was
the
first
one.
A
I
looked
up
and
my
main
focus
has
been
object:
storage
for
high
energy
physics
experiments,
but
we
use
it
for
a
lot
of
stuff
on
site.
So
this
is
rau.
This
is
an
Oxfordshire
in
the
UK.
The
STFC
runs
sort
of
large
experimental
facilities
that
other
people
can't
afford
to
run
or
are
too
big
for
them
to
run.
A
So
we
provide
these
to
universities,
businesses
and
industries
for
them
to
use
for
research,
I
work
in
the
scientific
computing
department,
which
is
involved
in
facilitating
those
yeah
providing
compute
and
storage
for
those
experiments,
as
well
as
international
collaborations,
so
yeah.
So
this
is
Echo.
This
is
the
largest
surf
cluster.
We
run
yeah,
it's
been
in
production
for
I,
think
two
years
now,
and
it's
currently
just
ticked
over
the
40
petabyte
mark
of
raw
capacity
and
its
primary.
All
the
data
pulls
the
EC,
a
plus
3.
A
So
that's
29,
petabytes
of
usable
storage,
the
main
user
by
I
guess
amount
of
space
used
at
the
LHC
experiments,
so
high-energy
physics
experiments
in
CERN,
but
we
have
other
users
as
well,
and
we've
sort
of
been
bringing
more
users
more
users
onto
echo
I'm,
not
really
going
to
talk
a
great
deal,
a
detail
about
what
echo
is,
what
the
hardware
is
and
how
we
use
it.
I
talked
about
that
a
lot
last
year
in
Beijing
and
I'm
gonna
link
that
talk
at
the
end.
If
people
are
interested.
A
So
yeah
I'm
who's
using
echo,
so
we've
got
the
existing
high-energy
physics
experiments
at
CERN.
We've
started
storing
data
for
existing
experiments
that
have
been
interested
in
using
echo
or
something
like
echo,
so
that's
Dimond,
which
is
on
site
and
that's
like
oh
yeah,
and
then
we've
got
new
experiments
and
communities
that
are
interested
in
using
something
like
echo
for
their
data
storage
route,
yes,
sort
of
a
more
a
more
object,
storage
type
model
than
the
traditional
scientific
data
storage,
so
yeah
so
yeah.
A
Since
the
last
time
we
talked
about
echo,
a
few
things
are
kind
of
obviously
different
yeah,
the
main
one
is
it's
a
lot
bigger
so
that
that's
the
graph
of
the
capacity
sent
in
the
last
like
15
months
or
something
and
yeah.
When
I
talked
about
it
last
it
was
ten
just
over
10
petabytes
and
it's
now
yeah,
it's
tripled
in
size.
We
I
think
we're
a
few
OSDs
below
5,000
diversities
currently,
but
we
did
did
reach
that
and
yeah
it's
a
hundred
and
eighty
storage
nodes,
and
this
is
yeah
and
it
is
still
growing.
A
We
have
more
Hardware
coming
we're
sort
of
aiming
for
a
steady
state
of
around
6,000
or
stis
for
this
cluster
and
yeah,
obviously
we're
not
just
making
it
bigger
for
fun
for
our
own
amusement
we're
storing
more
data
and
yet
currently
there's
20
petabytes
of
scientific
data
in
it.
So
that's
28
petabytes
of
raw
capacity
used
on
this
cluster.
A
So
what
do
I
want
to
talk
about?
I've
got
two
topics:
I
kind
of
want
to
touch
on,
and
the
first
one
is
the
title
of
the
talk.
It's
managing
OSD
utilization
variants
app
at
this
sort
of
scale,
I'm
going
to
talk
about
what
I
actually
mean
by
that,
what
works
or
what
worked
for
echo
and
what
didn't
work
for
echo
and
sort
of
explore
how
that
went,
and
then,
with
the
second
half
of
the
talk,
I'm
going
to
be
talking
about
what
its,
what
it's
been
like.
A
Having
that
massive
increase
in
the
size
of
the
cluster
while
continuing
to
run
production
work
and
how
SEF,
and
particularly
how
Crush,
how
the
actual
data
placement
worked
at
that
scale,
whether
whether
it
worked
as
I
wanted
it
to
so
yeah.
It's
a
collection
of
words.
What
does
it
actually
mean?
What
do
I
mean
when
I
say
this,
and
why
did
it?
Why
do
you
guys
care
about
it?
Why
do
I
care
about
it
so
crush?
Is
it's
sort
of
the
heart
of
Ceph
its
controls?
A
What
placement
groups
end
up
on
which
OSDs
and
which
data
ends
up
in
on,
in
which
placement
groups,
if
you
yeah,
if
you
randomly,
distribute
placement
groups
onto
our
SDS,
you
end
up
with
a
normal
distribution
of
your
OSD
utilization
or
placement
or
explore
OSD.
So
that's
what
a
normal
distribution
looks
like
or
a
Gaussian
or
a
bell
curve.
You
can
call
it
a
number
of
things
and
so
yeah,
it's
it's
a
histogram.
You've
got
your
frequency
up
the
side,
your
a
number
of
our
C's
and
you've
got
your
utilization
along
the
bottom.
A
A
These
are
the
ones
that
in
general,
have
less
placement
groups
and
the
average
OSD
in
the
cluster,
and
if
you
can't
figure
out
how
to
get
more
placement
groups
onto
that,
OSD
you're
never
going
to
be
able
to
use
the
space
in
it
and
it's
always
going
to
be
less
full
than
the
rest
of
the
population
and
it's
wasted
space
and
then,
on
the
other
end.
You've
got
your
over
full
ones.
A
You
want
the
center
to
be
a
nice
big
spike
with
very
little
spread.
Okay,
just
a
little
bit
of
a
reminder
and
I'm
gonna
talk
about
standard
deviation,
a
lot,
it's
just
a
easy.
Well,
it's
a
it's
a
way
of
defining
how
much
spread
there
is
in
your
variation
or
you'll,
get
the
variation
of
your
population
and
the
variance
is
just
the
square
of
that
and
yes,
I
will
be
using
those
terms,
basically
interchangeably
throughout
this
talk,
so
apologize.
A
So
what
does
it
look
like
in
reality,
I've
said
it
should
approximate
a
normal
distribution.
Is
that
true,
so
yeah
pretty
much
so
this
is
Echo.
Last
last
year,
sometime,
there's
3,000
OS
T's
in
the
cluster
at
this
point,
and
we
have
something
that
looks
very
much
like
a
normal
distribution.
I
think
my
favorite
part
about
this
is
the
incredible
spread
that
echo
is
demonstrating.
A
A
A
If
you
were
to
take
a
take
an
example
sort
of
a
fairly
absurd
example,
if
you
had,
if
your
average
number
of
placement
groups
per
OSD
was
2
and
so
crushes
pseudo
randomly
distributing
its
placement
groups
onto
OSDs,
you're,
relatively
likely
to
get
OSDs
given
enough
OSDs
relatively
likely
to
get
OS
T's
with
no
placement
rates
on.
So
your
spread
is
all
the
way
down
to
zero
percent
utilization
and
it's
probably
a
similar
amount
over
the
top,
because
you're
likely
to
get
doubled
the
amount
as
well
to
go
to
the
other
end
of
that
example.
A
If
you
have
hundreds
and
hundreds
of
placement
groups
per
OSD
you're
still
going
to
have
outliers
you're
still
going
to
have
a
normal
distribution
of
OSD
utilization,
but
it's
in
it's
essentially
impossible
to
get
an
OSD
that
has
zero
placement
groups.
It's
just
it's
very,
very
unlikely
and
similarly
you're
very
unlikely
to
get
an
OSD
that
has
somehow
got
double
the
number
of
placement
groups,
so
you
you
bring
that
spread
tighter
together
and
that
improve
your
distribution.
A
Having
said
that,
and
having
thought
about
that,
it
doesn't
necessarily
you
can't
use
that
with-
or
at
least
we
found,
we
couldn't
use
that
with
echo
to
improve
our
distribution.
Yeah
I
think
the
way
the
mass
works
is.
You
need
to
quadruple
the
amount
of
placement
groups
to
have
the
stat
your
standard
distribution
of
your
deviation
for
echo,
to
get
the
standard
deviation,
we
were
looking
for
which
was
somewhere
around
2.
A
That
was
what
we
had
calculated
before
to
be
able
to
run
the
cluster
as
full
as
we
needed
to
we
needed
to
increase
the
number
of
placement
groups
place
somewhere
somewhere.
I
think
it
was
around
25
times,
which
was
a
pretty
yeah.
We
couldn't
do
that.
We
couldn't
have
that
many
place
from
groups
and
run
the
class.
A
The
the
OSDs
were
simply
too
overloaded,
so
we
had
to
explore
other
ways
of
doing
that.
So
I'm
sure
this
is
not
used
to
anybody
in
here.
If
you
fill
up
an
OSD
any
plate,
any
pool
that
has
placement
grips
on
that
OSD
is
now
full.
You
can't
write
any
more
data
to
it
because
you
can't
control
which
OSD
or
which
placement
group
the
data
is
going
to
end
up
in
and
yeah.
This
leads
to
this
statement.
A
Looking
at
your
general
usage
statistics
on
a
large
cluster
or
in
any
cluster,
but
specifically,
if
you've
got
a
large
cluster
with
a
large
variance
in
OSD
utilization,
the
numbers
can
be
incredibly
misleading.
So
if
you
look
at
the
example
here,
yeah
we've,
the
cluster
is
just
over
half
full
in
terms
of
raw
use
capacity.
But
if
you
look
at
the
fullest
orestes
we're
yeah
we're
over
80%
and
if
you
were
to
try
and
write
another
8
petabytes
or
so
into
that
cluster,
you
would
run
out
of
space
very
quickly.
A
I'd,
say:
there's
probably
a
space
for
another
1
or
2
petabytes
before
your
fullest
OS.
These
are
well
it
like
approaching
a
hundred
percent
and
so
yeah
it's
they're
not
particularly
useful
to
statistics.
Oh
sorry,
they're
not
useful
numbers
to
look
at
if
you
have
a
lot
of
variation
in
your
OSD,
util
and
utilization,
and
because
of
this
because
we
had
a
production
cluster
that
we
were
throwing
Hardware
into
and
we
were
throwing
it
in
because
we
needed
more
space.
A
We
had
more
users,
we
needed
to
get
access
to
the
space
and
we
need
to
get
access
to
it
quickly.
This
was
a
problem.
This
was
something
that
was
making
my
life
kind
of
interesting.
So
what
do
you?
How
did
you
control
the
utilization?
And
so
when
we
were
doing
this
last
year,
this
was
the
tool
we
had
I'm.
Sure
again,
you
guys
are
probably
aware
of
this.
This
isn't
not
something
I
would
recommend
using
now
with
luminance.
A
We
have
much
better
tools
that
I'll
start
talking
about,
but
this
is
what
we
had
to
and
so,
as
a
quick
recap,
the
OSDs
all
have
re
weight
values,
it's
separate
from
the
crush
weight
and
it
can
be
between
1
and
0,
and
reducing
the
real
weight
value
will
reduce
the
number
of
placement
groups.
It's
a
yeah
to
go
into
the
specifics
of
what's
happening
at
the
crush
calculation.
Your
the
OSDs
are
effectively
competing
to
be
in
the
placement
groups
they're
drawing
straws,
which
is
why
the
algorithm
is
called
straw.
A
Horse
draw
2
and
their
straws
are
sort
of
functions
of
the
PD
ID,
the
OSD
ID
and
a
few
other
things.
The
rework
value
just
comes
in
at
the
end
and
is
multiplied.
The
length
of
the
straw
is
multiplied
by
the
rework
value
which
will
shorten
the
straw
and
move
placement
groups
of
it.
It's
a
fairly
simple
concept
and
rework
by
utilization
uses
this
to
rewrite
the
fuller
re-weight
down
the
fullest
diversities,
to
try
and
ensure
your
data
becomes
better
balanced.
A
What
does
that
look
like
on
echo
on
the
scale
of
echo
so
you're
about
to
watch
what
will
be
I?
Think
a
month
of
echo
in
I,
don't
know
maybe
30
seconds
or
something
and
you're
gonna
see.
We
run
I
think
about
seven
large:
rework
the
utilization
runs
throughout
this
to
try
and
improve
the
spread,
and
then
there's
also
a
bunch
of
other
stuff
going
on
which
I
will
talk
about.
So
it's
pretty
obvious
when
you
run
a
rework
but
utilization.
A
It's
a
very
simple
thing:
it's
pulling
the
top
off
the
distribution
and
moving
those
off
the
end
off.
It's
moving
those
full
OSDs
back
into
back
into
the
mean,
there's
also
a
lot
of
other
stuff
going
on
you'll,
see
like
the
the
mean,
is
sort
of
bouncing
up
and
down.
There's
data
going
in
there's
data
coming
out.
The
clusters
got
a
lot
of
stuff
going
on
we're.
Doing.
Placement
group
increases
at
this
point,
which
is
shuffling
data
around
internally
we're
adding
Hardware.
A
This
is
just
sort
of
small
additions
of
storage
nodes,
so
there's
a
lot
going
on
and
if
you've
been
keeping
an
eye
on
the
standard
deviation
on
the
top
left.
We
started
at
about
11,
I,
think
and
I
think
this
clip
ends
there
at
about
8,
so
we
have
managed
to
reduce
a
standard
D
we've
managed
to
improve
the
spread
of
echo
through
this
month.
However,
as
I
said
it's,
it
was
a
busy
month
for
record.
A
A
This
makes
it
yeah.
This
makes
it
not
necessarily-
or
this
made
it
not
work
very
well
for
us,
echo
was
always
gonna
have
a
lot
of
churn.
Just
the
Whitner
model,
we're
always
adding
hardware,
we're
always
sort
of
bringing
new
experiments
online.
There's
always
going
to
be
data
shuffling
around,
and
if
you,
weary,
wait
by
utilization,
couldn't
allow
us
to
sort
of
do
all
the
operations
we
needed
to
and
keep
the
utilization
in
check,
and
it
wasn't
really
gonna
work
for
us
yeah.
A
So
there
are
a
couple
of
reasons
why
I
think
it
wasn't
working
for
us
as
I
said
sort
of
the
general
churn
was
an
issue.
You
can't
add
weight
easily.
If
you
have
a
very
static
cluster,
you
can
add
weight
because
you
can
weight
everything
down
and
like
go.
You
can
run
real
weight
by
utilization
enough
that
the
average
is
weighted
down
and
the
underutilized
stuff
is
then
waited
up
by
still
being
at
full
weight.
A
Yeah
again
that
wouldn't
wasn't
really
going
to
work
on
EKKO,
we
weren't
finding
the
cluster
were
staying
static
for
long
enough.
That
relays
are
actually
able
to
have
that
kind
of
effect
and
then
finally,
the
last
point
is
the
issue:
I
think
in
particular,
with
EKKO
for
using
re-weight
by
utilization.
Is
it's
a
yeah?
It's
sort
of
a
continuous
thing.
It's,
like
you
instinctively
say
well,
if
I
weight,
something
down
by
a
few
percent,
I
expect
a
few
percent
of
data
to
move
off
it.
A
At
this
point
in
echos
existence
it
was
it
had
very
large
placement
groups,
so
I
think
individual
placement
group,
with
around
5%
of
any
particular
OS
DS
total
utilization.
That
was
the
big
data
placement
groups,
and
so
each
OST
would
only
have
a
handful
of
those
placement
groups,
and
so,
if
you're
trying
to
move
these
off,
you
don't
know
the
specifics
of
the
the
individual
calculations
that
determine
how
those
placement
groups
ended
up
on
those
OS
DS.
You
don't
know
the
length
of
the
straws.
A
A
A
When
we
had
sort
of
come
up
with
echo
and
got
some
money
to
buy
Hardware,
we
had
got
some
money
to
buy
an
amount
of
hardware
to
provide
for
the
scientists,
sorry
an
amount
of
storage,
and
we
had
based
that
on
the
assumption
that
we
could
run
a
cluster
with
an
average
utilization
of
85%,
it
seemed
fairly
achievable.
It
may
have
been
ambitious,
but
yeah.
It
was
something
that
we
thought
thought
would
be
would
be
achievable.
A
However,
in
practice
that
was
not
what
we
were
managing.
Yes,
that's
my
trying
to
trying
to
explain
what
would
happen
if
you
tried
to
run
that
distribution
operate
five
percent.
If
you
try
and
move
the
average
up
to
85
percent
you've
yeah,
it's
not
going
to
be
a
fun
cluster
to
try
and
run
at
that
point
and
so
yeah.
This
was
a
little
bit
concerning.
A
There
was
much
head
scratching
and
well
I,
don't
know,
but
luckily
the
Ceph
developers
had
been
thinking
about
this
and
up
map
came
along
so
just
very
quickly.
What's
up
map
our
map
is
a
it's
a
different
part
of
the
crust.
It's
a
new
part
of
the
cross
calculation
that
happens
after
you.
You
still
do
the
crush
calculation.
You
end
up
with
a
set
of
OSDs
for
a
placement
group.
You
then
check
in
the
up
map,
which
is
basically
a
hard
override,
which
says.
A
Having
the
ability
to
actually
forcibly
move
placement
groups
from
somewhere
to
and
and
another
explicit
location
is
not
something.
We've
ever
been
able
to
do
in
set
before
and
from
sort
of
like
a
few
a
year
of
managing
a
large
set
cluster.
It
was
one
of
the
pain
points
of
if
you
got
placement
groups
moving
in
directions.
You
didn't
want
them
to
move,
especially
if
you're
thinking
about
full
OS
DS
you,
you
can
reweighed
stuff
down,
but
it
won't
necessarily
move
that
placement
group
that
is
moving.
A
The
balancer
uses
up
maps
to
move
placement
rips
around
to
balance
your
cluster,
as
the
name
would
suggest
it
has
straight
yeah
straight
from
the
beginning.
It
has
the
potential
to
be
a
lot
more
efficient
than
something
like
relay
utilization
purely
because
it
can
as
well
as
moving
placement
ribs
off
full
osts.
It
can
move
them
on
to
empty
OS,
DS
or
underutilized
orestes.
So
you
can
sort
of
you
reduce
your
spread
from
both
ends,
which
is
useful.
A
Dan
gave
a
really
good
talk.
It's
ft,
Berlin
I
will
link
those
slides
at
the
end.
A
lot
of
the
work
I'm
gonna
describe
is
based
on
what
Dan
suggested
would
be
a
good
way
to
do
it
and
yeah
so
I
will
link
those
at
the
end
and
they'll
encourage
anyone
to
check
them
out
if
they're
they're
interested
so
started
off
by
having
a
bit
of
a
test
on
one
of
our
small
clusters
and
yeah,
it
did
exactly
what
it
was
meant
to
do.
It
was
a
very
it
was
a
very
convincing
test.
A
We
went
from
yeah
sort
of
a
fairly
standard
spread
to
a
very
yeah
I
mean
it's
no
longer
really
a
normal
distribution.
At
that
point,
it's
just,
although
I
Steve
being
within
2%
of
utilization
of
each
other,
which
is
exactly
what
you
want
when
you're
running
a
big
sf+
than
you
as
yeah.
You
really
don't
want
to
be
thinking
about
distributions
and
standard
deviations.
A
You
just
want
data
all
Europe
diff
to
be
the
same
fullness
so
yeah
after
this
it
was
a
very
easy
decision
to
switch
echo
over
to
using
up
that
balancer
and
that's
what
we
did
so
I
think
this
clip
is
going
to
be
about
four
days
of
balancing
in
total.
So
we
start
the
standard
deviation
somewhere
around
eight
and
then
it
all
just
sort
of
goes
Zoop
and
your
cluster
is
a
lot
better
balanced.
So
that's
the
standard
deviation
of
two,
which
is
pretty
good.
A
However,
it
wasn't
perfect
it
wasn't
that
perfect
distribution
that
we
saw
on
the
dev
cluster
and
I
did
a
bit
of
poking
and
it
turns
out.
We
had
run
into
an
issue
that
affected
large
numbers
of
OS
T's,
which
had
already
been
found
by
Dan
and
have
already
been
fixed,
which
was
cool
and
we
just
waited
for
the
back
port
and
then
turned
it
on
again,
and
so
what
happened?
So
again,
this
is
I,
think
another
48
hours
of
balancing
and
yeah.
It
worked
really
well.
A
Actually
I
was
pretty
impressed
with
it,
so
this
is
yeah.
This
was
kind
of
I'd
spent,
probably
the
best
part
of
a
year.
Looking
at
these,
all
these
histograms
and
being
like,
oh
no
I'm,
how
we
have.
How
is
this
ever
gonna
work
and
then
I
come
in
on
Monday
morning
in
there's,
2,000
OSDs
in
like
one
percentage
point
bin?
That's
like
that's
that'll.
Do
it.
A
So
yeah
it
worked
really
well.
Our
total
utilization
spread
had
dropped
to
5%
so
compared
to
the
sort
of
40%
we
were
managing
at
a
push
earlier.
This
was
yeah.
This
was
incredible
and
the
sort
of
the
thing
you
really
care
about
is
the
distance
between
your
the
average
of
the
distribution
and
the
fullest
OSD.
A
Because
that's
the
thing
that
controls
helpful,
you
can
how
far
up
you
can
push
the
average
the
average
cluster
utilize
it
or
sorry,
the
average
OSD
utilization
before
your
cluster
is
full,
and
so,
if
you
imagine
trying
to
run
both
of
those
clusters
at
85%,
full
one
is
completely
possible.
The
other
one
is
definitely
not
possible,
so
it's
that
sort
of
it
was
that
sort
of
difference
that
sort
of
extra
Headroom
you
get.
That
is
really
really
useful.
Yeah
sort
of
a
rough
eyeballing,
the
distribute,
the
new
distribution.
A
We
had
approximately
six
petabytes
of
extra
space
to
play
with
now
we
could
comfortably
run
the
cluster
above
80%
85%
yeah.
We
can
run
at
80%
without
even
changing
the
default
full
ratios,
which
is
really
cool,
and
once
we
get
more
comfortable,
we
can
push
it
higher
and
higher
and
once
we're
happy
that
the
up
map
balancer
is
is
working
as
expected.
A
Yeah
I
think
the
thing
that
I
found
really
interesting
about
this
was
how
efficiently
it
did
it.
So,
as
I
said,
I
it's
been
months
trying
to
balance
the
cluster
using
re
way
by
utilization
moving,
yes
actual
petabytes
of
placement
groups
around
and
then
having
the
move
back
or
move
in
the
wrong
direction,
and
it
was
sort
of
a
it
was
very
much
a
guessing
game.
This
took
a
week
of
combined
combined
backfilling
to
balance
28
petabytes
of
data,
which
was
yeah
pretty
amazing
and
the
cluster
was
in
production.
A
Throughout
this
whole
thing,
this
is
kind
of
it.
Oh
yeah.
This
is
kind
of
hard
to
look
at
I'm,
sorry,
but
the
yeah,
the
red,
is
the
rebalancing
traffic.
So
we
are
sort
of
doing
anywhere
between
well
averages
sort
of
30
up
to
50
at
the
end,
but
we
were
doing
sort
of
20
to
30
gigabytes
of
read.
Traffic
production
read
traffic
throughout
the
whole
thing,
with
no
issues
and
that's
sort
of
fairly
standard
for
echo
and
4sf
cluster
like
this.
A
So
yeah
up
map
was
really
cool
for
us
see
now,
with
the
rest
of
the
time.
I
want
to
talk
about
what
it's
been
like,
adding
this
many
OSDs
to
a
cluster
I'm,
not
actually
going
to
talk
about
sort
of
the
administrative
sort
of
thing
like
what
what
deployment
tools
were
used
or
anything
like
that.
I
want
to
talk
about
how
Crush
handled
it.
A
A
However,
I
think,
with
all
of
these
things,
there
are
a
few
things
you
don't
expect
to
happen,
and
one
particular
one
I
found
quite
interesting,
mainly
because
it
was
such
a
headache
when
it
happened
and
then
I
had
such
a
such
a
sane
explanation
when
I
figured
out
what
was
actually
happening.
So,
let's
look
at
what
adding
OS
DS
to
a
cluster
looks
like
so.
This
is
towards
the
beginning
of
last
year
and
Echo's
mm
OST
at
this
point.
A
So
this
is
right
at
the
beginning
of
what
we
started
doing
and
I
think
throughout
this
clip
it's
about
a
month
or
just
under
a
month,
and
we
add
700
OSTs
and
the
OS
DS
were
adding
and
are
slightly
bigger
in
terms
of
space
but
they're
normal
Isis
they're,
all
the
same
height
so
yeah.
So
this
is
adding
sort
of
like
batches
of
OS
DS
as
we're
sort
of
feeling
out
how
the
deployment
process
actually
works.
Ever
all
of
the
new
OS
DS
are
coming
in
at
halfway.
We're
not.
A
We
haven't
like
been
able
to
exhaustively
test
the
hardware
at
scale,
so
putting
them
in
at
half
weight
means
they've
not
got
as
many
OSDs.
We
can
check
the
memory
usage
check,
the
CPU
make
sure
it's
all
ok
and
then
once
we're
happy
with
it,
we
can
wait
everything
up
to
100%
and
the
sort
of
the
distribution
or
sort
ago
sure
I
had
a
lot
of
fun.
Making
these
animations
it
was.
It
was
an
enjoyable
couple
of
weeks
but
yeah,
and
so
that's
that's
what
it
looks
like.
So
you
can
see.
A
Yeah
we've
gone
from
the
cluster
of
the
way
can
I
do
that.
Yeah,
the
distribution
at
the
beginning
was
sort
of
around
70%
full
and
then
at
the
end,
I
can't
skip
the
end.
Sorry,
but
the
distribution
at
the
end
was
about
well
substantially
less
full
60,
something
with
percent
full.
So
what
was
that
so?
That
was
adding
50
percent
of
extra
capacity
to
a
production
cluster
in
under
a
month.
It
was
a
twelve
petabyte
cluster
at
the
beginning
of
that,
and
we
added
six
petabytes
of
orestes.
Do
it
yeah
this
was
and
yeah.
A
This
cluster
was
fairly
full.
It
was
70
something
percent
full
when
we
started
the
operation,
so
there's
eight
petabytes
nearly
eight
petabytes
of
data
in
the
cluster
that
has
to
be
shuffled
around
yeah.
A
third
of
that
data
has
to
move,
see
yeah.
This
was
all
done
while
the
cluster
was
serving
production
traffic.
There
was.
This
was
not
noticed
by
anyone
yeah,
we
didn't
have
any
complaints
about
this.
It
was
all
handled
properly
by
SEF
in
the
background,
which
is
really
good.
A
There
was,
however,
some
unexpected
placement
group
movement
so
where
the
arrow
is
pointing
I'm,
not
sure.
If
anybody
noticed
the
bulk
of
the
existing
OSDs
sort
of
are
moving
down,
get
the
occasional
one
that
kind
of
tries
to
make
a
break
for
it
decides
it
wants
to
be
more
full,
which
was
yeah
fairly
problematic
for
a
full
cluster.
When
yeah,
when
we
started
adding
hardware
and
you've
got
OS
DS
that
are
80,
something
percent
full
suddenly
trying
to
get
more
full.
It's
a
little
bit
confusing
and
so
yeah.
A
That
was
the
title
was
my
initial
reaction.
When
I
was
trying
to
figure
this
one
out,
so
I
did
a
little
bit
of
working
out
what
was
actually
happening
and
it
turned
out
yeah
most
of
the
placement
groups
were
doing
exactly
what
I
expected
them
to
do.
Most
certain
most
of
the
data
movement
was
in
the
direction
I
expected
it
to
be
in.
We
had
placement
groups
moving
off
existing
OSDs
or
out
of
existing
crush
buckets
and
into
the
newly
added
crush
buckets.
A
However,
if
you
look
at
the
script
output,
we
had
some
placement
groups
moving
on
to
newer
onto
existing
hardware
or
surround
to
existing
storage
nodes,
so
yeah,
which
isn't
really
expected,
isn't
really
what
I
expected
the
only
place
they
can
have
come
from
they
can
have
come
from,
is
the
other
existing
storage
nodes
and
that
kind
of
violated.
What
I
thought
straw
to
should
be
doing
yeah.
A
The
assumption
is
always
that
it
will
do
optimal
data
movement.
You
have
your
yeah,
you
you
wait
up
and
a
bucket
data
will
move
into
that
bucket.
It
will
not
move
between
existing
buckets
you're,
not
changing
anything
in
the
new
buckets.
There's,
there's
nothing.
There's
no
reason
for
placement
groups
to
move
between
those
days,
so
it
turns
out
int
hadn't
thought
about
it
hard
enough
and
it's
sort
of
yeah
I
was
thinking
about
it
too.
Simplistically
and
yeah.
You
kinda
need
to
step
back
a
little
bit
and
think
about
what's
actually
going
on
here.
A
So
when
a
placement
group
has
a
set
of
OS
DS
and
that
set
of
orestes
must
obey
the
failure
domain,
I'm
sure
that's,
everybody
is
happy
with
that.
That's
it's
not
not
particularly
revolutionary
and
so
crush
has
to
deal
with
that.
Crushers
do
not
return!
You
sets
of
OS
DS
that
are
on
the
that
have
share,
have
OS
T's
on
the
same
host.
Assuming
your
failure
domain
is
on
the
host
so
crush.
Does
this
it
sort
of
it
will
do
it?
A
Does
the
calculations
one
by
one
to
produce
each
OS
D
for
the
set,
and
so,
if
it
does
a
it,
produces
an
OSD
and
then
it
produces
to
the
next
OSD
and
that
OSD
collides
with
the
first
OSD.
It
has
to
redo
the
calculation
and
it's
just
got
a
value
that
increments,
which
changes
the
output
and
produces
a
new
OSD
or
a
new
crush
bucket
I've
kind
of
generalized
on
this
slide
and
assuming
that
doesn't
collide,
that's
then
in
the
sets,
and
that's
your
that's
your
set-
and
this
is
great.
A
A
However,
it
leaves
still
an
interesting
edge
case.
That
is
what
I
was
seeing
here.
So
when
you've
got
these
you've
got
this
collision
and
the
second
OSD
has
to
choose
a
new
value.
What
happens
when
that?
First
OSD,
the
one
that
caused
the
collision
originally
leaves
the
set
so
say
you
add
a
whole
bunch
of
new
hardware.
You
get
placement
groups
moving
off
existing
OS
DS
if
one
of
those
existing
OSTs
in
a
set
was
causing
the
result
to
be
changed
further
down
the
set
due
to
a
collision.
A
The
only
deterministic
thing
to
do
is
to
revert
that
calculate
or
redo
that
calculation
and
without
the
which
now
doesn't
contain
the
collision.
So
what
that
means
is
so
if
you
look
at
the
example
here
this
yeah,
this
is
a
placement
group
that
I
was
looking
at
at
the
time.
So
shard
zero
is
the
first
first
item.
A
The
acting
set
is
what
it
was,
and
then
there
is
what
it
is
now
I've
added
new
hardware,
so
we've
gone
from
1811
to
3001.
So
we've
got
this
put
this
placement
rope
has
moved
on
to
one
of
my
new
OS
DS
and
that's
fine
I,
that's
very,
very
understandable.
Well,
you've
also
got
is
OSD
one
one,
one
that
has
now
been
replaced
by
another
OSD,
that
is
on
an
existing
host
they're,
both
on
existing
hosts
they've,
seemingly
just
switch
randomly
now
that
I've
sort
of
talked
through
it.
A
It
might
be
apparent
that,
because
OSD
1811
in
1833
share
a
host
1833
actually
was
the
original
answer
for
that,
but
that
shard
for
in
the
set
so
crush.
Originally,
when
it
ran
said,
oh
I
want
1811
1833
to
be
in
that
position,
but
that
collided
with
1811
and
so
crush
had
to
increment
the
retry
value
and
produce
a
new
result,
which
was
one
one
one,
and
so
what
that
means
is
yeah
and,
and
so
the
set
the
set
contained
one
one
one,
but
once
the
the
OSD
that
was
causing
the
collision
was
removed.
A
So
yeah
this
was
a
bit
of
a
pain
point.
I
should
add
at
this
point
this.
The
amount
at
which
you
will
see
this
happening
entirely
depends
on
your
cluster
in
the
size
of
your
placement
ropes,
it's
the
relationship
between
the
size
of
your
OSDs
and
how
many
failure
domains
you
have
so
the
more
likely
you're
more
likely
to
see
this
with
big
placement
groups
with
few
failure
domains
if
you're
more
likely
to
see
collisions.
You'll
get
more
of
these
collision
replacements
happening
in
each
of
your
placement
groups,
so
yeah.
This
is
once
you
understand.
A
What's
going
on,
it's
not
that
scary,
you
can
rewrite
the
OSDs
down.
Placement
groups
will
move
off
them
if,
if
you've
got
sort
of
20%
objects,
misplaced
and
a
few
thousand
placement
groups
and
backfill
weight,
it
can
be
take
a
little
time
for
these
things
to
start,
but
in
general
it
kind
of
just
does
just
manage
itself.
A
However,
you
do
keep
getting
woken
up
in
the
middle
of
the
night
by
OSDs
going
getting
fuller
and
fuller,
which
can
be
annoying.
The
main
reason
why
I
found
this
or
I
think
this
is
quite
can
be.
Quite
troublesome
is
there's
no
visible,
the
other
there's
no
osts,
don't
keep
track
of
how
much
data
is
moving
on
to
them.
So
there's
no
warning
that
an
OSD
is
trying
to
fill
itself
up.
It
will
just
do
it
and
then
it
will
hit
one
of
the
ratios
and
then
trigger
a
health
one
or
a
health
error.
A
Having
a
couple
of
tools
like
this
and
then
sort
of
building
loops
around
them,
and
you
can
kind
of
iterate
over
your
fullest
OSDs
when
you've
got
weird
placement
group
movement,
make
sure
you're
moving
placement
ribs
off
and
make
sure
things
aren't
going
to
end
up.
Full
I
will
link
these
at
the
end.
If
anybody
is
interested,
they
are
very
hacky.
They
were
written
for
a
purpose
very
quickly
and
they
should
work.
Maybe.
A
So
yeah
right
so
finally,
one
of
the
other
things
Dan
mentioned
in
his
slides
was
adding
OSDs
using
PDF
map
so
that,
very
briefly,
you
sort
of
you
add
in
all
your
ideas.
You
up
map
all
the
placement
groups
off
them,
and
then
you
let
up
map
you.
Sorry,
you
let
the
yeah
you
let
the
up
map
balancer,
move
those
up,
remove
those
up
maps
and
move
placement
routes
back
onto
them.
It
gives
you
much
more
control
over
the
operation.
It's
it
sort
of
is
yeah
rather
than
making
a
big
bulk
re
weight.
A
Where
you
end
up
with
ten
an
object,
misplaced
and
that's
like
a
week
of
backfilling
you
can.
The
up
map
balancer
can
make
sure,
there's
always
like
a
fraction
of
objects,
misplaced,
which
means
that
you're
not
having
placement
groups
waiting
for
backfill.
This
means
that
if
you
then
hit
OSD
failures
or
anything
else
which
you
expect
with
the
cluster
this
size,
it
will
there's
there's
no
large
backfill
queue
that
the
degraded
placement
groups
get
stuck
in.
Everything
can
start
straight
away
and
you
can
return
to
a
a
plus,
a
non
degraded
cluster.
A
A
lot
quicker,
so
yeah
swing,
conclusion
sort
of
yeah
the
OSD
edition
this
year
went
really
well,
it
was
kind,
it
was
a
little
bit.
We
we
hoped
it
would
work
well,
and
it's
kind
of
is
the
thing
that
was
sort
of
sold
to
what
that
sold
set
for
us
is
the
whole
idea
of
this.
These
data
placement
algorithm
that
didn't
have
sort
of
they
only
had
optimal
data
movement.
A
When
you're
expanding
clusters,
it
was
always
the
model
we're
going
to
have
with
echo
is
always
going
to
be
new
generation,
small
new
generation
of
hardware
every
year,
and
so
we
need
something
that
can
cope
with
that
and
we
can
add
Hardware
in
and
not
have
not
have
to
completely
change
our
data
distribution
over
our
cluster
ofd
utilization
variance
has
been
a
big
part
of
my
year.
It's
been
something
I've
been
thinking.
A
A
lot
about,
I've
learned
a
lot
of
interesting
things:
I've
had
to
go
back
and
sort
of
re
rethink
about
how
Seth
actually
works
things
I,
thought,
I,
understood,
I,
definitely
didn't
understand,
which
has
been
fun
and
the
at
map.
Balancer
has
kind
of
solved
all
of
those
problems,
so
I
maybe
could
have
just
waited
nine
months
and
waited
for
up
map
balances
to
come
out
but
yeah.
The
up,
my
balancer,
is
fairly
amazing.
It
works
as
expected.
We
have
petabytes
of
extra
usable
space
on
this
cluster
and
yeah.
A
A
A
A
Young
so
yeah
we
so
we
have
the
luxury
with
echo
of
not
our
cluster
is
not.
The
cluster
use
case
is
not
latency
sensitive.
This
is
all
high
throughput
object,
storage
for
high
energy
physics
and
other
stuff,
so
we
don't
really
track
that
sort
of
stuff
I.
Don't
we
haven't
seen
anything
significant
I'm
prepared
to
believe
that
may
have
been
a
change,
but
not
anything
noticeable
for
us.
A
B
A
A
It
controls
the
OSD
distribution.
Well
enough
that
I
don't
think
we're
going
to
be
running
into
those
problems.
You
will
always
have
problems
and,
as
I
said,
the
way
crush
works
you
will
always
have.
You
will
always
have
the
edge
cases
where
placement
rates
move
in
unexpected
directions.
You
can
use
up
map
to
rather
than
having
to
re
weight
stuff
down.
You
can
now
use
up
map
to
move
placement
routes
if
they're
moving
in
the
wrong
direction.
A
If
it's
critical,
if
it's
sort
of
the,
if
yeah
effectively,
if
you're
in
an
emergency
situation,
you
still
need
to
be
able
to
identify
those
placement
groups,
and
so
the
sort
of
the
stuff
I
put
up.
There
was
more
about
being
able
to
identify
placement
group
movement
and
like
what
that
meant
for
OSD
utilization,
which
will
always
be
a
problem
I
mean
at
map
has
made
our
cluster.
We
can
run
it
a
lot
fuller,
but
we
will
run
it
a
lot,
fuller
and
we
will
run
into
the
same
problems.