►
From YouTube: Ceph Developer Summit Quincy: RADOS
Description
00:00 - Beginning
00:23 - Dashboard and rados: overview of current status and next steps
34:11 - Review crash telemetry panels: https://pad.ceph.com/p/telemetry-crashes
48:05 - Immutable Content Optimizations
1:10:37 common mgr pool
1:46:14 structured confvals to generate document (and c++ source code): https://pad.ceph.com/p/confva-yaml-doc
2:02:08 - automated auth key rotation: https://pad.ceph.com/p/auth-key-rotations
https://ceph.io/cds/ceph-developer-summit-quincy/
A
B
I
think
you're
going
to
run
the
demo,
we're
basically
going
to
add
him
around
the
support
of
rather
specific
stuff
in
the
dashboard
and
for
the
future
things
or
things
to
come.
Well,
I
will
probably
will
again
to
get
your
feedback
and
suggestions
and
also
know
about
the
things
that
are
new
to
brothers
and
that
you
can
see
their
key
for
usability
and
and
that
an
operator
should
should
have
in
the
in
the
dashboard.
C
C
C
C
C
C
Right
now
I
have
no
osd,
so
I
will
proceed
to
create
some
osds
like
I
can
go
to
the
primary
devices
not
and
right.
Now
this
is
the
current
way
of
adding
waste.
This
is
to
try
to
to
apply
some
filters
in
order
to
proceed.
So,
for
example,
I
will
go
for
the
type
and
select
hdd,
so
I
have
applied
a
filter.
B
Regarding
this
specific
workflow,
we've
got
recently
a
request
from
downstream
qe
on
being
able
to
specify
the
path
for
the
filter
in
the
track
group.
I
think
that's
supported
by
the
specification,
but
I'm
not
sure
how.
How
useful
is
that
for
if
there
are
specific
use
cases
where
we
can
foresee
that
a
user
would
need
to
filter
by
a
specific
path,
or
apart
from
the
other
fields,
that
we
are
exposing
right
now.
E
I
think
that
depends
a
bit
on
what
you
want
to
do
right,
if
you,
if
you
have
a
day
one
situation
and
you
want
to
create
a
lot
of
force,
these
on
a
lot
of
hosts,
then
specifying
the
path
might
be
a
bit
problematic
and
on
the
other
hand,
if
you
just
want
to
add
a
few
of
these
on
on
one
specific
new
host
or
so
then
having
the
password
make
sense.
So
I
think
we
kind
of
need
both.
F
B
E
E
D
It's
almost
like
you
want
to
go
to
the
device
inventory
list
and
you
see
a
disk
that's
available
and
you
want
a
button
that
says,
create
an
osd
out
of
this
disk
and
it
just
like
it
just
does
it
right,
like
that's,
what's
going
to
happen
when
you've
like
replaced
the
disk
or
you've
like
swapped,
something
or
whatever
you
just
want
to
like?
Go
okay
where's,
the
disk
there
it
is.
G
E
G
C
E
G
H
By
the
way,
I
noticed
something
like
sca
listed
in
some
examples
here.
I
think
you
want
to
be
careful
about
using
persistent
name
names
for
the
disks
as
well.
Things
like
sda
or
scp
are
liable
to
change
with
hardware
changes
or
cross
reboots.
Potentially,
I'm
not
sure
if
this
is
like
a
split
that
needs
to
be
considered
in
cydia
or
dashboard
or.
E
C
H
G
C
H
G
D
C
B
C
C
D
G
A
G
I
G
D
D
C
C
B
C
C
G
C
C
B
G
D
G
D
A
Yeah
everything,
essentially,
you
can
overwrite
the
some
of
the
defaults,
but
under
the
hood
it
auto-tunes
all
the
values
at
the
moment
with
pacific.
We
also
have
new,
like
employee
based
profiles,
which
we
could
add
to
the
dashboard,
which
will
always
have
a
default.
But
if
the
user
wants
to
change
the
profile,
it
should
just
be
like
select
abc
from
abc
and
we'll
do
everything
under
the
hood.
So
all
the
recovery,
sleep
and
max
backfills
everything
should
just
go
away.
A
B
A
D
Just
it's
sort
of
a
separate
comment
on
just
the
ui.
I
wonder
if
it
would
it's
a
little
bit
confusing
to
me
that
there's
this
drop
down
with
cluster
wide
stuff,
but
it's
on
a
page
that
has
all
these
ocs
listed.
So
when
a
dialog
pops
up
not
obvious
whether
this
is
like
came
because
you
clicked
on
something
on
the
lsd
or
whatever.
I
wonder
if
it
may
would
make
more
sense
of
these
cluster-wide
options
were
just
like
different
tabs.
G
D
B
G
G
B
C
J
G
F
A
To
comment
the
ones
we
can
get,
we
can
have
some
more
iterations
over
these.
I
think
the
whole
osd
configuration
and
how
we
display
stuff
and
what
is
important,
what
is
not
we
can
have
separate
discussions
in
the
interest
of
time
I
just
want
to
you
know
we
have
13
topics
and
we're
almost
like.
You
know,
half
hour
into
the
meeting.
What
what
other
major
topic
like
areas
do
you
want
to
cover,
except
for
the
ost
section
in
the
dashboard
ernesto
and.
C
Yeah
yeah,
we
were
doing
just
showing
the
cluster
area
of
the
diver
in
the
sense
that
if
there
is
any
missing
feature
that
you
think
that
from
the
rados
team
should
be
there
here
in
the
driver-
and
you
have
noted
or
noticed
that
something
is
not
there
and
should
be.
For
example,
it
can
be
a
great
feedback
for
us,
because
if
we
go,
if
we
finish
quickly
the
walkthrough
you
have,
we
have
the
christmas
viewer.
C
You
have
the
also
the
manager
modules,
so
you
can,
for
example,
select
a
module
from
the
dashboard.
Well,
you
have
some
the
details
also
about
the
settings,
but
the
you
can
click
on
edit
and
then,
if
I
want,
I
can
edit
some
settings
and
this
will
be
reflected
like
a
theft
config
down,
for
example,
if
I
want
to
put
the
I
will
do
a
secret
key
and
the
access
key,
I
can
do
it
in
a
graphic
manner
in
order
to
have
access
to
the
object
gateway.
C
For
example,
then
you
up,
you
update
and
it
gets
updated
and
you
do
a
safe
conflict.
Then
you
will
see
the
changes
and
for
in
the
list
of
the
manager
model,
we
can
see
what
are
enabled
and
what
are
always
on
example-
and
I
don't
know
if
we
should
add
any
other
relevant
info
here
about
the
modules
about
modules,
or
we
are
okay
here,
for
example,
can.
B
C
B
B
C
C
K
K
C
B
G
G
B
C
D
J
I
think,
for
the
sake
of
time
I
mean
there's
a
lot
of
items
on
your
list
today
and
I
I
you
know
we
got
on
the
agenda
because
you
know
we
basically
you
know,
don't
think
of
enough
people
use
this
and
give
us
feedback
so
overall
we'd
love
to
see
that
we'd
love
to
see
you
come
to
the
dashboard
meeting
and
use
this
on
occasion
give
us
that
feedback
we
need.
Otherwise,
this
product
won't
be
what
our
customers
need,
let
alone
our
engineers.
I'd
love
to
see
everyone
use
this
on
a
regular
basis.
J
I
mean
so
overall,
please
send
feedback.
You
know,
please
help
us,
you
know
you
know
with
the
features
that
we
require
moving
forward.
I
mean
it's
an
absolute
must
for
for
that.
Otherwise
you
know
it
will.
It
will
not
be
successful
and
you
know
at
some
point
I
mean
sebastian's
going
to
talk
about.
You
know
dashboard,
it's
who
say
and
their
successes
it
may
be.
Hopefully
our
next
dev
meeting
there'll
be
some
interesting.
You
know
discussions
at
that
about
you
know
suse.
J
D
A
C
A
A
M
So
yeah
I
linked
another
ether
pad
that
has
a
few
links,
there's
a
link
to
a
sentry
dashboard.
That
has
sorry
so.
First
of
all,
I
don't
know
if
everybody
knows
we're
collecting
telemetry
data
and
some
basic
data
about
the
cluster
and
data
about
crashes
and
sometimes
data
about
the
disks
of
the
clusters
and
now
we're
going
to
focus
specifically
on
the
crashes
data
collected.
M
N
G
M
There
is
an
issue
with
the
importing
the
older
events
and
there's
another
issue
with
the
latest
seven
days,
but
we're
handling
that
the
main
idea
is
to
integrate
the
reviewing
of
these
crashes
daily
when
we're
doing
a
bug
scrubbing
so
that
we
actually
get
value
out
of
it.
So
I
don't
know
niha
josh
state.
If
you
want
to
suggest
ideas
of
how
we
can
actually
integrate
looking
at
them
daily.
D
Them
yeah,
I
mean
I
ideally
we'd,
have
this
nice
there,
the
redmond
plug-in
for
century
is,
is
pretty
useless,
so
I
think
we
need
to
probably
write
around
just
so
we
can
link
these
to
tickets,
but
even
without
that,
I'm
hoping
we
can
incorporate
this
into
our
bug
scrub
routine,
so
that,
in
addition
to
looking
at
the
tracker
issues,
we
also
look
here
and
you
know
iowa.
We
can
ignore.
Obviously,
but
you
know:
here's
random,
a
random
issue.
D
Here's
some
more
stuff
version,
the
the
frequency
over
time,
and
ideally,
ideally
we'd-
have
this
link
to
the
trackers
you
see
be
able
to
see
until
then
like.
Maybe
we
can
come
up
with
some
like
we
ignore
it
once
there's
it's
solved
or
resolved,
or
I
don't
know
it's
kind
of
hard
to
link
it
to
the
without
actually
having
it
linked.
I
guess
that
isn't
quite
like
you
can't
have
a
custom
state.
D
D
A
And
then
yeah
I
mean
I
at
least
the
way
I
use
sentry
at
the
moment
is
not
like
to
do
bug
scrub,
but
when
I
see
issues
that
can
be
used
that
can
be
tracked
using
sentry.
I
do
link
it
back
to
the
tracker
or,
if
I
open
a
tracker,
I
link
the
entry
event
associated
with
it,
so
anybody
looking
later
can
go
and
see.
You
know
what
is
the
frequency
or
when
it
started
occurring
and
all
that
kind
of
stuff
but
yeah.
A
G
D
G
A
That
is
a
clear
data
point
of
what
we
need
to
focus
on
or
if
there
is
a
regression,
particularly,
I
think
the
questions
we
had
have
was
like.
You
know
how,
when
do
you
start
tracking
or,
like
you
know,
we
currently,
we
have
like
these
14
days
of
crashes
that
we,
you
know
then
make
that
more
useful
and
have
separate
sections.
A
M
M
Yes,
okay,
cool,
so
that's
the
other
dashboard
has
all
sorts
of
panels
with
the
statistics,
according
to
versions
and
other
other
parameters
as
well,
and
I
just
wanted
to
focus
on.
Maybe
this
is
the
most
interesting
part
you
can
see
how
many
new
crash
we
call
it
fingerprints,
because
we
try
to
group
together
similar
crashes.
M
M
M
M
You
can
see
just
the
crash
id,
and
here
you
have
the
actual
trace
that
wasn't
filtered
and
all
of
them
captured
details
from
that
crash.
So
it's
easily
browsable
and
by
the
way,
if
you,
if
you
see
a
certain,
a
certain
function
that
catches
your
attention,
you
can
just
click
on
it
and
you'll
see
all
of
the
fingerprints
that
actually
contain
this
so
and
you
can
keep
filtering
by
more
functions
on
the
on
the
sectors
that
has
same
one.
A
O
O
You
hear
me
yes
cool,
so
there
we'll
try
to
keep
it
short,
because
it's
kind
of
a
little
bit
out
of
topic.
So
there
are
three
of
us.
First,
massimo
is
from
university
of
peace,
and
his
focus
is
on
streaming
data
and
high
performance
computing,
and
there
is
also
nikola
dondrimo,
who
is
here
listening
and
he
is
working
for
software
heritage,
which
is
an
initiative
to
collect
all
the
course
code
available
in
the
world
and
produced
by
humanity
and
keep
it
safe.
O
O
O
O
All
this
wouldn't
work
at
all
if
it
was
read
right
objects,
that
is
it's
not
flexible
enough,
but
the
commonality
of
all
these
workloads
is
that
they
are
immutable
objects,
and
so
we
can
probably
we
can
leverage
this
property
by
packing
them
together
and
that
actually
works.
There
is
no
drawback,
so
for
software
heritage,
what
we're
doing
this
year
is
to
use
ceph
and
pack
the
objects
into
an
rbd
image
so
that
we
we
get
the
benefit
of
everything
that
we
just
talked
about.
O
So
there
we
go
what,
if
seth
provided
something
that
would
help
this
use
case
so
that
software
heritage
facebook
link
it
in
just
need
to
use
ceph
out
of
the
box
and
it
just
works.
There
is
no
space
and
solidification
and
if
you
want
to
enumerate
all
the
objects,
it
goes
as
fast
as
if
you
were
doing
an
rsync
on
100
gigabyte
worth
of
volumes,
and
I
don't
know
if
it's
possible,
because
I've
worked
and
before
I
go
into
the
ideas
that
we
got.
I
would
like
to
probe
you
everyone
here.
H
I
I
Well,
zipper
is
part
of
an
extension
system
being
developed
for
rgb
that
allows
builds
filtering,
basically
stackable
drivers
and
things,
so
you
could
yeah,
I
think
who's
imagining
this
is
the
driver.
Filter
photo
driver
for
rw,
but
I
think
in
general,
in
in
rgw,
is
where
we
would
want
to
put
this
we're,
definitely
interested
in
incorporating
packing
support
and
we
and
we
some
challenges
in
the.
I
D
I
think
the
one
of
my
questions,
I
guess,
is
how
this
is
used
like
in
in
your
case
the
week
when
you
prepare
these
big
rbd
images.
Is
there
this
whole
sort
of
offline
phase
where
you
back
pack
it
all
together
and
you
build
an
index
and
you
put
the
index
at
the
front,
and
then
you
import
that
into
exactly.
D
That,
once
this
goes
once
this
is
in
ceph
like
you
want
the
index
like
in
the
front,
so
you
can
do
best
listing
of
the
names
of
objects
and
look
up
a
specific
object.
That's
in
this
big
blob
and
that's
a
little
bit
challenging
to
do
if
you're,
like
sort
of
slowly,
you
know
doing
these
over
time,
but
yeah,
I
think
it
would
be.
It
would
be
helpful
to
have
maybe
a
set
of
a
set
of
constraints
to
better
understand
how
it
would
be
used
like
what
are
the.
D
What
are
the
ingestion
works
loads
like?
Is
it
like
a
trichlophiles
that
are
sort
of
randomly
accumulated,
or
is
it
like
a
serial
import
of
a
linear
sweep
of
a
file
system
or
whatever,
and
then
also,
then,
what
are
the
like?
The
subsequent
workloads
like
the
batch
export
or
static
file,
or
so
on.
So
I
think
those
understanding
what
those
workloads
look
like
and
what
the
expectations
are.
There
would
help
drive
with
the
design.
O
O
I
Well,
I'm
not
sure,
I'm
not
sure
that
the
film
driver
explains
all
of
it,
for
what
rw
you
do,
but
rgb
might
be
using
that
filter
in
order
to
in
order
to
do
listings
or
to
incorporate
or
incorporate
edits.
There's
also-
and
this
is
this-
is
valuable
this
this.
This-
is
this
this
this,
your
your
your
posts
just
develop,
reminded
us
to
think
about
this
more.
We
have
other.
G
D
I
mean
we
have
had
some
discussions
around
in
a
raido's
pools
that
have
different
access
semantics,
where
they
might
be
immutable
or
when
you
create
a
new
object.
You
know
that
it
has
a
unique
name
or
something
like
that,
because
those
would
affect
the
way
that
the
replication
and
recovery
algorithms
work,
but
those,
I
think
those
are
different
because
they
don't
really
address
the
problem
of
tiny
objects
and
bulk
import
and
export,
and
so
on.
So.
I
O
O
I
guess
that's
where
I'm
a
little
hesitant,
because
when,
when
I
thought
about
it,
the
benefit
of
packing-
let's
say
at
least
one
million
object
together-
that
having
at
least
10
gigabyte
worth
of
something
that
you
can
mirror
seemed
like
a
good
way
to
change
the
order
of
magnitude
of
the
problem,
and
I
I
still
don't
see
if
rgw
can
provide
that
kind
of
change.
If
that
makes
any
sense.
O
D
I
I
don't
think
zipper
does,
but
but
by
itself,
but
zipper
is
a
way
that
there's
a
way
to
do
inline
edits
of
the
other
requests
to
direct
it.
To
do
something
specific,
but
my
sense
is
this
probably
involves
I
mean
I
want
you
to
jump
in
here
too.
We've
talked
about
this
a
bit
and
and
the
the
object
manifest
that
rw
has
a
lot
allow
for
objects
to
have
alternate
structures
that
can
be
arbitrary
and
an
async.
I
Another
piece
probably
would
be
asynchronous
processing
after
ingest
to
ingest
these
into
their
pack
form
or
to
do
later.
Compactions.
If
we,
if
we,
if
we
make
edits
your
system,
never
makes
edits,
but
others
might
so
that
and
zipper
gives
you
zipper.
I
think,
gives
you
inline
access
to
territory
target.
You
know
to
make
more
flexible
access
patterns
instead
of
rhw.
If
we
see
that
we're
looking
for
something
that
has
a
more
complicated
as
an
alternate
access
path,
I
mean
other
than
our
bucket
index
or
or
then
we
have
some.
P
Yeah
well,
yeah
zipper
is
not
going
to
solve
everything
magically
just
start
the
framework
where
everything
will
need
to
be
created
right.
That's
one
thing,
just
a
side
note
that
access
pattern,
where
you
have
multiple
separate
objects,
and
then
you
can
read
everything
in
one
go
reminded
me
of
this
with
the
terribles
with
large
object
implementation.
P
I
Because
yeah
the
ultrasound
of
implementation,
but
yeah,
but
it
does
do
that
and
and
then
the
pieces
are
all
like
you
know,
as
in
your
workflow,
the
pieces
are
sort
of
at
the
inverse
of
it,
but
in
switch
object,
storage
like
these
live
objects.
The
pieces
are,
the
pieces,
are
themselves
objects
that
something
uploaded
and
then
and
then
and
then
it
uploaded
a
rule
for
combining
them
right.
R
R
R
But
you
need
to
look
very
carefully
at
whether
we
are
trying
to
what
whether
objects
are
pre-packed
before
rgw
gets
them
or
whether
we
can
afford
to
like
write
a
bunch
of
four
kilowatt
objects
into
r2w
and
then
have
rgw
like
read
them
out
and
rewrite
them
into
a
packed
form
and
delete
them.
Whether
lookups
can
afford
to
go
find
light.
R
Do
the
like
rgw
generic
object,
look
up
to
find
the
the
packed
location
and
then
like
look
at
the
the
index
of
that
packed
object
and
then
do
the
actual
read,
and
that
and
that's
my
concern
with
with
something
like
this,
especially
if
we're
trying
to
like
do
a
generic
implementation.
We
expect
to
work
across
a
large.
A
large
range
of
anything
is
that,
okay,
because,
like
I
think
the
haystack
paper
had
a
lot
of
math
to
demonstrate
that
that
every
that,
what
their
constraints
were
and
and
that
their
design
like
hit
them.
O
O
O
So
currently,
there
are
an
ecosystem
and
they
know
postgres,
they
know
mysql,
they
know
other
backends,
they
don't
know
ceph,
and
I
was
wondering
and
that's
where
massimo
is
most
interested
in,
if
steph
has
so,
it
could
be,
maybe
a
rgw
that
allows
to
plug
in
this
kind
of
back
end,
so
it
made
available
as
a
stream
of
objects
somewhere
which
would
help
again
with
mirroring
and
again.
The
speed
of
this
mirroring
would
matter.
O
G
H
H
H
But
the
bright
side
is
that
these
are
all
steelers.
So
if
any
time
one
goes
down
restarted,
it's
not
a
very
complex
operation.
It
doesn't
require
a
much
complex
recovery
and
the
the
key
piece
I
mean
that
you
need.
There
is
a
like
a
proxy
layer
like
a
shape
proxy,
which
we
already
often
deploy
with
the
dashboard
to
be
able
to
stand
in
front
of
whatever
ek,
endpoints.
We
may
be
exposing
there,
but
that's
something
that
we
already
need.
H
H
B
B
B
Well,
right
now
there
are
some
existing
caches
like,
for
example,
the
controllers,
the
http
controllers.
There's
this
view
cache,
which
is
caching
things,
for
example
like
the
rbd
images
or
the
pools,
and
apart
from
that,
I
think
we
have
some
caching
in
in
the
front
end
as
well,
we're
also
avoiding
the
browser
to
pull
in
the
backend.
So,
but
I
think
in
the
case
of
the
dashboard,
we
will
probably
need
to
bring
this
kind
of
layered
approach,
because
there
are
multiple
places
where
this
dashboard
can
have
multiple
users.
H
H
The
doesn't
address
is
models
that
might
potentially
need
to
scale
beyond
a
single
process.
I'm
not
sure
that
we
have
any
of
those.
Currently,
I
think
the
one
that
has
been
a
bottleneck
is
prometheus
and
maybe
a
better
approach.
There
would
be
to
change
the
way.
We're
reporting
media's
data
to
give
it
directly
to
prometheus
itself
from
each
of
the
nodes,
instead
of
funneling
it
all
through
a
single
or
even
a
few
of
energy
processes,
I'm
not
sure
yeah
how
much
value
we're
getting
at
that
extra
processing.
There.
D
D
Because
I'm
wondering,
if
that,
if,
if
the
approach
we
take
to
scale,
the
manager
is
to
basically
separate
separate
out
modules
into
separate
manager
processes,
then
that
limits
our
the
the
ceiling
to
what
we
can
get
is
like
the
number
of
manager
modules
right
like.
If
we
have
10
manager
modules,
then
at
most
we
can
scale
10
times
bigger,
but
I
think
the
reality
is
that
we
don't
actually
have
that.
Many
only
a
few
of
the
manager
modules
are
actually
ailing.
H
G
D
Yeah,
I
guess
it
seems
like
if
we
sort
of
address
those
piecemeal,
then
there's
only
like
by
count
on
one
hand
like
the
remaining
modules
that
are
problematic,
and
so
I'm
wondering,
if,
like
the
cost
benefit
of
the
complexity
of
breaking
these
things
out,
is
going
to
be
limited.
The
benefit
will
be
limited
by
one
hand.
I
guess
that
makes
sense.
H
I
D
G
G
D
H
C
Yeah,
for
example,
creating
countries
or
of
a
lot
of
osds
a
lot
of
rbds
thousands
of
buckets,
for
example,
for
for
dashboard.
I
can
imagine
several
several
testing
and
and
retrieving
info
that
info,
for
example,
retrieving
thousands
of
buckets
10
000,
for
example,
or
1000
rpd
images,
because
we
are
doing
listing
and
we
are
trying
every
time
to
to
improve
the
way
we
retrieve
the
info.
C
I
I
don't
know
I
don't
know,
for
example,
the
an
upstream
consumer
like
the
cern
I
it
would
be
great,
at
least
for
example,
I
don't
know
if
the
the
stress
that
the
cern
put
on
clown
staff
is
something
more
realistic.
I
don't
know
what
are
the
exact
figures
that
the
cern
is
using
for
safe?
I
don't
know
how
many
or
is
this,
how
many
rpds
or
are
you
having,
but
if
we
can
retrieve
this
data
and
create
some
more
resistant.
A
H
A
Q
G
D
Of
fake
osds
that
are
sending
manager
reports,
whatever
it's
whatever
it
is,
I
guess
my
like
high
level
comment.
Thought
is
basically
that
seems
like
there
are
sort
of
lower
hanging
fruit
where
there
are
existing
issues
that
are
sort
of
independently
addressed,
make
the
current,
even
with
the
current
architecture,
just
perform
better
like
the
notifies
yeah.
H
D
H
I
agree:
it's
not
the
first
first
thing
to
do.
I
think
I
definitely
want
to
introduce
the
progress
module,
insights
notification,
consumption
first,
but
I
think,
like
the
we've
already
seen,
that,
like
the
balancer
and
the
other
modules
doing
a
lot
of
cpu,
so
it
might
be
time
to
think
about
that
splitting
out.
Even
with
a
single
you
know
single
host
into
multiple
processes.
D
H
H
G
H
A
A
I
guess
one
more
point:
before
we
move
to
the
auto
scaler
I
wanted
to
bring
up
this
stage.
We
had
this
discussion
about
using
a
common
pool
for
some
of
the
modules
like
insights
and
device,
health
and
stuff.
I
think
that's
still
a
good
idea.
We
just
have
like
one
manager,
module
pool
versus
you
know
different
rules
and
inside
it's
doing
its
own
thing.
D
D
A
N
N
So
the
new
algorithm,
how
it
works
is,
is
that
just
in
a
high
level
term
is
that
it
starts
out
with
like
a
full
complement
of
pgs.
So
that's
like
so
how
much
pg's
the
post
in
general
should
be
used
so
like
it
depends
on
the
amount
of
like
osds
and
the
monitor
target
pg
per
osds.
So
let's
say:
there's
like
four
osd,
so
four
times
like
100
would
be
400
and
assuming
the
the
replication
size
of
each
pool
is
like
one.
N
The
full
complement
in
this
case
would
be
like
400
and
let's
say
there
are
four
pools
and
it
so
when
you
start
out
with
four
pools,
each
pool
would
get
400
divided
by
four.
That's
like
a
hundred
pieces
each-
and
I
guess
like
if
you
round
it
up
to
power
of
two
there's
128
pgs
each
but
let's
say,
pool
one
started
using
50
of
the
capacity
of
the
space.
So
how
it
works
is
that
pool
one
would
get.
N
Fifty
percent
of
like
the
full
complements
of
pg's
in
this
case,
is
two
hundred
to
round
it.
Up
with
power
of
two
is
two
fifty
six
and
the
rest
of
the
pools,
whose
two,
three
and
four
would
get
so
the
200
divided
by
three,
which
is
like
67
p
like
yeah,
so
it
would
get
like
64
pgs.
So
that's
just
how
it
works.
Assuming
like,
let's
say
stuff
like
the
bias
is
like
one.
The
problem
with
this
that
I
encounter
is
that
when,
if
you
can
see
in
the.
N
There's
a
problem
with
when
you
for
rgw
with
this
new
r
scaler
it.
So
when
you
start
out
the
pool
for
the
device
health
monitor
starts
up
with,
like
128
pgs
and
when
you're
creating
like
rgw
pools.
It
did.
The
device
health
monitor
did
not
scale
down
in
time.
So,
therefore,
that
hits
like
the
cap
for
mon
max
pg
per
osd's
cap
and
one
of
the
solutions
that
I've
been
working
on
and
fixing.
That
is
just
to
create
a
pg
num
max
on
it's
another
feature.
N
D
H
So
we
do
have
like
ways
that,
like
ffs
and
rgw,
create
their
metadata
pools
and
they're
already
applying
certain
settings
to
them
so
that
we
could
apply
similar
kinds
of
mac
settings
for
those
metadata
pools.
At
least
I
was
thinking.
Maybe
this
max
should
be
like
a
percentage
of
the
like
pg
budget,
rather
than
a
fixed,
absolute
number,
though,
or
like
a
large,
a
very
large
cluster.
You
might
want
to
have
a
more
parallelism
for
your
metadata.
A
That's
yeah,
that's
that
definitely
sounds
better
than
a
hard
cap.
What
max
should
be
like
yeah?
What
you
said
is
right.
I
mean
if
we
know
what
the
application
is
going
to
be.
Maybe
we
can
determine
those
gaps
based
on
whether
it's
rgw
data
pool
or
metadata
pool
whatever,
but
it's
just
like
a
generic
pool.
The
user
is
trying
to
create,
for
those
it'll
be
hard
to
determine
what
those
gaps
should
look
like
or
that's.
We
just
have
to
go
with
something.
H
G
D
Seems
like
there's,
I
mean
there's
no,
there's
no
substitute
for
actually
having
some
information
about
what
the
pool
is
going
to
be.
If
we
can
somehow
induce
a
user
to
set
the
target
ratio
on
a
pool,
then
we're
going
to
like
pick
the
right
number,
the
first
time
and
then
we're
not
going
to
have
to
like
glitter
merch
cool.
But
in
the
absence
of
that
information
we
basically
have
two
options.
D
H
H
Yeah,
like
I
think
when
we
were
hitting
this
and
testing
it
was
actually
it
wasn't,
even
a
pool
that
was
being
created
with
a
particular
pc
name.
So
it
was
just
using
the
default
low
number,
but
because
the
articulator
had
already
kind
of
used
up
the
entire
budget
and
because
of
like
the
rounding,
it
ended
up
being
close
enough
to
the
cap
that
we
just
went
a
little
bit
over.
D
G
D
D
D
I
guess
the
the
one
other
sort
of
element
in
the
room
is
that
this
was
such
a
shock
when
it
when
it
happened
during
when
I
was
testing
the
pacific
upgrade
because
as
soon
as
the
manager
upgraded,
which
was
like
the
very
first
step
of
the
upgrade
suddenly
pg's
went
crazy
right.
There
was
a
bunch
of
splitting
all
the
whatever
everything
went
nuts,
and
so
I
think
on
upgrade
like
I
don't
know
that
we
want
to
like.
H
D
Yeah
or
even
like
an
auto
scaler
profile
or
something
so
there's
like
the
conservative
profile
that
has
the
current
behavior,
where
things
start
small
to
scale
up
and
there's
like
a
yeah,
a
new
profile
that
we
introduce.
That's
this
new
behavior,
but
isn't
the
default
for
upgraded
clusters,
but
it
is
for
new
clusters
or
something
like
that.
Yeah.
G
A
A
H
So
that
making
up
related
to
the
auto
scaler
was
that
we
have
some
of
these
warnings,
like
the
learning
about
object,
skew
and
I'm
not
sure
if
there's
maybe
a
few.
Others
like
this,
that
we
introduced
before
the
auto
scale
existed
to
try
to
warn
people
about
pgs
being
imbalanced
or
not
having
too
few
or
too
many
or
something
like
that.
H
H
A
A
D
H
G
R
H
A
A
A
A
E
T
Oh
yeah
sure
so
the
change
that
that's
been
merged.
Basically,
what
that
does
is
is
that
we
dynamically
change
what
amount
of
logs
are
trimmed
before
this
change,
we
used
to
have
an
upper
bound
specified
by
paxos
service
trim
max.
So,
according
to
the
log
ingest
trade
right
now
we
just
or
change
the
max
accordingly.
T
A
I
guess
I
mean
what
what
I
sure,
as
pr
does,
is
much
better
than
what
we
had
earlier,
where
somebody
had
to
manually
go
change.
Some
setting
after
the
monitor
had
already
the
monitor
db,
had
already
filled
up
to
maintain
the
ingest
rate
versus
the
trimming
rate.
But
I
guess
the
bigger
question
is
that
is
there
a
need
for
all
this
to
be
stored
in
you
know,
go
through
access
and
store
them
on
db
or
not
or
like?
What
is
the
historical
significance
of
something
that
it's.
D
D
Suspect
that
a
fairly
big
part
of
the
problem-
maybe
not
if
yeah
one
part
of
the
problem,
is
also
just
that
the
way
the
log
monitor
is
implemented
is
like
not
efficient
at
all
like
it
has
a
it
has
like
a
there's,
a
log
summary
class
or
something
that
has
like
the
last
100
entries,
and
it's
basically
that
entire
structure
is
rewritten
on
every
commit
like
it's
just
it's
just
totally
the
way
it's
persisted
is
just
totally
stupid,
and
that
was
just
because
it
was
like
expedient.
I
think.
L
But
I
think
the
way
how
we
are
using
the
the
cluster
log
is
the
wrong,
because
I
think
the
large
the
cluster
lock
is
for
random
message.
Human
readable
message,
which
is
very
critical
and
which
should
get
the
attention
of
the
administrator
immediately,
which
should
up
in
the
of
course
call
for
human
intervene
at
this
very
moment.
Instead
of
for
some
some
slow
operations,
I
think
the
flow
operation
should
be
should
be
to
be
sent
to
manager
to
module.
For
example,
the
earth
module
you
you
created.
L
It
could
be
repurposed
for
for
collecting
the
subscribers
subscribe.
The
slow
message,
for
example,
when,
when
the
alert
list
module
notice
that
there
is
some
slow
operation
going
on,
it
is
supposed
to
collect
the
details
from
the
from
the
demons,
which
is
that
which
has
low
messages
and
log
into
a
local
time.
L
Local
database,
which
is
not
hosted
by
by
civ
cluster
and
later
on
the
when
the
as
major
as
a
major
administrator
log
on
to
the
system
using
the
dashboard
or
something
he
or
she
will
notice
that
something
goes
wrong
and
it
will
close
the
data,
along
with
the
timestamp
and
figure
out
what
was
going
on
and
and
looking
to
the
log
or
whatever
he
or
she
can
have
done.
Instead
of
looking
at
a
class
lock,
prologue
is
for
something
a
a
system
is
going
wrong
or
or
something
from
something
big
happens.
H
So
I
think
that
that's
the
thing
I
like
with
the
slops,
though,
is
that
that
information
is
very
helpful
in
the
cluster
vlog,
because
it's
very
it's
a
fairly
simple
place
to
match
up
what
is
going
on
with
the
attack
cluster
state.
I
don't
think
we
need
to
have
all
details
of
every
single
slow
request,
which
is
what
we
currently
do.
L
H
D
G
A
S
I
would
say
that
you
need
to
know
which
pgs
are
experiencing
slow
ops,
so
even
just
a
stat
line
in
the
pg
stats
would
do
the
job
you
don't
even
need
to
list
the
slow
requests
at
all
yeah
like
aggregated
just
this
pg
is
as
of
the
sosd
mapbook
or
as
of
this
reporting
interval,
seeing
slow
ops.
That's
it.
S
A
I
guess
the
idea
is
that
if
you
don't
have
the
ability
to
lie
right,
but
if
you
don't
have
the
ability
to
do
live,
debugging
or
like
capture
the
state
of
the
usd
at
that
moment.
Having
that,
like
the
initial
information
of
you
know
where
the
initial
store
slopes
were
or
why
they
were
in
the
cluster
log,
as
an
afterthought
is,
has
been
often
useful.
S
C
G
S
H
D
L
L
S
I
mean
the
managers
could
offer
a
general
purpose.
Sampled
logging,
yes
mechanism,
without
needing
to
go
through
the
monitors
access
system.
I'm
saying
we
could
have
not
a
logging
system
that
isn't
the
cluster
log
that
serves
the
same
purpose.
The
cluster
log
currently
does,
but
without
screwing
up
the
monitors.
G
D
H
D
I
think
it's
nice
because
you
have
a
there's
a
log
last
command
that
like
dumps,
the
last
few
messages,
and
actually
one
of
the
things
that
I
wanted
to
do
was
change
it.
When
you
just
select
w
you
can
give
it
the
channel
name.
I
wanted
to
combine
that
with
the
logs
command
so
that
you
could
do
you
could
do
like
an
equivalent
of
a
tail
dash
f,
do
do
it
logs
what
the
channel
is
number
event
recent
entries
and
then
also
block
and
pull
or
whatever
to
follow
the
log.
D
R
H
R
H
L
G
G
H
D
H
H
A
L
L
I
propose
an
solution
that
we
can
add
is
the
option
in
engineering
file
with
with
a
predefined
scheme
which
is
flexible
enough.
So
we
can
even
write
some
c
simple
code
in
it
and
it's
inflexible
structured
enough.
So
we
can
use
a
python
script
to
extract
the
interesting
information
from
it
and
generate
generated
rc
file
and
the
more
interesting
things
that
we
could
generate
different
option
versions
from
a
single
source
of
yaml
file.
H
L
F
G
A
D
D
L
G
D
I
H
L
I
think
we
could
start
with
some
some
basic
commands,
but
by
first
step,
is
to
to
convert
the
the
option.cc
to
a
jumbo
file
and
whip
up
as
python
script,
to
generate
the
cc
file
and
legacy.h
file
from
this
yaml
file
and
then
expand
this
script.
So
it
can
can
can
generate
the
subset
of
the
options
and
render
it
using
a,
for
example,
a
ginger
template
to
rst
and
render
it
using
things
later
on.
We
can
expand
it,
expand
it
by
adding
more
labels
and
schemes.
H
H
A
D
Yeah
yeah,
so
this
was
ignored
for
a
long
time,
because
we
didn't
have
encrypted
sessions
between
demons
and
the
monitor,
and
so
any
sort
of
automated
key
rotation
would
just
start
sending
keys
over
the
wire.
That
was
actually
would
be
worse,
but
now
we
have
messengers
to
secure
mode,
so
no
more
excuses.
I
think
that
there
are
two
big
challenges.
D
D
R
D
P
D
D
H
I
mean
the
other
stuff
like
that.
Separate
would
be
the
discrete
key.
I
like
the
luxe
encryption
keys.
Is
there
some
kind
of
version?
I
thought
that
was
like
some
kind
of
version
of
lux
the
the
way
you
could
rotate
those
keys
by
having
like
having
a
key
that's
encrypted
with
another
key,
and
you
rotate
the
encryption
key
or
something
like
that.
D
G
D
Could
be
rotated,
so
probably
a
similar
mechanism
would
need
to
be
used
in
that
case
too,
that
one's
probably
a
little
bit
easier,
because
it's
there's
only
like
one
place
where
the
dmcrypt
is
like
started
up
and
so
like
identifying
the
path
that
needs
to
try.
Two
different
keys
would
be
a
little
bit
simpler,
but
I
think
even
then
it
just
it's
a
similar
versus
question
of
like
which.
D
D
There's
a.
I
have
a
question
down
here
about
that,
but
right,
so
I
I
guess
there
are
these
two
options
right
either
you
have
like
the
old
key
and
the
new
key
both
available
on
the
monitor
and
the
client
will
use
it
whatever
key
it
has
and
either
if
it's
old,
it'll
work
and
if
it's
new
it'll
work
or
you
flip
it
around-
and
you
have.
D
Kind
of
leaning
toward
option
one
having
it
done
on
the
monitor,
because
they're
sort
of
the
way
we
can
narrow
down
the
cases,
and
it
might
even
be
that
the
idea
of
having
multiple
keys
associated
with
the
same
auth
entity
might
actually
be
a
good
thing.
Also
in
general,
I'm
not
really
entirely
sure.
D
D
Actually
you
can
do
it
a
key
rotation
so
that
when
you,
when
you're
renewing
your
ticket,
you
could
try
using
the
new
key
and
it
would
transition
seamlessly
from
the
old
key,
so
that
you're
sort
of
you're
an
exist.
An
existing
session
would
be
able
to
transition
from
an
old
key
to
a
new
key
without
being
interrupted,
and
that
I
think,
needs
a
more
careful
read
of
the
auth
code
to
see
exactly
how
that
would
be
implemented.
I
don't
think
delia
is
here.
D
Unfortunately,
but
we
probably
need
to
brainstorm
on
that
figure
out
exactly
how
that
would
work,
and
that's
assuming
that
we
actually
want
to
be
able
to
do
this
without
restarting
demons.
If
we,
if
we
think
it's
tolerable
to
just
if
you
rotate
every
six
months,
for
example-
and
you
just
like
restart
demon
demons
as
you
do
it,
then
that's,
then
you
can
do
sort
of
a
simpler
solution.
H
D
Yeah,
so
I
think,
I
think
being
able
to.
I
think
it's
an
it's
a
good
idea
to
teach
them
on
client,
basically
how
to
and
the
author
protocol
or
whatever
how
to
transition
from
an
altitude
to
a
new
key
while
it's
online
and
so
there's
sort
of
some
hypothetical
scenarios
here
like
the
client,
would
generate
a
new
key
and
then
store
install
it
on
the
installed
on
the
on
the
monitor
and
then
it
would
re-authenticate
using
a
new
one
and
then
the
monitor,
perhaps
as
soon
as
it
sees
that.
D
D
There
are
a
few
things,
there's
sort
of
another
question
of
how
you
actually
update
the
key
on
the
client
like.
Would
you
want
stuff
adm
to
like
update
the
key
ring
file,
or
would
you
want
the
process
that,
like
the
mod
client,
for
example,
that
you
just
told
to
go,
do
this
sort
of
online
key
change
to
rewrite
its
own
key
file
key
ring
file
and
in
the
case
of
blue
story,
you
also
have
to
remember
that
the
key
ring
file
is
actually
on
a
temp
fest.
D
D
And
then
the
last,
the
last
challenge
is
around
kernel
clients
because
that's
like
sort
of
a
separate
implementation
and
it's
a
little
bit
more
awkward,
because
you
know
mount
that
stuff
probably
needs
to
be
able
to
accept.
If,
if
you
go
the
two
keys
on
the
client,
it
needs
to
have
those
keys,
but
the
kernel
is
not
going
to
rewrite
its
keyring
file,
so
you're
going
to
have
to
have
some
other
like
helper
utility
or
whatever.
D
That
runs
on
the
client
host
that
you
know
pokes
sysfs
or
whatever,
to
tell
seth
to
go
change
its
key
and
then
updates
wherever
the
key
ring
file
is
stored
in
etsy,
ceph
or
whatever
it
is
so
it'll
be
a
little
bit
tricky.
We
need
to
make
sure
that
we
standardize
the
way
that
those
kernel
clients
are
managed
right
now.
I
think
mount.set
doesn't
take
a
traditional
keyring
file
which
has
sort
of
been
a
future
ticket.
Q
D
D
D
D
D
D
Q
We
don't
allow
access
to
etsy
so
where
you
can
read
only
access
to
edc,
but
I
don't
think
we
do
that
with
sub-adm.
In
fact,
I'm
not
sure
what
this
is
kind
of
off-topic,
but
I'm
not
sure
what
kind
of
protections
we
have
in
the
system
to
unifiles
we
generate
with
cdm.
Maybe
that's
something
we
should
look
at
because
there's
a
divergence
there.
D
D
In
general,
nothing
gets
or
demons.
At
least
nothing
is
demons,
don't
use
etsy
stuff
at
all.
The
only
time
separation
uses
epcs
is
when
you
use
the
shell
command,
and
that's
mostly
just
so
that
you
can
have
if
there
is
an
etsy
stuff
on
the
host,
that's
sort
of
like
the
default
cluster
that
the
fediam
shell
will
bring
up.
So
you
have
to
pass
all
the
arguments
all
the
time.
D
D
Okay,
well,
I
guess
the
action
items
are
probably
just
to
knee-high.
You
could
check
the
source
and
see
if
there
are
any
like
additional
constraints
and
then
I'm
going
to
lean
towards
seeing
if
option
one
it's
feasible.
If
there
are
a
small
enough
number
of
paths
that
the
monitor
can
do
it,
I
think
that's
probably
going
to
make
a
little
bit
more
sense
and
then.