►
From YouTube: Ceph Developer Monthly 2021-10-06
A
B
B
Okay,
so
I'll
just
be
giving
some
updates
about
the
telemetry
performance
channel
today,
some
background
again,
you
may
have
heard
this
already,
but
I'll
just
go
through
it
really
quickly.
What
is
telemetry
the
telemetry
module
collects
anonymous
data
about
users
clusters
and
this
data,
which
is
displayed
on
public
dashboards,
is
used
by
ceph
developers
to
understand
how
people
are
using
their
clusters
and
to
identify
any
problems
that
users
are
experiencing.
B
The
telemetry
module
is
broken
down
into
several
channels,
each
of
which
collects
a
different
set
of
information.
So
the
basic
channel
provides
some
basic
information
about
the
cluster.
The
crash
channel
provides
some
information
about
daemon
crashes.
The
device
channel
provides
some
information
about
device
metrics,
the
ident
channel,
if
users
opt
in
provides
identifying
information
about
the
cluster
and
the
perf
channel,
which
is
what
I'll
be
talking
about
today,
provides
performance,
metrics
of
a
cluster
as
developers.
B
What
is
the
perf
channel
currently
collecting,
so
the
first
set
of
metrics
I'll
talk
about
today
that
we
are
collecting
are
perf
counters,
where
these
the
perf
counters,
the
way
we're
deriving
that
is
through
the
manager
module.
There's
a
method
called
get
all
proof
counters
where
we
can
fetch
those
counters
and
they're
filtered,
where
only
counters
that
are
considered
useful
or
higher
priority
are
being
collected.
B
Each
of
the
sets
of
perf
counters
are
divided
by
metadata
servers,
monitors,
osd's
radius
gateway
and,
as
one
example
of
what
somebody
might
gather
from
these
perf
counters
is
if
you're
looking
at
the
blue
store
priority
cash
counters.
These
counters
are
useful
for
diagnosing
problems
with
memory.
Consumption
in
osds.
B
Another
set
of
information
we're
collecting
is,
I
o
rate
irate
is
being
derived
by
fetching
fetching
it
from
the
manager
with
the
get
method
and
we
are
collecting,
I
o
rate,
to
gain
a
better
understanding
of
the
changes
in
osd.
I
o
activity
between
telemetry
reports
in
this
json
skeleton
of
what
that
might
look
like.
I
have
on
the
side
we
can.
B
B
This
is
data
aggregated
from
fetching,
a
pg
dump
from
the
manager
module
and
it's
it's
aggregated
by
pool.
We
are
collecting
stat
sum
per
pool
to
learn
how
placement
groups
are
managed
in
a
cluster.
However,
we're
currently
aggregating
these
metrics
by
pool,
but
we
decided
through
discussion
that
it
would
be
better
to
collect
this
information
at
an
individual
pg
level
rather
than
at
pool
level.
So
that's
a
work
in
progress
right
now
so,
and
this
might
look
a
little
different
to
you
as
we
move
forward
with
those
changes.
B
Another
important
thing
I
want
to
address
about
stat,
semper
pool
is
that
we
were
originally
including
pool
name
in
in
this
report.
However,
we're
removing
pool
name
in
a
pr.
I
have
open
right
now,
because
that
could
potentially
contain
sensitive
information,
which
is
goes
against
the
anonymous
integrity
of
telemetry.
B
The
original
intent
was
to
provide
some
context
about
how
the
the
pool
that
we're
collecting
in
what
way
it's
used.
For
instance,
is
it
a
manager
pool,
and
there
are
certain
default
pools
that
we're
interested
in
and
knowing
that,
but
the
way
it
was
implemented,
it
could
collect,
pool
names
of
that
people
create
themselves,
which
is
not
what
we
want.
B
B
We
want
to
collect
mempool
stats
to
see
the
number
of
bytes
and
items
that
each
type
contains
and
from
that
derive
which
types
are
consuming
pool
resources
and
the
addition
of
this,
the
mempool
stats,
is
addressed
in
a
pr.
I
have
open
right
now,
if
you're
interested
in
seeing
the
implementation-
and
I
have
a
skeleton
of
what
that
might
look
like
it's
really
similar
to
the
output
of
what
you
might
see
from
running
stuff,
osd
dump
mempools
another
metric,
we're
interested
in
adding
is
a
compaction,
roxdb
compaction.
B
B
The
skeleton
I
have
json
skeleton
I
have
to
the
side
is
just
about
my
interpretation
of
what
that
might
look
like
this
doesn't
exist
yet,
but
once
we
are
able
to
extract
that
data,
which
adam
is
currently
working
to
do,
he
did
a
lot
of
work
to
extract
those
counters.
This
is
what
it
might
look
like
and
that's
our
work
in
progress
extracting
that
data
that
we're
going
to
discuss
at
the
later
telemetry
huddle
this
week.
B
B
That
customers
are
curious,
so
questions
that
need
to
be
answered
about
this
idea
to
address
unavailable
data
should
we
collect
metrics
per
pg
rather
than
per
pool.
The
answer
to
that
is
probably
yes,
and
if
so,
which
metrics
do
we
need
to
collect
and
then
to
answer
the?
How
often
of
that
question
does
a
measurement
or
counter
exist
that
is
being
reset
to
help
us
distinguish
between
unavailable
data
occurrences?
B
Can
we
miss
the
window
when
these
occurrences
happen
and
then,
lastly,
what
are
some
possible
reasons
for
data
being
unavailable?
That
would
be
possible
for
telemetry
to
detect,
and
this
is
this
idea-
is
the
probably
the
least
fleshed
out
idea
we
have,
but
it's
still
really
interesting
to
talk
about.
So
maybe
you
all
will
have
some
ideas
about
what
kinds
of
metrics
would
be
helpful
in
detecting
occurrences
of
unavailable
data,
and
that's
all
I
have
to
share
for
this
presentation.
C
C
B
D
Yeah,
that
was
the
idea
stage
so
for
like
things
like
pg
log
length
and,
like
you
know,
recovery,
stats
and
scrub,
stats,
etc,
pg
level
things
are
more
useful,
but
I
think
the
general
idea
was
there
are
some
metrics
that
make
sense
on
a
pool
basis.
Like
you
know,
a
number
of
bytes
and
number
of
rights,
all
that
can
still
be
useful
on
a
pool
basis,
but
there
are
certain
metrics
that
if
you
aggregate
on
a
pool
it
doesn't
make
too
much
sense.
So
for
those
we
need
the
pg
level
stats.
B
Yeah.
Okay!
Yes,
thank
you,
so
yeah,
maybe
we'll
decide
that
we
want
to
keep
stat
some
per
pool
in,
but
either
way
we're
missing
that
pg
level.
You
know
logs
that
we're
we're
not
getting
here.
So
that's.
It
could
be
an
addition,
or
we
may
not
necessarily
remove
this
all
together,
but
yeah
either
way
we're
not
addressing
things
at
a
pg
level
when
we
should
so.
B
Yeah,
we
could
do
something
like
that.
I
I
know
that
we
were
interested
in
collecting
pg
log
length,
so
I
I
need
to
go
through
and
see
what
were
what
we're
able
to
extract
from
pg
dump,
because
that's
where
we'll
need
to
look
most
likely
and
depending
on
what
we
can
extract
from
that,
we
can
include
some
more
helpful
data
that
could
be
processed
later.
E
Sure
one
of
the
things
too,
that
that
might
be
important,
too,
is
that
we
may
not
care
about
pg
log
length
until
we
do
right
like
if
it's.
If
it's
you
know
crazy,
so
it
might
be
something
where
we
can.
If
we
see
something
that's
really
outside
the
norm,
we
want
to
record
it,
but
otherwise
we
don't
care.
D
So
yeah
like
having
a
threshold
right
that
if
it's
above
threshold,
that's
when
we
start
capturing
those
but
yeah,
I
think
it'd
be
useful
exercise
to
go
over.
You
know
what
stats
are.
Are
there
in
the
or
pool
that
we
are
collecting
and
which
of
them
actually
don't
make
sense
on
a
purple
basis
and
need
to
have
a
separate
aggregation
or
a
pg
level?.
B
F
I
think
it
will
be
helpful
to
to
to
have
a
pilot
on
the
lrc
and
see
exactly
the
reports
that
we
can
have
from
this
cluster
and
have
it
evaluated
and
see
if
it
makes
sense
to
aggregate
some
of
them
or
have
it
separated
per
osds
or
rpgs.
And
and
if
this
data
is
actually
useful.
Because
if
we
have
everything
aggregated
and
we
cannot
derive
real
problems
that
we
see
in
the
cluster,
then
it
might
be
useless.
D
So
yeah,
I
guess
before
we
answer
these
specific
questions.
My
general
question
is
you
know?
What
do
we
want
to
infer
like
data
can
be
available
for
a
minute
and
data
can
be
available
for
hours.
So
I
guess
we
only
care
for
long-term
data
unavailability.
If
there
is.
You
know
that
that's
the
pattern
in
my
mind
that
I
think
will
be
more
useful,
but
I'm
curious
to
know
what
other
ideas
folks
have
around
data
and
availability
and
what
that
could
tell
us
from
from
capturing
the
this
kind
of
data.
G
E
No,
no,
it
was
a
very
kind
of
high-level
statement,
basically
just
that
customers,
I
think,
wanted
some
kind
of
indication
very
generically
of
when
data
was
unavailable
and
for
how
long.
But
I
I
don't
think
that
there
were
necessarily
any
specific
requirements
around
that.
Just
I
think
the
thought
right
now
is
more
or
less
that
something
rather
than
is
better
than
nothing.
So
I
think
that's
up
to
us.
A
Yeah,
you
could
think
of
it
like
a
looking
at
like
a
stuff
cluster
sla
like
what
what
is
the
availability,
that
a
cluster
is
providing
over
time.
So
what
fraction
of
the
time
is
it
unavailable?
Is
it
like
one,
nine,
two
nines,
three
nines
and
nines,
probably
not
that
many
lines,
but
that
depends
a
lot
on
the
hardware
in
the
setup
right.
So
it
will
be
interesting
to
see.
E
D
B
B
D
Yeah,
so
I
think
that
that'll
be
a
good,
you
know
start,
I
believe
so
how
like
I,
I
I
think
we
have
the
total
number
of
pgs,
if
I
remember
correctly
in
the
in
the
telemetry
reports,
but
what
states
they
are
in
is
not
there,
so
maybe
just
have
that
to
start
with,
and
then
we
you
know
as
josh
was
talking
about
like
over
time.
We
see
how
you
know
what
what
the
distribution
looks
like,
how
long
pgs
are
not
all
active,
clean
or
like
in
the
desired
state.
A
D
A
H
A
H
A
B
C
I
was
just
thinking
that
even
independent
of
telemetry
one
thing
that
the
stuff
clustered,
it
always
shows
you
a
snapshot
of
what
the
availability
is
at
a
particular
moment,
but
there's
no
like
summarizing
over
time.
It
would
be
interesting
if
the
cluster
could
track
its
own,
like
availability
over
the
last
month
or
month
or
year
or
whatever
it
is.
A
F
Yeah
and
then
we
can
have
this
information
sent
in
the
report.
So
that's
what
I
was
curious
about,
whether
if
we
just
have
a
snapshot
is
it
doesn't
tell
us
enough
about
what
happened
in
the
meanwhile
in
the
cluster,
so
it'll
be
good
to
have
this
on
the
cluster
and
then
just
to
have
this
output
sent
through
the
monetary.
E
F
E
B
And
I'm
liking
that
this
is
a
concrete
thing
that
we
can
metric
that
we
can
take,
because
I
I
wasn't
sure
up
until
this
point
really
what
metric
we
would
need
to
grab
or
even
if
we
would,
we
would
need
to
add
a
new
counter
or
anything.
So
it
seems
like
we're
already.
We've
already
got
this
metric
and
we
can
extract
it
and
report
it.
A
In
terms
of
the
time
aspects
like
the
windowing,
that
sampling
is
certainly
much
better
than
nothing
like,
we
could
also
add
something
like
a
histogram
over
time
for
like
which,
which
dates
pgs
we're
in
over
the
course
of
a
day.
For
example,
like
this,
many
videos
were
in
this
state
might
have
been
clean
for
this
much
time.
E
B
E
B
B
I
No,
it
was
just
a
china
thing.
Paul
kushner
is
now
working
on
on
exposing
the
tech,
health
checks
via
prometheus.
So
this
will.
They
will
be
now
available,
not
sure
if
that
might
be
also
interesting
to
have
those
events
or
and-
and
I
don't
even
know
how
to
do
that
in
the
telemetry.
If
that
even
happens
that
the
the
pr
is
this
one,
so
maybe
it's
worth
it
just
to
see.
We
also
trigger
alerts
for
from
that
so
yeah.
I
A
B
B
D
Yeah
yeah,
so
I
guess
I'm
a
little
skeptical
about
slow
ups.
Slow
ups
can
be
for
a
lot
of
reasons,
and
I
I
I
think
a
lot
of
the
information
that
we
are
collecting
will
probably
not
be
enough
to
even
infer
the
reason
for
slops
for
certain
reasons
for
certain
things
it
can
be
like
you
know,
let's
say,
there's
slow
ups
because
of
a
network
issue.
We
have
indicators
of
slow
ping
times
between
ost
etcetera,
that
we
capture
it's.
D
It
is
part
of
a
separate
command
and
it
is
also
part
of
ceph
status.
So
there
we
can
infer
things,
but
in
certain
cases
it's
just
things
are
slow
momentarily
for,
for
you
know,
just
backlog
or
or
just
the
system
being
slow.
Those
kind
of
things,
I'm
not
very
sure,
slow
ops
are
going
to
be
really
useful
to
capture
there.
B
D
Think
yeah
that
let's
consider
that
as
phase
space
do
or
or
something
because
that
wouldn't
need
us
to
at
least
have
these
bare
bones
burst
right
if
pgs
are
not
available-
and
we
know
that
from
the
pg
states-
it's
it
is
expected
to
have
slow
ups
right
so
that
that
time,
slow
ops
is
not
going
to
be
very
useful.
But
if
the
pgs
are
are
active,
clean
and
still
there
are
slow
ops.
Those
are
the
cases
where
you
know
there
could
be
a
network
thing
going
on
or
just
like
a
background.
D
H
So
I
I
think
this
is
a
very
useful
metric
or
or
just
you
know,
just
just
recording
that
the
slow
ops
warnings
have
been
there
for
this
much
time
is,
I
think,
very
important,
because
when
someone
complains
about
again,
you
know
my
my
workload
just
was
stuck
for
for
a
while
and
now
it
kind
of
resolved
itself.
We
want
to
know
you
know,
and
we
want
to
see
that
the
time
and
maybe
the
number
and
see
which
which
pools
were
affected
and
things
like
that.
A
Yeah,
I
think
we
have
some
thoughts
about.
I
think
I
was
talking
about
some
different
ways.
We
could
try
to
build
in
work
analysis
capabilities
to
try
to
figure
out
where
those
are
coming
from.
I
think
that
that's
maybe
a
larger
project,
and
we
do
have
some
data,
though,
with
the
channel
coming
in
about
latencies
and
histograms
of
latency
over
time,
so
that
that'll
also
give
us
a
view
of
like
if
there
are
there's
tons
of
slowness,
we'll
see
that
it's
from
the
perf
channel,
regardless
of
the
slow
openings
or
not.
B
Yeah
and
I
think
yeah,
I
think
that's
a
good
point-
that
we
can
use
the
data
that
work.
The
other
kinds
of
data
that
we're
getting
to
diagnose
the
slop,
because
the
sense
like
that
is
that
it
could
be
caused
by
the
slop
itself
just
implies
slowness,
but
that
slowness
could
be
caused
by
any
number
of
things.
So
it
might
be
better
to
know
those
any
number
of
things
rather
than
just
the
slow
up.
But.
D
B
That's
a
good
point
about
collecting.
Both
too
I
mean
if,
if
slow
ups
are
indeed
accessible
or
available
to
collect
by
telemetry,
I
would
have
to
look
into
how
we
could
collect
that,
but
I
wouldn't
be
opposed
to
collecting
both
and
then
you
know,
we've
discussed
it.
So
it's
always
better
to
have
a
little
bit,
not
too
much
information,
but
a
little
bit
more
information
than
a
little
too
little
information.
So
you
know
it
could
still
be
helpful.
A
B
F
F
C
C
B
C
Reason
to
necessarily
duplicate
it,
but
I
mean
that
brings
up
the
other
point,
though
I
wonder
if
it
would
make
sense
to
just
briefly
summarize
what
we
came
up
with.
I
guess
a
couple
weeks
ago
in
the
telemetry
call
about
how
to
how
to
version
the
the
channels
to
deal
with
that,
so
that
people
don't
have
new
things
start
getting
sent
that
they
haven't
opted
into,
but
also
they
aren't
sort
of
forced
to
turn
things
off.
If
something
new
comes
along
that
they
don't
like.
C
F
C
Think
what
we
decided
is
yeah.
We
went
around
in
circles
on
that
a
bit
I
think,
and
basically,
instead
of
having
a
channel
be
on
or
off.
If
it's
on
it's
on
at
a
particular
version
and
initially
we
said
releases
because
normally
drill
this
up,
but
we,
if
we
just
make
it
a
version
string,
then
that
would
also
let
us
backboard
things
if
we
wanted
to
and
then
you
could
opt
opt-in.
C
But
basically,
if
you,
when
you
enable
telemetry
on
a
channel
or
enable
a
channel,
you
can
either
just
turn
it
on,
in
which
case
little
by
default,
opt-in
to
whatever
the
current
version
is
or
you
can
opt
into
a
particular
version
of
whatever
that
channel
is
so.
You
could
opt
into
what
was
being
reported
in
octopus
instead
of
in
pacific,
for
example,
and
then
there
might
also
be
an
option
where
you
can
opt
in
to
like
any
future
version
too.
C
Although
I'm
not
sure
that
anybody
would,
it
would
be
wise
to
suggest
that
people
do
that,
and
then
I
guess
that
the
missing
piece
is
that
we
would
need
a
programmatic
way
to
like
a
cli
command
or
whatever
it
is.
That
would
dump
like
a
nice
summary
of
exactly
what
is
added
in
any
particular
channel
and
version.
C
So
you
would
see
that
in
the
basic
channel
in
16.2
we
added
this
field
and
so
it'd
be
like
a
json
thing
that
would
dump
a
bunch
of
strings
that
would
like
just
describe
what
it
is.
That
is
added
so
that
when
you're
presenting
a
dialogue
or
when
you're
asking
people
to
that,
there's
new
challenges
that
we'd
like
to
report.
Do
you
want
to
accept
or
reject
it?
Then
it
will
actually
show
you
what
what
it
is
that
you're
that
we're
asking
to
collect?
C
C
C
B
C
G
B
C
For
a
smart
sport,
okay,
so
here's
the
pad
here
so
basically
I
just
want
to
give
a
sort
of
a
status
update
on
where
things
stand.
All
the
notes
from
our
previous
discussion
that
we
had.
We
had
a
a
call
with
to
leave,
maybe
like
a
month
ago,
where
we
talked
about,
we
were,
it
was
in
the
performance
school.
I
think
where
we
talked
about
the
the
current
code,
that's
in
the
tree
and
what
was
what
should
we
change?
C
So
I've
been
working
on
a
pull
request
that
makes
most
of
these
changes
and
that's
basically
ready
to
go.
It
has
a
bunch
of
patches
on
top
that
address
the
most
recent
review
from
igor,
so
those
aren't
squashed
yet,
but
as
soon
as
he
takes
another
look,
then
I'll
squash
this
down
and
I
think
he'll
be
ready
to
merge.
C
So
the
basically
the
the
current
status
as
as
of
all
of
that
is
that
the
all
of
the
internal
metadata
tracking
is
implemented
so
that
we
can
and
a
simple
cleaner
is
implemented,
and
so
we
can
actually
booster
will
successfully
run
on
an
smr
hard
drive
at
least
on
one
of
the
simulated
ones.
We
don't,
I
don't
think
we
have
actually
tried
plugging
in
an
actual
actual
one.
C
C
I
think
for
now,
for,
like
all
the
use
cases,
that
people
will
probably
use
these
drives
for
there's
going
to
be
enough
space
in
that
conventional
zone
for
it,
but
there's
lots
of
things
you
could
do
in
the
future
to
make
it
like
spill
over
better,
so
that,
if
you
had
a
lot
of
metadata
would
spill
into
the
initial
zones,
it
is
possible
to
put
rocks
to
be
on
top
of
those
that
part
of
the
drive
without
any
real
trouble,
but
yeah.
So
it's
basically
it
works.
C
You
can
run
the
seth
test,
object
or
object,
store
tool
and
it'll
drive
to
completion.
Some
of
the
tests
get
skipped
because
the
tests
are
making
checking
like
statistics
and
other
specific
bugs,
and
so
on
that,
like
don't
really
make
sense
with
the
smr
drives,
so
there's
some
tests
that
are
skipped,
but
otherwise
it
looks
like
it's
in
pretty
decent
shape,
but
it's
totally
none
of
the
like
policy,
stuff
or
tuning
is
particularly
smart.
C
So,
as
far
as
what
the
next
steps
would
look
like,
I
think
the
first
one
is
figuring
out
how
we
want
to
build
right.
Now
you
have
to
link
against
this
lib
zvd
library
in
order
to
use
the
smr
drives.
There's
a
copper
repo
that
kifu
created
that
I
was
using
for
my
testing.
There
are
currently
no
debian
packages.
C
I
think
probably
what
we
want
to
do
is
just
either
build
some
of
these
ourselves
or
go
poke
somebody
and
get
those
into
whatever
distro
so
that
we
can
just
build.
Have
the
standard
bills
that
we
do
link
against
this
package.
We
could
actually
even
just
not
looking
against
it
in
the
debians
that
might
even
be
okay,
because
the
container
image
uses
is
based
on
centos
and
so
as
long
as
the
support's
there
and
most
people
are
using
that.
C
Then
we
capture
most
users,
at
least
for
the
short
term,
but
once
we
do
that,
then
we
can
have
pathology
tests
that
exercise
this
code
by
like
running
the
stuff
test,
object,
store
tests
against
the
simulated
device,
for
example,
and
so
on.
Then
people
can
actually
test
deploying
it
on
real
hardware
and
I
think.
C
We
have
all
the
pieces
we
need,
I
think,
the
copper
repo
that
keeper
has
a
sufficient
for
that.
We
just
need
to
change
the
container
image
build
for
for
master
for
quincy.
So
I
think-
that's
that's
probably
my
suggestion,
but
I
wanted
to
get
feedback
on
whether
that
makes
sense
to
add
that
dependency.
C
So
it's
easy
to
get
the
lib
zbd
installed
in
our
container
image,
because
we
we
have
the
package
and
we
can
just
add
it
in,
but
it's
a
compile
time
option
that
compiles
all
the
stuff
in.
So
if
the
library's
present,
then
it
enables
all
this
extra
code,
which
means
that
it
also
needs
to
be
part
of
the
like
standard
rpm
build
process,
which
means
that
it
has
to
be
installable
for
everybody
who
downloads
like
the
centos
rpms.
C
C
A
C
Okay,
so
probably
need
to
have
a
conversation
with
can
or
keep
or
somebody
to
see
what
what
makes
more
sense,
but
I'm
kind
of
I'm
a
little
bit.
I'm
inclined
to
do
this
sooner
rather
than
later,
because
then
we
can
start
doing
the
testing
and
we
can
do
it
just
on
master.
We
have
to
worry
about
all
the
old
gestures,
because
this
is
all
new
code.
C
Yes,
that
is
that
is
my
understanding
is
that
is
the
way
forward.
There
is
some.
There
is
a
kernel
driver
that
allows
you
to
interact
with
smart
drives
like
via
some
some
weird,
like
fabricated
file
system,
where
each
each
zone
becomes
a
file
that
you
can
append
only
to,
and
then
you
delete
whatever
there's
like.
C
A
C
J
C
B
C
Target
okay,
so
that's
that's
the
first
thing.
The
second
thing
is
the
cleaner,
so
the
cleaner,
that's
currently
implemented
is
like
the
simplest,
cleaner.
Basically
that
you
could
implement
so
it
just
sort
of
right
now
in
the
tree.
It
wakes
up
like
very
aggressively
just
so
we
exercise
the
code,
but
it
that's
not
actually
what
you
wanted
to
do
so,
there's
a
lot
of
policy
around
deciding
when
to
clean.
That
needs
to
be
tuned
and
it
doesn't.
It's
not
super
efficient
like
it'll.
C
Take
one
object:
it'll
read
the
whole
object,
the
parts
of
it
that
are
in
the
zone
you're
trying
to
clean
and
I'll
rewrite
them
somewhere
else,
but
it
doesn't.
It
doesn't
deal
with
clones
very
well.
So
if
you
have
a
cloned
object,
it'll
basically
rewrite
that
data
twice
for
both
objects.
It'll
read,
read
it
and
rewrite
it
separately,
so
it's
not
suitable
for,
like
any
greater
schools
that
have
snapshots.
For
example,
it
doesn't
like
think
about
compression
or
trying
to
preserve
any
of
those
things
it
just
sort
of
blindly
rereads
and
writes.
C
C
The
rewrite
behavior
is
efficient,
if
you
add
all
the
tuning
and
when
to
wake
up
and
throttling
and
stuff
like
that,
but
yeah.
I
think
this
is.
This
is
the
primary
thing.
This
is
the
primary
area
where
I
think
to
leave
business
students
want
to
focus
is
like
all
the
policy
and
testing
and
performance
discussion
whatever
around
when
when
and
how
it
should
behave
and
less
about
like
the
the
blue
store
implementation.
Details
of
you
know
how
metadata
is
modified
and
so
on.
C
Yeah
there's
there's
like
one
other
thing
here:
there's
like
one
lookup
method,
that's
needed
in
blue
store
so
that
we
can
go
from
an
object
to
a
handle
to
the
collection,
which
is
something
that
you
normally
never
have
to
do
in
blue
store,
because
then
the
osd
are
going.
The
other
way
around,
but
current
implementation,
just
like
enumerates,
tries
every
collection
to
see
which
one
matches
the
object.
So
there's
I
need
to
fix
that
still,
but
that's
a
relatively
small
piece
then
there's
the
conventional
zones.
C
How
do
we
want
to
use
them
right
now?
They're,
just
given
over
to
bluefs
entirely
and
then
blue
store
data
uses
the
sequential
zones.
It
is
possible
for
bluefs
to
use
the
sequential
zones
like
the
sst
files
are
always
written,
sequentially
and
red
ones
and
like
actually
they're,
very
smart
friendly.
So
that
would
be
pretty
easy
thing
to
do
and
it
would
even
be
possible
for
blue
fs's,
like
internal
logs
or
the
the
rocks
to
be
log
file
to
be
written
on
sequential
zones
if
you
really
wanted
to.
C
C
Cheaper
yeah
actually
right
yep
and
it
is
doable,
but
it
is
sort
of
a
tedious
weirdness.
I
think,
for
these
smr
drives.
It
doesn't
matter
because
there's
always
going
to
be
that
conventional
zone
and
there's
always
going
to
be
plenty
of
room
there
for
those
rocks
to
be
log
files,
even
if
that
were
the
only
thing
that
we're
going
there.
C
C
So
right
now
we
have
ssd
and
hdd
are
the
two
automatic
classes.
Basically,
but
we
could.
My
suggestion
would
be
to
have
an
ssd,
an
hdd
and
an
smr
hdd
or
just
smr,
or
something
like
that,
or
these
smart
hard
drives
so
that
you
could
have
you
know,
crush
rules
that
take
that
inside
by
using
just
the
smr
drives
for
the
bulk
storage
pool
or
or
even
just
so,
there's
some
visibility
when
you're,
like
dumping
posties
or
whatever.
K
C
Yeah
I
just
was
also
just
sudoing
before
I
ran
the
test
yeah.
It
was
annoying.
That's
a
good
question,
I'm
not.
We
also
that
might
affect
also
the
osd
whether
it
can
like
when
you're
resetting
the
zone
pointers.
I
wonder,
I'm
not
sure
if
you
have
to
be
super
user
or
not,
so
the
osd
might
not
be
able
to
drop
privileges
as
well
yeah,
I'm
not
sure
about
that.
G
G
G
It
suggests
a
simple
service
monitoring
system
which
collects
a
matrix
at
a
target
at
given
interval
which
can
be
configured
and
it
has
it
its
old
rule,
expression
and
query
language
which
we
can,
which
we
can
trigger
and
get
the
exact
result
based
on
based
around
mattresses,
which
are
exposed
to
exposed
at
certain
points
like
exposed
point
to
concept
yeah.
So
another
thing
is
important
about
prometheus.
Is
it
basically
collect
metrics
via
a
pull
model
where
you
expose
your
metrics
at
http
web
server,.
G
So
next
one
what
what
we
need
to
do
to
to
be
consumed
by
whatever
metrics
we
expose
to
be
consumed
by
prometheus.
So
what
we
need
to
do.
First,
we
need
to
add
instrumentation
to
our
code,
whatever
code
like
whatever
metrics
we
want
to
expose,
we
we
need
to
instrument
them
and
these
we
can
do
via
prometheus
client
libraries.
G
So
it's
a
completely
related
to
rbd
and
our
code
base
agency.
So
we
need
to
look
at
prometheus
client
library
which
matches
c
plus
plus
language,
but
unfortunately
there
is
no
simpler,
plus
client
library
official,
but
basically
how
this
client
library
will
help
to
help
us
in
exposing
internal
mattresses.
G
So
what
this
c,
plus
plus
promethia
cpp
library
does
it
it
is.
Actually
it
is
very
simple
and
a
small
library
which
is
completely
written
in
c,
plus,
plus
and
internally.
It
uses
a
civic
wave
to
as
a
http
wave
service
web
server,
and
it
has
three
main
component:
one
is
core:
another
one
is
pull
and
push,
so
it
is
complete.
G
Module
is
divided
into
three
parts,
so
basically
core
is
a
more
responsible
and
where
all
the
all
the
instruments
instrumentation
and
how
to
expose
metrics
and
each
and
everything
is
code.
Is
there
and
pull
model
where
we
can
use
this
library
wherever
you
want
to
pull
mattresses
from
a
particular
endpoint,
such
as
in
our
case
prometheus?
So
that's
why
we
will
be
only
interested
in
these
two.
These
two
components.
G
So
core
component
has
two
main
component:
one
is
registry
manager
which
actually
has
tools
and
instrumentation
to
implement
a
global
registry
where
we
can
capture
and
put
all
the
mattresses
and
another
one
is
matrix
in
point
where
it
provides
interface
to
basically
implement
and
expose
those
matrixes
to
a
particular
defined
target
where,
by
default
target
for
prometheus
to
pull,
it
is
slash
matrices,
so
it
does
the
same
it
actually
by
default,
exposes
all
the
matrices
and
bind
these
to
a
particular
endpoint.
That
is
less
mattresses.
G
So
in
this
experiment,
actually,
as
we
can
see,
this
complete
data
pipeline
is
divided
into
two
parts.
One
is
producer
and
another
one
is
consumer
side,
so
consumer
side
basically
starts
with
controller
and
controller,
actually
calls
the
export
and
it
sends
the
matrix.
Data
to
exporter
and
a
consumer
side
begins
with
prometheus
pull
request
and
whenever
prometheus
server
sends
a
pull
request
to
exposed
http
point,
it
actually
defines
and
gets
a
collectible
interface,
and
it
has
a
registry
inside
where
we
register
for
a
particular
endpoint
and
it
fetches
data
from
there.
G
You
can
say
a
producer
side
where
collector
is
collecting.
All
the
data
and
exporter
will
and
sorry
and
collector
will
export
this
data
to
exporter
and
exporter
has
a
job
to
basically
expose
expose,
http,
endpoint
and
and
give
the
data
to
collector,
where
collectors
will
have
only
one
duty
to
to
gather
all
these
data
and
wait
for
prometheus
pull
request
to
fetch
data.
G
G
I
Hey
sunny,
yeah
well
looks,
looks
really
nice.
I
I
would
like
to
ask
question
about:
what's
the
plan
for
high
availability
here,
because,
for
example,
right
now
in
with
the
promise
exporter
stats
attached
to
the
manager,
we
currently
have
three
well
or
in
some
scenarios
we
can
deploy,
for
example,
three
managers,
so
we
have
kind
of
a
replica
three
or
three
possible
instances.
So
what's
the
model
or
high
availability
model
for
this.
G
I
Okay-
and
I
have
also
questions
about
security-
and
I
mean-
and
that's
what
we're
also
dealing
with
with
sdp
and
well
probably
as
rdw
fox
no
to
that's.
Well,
quite
an
issue
right,
so
are
you
planning
to
support
tls
just
for
the
encryption
of
the
endpoint
and
also
is
there
any?
I
mean
security
measures
to
control,
because
that
this
basically
might
turn
the
diamond
or
its
demo.
I
G
Yeah
sure
this
is
a
really
good
suggestion
and
I
haven't
looked
at
the
security
part,
but
it's
in
very
initial
phase,
so
we
are
gathering
all
the
things
all
the
feedback,
and
we
will
just
go
through
this,
though
so
really
thank
you
for
this
one.
I
haven't
considered
it
but
yeah.
Definitely
we
have
to
do
it.
L
H
M
H
H
H
H
L
H
H
You
know
write
up,
write
up
some
documentation
as
far
as
what
is
exposed.
Perhaps
you
know
something
that
would
closely
resemble
a
schema,
but
in
in
in
general
the
the
pooling
side
of
it
is
going
to
be
outside
of
ceph.
I
I
think
that
the
existing
prometheus
module
in
the
manager
probably
stay
for
a
while,
because
not
everything.
H
B
M
Well,
it's
just
just
to
be
just
answering
casey's
question
directly
minor.
I've
heard
this
described
in
a
couple
couple
of
times
by
was
some
people
working
in
the
storage
management,
team
and
yeah.
My
understanding
from
that
is
that
this,
the
strategy
of
prometheus
is
that
it
can
scale
up
into
multiple
and
multiple
daemons
that
can
do
this
collecting
work
in
parallel.
H
L
H
L
H
It
but
yeah
so
if
rgw
is
structured
around
pushing
this,
this
sort
of
data,
then
yes
and
probably
need
to
be
restructured,
but
I
think
for
like
in
general,
even
even
for
like
even
if
you're
you're,
even
if
you're
pushing
today.
Turning
that
into
a
pool
doesn't
seem
like
it's
gonna
be
hard
because
you
could
just
like.
H
H
Yes,
sebastian
was,
is
it
was
actually
involved
in
in
these
discussions?
I
I
don't
have
the
specifics,
except
that
what
I
seem
to
remember
is
that
it
is
actually
much
easier
to
do
this
in
a
kubernetes
environment
than
it
is
outside
of
kubernetes,
and
so
the
problem
is
actually
you
know
raw
sfadm
installs.
The
kubernetes
is,
is
you
know,
taken
care
of
almost
automatically.
H
Because,
again,
this
is
how
the
rest
of
kubernetes
works.
So
it's
not
an
issue
there.
It
is
an
issue
for
non-kubernetes
installs,
because
you're
right,
we
need
to
like.
Suddenly
you
need
to
get
that
you
know
have
a
way
to
get
the
list
of
endpoints
and
you
know
somehow
get
that
to
prometheus
or
prometheus
needs
to.
H
H
L
M
Right,
sorry,
yeah.
I
agree
with
that.
That
sounds
like
a
very
simple
thing.
I
mean
one
of
the
things
we
I
don't
know
this.
This
isn't
being
addressed,
but
one
of
the
things
we've
been
hoping
to
see
from
this
is
additional
types
of
counters,
and
it's
not
really
the
same
thing
but
but
it
seems
like
it
relates
in
terms
of
the
work
that
gets
done.
M
I
don't
think
it
makes
sense
to
try
to
change
how
manager
understands
counters
if
we're
going
to
have
another
system.
At
any
rate,
there
are
a
lot
of
users
of
of
hw
have
asked
for
things
like
yeah,
sparse
counters.
Essentially,
you
know
the
number
of
views
that
number
most
active,
buckets
or
as
well
as
most
active
users
are
on
and
quarterly
stuff,
and
things
like
that
are
things
that
people
have
asked
for.
H
Yeah,
I
think
they're
called
labels
and
yeah
it's
it's.
I
mean
it's
fairly
expressive,
so
it's
it
doesn't
map
one
to
one
to
to
self's
performance
counters,
and
that
is
actually
one
of
the
things
that,
by
eliminating
this
performance
counters
bridge
that
we
currently
have
where
we,
where
the
manager
module,
takes,
set,
perf
counters
as
input
and
then
outputs.
You
know
prometheus
http.
H
H
It
would
actually
be
more
expressive
because,
instead
of
going
via
the
intermediary,
you
could
directly
encode
exactly
what
you
want
with
the
labels
that
you
want.
You
know
these
are
like
key
value
pairs,
and
so
I
think,
as
far
as
expressiveness,
we
would
not
lose
anything,
but
rather
gain
quite
a
lot,
because
she
could
expose
histograms
and
other
fairly
advanced
things
there.
Prometheus
has
built-in
support
for
them.
G
Basically,
four
things:
a
matrices
type
like
counter
gauge,
histogram
and
summary,
and
it's
a
job
of
actually
this
one,
a
client
right
ready
to
implement
all
these
kind
of
mattresses
that
you
can
put
in
into
your
instrumentation
and
map
this.
So
so,
I
think
we
need
not
to
worry
about
too
much
about
actually
what
kind
of
type
of
mattresses
just
we
need
to
worry
about,
actually
how
we
are
going
to
map
on
the
required
matrix
that
we
want
to
expose
with
these.
H
A
H
Yeah,
I'm
just
I'm
just
like,
as
mentioned
in
the
comments.
Maybe
we
should
capture
the
discussion
in
the
chat,
but
both
myself
and
matt,
and
I
think
others
feel
that
bringing
in
bringing
back,
I
should
say,
civic
website
module
as
part
of
the
prometheus
cpp
library
which
sunny
has
been
experimenting
with,
is
not
undesirable,
and
even
though
it
is
not
going
to
be
used
to
an
extent
that
it
was
once
used
by
rgw,
which
is
you
know,
by
sharing
or
handling
actual
data.
H
This
is
just
some
text
that
would
that
we
don't
expect
any
performance
issues
with.
It
is
still
something
that,
if
possible,
we
would
like
to
avoid,
and
so
one
thing
that
was
one
idea
that
was
coded
around
in
the
past
was
using
a
boost,
beast
web
server
for
this,
but
there's
nothing
out
there
that
combines
the
you
know
the
primitives
endpoint
library,
with
with
beast.
H
And
have
a
look
at
what
what
it
does
and
maybe
that's
enough
for
our
purposes,
because
we
do
need
like
this
work
needs
to
land
sooner
rather
than
later,
so
we
one
we
the
reasons
that
we've
decided
to,
or
I
I
guess
that
we
went
with
exploring
this
existing
library
instead
of
just
writing.
Writing
something
from
scratch.
H
Even
if
it's
used
just
for
serving
up
this,
you
know
premieres
data,
it's
still
a
web
server,
it's
still
an
attack
vector
and
having
that
and
maintaining
that,
especially
if
we
end
up
just
cutting
pasting
from
from
open
tracing
or
some
other
project
is
going
to
be
a
problem,
because
no
one
is
going
to
update
that
so
converging
on
boosts,
beast
long
term
is
definitely
the
goal
on
how
to
get.
There
remains
to
be
a
question.
G
H
H
Yeah
and
it
looks
like
casey's
volunteering
to
get
us
some
help
so
yep.
Let's,
let's
focus
on
that
and
see
how
far
we
can
get
in
the
coming
weeks
there,
because
I
think
it's
if
it
can
be
done
in
the
in
the
short
term,
then
it
is
definitely
it
is
definitely
the
right
way
to
do
it
because
we
spent
a
long
time
getting
rid
of
david
webb
and
bringing
back
us
not
really
undesirable.
H
B
H
H
H
H
B
H
H
They
are
all
conceptually
similar
in
that
they
provide
a
disaster
recovery
mechanism
and
the
high
level
goal
is
to
it's
not
so
much
down
to
the
prometheus
metric
types,
but
you
know
deeper
into
like
the
individual
metrics
themselves.
You
know
so
that
things
like
how
far
is
is
your
mirroring
behind
so
is,
is.
B
H
M
Yes,
I
don't
want
to
open
up,
and
I
can't
wear
them
feels
like
that's
a
pretty
opinionated
statement
that
that,
and
it
seems
like
it,
leads
in
the
direction
of
asserting
that
there's
that
the
replications
are
all
semantically
similar.
If
and
if
it's
except
they
actually
are
kind
of
similar.
It
makes
great
sense
to
to
to
try,
to
you
know,
have
common
common
com
sort,
some
common
idioms.
M
So
to
that
extent
I
you
know
I
would
agree
with
that,
but
I
wouldn't
want
to
at
the
overall.
You
know
that
the
overriding
goal
is
is
to
hide
the
differences
between
different
different
different
protocols
that
use
ceph
and
happen
to
be
replicating
because
we
might
well.
You
know
we
might
want
to
vary
the
semantics
even
more
than
we
already
are,
or
we
might
discover
that
there
already
is
enough
variance
that
we
care.
H
Yeah
matt
and
the
goal
is
definitely
not
to
preclude.
I
mean
it
would
be
up
to
the
individual
demon
to
you
know,
send
whatever
metrics
it
it
deems.
You
know
necessary,
really,
really
the
the
short
to
medium
term
goal
here
is
because
rbd
and
suffice
mirroring
at
least
to
an
extent
that
that
they
are
going
to
be
used
by
kubernetes
are
both
based
on
snapshots
and
the
the
mechanisms
are
really
similar.