►
From YouTube: Ceph Developer Summit Quincy: CephFS
Description
00:00 - Quincy open issues
00:59 - Dashboard and FS/NFS
14:05 - mds_memory_target
18:19 - FScrypt integration
24:32 - async rmdir/mkdir and link/rename
30:55 - recursive unlink
42:14 - MDS rolling
50:00 - CephFS mirroring metrics (in line with RBD mirroring metrics)
B
B
B
This
is
how
it
looks
like
I'm
running
a
via
start
cluster,
so
some
things
would
be
a
slightly
different
compared
to
as
fading
one,
but
what
I
think
basically
will
work
for
for
the
purposes
of
this
demo,
so
well,
basically
in
via
star
there's,
this
default
system.
So
currently
this
is
what
you
get
in
the
dashboard.
I'm
also
enabled
a
couple
of
mds,
so
we
have
an
active
standby
one.
B
That's
currently
the
info,
also
the
bulls
associated
with
this
file
system,
and
we
have
this
chart.
I
think
there
was
a
proposal
for
removing
this,
because
we
have
the
grafana
ones,
but
this
is
coming
from
the
information
in
in
the
manager.
So
well
it's
free
to
get
that.
So
we
may
leave
that,
apart
from
that,
we
have
the
clients
right
now.
There
is
no
client
connected
here,
the
ad
directory
listing.
B
So
we
now
have
are
able
to
see
a
list
of
directories
and
also
we
have
the
possibility
of
creating
snapshots,
so
the
snapshot
comes
with
with
a
pre-built
or
free
suggested
name
for
that,
based
on
the
date
and
for
the
directories.
We
also
have
the
ability
to
set
quotas,
so
basically
we
can
have
set
course
for
years
and
that's
mostly
it
regarding
the
operations
for
the
file
system.
B
There
is
this
dashboard
as
well
grafana
one
and
what
else
is
missing
here,
so
that
would
be
for
pure
ffs
if
we
go
to
the
nfs
which
we
didn't
demo
for
for
rtw,
so
we
may
use
this
also
for
demo
in
this
as
well.
This
is
how
it
looks
like
this
is
created
by
yesterday.
So
with
stevie
m.
It
will
be
yes
a
bit
different.
I
cured
this.
B
This
one
point
alf
also
has
been
recently
investigating.
It
seems
like
we're
able
to
create
nfs
v3,
though
I'm
not
sure
if,
from
the
dashboard
perspective,
the
nfs
with
remote
point
is
there,
but
I'm
not
sure
if
the
ganesha
diamond
we
haven't
tested
that
yet
is
fully
working
with
with
this
config
and
not
sure.
How
is
this
working
with
the
volume
manager
or
the
nfs
new
module
so
we'll
need
to
double
check
that.
So,
basically,
you
can
create
a
an
export
from
here.
B
D
B
I
think
it
probably
it
will
be
worth
it
to
have
a
follow-up
discussion
on
this
because
there
are
different.
Well,
you
know
upstream
downstream
discussions
around
this
topic,
so
at
least
that's
where
it's
possible
to
do
that,
we
can
simply
disable
that
hide
that
or
whatever.
So
it's
just
for
our
convenience.
E
Yeah
and
also,
I
would
probably
specifically
want
to
be
using
nfs41
or
greater
4-0,
actually
quite
a
different
protocol,
and
we,
you
know
it
may
be
fine
on
4-0,
but
it's
not
well
tested
and
one
thing
that
doesn't
give
you.
If
you
use
4-0
and
then
you
your
grace
periods,
have
to
last
the
full
length.
You
can't
lift
the
grace
period
early
if
the.
If
the
server
goes
down,
it
has
to
come
back
up,
so
you
don't
and
that
that
applies
to
any
you
know
serving
the
cluster
as
well.
C
In
that
cluster,
that
you
have,
I
think,
it's
user
defining
here,
yeah
well,
in
this
case
it's
usually
fine
yeah
yeah
yeah.
I
think
that
orchestrator
is
only
worked
with
before,
but
as
we
allow
also
the
user
defined
it
exports
until
now
at
this,
they
can
set
this
this
way,
but
well,
this
is
a
one
of
the.
The
debates
is
if
we
we
definitely
dropped
this
for
quincy
or
or
what
two
okay.
A
B
B
Okay,
I
will
resume
this
and
yeah
just
create
a
sample
there,
for
example,
so
yeah
this
form
will
create
this
direct
directory
in
surface,
so
we
can
just,
for
example,
we
may
drop
the
nfsp3
and
then
we
lose
the
tag
option.
We
put
another
name
for
the
export
just
to
and
let's
select
this
option
and
that's
it.
So
we
have
the
other
export
created
here
and
if
we
go
to
the
file
system,
the
directory
is,
is
there
so
there's
the
sample
directory
created
there?
B
Currently
we
don't
support
the
creation
of
files
here.
I
don't
really
know
the
reason
for
that,
but
well
we
may
start
adding
more
fine-grained
operations,
maybe
also
dealing
with
objects
and
are
yeah,
it's
something
that
we
haven't
considered
yet,
but
as
we
progress
towards
more
fine-grained
operations,
we
may
include
that
as
well.
Just
kind
of
some
way
of
checking
there's
some
objects
and
I
think
that's
all
of
it
not
sure
if
you
have
any
questions
about
this
is
any
thing
that
you
would
miss
here
from
the
cfs
or
nfs
perspective.
A
A
So
yeah
you've
already
got
the
the
you
know
being
able
to
snapshot
individual
directories
with
arbitrary
names.
That's
that's
great,
but
I
think
the
next
step.
There
would
also
be
the
scheduling
the
performance
metrics.
I
think
we're
I'm
trying
to.
I
can't
recall
like
where
exactly
mds
is
sending
those
metrics.
I
think
it's
in
the
manager
report
message
that
periodically
gets
sent
sent
out
that
hasn't
been
touched
since
john
spray
was
working
on
this.
A
G
G
G
We
use
a
dbus
mechanism
to
update
the
interfaces
rather
than
using
a
watcher
on
the
redis
url,
even
though
we
stick
the
url
in
a
random
object.
So
I'm
wondering
if
we
should
be
updating
we
we've
recently
updated.
You
know
to
use
this
ffs
sub
volume
module
rather
than
the
old
volume,
client
library,
and
we
have
interest
in
staying
up
to
date
with
what
what's
happening
here.
G
So
I
just
want
to
you
know.
I
don't
want
to
divert
the
conversation
about
the
dashboard.
The
ui
aspects
of
it
are
primarily
of
interest
to
us
from
a
read-only
cloud
administrator
perspective
rather
than
in
terms
of
updating
exports,
all
of
which
is,
you
know,
api
automation,
but
underneath
here
there's
a
different
api.
Clearly,
I've
played
with
the
dashboard
and
the
orchestrator
with
pacific
a
bit
with
seth
adam
and
with
and
they're
both
both
more
up-to-date
than
what
we're
doing
so.
G
G
And
the
other
aspect
of
this
is
you
can
deploy
ganesha
active
active
with
the
orchestrator
and
that's
complicated,
but
you
know
for
us:
if
we
lose
a
node
system
d
is
not
able
to
handle
that
migration
to
another
node
or
something
like
that
today
and
we
had
talked
about
kubernetes
and
one,
but
you
know
simpler
than
that
problem.
Right
now
is
just
the
export
maintenance
issue,
and
that
may
be
the
place
to
start.
A
Yeah
that
the
nfs
plug-in
right
now
also
does
all
the
orchestration
handling
for
setting
up
nfs
clusters.
But
I
think
there
is
interest
to
to
make
it
work
with
statically
defined
set
of
nfs
servers,
which
is
something
both
the
dashboard
for
a
legacy,
support
perspective
and
also
manila
is
interested
in,
but
that
work
hasn't
been
scoped
out.
Yet.
G
B
A
So
the
the
tracker
tickets
in
the
agenda,
unfortunately
this
this
pr-
is
the
the
pr
associated
with
this
take
its
atrophy
quite
a
bit.
It
means
revived,
but
it's
a
fairly
simple
vr
that
just
dynamically
modifies
the
mds
cache
memory
limit
in
response
to
the
total
memory
usage
of
the
mds,
which
would
be
defined
by
or
the
the
target
would
be
defined
by
the
this
new
mds
memory
target
variable
analogous
to
the
other
ones
in
the
lsd.
A
F
One
of
the
tv
items
also
frequency
for
stephanie
anna's
to
have
stuffed
idioms,
setting
these
limits
and
be
able
to
scale
them
automatically
to
the
size
of
the
node
or
whatever.
It
is
so
it'd
be
nice
if
it
was
using
a
consistent
setting
across
all
the
different,
even
types
but
very
nice
to
have
that's
all
yep.
D
E
F
It's
something
that
mark
built
for
bluestore
that
allows
you
to
use
that
it's.
If
you
have
a
pool
of
memory,
that's
consumed
by
multiple
caches,
you
can
set
various
policies
around
like
how
you
would
like
that
memory
to
be
used.
So
like
these
caches,
you
know
at
least
this
much
memory
and
then
above
that,
then
they
scale
at
this
ratio.
And
then,
if
it's
above
a
certain
point,
then
this
one
gets
all
of
it
or
whatever.
It
is.
You
know
sort
of
this
tiered.
F
D
F
A
D
E
At
least
in
the
near
term,
if
you
have
to
do
a
cap
revoke
that
you're,
probably
doing
memory
allocations
and
making
it
even
worse
right,
you
know
so
yeah
I
mean.
Is
this
really
a
bug
I
mean
per
se?
I
mean
it
almost
looks
like
it's
like.
If
we're
not
hitting
the
mbs
cash
memory
limit,
you
know
correctly,
I
guess
maybe
we
should
be
just
our
math
is
off.
F
D
A
E
Yeah
I
mean
I've
got
the
file
names
piece,
pretty
much
done
and
we've
got
dang
did,
though
I
guess
zhang
picked
it
up
that
started
it
and
then
finished
it
up.
It
did
the
alternate
name
feature
that
we
need,
so
we
need
to
be
able
to
give
entries
a
secondary
name
in
case
they're,
very
long,
because
we
can't
just
hash
or
encrypt
and
hash
them
at
that
point,
or
we
can't
just
encrypt
in
base64
and
code
them
at
that
one.
E
E
What
I'm
stuck
on
now
is
the
fact
that
the
mds
is
what
handles
truncates
for
the
content
encryption,
and
so
when
we
set
size
to
a
particular
length,
you
know
the
mds
will
come
back
and
truncate
the
thing
down
to
that
bite,
but
that
could
be
in
the
middle
of
a
crypto
block
and
at
that
point
we
can't
decrypt
the
tailbone
file.
So
we
want
to.
E
We
have
to
ensure
that
we
that
we
somehow
teach
the
nds
to
round
up
to
the
to
the
end
of
the
next
crypto
block,
as
we
do
this
that
you
know
stuff,
there's
a
couple.
Different
approaches
we
could
do
and
I
think
greg
is
a
great
farm-
is
going
to
help
out
with
some
of
this,
but
but
any
case
we've.
We
got
a
meeting
scheduled
for
later
today,
just
to
discuss
some.
You
know
where
to
go
with
this.
So
any
case.
E
E
For
instance,
it
will
just
write
through
to
the
you
know,
synchronously
through
to
the
to
the
nbn
or
to
the
server
whenever
it
needs
to
go
to
the
osd's.
Whenever
it
needs
to
go,
do
a
write
or
a
read.
If
we
do
it
right,
we
can't
in
that
situation
we
need
to
be
able
to
do
a
read,
modify
right
cycle
because
we
have
may
have
to
slurp
in
the
beginning
and
end
of
a
crypto
block
at
the
beginning
of
crypto
blocks
to
make
sure
that
we
can
handle
those.
E
So
we've
got
some
code
that
does
all
this
it's
pretty
complicated
and
I'm
hoping
I
can
clean
it
up
a
little
bit
before
we
it's
ready
for
merge.
So
I'm
hoping
that,
maybe
by
the
summer
we'll
have
something
that's
ready
to
go,
but
we
need
some
mbs
support
for
the
truncate,
but
but
yeah
the
file
name
portion
actually
looks
pretty
good.
Now,
it's
working
really
fairly
decently.
E
Luis
enriquez
says
that
he
was
hitting
a
problem
with
it,
but
where
you
occasionally,
but
he
and
he
was
trying
support
to
track
it
down.
But
I
remember
back
from
maybe
I
haven't
been
able
to
reproduce
the
problem
so
anyway,
that's
where
we
are.
I
don't
have
a
lot
to
discuss
here
with
it.
I'm
thinking
this
anyways.
A
Yeah,
I
think,
as
far
as
the
direction
we're
going
to
go
with
handling
truncate
right
now.
The
preferred
solution
is
having
two
sizes
profile.
One
is
the
the
actual
size
that
we're
already
using
and
the
others
the
a
size
that's
used
for
the
purposes
of
truncate,
so
the
mds
will
will
not
truncate
off
the
last
few
bytes
that
we
need
and
they'll.
Both
both
sizes
will
be
protected
by
the
same
locks
or
maybe
I'm
sorry.
I
have
that
reversed.
E
We
could
we
could
even
take
that
feel
that
the
the
you
know
the
encrypted
or
that
the
crypto
enabled
clients
use
and
encrypt
it
too.
So
we
could
cloak
the
the
full
size
of
the
file,
even
if
we
wanted
the
cache
there
is
that
you
need
to
when
you
do
a
crypto,
the
the
crypto
blocks
have
to
be
at
least
16
bytes.
So
you
know
we
would
be
consuming
an
extra
word
in
the
in
the
inode.
You
know
somewhere
to
do
this.
D
A
So,
for
also
for
those
who
aren't
aware
with
the
you
know
about
this
project,
we
didn't
really
introduce
it.
The
fs
crypt
is
the
is
in
the
is
in
the
kernel
tree
and
it's
a
library
for
it's,
a
generic
library
for
file
systems
to
use
to
encrypt
files
and
file
names
within
the
file
system
currently
supported
by
ext4.
I
think
it
was
originally
a
project
for
android
and
we're
now
trying
to
use
it
in
ceph.
E
E
E
D
A
D
A
Neither
neither
antar
nor
arm
dash
rf
actually
do
those
have
syncs
so
user
perspective.
It
would
be
an
instant
change
so
that
that
would
be
the
direction
we'd
like
to
go.
So
the
next
step
is
to
get
it
getting
armed
or
make
their
definitely
asynchronous,
and
then
link
and
rename
is
sort
of
a
nice
to
have
stretch
goal,
because
that's
those
are
operations
that
are
used
by
by
rsync,
for
example,
in
some
configurations.
A
E
E
So
we
have
to
make
sure
that
that
doesn't
happen,
because
then
that
rm
door
won't
work
if
there's
a
file
in
the
directory,
and
so
that's
the
hard
part,
and
that's
why
you
know
we
have
done
armed
or
to
be
a
synchronization
point,
but
we
you
know,
maybe
we
need
to
revisit
that
and
think
about
how
to
do
it.
I
I
don't
know
a
way
right
off
hand
to
make
that
simple.
I
haven't
really
figured
out
anything.
Don't.
A
We
already
have
that
issue,
though
like
if,
if
I
have
two
applications
that
are
unlinking
files
in
a
directory,
and
then
you
know,
finally,
all
the
at
least
from
the
client
perspective.
All
the
files
have
been
deleted,
then
one
of
them
does
an
armed
or
one
of
the
applications
says
an
arm
there,
isn't
that
armed
or
already
ordered
against
the
other
unlinks
in
flight.
E
E
I
don't
think
about
it.
I
don't
know
I
mean
it's
worth
experimenting
with
be
able
to
probably
please
maybe
draft
something
and
throw
it
together
and
see
if
we
can
make
it
work.
I
don't
know,
but
it's
a
little.
I
mean
it's
not
too
hard
to
do
even
to
fix
the
client.
To
do
this,
I
mean
we
there's
some
guard
rails
in
there.
Basically,
that
keep
you
from
proceeding
at
you
know,
barriers,
basically
that
keep
you
from
proceeding
until
all
the
dentures
have
you
know,
synchronously
been
deleted
from
the
mbs,
but
we
can.
E
We
could
remove
that
and
see
how
it
goes.
I
I
hit
problems
with
this.
You
know,
I
don't
think
it's
you
know,
but
you
know
again.
The
catch
is
that
when
you
issue
these
things,
you
know
the
unlinks
come
back
very
quickly,
and
you
know
the
the
calls
are
still
very
much
in
flight.
You
know
that
may
not
have
been
transmitted
yet.
E
So
it's
not
too
hard
to
imagine
that
the
socket
buffer
you
know
for
the
rn
dirt
process
that
you're
calling
ends
up
ordered
you
know
getting
the
mutex
for
the
socket
before
you
can
send
the
the
unlinked
you
know
gets
his
chance
to
send
it
yeah.
We
have
a
lot
of
competing
new
texas
stuff
too.
So
it's
hard
to.
I
don't
know
which
way
that
it's
gonna
fall
out
so
anyway.
I
when
I
did
some
experimenting
with
a
while
back.
I
hit
problems
where
you
know
director
is
not
empty.
E
D
A
A
A
All
right,
the
next
one
is
recursive
unlink,
which
is
kind
of
similar.
This
is
an
idea
to
just
add
an
actual
rpc
to
the
to
the
mds
that
does
a
recursive
on
link
of
a
subtree
one
of
the
main
target
uk
use
cases.
We
have
at
least
from
an
immediate
sense,
is
the
the
volumes
plug-in
be
able
to
recursively
delete
an
entire
sub-volume
without
having
the
manager
have
to
list
and
rm
the
entire
sub
volume
in
an
asynchronous
fashion.
A
It
could
just
one
off
shoot
off
the
rpc
off
to
the
mds
and
let
the
mds
chew
through
it
over
time
and
thank
you
is
planning
to
work
on
this
hopefully,
but
for
this
this
release,
I
think
it
actually
supporting.
It
will
be
a
little
weird
for
the
mds
or
require
some
changes,
because
the
directory
assumes
that,
like
every
directory,
that's
actually
in
the
stray
is
already
empty.
A
A
For
the
entire
recursive
subtree,
you
wouldn't
be
able
to
iterate
through
all
the
files
to
update
the
link
counts
until
you,
actually
until
the
mds
actually
performs
the
unlink
to
make
this
available
across
all
of
our
drivers
and
not
just
as
a
special
purpose
rpc
and
the
lips
ffs.
I
was
thinking
we
would
also
plumb
in
support
by
maybe
having
a
dot
trash
directory
that
you
can
move
hidden
in
both
the
views
and
in
the
kernel
client
that
would
be
translated
into
that
recursive
unlink
operation.
H
A
H
H
Unlimited
file,
you
have
a
file
in
in
slash
toolbar
and
it's
and
there's
a
hard
link
outside
to
one
of
those
files
and
you
recruit.
Then
you
do
an
issue
and
recursive
delete
on
foobar.
You
need
to
like
move
the
move.
The
file
up
or
something
out
like
like
you,
can't
just
sit
in
the
existing
purge
queue
and
then
go
away
because
something
else
might
be
pointing
at
it
yeah.
H
H
I
don't
remember
what
invariants
we
hold
around
around
the
link,
counts
and
and
their
locations,
because
likely
because,
like
the
vet,
like
the
current
back
trace,
might
on
on,
it
might
be
out
of
date,
and
so
we
like
look
at
it
and
say:
oh
it's
in
the
recursively
deleted
section.
We
can
just
throw
it
away,
but
actually
got
moved
out.
F
H
F
H
H
H
A
D
A
A
That
is
a
problem
right
now
the
stray
directory
can
grow
unbounded.
If,
if
there's
you
know,
you
have
a
huge
sub
tree
of
hard
linked
files
and
one
of
those
sub
trees
gets
deleted
and
if
the
mds
never
touches
the
the
remote
entries,
it'll
it'll
never
do
the
reintegration,
so
the
strategy
just
keeps
growing,
so
it
made
make
sense
to
just
have
the
mds
periodically
go
through
the
entire
file
system
so
that
it
can
do
that,
reintegration
if
it's
not
driven,
naturally
by
client,
io.
F
A
Topic
all
right
next
step
is
mds
rolling
upgrades,
so
this
has
been
a
kind
of
troublesome
part
of
cfs
for
a
while.
Now,
in
fact,
we
have
a
rather
complicated
upgrade
procedure
for
cfs
that
users
that
users
should
follow,
involving
pretty
much
turning
off
all
mdss,
except
the
one
active
on
a
file
system
and
reducing
max
mds
to
one
and
then
actually
doing
the
upgrades
of
all
the
mds
daemons.
A
Instep
ffs,
we
only
use
the
incompat
feature
set
in
the
compat
sets.
So
that's
that's
the
only
thing
I'm
talking
about
here
that
works
pretty
well,
it's
something
I
want
to
backport
specific.
I
promised
it
would
be
in
pacific
for
rook,
because
the
tearing
down
all
the
standbys
turning
them
off
is
is
a
rather
onerous
upgrade
procedure
for
the
orchestrators
to
have
to
deal
with,
and
this
gets
us
pretty
far
to
making
that
process
a
little
easier.
A
The
next
step
in
in
supporting
rolling
upgrades
is
to
make
it
so
the
mds
you
can
have
mixed
versions
of
the
mds's
mixed
active
mixed
version,
actives
on
a
file
system,
and
I
think
to
support
that.
It's
going
to
require
some
significant
changes
to
the
to
the
mds
map,
some
of
those
things
I've
laid
out
in
the
ticket
and
then
there's
been
some
efforts
to
version.
A
F
A
F
A
A
A
F
D
F
F
D
E
F
D
A
A
Maybe
add
that
to
the
caps
list
that
indicates,
you
know
what
what
kind
of
availability
zone
it's
associated
with
and
and
whatnot
and
then
you'd
be
able
to
block
lists
an
entire
set
of
auth
credentials
with
or
instances
that
are
using
those
auth
credentials.
Based
off
of
that
tag,
I
think
that
was,
I
don't
recall
the
exact
details,
but
there
was
discussions
on
the
mailing
list.
That
indicated
I
was
not
that
wouldn't
be
quite
workable.
A
In
particular,
you
need
to
be
able
to
on.
I
think
one
of
the
main
challenges
is:
you
need
to
be
able
to
unblock
list
new
instances
so
or
if
a
new
instance
comes
from
that
cluster,
that
it
should
not
be
block
listed,
so
you
need
to
perhaps
some
kind
of
a
generation
id
epic
for
for
how
you
block
lists
an
entire
group
based
off
of
a
tag,
and
then
I
think
it
development
discussions
kind
of
stalled
there.