Ceph Ceph Day Boston 2014, 25 Jul 2014

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: RH InkTank Ceph Day Sessions Casey Bodley COHORTFS

Description

Ceph Day Boston 2014
http://www.inktank.com/cephdays/boston/

A

Hi everybody, my son, how's everyone doing. My name is Casey badly. I am at covert FS working on NFS on top of Steph.

A

So title is a scalable metadata over NFS.

A

So why do we like NFS? It's a IETF standard, so there's a lot of implementations out there. It's well established there's a client available on all the major platforms and it's extensible and it can adapt to future needs.

A

A

So there's many techniques for scaling data we've got rate and object, striping crush for placement. So what can NFS do to take advantage of those techniques?

A

It's parallel NFS. The NFS was introduced in an A+ 4.1 and it allows bffs clients to speak directly to storage devices.

A

So PFS has different layout types for black volume devices, OSD switch speak to 10, and the file layout is a generic layout that uses the NFS protocol. So you can basically run a NFS server on top of your storage device.

A

And that's what it looks like, instead of having a single PMF or single NFS server.

A

Thank you. You have one metadata server which serves layouts and metadata, and you have your array of storage devices. Data servers, so a PFS layout is state given by the MDS that describes how file is placed on the devices.

A

There's a layout get and layout return, and the MDS can recall it with layout, recall and conflicts or changes to placement and there's a consistency via layout commit. So you do your rights and commits to the individual data servers and then layout, commit to the MDS will update the size. The metadata and soapy NFS is optional.

A

If something isn't working, if you can't get your layout, if a right to a data server fails, you can always fall back to the MDS.

A

So what about metadata still have just the one MDS and their workloads that will stress that, like large directory shared by many clients, I'll trying to read it, workloads that create lots of files and operate on men on many threads and so systems like SEF, have techniques for scaling metadata, distributing it and load balancing, but NFS doesn't have a way to take advantage of that currently. So this talk is about P NFS meta stripe, which takes the ideas of P NFS and applies it to metadata.

A

So you can mount an initial metadata server and get layouts which direct you to the other metadata servers where your files and directories are located.

A

So we are working on an ietf draft, 4p, NFS, meta stripe. It defines to layout types. One is for the striping of files so finding out where an inode is located in your cluster and the other is directory striping. So finding where a directory entry is going to be.

A

So, as I said, the client mounts the initial MDS and it's directed to the others.

A

So the file handle striping layout is a single layout that covers the entire file system, so that differs from the pmf s case where you get a layout on a individual file and we use NFS file attributes to provide hints to the client about where the file is placed and that allows the client to talk directly to the MDS that is authoritative for that file.

A

So because it's based on hints, we don't have to recall this file handle layout on changes to placement so directory striping is the other, possibly more interesting one which describes how directory entries names are striped across the directory. So right now it's just simple hashes over stripes we'd like to add support for SF frag trees.

A

But this one is a layout / directory.

A

It's a recall, done changes to placement, and when you create a directory, you can give hints for how it should be striped how many files you expect to put in it, and so the operations that we can take advantage of in meta stripe are AP Reader, which allows the PMF s client to read all of the stripes in parallel. So.

A

The normal reader uses cookies to index into the directory, but those are all separate for each stripe now. So the clients in charge of enforcing the ordering to support, toddler and seek der other operations are directory modifications that we can do in parallel and we use layout commit to provide consistency for these. So we can send bulk, creates renames removes without having to update the inode some time and on layout commit. We restore the consistency.

A

So this gives us parallel operations, so the prototype that I've developed for meta stripe is based on the open source, NFS ganaches server available on github.

A

They have the existing support for staff and for the client I'm using the PI NFS test suite, which is a Python implementation of NFS, with a lot of great testing, so I developed a set of tests for all of the métis tribe operations, it's available on Linux NFS, and so this is what the prototype looks like each metadata server as an instance of ganesha and the ceph MDS, and the MDS talks to the the ceph components as usual, and it's driven by the PI NFS client.

A

So it changes required for lips ffs, which is the client library force F, that Ganesha links to provides callbacks and changes to the MDS map, so that Ganesh you can track where the metadata devices are and provide layouts on them, and it also gets callbacks on direct replacement for restriping events, so that I can update the directory, layouts and so changes in the set MDS to support this. Since the original draft does not support stuffs fragmentation, we replace Zeff, stir fragments with simple stripes.

A

We added / stripe blocks instead of idle. Oh I know blacks, so that we can operate on stripes in parallel and.

A

4M time updates, we do asynchronous updates and their batched.

A

A

And we use these cap updates for the striped blacks, instead of invalidates so that the client can maintain its cash instead of blowing out everything and changes.

A

So for the parent stats, which is the UM time the size, information and the recursive stats for quotas and stuff, are done, asynchronously they're, queued up and the md cash is pretty dirty journal. Parents function and their son in batches to the MDS. That's off for that I know'd so since they're asynchronous, it only provides week. Consistency for the directory em time layout commit is what's going to sink it, but that is incomplete at the moment in terms of caching.

A

By adding capabilities to the stripe, blacks, we allow the client to cache the directory entries for each stripe independently and they are not invalidated. When I note changes and for the the cap updates.

A

We send those synchronously on changes so that we update clients, for example, adding and removing files, since their last cap update, allows them to stay synchronous or consistent.

A

So future work looking to implement the layout, commit and get the M time, consistency right and add, support force, F frag trees, both in the ITF draft and to implement the SEF MDS. In terms of that, so we're going to leave QA until later, but that's all I've got. Thank you very much.