Ceph Cephalocon Barcelona 2019, 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rook Deployed Scalable NFS Clusters Exporting CephFS - Patrick Donnelly & Jeff Layton, Red Hat, Inc.

Description

Rook Deployed Scalable NFS Clusters Exporting CephFS - Patrick Donnelly & Jeff Layton, Red Hat, Inc.

Rook was developed as a storage provider for Kubernetes to automatically deploy and attach storage to pods. Significant effort within Rook has been devoted to integrating the open-source storage platform Ceph with Kubernetes. Ceph is a distributed storage system in broad use today that presents unified file, block, and object interfaces to applications.

This talk will present completed work in the Ceph Nautilus release to dynamically create highly-available and scalable NFS server clusters that export the Ceph file system (CephFS) for use within Kubernetes or as a standalone appliance. CephFS provides applications with a friendly programmatic interface for creating shareable volumes. For each volume, Ceph and Rook cooperatively manage the details of dynamically deploying a cluster of NFS-Ganesha pods with minimal operator or user involvement.

About Jeff Layton
Red Hat
Principal Software Engineer
Raleigh, NC
Jeff Layton is a long time Linux kernel developer specializing in
network file systems. He has made significant contributions to the
kernel's NFS client and server, the CIFS client and the kernel's VFS
layer. Recently, he has taken an interest in Ceph, in particular as a
backend for other network storage protocols.

About Patrick Donnelly
Red Hat, Inc.
Senior Software Engineer
Mountain View, CA
Patrick Donnelly is a senior software engineer at Red Hat, Inc. currently leading the global development team working on the open-source Ceph distributed file system. Patrick has been a speaker at several events presenting recent work on Ceph, including Cephalocon APAC, various Openstack Summits, CERN, and Vault Linux Storage & Filesystems Conference. In 2016 he completed his Ph.D. in computer science at the University of Notre Dame with a dissertation on the topic of file transfer management in active storage cluster file systems.

A

Okay, let's get started everyone, hello, I'm, Patrick, Donnelly, work at Red, Hat, the SEPA, fest tech lead. This is Jeff. Lee he's did a lot of the work on this project. He's got a you want to give your background. What.

B

Been a Red Hat for quite some time and work on storage of all sorts.

A

A

All right so we're going to talk about deploying NFS clusters using rope to exports ffs to give a little introductory information on sefa fest for those who aren't already familiar with it. Sefa fest is one of the first use cases, first, F, actually Lawrence Livermore, when they were first giving the grant to Sage what they really wanted, a file system and what they ended up. Getting was an object store so sympathize with the first thing that we're going to build on top of the object store that provides a POSIX distributed file system.

A

Some of the more notable features of it is that the MDS is form a cluster and you have all the metadata stored separately and that the the clients have direct access to all all the file data to do reads and writes they don't have to talk to them. The essence to do those operations.

A

So the clients and the MDS also, it's also notable that they cooperatively, maintain a distributed.

A

Cache of the metadata, including I, knows, and the directories the MDS hands out, capabilities which are similar to leases and other systems that give the clients permission to actually do, writes or reads on a given file, and that's how you don't have any kind of inconsistencies, especially with caches, where one client will be trying to write to a file and maybe but for those rights, while another clients trying to read them and they disagree on on the state of the file and because of the way the system is set up.

A

That way, you have a certain degree of trust with the clients and how they behave that you, you were trusting them to to obey what capabilities you've handed out to them and they're not going to write to a file when they're not supposed to.

A

We built two clients up to now or not up to now, but the two primary clients that are in use today are the step step views and the kernel client.

A

The set views. Clients typically are test bed for new features. It's also used sometimes for things like quotas prior to mimic the kernel clients where you're going to get all your performance.

A

Yeah, so why on earth do we want to put an NFS gateway in front of us ffs? So some of the issues with having these trusted clients and well behaving client and clients that we assume are going to behave well? Is that not all these clients may be able to speak stuff properly, especially with the latest versions of Ceph? They might have a really old kernel that may be known to have bugs it's, not gonna work behave properly or it's not or the the clients running on an operating system that is running as effusive.

A

You know we don't know anything about. This may have some modifications. Another aspect is security. There might be a firewall protecting the stuff cluster from say, whatever computer is being done, maybe the Ceph cluster is operating as a storage appliance for some research university and you don't want you know the students have access to whatever data might be on there, such as like biometrics and things that need to be protected by HIPAA regulations and rules.

A

Another security aspect is, you might want to protect the data of vias, some authentication mechanism, that stuff doesn't yet support, for example like Kerberos, although that will this slated to change in the future with octopus and then finally, one of the more the use cases we really had in mind when we began this work was OpenStack Manila, where we have 10 of VMs that are wanting to cooperatively use a particular volume and share part of the shared file system, and you don't want other tenant VMs to be able to talk to that.

A

To manipulate that Sherriff volume, and you also don't want those tenant vm's to have raw access to the storage network. You want them to go through some gatekeeper service like an effete, an NFS daemon, to prevent them from manipulating data that they're not supposed to or operating an old client that we know has.

A

So, put that on my hand, this off to Jeff, so.

B

We've had active passive gateways on top of set fest for a long time. Essentially what what that means is that we have- and these are pretty common and and have been for a couple decades now, but you build those you can build a cluster with four NFS. That is where you have, you know main one main server and what sort of standby host and then the main server fails. The fails over to the other node.

B

Those are those tend to scale pretty poorly one of the problems is that you know at some point you're going to hit the cake. You know all you can do and with a single server and at that point, then you know you sort of hit a wall in performance, so we've had this and for quite some time now at least and we've had a way to do this and since Ceph luminous as well in an active passive configuration.

B

You know we typically you rely on some sort of third party clustering software like pacemaker or core sync. You know, and then people built active passive NFS clusters with stuff, like you know, Veritas clustering and that sort of thing, but what we really want to get to is an active, active deployment. So- and you see here, we've got you know, and this other slide we had. You know all the all the clients you know are various clients here are connected to.

B

You know one single node and there's no, this other nodes just kind of sitting there not doing anything, and so then, but really what we want to do is be able to scale out linearly. You know so that we can have multiple clients connected to different nodes and still have consistency, guarantees and stay full and you know and allow them to manage state properly across the whole cluster.

B

So when we embarked on this project, we had a number of goals and if anybody could text might be a little too small but any case yeah. Essentially we wanted to be able to scale out the cluster. You know linearly, so you know if we decide we need. You know a bit more capacity. We can just add a new cluster node and keep running another goal we had was we wanted to be container izybelle.

B

You know with it's a new world now with containers and kubernetes and that sort of thing, and we wanted to be able to leverage that infrastructure to make it easy to deploy these things. It's nice to be able to go into cooper, Nettie's and just say everything you know, give me another cluster note.

B

We also wanted in order to do that, we decided to focus on NFS before 1 plus I'll talk about the NFS protocol here in a minute, but the for one gives us a number of features and capabilities that earlier and fs versions didn't have and then also we wanted to sort of minimize the need for third-party software like pacemaker. If you have, we already have a cluster, we have couple of clusters right, we have kubernetes and we also had SEF. We didn't really want to add a third one too.

B

In order to handle this, we feared that what should be enough.

B

So our you know, demon of choice for using NFS is NFS Ganesha, so it's an open source, NFS server runs all in user land and so that's kind of nice. It makes the containerize wolf portion quite easy.

B

It has a plug-in back-end, so we can plug in for different different file systems and one of the ones we can plug in that we're concerned with here is SEF uses a lips F of S, which is Patrick, talked about the fuse client.

B

It really is built on top of a library called libs, ffs and so connection plugs into the same library and uses the same functionality to to do its thing, and we also have the ability in Ganesha to do to store some of the out-of-band information in in rat us directly, in particular, for NFS.

B

Before and I'll talk about this again here in a bit, we need to be able to keep track of what clients are have state at any given time, and we also need to track some, and we have also tracks some configuration information in in rat us as well. So all this kind of boils down that. It's all pretty amenable to containerization. We run all this in user land. We don't need any.

B

You know we're not storing all the configuration recovery info and rat us. So there's really no permanent information. We need a store on a local disk or anything. We don't need any really like writable local filesystem storage. It runs completely unprivileged. We don't need any root privileges or anything like that and that in the container and if the thing falls over and dies, we can just spin up a whole brand-new container from the read-only image and not have to try to do that sort of failover dance.

B

That's pretty common with traditional clustering, so uh step back in for a second I'll talk about the NFS protocol, so the NFS protocol is pretty old as came started in the 80s best son early versions were all stateless, so they didn't have any state at all.

B

They you you it's, you know we didn't keep track of open what files were open or whether locks were held or anything like that, and those were all done by bolt-on protocols that were added to the side and then in the early 2000s we had NFS before was I was presented, and for that we needed it to be stateful, and that was mainly done so we could handle locking in the core protocol and then about a decade after that we have a new protocol that came out nfsv4 one so in and for that it added a number of features.

B

One thing was a sessions layer which made it easier to do exactly one semantics which was necessary for NFS, and then we also it added something that we rather liked for this, which is added to reclaim complete operation, see prior prior NFS protocols, didn't tell the server when they were finished, reclaiming objects. So you it never knew what if another reclaim operation would come in at any given time, and so you always had to wait for the full grace period. We'll talk about that again in a minute.

B

Reclaim complete operation makes it so that we can keep track, of which clients are connected back and reclaimed and completed the reclaim. And if that happens, we can lift the grace period early so, and it also has some features for clustering and and migration so that we want to be able to use in this configuration. And then we also have an emphasis before that's come out fairly recently, and it's mostly just smaller features on top of e41. But it's pretty similar to protocol.

B

So the big challenge with any NFS server is that you have to deal with recovery. So most NFS servers don't track ephemeral, state on unstable storage. So what happens? Is you you know if the machine crashes and comes back, the clients have to have to reconnect to the server and tell it you know: I had this file open, I had this lock I had this delegation and so a lot of you know. Commercial NFS servers will do this by tracking detailed State on stable storage.

B

So if, but that turns that tends to be pretty expensive performance wise, so Ganesha heads. So what we do instead was we we just track. We keep track of what clients had state, that's NIT, which is necessary for certain.

B

Certain corner cases that can occur if, if the machine reboots and comes back after a network out after a network partition, kadesha has had support for storing this in rat us and single server configurations for a long time it had some bugs, but well you clean those up pretty well now.

B

So anyway, when the machine comes back, you have we established, what's called a grace period at the beginning and then when the NFS server comes back, I should say and that during that time no new state can be established.

B

So if someone tries to do an open, they'll get an error back that says, try again a little later, and so the only the we only allow reclaim requests during the grace period, so we're allowing clients to come back in and grab stuff that they had before, and we only allow reclaim for clients that were present and held stayed at the time of the crash, and that's so in order to track that we have to keep track of what clients were holding that stayed on stable storage.

B

So when the thing comes back at knows so, prior to doing to a client doing its first open, we set a stable storage record for it and if there isn't one already and then we remove it after the last file is closed or when the client state expires or we the clients, lease expires. And then, when we go to lift the grace period and move into normal operation, we have to atomically replace that database.

B

Replace the old client database with the new one.

B

So another way to think about this is that we have actually multiple epics, so you can think of a you know, a sir life of an NFS server as a series of reboot epochs. We start out comes up it's in grace period, we transition into normal operation and when it crashes again it comes back up.

B

It's got a new epoch, so you know, and so what at the start of each new epoch, you know as the clients perform their first open, we set a record for them in the database and then during the grace period, allow clients that are present in the previous database to do the reclaim and then, whenever we are ready to move into the new grace or out of the grace period, we do that atomic replacement and so that the next time a new boot we know exactly who was in the who was present at the time that the crash happened, and the key point here is the reason why this matters is.

B

Is that that we can apply this epic concept across a cluster of servers.

B

So this is new. In Ganesha we've we wrote a new rat us recovery back end for this, for the four Ganesha that does keeps track of a coordinating database within rat us in a rat us object. That's to allow multiple servers to coordinate what epoch is President Roh epoch were in at the time in which epochs were allowed to recover from.

B

So we do that with a couple of 64-bit integers. When we have the current epoch, one we're in now, and then we also have one for a recovery epoch and then when, as we move from from the grace period and into the normal operation phase, we zero out the recovery epoch to indicate that we don't we're not allowing anymore reclaim so, but we also need a way for the server's who are loosely coupled to be able to indicate what their needs and their current status. So we have a flat.

B

We have a set of two flags per server that we also track. One is the need grace flag. So when the server comes back up, he says what I'm just crashed and came back. I need a grace period.

B

It's the other servers can then look at that and then, when as they move into the grace, they can then go start enforcing a grace period that they probably won't, allow any reclaim, and then they can say when they are enforcing the grace period, they can go and set a flag, saying hey I'm enforcing now, and then at that point the server that is, crashed and come back and look and see and say: okay, is everybody in the in the cluster now enforcing the grace period, great I can go.

B

Do a lot reclaim to happen, so the idea here was to to move away from sort of a traditional way of doing this sort of thing. Where we have. You often have a central controlling entity that says: okay, everybody go into grace period. Now everybody come out of grace period.

B

Now you know that's a single point of failure, so I wanted to get away from that and do this in a way so that we have, you know, apply some really simple logic and information so that each node can sort of make its own decision about what it should do at that time.

B

So so that's you know interesting from the recovery aspect of it, but we also have some other issues here too I mean when we look at the sort of configuration, we're really layering NFS layering NFS over any clustered file system.

B

You, you know we are dealing with layering a network file system on top of a network file system effectively, and so when we do that we have their sessions. You know each each network file system has its own sessions sort of layer and in case of sefa fest, both of those session and NFS they're, both lease based mechanisms, so their protocols are stateful. We have leased leases that we have to deal with where you know the clients have to maintain regular contact with the servers to make sure they haven't dropped off the net.

B

So you know we have some real challenges with this. In that you know we have to deal with it, but maybe interlocking timeouts with some of these sessions. So in effect you know. Here's an example like on here is that you know we may have a Ganesha's server that crashes and then we start up a new one right. Well, so the client issues a request to do an NFS reclaim that new server then comes back and says. Okay, you know I'm gonna do an open through lips ffs.

B

But now are you stuck because the SEF efest client, the previous F of S client, that is now dead, is holding the caps that we need that the news client needs in order to do the open, and so you know the the MDS session. Timeout is defaults to 60 seconds and usually our NFS grace periods are 90 to 120 seconds. So by the time that other client expires, we've already eaten into a half the grace period and it can't be worse right.

B

You know, if we have, you know it may maybe there are other timeouts to deal with. We have leased timeouts and stuff, so the clients may not even come back immediately when the server reboots. So we have you know by the time we get around to everything being cleared out. It could be, you know, the grace period could be all the way gone and the session could also die. You know if we have. It takes a long time for the server to reboot and come back.

B

You know the MDS session could die before the server's returned at all, and at that point you know, client another clump. You know another stuff. A fest client could have raced in grabbed a lock. You know it's a file lock done some changes and drop that lock and then the server coming back would be none the wiser and that that's bad, because then we're looking at you know silent data corruption.

B

So we added another feature to not Canales. We an interface to do to allow set clients to reclaim client sessions. So for now it's so in a you know: Seth assess client drops abruptly.

B

He just drops off the net, let's say crashes: the MDS will keep the session around for a little while in case, because he doesn't know I mean it could just have been the you know, network hiccup or right right or something so he's gotta wait a while for the Ceph efest client to come back, and but you know all the stateful info for this, for this session is, are all the stateful info for the that the client holds as as tied to those sessions.

B

So we added new interfaces to lips efest, allow the connection to request that it's old session be killed off immediately, so that allows us that gets us past this point where we don't have to wait around for that MDS session timeout to in order to be able to get caps that we need to reclaim info. It also allows us to do things like extend that timeout so that we have a little extra time in case we have a real. You know real major problem that might prevent the server from rebooting and coming back pretty quickly.

B

So eventually we want to be able to extend this interface to that sort of long term plan is to extend this interface so that we can allow the client to so. We would allow the MDS to sort of hold those caps in in escrow for this for the new client coming back, so he would be able to reclaim against those old caps. So wouldn't so in you know, here's my token animation here. So here we have a you know: a single server, NFS server, Ganesh, the server that's connected to the Seb's cluster.

B

He dies news, client comes back or we spin up a new client, our new NFS server over the same node ID, and then he talks to the Ceph cluster. That says: okay go ahead and kill off that session. Now boom it's gone, so that allows him to sort of kick off and get back to running quickly.

B

Eventually, we, like I, said we want to be able to allow it to do things or to be able to reclaim stateful info from what that was held by the previous client, as that would allow us to avoid putting the whole cluster into grace. The other nodes could then just keep chugging along because we would know there'd be no conflict. I think I was going to turn back over to you at this point. So I'll turn back up to Patrick yep.

A

So the next challenge we wanted to address was scalability, and this was one of our long-term goals, so in in a lot of the NFS Canisius setups that are in the wild, it's currently an active/passive configuration and what we wanted to do it. Allow was having multiple NFS Ganesha servers in an active-active configuration that allows us to scale out horizontally for what the file system being exported. So you could have clients talk to multiple NFS servers.

A

Another goal we would like to do is be able to scale in the up in another direction to by having an NFS cluster for each volume that is being exported within cephalopod volume, or nowadays we call sub volumes, which would just be a directory tree that we want to export with Ganesha.

A

Another way we could achieve scalability is by having an NFS cluster for each of those directories, sub volumes so that, if assuming these you know these data sets are independent, you can just have clients relegated to a particular part of the sub tree and only talking to a particular set of Ganesha's.

A

Another challenge we needed to address with deploying these NFS gateways was handling failures through in dealing with IP migrations, so that when one NFS Ganesha server fails and another needs to take over for it, you need to move the IP address because NFS clients don't know how to have their their IP address switched with SEPA fest. This has not been an issue with them.

A

Guesses, as you know, and as you may know, the MDS has also have standbys, and so when the active MV has fails for whatever reason and the monitors promote a standby to an active, the monitors distribute a new MBS map which goes to all them yeses and also all the SEPA fest clients, and they know what the new IP address of the new MDS is.

A

There is no such service built into NFS, so the NFS clients have no idea that as something is taken over, so you need some sort of infrastructure service to actually move the IP address over to another Ganesha daemon.

A

So this is where rook and kubernetes come in requests and communities are kind of a direction that stuff was already heading in, and luckily kubernetes also deals with the the issue of IP migration for us during failover, and so I made it an attractive platform to target for this work.

A

So, as you may have saw, seen in other talks, Brooke is a is a a cloud native storage Orchestrator. It's an operator that runs within kubernetes and manages your SEF cluster sets up sets up. The cluster manages it through its lifetime.

A

For our work, we're also taking advantage of this new paradigm within SEP, where we're the scepter itself is also aware of the orchestrator running under it, and in this case it would be rook and it can actually issue commands to the to the orchestrator to change how the SEF cluster is deployed, for example, set up new STS for this work.

A

What we're looking to do is we want the SEF cluster to actually manage the NFS daemon, so the NFS gateways are now a first-class citizen within SAP, and so we're gonna take advantage of the orchestrator to have settled rook to please you know, deploy me a new NFS, Ganesha gateway container.

A

So rock is taking or is handling all the details of actually deploying the container. So now Steph doesn't have to worry about alright, so I need to I want to deploy Ganesha's on these three servers and eventually I'll get around using them. We don't have to deal with that nonsense. Rook will actually take care of the details of deploying those on demand again we're taking advantage of the orchestrator. So this is gonna.

A

Let us scale up and down the NFS Ganesha servers and also in the dashboard is going to allow us to set up the exports for the Ganesha demons. This was a somewhat a queer way of setting this up, but when the manager actually wants to configure the exports for the the Ganesha servers, it's going to issue commands against the dashboard API, which will actually go out to rathaus and set up the configuration files for the for the Ganesha's cluster.

A

So, what's interesting about this is that it's going to allow us to set up Ganesha servers on demand for a given sub, william that we want to export, and it's also going to allow us to scale that horizontally with the number of active actives according to how popular that sub volume is initially we'll just let the operator of the cluster set statically, how many Ganesha's they want to set up. This is kind of a direction we want to go in as the set project as a whole.

A

We're you know against F is sort of managing itself and how it's being deployed and being able to dynamically modify itself. This sort of you know software-defined, storage right now, we're looking at you know how we can do this with Ganesha and exporting, but we also want to look at this within cephus in other aspects, for example, with deploying MVS is being able to have SEPA fests. You know, do some kind of introspection and detect that it needs to act.

A

It could benefit from having more MTS's and dynamically, go out and add those and then at some point in the future, when probably with an octopus, when we have multiple multi file system support, and so you can actually create volumes which would be multiple file systems, the Ceph cluster itself, the manager is going to be deciding that I need to deploy more MTS's in response to creating new file systems, and so, as far as the vision for the future we're not quite there yet with Nautilus. A lot of the the building blocks are in place.

A

We just have a few things to clean up and then back pour it into Nautilus. So you can kind of expect it within a few minor releases. Eventually, where we want to get is we can create a sub volume which is again just a little directory tree within cephalus. In this case, the file system is called vol a and the sub directory that we're creating is called sub vol 1. This is a new volume manager plugin that we were building within stepheson.

A

It's going to allow you to have a single source of truth and essentially managed in API for actually managing the volumes within cephus and then eventually, after you've set up the sub volume we want to be able to just as simply as that say, turn on. Nfs sharing and the Ceph manager goes out. Configures the Ganesha exports for you securely sets up the authentication credentials for that Ganesha's server and turns it on and kubernetes will deploy for force F and it just works.

A

You don't have to do anything else, so we're taking advantage of the container deployment to get as much scalability as we need in response to demand and setting it up as as simple as possible for the operator.

A

B

On the second second bullet point here is we're talking about adding the support for before migration. So one of the problems we have now is, like patrick said, the NFS clients are sort of bound to a particular IP address, and so what we want. But there are mechanisms in the NFS client in the kernel note of the Linux kernel in it, this NFS client that will allow it to move to a different one, and so we could leverage that to be able to do things like scale down a cluster.

B

So if we've had say you know ten, you know a three or four node cluster and we want to scale down to you know one less. We could migrate everybody off of that node and on to other ones, and then they could go and and then we could just decommission that note, and then we were also talking about optimizing the grace period for subvolumes. So we really don't need to put everybody into grace and if they are not exporting the same things, so we want to be able to.

B

We could sort of reoptimize this work so that so that we only confined the grace period to nodes that are that are exporting stuff where that a rebooting node is also exporting and then finally, we want to eventually would like to do some sort of SMB integration, probably using samba, and we have some ideas about how to do that. But it's a bit of a bigger project and with that, that's the talk. So uh any questions.

B

Okay, no questions: okay! Well! Thank you for coming.