OpenZFS OpenZFS Developer Summit 2013, 19 Nov 2013

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OpenZFS Developer Summit Part 8

Description

http://www.beginningwithi.com/2013/11/18/openzfs-developer-summit/
ZFS Multi-Tenancy (Robert Mustacchi), Max Bruning of Joyent on ZFS diagnostics. Examining on-disk format (Max Bruning); Closing, Future Plans (Matt & Co.)

A

Come sit down, eat your food and we have two more talks, say brian.

B

And chris george, hey karen, hey everyone.

C

A

Sorry so uh rob you're gonna talk about multi-tenant vfs.

C

Yeah, I'm gonna talk about uh a bunch of the challenges and things we've done so far and uh things we'd like to do in the future and I'll try and keep this brief. So people can get back to eating and beer. So basically there's two different problems. We have because we run a multi-tenant environment. Folks are running in different virtual machines. Some are hardware virtualized, some are just using os. Virtualization basically have two main problems we want to solve. The first is that we want to have some notion of.

C

uh Let's say you wanna have submission of I o fairness. um This is also true for all the other resources you have, but um what's really relevant here is basically disk io and, while in straight disk, io isn't as as useful as looking at it at the zfs and vfs layer, because.

D

For most of our customers, what they actually care about, isn't what the disk io latency is it's actually what the vfs.

B

C

Latency is because, if they're doing mostly serving web pages say, what's that's going to be cached reads coming out of the ark, so they don't really care about what this gets. More importantly, they shouldn't care about what the discs are actually doing, um and, of course one thing is that, as you buy more, you should be able to get your entitlement of a share of the box. So one of the first things we basically looked at doing is um we did this.

C

uh This is work by uh bill fiesty and jerry jelenic, with some most recent enhancements with from brian, obviously,.

E

C

The notion of a per container I o throttle, so what that means is that, for us, everything is basically every customer, so every guest uh vm runs inside of some os level container, whether it's ultimately just running a hardware, virtualized guest or not, and we want to basically provide some of I o fairness. So what that effectively does? Is it basically cues in office? Some of the current zfs information basically looks at the percent utilization.

C

We then say if the disks are actually not very utilized. Who cares just like everyone can go use this because there's no contention, but once we start uh creeping up towards the 70 or 80 mark we're going to start kicking in and basically we're going to use the notion of shares. Basically different different containers have different weights. Obviously that dictates how often they can do io and those that have less basically will get throttled for heavier. I o operations so.

D

C

Basically, a small town he's basically just trying to do a ton of rights we'll end up having to actually inject up to about 100 mics of basically busy weight on various vfs operations, the idea being that, ultimately, you want a little guy who's, just trying to get a single.

A

Reading to be able to actually proceed so you're actually doing this at the vfs.

C

Is all this measurement and everything happening in.

A

The vfs layer, rather than no.

C

This is actually all part of zfs okay, so um we basically have running case stats that are collecting, um basically we're using loosely keying off the utilization and basically of running case stats on the containers for both uh logical operations and the size of those operations.

A

But you're answering the delays in like cfs reads: if that's right.

D

Yeah, it's not in the.

A

C

A

As opposed to that,.

C

In through the vfs, which will drop you into the entry point of zfs.

A

Where are you talking about percent of members.

C

Are talking about the percent of arc? That's used, sure, that's something! That's not there yet and we'll talk about things. We'd like to go do in the future, but yes, obviously you're in a multi-tenant system having control over how much percentage of the arc someone is using. So you said busy weight, you mean you actually like, like you'll stay about 100 months, we'd rather have okay!

C

Well, because if you do because it's a very small amount, but it's just enough to basically allow other people to basically get disguise and get other stuff scheduled and we'll also kind of play a little bit of games with the scheduling.

C

So that's one aspect, the uh you know the other big aspect that we have um is. Basically you want to have basically controls on zfs operations. Yeah, as we all know, um if you create ten thousand snapshots on a box, the box is gonna, be kind of sad. Similarly, if you can create you know, ten thousand file systems.

E

You basically push up.

C

The number of file systems and snapshots things can get uh pathological. We also don't want, for example, customers to be.

D

C

Enable youtube um there's a lot of reasons for that.

D

C

Need to ask: well then you need to learn, um and so one of the things that we do with we have delegated data sets that we pass into these zones so part of the work that we're. Currently it's currently ongoing. It's on review and see if I supposed to have the notion of limits on for the number of data sets and snapshots that can happen in the data set hierarchy. So that way you can easily delegate a data set to a customer, say you're allowed to have 10 snapshots.

C

So you know that even if you have 100 customers on a box that you're not going to basically run into pretty egregious behavior, if you have a question.

C

And then so those are the kind of the two main things that we're doing today. um Those basically allow a lot of functionality for basically untrusted customers to do what they want um kind of the open things we want to do. One of the biggest things is basically multiplying arc, basically having a notion of having shares in the arc, so you basically could have the obviously the arc is going to be growing and shrinking, but you want to basically have some slightly more intelligent choice over basically what data sets are going to be affected.

C

That's kind of one idea: we have that brandon's kind of toyed around with a little bit. The other stuff we want to do is make receive a little bit more um trustworthy.

C

So right now um you know cfs receive is send receive, is great, especially if, as long as you can guarantee that zfs generated the data, um so we kind of want to do some receipt, fuzzing and kind of really solidify that such that you could actually say, take a zfs snapshot that someone created on their own machine at home and basically be able to receive that elsewhere and not have to worry about militia a malicious user who's. Basically crafted a pathological sensor.

A

So in that case, are you concerned with um making it not panic the box or are you concerned with them like not creating a set uid root, binary right.

C

Those are very different, we're not concerned with the contents of the stream mark as much as the correctness of it so yeah, so that won't panic the box that once also you trick us into saying. Oh, we have some. You know five petabyte file that doesn't obviously doesn't look like. We.

A

Did we've done a few fixes in this area, because we have like a test uh test case where it just like, injects random corruption into your sense stream? um So we've done added a few more checks there. I think a lot of that is just around like being much more defensive about the today that you.

E

Receive the biggest one is, though, that, like the object, the objects that you send and the sense room are just data, so you can send a directory which is a zap object, and we don't really have a view in the fact that it's supposed to be a zap object. We just write it out and then we don't find out until someone goes to read the object that it's a zap object. But then.

D

E

D

That's the correct example:.

E

C

That's so much those are the kinds of problems. Obviously we want to go and fix that way. One customer can't obviously come and take out a box, so those are the kinds of problems that we deal with in terms of just multi-tenancy and adding we're always looking to add more observability up through the stack in terms of being able to see things on a per container basis, in addition to per data set basis, thanks.

C

B

B

Max other maps.

A

He said 10 minutes ago or from this I think he mentioned this inside, so I thought I'd show a little bit of what I've been doing.

A

um And I didn't really introduce myself before I'm I'm max. I work for joint and I do training mostly when I started joint. I was looking at a colonel hang and I worked on kvm the kvm port uh for a year.

A

um I started it did about four or five months, and then robert here and uh brian cantrell joined me and it went much faster after that, um and then I went back to what I've been doing for many years. Actually, there's some people in here who've taken classes with me and various places like india and boston, and where do I know you from from you're going in office, oh from okay, much more recently, um so there was the other max. Is he still here right, you're, the other max.

D

A

So I you you actually had as one of the topics uh to talk about education for zfs developers right, so I'm gonna make a shameless plug. I teach a zfs internals course. I've done it maybe 10 12 times now, I'm slowly uh after several years of studying it learning the dmu still not there all the way, but um and what I do in that is. uh I spend the first day, basically doing nothing but looking at zfs on desk- and I I'll use zdb for that.

A

But uh I've also been late more lately using this mdb that I modified and I'll explain what I did. I took the I took mdb and I basically this is back in nineteen, no 2005, 20 22, no 2008 2028 2008, so I took I took mdb and modified it so that I could run it on a raw disk and look at the data structures on the disk.

A

Okay. This is not necessarily the right way to do it. I talked briefly with robert about this. Robert suggests, a new target and I've kind of flip-flopped back and forth, and now I think new target is the right way to go, but yeah get on with it. um So let me show you um so how do you you mentioned uh earlier, like that? It doesn't know how to decompress stuff.

C

So all the metadata is compressed.

A

This seems like it's a pain in the ass. Not everything is decompressed. There are occasionally zap objects which are not compressed. No clue. Why not? But I I've got some here. Okay, um so I mean. Are you saying that you can only look at those zap objects? Oh.

D

A

Look at everything: okay,.

D

So how does that work? Well, no! What I do is I.

A

I take the the thing I want to look at and I use uh mbd, I'm sorry dd or um I've or I'm sorry zdb. You use zdb to read or.

D

A

It well, the thing is zdb: if uh you're having problems with the pool, zdb, minus uppercase r doesn't work well.

A

So what I, I wrote, a little z, uncompressed tool that you give it the physical size, logical size, an offset on the desk and it uncompresses it for you and you can optionally, give it a decompression algorithm in it and then I'll put that output in another file and then I'll run mdb on that file.

A

So the file only contains my uncompressed data object, whatever that is um so at any rate here.

A

Just so you can see what I'm talking about here so here I'm trying to look at um a pool called zones and this pool. Basically, we cannot import and we do get some nice warning messages and um then it dies with this assertion, even though I'm using uppercase aaa to try to have it not die from this. uh It didn't matter it's dying in the assertion and in fact all this does all it gives me.

A

I tried putting it into a file, but then I realized I'm on a ram disk, because this is smart os and I don't have any pool available. So it's a ram disk, so it just fills up. The file system stops after some small amount of data, but then I what I did was I tried dumping it out to my screen and I get basically it dumps for a long time and then it dies with this and it dies with this right after dumping the space map.

A

So it doesn't get me anywhere, I'm not getting there with zdb. So.

A

So here's, basically what I'm doing I I'll run mdb on the the raw disc that contains this pool, and I I could run it if I have multiple discs, I can run it on different discs, and but I'm not going to get into that. I basically pick up the kernel's ctf information. That's what this load ctf is doing.

A

I load a module that has, among other things, the super block, walker and a couple other things not all of which work I had to modify the zfs module. That comes with the system. I had to add four characters: zfs back quote on at one place, and now I can use. I can look at all the cfs data structures and here I'm going to walk the uber blocks and print each of them out.

A

So what I end up doing with this is I basically I can type it. I can look at just the transaction group, id type it to something that does the sort to give me the highest transaction group id and.

A

There I did that, and here I'm I'm dumping out the uber block with the highest transaction group id and from this I get a block pointer and this block cleaner is at this location on on in on the desk. This bite offset on the desk so I'll. Take that.

A

And dump it out like so so this looks just like you would see with mdb and I can see. Oh it's an l0 object set so from here. I can go and dump out. So what I do is I try doing the zdb here, but uh this won't work and with the minus c it doesn't help the pool's exported. It doesn't help.

A

Eventually. I think this time I might have ended up doing dd on it, but.

C

B

Find it here, the db is right up there.

A

Yeah: here's the dd.

A

And yes, I seek here and dump out 512 bytes, because it told me that the upset fizz tee for the moss is 512 bytes. So I dump it out, um and here I tried. Oh, this is still not right, because I'm trying to decompress until I re looked at my code that I wrote a while ago and realized that.

A

Basically, I could decompress directly.

A

Get there eventually.

A

Okay, so here on uh I've, oh I got it back up right. There.

A

So here I'm dumping out parts of the moss. I'm sorry the object, fizz t. So this is the decompressed block that the uber block points to so I'm doing this by hand walking the data structures on the desk it's time consuming, but I can find everything I found history. I can look at the space map. I can look at everything on the disk this way. It's there's some things I want to add.

A

um I mentioned I'd like a walker, I have actually written a command that, I can just say, cold call number block and it should give me the current uber block which doesn't work. In other words, the command is broken, but uh this is where I'm going to go with. This, eventually, is uh what I really like to do is say: uh um clone call, number block, pipe uh cold, calling uh walk data structures and it walks everything.

A

But then I realized that basically, then I had written zdb, because that's what cbd does the nice thing about? This, though, is that I can I can. I can find things so I right now what I'm really looking for is, uh for instance, this here. Where is this? What is it coming from? It's got to be in the zap object. Where is that zap object? What's going on with this?

A

Zap object has a uh object id and if I can look at the object idea and try to figure out why it's complaining, that's it. This is great. You should totally finish it yeah. I I I like. I said I wrote this several years ago and there was no interest in it and what's happened over time.

A

It's very interesting is over time now I get an email, maybe two times a week asking for help recovering files and in fact, five years ago, when I didn't have any work, it would have been great to get that email and now I do this yeah I'd like to finish this yeah interesting stuff. For me, okay, thanks guys thanks.

A

I realized that we've lost a few people to the to the food, but um thank you all for coming. The last thing that I wanted to ask people is just um kind of for a evaluation of this day, and like was, I hope that today was useful. Would you like to see like another event, just like this uh in six months or a year? Would you like to see something? That's like more days, more involved, more um customers.

A

You know like something: that's more oriented towards um giving presentations to people about the work that you've done on zfs that we might already know about because it's in the source space, but um people using zfs might not know about um any thoughts or opinions.

A

I'm gonna ask for a show of hands if nobody has an appeal. Okay, so who? Who wants to do this uh event again in one year? Okay, six months, okay,.

D

Six months over a year.

A

um Who would like to do an event that is like maybe like this, but in addition, maybe, instead of the hackathon, have a customer oriented day where it's presentations for customers. We get a bunch of sysadmins, and you know people like that in here. Other kind of enthusiasts who are not developers is that I know there's a lot of venues for this kind of stuff already, so I don't want to. I don't want to replace this emotional would be interesting all right so, and would you be interested in speaking at such an event.

D

Okay, possibly.

A

Yeah, but I think that's that's great feedback. um The main thing is, I don't want to like be replacing other events that are happening like there's already lots of great um conferences, and you know all that are kind of specific to each of these communities. Freebsd has a bunch of conferences.

A

uh Linux has a bunch of conferences and uh lumbos has uh you know a couple conferences, so you know, I don't think that we want. I still want all of us people here to go still talk at those conferences, but um if there is interest, then we'll we'll look at see what it's gonna take to plan a bigger event for next year or in six months. I.

B

Think the market's there about six months, okay,.

C

Yeah is there any place that these people are already gathered, these customers and you bring up a good point there. What if there.

A

Is a conference for development, non-developer users.

C

That we could just go to what was the crash economy yeah.

A

I mean we could certainly crash a conference or try to um you know. The question is like who's and when because a lot of these conferences are um more domain specific, so, for example, we could we could try to crash uh bsd can which is held. You know every may in ottawa, canada. You know it's four days, there's like two days of um developer summit and then and uh and tutorials and then two days of new customer oriented um conference. So we could ask them like hey. Can you add one more day?

A

That's just zfs and we can kind of like blog on to your conference. um You know they might be able to do that. If we come up.

D

With the money, and why not.

A

Can we take two hours of the customer day I mean yeah, I mean we can take two hours of the customer day just by reporting talks. You know like that's. That's that's easy. um The the issue is more like. Should we create an event of this scale where it's like an entire day of zfs.

B

So there's two scale in la in late february. Is you can run a full track day of zfs during their three or four days that they run quite cheaply? It's getting bigger every year as a nice crowd. They I mean we got involved because they they specifically asked me for liberals and smart os speakers. I'm sure they'd be delighted to work something out for cfs uh and possibly use next lisa next year.

D

All right, lisa that would take a lot.

B

A

All right so would people be interested in doing in like, in speaking, at a zfs track at scale in february.

D

Yeah sure it's.

B

C

A

um So, thank you all for coming. um The beer will be open soon. uh Chris has selected the beer. He is our chris and basil. I don't know steve basil here. um The the beer is an amazing selection of some of the best beers in the world. So ask chris more about it. We didn't go that high okay, some of the best beers that probably most of you will have ever drank. um Unless you work for deltix and go to christmas, private tastings.

A

So great enjoy enjoy the beer enjoy the evening and tomorrow, 9 30, we'll start the um hackathon. So talk about your hackathon ideas. Try to find some people that are interested in your hackathon.

E

One thing about the beer, too: um there's six packs of things, big things, but there's a couple like just big bottles and a couple of small bottles of the same things and I'll pull out some small glasses for some of the fancier things. I guess that we don't have enough for everyone to have their own like private model, um but I'll just put those on right here with some glasses cool.

A