OpenZFS Leadership, 24 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: August 2019 OpenZFS Leadership Meeting

Description

We discussed OpenZFS DevSummit talks; the Linux Plumbers Conference; 32-bit kernel atomic reads; zfs share -a on Linux; and dedup improvements.

Detailed notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit?ts=5d125dd0#

A

Yes, thanks for joining us, August, 20, 1920, first leadership meeting we have a few things on the agenda. I will kick it off with the opens divest developer summit. So yesterday was the deadline to submit talks for the opens defense developer summit. I got a whole bunch of interesting talks. That I think you guys will all be excited to hear it. But if you were thinking about somebody's talk and didn't, then let me know right away.

A

We're gonna I'll send out I'll finalize the schedule in I think two weeks.

A

So let me know within that time. If you forgot to send me your clock on time, did anybody think that they were gonna? Send me a talk, but then did not or want me to reconsider their talk? Okay, hopefully everybody found my email. The next topic was about a different conference. That's there for my to ask about.

B

Hi everyone, George, Wilson and I will be at the mix: plumbers, September, September, 9, I, think September 13, or something like that. It's basically a conference that deferred like a general developer and if you're working on Linux- and you know you have stuff to talk about or discuss and connect with. Other individuals like this is the place to be towards are not something that sounds like an open discussions for guarantee binding specifically, but in resources to carry father folks from office from this community will be also there.

A

Anybody else playing I think you know going to a conference.

A

All right: well, if you hear about any of your co-workers going, let's start from read George know you guys, can meet up and talk, see a pep talk to the fest or started you guys about how to engage about ZFS with the Linux kernel development community.

A

Cool the next topic that we had was I. Think from Andre. Are you on? Yes, you are cool about a 32-bit non-atomic operations. Do you want to kick that off under it.

A

Maybe he's looking for his new vine hi all I'll give any introduction in see if one reckons left. You so I think that the the issue here is that, if you're running with a 32-bit kernel, let's think about it, go ahead. Go ahead on during.

C

Operations modify values that we.

C

Reference counts or various stats, but to read them. We use plain plain reads like 20 assignment operation, and it seems that that works very well on 64-bit platforms, because.

C

And for yes and I mean that those Atomics atomic variables 64-bit as themselves so that work works well because 64 is a natural integer, size, natural processor, integer, size and I. Think that usually or maybe we even enforce it- those variables are properly aligned in memory. So it's not a problem, but on 32-bit platforms, those plane reads: I think they I implemented as two instructions to fetch two 32-bit values from memory and then manipulate them further.

C

So even if asserted two-bit platform can provide a correct implementation of 64-bit Atomics, there is no guarantee that those 34 bit atomic updates would play well with those plain reads of the values and that's my concern at this point.

C

It's mostly theoretical and I think that probably situations where that could cause a problem are quite rare, because it's not often that we like cross that boundary when, like both halves of so 64-bit value, get updated at the same time, it's very rare and for some things, like reference, cons, I think it's extremely rare like when you have that many references, except maybe for some stats, where, like we use a reference, counts for stats.

C

But I wanted like to hear general opinion if we may be sure that should add some atomic read operation instead of those plain plain excesses: duane reade accesses to 64-bit Atomics. So that's yeah! That's it.

A

Yeah I mean Andre that I think I agree with what you said. A bunch of those are probably my fault from way back in the day and I. Don't I, don't know for sure that those can cause errors and, like you said in most of them, are in there in, like code possible, we're just kind of reporting something rather than really taking an action based on it. Wasn't it but I mean he would still be wrong to give him the wrong the wrong answer there.

A

So probably we should be introducing some atomic 64-bit read operation that would be able to be atomic on the 32 platforms using a similar kind of mechanism as the atomic increment with return value, like you know, committed with, like atomic increment by zero in return, new value underlying operation, I, actually.

D

When I was working on the ZFS compatibility layer a few years ago, I did this I used compacts gjb to do the atomic reads and it's it's super easy to implement. It just makes it a lovely, because you have to put that everywhere. You want to read the variable.

D

The other question is how many 32-bit platforms do we actually run on these days? I mean I. Imagine there are still some, but it is worth it's a question. We're thinking about yeah.

A

I think that there's probably a bunch of problems that it does run on like on FreeBSD and Linux, as you mentioned, but how many instances there are of those is probably a very small. But if some of you, if somebody cares about those, then by all means like go, do it.

C

Yeah I think that I don't have very good stats and I'm not sure like. If many people are on a ZFS on 32-bit weights, a PowerPC or something like that. But I think that I 386 is certainly supported and I have heard from people who use the FS on that on FreeBSD on that architecture. So they intercept a two-bit architecture.

E

I just wanted to add that I wrote a like a separate thing, just as part of a side project, and it it I had the same kind of issue and I was able to quickly find and fix all of the issues.

E

Just you know, or just by doing a lot of copying and pasting by taking the by taking the atomic structure and instead of having it be like the kernels actual atomic I made it a struct that just wraps only the kernels atomic structure, and then you can't set it with just a regular value like you can't just assign it to I, don't know 63 or whatever you want it to be, and so it makes it so that it won't compile.

E

If you try to do that in the future, and it may be, and therefore it made it really easy to find all the places where, where that wasn't true, just throwing that out there yeah.

A

That's a good idea for like a compatibility layer or something that we would use for those Atomics instead of just using you insisted for us to some struct that you have to go through like a ZFS, increment atomic cuz if it's decrement atomic as if, let's get atomic kind of robbery routines on the struct. That's cool I like using the compiler to tell us where our bugs are yeah.

C

Okay, so I think that I see a path forward and I. Think I will discuss this issue further. First, with FreeBSD.

C

Developers in terms of what we want to do and what actual platforms we still want to support for ZFS, and maybe we will try to sort of spearhead this effort on FreeBSD and then propose some changes. Common for all platforms, cool fall, flavors of ZFS.

A

All right, thanks George, do you wanna talk about the deepest share, a problems and hockey solution that you've.

F

Sure so I think we wanted to kind of just share a little bit of kind of what we're learning with with Sharon, FS and and Linux, and also kind of plug a lightning talk that we submitted for the open, ZFS developer summit.

F

But so what we've seen most recently is instances where, if you're, using like the share and FS property and relying on the ZFS share a functionality which is a service to system D service on Linux, if you're, using that, you can run into situations where you can get stale file handles I, don't know how many people are from the Linux side are actually doing this, but definitely on del phix. We we saw this.

F

You can get it from either restarting the NFS server or even rebooting like that. If the NFS server, so in both cases it kind of show up and what we uncovered is kind of, like the way that the ZFS functionality was kind of plumbed into systemd is a little bit cumbersome, but there's there's not a real natural way to do it. It's it's a lot cleaner.

F

On illumos, where you know we can rely on kind of the SMS services and dependencies to be able to plug into it and there's closer hooks into like the NFS functionality directly from like look share that doesn't exist very cleanly on Linux and so as part of like the investigation. What I found was that the way that the service dependent she's laid out as we have when system boots up, you have like a mount D service.

F

There's an NFS server service and those things will start out first and the ZFS share service wants to start it after NFS server is up. It primarily does that, because the way that in Linux you share services and when you, when your runs yep sure a is just Forks and execs a bunch of like export FS commands. So it needs to have like the NFS service running so that anything that you're exporting now gets registered and can get shared correctly.

F

The problem is that if a client happens to try to you know either remount or request a file system that I had mounted it'll get response back from like mount D and the file system itself may not be shared. Yet, especially if you have a lot of file systems that you're trying to share out, you had now have RPC mount D and an F SD that can respond, but you haven't completed the the share a component of it.

F

So you end up with a nice tail and what's weird is like you know, it's a momentary, stale filesystem. If you try to access it after the share has completed all of a sudden like it comes back, although there's a lot of applications that don't like that at all. So so when we started going about trying to figure out like okay, how best to deal with this, we came up with a pretty hacky approach, but it turned out to actually work rather well and that was to introduce a new component to ZFS share.

F

So, instead of using ZFS share, a I introduced, a ZFS share G for generate, and the idea here from the Linux that is to try to make ZFS like closer tie and have a closer tied to the way, the NFS exports, and so what I'm doing now is actually creating an exports file. Instead of relying on export FS the command to actually export these individually, that can then be consumed by the ZFS.

F

Sorry, the NFS server and I've changed the dependency ordering so that now on boot-up, you simply run and generate this file and then have NFS server consume it so you're guaranteed to always have the file systems exported by the time those services are up and running. So you you no longer get the stale file handle it's kind of a hacky solution, as as Matt mentioned it's, it was a prototype to kind of get us to get us out of a really kind of nasty condition, but we found some really nice benefits from it.

F

In particular like we, you know running this on a system to have a lot of file systems. We went from like the share service. Taking 300 seconds to now taking 7 seconds, our plan is to do a lot more redesign in this space that we, we truly believe, will benefit the Linux community.

F

But if there are others and that are doing things on Linux and find themselves in situations where they're seen these kind of weird things feel free to free to reach out to the del phix team, we can kind of share what we have right now. It is like I, said kind of a hacky solution, so it's not completely clean and and we're not ready to upstream it yet.

F

But it's the beginning, part and probably the foundation for a much bigger redesign of the way that the sharing works on Linux that will be coming soon and what we'll go into more detail on the stuff at the at open, ZFS developer summit. There's a lot more problems that we've uncovered in the NFS space that Don and I can share. Yeah.

A

I think one thing we were wondering from folks in this hall is: if other people have run into this problem, and if you have solved that problem in similar or different ways, then I think that would be good to know both from are like how do we design a real solution to this, and also you know, they'd, be interesting to work with George and dawn on the talk they're gonna do for their conference.

A

E

Will say that at Daddo we have solved the NFS compatibility or the NFS use with ZFS stuff by just using NFS as it is without, like the standalone version you're not using the sheriff s property at all, then that we're not using any of that. So if it might be nice, if we could, there might be some benefits there, it might be nice if, like you know, we could actually have some control over that, because you know for for what it's worth.

E

We have had issues with NFS in the past, as I think everybody adds at least to some extent, but having some control over it. Just because it's a part of the file system would would probably be a good thing. So thank.

A

You so Tom, are you like putting stuff into @c, share, tab or exports yeah.

E

At the exports yeah.

A

Ok, so you just put stuff into Etsy exports for your ZFS file systems and then let the and, if that server deal with it there, which is like kind of similar to what George is doing. But George is just saying like we're going to generate that file like at boot time from the proper decent ZFS right, yep.

E

Yeah, but we have like a whole like again, we just have like a separate thing which I forget what we call it I think we call it like X exports manager or something that you know that's whole job is just to do that because we didn't trust, didn't anybody else to do it right so yeah.

G

In general, what George is doing sounds similar to what FreeBSD does where we have e to c ZF s, exports. That is all the sharing of s lines from all your file systems and then mount you just when you change one set of s: reloads Mountain, D and it reads the file and update this list of exported.

G

F

Does FreeBSD generate that Alan is there like? Is it like whenever ZFS like ZFS share? If you do ZFS share and if s it automatically generates that file for you? Yes,.

G

And so I think your concept of having one command refresh at all might actually be a slight improvement. Yeah.

F

I think ultimately interesting yeah. Ultimately we want to be able to like have this. You know individual updates and share NFS to actually do the same file, so it sounds like there might be a convergence happening between like what we're proposing for Linux and what FreeBSD has done. That's cool.

A

B

Yeah, it sounds like.

A

We don't fix, should take a look at what was done previously to see if, like they already did a bunch of this stuff, they were thinking about doing and.

H

We'll say in freenas, which is based on freebsd here is we avoided a huge class of problems, because freenas would never explore things like slash user, nor would it mount slash user from an NFS share, and so we were able to move that up a sharing way later in the boot process and kind of work around a lot of this I. Don't know that that's generally purpose applicable or not don't think it would be, but but we sidestep the problem as well, but.

A

That would probably work for like the dumb phix appliance. Like you know, we are doing that either. This is just for data, you know yeah that are cool, but I agree that it might be may make it hard to integrate with, like the typical installation of especially Linux right.

H

Where you could NFS mount user yeah.

A

Where you could share it, you know, let's share that.

A

All right any any more comments on Sharon if s.

A

All right, in that case, I'm Alan, did you want to talk about D, do stuff just.

G

Very briefly, yes, we have a client who's using D dupe at a very large scale, like a hundred terabytes of data or so and they're running into all the problems that everybody knows about, and they it looks like well, actually have a budget to try to fix some of it, and so we're looking for anybody who might be interested in working with us, buy a contract or whatever to actually implement some of the ideas you proposed in 2017 I think it was during the hackathon of having a ceiling the limit on the size of the DDT, the idea of pruning entries that only have that haven't deduced, yet that I only have a single match to free up space in the DDT and then the idea of doing the log based approach of the on disk format so that you have you know a thousand x, less chops.

G

Every time you have tainted DDT.

G

So if anybody would be interested in working on that, if you get in touch we'll talk more I.

A

Am NOT interested in working on that, but I am happy to kind of you know, go over this stuff that I had proposed and kind of help guide with next steps in one knife, if somebody is gonna, be working on it and I'd love to see that come to fruition and.

G

This will be done directly against the opens that I festmaster I. You say that said if I some Linux right now but against master and then the client will want it back ported to 0.8, but that's their problem, but this would be open to everyone, as as it happens, to.

A

All right that brings us to the end of our scheduled agenda.

A

Other topics that folks would like to cover today.

A

There's a few things on the backlog: I, don't know, I haven't heard much questions about these recently. So there's the feature platform or platform feature matrix, Jesse called people just quit on I know we discussed this may be the last time or two times ago. I think this is still useful, but less useful than it was before. Now that FreeBSD well, FreeBSD is, is you know, swishing to be based on the Linux code, so it'll kind of pick up all that stuff once that happens and I know that Jerry has been updating.

A

This Jerry jhamak isn't updating this based on his work of bringing new features to illumos, which is made a bunch of progress on that. But I think that this is fairly accurate now and I think that we have people kind of working on getting everything in sync just great.

A

But I can move fast, clinician to the integrated of deductions, other questions or things that folks would like to discuss. Well, thanks for the link Alan.

G

You had to go, dig it up, I wasn't sure exactly where it happened. Yeah.

A

I'd, like the time when I'm working in some unfamiliar card base, that's like the hardest problem. Half the time think I know that this is happening somewhere in this million lines of code. But where is it happening?.

A

All right, then, we will have a record early ending of our meeting, and the next meeting will be, let's see.

A

September 17 at the same time is that right, sir yeah.

E

The last minute we had was at 11:00 so yeah, so.

A

It'll be September 17 at 1 p.m. Pacific, the same time as this one and in between now and then you should see.

A

It's also note the acceptance, notices for speakers for open ZFS and then that will be posted on the website so and then we'll start advertising more generally for folks to register registration is open now, but we'll start kind of publicizing and pushing more once we have the speakers, I think that'll get people excited when they can see. You know all the cool stuff we'll get to learn about at the conference and we'll see you in four weeks, Thanks I just.