OpenZFS Leadership, 12 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: October 2022 OpenZFS Leadership Meeting

Description

Agenda: releases; docker overlays; hackathon ideas

full notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#

A

B

A

Start with that, then uh we're two minutes after so we can get started.

C

Okay, oh I, just dived into this area pretty recently, but uh problem series exists around for three years or about uh the problem. Is that original Docker driver or native one for ZFS written in goal and written pretty terribly? It runs the forks, separate processes to do things like ZFS list that takes forever and practically burns down the system, uh science. We have now go developers and nobody else dive into this.

C

Our middleware team found us a solution with using overlay FS on top of ZFS, which seems works much better and that's the way you use it for other file systems, and at that point we found that there were already patches hanging on on GFS for like three years to implement couple features required for overlay, FS, actually work on top of ZFS.

C

Oh it's! It's not terribly specific to GFS just couple requirements uh that there are two PRS I mentioned: the first 9600. It's practically changes how ZFS handles um a rollback, I think right or a rollback yeah that how it wipe Direct locations in in the Linux kernel uh seems pretty straightforward. You just need somebody who is good, Linux, VFS understanding to say yet. Yes, that's that's correct. That's the right way to go, so there are one cosmetic thing that you'll be able to fix.

C

Just somebody can do it if otherwise, the pipes looks good like it's, it shouldn't affect anything else and it's not exactly.

C

Just probably that's the right way to do sync: uh instead of making a directory cash regularly go to ZFS, this is fully disability. This is valid for interesting. Jcfs in case of rollback, can just go and wipe all the directory cache in one go. That would be probably also more efficient, so I think it's just a good push to go. It's just my qualification in that area is not enough, just to say how would I deserve. We need somebody to take a look and just push the final button.

C

There's a patch yeah, okay. Well, there were some comments from Matt and Brian, but they were a while ago. So some some more comments from them or somebody would be good. The other patch is more complicated. uh It's odds, uh support for couple, Linux, specific flux to rename apps to which allows some fancy things such as swapping two files, atomically within one operation and seems like rename and create placeholder. Instead, most of things are used by our layoffs, but again not specific to Overlay fs and one of problems.

C

There is that it requires couple more transaction types to handle those operations in zeal and right now it's implemented only for Linux and it reduces couple more features which are implemented only for Linux, which means once you do this operation. One of those operations pool can never be imported in anywhere else.

C

So a couple Solutions I see one is easy: Implement those things for other operating systems, which is not very useful. They require work. Otherwise I think it would be just easier for everybody to just to drop those additional feature: flags and just replay Zeal as good as gun. Up to the point, it's supported, I think switch of operating system is not doesn't happen so often to bother so much. Okay, we'll lose couple seconds of activity on previous OS of activities that only applicable to that OS and nothing else and I think it would be easier.

C

Just I commented there, but again, if somebody has other ideas who wish to review in those two patches, we are actively interested and actually would be happy if, after three three years of waiting, it would land within a month.

A

Yeah I was looking at the rename at to one as well uh for my presentation at the um staff Summit coming up, um because it's again something that relates to the container support that Clara added the Linux namespace delegation, stuff for um and yeah, especially the case where you're doing a rename, where you're also creating a whiteout for the old version of the file, so that on the overlay app, that's the other lower file doesn't show through.

A

um In that case, it's interesting that Linux implements the whiteout by making it a character device with a specific ID or something. Oh.

C

Yes, zero, zero IDs.

A

Yeah, whereas yeah I was looking at something similar on FreeBSD, where we would possibly add whiteout as an actual like V, node type or whatever, uh but that would require income implementing or incrementing the version number of the the Z of s posix layer, the SPL so like the ZFS version, as opposed to zpl version or ZFS upgrade versus Z4 upgrade.

A

You know how the file system has versions up to five, and then we haven't changed anything since forever, whereas the pool we did feature Flags after version 28 or whatever, but for the file system itself, we've not ever had to change anything since I.

D

Remember, hearing about all that being updated to version 1000 When Future frogs were originally being developed, but it just never happened.

A

Yeah and I think there's another PR somewhere that talks a bit about the idea of having feet. uh Sorry, it was on the wiki. There was a an idea, a project proposal for the hackathon of uh implementing feature flags for the the zpl part of it as well. um I, don't know what the right thing to do. There is um the Linux specific way of doing whiteouts seems slightly sub-optimal, but I don't know uh if it's worth the pain of you know bumping the version number of the zetiel.

C

Why do you think it's worth election, but what is incompatible there I'm not sure I'm getting well.

A

uh What we'd want to do is add a new object, type like right now we have you know regular file, directory Sim link, Etc uh we'd, actually add a type for whiteout.

C

uh Those separate type which would not be a laser character device, no other different type on FreeBSD use.

A

It right or yeah if we added a type that way each OS could decide how to implement that. Where on Linux I can be the character device thing around, FreeBSD can be a regular whiteout like we already have support in ufs for uh or whatever.

C

Oh yeah, but both probably are supported already. You know a check character device should not be a problem anyway and I. Guess that uh different type on freebies I guess it works, but I never used. I have no idea yeah, but just or you meant some something more unified that would translate on different toys. Different things, Yeah.

A

So, basically, if ZFS knows that that object, ID is type white out and then in the OS specific code, it can be how we expose that, whether it's you know setting the flag on a v node or whatever, or making it a character device or whatever.

C

A

Is is more consistent that way.

C

I wonder how otherwise overlay fs and Union FS are compatible to each other, except the White House right.

A

It's not worth it, but.

C

Yeah at this point, um it just got quickly Burning uh from a Docker site on Linux. We never used it on freebies, G and honestly. I would prefer maybe to implement the native ZFS way. We have proper Docker driver implementing it uh properly from whatever language, just through GFS native calls, without parsing the fs list, outputs and maybe then it would be nicer, but just we have no gold developers to do that. Stuff.

C

So I'd like to just attract people to those too no and the first just needs to review the second. So we need to decide whether we need features to handle those things. What people's opinion about maybe just drop the features and, as I've told in my last comment and just replace Zeal as much as as good as gun. Otherwise, we'll get a quote. We can never test cross platform, so it's but it's theoretical. It would be better, but practically it's spend of time and inability to test and just difficult.

A

A

So each I guess each item in each transaction in the cell is supposed to be separately Atomic, so decomposing the rename to swap thing into just two or three renamed or whatever, wouldn't actually work. The way we want so.

B

Yeah I think uh some kind.

A

Of a flag to just say this is skippable, if you don't understand it or whatever, maybe it makes sense there.

C

No, like uh I'm, not sure it's keepable, I think it's we just can't. We don't know what happened with next operation. Next operation May depends on the separation, so it's.

B

Practically we can.

C

Replay replay look as long as we understand everything as soon as we find first, not understandable. We can just dump in locks. Okay, we got some, so we don't know actually.

A

Stopped replacing yeah, we started playing the Zill and you know most times when you're importing there isn't a zild replay anyway, like if you're switching os's. You normally do that with the nice export and so there's nothing to replay. But if you crashed on Linux and then imported on FreeBSD, you would get a little warning and at least you'd be able to import the pool. Oh.

C

Yeah I would prefer that versus a requirement to install back Linux import export and then everything so I would prefer that pay for that small data loss.

A

Yeah I think the the number of times that will come up is small enough that it makes sense to do it. That way, especially versus the pool, is unimportable.

C

A

All we hit, but yeah um I know the the second PR you're talking about there. I was hoping to get people to to look at as well uh as part of my talk, so I'll I'll push on that for you.

A

um Next I think Richard you've had uh a bug you wanted to talk about.

D

Oh yes, so this actually came up the day when I was testing the stop playing and wanted to populate uh L2 orc uh from a file system that was already populated as I just reloaded to Kernel modules.

D

So I I had ordered kernel modules built from master and then ran this command, which basically finds all the files uh that are in a directory in any of its children and then cats, and to Dev no uh with 256 CAD operations in parallel and all the cat processes ended up deadlocked and the um killer triggered and ZFS Swiss also deadlocked. The issue appears to be one of the locks that we're taking Arc buff access.

D

Well, I. Don't have more information right now, so I remember seeing on the agenda. There was something regarding tagging uh version 2.2, which would be awesome, but until this bug is fixed, we can't tag for master, as this is well, it's not present in the current state of Maurice, but it is present in master and we just can't release this sort of progression.

A

Yeah and you, you said, you suspect, it's the change to the debuff hash scaling. Oh.

D

Yes, I do suspect it's that debuff has uh arrays. There were some similar complaints about that patch when it was proposed for 2.1.6 and it ended up being dropped from that, so that that's the main candidate at the moment, um I'm not sure why maybe there's something about how Linux might not like having its uh locks in a virtual mem in kernel, virtual memory, I'm, not sure.

A

D

I, don't have anything else to say about that, except we can't tag 2.2 until this is fixed.

A

Yeah it'd be great if we could uh get a a smaller reproduction case so that we could rank a test that will fail for it uh so that we can track it easier.

D

Yeah I was thinking it would be great to have this in other DFS test Suite, um but it I need to do more digging.

A

Yeah I do think it's just accessing that many files, concurrently, where, if we just created a directory of a thousand files and XR capital, P 256, it would hang again or.

D

uh Maybe I I haven't tried yet I haven't had time. Okay,.

A

uh And then yeah my question was about: are we still planning to do a 2.2 release soon, uh but without Matt or Brian? Here, I, don't know uh if anybody else knows anything about that.

A

uh I know Alexander. That was something you had asked about back in the spring or summer, so to try to align with your releases right.

C

Oh yeah, we are uh oh we're already working on a new release of tuna scale, which is now based on 2.1 still so, but it's uh it's so we are right now releasing beta 2. It should be rc1 soon so like would it be released right now or we could switch on it before the rc1? If, if we don't, then we have to stick for 2.1 for longer and backboard thing. There right now would be the most convenient time like months after maybe already less convenient or late. So.

A

Yeah uh I, don't know I'm, don't I didn't look last time to see like how many release candidates and how long between, when we started tagging a release and we actually pushed it out like when we did 2.1 what that time. Scale was like, uh but I expect that getting 2.2 out in less than a month is probably no.

C

I I, don't know exactly a release. I mean more like a bridge point right that we could follow. So science will also also have time between well during our RC cycle and so on. But at least we need some point. We could track some Branch yeah.

A

I think that's reasonable to have done in the next couple of weeks, especially if we plan to have it released before the end of the year. We need to get that done.

B

A

Is there anything else anybody want to talk about.

A

John, did you get finished all the NFS uh stuff you? You were working on.

B

um Yeah yeah, that was good and it we pulled it back in, and so we have it on our end.

A

Yeah I think we got that all working, so it works the same way on FreeBSD as well. So that's good.

B

E

B

Did have a quick question, um let's, regardless of the upcoming Summit, did we talk at all on one of these uh calls about the hackathon or any goals for that, or is there like a spreadsheet for ideas.

A

Under is usually a spreadsheet I don't have a link to it. Handy.

A

uh And I know there's a Wiki page, although the Wiki page I think is from five or more years ago,.

B

But yeah there's still some good ideas.

A

On it, as it turns out, um yeah don't have.

A

The latest uh spreadsheet, for that.

B

Right there I'll ask on our end to see if anybody has like if Matt or someone has ideas as to where that lives.

A

Yeah I think it's just in somebody's.

A

uh Public facing doc service account uh is where it looks like. There's, yeah I got the one from 2021 and 2020 and a couple other times, but it doesn't look like there's a new one. Yet.

D

So we're going to talk about, there is one thing that occurs to me, which is um some people might have noticed that I've been doing a lot of static analysis stuff lately, um I know, I've been somewhat subtle on the YouTube tracker, but um so I've actually been uh researching. Defense attack, analyzers and I was like I want one that has uh X Y and Z, and it's uh Brian mentioned to me. He would love it.

D

If so, we could have every pull request, static, analyzed and after searching uh through what's available, I finally found one that seems to check all the boxes. I mentioned this uh to you all in the other day, um so it's called sonor Cloud, it's free for open source projects. It can integrate with GitHub to run on POS and even branches. It allows you to dismiss false positives. You can second custom rules and they have one uh who saying that go to is bad and that that needs to go and it can also check multiple languages.

D

So the Python scripts can get some attention too. um So I'd like to set that up in the near future, uh so our pull request will be static, analyzed and any regressions can be caught sooner rather than later.

B

Richard is that something we can install locally so that if we do get a fault or a failure, we can iterate locally or.

D

um Not uh well it's a cloud-based thing and they have a local version, but the C version actually requires you to pay. uh So it's not something we really could run locally.

D

um There is something else called code Checker that can be run locally, that hooks into uh Crane and static analyzer and a few others, but that's something I'm still researching.

B

D

Now, if you were to run scan build to get Crane's results, you just get inundated with a bunch of reports with over 100 reports and it's not very clear to tell whether or not anything is uh new. Aside from the issue account going up, so it's just not useful for local use, otherwise, you're accessibility looking for issues, um but that's something I'd like to find out in the near future.

B

Yeah, if I'm not mistaken, when you open a pull request, you can Target a specific check right, so I could say, run this static analyzer, so I don't have to run all the other tests. If I'm trying to iterate on it.

D

um They have some tools like that: spot. Stick commercial, but.

B

I'm I'm, referring to like there's a tag Alan. Maybe you know like when you, when you check in a a change, you can tag it so that it runs a specific test.

A

B

Know uh all right, Brian's, probably the right person to ask.

D

On that, oh that's! Why? You're in the ZFS test, Suite.

B

No, not in the test Suite just like on the commit itself, you can say like I, don't want you to run this test um and ignore all these other tests. So.

D

That you don't remember those.

B

Okay, that'd be something I would I would uh advocate for because yeah, if you do get when it does fail like, then how do you know you fixed it other than writing all the tests? You know that seems like yeah.

D

Well, it should be very quick in terms of getting feedback, um but anyway, this is something that I'm planning to look into in the near future. I hope we will have that uh maybe around the time of a summit, and then we can iterate on that.

B

Richard, how many pull requests have we opened so far for this, it seems like there's been a half a dozen at least you.

D

Know that's why I said I was being subtle. um That was a joke as well. Well what in particular static analysis, stuff or yeah.

B

The ones that you've the the ones this has helped catch things.

D

um That's a good question: I haven't counted but I'm, going to guess uh between 30 and 40.

B

D

And some of these poor requests, Owens single issue things basically I tried aggregating different issues whatever and some.

A

Like all the class of issues, kind of thing.

D

Yes and then we actually can see a few commits and the uh histories where I said uh yeah I meant to solve that, but the other commit but I miss these cases. uh But um yeah.

D

There are quite a few things that have been uh fixed and actually, if you were to go on or there's a comedy link on uh the main page for the GitHub repository and if you click on it, you can actually see some information watch, a defect density and if you scroll down, you can see uh defect density over a period of time. So you can see we were at. It was actually a little higher than 0.65 when I started, but the day I started.

D

I, basically just made a bunch of improvements and that actually lowered the defect density from any 0.71 and then over time. You see it drop to 0.57 to 0.3 to 0.22, and it's been stuck there for the past week as some of my uh pull requests, uh which fixed into a number one I needed, I'm a business which I did on a week last weekend and the other one is a bit scary because I'm touching Lua, so um it might take some time before I can be committed, but there I think.

D

Okay, there are 106 outstanding issues, I think the uh pull requests I have opened, solve 23 of them so and then there are many other ones that not necessarily uh bugs sometimes it's a false positive but I'm like oh, is this an opportunity for some cleanup and uh other times I'm still trying to decide on and so on and so forth, but uh yeah a lot of these things have been analyzed and something that actually surprised me was that I actually had expected coverty to catch quite a few uh issues that without some things with false negatives, and there are certain things that it should have bought because it caught other things like them: yeah, I, just didn't and others said: I can analyzers quote them.

D

For example, Klein and gcc's uh based in static analyzer, which actually surprised me, um but I thought that was somewhat interesting and I'm actually of the opinion. I've actually made two observations. Well, uh I guess conjectures uh one is at about half of the reports are all false positives.

D

The other is that for every report we have, it probably is at least one more bug in the code that isn't reported um right. Otherwise, that's my thinking at the moment. I think that's everything. I have to say about that.

A

uh Alexander, just looking through the old hackathon ideas, I saw one here that relates to another docker-ish uh thing, which is support for ID mapped mounts or basically, if you clone a data set being able to override the user IDs that own all the files, so that you could, you know, clone a data set and give it to a different user or a different container without having to actually you know, recursively, Cho and all the files and just handle that at the file system, level kind of like how you know if you mount a MS-DOS FS, you can just say instead of root, all these files are owned by this user.

A

uh Is that something that would fit into your use case at all or just.

C

It wasn't uh arise to my point. I may ask other guys, could you throw me a lean copier or whatever it is I can do that.

C

A

A

Christian, you came in late. Is there anything you want to talk about.

E

A

A

Everybody's out of things to talk about, then I guess uh obviously see most of you at the developer, Summit and then the rest of you in four weeks.

E

Okay, Ellen sorry I have I, have something I'm on my phone right now, so I didn't I, don't know when to bring it up. I asked an IRC a question about this: credential struck that we hold in the Linux specific code Brian. If you could answer a bit.

A

Sorry, okay, but.

E

Yeah yeah does anybody else know.

A

On Linux about the the what's it called CR hold the lock.

E

A

E

Richard already.

D

But I just said everything I knew, which isn't enough very basically. My answer was this needs to be inspected.

E

A

Are you seeing a particular issue there.

E

Was mostly uh I ported some some code, uh some internal code and was wondering okay? What are what are the rules about this CR hold and why are we doing it in some places, but not in others where it would actually be more appropriate? According to my understanding, so.

A

D

Basically, we inherited uh Prudential infrastructure from open, Solaris and just hook the links infrastructure into it and I I don't really well.

D

My guess is that people have just done whatever worked since then, even if it's not necessarily uh according to whatever the rules were in open, Solaris and in Linux, and it just happened to work and as far as I know, we don't really know much about it.

E

Yeah, the weird thing is, as far as I can tell in sudaris: we actually don't do the Sierra holds, because that is done by the EFS code. uh I mean I, don't really know open sonares, but that's what I gathered from from looking at the code on GitHub, uh so yeah yeah I, tried to reach Brian and talk. This through would be worth a cleanup, I think yeah.

A

Yeah I'll poke Matt about getting the uh hackathon idea. Spreadsheet started uh because there's a bunch of good ideas from previous years and I think I have a couple of new ideas as well, and uh hopefully we can get some interesting stuff out of the hackathon.

A

All right thanks everyone and see you in four weeks.