OpenZFS Leadership, 1 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: April 2023 OpenZFS Leadership Meeting

Description

Agenda: Block Cloning update; 2.2 release;

full notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#

A

All right, it's one o'clock, I guess we should get going with this. uh Welcome everybody to the April 25th open, ZFS, meeting uh I see we got a couple things on the agenda today. um Let's see, there's.

B

C

A

Mentioned he wanted to go first, so uh if you're, if you're ready, maybe we can uh launch into that. Yes.

D

Okay, so um let me start I will share my screen. I want to show you guys something and uh also talk about block cloning.

D

Can you see my terminal.

A

Here again, okay.

D

So uh uh when blood cloning was merged to freebies they had there were some issues.

D

uh There was one data, corruption with embedded blocks and we were overwriting eight bytes when the when we had very little data so between, like 60, bytes and 112. This is where we use embedded block on this specific region that was overwritten.

D

So that was a bit unfortunate. uh This was uh fixed, uh but uh there is some other issue that were found. Those were fixed as well, except for one and this one uh turned out to be a bit harder to fix, and it made me to step back with uh with the implementation and and think about it more because the debuff life cycle is extremely complicated and there's a lot of like Corner cases that we need to uh that.

D

We need to handle, and so this specific Corner case was when we cloned a block, and then we wanted to partially overwrite the Clone in the same transaction group. So, for example, this is totally different case than clonica block and fully overwriting the the Clone block. uh Those are different cases. There are many coronary cases like this, so uh I started to work on uh on adding tests to uh to ZFS test suite, and there will be a lot of tests.

D

Unfortunately, but I I really want to like properly test everything. I can show you uh like, for example, some things I'm going to test, and this is really not not full list, but they want to test like all the interactions were, for example, there is no, the block is destination block is not cached, we clone it. We, for example, override it.

D

Then we read in the same transaction group and then after exporting and importing the pool, we read it again so uh and this is done in a loop, but there are multiple cases like this and basically I I have like four loops and try all those cases uh one by one. So this one Loop is around like 100 tests, uh then we start from overwrite we clone and then read. uh This is a different case. Then we have uh clone overwrite and clone again and then read and stuff like this.

D

So there are many cases uh that are not obvious, and- um and this is uh I would guess this, like 10 of stuff I, want to test I have different tests for like embedded blocks holes.

D

uh We have to test encryption, we have to test interaction with the duplication and- uh and there are many cases, many more cases that need Loops like this uh to be tested, and uh it all starts from.

D

uh From this, so this uh this small change that the no field state can transition to different state, not just be evicted, but actually it can be transitioned to other uncached or read, fill and cached.

D

It changes a lot we need to like handle those cases, because normally, when the block is cached, we assume that we have the data and if we have a clone, uh we already have dirty record, but we have no data, so we need to special case all those uh situations to to handle this to to read the data from the Clone if needed. So if we want to read clone block or we want to override a partially override clone log, then we have to read it.

D

We have to change the debuff state under it, dirty it again, if needed and stuff like that, so uh I spent a few days trying to understand the debuff life cycle and I think I uh a much closer to understanding it, not sure if fully yet, but uh but I think it's much much better than I will be sending the pull request soon.

D

Once I finish, the tests uh I would want to publish this, and hopefully that will help. But another thing: I um block learning was enabled by default, and this was uh this caused some problems because people started to upgrade pools and there is not truly uh going back.

D

But I think uh well. The problematic code was when we cloned the blocks. So uh what I would like to do is to uh I implement it in freebie they implemented a suite that turns off block cloning.

D

Even if the pool was upgraded and we cloned some blocks, we can still disable it and in 3bd, will keep it disabled by default for now, uh but we can still free clone blocks, so uh people don't have to like worry about uh I, don't know going through some like uh restoring the data from the backup before Zippo upgrade and stuff like that.

D

So I think it's like uh like good solution uh in case that people already upgraded pools and and still it's good when people want to try block cloning, they can just enable it and just uh just to do just try it out and see if it works and uh and disable when they want so the freeing code I think it's much less complex, so that should be okay.

D

So that's block cloning, uh not sure if you guys have any questions.

A

Yeah thanks for the update, so so how was that disable switch implemented on FreeBSD? Sorry.

D

Bsd to assist control, so it's only for FreeBSD. For now it's it was committed to FreeBSD, uh so uh I'm sure there is equivalent in Linux for configuration like that, but this was yeah. This was freebies this specific and it's only in freebies. It wasn't uh uh I didn't send pull requests with this one. Yet yeah.

A

So, on the Linux side, this is one of those things we haven't wired up. Quite yet all right, so I don't think anybody who's used the black cloning functionality there yet and it's not wired up today. This is calls, so that's also on the to-do list um for us to work once we get all the other bugs nailed down, but it's great to see the careful test Suite being put together for it. That's excellent.

D

Yeah I think I underestimated the Buffalo cycle, initially, unfortunately, uh but uh hopefully now, I have a better understanding and uh okay, um so uh we discussed rate limiting uh I think on the last call, uh and uh there was a lot of pushback to to implement rate a meeting to be hierarchical, so in order so to be able to to configure rate limits. uh For example, some like uh a single limit on one of the top data sets and then basically just uh divide this limit into some uh lower data sets and initial implementation.

D

Didn't do that and uh and again I had to uh but I'm grateful for for the motivation for the pushback, but I I had to like uh yeah think about this, some more because one one of the concerns was uh performance and log contention.

D

If we decide to do this uh because what we want is to work for this to work exactly as quotas, so we can configure, as you can see here, higher limits below, but we still want this higher limit to be enforced and not the bottom one so I think with quota it works the same. You can configure higher quota than your parent, but still you have to uh you won't be able to go over the Department Squad and uh so- uh and we also so this is. uh This is prototype of the new design.

D

It also includes uh using separate properties for uh separate properties for all the limits. So now we can uh in theory we could do this more granule. So, for example, include separate rate limiting for metadata operations.

D

That would request that I didn't want to pack everything, because the the previous implementation had like a single rate limit property and every all the limits were squeezed into this one. So I didn't want to overload that uh I'm, not sure I'm, not yet convinced if we want to have separate rate limits for metadata operations, but it's definitely now much easier and much cleaner to do and uh with this new design and the performance uh I think I found a way to to avoid log contention.

D

uh I would I will use readlock around the structures, so all the iOS shouldn't uh should be able to to run concurrently, no matter on which data set you you work on. There is a bit much more work to do because we have to update the structures at every single uh level that is involved in rate limiting.

D

So if you configure like three rate limits uh on three different data sets going down, then we have to update those three structures, so it will be a bit more expensive, but uh if you have like two three, it shouldn't really matter uh from my tasks. If, if you have Freight limiting configured to very high number, there shouldn't be a difference between having create limits and having having no rate limiting and having great limiting configured to really high numbers.

D

There shouldn't be much of an impact on performance, so let me show you how the hierarchical limits work.

D

So we have those configured so at the top we have 10 megabytes of read bandwidth configured then, on raid bus we have six megabytes right through eight megabytes and rate for bar 100 megabytes.

D

So if I try to read just from a single file from rate.

D

Let's say 50. I should get 10 megabytes right.

D

Yeah so 10 megabytes, then, if I read from uh like uh bus, I will get.

D

I will get six megabytes right and uh but if I read concurrently from Buzz and Foo, one have six megabytes one half eight both should do around five megabytes because the higher limit is 10 megabytes right. So let's try to do this.

D

So we will be reading from those two. So if we would read individually from each of them, there will be six or eight megabytes, but because we are reading concurrently from both we should obey the higher limit, which is 10 megabytes. So each read should be around five megabytes.

D

So that's the and what we also gain with those hierarchical limits is that we can treat each limit separately. So we could configure like read limits here and operation limits higher or something like this. You could we could mix and match which wasn't possible with the uh with the initial implementation.

D

And the last thing is that when you, when we try to read from full bar this one, even though it has a hundred megabytes, it should obey the higher limits, which is eight megabytes.

D

Yeah, so uh that's pretty much, it I think this uh prototype works pretty well and I think it meets the. uh It means the feedback from the from the last call any questions.

C

Paul, you said you're using read blocks about how your updating, stats and bunch of atomics.

D

No, so actually so to update each individual structure. I use a ride lock, but right lock is so, for example, let's say we would have oh something like this.

D

And we will configurate limits only here.

D

D

So now we have.

D

So now we have rate limits configured here and here, but not here not here and not here correct, so we have two rate limit structures so uh to update each structure. So those two we need right lock, but only on this individual structure.

D

So there will be a lot contention going up when we update the structure, but when we, uh when we go up and look for those structures- and there is only read lock to go through uh through the entire hierarchy because it can be complex, I guess.

C

So for uh for those who don't use any rate limits, there will be uh so a red lock acquisition for every parent data set for which is not exactly free, but.

D

But it will be hard to measure I guess.

C

Well, I just remember: there was a I read the right look and I read mostly lock, so it mostly could be more efficient. Even so.

D

But we need right lock when we like do renames uh when we configure limits and stuff like that, so those are uh rare operations, so we we could use, read most logs. Definitely uh that we can. uh If we still have a little bit of time, we can uh try to do a simple test.

B

D

Let's see so now, there is no rate limit configured at all.

D

Okay, so now this is basically Spurs file, so it will be quick, uh but now, let's try to configurate limits to some High number.

D

So we can use Terra uh yeah, it should be, it should be close, but of course yeah. If, uh if you have like complex hierarchy- and we have to update like we have a rate limits configured on the top data set, we will need to. There will be some log contention, updating the structure from all dials coming coming up.

D

But of course, even if you have all those like a single data set and all the all the files in a single data set, you would still have the exact same log contention on this structure, uh because every I o would need to update the structure.

C

So, for updates it's it makes sense for those who enable rate limits. They kind of agree to overheat, I'm more worried about case when the rate limits are not used and we have multiple concurrent threads and uh right. You know in parallel, trying to supply.

D

C

Same lock, even for read no.

D

No, so actually I think there is a shortcut so uh because each data set uh points at the rate limit at the data set. That has the right limits configured so, for example, this data set has a pointer to this one. Sorry.

C

If there's nothing, there will be nothing yeah.

D

We don't acquire any locks, so it's if it's not configured, then there is no overhead at all.

D

Okay, any other questions.

D

Okay, uh thank you for letting me go first, because I have to leave early, so thank you guys. Take care no.

A

Problem thanks for showing us the current status and stuff, it's pretty cool. Thank you. Bye, guys.

D

A

Let's see uh next on the agenda, what do we got? I'll, probably hop back to the top here, so an update on uh the GitHub action, Runners right and some work being done. There is Tino.

C

A

A

Maybe not so even a little bit of background I know a little bit about this um there's been some work to you know, stand up some additional CI infrastructure to get better test coverage on some other architectures uh Power, PC um ar64 or that kind of thing and there's some resources available at openstack, so I know I'm sure Tino can talk about it more, but the hope is to be able to take advantage of that kind of functionality, and you know set up some new Runners and just get better test coverage there, because right now we're still pretty x86 Centric in the test coverage.

A

So um I don't know about any more of an update than that other than as a work in progress. So maybe maybe we can Circle that back to that. If he's uh jumps on the call later um so something I do know something more about. Is the upcoming 2-2 release? Let me talk about that for a little bit, so we've got pretty much all the core functionality in now in the master Branch for the 2-2 release.

A

um There's a lot of good stuff in there. A bunch of cool new features, I mean we've been accumulating them for for quite a while now, so the plan is to cut a 2-2 release Branch, um hopefully in the next couple of weeks, the sooner the better so expect release candidates out, for that shortly um to start stabilizing and testing, they should be in good shape.

A

They'll just be branched off the master Branch as a 2-2 release and I'll put out um fairly frequent RCS and when we're go through a couple of those and are happy with the stability uh we'll tag, a final 2-2 release, I think things are in pretty good shape at the moment, um but you know we'll give it a little bit more testing all right on a stable release branches. So if people are available to help test that and make sure it's really completely solid, that would be great.

A

um Are there any questions about that? Maybe I can address.

E

A

Okay, then I guess just watch for a release, Branch here coming up and then hopefully near future.

A

um So following the 2-2 release, or at least making that Branch, um the other noteworthy thing is getting in some major new features and one of those that you guys talked about last week. Sorry I missed that, but that's getting the OS X support in all right. So once we cut that Branch um I think I'm watched last week's call and I agree. It sounds like a good plan forward to get all those changes merged um and get them stabilized and that'll be a good time to do it.

A

uh After that release branches cut, and then we can, you know, just move forward with that and get it integrated and uh you know, stand up any testing infrastructure. We need eventually, but it'll, be great to see that kind of stuff get integrated, so that should be coming up too I. Just wanted to. You know mention that briefly, as an update.

A

But let's see what else I feel like I'm blowing through these kind of quickly, so people have questioned their topics. Let me know uh I.

C

Think the only other thing really.

A

Here on the list um and I don't have much to say about it: either is uh raid Z expansion um I'm, not quite sure, what's going on uh with this one, maybe others have some more insight, um but I know this is still. You know, work that Matt did. That needs to be uh he's more code. Review needs more testing, uh it needs to be rebased, so there's still a bunch of work there to do time to get it finalized. I know.

A

uh People have, you know, done some uh testing with it, but it's still not quite wrapped up, but it would be great to push this work forward and get it rebased and tested, because I know this is a feature. A lot of people are keen on. It. Just hasn't quite come together yet, but it's a.

E

A

A

So I don't know if there's other things, people want to talk about or questions um I guess we've got some.

C

E

For that still.

A

E

Brian I want to touch base very briefly um about the issue. I opened last week, um the pr the deadlock.

D

E

Issue yeah, given that you have the the revert lined up and it looks like Richard thinks it is his issue. uh Are you going to go ahead and revert that that, for now, yeah.

A

That was my plan was to go back and revert that right away. I was looking at the test results just this morning, for that revert and, interestingly enough, it seems like z-test is failing reliably.

C

A

C

A

I was about to merge it and then I got stuck. Oh, my God. We got to look at those test, failures and see why so so there may be some other conflicts or things that need to be in a while in there before we can get it uh integrated.

C

But yeah I.

A

Think we should revert that and whatever else may inadvertently depend on it, that we didn't notice, yeah yeah all right, but um so I will try to take a look at that or or I. Don't know if Richard might take a look at it too or if you want to take a look at it. That'd be great, but yeah I re-ran the z-test a couple of times and they're failing pretty reliably but I haven't had a chance to pull down the exact traces and look. Why.

E

To see if it's the same deadlock, yeah.

A

I mean who knows right, hopefully not yeah, but yeah I was I, was ready to revert it and say that's an easy thing to do. Take it out, but uh yeah we got to do a little leg. Work. First, okay sounds good.

A

um I, don't know of any other significant issues in the master Branch, but if you guys know of anything, that's you know open bugs are issues that are major. Please let me know um we'll want to get those fixed um and you know obviously get that was fixes folded into a release, but I think we're in a pretty good spot. For that.

A

um Well, I, don't think I have anything else. Unless there are other topics here, it might end up being kind of a quicker meeting this week or months.

A

All right well nobody's got anything else, um but.

B

Sorry I was on the phone.

E

B

um I missed part of what you were saying there, but it's uh could you just reiterate for where the what the plan for 2.2 is.

A

uh Cutting release Branch shortly, okay,.

B

So everything we can get forced export in there or do you think, would you rather not.

A

I would like to get it in.

B

There, if possible, so.

A

Like um if we my.

B

A

Looking at the pr is, there's like one outstanding bug: yeah.

B

We fixed the other two over the weekend and there's just the one where, when we cancel the Zio and it invokes the the done callback, uh it's expecting there to be a reference left and there wasn't and we're just trying to debug that. But we solved uh thanks to your help, actually figured out the one where uh it was never actually getting into the force unmount code, because the unmount happening first was hanging because of the suspension and wasn't doing the right thing. Yeah.

A

I think the fix you guys settled on there was a good idea, adding the additional axles. um I was going to mention suggesting that of the pr at the time, but then I forgot so I'm glad you guys ended up in the same spot yeah it.

B

Was uh because we originally designed this on FreeBSD uh back in was that like 2018 or 2019, um where force unmounting of a file system is the thing you can just do uh it didn't? Have this particular problem, um but making it work on both has always been a goal. It's just. We hadn't managed to figure out where it was getting stuck, uh but now we have, and so yeah that's just that last one and hopefully uh between Marius and myself, we can get that sorted up very shortly. Yeah.

A

If we can get that merged, then I think we can Branch the 2-2 release right away. Okay, please scan it out for folks, I I know that's uh you said the feature that's been worked on for a long time now and I know a lot of people want it. It's on my short list of killer features too.

B

Yeah and I think most of the other stuff we have in Flight will be longer and can wait for the next thing. Yeah yeah.

A

I think we've got a bunch of stuff queued up for the next major release. We have no shortage of cool features on the horizon, all right, they're just going to take a while to come together, um but.

B

Yeah, uh because Alexander did you already talk about what we were looking at or.

C

Oh I I was going to ask her. What's the plan of Brian, how long do you want to marinate her before a release actually.

A

I mean I think a month, maybe would be reasonable if we put out RC's weekly or something like that. You know we get whatever fixes we need in there. But if we cut an RC branch and we let it marinate a month or so I think we're probably in good shape, because that the master branch has seen a lot of testing and it's pretty solid. So um but once we've got RC, the people again can jump on and kick the tires there and it always helps build confidence.

D

A

I guess the only other thing on a release front worth mentioning. Is uh we put out a 211 release just a week or so ago, as well with a bunch of back ports and stuff, so the stable release, kind of update so I expect that to roll out places.

C

Even 12 11's already.

A

It'll probably be 12 too, at some.

C

Point I wasn't 11 Target I think it was good, it was merged.

A

Yeah yeah, so 11 was released and then we started accumulating patches for 12 already so.

C

There's not not too many, but a handful here.

A

A

I'm, sorry Alan, do you have something.

B

Else no uh I think that was I. Think almost everything else we had pending has been merged now yeah, it's quite stuffed quite a lot of stuff land. In the last two weeks, yeah.

A

It's been busy but I think we're in a good spot.

A

um Definitely ping me when you've got a update to the forced export patch and I'll, give it a kick the tires there again and make sure it works out for on.

B

Our homework great yeah, that's I, think the hardest thing about testing it is uh Hardware always behaves differently than not Hardware yeah.

A

And everybody's Hardware seems behave a little bit differently, so getting a wide range of systems is helpful for sure yeah.

B

Because yeah, actually, the thing we wrote it for originally was for emulated disks uh kind of a S3 like thing that, like what uh the devil's people were working on.

C

B

That's it for me.

C

A

Well, if we don't have anything else, uh I guess we can call it a meeting and you guys can get an hour or half hour back.

A

All right: well, thanks everybody. Thank you.