OpenZFS Leadership, 22 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: June 2022 OpenZFS Leadership Meeting

Description

Agenda: DirectIO PR; ZIO taskq's for multiple pools; multiple encryption keys

detailed notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit

A

All right, let's get started, um welcome to the june 2022 open, zvs leadership meeting.

A

um I think we have one item on the agenda uh update from brian atkinson on direct, I o um any other items that folks want to add to the agenda now.

B

Yeah, I have some ideas on how to have the interface for having multiple user keys for encryption.

A

Awesome all right, let me add that to the agenda.

A

Somebody is adding it.

A

um Then I saw also some added zeo task view scaling. uh Any other items folks want to add now.

A

All right, then, um why don't we get started with brian's update on direct? I o.

C

Hey so uh the last, I think probably anybody who's been updated on. This was the developer summit and good news. I think that the linux side is completed now, first and foremost, I'm here to solicit reviews on the pull request.

C

There was a couple.

A

I am not hearing brian, oh, it looks like alan is still moving yeah.

C

Hey, can you hear me there.

A

C

Go sorry, apparently, my internet connection is unstable at the moment, um but I don't know what the last thing you guys heard, but there was, there was one part uh we had talked about were doing stable pages, and that was the concept of say: we've taken a user's buffer, we've met or pinned it in. You know using the kernel and we're going to directly write it out. Well, they could still change the contents of that buffer.

C

Unfortunately, after going down many kernel rabbit holes, it doesn't seem you can actually do that through the kernel. uh The main trouble comes in with the users pages, possibly being anonymous pages, which is commonly what they would be. So this just means, like the pages aren't backed by a file or anything so like if they malek put fill them up and send them.

C

There's no way in the kernel to write, protect the page table entry of anonymous page at least no way. I could figure out how to do so. That's very unfortunate. So what I wound up doing instead is I added a module parameter, specifically just cio direct write, verify and what it does is. If this is enabled it's another stage in the cio pipeline, so right before it goes to say: okay, let's update the block pointer in the done function, because we've successfully committed this out.

C

If you get a checksum failure there, it's like no, don't do that. This is failed and what I wind up doing is sending a z event or z. You know, so it can be uh seen that way and also added as equal status, lowercase d, and you can see these checks on verify failures that happen there, but it's never committed to disk. In that case, it's just like no something changed, and so we're not going to do this anymore, so it just bails out and returns the en val.

C

In that case, um the good thing is that does work on freebsd, though stable pages do work so, unfortunately, with linux, we're kind of stuck.

A

So the semantic that you would be implementing is basically like your application is required to not modify this while the right is in progress right, but we have this safety, that's going to double, check it and fail that and fail the rate exactly.

A

um The alternative would be to not require that they abstain from writing or saying from storing to that memory, while the raise in progress but- and the only way to do that would be to like to be copy. The data somewhere.

C

Else exactly yeah, so I mean it's least a safety net. In there and again the the cool thing is, I know one concern we had. Is you know if you do, if a user does unintentionally manipulates it and it writes it out, um there is the idea. Well, you have a checks on failure. You go to read it. It's like hey, something's, all wrong here and there's no real way to coordinate that back to what caused this is it.

C

You know an actual hardware thing going on at least with this module parameter in the z pool uh status. Now it can be like no does exactly what happened. We didn't commit that right so, like please, don't manipulate your buffer, you know and would the check be on by default.

C

I have it off for now, um because I kind of view that, as like well you're doing like an extra checksum on every right, it runs.

A

C

Yeah, so I mean it's costly, I I would like to have it as a thing where it's like you opt into it, but I mean I'm I'm vote for people telling me that no defaults on and if you're feeling comfortable enough, don't do that.

A

Can we tell at that point in the cio pipeline if it was a direct, I o right and yeah.

C

Only do it in that.

A

C

That's all I do yeah it's because I do the dmu write policy. I flag it. It's like there's a directio right here.

D

So that's the only commentation to what's the ramification to the pool in the event of these right failures.

C

D

C

D

They're just they're just kind of ignored, so like we get right errors, then everything just continues driving on and yes.

C

Yep yeah and there's cleanup done as well just through the normal cio done right. So if we set it, it's em val. It's like all right, clean up the meta slab. It says, though, the right never happened.

C

At least that's how it should work pretty sure when I told but again this is where reviews come in, make sure I didn't mess that part up, but everything I've tested and looked at it's like all right. We seem to clean up correctly. We use attorney and ballot that state, so consumers, in that case.

D

Would they, unless they're watching for these checks on errors, wouldn't realize that their rights didn't actually happen.

D

Oh, the ian val goes all the way back up.

C

D

So all these all this is done synchronously at the time. The right is happening, there's never like, and it's never an async path right. Exactly: okay, yep.

C

Then again, there's a they also added that event like it's there as well. um I put a test case in there as well to test this out for linux, as well as the freebsd side. Again, the good news. We can write protective freebsd, no problem, so I've added a case for that.

C

um That seems to be working, fine and the only other thing that's kind of lingering. At this point there is this one freebsd error, which I'm also soliciting help for that here as well.

C

um For some reason, just with reads uh the first like like if you're doing, multiple direct, I o reads, requests one after another, the first one for some reason when it goes to fault hold, the pages seems to fail quite often, and I have no idea why there's just a failure that returns up from the bm fault, quick, hold pages from freebsd, and in that case, because it's like the the linux compatibility freebsd code, they return to e-fault, which is what I do in that case and percolates back up to the vfs layer, and then it sends it back down.

C

But it's two separate reads of smaller chunk sizes.

C

So it's super strange. I don't know why that's happening.

C

But I do have a separate branch that I'm can point free of bsd people to with the test case that also don't steam message like hey here's, some printouts. That shows exactly where we're failing.

C

You know. While it works, it's not working correctly, so any help. There would be great because I think that's really the one lingering issue at this point other than people commenting on the pull requests, trying it possibly breaking it.

A

um So for the verification thing, um is it faster to do this double check sum versus just versus copying the data, the performance.

E

The same thing.

C

The performance is, what kills you on the b copy or the the copy.

D

So yeah it's worse than the check summit calculation.

C

Which is the whole reason we started doing the odor direct. You know it was the memory copies that were killing us.

F

Yeah is it still worse if you have one of the stronger checksums on, or is that just true for say, fletcher iv.

C

Yeah I mean it may yeah I mean I haven't, run a whole sweep of like what's the cost penalty associated.

D

Is the um can the copy be like opportunistic like? Can you detect at the time that somebody's actually making the second modification and only do the b copy, then.

A

You have to be able to detect that yeah yeah.

D

G

What I'm wondering.

D

Like can you do that.

A

Yeah, like I mean I wonder rather than so, I guess I have two questions. One is around: what should the default behavior be and the other is um if they do not if they're, if they, if, if we are verifying by default, then is the right behavior really to give them eio or ian val, or is it to just like?

A

Do the b copy, then, like you do the checksum? You do the second check sum and it fails, and then you say: oh it chuck's been filled. So now I'm going to be copy your data and then write it out. I'm going.

C

A

About that then.

C

A

C

Send it back to the pipeline, though yeah.

A

Yeah yeah, I mean um I'm sure, that's like more complicated to implement, but um I guess one of the questions is what's, which is the right data to write the original version or the version they just modify it again. Either version is correct.

G

A

The old one we probably.

G

Don't have anymore if they overwrote it, though yeah.

A

It's just that the checksum has to match it right all right, so you're saying that when we get the checksum failure, should we reset the pipeline and write out that new version with that correct checksum rather than ring out yeah yeah I mean, or you could like reach instead of copying, you could just retry loop right, just be like well I'll retry again until I get two check sums the same, and but I mean obviously that could infiltrate yeah, yeah and.

C

There's still, no, even after you do the checksum verify say it passes, they can still go in there.

A

C

Like if you're just like.

A

Twiddling like flipping some bit on and off yeah.

C

A

You could have the same checksum before and after, but in the middle it had some other value. That is not the value that was checksum, so I mean maybe, given that you know, maybe it doesn't make. Maybe the e in val does make sense, because you know we're. We can't guarantee correctness right in the face of concurrent stores. We can just try to detect some.

A

You know modifications and you know maybe what you so given that you know. Maybe what you do is you say for the question it should be on or off by default. Maybe it's maybe it's not on or off by default. Maybe this is probabilistic and you say the default is we're going to check one percent of all of your rights or you know some percent, um so we're gonna double check some.

A

You know one percent of your rights and then let you know if you're doing the wrong thing right. The requirement still continues to be like you have to not modify it, but we're just going to give you like a little bit of a double check so that you know you know you're not get. If you check some error later on and you see that the counter was bumped, then you know that, like the application probably is doing bad things and if, if the counter is not bumped, you don't really know for sure either way.

A

But even if we don't check that, even if we double check something every time we couldn't really guarantee that either way.

C

If the count is zero, so you're saying possibly just periodic hey, let's try the check real fast.

A

Yeah, um because.

C

It doesn't have the cost.

A

Of being on all the time, but you don't have the advantage of if we have it off by default, no one will ever turn it on yeah, like you have the performance, the performance would be probably negligible um performance impact of doing it. One percent of the time would probably be negligible, um and that way you have at least some chance that people who are just using it like naive. The concern is naive. Users right, like naive users, are like, oh great, like oh direct.

A

Let's like let's go direct it that'll make things, go faster right, like everybody on the internet, says user directs make things go fast. They don't know what they're talking about they don't know. They may not even know that you're using zfs or whatever, but, like I read some advice on the internet. That says oh direct means go fast, so I'm going to use odirect and then like, and then you know a month later. Oh zfs is just full of trucks and errors. My discs are fine.

A

Every like this is all broken right, um and so we want to like have some kind of safeguard for those naive users, um which is why I think, having some kind of verification, maybe not 100, of the time verif like double check some, maybe it's one percent. Maybe it's 10, like I, don't know um some kind of verification for those naive cases so that at least then like, if we get inundated with like uh zfs, says unexplained check some errors. We can be like hey check this count.

A

Oh the count, the count of you know wrong. Double check sums is, is non-zero uh you're using okay you're using o direct wrong? So yeah? You know.

C

Yeah and I do percolate that counter up to the top level of vw as well, so it's not like per I mean I don't know yeah. I felt more into like intuitive to me, like top level counter, but yeah yeah. I guess the the heuristic for um determining that percentage.

C

I think we'd have to think on.

A

Yeah, I think you would probably just like set it as high as possible to value this as high as possible, while not impacting performance yeah. You know, measurably.

A

Okay, I mean think about like for every 100, writes you're, doing 100, checksums, plus a lot of other stuff yeah. So.

C

A

Like if for every 100 rates, you're doing 101, checksums plus a lot of other stuff, you know the same amount of other stuff. You know the impact performance impact is no more than one percent right.

E

The question is like how much absolute overhead do you get with a checksum because, like if you look at it in a latency histogram, then say a 50 50 split and we'll have one peak at the latency without doing the extra checksum and one pic the other one. So if you have you're doing something, latency critical, you're gonna end up with, like always assuming that you're check something right so on average, on average matt you're right. But if you care about like p and 90 whatever.

A

99 then yeah yeah.

E

A

Care about like p99 or p99.9, it could matter.

A

I I agree that that is true. I hope and assert that it is not important, but if, if people know that it is important, then uh then definitely contradict me. um They can promise to use direct. I o right and set it to zero percent yeah.

E

A

I mean look if you're. If you're well informed, then you can change the value to whatever. Then you know you're informed enough to change this tunable to whatever you want.

C

So we still keep the tunable, but that's the adjustable. I guess scale.

A

C

G

C

So often we're going to double check on this count. Okay,.

E

Can we make it so individual application threads can opt in or do we lose the context? I guess we lose the context deep in the cio pipeline right. We don't know which, which threat issued the call.

A

uh I mean you probably could like you probably could do it on a per data set basis or something. um I think it would probably be fine to just have a system wide tunable, okay,.

A

Since you know we're we're still like it, this doesn't the tunable doesn't control any kind of correctness behavior. What the expected interface is. It's just like you're always required to not do stores on direct like. While you have a direct, I o right in progress and we're just doing a little double check for you.

A

um If you get it wrong, then what's going to happen, is you're going to get? Is that like the right will succeed and when you go to read it you'll get a check. Some error so, like you, told us to write this data, but you can't actually read it back, um so you know we're just trying to help you not get we're trying to help. You know when you might be in that circumstance.

A

So that you can know what to fix right when you get when you're like oh, I got a lot of tricks and errors. Then you know, oh, what I need to fix is not my disks, but rather my application or my use of odirect.

C

Okay, yeah, I think it makes sense to change it to account parameter then yeah.

G

So I guess I just add that I think most of the applications on linux already have this expectation right where they know they shouldn't touch something that's doing direct. I o right, while it's in flight. This is like not.

A

In the degrees file systems.

G

A

Right, like your results, would be non-deterministic.

G

If you're modifying it while you're writing it out right so yeah, I think most of the main consumers of this interface that like already exist right. The big ones already know this and do things right. So I guess we're mainly worried about the someone writing a new code or turning this.

A

On oh direct means go fast and I want to go fast, so I turn it on and I don't really know anything about the application. I'm just like you know. Sys admin and I have a bunch of random applications.

D

A

D

Kind of like the behavior is for other file systems on linux. If you use odirect and concurrent consumers.

C

Well, typically, there's not a checksum associated, so it's kind of it is what it is right. The data is disk just kind of right now and then.

A

We'd be doing as well as they are yeah exactly exactly.

D

Like is there so, in other words, is there a value in us like telling them that they're doing it wrong when the typical you know, consumer of other file systems effectively, would never know they're doing it wrong other than they just get like busted data.

A

Yeah I mean you could argue like hey, you want to do a direct we're going to turn your checksums off yeah. I'm not sure, that's a great idea, but I think that there's an argument for.

E

It okay, the problem is ultimately that we don't attach to the block on disk or the block pointer that points to the block whether the block has been written by o direct or not. If we had that bit of information, then we could like surface to check some errors in the correct way differentiate between all that.

E

I don't want to argue changing on this format, though, but I think that is the crux of the problem.

C

I think we did talk about that at one point. This seems like an easier.

A

C

A

At that point, though, like if you know it was a direct, I o right and you get back data with the wrong checksum. You still don't know if that was because they modified it after you generated the checksum or because the disk actually mangled that.

E

Is not true that was true.

A

It's a hint that you know you can tell us.

E

This might be because.

A

Of that, but that's about it.

A

Yeah and if we wanted to do that, I suppose the way we could do that was the um the per data set feature flag activation thing like we did for z, standard, where you know we flag a data set as soon as any block's ever been born that use direct. I o for write and we'd, be like you know. If you get to check some error here, maybe it means something, but I don't know if that makes sense.

A

In my opinion, I think it'll be sufficient to do the you know because, like we're saying you know applications, you know they they wouldn't have deterministic behavior. If they were doing this thing, if they were doing the concur modification today so like they are very likely to not be doing that, and so you know we can just kind of assume that that is fine, but we're you know like in zfs. We try to go above and beyond. We have checksums.

A

We assume that you have the checksums on, um and so you know part of that doing better than other file systems is like we're, gonna check and make sure and record.

A

If you have this double modification, you could also argue like maybe in the double modification case like well, I mean, I guess getting the error back to the application, would would give you a good clue like it would make sure the application dies sooner rather than later, and then surfaces the error rather than just having as yet, rather than just like recording on disk so yeah, I like what you've done there.

A

I had another thought, but I've always lost me. I think that this is um this approach seems good and like better than what other file systems are doing. Oh the other thing I was going to mention. um If we wanted to go even further another thing that we could do to help folks would be to have a tunable.

A

That's like uh don't verify like don't verify, checksums on read, and so that would let you like. If you ran your application, you did these rights and it did the concurrent modifications now you're like oh I'm getting checked some errors, my data, isn't there I'm so sad, we could say well, you can set this this tunable to just say like ignore the checksum error and give you the bad give you the data that is inconsistent, which is just as good as what other file systems are doing.

A

C

A

Done as a patch for zdb, specifically to extending my hackathon project to be like, I have this file, it has check some errors extract. The whatever is there anyway.

E

A

E

Throwing in this, so there is actually like a practical concern. For example, if you use service and receive somewhere, I think I think if we encounter it check some error doing zeros, then we stop the send right. I don't know whether there is a flag to tell it to send corrupted data there isn't there is yeah yeah yeah.

A

It replaces the data with a pattern.

A

Anyways, I think that's kind of not required, but um be something nice if, if if this is even at all like relevant, I mean- hopefully none of this is irrelevant and we're just like going the extra mile. That is not really required, um but I think especially given that you've already implemented this. The like double check sum checking and all that stuff um keeping it in and having it on.

A

You know, one percent or a few percent of the time uh would be nice.

C

Okay, yeah and it seems like the count I mean, makes sense and we can just set a default of. I don't know 100 or something yeah and just I mean I'll, have to go back and update the code, obviously to increase the counter, but that's not hard, it changed the module forever. I mean just to make sure it makes sense um yeah.

C

So the only other thing that could, I would bring this up here, while I'm thinking about it, it can trip you up in this case, like okay, the user is adjusting the buffer out from under you, it's like in the compression path and I'm sure this happens in other places too, but for sure on compression path like if you have you know, debug on and the data has been manipulated after the compression has taken place, then the abd compare will panic on you because it's like well, that's not what I just you know created a compression on, but that's just you got debugging code on at that point and if we wanted we could, I could just you know, add another parameter to it.

C

You know the abd call just be like this is a direct. I o right don't sit there and do that during debug or we can just be like. Well again, I don't know anybody. That's running debug code on production anyways.

C

I could be wrong, but.

C

um I could catch a developer if they that were to occur, but again I don't know why they would be manipulating that.

A

I guess again: that's that's pointing out that they're doing it wrong yeah, and so it's probably a good thing rather than the bad thing. Yeah yeah, although like it's not our bug, so we probably shouldn't be panicking.

C

And again it's if we wanted to adjust it, it could simply just pass in another parameter: abd call yeah.

A

Yeah I mean if you're using compression like you're already paying like that's a pretty big cost and you're already kind of copying the data around, and that's just that's just it once.

C

It's compressed right, it's just it's that abd compares to the original buffer when it turns it back. It's like hey that doesn't match it's just debugging code.

A

Like does it make sense to do zero copy, um direct, a or writes if you're doing compression at all, like? Maybe if you have compression on, we can just be copy your data first and then compress it. um My concern is like uh if you're, concurrently modifying it.

A

What if you're like does? Are we sure that the compression like do all compression algorithms access each byte of the input buffer? Exactly once?

A

If not, then? Presumably, if the buffer is changing out from under the compression algorithm, then the output results are completely garbage right and so like. In that case, you might have something where it's like yeah, like I did, the compression it it output garbage.

A

Then I did the checksum I'm texting the compressed data, which is like obviously correct, like the the checksum will match. I write it out. I read it back in I'm like great, like I checksum is correct, um but then, when I go to decompress it, the decompressor is like this input is garbage.

A

um It explodes which looks very much like a zfs bug.

A

That seems kind of concerning to me.

C

C

Don't confuse when it decompresses it, it would blow up how, though, because I mean the checksum is still valid right. Oh yeah,.

A

The checksum is valid and correct, but the input is garbage because the compression, because the compression code, like the compression code, probably assumes that the input buffer isn't changing while it's doing the compression and if it is changing, then probably all bets are off as to its output.

A

Right, like you, could imagine a compression thing where it's like you know it's building some table. That's like. ah I noticed this string at this offset and then later on.

C

A

You know uh is this: is the string at the current offset the same as the string at this rate and.

C

A

A pointer into the earlier data right and so like it might make the wrong decision about whether this data is the same as that data, because it's like that data has changed between when I first looked at it and when I'm looking at it now.

C

A

um And I mean maybe the output would be like uh decompressible, but just give you the wrong results right, yeah and the checksum would be right and you'd be like great, like it checks them correctly. Here are the results? Here's here's your data back and it's just like it's it's neither the it's it like. It is no version of the correct data. It's not like. We got your data as of some point in time. It's just like completely mangled, because the compression decompression, like you know, lost its brain.

F

hmm Wouldn't the checksum still fail, then, because it checks them before it applied the compression no.

A

The checksum is the only.

F

A

Oh after the compression, that's.

F

Right yeah, sorry, I remember this now: yeah.

D

F

Think that's like for this.

A

D

On the compression, because we create another buffer, I think we've already copied the data and we do like a push transform and then that gets popped off later. But like.

A

We cop we, we compress from the input buffer into a new buffer. I think yes, sometimes we create a new linear buffer, because.

C

The compressor.

A

Can't consume an abd so sometimes.

C

A

Data first then compress it into another buffer.

F

Yeah, I believe we do that all the time. Actually I.

A

Don't think any of.

F

The compressors.

A

Support reading abds, yet no so yeah all.

F

A

We just if we make sure that we do that so yeah, like let's copy the buff, let's copy the data before doing the compression sounds like we're already doing that in some.

C

A

C

I think that in.

A

C

It's abd borrow buff copy so like what I was saying is when you do the abd return, buff, that's just so the debug kill or gets you there right, because it's like it's a contents of change. It's like so.

A

What what I'm saying is like abd bar above copy does not always copy the data into another buffer right right. It doesn't if the buff, the original buffer, is already linear, linear, yeah yeah. Exactly so we'll.

A

What if it's a scatter with exactly one entry, it's.

C

A linear page, we do ryan help me remember this. I can go look at the code and try it over, but I think what we do is just a linear page. It's still a scatter.

G

C

That's my recollection.

G

Too, as we still treat it as a scatter, a special kind of scatter, which is one page and map it, so we can map it and then copy it. I don't think we copy it. I think we're we try to be clever and don't so. We might need to make sure we make a copy there yeah. So.

A

I think that we should just change that to say you know we always make the copy, and maybe the way to do that is like in this code path, rather than using abd bar above copy. You use like the abd iterate function and do the copy ourselves it's like. We know. We definitely know that a copy was made because, like we allocated the buffer, we.

C

A

The abd we copied the data into the buffer, and now it's stable data that we're passing to the compressor.

C

I guess do we know how big of a performance hit? That would be say if we're linear rougher to begin with, though,.

A

Well, we only need to do this if it's not, we only need to do this if it's a user data buffer right like right, you only. You only do this so you're, saying.

C

F

A

If the abd is pointing to user the user's address space, then you do this. Otherwise.

C

Yeah we flag, so we could do that. Okay, yeah! I can go in there and adjust that code.

A

Or maybe I mean, maybe you could even do this could be part of abd borrow. Buff copy is like if the abd knows that it's user data, then it's like, oh, if you're doing var, buff copy, then and it's you and it's pointing to user data, then we're definitely allocating a new thing. Copyright.

C

Yeah, absolutely okay, yeah.

C

Yeah, but still I mean we, we I'm glad we talked about all that that was good, uh but the the return buff still could trigger. Also.

A

But that's because.

C

The source is still changed out from under you, and I remember like in that debug code they're. All it's doing is like. Does the source return.

A

C

Do they match yeah they're.

A

Not a big deal.

C

A

Would just I would change that to not make that assertion if the, if it's yeah, if the abd is pointing to user data yeah right like I, don't think that we want to.

A

I mean it is a corner case. It is a debug thing, but I still would rather not make zfs panic because use like applications misused. It right.

G

Yeah, I agree, even with the debug build there shouldn't be anything, an application can do to cause it to panic. Yeah I mean.

A

Panic should users should be able to know like if the system panics in zfs, I could file the zfs bug because there's a problem with zfs.

C

C

So yeah, I think that was the only major things I had with it and obviously, like you know, in reviews people look at it and taking it for a test driving, I mean again, I feel pretty good about the linux side. Making these modifications makes 100 sense um and yeah if we just get anybody to look at that, one lingering error and again I have a branch, my own branch- I can point them to it's like hey run this there's the message: output, at least to show where the failure is starting to occur.

A

Yeah, this is great thanks a lot for your work on this. I know it's been a long time coming.

C

Yeah sorry, it took so long.

A

That's all right.

A

um Yeah thanks for bringing up these issues here, I think um I mean. Hopefully none of this really matters because applications aren't doing it, but I definitely feel better. Knowing that you know, zfs is kind of doing as best as we can um with these kind of tricky semantics, yeah, absolutely.

A

All right uh next agenda items, um zio task, you scaling from multiple pools yeah that was mine, um so we have a tunable zeo task. You batch percent uh defaults to 75 percent and defines how many threads we do. The zio processing on, which is the bulk of that is compression and checksums and things of that nature.

A

So it defaults to 75 of your cpu course the idea being to leave some system time left over, so that, if you're writing to a highly compressed data set or something that you know your system's not going to not be responsive because all the threads are busy doing compression uh the problem. Is we create that many task queues for each pool?

A

So if you had two pools- and you say where zfs sending from one and receiving to the other uh you'd be using a hundred and fifty percent of your cpu cores for decompression and compression, if you were, you know, I think the user was reading from a pool that was jesus compressed and writing to one that was it's standard compressed uh or you know. In other examples, you know.

C

A

Has seven pools, uh then that's a lot more threads than they actually have cpus yeah um is. Did you have a solution proposed for this? uh I was curious. What other people thought the right thing to do there, like with the dynamic mode that we have, we could try to. You know lower the number of threads per pool when you add more pools, but you know, sometimes people have some pools that aren't busy and some that are and then how does?

A

How do you make any of this make sense in the case where you know I have two really busy pools and then I have my boot pool and I don't do. I want the same settings for all of those and how does that work.

A

um I I don't really know um george uh might have thoughts about this, but uh would it make any sense for those task queues uh to be global rather than purple, or is that just inviting deadlocks galore.

D

I mean that seems that seems like the reasonable approach, but then, like you know like I wonder what things like you know like when you have our pool and then a really busy pool like. um Could you get into like that deadlock scenario where, like the real busy pool just, is like starving out our pool.

A

Yeah like like what, if you have a I mean, if you have a pool, that's based in another pool, you know we're not really like.

C

That's not like super supported.

A

But that seems like it could be problematic and.

H

A

If one of the pools is hard drive based and has high latency and the other is nvme based and has really low latency, I mean I don't think that there's a there there's a. I doubt that there's a real starvation issue, because the task is our fifo.

A

Like I mean you know, they're work, stealing and whatnot, but like more or less it's like more or less fifo. um So I would think that the fairness would be reasonable. My concern is with like there's dependencies because like right now you know you can't have dependencies from one or what is it like uh within one ta within one zero type like say reads: you can't have a read that depends on another read you can ever read. That depends on the right or right. That depends on a read because those are different task.

A

You groups, but my concern would be like if you have multiple pools and you're doing like say this like stand and receive from one pool to another, is it possible that you introduce a dependency? That's not allowed where it's like.

A

You know I'm doing a I'm doing as if a so I'm doing a read, I mean I guess you shouldn't right, like you shouldn't need to because the at the cio level there is no interaction between the pools. Unless you have a pool base in another pool, then then you're probably out of luck.

A

um I guess the other question is inside the task queue. Does it have the information about which pool it's for or does it avoid that? Because it knows that the whole task queue is for one pool like do we.

C

A

Every data structure to attach a bunch, more information to it in that case, if we share a test queue between all the pools, I don't think you need to attach any more info.

D

I think the problem you may encounter is that now, like some of these task, q locks, which are already really hot, get really really really hot.

D

um So alan, when you kind of like, is this something you kind of encountered in practice or yes,.

A

So uh in two different cases, uh the one case was just a user doing the center received from one pool to another, and the system was basically becoming almost unresponsive because they're using 150 of the cpu cores for compression decompression, because two pools each using 75 and the other case was just the customer that has seven separate busy, pools and uh noticing you know their starving each other for cpu time or just overall we're creating.

A

You know they have 112 cpu cores and they're, creating task queues for 75 of them times, seven pools and it's just a lot of extra moving bits.

A

uh But because that is a tunable you can only set before you import any pools it gets hairy to. You know, try to adjust it as you import, more pools and so on.

D

Yeah and for like a customer like that, I mean: did you get a sense for what they would like to see like? Would they want to see all the pools like reduced the number of like right threads or are they wanting something like? Oh, I have these three pools that are more critical, so I really want them to have more cpu than the other. You know uh four pools.

A

uh In this seven pool case they're all the pools of equal importance to them, but I can definitely see that being for other people yeah, I'm almost thinking like. Should we have a z-pool property that controls this and then, if it's not set, we default back to the system-wide tunable or something and does that make sense, or does that not make sense?

A

You know that helps maybe with the r-pool case where it's like my r pool only needs you know x, threads or whatever, but all my.

C

A

Need you know, 25 of my cores or whatever.

H

The problem we have, if we have so many different types of io, that you would have too many knobs to do- that's probably not well.

A

Yeah, that's. uh This is just the zio uh batch test queue. That's not even mentioning the you know. We got what the read normal and read high priority uh issue and interrupt and then all those for write null uh ioctl, etc. Yeah um I mean I think that there's not an obvious solution here. uh If you, you might try making it a system-wide task, you rather than per pool and like see if that seems to work.

A

um Yes, the other option is making that tunable more dynamic, so you can change it at runtime and just lowering the you know when it's uh the dynamic task you code goes to be, like you know, should I spawn more if you're like? Actually I'm over my quota, I should shut down uh test queues as they become idle or before, like.

D

A great hackathon yeah feels like a great hackathon type project to go even for like either of those options right like to go as a global cio test queue uh or a global task implementation, and then one that like allows you to adjust.

A

Cool: um let's, uh let's go to the next topics just so we can try to give like five minutes for each of these other two topics: um the next one, multiple user keys for encryption.

B

Yep, um so I was looking at basically the user facing documentation. What not- and it looks like the the cleanest way to do this from the user's perspective, would be to use the at notation on the properties uh and have multiple slots. So you would have like key format at zero at one et cetera for slot zero slot, one for different user keys and obviously all that would be stored in the metadata and we'd just wrap the encryption key, multiple different times, which is similar to what we're doing now.

B

Just do it more times and then you'd add a command for zfs, add key, which would uh allow you to add an additional key um zfs for move key that would prompt for a key that would then find the slot that is using that key and remove it or remove slot, which would then just clear that slot is the high level of that it's largely modeled off of what luke's is already doing with their multiple key slots.

B

I believe they offer eight key slots. I was going to go with that until I saw a technical reason otherwise, um but that's the the high level view of that.

B

I can show what I added in the man page. If, if you want me to.

A

Sounds reasonable, I think um yeah. If you want to get feedback kind of before you before it's ready for a code review, you could um maybe open an issue or push the mailing list with the with the man. Page oranges,.

A

B

That's pretty much what I've got.

A

Any questions or comments on that.

A

All right, then, the last one is one one: six, seven, nine, I'm not sure who, whose was this. That would be me.

F

That's that's the bug where, if you do receive an encrypted, send it panics in a no b reference.

F

um I have two solutions that are implemented in work and I was wondering if I could get opinions on which of them is less gross. um But basically the bug is that um there's a special code path for in-place decryption of um d-nodes in um the arc read code.

F

And what can happen is that you can have an arc right that happens and dumps out the buffers while it's going through arcread. Well, it's debuff right, which calls are great but anyway, and so you can be trying to do a read on one of the buffers while somebody has temporarily said it to null and your life is very sad.

G

I have two different solutions:.

A

G

Structure is being concurrently, modified or somebody might set it to know while you're trying to work on it.

F

uh Specifically, what happens is that the.

F

The p, a b d and r e b d in uh arc, buff header, get set to null temporarily while going through the path of arc. Right ready are great, are great done and they don't really check much before they do that. They just said it so like it sets it to null in, I think, ready or so and then change and fixes it in done.

A

And I don't know what the locking is supposed to be like there, but should it be holding the arc hash lock on that entry doing that, so that you would.

F

Think that would work. It does not work.

A

F

F

But I tried that and it does not help.

F

The two solutions that I have at the moment are that I have something that refactors the arc right code path, so it doesn't temporarily set it to null um it's somewhat invasive, but it works.

F

um The one I think I prefer is that um the thing that does the decryption in place doesn't really actually need to reference that buffer. um So I just added handling that notices. If the buffer is null and if it is, it just goes and takes it just allocates a temporary buffer.

F

um I could do that unconditionally, but sticking an unconditional allocation and didn't necessarily seem like the best performance choice.

D

The code path, where we're we're unencrypting directly.

F

Into the buffer, is that the case, specifically, what happens is that um we try to uh use. We try to decrypt directly into the buffer. Yes, that's what that code path is specifically for, and that's that's happening in the context of a receive.

F

um The bug only happens in the context of a receive, um because I mean.

D

Like what I'm thinking about is like, if the, if the header is already in right, then how is it discoverable? um It should be an anonymous header right and we would have released it as part of the write.

F

I can go into detail on how this happens, but basically, if you trigger the quota recalculation code that goes and dirties everything um there's a window where this can happen, I don't have a good explanation for why honestly.

D

But I mean, I think, that's probably why there's no there's no locking in place, because the assumption is that you shouldn't be able to like find this buffer.

F

D

D

um That's uh yeah interesting.

F

I I would prefer a better solution than this, but um I will take not panicking as a starter.

D

I mean I I would be. I would be interested in kind of like the details of like I have a right.

C

Now kind of I can link.

D

That um um and and then maybe I I can provide some some better alternatives, but um yeah I'd like to kind of understand the scenario first, but I'm I'm cool with what you know if you're doing like a temporary copy. If that's a solution, you know to keep us from panicking. Maybe that's what we need.

F

Silliest thing is that it's not, it doesn't even need to copy really well anyway,.

D

You just need to read not no.

F

F

It needs a buffer that is not null somewhere yeah.

D

Yeah feel free to send me like.

F

Your write up or point me at it. I just linked it in the notes. um It's not everything, but that's most of it, but.

F

Well, it actually just it mostly summarizes what I just said not in much detail. I have a lot of experimenting and things to do and don't work um mostly don't.

F

A

Cool um it sounds like we need to dig into this a little bit more, at least, to to understand thanks sturge for um volunteering, to take a look at this rich.

A

um I think we're almost at the end of the meeting time. um I wanted to just mention that we're we're still working on planning the open zfs conference, um we're looking at dates. uh It looks like it might be in early november, um we're we're still trying to plan an in-person conference, uh as folks have indicated from the last meeting um and uh we're still, the opportunities are still available to help out. uh If, if anybody can help with the planning and logistics, um that would definitely be appreciated.

A

um We're kind of going forward uh with the resources that we have and uh trying to do, the best we can um with uh with getting the the venue nailed down and then we'll work on catering and stuff.

A

um So looking forward to seeing you all uh in the fall and hoping that kovud will be uh calm, then um in the meantime, the next meeting will be in four weeks, which is uh july 19th. um We uh delphics folks have a company event during that time. So I think we might not be uh some folks might not be able to attend, then um brian or brian ballendorf. Would you be available to lead the meeting on that time on the 19th.

G

Yeah, that should work for me all right: cool.

A

A

uh Great well uh thanks everyone and um goodbye from sunny san francisco, where they it's reached, 95 degrees on the solstice today, which is why my I'm on a different camera, because my camera overheated, so I had to fall back to the macbook camera, so have a great uh enjoy your summer and we'll see you again in four weeks, see ya.