OpenZFS Leadership, 17 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: December 2021 OpenZFS Leadership Meeting

Description

Agenda: ZFS Interface for Accelerators; Native encryption; https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#

A

All right, let's go ahead and get started, uh welcome to the first open, zfs uh leadership, meeting, postcard, post uh dev summit. um I hope everybody had a great time at the development summit. uh Four weeks ago um we have a couple of great ideas on the agenda today.

A

uh First one we were just discussing um so uh folks who attend regularly, may not benefit from this much, but there's a lot of people who can't attend or attend sporadically and the meeting notes are really useful to them um to see what was discussed and and decide. You know when they need to go, look and read, watch the video for the details of something that was interesting um thanks a lot to seraphim for taking those notes for the past uh several years.

A

I think, um and uh he needs a little break- he's been at it for quite a while. So uh we're looking for volunteers and thanks to christian for volunteering to do to do that. um That's much appreciated and uh I'll send out those meeting notes. Folks, don't haven't seen them already. I send them out to the mailing list developer mailing list uh after the after the meeting.

A

All right cool thanks a lot christian, um so the other things uh jason um jason lee uh had some things to discuss about using zfs with hardware accelerators um and I think, jason. You said you had some slides. You wanted to share to talk about your idea.

A

I see you're there, but you're muted, still connected on a phone can.

B

You hear me now there.

A

C

Cool, can you see my screen.

A

C

Alrighty so uh uh hi everyone, I'm like matt, said I'm jason lee from los alamos uh and the plan for today is just to talk about the zfs interface for accelerators and a companion piece called the data processing unit services module.

C

So this is going to be super high level and is more meant as a feeler to get people uh interested in our work and to see if we can uh to see how many people are interested and maybe get comments on it. When I eventually open a pull quest in the open, dfs repo.

C

C

The motivating reason for why we, this work exists, can be summarized by this graph graciously provided by brian atkinson. uh At the lab we like to run ray z2 z, pools with compression and tracfoning enabled. Specifically, we like to run gzip as our compression algorithm on this graph. You can see that with no compression we get about eight and a half gigs, a second of throughput across a large number of io threads, targeting a single file and slightly under that line. We have lv4 compression, which is about at a 7.5 gigs a second.

C

So we have some performance degradation due to the fact that there is now overhead for compression, but it's relatively small about only a 10 drop or so. However, at the very bottom of this graph, you'll see a nice straight red line, which is the jesus throughput.

C

uh Well, uh that's a massive drop-off which we would prefer uh to not have so, and the basic reason for this is because compressing data fuse up on a general purpose. Cpu is just really really slow.

C

Fortunately, there are devices called computational storage devices, storage devices and confidential terminal storage processors which are currently being developed. They have hardware implementations of operations such as compression and checksuming, and erasure coding, which happen to be to overlap incredibly well with the features that vss provides, which makes vfs an ideal candidate for offloading operations.

C

uh As a side note, I do know that the intel there is code for intel qet operations. Unfortunately, that doesn't run on my.

D

C

So that's a slight problem, so what was done to enable accelerators for all machines was to add a trim layer to zfs so that data might can be offloaded onto hardware accelerators, while the ios and their corresponding abd still follow the zfs code paths.

C

uh This allows for alternative implementation of the same functions so that, if, for whatever reason accelerators are, are not available, the data sitting on storage can still be read out or read in or out, because all of the data looks like as if it was.

D

Written by zs.

C

With no changes.

C

So uh here is a diagram of the right pipeline, with the i8 enabled at the top uh is a highly simplified diagram of the cio right pipeline on the left. We get a cio plus, it's abd. We send it to compress uh do a bunch of checks. uh Do it transform push, I think uh with and then pass the compressed data to check something raise z, uh the v does disk or video file code to issue the I o and.

D

C

Data lands on storage, which in this case is nvme so with via enabled what one data enters the uh the I o right, compress function.

C

The function, the the zfs shim layer will see, hey via is enabled easy. Ia compression is enabled this data is not offloaded. So let's offload the data onto the off the hardware accelerator and compress it. So when this data is offloaded, uh the via interface returns, a handle for for zfs to keep track of this handle will be stored in the abd as a void star. So uh so it's opaque and vfs has no idea what it is and can't do very much with it except pass it around.

C

So then the zio is passing to the trapsome uh stage and the handle is passed into the via checksum function, which in turn calculates the checksum of the data and then, similarly for uh rate v, the date the data is went into trunks and the parity columns are calculated uh and for rate, the.

C

In addition to the abd data itself, it also will hold a uh another zia handle inside the razero script.

C

And finally, the data is passed into the the vw disk or uh pipeline and written out to the to storage.

C

So just during normal operations, data will never be brought back into memory other than minor stuff like uh return codes or checksums, because checksums are needed in block pointers, but in general data will that the actual data in the abds will never return to to memory. So there will not be a cost of constantly transferring data back and forth between or on the pcie bus.

C

However, there are errors in in the operation, for instance, he passed in it. He passed. The data is passed into the checksum function when you have request flexure for and this hardware accelerator for some reason doesn't support, flexure 4, then, yes, the data will be passed back into memory and then offloaded later. But hopefully this is not a normal occurrence, so you shouldn't incur that cost too often.

A

This is really cool. Can I ask a question about that? Previous slide sure. um So um it looks like in the diagram here like the um the xeo. The zio pipeline is still the thing. That's like driving this, so every stage of the pipeline is still executed like normally and then um it's just passing this hand like after the compression.

A

Then we go back to doing the pipeline. It goes to the next stage. Maybe the next stage is checksum and then um and then it's going to pass the handle to the zia. You know um call like callback or whatever is that? Do I understand that correctly.

C

A

Correct, okay and um and how integrated is this with the abd stuff like, for example, let's say that um we add some other stage in here into the pipeline that needs to process the data, um and maybe it needs to process it on the main host cpu uh by by like just like calling the abd you know, iterate or abd copy um kind of functions. Is that gonna is the abd code then going to know that, oh, like I have this zia handle?

A

I need to like copy it back from the uh you know, the external board memory into the main memory, so that I can do it or like do we risk um like looking at the main memory, which is now out of date, because it was you know not we don't see, we don't have the compressed version or whatever.

C

uh So it is not integrated with abd's at all, except for that uh zia handle.

B

C

The abd, uh if you want, for, if you want to bring data back, you pass a fully allocated abd into the callback and, assuming I this is implemented correctly. The data will be unloaded into the abd according to whatever uh abd type. It is whether it's linear or scatter or okay,.

A

Cool yeah, I guess my main concern is just like. um Like you know, the normal case is going to be like not having this external processing unit and we want to make sure that we don't like break the use of the external processing unit by like adding something that you weren't thinking about into the ziya pipeline.

A

um And so it would be nice if there was a way to for the like at the zio layer to have some kind of error checking or something to make sure that um the data like if there is a handle, then you know and you're trying to get the data. Then you have actually like uh copied it back. Like have some assertion that you know you you already copied it back or or some way of checking that you know what I mean.

E

Can we do something like.

A

E

You copy it out to the cia, you like freeze the in-memory ones, yeah yeah, and so you know that you're never allowed to touch that until you've copied back. If you need to to modify it, yeah.

C

uh So the current check I do is simply checking whether or not the cia handle is null if it's null it was not offloaded. If it isn't sorry, if it's not null, it is offloaded. If.

C

Is currently in memory.

A

Okay, so yeah so basically have some check where it's like. Oh you know, you're trying to do abd copy whatever or abd iterate whatever. um If their handle exists, then we explode or we explicitly copy it back or something like that.

C

Right but it that that that code would not be in the abd itself. It would be part of the small modifications inserted in front of the operations.

C

Like right before doing.

D

C

You first check whether or not this data is offloaded or not.

A

Yeah, but what if I'm like, do we.

E

Add the check to the the function that pushes the pipeline forward for each step or whatever so.

D

E

If we add a new step in the pipeline, we don't have, we won't accidentally forget to add that is the ia handled null check to it right. That's what you're saying that? Yes, I think.

A

Yeah, I'm sorry, I don't want to like rattle too much on this um it, but uh maybe just take the high level point that, like let's find ways that we like this sounds really cool. It sounds like you've integrated it really slickly with the zio pipeline um at a high level. That's just additionally think about like how we can add safeguards to make sure that we don't you know accidentally get confused about where the data is and then you know operate on still data or something because of that.

C

Right right, these are the comments I've, I'm hoping to get like, like I guess now or if I ever or when I.

D

C

Chance to open a pull request.

C

But uh right this uh presentation is just super high level. Here I did a thing.

A

Yeah this is this is really neat I like it um feel free to go on to the next slide. If you want.

C

All right, so the other thing we did is offload uh re-filtering. So normally, when you re-silver.

F

C

Data each uh stripe- or I guess row in dfs terminology- is read in and then the the data is reconstructed using the data that was read and the checksums are verified, verified and if it is successful, the parity bits are property.

C

Columns are recalculated and compared against the existing data and if, for some reason, there's an error, you.

D

C

You reconstruct the columns using, I believe, permutations of the data that was drawn and similarly to the right pipeline. All of these operations are mirrored in the in the uh hardware accelerator.

C

You offload data from right before reconstructing data and instead of calling the zfs reconstruction functions instead called the zi8 raid z reconstruct function to calculate the reconstructed data in with from within the offloader and similarly recalculate the checksums and recalculate the parity columns and then.

C

Eventually, issue any repairs that have to be made to 2d storage uh through zia, instead of through the vfs vw disk or the file functions.

C

uh This also sounds.

A

Very very cool and very tricky um just uh background info, how like uh the razer reconstruction is non-trivial and non-standard. um How is this like? A is this data processing unit like an fpga that you've like programmed to know how to do the recon, like that raid z, math stuff um in parallel, or how does that even work.

C

uh So zia does not control the mapping or.

G

C

Of that stuff, all it does is take the actual. The payload of the abd.

A

C

And operate on that, so if so, zfs will still hold the mappings and.

A

Yeah, I guess my question was like you know, when you're operating on that to do the re to do the like reconstruction math, how do you tell it how to do the reconstruction map?

A

Like I don't know anything.

D

About these cards.

A

So or whatever it is, data processing units.

C

uh Or actually, could you repeat that so you're saying that the vfs rate up the rate calculations are not standard.

A

Well, they're not like I mean they're not like uh something that was in use before zfs. As far as I know like that,.

B

C

I presume we are telling the plan is to tell the accelerator vendors to implement grade z correctly, so that vfs alone can still read the data that they write.

A

Oh okay, so it's it's not, um let's see so you're just going to get the vendors to implement like you're going to send them. You know raid z, that v raid z dot c and be like hey cut up this math and hardware.

A

C

I don't hold me for this, but probably.

A

Okay, oh that's cool yeah. I mean that that math, it's not like uh it's, certainly not impossible. I mean, uh I know that it's um like vectorized uh routines have been written for a bunch of different. um You know vector instruction sets uh to accelerate the raid z reconstruction, um which is you know, pretty complicated, especially when you get to raid z3.

A

You know with three discs died. uh It's it's pretty involved.

B

This sounds cool.

C

Cool uh so right, so uh the original plan was to uh write this shim and then have it link into an existing uh bit of glue code for that talks between accelerators and the via shim. Unfortunately, that was just not a good design. It worked to get me started, but it wasn't extensible, for instance, if we want to if we have multiple accelerators and we want to switch between them, we couldn't just do that by linking vfs directly to the accelerated glue code.

C

Instead, we created the data processing unit services, module which acts as a registry. For uh then vendors implement what we call providers, which is the glue code that talks to their accelerators, uh which allows us to turn accelerators, often on from uh or switch between them or uh or any number of other operations. That.

A

uh So yeah, like you, might have one accelerator that does gzip and one that does raid z, or maybe you have like two that do gzip and three that do raid z and you need to, like you know, multiplex over them right. Oh.

C

Oh uh so that that could work, but that was not the plan. We were hoping to keep data offloaded on the same accelerator throughout throughout its lifetime. So crazy copies wasn't right.

A

So you would have like um the idea is that you'd have multiple accelerators, each of which provides all of the functionality that you need. Yes,.

C

Okay and then z, pull one would use the one accelerators equal to you would use a different accelerator.

A

C

Reason that you couldn't like have.

A

Is there any reason that you couldn't just have one big z pool and, like you know, load balance across your accelerators.

C

uh Probably not we, I guess we haven't really thought about that. We're still we're still trying to sort of get all the issues ironed out so.

A

Yeah in the general you use case like it would be nice to have flexibility there right like it would be nice. If, right, you could have any number of storage pools and any number of accelerators, and it just like uses them to the maximum that we can right.

C

So uh this diagram just shows the general ecosystem of how, via will be used with the data processing services, module or so commands coming from user space on the left into zs, with the ia enabled which in turn sends uh data and commands down into the dp usm, which talks to the the glue code, the providers, for instance, if a software provider or bluefield 2 provider, which in turn communicates down to pcie bus to wherever the accelerator.

D

C

Which then finally writes to the backing storage, uh so the the dpsm has uh two interfaces.

C

One side will be used by the caller, for instance, zfs, and it will have a non-restrictive license so that it doesn't clash with, say, ac dl and the provider side will have a gpl license which so that implementers may use the linux kernel, functions that are gpl'ed.

C

uh So uh that's all. I really had.

H

C

Hopefully, people are interested- or I guess, find this interesting and we'll provide tons of feedback about this.

A

Yeah, this sounds really neat. um How uh how would this relate to the um intel uh affair, the acronym now, but the qatar qat yeah, like the qat offload like, would that could that become one of these providers that plugs into the d? P? U s m or would that have to remain kind.

C

Of a separate thing, I believe so uh the I've only seen the qrt code in a few places, but the modifications weren't terribly big.

A

C

Imagine you could just extract it and write a provider and that's it.

A

Yeah, I don't remember how exactly the memory management with the qat worked in terms of, like um you know, providing the by copying the buffer and then like it does the do you get a handle that kind of lives across multiple invocations or or what right? But I guess even.

D

A

As flexible, hopefully, you could still fit into this api, even if the performance wasn't as good as it could be with one of these custom designed um boards right.

A

um I uh like this. This is very interesting to me, but it's not really in my like personal use case, um so I'd definitely be interested to hear from other folks about uh both technical like how this would work, and also like, is this interesting to you, um and uh I imagine people might be curious about like how they could get their hands on these fancy accelerator boards.

C

uh I'm not sure how people would get their hands on the accelerator boards might have to talk to uh brad settlemeyer.

F

um Might be a question about like the the memory model, so the idea is every like all the accelerators will be attached to pci express there. There's no idea for like tighter co-processors, which have appeared in more recent arm processors, for example. There are like arm processors with pretty tightly integrated, fpgas and so on. That could be used.

F

If you consider those or is it fully limited to pci express.

C

Well and it's it wasn't designed with pcie specifically, but that's that was the general thoughts since nvmes are behind pcie buses. Sure, okay,.

A

It seems like earth.

C

Yeah, if we can figure it out, why not I'm sorry.

A

I was just gonna see it seems like architecturally. The only thing that might be really different. There is the need to copy the memory out to external.

A

You know external memory store versus, like presumably those more tightly integrated things would would just be using main memory.

F

Yeah exactly that was my thinking. Yeah.

A

But you could imagine like maybe that you know maybe the like handle is just like. Oh, the handle is just like a pointer to main memory and the copying to and from the handle is just like here's, the pointer. So it seems like the general thing that's being designed here, would probably work just fine for a more tightly coupled accelerator.

F

Yeah I mean like if I in the product interpreted the first slide correctly then or one of the first slides, then the idea was to directly copy like tell the nvme device copy from the accelerator onto the nvme device as possible. In that case like we have to make sure that it's dm a copyable memory on the outside and so on. A few implications there, but in general yeah, should be flexible enough.

A

Other questions or comments.

G

um I guess my question would be how far in the future, do you think you see this being minimally viably usable because it sounds like a bunch of it is still at least pending. Let's go tell the hardware vendor to go implement it, um which sounds like a longer timetable.

C

uh So uh we do in fact have one prototype piece of hardware that we're trying to implement a provider on so hopefully not far in the future.

E

D

E

Just one other comment: um I know with the qat there was a maybe a minor issue with gzip. In that the way they implemented, it meant that the same block compressed with the cpu would have a different checksum than the block compressed by qat like it was decompress compatible, but it wasn't actually bite for byte the same as if you would compress it on the cpu. They used a like a slightly different table for for the gzip that resulted in it.

D

E

But not actually the same, and there just might be something you want to watch out for or something maybe you can test for when when implementing gzip and hardware on on this.

C

A

And I think that's I mean that's that isn't like a hard requirement. No, it's just something to be aware of. Like.

D

If it could be.

A

Exactly the same as the software encoding, then that would be nice, but if not, you know it's just something to be aware of yeah yeah.

A

C

Would that cause problems or like if the chest form is correct?.

E

Does it matter uh only in the case of like compressed arc is turned off and you have an l2 arc so not in most deployments, don't.

C

Matter I see it: okay, yeah.

A

There's actually.

F

Ellen, didn't you suggest anywhere like I don't remember if it was you, but you suggested to turn on compress like require compressed arc at some point.

E

We looked at it, but uh there are enough use cases where it makes sense.

D

E

Off that uh yeah, I'm looking at other ways to try to solve that problem.

A

Yeah, I would still love to get rid of get rid of non-compressed arc despite those use cases, but maybe that's not practical. I don't know.

G

A

G

Being bite, identical, isn't required boy. Have I got a bunch of upgrades to submit prs4.

E

Yeah, maybe someday, we can figure that out.

A

D

One question for the project matt: um as you know, assuming that we see kind of vendors coming into the space and kind of creating these uh these hardware platforms, how does the project ensure that, as we make changes, we're not breaking these things?.

A

Yeah, I think that's a great question. uh I see on the slide. That's up now. You have uh something called a software provider, um I'm hoping that. Maybe that's like you know something that could that can be a provider that hooks into this interface but is implemented in software and doesn't require any special hardware.

A

um That would be really nice for testing.

A

uh You know if we could have something like that that where it's like yeah, uh technically it's just in software technically uh you don't need to copy it off anywhere else, but for testing purposes we're actually going to copy it to some other memory so that we can catch any bugs related to uh you know, copying or not copying um from the main abd buffer.

A

I think that would be ideal in terms of testing.

C

I will have to check with uh our legal people for about that.

A

In terms of like open.

B

A

The software provider or what.

F

A

Well, presumably, it's gp uh like by making a gpl too, aren't you like intending that all the providers have to be open source.

C

C

Well, uh oh, how does this work? uh I.

A

Mean I'm not a.

C

F

A

um But talk to whoever you need to but like it sounds like. That would be a great way to enable.

D

Testing yeah, I think, that'll, make it easier.

D

uh You know to even like get some of the stuff into into the main repo, because otherwise, like I can just imagine you guys you're going to be like fixing bugs, as somebody comes out with some new thing in the pipeline and all of a sudden like this, this stuff just starts blowing up um and unless you're tracking those changes closely, you may not recognize them, and then you know, god forbid, like you end up with some corruption or something.

G

Yeah right, I should mention, I don't think the qat stuff has anything in the test suite right now.

A

Yeah, that's probably already the case with the.

E

D

Because it would.

E

Require the hardware to be able to test it.

D

Yeah that way, it would be nice if, like if this becomes kind of a generic framework that- and we do have something like a software provider or a test framework or something. Then uh we like get rid of qat, bring it into this framework and and then provide ourselves some protection there and and consumers of that protection.

A

Yeah I mean I I could imagine that this becomes like like there is no non-dp usm. Now usm is just like that's the interface for doing um you know doing like bulk operations on data and all the current software implementations move to the other side of that api as part of the software provider, and then it and then all that you have is a switch. That's like oh, like are software providers operating in place, or are software operators gonna like be copied out to another buffer for testing purposes?.

A

um That would be, that would be cool.

A

um The other kind of high level question I had was: um do you know if these uh hardware providers are going to be making like selling this hardware to the general public, or is this a special you know for for government consumers only kind of thing.

C

I do not know okay.

A

Yeah, I think that um getting it like accepted upstream, um if, if there were, if the only use case of this was um if you're a special entity that can purchase this special hardware, then you can use a special thing. Then I think that it would be a hard sell to get it integrated upstream, but um if, if either you can like hook into software providers hook into the qat, that's generally available- or if this special hardware is um you know is- is purchasable by other entities.

A

Then uh then I think that there's then it makes sense to get this upstream so that people can use it.

C

A

um Hopefully, that you know gives you some decent.

B

A

On you know what you know, maybe if you find out that oh, like they're, not going to sell it generally, then that means that you know you need to put some more emphasis on the software providers or hooking into other hardware accelerators that are generally available, um and you know, and and of course it also just happens to work for your use case as well.

H

A

um Like I mean, I see, you have gpu provider, like obviously there's a lot of gpus you can buy in the open market if somebody's gonna write, like a you, know, raid z, uh vectorized math, for for for gpus. That would be really cool too.

E

And I'm I'm sure that.

A

E

256 one is about as optimized as it can be at this point right for gpus.

E

Because that's what bitcoin insulin uses all right.

A

Yeah, maybe you get one of those bitcoin specific cards and use it for your shot. Calculations.

A

Cool, uh do you have any other questions or things you want to get feedback on jason.

F

I have one last.

A

F

uh So, as I understand it, this dp usm will be a separate linux. Kernel, module and cfs would just use the exported interface of it. Just like any other file system would use it like. Will this dp usm do any kind of resource multiplexing, or will it expose like like what? What will the interface look like for zfs?

F

To give me an accelerator for gzip, give me an accelerator for raid z, maf, whatever like, if it handed out that accelerator will another z pool, be able to use it, or is that accelerator gone now, and how do you handle like multiple z, pools on the system with a single accelerator.

C

uh I believe currently in theory we can have multiple users of the same provider.

C

I haven't really tested that out because, as I mentioned before, we're still getting things to work, but uh I don't believe there's any reason why we can't use a provider but have multiple.

F

Depots use the same provider: okay, then the provider has internal parallelism or whatever, because otherwise, like the operations would get slower as work on the accelerator is pending right. Okay,.

C

Right, we should probably add like some status checkers for for our hardware accelerators into the dp usm, or something like that to make sure it doesn't get overloaded.

B

Yeah some kind of observability into the q dot. Right because I mean at some.

A

B

It may become the bottleneck and then you want to back off and say: don't put everything on accelerated right.

E

Well, like I know, I think, with qat. It only made sense if your buffers were big enough. So if you were trying to check some only for 8k the overhead of copying out to pcie and back wasn't worth it. But if you were check something 128k, then it was, and it might make sense to be able to decide which ones to offload and which one's not to on some threshold.

E

Although part of the the design here is to you know copy it off once and do all the operations on it instead, so it definitely reduces that the cost there.

A

Cool thanks a lot for this presentation, jason. I really appreciate you like coming and bringing these ideas to the the team and and getting the feedback early on, and you know and engaging with the community. I think that um you know there'll be a lot less surprises later on. um You know when, when you want to get it integrated so um right next steps. uh Could you send this a link to your slides to me just so that we can.

C

A

I don't have this online, can I just email it yeah yeah. You can just email, a pd, any email, a pdf or a powerpoint or whatever to me and then I'll. I can put it up just as part of our meeting notes, okay um and then uh yeah, I mean feel free like when you make progress on this and you have more questions. uh You know feel free to come back to this meeting and um and have another discussion or uh open a pull request whenever you're ready.

C

All right cool thanks.

A

Cool um looks like we have a few more things added to the agenda. um I'm gonna, let's see, um let's do let's do uh is rich. Are you here? um Do you want to talk about the encryption stuff and then we'll end up we'll close out the meeting with the requests for code reviews it works for me.

A

Cool, so I don't know if you have enough, if there's enough time to go over everything that you wanted to today, but why don't we at least get started? It's fine.

G

I can I was planning on summarizing it very briefly. Anyway, um all right.

G

So it turns out encryption has bugs surprise. um I know it's shocking right, but um it I I went through. um I can share the spreadsheet I produced. uh I went through all the bugs tagged encryption and I gave them a first pass and it turns out that they all fall into one of like three categories: more or less there's a couple of outliers, but broadly um the big one uh is.

G

uh I wish I'd made slides thinking about it. It didn't occur to me that that would be useful here. Whoops. uh Sorry, the big one um is a bug where you end up with somebody's, not rough counting d notes correctly.

G

um So you appear to end up with um two people thinking they have the only reference to a d note and one of them frees it and then the other one is very sad.

G

um Yes uh and as a result, this can produce all sorts of different stack traces, because you know one of the threads is in the middle and already checked. This is all valid, and then it comes back and suddenly it is not.

G

um I've been looking into this for a bit. I have had some trouble reproducing it on anything, that's useful. For some reason, I've only been able to reproduce it on random, not x86 architectures. All the reporters are from x86 architectures, but you know I have a zfs test suite loop. That will do it inside of a day on.

G

You know my spark board, and I left it running for two weeks on the next 86 board and it's perfectly happy with its life choices.

G

So you know software um very annoying, but uh all right, I should have led. I mostly came here to ask for like if anyone would be willing to spend a bit of time and offer insight, because I am not at all familiar with this code. I have tried reading it, but it's somewhat difficult to run down what might be going wrong um and the bugs at least yeah, the all the bugs that I'm looking at seem to go back to when encryption was first integrated, um like the one I'm talking about right now.

G

The earliest report is pre 0.8.0.

G

um So I can't really bisect my way out of it.

A

Yeah, and is this bug that you're talking about related to um send and receive yeah, or is that not on the scene.

G

I think maybe so like I haven't seen any proof that it has to be sent received to get this bug. I have only seen it reproduce with send receive, but you know um the other one that I'm going to mention is a buck is a set of bugs with send and receive explicitly.

G

So you know until I have a root cause. I'd probably.

A

Got it all right, yeah, that's also annoying, but um and.

G

Not too surprising uh a but a bug which seems to be the same thing on a lumos ate someone's data. So you know that's good.

A

Yeah, that's not great.

G

um So if anyone would like to ping me after the meeting and like talk about this, I would love to. um It would be nice to get these resolved because people keep coming and reporting them.

A

G

We keep going yeah, we know um the other one. uh Well, the other one is a set of bugs where you do incremental send receive and it nukes your data set. So you can't open it anymore.

G

There seems to be at least two different ways. This works.

G

One of them is the user object, counting problem where it. If I understand it correctly, it copies the user object, counting data that it got from the send stream in and because um I think it's the mac doesn't match, then when it goes to try reading it, it gets upset later.

G

um There was a pr that fixed this, but it was broken for some git release or some git versions that never made it into a release. So it got reverted.

G

um I don't know why it didn't get fixed again since, but um I was going to look at these in order and I've been looking at the first one, mostly so far,.

G

The other bug with incremental send receive is that apparently sometimes, if you like, did a change key or change key dash, I on the receiver, and then you go to send back the data or receive data having done that, it will clobber the wrapper key incorrectly.

G

So the next time you go to do load key. You will be sad because the key will not match what is actually encrypted on disk.

G

um Like it will update what key it thinks it should be using to unlock, but not actually re-encrypt the actual key or something.

A

um Yeah, those all sound like uh tricky bugs.

A

um I think this is a great opportunity for like someone or some company, that's using encryption to you, know, step up and be a leader in this area uh like it's hard for me personally, like I know a little bit about encryption and and probably more than I would like to about d nodes uh and center receive. um But you know I have a lot of other stuff to do and I don't and like I don't use encryption at my job, um so uh there's probably other people who are using encryption at their jobs.

A

That might be hopefully more motivated to take this on, and you know, lead us to a encryption, zfs, encryption correctness, wonderland.

G

Oh, I would hope so, but I'm not actually convinced anyone is using it at a large company. um I don't know about that either that that's the problem.

F

Yeah rich, we talked about this a few days ago uh and I didn't get whether you got a working reproducer for at least oh.

G

F

G

You did suggest that I make a minimal reproducer. I discovered that, while it does reproduce happily on a kvm ppc64 guest, it does not reproduce happily on a queen. You ppc 64 gas, which rather limits how useful making such a guest is.

F

Okay, so you need power, pc hardware with kvm on top and then you can reproduce it in a vm. It's very easy.

G

Any ppc 64 vm that appears to be okay right now. I do not know why, um but it definitely reproduces very readily on one and not the other.

A

Yeah, but it sounds like if somebody you know had the time to work closely with you, then you know you could iterate together on getting more info from that system. That does reproduce it.

G

G

I have a number of vms and real hardware that reproduce it. The problem is that the list of ones that kernel debuggers work on is much more limited.

G

um Well, printf debugging! Here you come yeah that that's where I've been.

E

uh Have you considered that the is it related to how much memory the machine has, and maybe you could re-purchase on a faster machine if you just tricked it into thinking it had.

G

Less memory I did because it does seem to be related to free memory on one vm. I have it only reproduces right before the kernel proceeds to permanently scream and yell, because it has run out of memory for higher order allocations, um but I have reduced the amount of ram on the vms, and linux has a nice flag on real hardware that you can say. I have this much ram.

G

I promise um it does not appear to significantly impact the outcomes it might make it faster on the ones that reproduce it normally, but I don't sample size hard because variants high.

G

um Like the spark I have, for example, is one gig.

G

But producing a vm with that does not necessarily make it reproduce.

G

It's cumbersome, um which is why I was hoping to find somebody who might be more familiar with the code to look at it um or at least help me look at it. I should say: I'm not trying to punt it all off to someone.

G

And the funny thing is, I don't even use encryption either.

A

I definitely appreciate your investigation into this and and bringing attention to it rich. Thank you.

G

I just wish I had something more productive to say then. Please help I've been looking at this for months now,.

G

D

In the curiosity, uh sorry matt really good. I was just gonna say out of curiosity. Do folks know if, like some of the like nas vendors or are offering encryption as a as part of their offering with zfs like ix systems, I mean? Are they using encryption.

I

Yes, we are using that not.

G

I

Long time, but we do know, that's optional,.

G

uh Do you use send receive to pass backups around.

I

Yes, I can say that uh that everything is great there. We saw some rough edges, but it's not terribly broken. I would say.

G

I am really curious what the underlying reason is that a lot of people don't have problems, and some people really do.

A

Yeah and it's there are some people that see problems repeatedly right.

G

Yeah, like there are some people, if you look in the issues who just come back and say, yeah, did it again today tomorrow, the next day. um But you know I've tried on my x86 hardware and it doesn't no yeah.

A

It's very interesting: it makes you wonder if there's something specific about their setup or is it just like very subtle? You know, like the timing and the block sizes and the pattern of you know: file creation and deletion that could be very hard to reproduce outside.

G

A

G

I've been trying to work through the combinatorics tree of what could this be, but.

A

Are they all on linux or is anybody.

G

Reproducing it on freebsd, I think all of the ones I've seen reproduce it are on linux. um Well, that's not entirely true right. I mentioned um all the ones that I was discussing before on linux, but I did mention the person or the um the people reproducing something that looks very similar on lumos for the one where the buffer gets the wrong ref count.

G

I'm kind of curious because the alumnus one got a bunch of activity and then hasn't been touched in four months.

G

So I don't know if they just migrated off and didn't keep looking or what, but uh do you know what the illumos ticket is offhand? um I think it's 14 0003.

J

It could just be, you just ran out of time to.

G

Yeah they said they were encountering it on a customer system, though.

J

Okay, so okay alex.

J

You know, and it looks like josh also did something I I can ask them.

J

I I know both of them. I used to work with both of them, so I can, I can see if they made any progress.

G

I would be curious to know. Thank you. I know that I looked and didn't see any commits land in smart, os or lumos about it that I could find.

J

No, I think they may have just ran out of time or with other stuff, um but y'all um alex is in australia, so I'll ping him this uh this evening, uh north america time. Sorry thank.

G

A

D

A

We're almost out of time for the meeting, but thanks a lot for bringing this up rich. I think you know, in the absence of anybody like picking standing up to to really take ownership of this. I think we should continue to bring this up at future meetings um just to to continue advertising this opportunity for uh for more engagement, yeah exactly- um and uh you know, to make folks aware of of you- know the work that you're doing and and that people are still hitting this.

A

um Obviously, encryption is like a big, a big, important feature of zfs, um but like a lot of the zfs features like the importance varies widely among people using it right so yeah.

D

I don't care about it at all and.

A

Other people it's like yeah- this is this- is really critical to my use case.

G

Like so many things.

A

Yeah um cool, I think in the last minute that we have let's uh let folks um solicit code reviews. I think there are a few things in here.

E

Yeah so mine's.

G

Just a small one go thanks.

E

Everyone uh basically, instead of using a static allocation inflation value of 24 times the logical size of the block, uh each v dev type advertises what its inflation would be.

E

So you know raid zed three is four times the a shift uh and it uses that value uh the the worst v dev in the pool uh to calculate you know using the the same math as before, and since we have access to the object, it also looks at what copies equals set to uh for that object set, and so with that you can generally get the inflation down to.

E

You know 1 to 3x instead of 24x, and that means, when you're running close to a quota, the performance doesn't, you know, fall off the rails nearly as quickly.

D

Ellen is that uh that inflation dynamic, as you add, like, b devs and b dip types.

E

Yeah, so if you every time you open a v dev, it checks and increments, the the worst v dev uh if the new v dev is, is worse.

E

So I just keep track of a pool wide. What's the you know worst uh minimum allocation size for any v dev in this pool and uses that value, and it changes the math to round up to the nearest that many sectors, rather than just straight up, multiplying the logical size by the worst case right, because if you end up meeting, you know seven sectors and the raid zed's gonna bump that to 10 you don't actually want to multiply the size by 10. You want to round it up to the next 10.

E

And I saw in the older delfix fork the the static 24 was just dialed down to. I think two uh because you're like we're, never gonna have dude we're, never gonna have uh raid zed and so on.

A

And that's worked just fine for us.

E

And and ditto was removed a while back, so the default should have been at least only 12 for the last while, but it doesn't even need to be that high.

A

Cool thanks that'll be a nice change to get in alec.

H

Yeah and I have uh an update last time- I talked in one of these calls um we're talking about the um corrective receive patch and that there was a test that was failing. I was trying to figure out um what was going on there, so all of that has been addressed, so it's ready for review again.

H

For I don't know, people that may not remember corrupt corrective receive enables us to heal permanent data, corruption using the consent file or something and there's a talk, that's linked in the pr for more context on this. You can go, listen to it and watch it, um but yeah any help in moving this forward or or anybody that wants to review. It is appreciated.

A

uh I don't see the talk like. Oh here, we go it's a little bit down. You might want to just copy those links into the like the first comment. You know that, with the description and everything oh yeah, yeah I'll do that I'll edit. It.

A

Because there's like yours is the last: the one of the links is the last one before there's 96 hidden items, so it's probably not long before that one's gonna get into.

A

I guess it gets hit enough. Where did hook doesn't want to hide it yeah, I think um yeah I don't know, I think it might just be like. Oh you know when there's more than, however many comments, I just it's just like you get the first three and then you the last seven and anything in the middle is just like.

A

Well, you have to click to expand, maybe yeah, I don't know I I only said because there's you know you, you only ever see the one like hidden items thing you don't see like I like, hidden, hidden and then here's one comment, then a bunch more hidden and then what does my comment right unless.

H

They were like.

A

Explicitly hidden by you know by somebody but they're like we don't want to like display. We don't want your page to be more than like you know, 20 meters, long or whatever, uh hiding all right cool. Well, thanks! Everyone um and we'll see you four weeks from today at the later uh meeting time.

B

Thanks man happy new year, everyone.