OpenZFS Leadership, 28 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: January 2022 OpenZFS Leadership Meeting

Description

Agenda: Encryption bugs; Compression algorithms

A

Hey everyone: let's get started with the new year open, zfs leadership meeting um welcome everyone. I saw that there were a few things added to the agenda um and then maybe we'll have time for open-ended q a so.

A

I think first thing we have is a continuation from the previous meeting discussion of bugs in encryption. um Rich. Are you here.

B

I think it was mostly a reminder that we, like shouldn't, forget, there's a bunch of encryption bugs outstanding and nobody has a commercial incentive to look at them right now, so yeah. I think that was the purpose of keeping that that agenda item alive, nothing more.

C

Okay, because I did get uh additional reports of that via the what's it called cincoid maintainer that he's getting a bunch of uh reports in his bug tracker about uh these encryption bugs as well one of the replication tools.

B

I think rich just joined so yes.

B

I did it's true.

A

Hey rich um uh were the things that you wanted to discuss about the encryption bugs it looked like. Maybe the few you added a few to the list since last time.

D

uh Yeah, just the one I haven't looked that much at it, but I was interested because it doesn't seem to involve send receive at all, um which is novel for the bugs currently reported in opengfs.

D

um It uses dedupe it does um it's not clear to me if that's related, um my guess, without having more information, is that it's just that using d makes it slow enough that it can lose a race, um and then it loses the same race that uh send receive can lose and free something. While it's in use.

D

I don't have that much to update on encryption. Actually, um I did spend a fair bit of time working on it before the holidays, and I found a lot of things that don't help.

D

But I wrote up a summary of the workflow that I think um is happening in uh one of the bugs 11679, but basically.

D

Debuff wright gets called and debuff reed gets called and debuff right finishes and frees buffers temporarily or frees the buffers temporarily at the end of the right um and then debuff reed, which is in the middle of running, suddenly tries to dereference one and is very sad with its life choices.

D

D

I tried to find a solution that was either simple or a big hammer like just locking around the whole thing, but locking with those buffers is complicated because sometimes halfway through it we'll release the mutexes and then retake them later.

A

um All right: well, it sounds like it's great that you have an idea of what's going on there, so we sounds like the next step is you're doing some testing. It looks like your tests may be passed.

D

um I tried something it made, it less likely to happen on some systems, but it still happens. um So what I was going to do next, but haven't finished, doing yet is trying to figure out what makes the encrypted received buffers behave differently here than everything else, or if it's just really, anyone could lose that race, encryption's just slightly slower and but that seems unlikely since I don't think I've seen anyone report the same crash without encryption.

A

All right well, it sounds like. Maybe the next step, then, is for folks who are familiar with the dmu internals to take a look at your analysis there. That would be nice. That makes sense um myself, probably being included among those.

A

um Gonna make a note to myself.

A

um And other folks who are interested in how the dmu works also input would be appreciated.

A

um It looked like you had another topic about compression that you wanted to bring up. I.

D

Did yeah um because it's been exciting some people for a bit and I'm also slightly interested because I have a pure outstanding about it. um Yeah that particular bug I linked is a bug. Somebody reported requesting a solution to the problem of you know: never updating the compressors ever because you know things will break um some discussion is slightly heated, though not.

D

I was surprised how not heated some of it was, which is nice.

A

um So uh maybe we could get on the same page about what exactly can break, um because I think that some of the um some of the heat there may be coming from you know maybe misunderstandings or disagreements about what exactly can break if that, if the compression, if the compressor, compresses things differently.

D

So my understanding, which I don't claim to be authoritative right, is that you know l2r gets sad if you.

C

End up l recompressing arc, specifically in the case where compressed arc is disabled, is where it would be a problem.

D

C

uh Because if the copy in memory is already compressed, it can just write it that way to l2 arc and it will match the checksum. That's in the block pointer when it reads it back from the l2r. You can check the block pointer and see if the text seems the same, but if the copy and the arc is not compressed, then uh if the block pointer has compression enabled, I think then it has to recompress it before it writes it to the l2 arc.

C

So when it reads it from the l2 arc and checks the check sum in the block pointer that it'll match, and it won't falsely report that the l2r returned corrupted data.

C

Although it's uh you know, it's not going to panic your system or something. If that does happen, it's just going to falsely report that your l2 arc is failing.

A

Yeah, so um that's one problem is this: if you, if you disable compressed arc and you have l2 arc, then it doesn't, it might not totally really work right, um but the the kind of fallback is relatively graceful.

C

A

C

This, the smarts we added for z, standard and we could add for lz4 in the future or something uh we would be able to say. Oh if the versions don't match- and we see this problem- we can, you know silently throw the data away and not use the l2 arc or never make it eligible for the l2 arc. Instead of you know, reporting a check, some error and making people think their ssd is failing.

A

C

I think the the other cases where this doesn't work are the ones that are probably more problematic.

A

Yeah I mean there's a bunch of workarounds that could be done there like um you, could check the checksum before you write it out. For example, you know you could be like. Oh you could say um you could just say. Well, you have compressed. You have uncompressed arc, we're not going to write stuff to l2. Sorry that just doesn't happen anymore or you can try to compress it and then try to decompress it. Try to compress it and then try to check sum it see if it checks the matches. What we think it should be.

A

If it doesn't match, then don't write it out. um We could also just be like no more uncompressed dark, so there's a lot of workarounds for that one. The other problem potential problem is with not right where, like. If you're writing the same logical data over top of itself, then- um and you have a strong enough checksum- then it'll it'll not do the right, because we know that your right is not changing the data and so um that wouldn't be effective.

A

um If the checksum sorry, if the compression changed um this seems like again like an even smaller corner case, I don't know that anybody besides delfix, is relying on not right.

C

Like I know a couple of backup providers that are, but not that heavily and they would be less impacted and that they would suffer once during the upgrade and then yeah it wouldn't be as big of a deal. Yeah.

A

For them, um so those are the two problems that I know of um the uncompressed arc, plus the torque and then the knob right um are there other problems.

D

The only other one I can think of which is sort of in the same vein as an upright it um conceptually to me at least, would be d-dupe right, because um you're hashing against the compressed blocks still there. So you have a similar problem where you don't save your space anymore.

A

Yeah that makes sense.

D

That's my understanding. I actually have not tested it, because if it's wrong, I don't understand how it could be wrong and not be causing all sorts of problems.

A

So in, in my opinion, uh all three of those problems are very minor and um like if it certainly if somebody wants to solve, if somebody wants to solve the updating, the compressor like more elegantly and more completely and and then that would be great, but personally I'd be fine with just like having all three of those very minor problems um and which, which would happen if we just like updated the compression algorithm and didn't do anything else. Basically, like didn't, do any mitigation at all. um You know those.

A

uh I think the the semantics that you would get are not are not that bad, that's my opinion, um but I I certainly would welcome others. So I see brian joined brian. Did you hear that kind of summary of the potential problems with um changing the compression?

A

C

A

C

E

Thanks yeah that drives up my understanding too. I know I originally was concerned. There might be some corner cases here with encryption too, uh but based on all the uh discussion and whatnot, but looked at that a little bit more and actually couldn't come up with anything particular that were corner cases, new corner cases for encryption. So I think I think those are probably the three cases um I think they're, all pretty minor too.

E

I guess the only one I'd be a little bit concerned about is the easy one to mitigate was the l2 arc one, and only because it would report itself as like lots of errors on your arc device, and that might prompt you to do something unwise.

D

C

Yeah and I think some work around like just you- know block if you have compressed arc off and we would have to recompress the block we just mark it as not eligible for l2 arc.

F

A

Yeah, that would be a very simple work around.

D

A

So rich does that like? uh How does that match up with with your thinking or what you were trying to achieve.

D

That all seems pretty reasonable to me like I personally, I don't use d-dupe, um except when people have bugs with it, um but I do use an upright, but I don't really care about a one-and-done cost for it. Honestly.

D

I guess my confusion comes from a.

D

Oh, I had been under the impression that um not having an elegant solution to this was what, for example, derailed the z standard pr for updating that, ultimately.

E

I think it was something we wanted to consider up front in the design of the z standard. Pr have some kind of versioning scheme, so we could tell in the future, make it easier to do this kind of upgrade.

E

So I think we wanted to have that baked in from the beginning, and then you know once we had that version information. You know we could do something smart with it in the future.

A

Yeah so for z standard we can like the l2 arc. Thing would be handled more more elegantly because you would know you could know like. um Oh, it was originally compressed with like compression compressor version. Two and now I don't have compressor version two. I only have compressor version three, so this particular block I can't put in the l2 arc when I have the uncompressed arc, um but other blocks like that work written with the compressor version. Three. I can do this trick of recompressing them and getting the exact same thing.

C

A

C

The downside quite a bit too yeah.

A

I still feel like look if you're, if you've done an uncompressed arc, if you've chosen to go down that road like that's, that is not a supported configuration. That's you know it's not guaran like it's not guaranteed to work well with everything else.

A

That's why it's not! uh You know a property, it's just a tunable.

C

You know, I think, our other concern of well. You know this pain once isn't a big deal, but if we're going to do this frequently, maybe it is, but I don't really see us updating the version of zed standard more than you know once a year once a major release or whatever. uh So I don't see like we're going to keep moving the target on people or something.

A

I mean I think that even once, every major release, or once year is like it's often enough that I mean it's often enough that it would be noticeable if there was much to notice. I think that the downsides like are just so minor and in in such corner cases.

C

Right and the fact that you know the win is obvious: with the 15 to 20, decompression performance improvement or whatever then yeah, it kind of becomes a no-brainer.

A

C

The only other thought that uh paul doadek- and I were talking about it and we thought you know- would it make sense to do something like we do for encryption, where we basically split the checksum into two. Basically, some of the logical block and the rest of the physical block, so that we could detect when the the logical block was the same, but the compression meant the physical block was different.

C

But how do we, you know, keep track of when that's the case in the block pointer, because we don't really have any free bit fields left at the moment. um Are.

A

There more there's.

C

Some free bits in there I think possibly um or you know, do we just use some of the undefined space to store an extra, smaller checksum of just of the logical block. That's just you know for a quick check uh kind of thing, uh because you know we don't necessarily wanna in the default case, suddenly make everybody's check some only half as strong as it was before by you know, making it only 128 bit everywhere instead of 256 bit by default.

D

I mean we don't really explicitly tell them that. That's what happens when you turn on encryption.

A

Yeah, I think that like if you, if somebody wants to implement that um it's obviously a lot more complicated, but uh that could could be good too.

C

But you know in the end, all it's doing is is mitigating those pretty rare corner. Cases like it might be able to make not write less of an issue, um because you could, you know, check and and be like. Okay, this is the logical block, is the same, and so we can skip it all together and not have the pain, yeah.

B

C

But I agree that the uncompressed arc slash alto art case is you have to be doing two slightly odd things together for that to even be an issue yeah? And I guess the other thing that rich has found is that you know it turns out. We've already been doing this a bit uh like we discovered a couple years ago that the qat offload for gzip actually results in slightly different output than the software version, and then rich just noticed that a lumos and freebsd's gzip is actually slightly different than the one in linux kernel.

A

Yeah I mean those are still pretty like it's probably unusual that people are taking their pools across those. I mean right.

E

And using gzip.

A

And using gzip um or like probably not very many people are using qat at all.

B

A

Those that are probably not going back and forth between qat and non-qet, but um so I wouldn't- I wouldn't necessarily take those as by themselves as licensed to do this. You know to everybody, you know, but um I still think that the the downsides you know these problems are so minor that it's just not that big of a deal.

E

I mean if the linux kernel ever does update their implementation of gzip on us too. It might just happen to us without us, knowing it right. One day, the new kernel comes out, they've, updated, gzip and now we're using a new one. um So getting ahead of that might be nice, but it seems unlikely. They would do that for the same reasons that we don't want to touch it right.

D

Actually so I mentioned this in the bug that I've made about it, but it turns out they did. They did something like what I proposed with lz4, where they updated the decompressor, but not the compressor, um but not because it changed the output, but because the performance was slightly worse on some architectures back in like 2006 and they've never touched it again.

D

So I don't think, there's necessarily that much friction from them on doing it. Just nobody's cared enough, but.

A

Yeah I mean as far as I'm concerned, um if somebody wants to open the pr to update the decompressor. That's like total no-brainer.

A

If somebody wants to open a pr to update the compressor as well, then I would say like test these corner cases that we think are problematic, make sure that they're handled in you know non-horrible ways. um You know uh like not right. It's like! Oh, you just don't get the knob right. We you get, the data gets written again and that's fine with the compressed ultra with the uncompressed arc.

A

You know, maybe adding that one liner to say you know, don't bother trying to write it to the l2 works so that you don't get these like false reports of checksum errors and.

C

A test that you know shows when it didn't work and then shows that it does work.

A

Or whatever, and then I'd be fine with integrating that.

D

That one liner would only work for. Is he standard right now, though? Right.

A

No I'm just saying like uh basically, you would oh.

D

If, oh, the one liner, if it has it, set at all okay yeah, if you.

A

Have uncompressed arc and you have compressed data, it just doesn't go to the l to work yeah.

C

Okay, just gets marked and eligible, and then you know we can refine that later to say: oh z, standard. If if the version is.

B

The current one.

C

Then we just not make it eligible and we can kind of refine it from there.

C

I think yeah, it's not going to be that hard, and I can help rich with that, um because I think, in the long run that's probably a little easier for us maintenance-wise than trying to maintain multiple different versions of z-standard.

C

Like you know, oh we're gonna keep the compressor at 145 but update the decompressor to 150 and then what happens when you know 160 comes out and and so on. We don't want to be carrying around multiple versions, and you know the one downside to the way we've imported zed standard of using the embedded. You know one giant c file version of it is that it compiles a lot slower uh because it doesn't have.

C

We can't use more than one cpu card to compile it, because it's one unit now I know in freebsd we noticed a big change in the the compilation time for z standard when you do it as one giant file instead of a bunch of little ones,.

D

C

About breaking that.

D

Up in the future, but.

C

But if we have two or three of those, then it gets a little crazy.

A

um Brian, uh what are your thoughts on like um acceptability of those pr's die potential? Pr's that I mentioned.

E

I mean that sounds fine to me. um I think it's a good way to move forward on this and it's all pretty reasonable. um Like I say you can refine the heuristics later. We want to make it smarter, but it's a corner case of a quarter case almost at this point already, so I think that's all pretty reasonable.

A

Cool uh any objections to this plan.

A

All right, then, I would encourage someone, alan or rich, to write up the pr and uh and and hopefully we can try to write up the pr and then bug brian and I, as soon as we open it, so that we can get it reviewed and integrated. um You know before, like those people show up um it's not that we don't want before you know it. It accumulates a lot of comments that.

B

Yes, are not productive. No, like.

A

I welcome people to come and say, like hey, isn't this going to break whatever you know? We do want that input. um What we don't want is uh for the amount of fud to exceed our ability to defend it. You know like we don't want.

A

We don't want to have a bunch of people coming and saying: um hey, like I think, maybe something might about this might not work, I'm not going to say what it is, but I'm scared of it, and then you know for the for the pr submitter to not have the time or energy to go and to go respond to every one of those and say like I'm, pretty sure it's fine, you know like what's the problem um and how they get kind of mired down in like bad feelings that are not um specific problems.

A

C

I guess it how many people are using d-dupe, because I guess they're actually, the ones that are probably going to feel the pain. The most.

A

I would guess, there's a there's, probably a lot of people using dupe. um They may not be developers that are on this call.

D

Yeah not enough to write ddt, log yeah exactly.

A

A

um So, which compressors specifically, are we proposing to update right away.

D

uh I was thinking lv4 and the standard because I actually have prs to do one of these and, as it turns out, someone else has written the other one. So.

A

Yeah, I mean that's, um I think, let's open them if lz4 um definitely would be impactful since that's the default.

A

um So you know if, if people that they use do um yeah adam says that he uses it and regrets it, um which I think is uh a lot of folks experience, but I I know there are people that use it and do not regret it, or you know that. Do you appreciate the space savings that it gives?

A

um Hopefully, this one-time hit for them is not a big deal, um but we can, you know, get comments and uh decide on how you know if we need to do more mitigation based on that.

C

I guess one question is for lz4: do we want to start tracking the version and stuff like we do for z standard, because then that'll be an on disk change, because we'll have to treat it a bit differently to store the version along with the block like we do for z, standard.

A

um We could I mean, I don't think, that's a requirement for getting this change in no.

C

uh Because it turns out, we haven't, come up with a clever way to use the version number yet.

A

Yeah uh I mean I, I would guess that you, it would be really nice if we could do it without changing the honda's format without needing a future flag. You might be able to do that like if you look at kind of the there's already a header on there, a header in the block that has right.

C

For the it's the compressed size, because the allocation is always going to be a shift based, uh and so it's so we don't feed the slack into the decompressor and break it. Yeah.

D

But I think that might be all the header has.

C

Yes, it's just a 32-bit big endian, encoded size.

A

Yeah, so maybe you would need it.

C

But, like you said, we haven't found a use for the version number yet so maybe we don't suffer a bunch of pain to future-proof something that might not turn out to be useful, although you know I just made the case for not carrying around multiple versions of these things, but I suppose we do mitigate the pain for people upgrading if it is a feature flag where they can opt to not enable the feature until uh they're ready for it, rather than just when they upgrade their version of zfs. It starts not deduping anymore temporarily.

A

Yeah, I think that that would be nice. um I mean that uh is. Is that kind of equivalent to just like introducing a new compressor right.

C

I think maybe what we end up doing is lz4 becomes lz4-legacy, and then we have a new lz4 that has the version number.

B

C

Is protected by the feature flag, so it doesn't.

E

Yeah uh I mean that works as long as you update lz4 like once right right. If we're ever going to do it twice or three times and make it a regular thing, that's going to become more problematic, but.

C

Well, once once we've done it once we have the version number information in the header, we can update it without having to change the thing, although we don't want to set the expectation that every time we upgrade the compressor, we let you choose whether you upgrade or not, because yeah again like I said we don't want to be carrying around uh multiple versions forever. Although lz4 is a lot smaller to carry around than.

D

So you would still only have to carry the compressor around yes, that too yeah um yeah. I was going to say if enough people complain, particularly because they're using d-dupe and don't want their d-dupe tables to be garbage at that point. um Well, if, if you're deduping.

C

Enough that it's helpful to you it will, you know, stabilize on.

A

Yeah you'll have some versions. You have two two copies of that logical data, sure sort of and yeah.

C

E

You might have a root surprise, though right, if you were expecting it to do, but suddenly you're you need twice the capacity you thought you did and you don't have it right, there's a thing that could happen with you. If you uh potentially writing a lot right, you're expecting everything to be do and suddenly it's not after an update.

D

Yeah, I I go back and forth on how strongly I feel that, like it might be reasonable to carry around the old compressor for a version or two and put big flashing notes in the release, notes, saying hey! This is going to change.

D

If, if you care don't upgrade until you have, you know flipped this bit and use the new version great.

A

What's? What's the cost of keeping the old compressor around like there's, there's a bunch of clothes.

E

At four, not much yeah, that's pretty cheap uh z standard is huge, though right.

A

Yeah, but it's just I mean what is the the me: is there a maintenance burden, or is it just like compile times, are a little longer.

C

uh A little bit of maintenance, I think, because you have to rename all the functions, so they don't conflict and that's mostly all scripted and so on. But that's a one-time kind of.

D

C

D

Going to say, with the standard, the import is all rewritten all does automatic, somewhat automatic rewriting of the symbols anyway. So you just edit how that works, um and then you write a shim but yeah.

E

I think the maintenance burden is pretty low for talking about one or two or three versions of it. I think where it gets problematic. If we end up with 10 versions of z standard in there right, it gets harder to maintain and makes the compile time longer rate. I'm less worried about the compile time, but yeah I'd prefer not to carry too many versions of it around.

C

Right and you know.

A

I think for elsin yeah, the hard part is writing the code to do the like. There are different versions: I've selected the most recent version I have either like I have a tunable or I have feature flags that tells me which one when I want to do. I feel, like that's stuff, that's tricky, but you do that once and then the fact that you have like a bunch of different c files for different versions of z, standard or lz4, like we never touch those.

A

It's not a big deal like as far as I know, we're never like. Oh, we have to go into the depths of every compression algorithm and change, something about it because we're refactoring something like that code is just like it gets through and once it sits there and we never edit, we never modify it. So um from that point of view, it seems like keeping around the old versions has very little cost.

E

I'd say the only caveat to that is if we start doing that, we're rapidly not going to be testing the older versions we'll just end up testing the newer version properly in all the automated testing. So we're okay with that not testing that older code ever because we never touch it right. um But if we do have a need to touch it, it's going to be painful if we have to go back and touch all the implementations to update them to handle some kernel change or something that a compiler flag or who knows what.

D

I mean I wouldn't suggest keeping an indefinite number of versions around uh like I said, I'd keep maybe one around if, depending on how frequently, we ever need to update anything if we need to update it again, punt the old one and put big flashing notices for like a major version release or two before you do it.

C

Yeah, I think the only problem there is that, generally people don't get to pick which version of zfs they use right, it's whatever their distro ships with.

D

Sure, um which is why it unfortunately suggests defaulting to the conservative option of like using the older one if they have an existing data set.

C

Or it's kind of the the advantage of a feature flag is that it's we use the new one by default. If you create a new pool, but if you have an old pool, it doesn't switch without you deciding to do that. Although you know people just blindly z-pool upgrade too right, you can't solve everything.

B

F

B

That the main point of contention is the d-dupe case.

B

um The thing I'm asking themselves is like, even if you put up this warning like, will the warning be actionable of any sort like if you rely on your pool being deduped and you don't have provisions twice the capacity like the case brian mentioned? What are you gonna do about it in the end.

D

um I mean at that point you would have to stay on a version that supports your version of the compressor for the lifetime of the system. If you don't have any way to migrate, your data around.

B

How about send receive can I send receive to a new pool? No right? Why not.

D

You're not passing your table around with send receive.

B

Oh right yeah and I could even avoid like doing compressed and received oh.

D

Yeah and you you'd have to forbid sending compressed at that point unless you wanted to write code to do recompression on the other side, ew.

F

Oh well, but in order for customers to stay or users to stay on a given version, that means we can't just simply say we're going to keep the last two versions around in the code right. They would never be able to upgrade cfs to new bits.

D

I think I might have missed.

F

A step in the logic there because, if like, let's say, let's say I'm on version 1.0 and we only keep the last two versions of lz4, and so when I upgraded zfs to you, know the next version, there's lz, 1.0 and 2.0 or rn the code base. As soon as we go to lz 3.0, it's only 2.0 and 3.0 in the codebase, but I'm using lz 1.0. So if I'm doing dedupe I'm getting burned right as soon as I do my upgraded bits.

F

That is true yeah. So it's like to me, I feel like it's either like it. Maybe it delays the pain point, uh but that that cliff point is coming um and it's either. We we, I feel like we either have to commit to say we're going to keep all the versions around or like. Why keep any uh around.

A

I agree with that.

C

Or at least you know, we keep the lz4 we've always had and with a feature flag- and you know you can opt out of that feature flag and just continue using the old one forever or you enable the new feature flag, which is. This is the new lz4 and it's going to keep updating and you agree to deal with it.

C

B

Of the way feature flags.

C

Are generally on by default, when you create a new pool, and so on that that people are opting into things without understanding it. In that case, right.

D

That's not new, though.

C

um Yeah, there's only so much we can do to meet everybody's expectations.

D

Sure I guess I feel like the topics of like lz4 and z. Standard might be different, because we don't lz4, doesn't show nearly as much for lack of a better term. Like I'm not saying we should carry security flaws if they come out in lv4 or whatever, but.

D

You know it hasn't been touched more or less, since it was integrated, and I don't necessarily see people continuously integrating new versions of it at the moment um without cause. Anyone see standard, though, is getting much more active updates with sometimes significant deltas.

D

So like I guess I feel like saying it's a one and done upgrade for lz4 might be viable or like one and done for a decade or something.

D

But yes, we do have a new lz4 pr in the queue it's mine. um But unless you mean that you just opened one either.

B

D

Or you know the standard might need or have good reason to get refreshed more frequently, and then this becomes a more frequent pain point.

A

Yeah, I think that that's a valid point- and you know lz4 being the default- means it's going to be more widely used. You know so maybe maybe that means that lz without e4 we always keep around the old versions and you can choose.

A

You know it's like there's lz4 v1 and this is lz4 v2 and you know we're going to change the default compressor to lz4 z2, but you know uh only for new data sets or something or only for new pools. I don't know. I think that you could you could kind of mitigate that and then say sure, we'll keep around the old versions of lz4 forever, but it's a different story for um z standard.

A

uh I feel like there's a lot like there's a lot of flexibility in the design space here, yeah. So really it's up to whoever wants to implement this. Like you know it seems like people, people a lot of people want to see this get done um so, however, much effort you want to put in to do all these various mitigations it's up to whoever implements it.

A

In my opinion,.

D

Well, whoever implements it and whoever agrees to merge it.

A

Yeah sure, like.

D

You're harder to.

E

A

Reviewers, if you're introducing like a lot of new mechanism, um you know- I imagine a lot of the comments are going to be about that new mechanism and how that works and the properties and the future flags and whatnot, as opposed to you know. If your pr is like you know, I add like I'm keeping the old file the old lz4.c, I'm adding a new lz4 v2.c and like it's just a new compression algorithm.

A

um That seems easy, easy to review. Likewise, if you're, like I'm just deleting the old file and adding a new one like that seems also seems easy to review um the the more mechanism we add the harder it is to review, but I mean hope I think that, hopefully we can get it get it done. If somebody wants to implement it, so I wouldn't you know I wouldn't discourage somebody who's motivated to do that.

F

What uh what advice would we give rich since he has an lz4 pr out now?

F

Do we say just replace it or you can't go in until there's some kind of you know until this problem has been addressed.

A

Is your pr um riches, your pr to just replace the old one, a little algorithm with the new algorithm and that's it.

D

The pr I currently have open is just replacing the decompressor of lz4 with the new one from you know the last year, um yeah it just punched the decompressor into a separate file, that's mostly untouched, um just upstream lz4 code.

A

Yeah, I think we can just merge that today. Like I mean we need to get it reviewed and whatnot, but that seems like there's zero impact.

C

Yeah, um yes, we might do it slightly if we're planning to pull in the entirety of 1.9.3 or whatever of lz4. uh We might want to keep the file in the shape that is upstream instead of breaking out the decompressor into its own file, uh but other than that. I think that.

D

That'd be the only thing that would change the way. We would lay out that pr I mean the the only problem with that is that I think the file the function names conflict with the existing ones. We have right um so you're, going to end up needing to like have giant commented sections which we can do, but.

A

um So I think that the the decompressor I think we can sort out those details in the code of you yeah with the compressor. um Does anybody want to volunteer to take on any of this work for the compressor.

D

So I also have a companion pr. I haven't opened that substitutes the compressor, um but it just replaces the compressor right. It just does the same thing as the decompressor, but pumps it into a different file.

D

A

I mean my advice would be uh like, let's see if, if somebody wants to do more than that, let's find out if nobody's going, to volunteer to do more than that say you know in the next month, then we should just take that we should just take what you have of replacing the lz4 compressor with the new version, and you know that's what that's what people were willing to implement? There's the semantics that we'll have um assuming that there are no dire objections from um d-dupe users.

C

Well, I'd be willing to help to do something along the lines of you know, importing the new one as a new compressor that adds the version number feature like we did for z standard, and this solves the problem of giving people a way to hold off the d-dupe armageddon or whatever, so that they can coordinate when they switch over to using new lz4 instead of old.

D

Yeah I mean that should be a simple change.

C

Yeah- and it's mostly just it- would keep the old one hooked up as lz4 for compressor uh and the new one hooked up as lz4 dash new or whatever naming we come up with, uh but they can both use the faster decompressor. So we get that performance gain on the decompression either way.

A

All right well, that sounds great to me. um Let's, let's see if we can see some progress on that uh before the next meeting um and if not then we'll then we'll reevaluate, like falling back to a simpler way of doing it.

A

um Cool well, I know we've got we've kind of gone around and around a bunch with this, um I'm glad that everybody was able to kind of get their thoughts out, um and even though this seems like you know, if it feels like we're spending a lot of time on a minor thing. um It's a minor thing that people have been. You know talking about and spending spinning a lot of wheels on for a long time. So if we can come to some kind of conclusion and get something integrated, then that'll be a great outcome.

C

Yeah I've been chewing on this since, like 2016., so.

D

Yeah, if you just want something integrated, you could, as you said, just merge that pr today and suddenly progress.

A

I think that's the the changing the decompressor um we.

C

Can I agree, that's it's a no-brainer, it's just the typical thing of well. If we're going to do that, we could do all this and then it quickly becomes bigger right and so maybe getting something is better than getting lost in the weight. So I think matt's plan makes sense of if we can come up with something before next meeting then great and if not, then we just go to easy ramp.

A

Yeah, I think the integrating the if we integrate the new decompressor this week, like that's fine, and then you know if, if that work gets reverted a month from now, because we're replacing both the decompressor and the compressor and reformatting the files or whatever, like that's fine,.

A

A

um Cool uh we're almost out of time. uh I I think this is on the agenda at least. Were there any other things that people wanted to discuss today.

B

I just want like for the um for the meeting notes. We just decided on how we're going to proceed with d2 plus s4 or the sa4 upgrade in general. I will have a separate decision on the standard. Is that correct.

C

I think so yeah.

B

C

So I think the standard we are a little bit less willing to carry around every version forever yeah, but with lz4.

D

C

Is so much lower that it's less of an issue.

D

I might go experiment with seeing how low I can get the impact of that, but that's we'll see.

C

Because that's the other thing is just because the newer version sometimes recompresses a block differently. It doesn't mean it always recompresses a block differently.

C

It's just in certain streams, now compress 10 better. That will definitely give you a different checksum, but I imagine that that only affects some fraction of the possible input blocks.

C

D

um For the gzip example, uh for example, right, it was more like 95 percent of the test blocks. I used the.

C

Compressor, I think, that's because it's a different table that it uses it's true, that that the same thing could happen in z, center.

A

Well, I'm going to go look at rich's, decompressor pr, um and I think that we can all look forward to some progress on this before the next meeting and um alan and rich. I definitely appreciate you putting in the you know the hard work to actually implement stuff and test it um and uh adam on chat for uh living dangerously.

A

I guess with uh the by testing the bits in the pr uh in these pr's um if anyone else uh is, is able to help with that effort, then um then get in touch with rich and alan uh on slack or in the prs.

D

I would also really appreciate anyone who wants to offer any insider help on the encryption thing, because I'm much less confident about how long that will take me without help.

A

Yeah, I I hear you there I'll try to take a look at the at your um analysis of the debuff stuff. Thank you cool. Well, uh our next meeting will be four weeks from today. um I think it's the same time. 1Pm. Let me check our news. uh uh Previous was 1 pm, oh, so the next one will be the earlier time uh no hold on. Let me double check, no all right.

A

So in december it was the earlier time this is the later time, so the next one will be the later time 1 pm, so the next meeting will be four weeks from now.

A

February 1st, at 1 pm thanks everyone.

C

I have one other question for the spa inflation thing. uh Somebody raised a point that it looks like maybe my assumptions about the what the formula needs to look like for that are wrong. um How do we actually? Is there a resistant code somewhere that actually calculates what it must? uh How much space a block is going? A a logical block is going to take to write out to a raid z3.

C

Because it's not.

B

C

Multiply the the logical size by four, it's a bit more complicated than that. But what is the right.

A

Is it the answer? uh When could it be more than that.

C

I don't think it could be more than that, but okay, is it usually less than that.

A

Sure, usually it's less than that, but it's supposed to be an upper bound right.

C

Right well we're trying to get away from overestimating by just multiplying the logical block size by 24, because it could be raid z3 with copies equals. Three and uh ditto.

A

Vlogs, but we got rid of the digital blocker. Your question is: if it's on raid z3, what is the maximum amount of inflation that it could be?

A

C

A

And and the answer of like definitely sometimes it could be 4x and you you would like to narrow that down in certain cases.

C

I think yes, that's, I think what I'm going for of like if it's a one megabyte logical block.

C

Is it most the time going to be much less than 4x, or is it only going to be marginally less sometimes.

A

uh I mean if gank blocks are involved, then it could be more than 4x right, it could be, it could be 4x. You know it could be have to gang multiple layers of ganging down to single sectors, uh and you know, and then plus the gang block, header header blocks and all that stuff.

A

In the worst case, right.

A

um If you want to ignore gang blocks or like make some guesses like 99.9 of the time, it'll be less than whatever, then you could probably.

A

Figure that out, like that, you know if, if the block size is n sectors, I mean you can do the raid z math to figure out like exactly how many sectors will be allocated.

A

A

Yeah I mean that that might be a reason. It might be reasonable to take that into account. Take into account the logical block size rather than just like the same multiples applied to all block sizes.

C

Yeah, well, that's what I tried to do, but I realized that what I have now is wrong, which is just it rounds the number of sectors up to the multiple of the raids that which doesn't actually account for enough parity in some cases.

A

um Well, why don't, I think, we're out of time for today um but uh feel free to ping me or um or we can discuss in the pr.

C

Yeah, take a look at your spreadsheet from the how I learned to love raid zed thing. Maybe that formula can help me comprehend, what's happening better.

A

Yeah I mean doing that. Math uh is probably like. You would need to call into the raid z code. You know I wouldn't try to reproduce that math.

C

No, I don't want to do that there, I'm just trying to get a a less egregious overestimation.

A

All right, cool I'll, see you all in four weeks.