OpenZFS Leadership, 4 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: January 2023 OpenZFS Leadership Meeting

Description

Agenda: encryption; mmap bug; on-disk format doc

full notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#

A

All right, let's get started happy New, Year, um welcome to the January 2023 open, zip resolution meeting a couple things on the agenda for today.

A

The first one uh was from Rob and I'm, not sure if you're on the call, um but he was looking for reviews or feedback and his pull request about Cha-Cha encryption.

A

um Are you here Rob.

B

I, don't see him he's uh from Australia, so the time zone might be difficult.

A

um It looks like Jorgen took a look and Richard have had a comment: Richard yeah, um but that was just kind of meta.

C

A

A

Yeah I uh I, don't really know, have enough context to comment on the material of it um we'll see if we can find some reviewers anybody interested in encryption.

A

uh Looks like this algorithm he's saying it's: you saw on Lower powered machines.

B

Yeah I think on machines that don't have accelerated AES, it's better.

B

It's uh I think also the default for what open.

C

B

Uses for the same reason.

D

Does anybody know, is this like breathing new life into encryption in CFS? Like do we have users now that are like?

D

That would be great if we had like people actually starting to use this more than just.

E

B

There's a good number of.

E

People using it.

B

And they're stumbling over some problems, although uh so far I haven't been able to isolate it down to much specific and every time it seems like we do. The error message changes to something like the crash changes, something completely different.

A

Yeah I think the next topic on on the agenda actually is related to that um about kind of stability of encryption in general.

B

Yeah, uh it looks like the the gang block change that I think you made recently Matt uh helped with some of that uh and.

F

B

Me of a separate thing of some number of months ago, a change was made uh so that we don't try as hard to avoid having gang blocks, and a few of our customers have noticed that on raid Zed, two and three pools uh getting a lot more gang blocks kind of unnecessarily, and they have to change the tunable to make it go back to trying harder before resorting to getting blocks.

B

Otherwise, uh you know really space inefficient, I, don't know uh it's not really related to the encryption, but it just reminded me when I was thinking.

B

Your fix for the the detection of encryption on gang blocks uh does seem to have helped uh in some of the cases we're seeing of people running into panics from encryption, um but I also wonder if it's related to are we making a lot more game blocks than we ever meant to, uh since that change, and maybe not noticing, do we keep is it is an easy way to get any stats on how often we create a gang blocks, you just like to have runtime counters for it or anything.

D

I think if they would do a scrub, you have counters, but.

B

I, don't know, didn't Alexander turn those counters off.

G

Oh I, disabled them, but code is still there just no interface to read them. You would need to address or something.

A

Yeah I, don't remember if I wanted to or did add a like runtime count of uh number of gang blocks.

B

I don't see one currently, okay,.

A

um Yeah, that would be a nice addition. It will allow us to quantify. The you know are.

B

We creating more of them than we than we think we should be.

B

I think the the one we were investigating the initial report was null pointer, D references in like lz4 decompress, uh but they kind of when we enabled assertions and so on that it tended to panic in other places much sooner, uh but it would seem that the the raw ABD pointer would get changed to null at some point and not uh contain the data anymore. When it, the buffer, was in the the state where it should contain the the raw ABD still.

B

uh But we haven't managed to track that one down uh it started hitting one where, when they use a clone for Docker, uh the M map isn't working properly. I think there's a different PR about this issue. I, don't know that it's necessarily related to encryption, uh but the an assertion fails about the the M map and ZFS copies of the data being in sync, uh and it's.

B

I think it's apparently derailing our attempts to to narrow down the encryption bug because we're running into other bugs.

B

But mostly so far, we don't have a great use case, for you know triggering the encryption problem on demand and uh it sounds a little related to what uh some lumos people found specifically having to do with clones. uh So we're gonna dig into that direction of it and see um because I think the other place people report, a lot of problems was send receive and when you do a send receive, there's a clone that happens under the hood right temporarily for the the temporary hidden data set or whatever.

A

Yeah like when you're receiving yeah.

B

You received into a clone and then swap it later um so yeah uh I wonder if that might be the the thread that we need to tug on and and see if that brings a bunch of these kind of disparate issues, all together.

A

Cool, oh yeah, Michael. Thanks for sharing, um there's a spreadsheet um that lists the various encryption bugs um it kind of tries to categorize them.

A

um So you know this kit kind of gets to Rich's question about, like is native encryption like do we think that it's really good enough and ready for prime time or do we need to like kind of add some official warnings around it.

A

um I, don't really have a good answer around that either um you know I, we uh I don't use it. Day-To-Day um I haven't been following the issues super closely, either, but uh definitely open to folks ideas or feedback on kind of save encryption and how how well it works or does not.

E

um Similar to what Alan was saying uh to I hope, my mic's worked.

C

E

Thank you uh similar to what Alan was saying. I was trying to look at a bug that somebody reported um about encryption, where you can end up with an embedded block pointer record on an encrypted data set. So that's good um and that works fine right up until you try to do anything with it and then the code goes. What uh and I tried to reproduce that and I accidentally reproduced a different bug which was about the point at which I thought I'd bring this up today because it was like.

E

Oh, uh there are sufficiently many bugs that if I try to reproduce one I may accidentally hit another that seems problematic, um I understand that a lot of people are using it fine and it works great for them and that's great.

E

But if you have a one percent chance of being jumped by a lion, it's still probably worth warning people. If there was no lie in Risk before metaphorically.

A

Yeah I mean uh you know: data loss is very bad right. So uh I think that if, if we have some data loss problems with encryption, then.

A

You know we should be taking those very seriously and trying to address them and if that's not possible, then you know putting some guardrails around uh the use of that feature to mitigate the impact.

A

um I, don't really have enough details personally to like make a call on that right now, yeah.

F

Alexander, do you have a sense of how many trunes users are using native encryption.

G

I, don't have any statistics, but it's uh widely announced and we switch from previous to Linux. We should migrate to Native encryption uh unless somebody you want to use SED on this encryption, so numbers should grow, but I don't think we have statistics on exact numbers.

E

I think you were saying at the summit that you hadn't seen anyone reporting problems with it, though weren't you. If I'm recalling right.

G

Oh yeah, obviously I heard from other from a few developers that everything better stay and terrible, but on other side like I, don't have any like huge Stoppers reports or anything that would tell that it's like more broken than not it's it's kind of working and uh we are creating some minor PRS polished here. Is there what I think was just one was just opened last week, so it's mostly working. Maybe there are some corruption somewhere, but again we have no reproduction or specific reports.

B

As far as I know, there have been no reports of crashes on FreeBSD, and that seems weird.

E

um At least at least the one that you're investigating Alan seems like a really narrow to me race, so I'm not entirely surprised. Something with a very different scheduling system has different properties for triggering it right for that one um I could try one of the ones that I think I know that I have a reliable reproducer for which is not a crash, but just it screws up setting the key and I would bet that would work on FreeBSD, but okay I haven't tried. It.

E

But you're right, that's true! Well, there's the lumos issue, which I guess could be unrelated. Technically, the stack Trace is the same, so I'd be really surprised, but.

B

I think it was specifically uh one of the illumos issues. I was looking at. They had a reliable reproducer, but it didn't reproduce using that reproducer on Linux or BSD. Okay,.

E

On the lumos, um the same failure mode seems to be happening, but it doesn't seem to reliably reproduce.

B

E

B

E

um What happens for them if I recall and in my experience for us, uh is that when we go to update the uh user accounting data on the data set um right.

B

So that one was only able to be triggered if you were receiving a send stream from a pool old enough to not have that activated already.

E

You can reproduce it reliably doing that, because then you can force the ZFS upgrade whenever you want right. um It still happens. If that's not true like that is I believe what happens with the failure. The bug that you're investigating is when it goes to trigger the recomputation of that it goes and touches everything um and that's how you do the recalculation and when it touches, object, zero.

E

um It somehow gets into a state where it's trying to do a right at the same time as it's trying to access it elsewhere and part of the right path it takes is that it sets the ABD to zero or no the contents of the ABD rather and then you know if you access it. While that's true Ben goes to Dynamite.

B

uh That sounds like one of the panics we're seeing with debug enabled is uh during I. Think it's Arc release that the ilm progress Isn't false, uh and so it sounds like like we suspected something with the ref counting where uh something is still using. This buffer I was in progress, but we're trying to free it. At the same time,.

E

Yeah, that seems consistent with what my expectation is for that. um So, at least in my experience, if you force user counting off, you can't reproduce it, um but that's not necessarily useful for everyone who wants to use that right.

B

uh Can you force it off once it's already on or.

E

I had a patch to do that, but if I recall the user who tried it reported other problems. Okay um with the patch just forcing it off, doesn't have problems right, um but uh that seems to be what was happening is that it tries dirtying it and um it thinks the ref count is zero when it's not and does something unfortunate okay, uh but in any case uh I, don't know what I'd phrase a warning like for this right like see.

E

If you were affected by this list of known issues, don't use the fs send, um um but.

E

With how many people are using it without a problem, it seems odd to put a warning of don't use this and would probably get a lot of questions.

C

E

But I'm not sure anything less is accurate enough.

E

I, don't I, don't have statistics I just know that people keep coming in and asking me questions about it being broken in various ways.

F

You are the world expert.

E

E

E

I just wanted to possibly put something like you know you you may wish to see if you're affected by one of these known issues with a pointer to Something in the docs listing a couple of the more easily hittable ones um or you know something like around dedupe was the original vision of like think carefully before turning this on.

G

But I think I tend to disagree in case because they're growing from design yeah. If we have problems with encryption, that's that's a box that should be fixed and just telling there are bugs like there are dragons it doesn't. It won't help anybody with anything. It won't help with debugging. It won't solve the problem.

G

It won't do anything. It's just bad publicity and that's all.

A

Yeah I kind of agree with Alexander um the the issues like it's hard to write, something that says like oh see, if you're affected by these bugs, but then like the bugs, don't necessarily the bugs aren't narrowly constrained they aren't like. Oh, if you're you know doing encryption and encryption replication and you've turned off. You know some are some. You know weird flags and you know. Okay, then don't do that, but it's just like you have you know we only all that we know is like sometimes the machine, panics and they're using ZFS encryption.

A

You know we haven't really narrowed it down like that. Much then um it's not that helpful to tell people unless we're just gonna say: don't use DFS encryption like we put it behind some flag or something, and it's not enabled by default anymore, which is I mean it's tough, because you know the cat's already out of the bag. Very people are using it yep.

A

um So that is not a great solution. Obviously the best solution is fixing the bugs, um but you know we need people that know enough about it and care enough about it and have the time to spend on it um and uh I. Think, unfortunately, like the original authors and um maybe also other folks, working at that company are not super active anymore in the community, so you know we need to, like you know, find some new experts find people who are willing to put in the time to um learn about this subsystem.

D

C

D

Sorry go ahead.

H

I think you should go.

D

I I was just going to ask like IX systems now supports and script encryption. Is that a relatively new feature.

G

Where you send the web UI signs, I, don't remember when, but a couple years, probably, but again we added it in parallel to existing uh software on this encryption and we're also in all our Enterprise systems, we asked uh by default. Cell full disk encryption drives, so it's not used in 100 cases, but it's in web UI, it's not behind any license or anything like anybody installed through nuts at home can start using it. It's like I, don't know any Gates for quite a while.

F

Okay, Alexander: do you know anything about the quality assurance steps that go specifically towards encryption?

F

It was an IX yeah with Nix.

G

On said, I don't know, I didn't dive so much. You know to our QE to say what exactly tested, but obviously there are some basic things of all the creation since replication things there's also obviously tested but I suppose this happens in some rare cases under some workloads we are not doing uh active a lot of benchmarks under encryption because for anybody who care about performance, we just recommend the CD drives since they cost the same, but don't have performance problems.

F

A

So I think uh you know currently we're kind of at a progress on this is stalled. A little bit sounds like um you know. Understandably, you know IX might be using this, but they aren't experiencing the problems. Maybe their customers are using it in a way. That's different than the people who are reporting problems are using it.

A

um So you know I understand them not wanting to like put a lot of resources into something. That's not a problem for their use case.

B

A

B

Clara is investigating one specific issue for a specific customer uh related to this, and hopefully we'll have we'll find something, and maybe it'll solve everybody's problem, but I'm not gonna, promise that um but we're you know looking at one specific uh cause of this, although every time we poke at it, it moves. So we're not sure uh which subset of the problems will be solved by by getting to the root cause of this. The specific one that we're investigating now.

A

Well, I mean that's that's better than nothing. Yes, um it's great that you guys are looking into something um is: is there a specific? Is there a bug, that's related to that, or is this just.

B

There's a gr somewhere, I forget it's: uh some, our customer joined an existing one because they had to vote the same stack trace and then they also got about six different stack traces from their Fleet of machines uh having to have something like 100 machines, running 2.0 or 2.1, uh and they seem to get two or three panics a week, uh and so we're poking at that trying to figure out what's going on there, but um I find the other one.

B

That's uh the nmap related one uh is a separate issue and is hampering us a little bit.

A

um And I'm are, are those ones the data loss ones or it's just like a.

B

Panic, no, this is just a panic and then reboot and everything's. Okay.

E

um To be clear, to the best of my knowledge, um the closest you get to a data loss bug with this currently is that sometimes it will set the key for the data set incorrectly, um so it will change what key the metadata says it's encrypted with without rekeying it um so, the next time you try to unlock it. It fails um other than any sort of just general rep um consequences of panicking in the middle of doing something, um I, don't recall, seeing any that actually caused data loss more than what I just described.

A

Okay, um so it sounds like the data loss may be slightly, the claim of data loss may be somewhat overblown or, like maybe I, shouldn't. That's like well.

B

I think the the illumos they specifically found a problem, but but that's true one's been able to.

E

B

That on Linux, so I don't know. That's.

E

True, the Olympus bug they did say that the metadata ended up horribly mangled after it happened. Sometimes that is true.

A

All right, and then this one where the metadata about the key got changed incorrectly, um is that this one uh issue number one one, two, nine four. Let me just.

E

A

Mounting encrypted data sets corrupting metadata.

E

uh There's one that I had: uh let's see one one, two nine four, you said.

A

A

C

Yeah, the spreadsheet that was kind of cool I.

A

E

um um There's I believe in a couple reports that are probably what I just described with a couple different ways: there's a very good reproducer for it, so it should be okay, demonstratable um I. Also there was another one that sort of had overlap in the failure mode that it looked like where it wouldn't unlock, but it was a different problem um where it didn't update the Mac for the uh user, accounting metadata correctly, um so I added a patch that just throws us out. If that happens, because we can just recalculate it. Okay,.

A

That one's fixed now.

E

Right, that's correct, um I think.

E

One one two, nine four was the one that I just had the pr for um I believe because it was yeah.

A

Because this was just marked well, this is marked as completed about a year ago.

E

Yeah, it got a fixed merge that caused the problem that I just had a PR for so it got reverted for that reason. Okay,.

A

E

It got revised and still had that problem, so I then had a PR and now so far it hasn't had a problem again.

A

Okay, so that one's okay, so the the whatever issues remain with the data set mounting, um are.

E

A

Which ones? Those are.

E

Yeah I should make that clearer.

A

Especially if there's a reproducer that seems like something that we could, you know address more easily than some of these other ones.

E

E

I can find this offline if it's going to take more than a second or two, but uh that's uh right, that's it 12614 I believe is the one that I'm thinking of.

E

It uh basically, if you change key or child and then receive it back, it resets the parents key.

E

Without re-keying, it.

A

E

Is a summary from my recollection of how this works, but.

A

Yeah, well maybe this is, you know, the list of bugs is is long and maybe a little bit daunting, but.

C

A

Fine, uh but this one seems like it would um like it's kind of a sweet spot of there's a reproducer and it's the severity. Is you know it results in you not being able to access your data, so that's very bad. um Some fixing this one might be both possible or plausible, and um you know high impact.

E

A

E

A workaround for it that I need to finish cleaning up, because there is a force change key operation, half plumbed in right now.

E

So if you know the key that you need to reset it to you can do that. Yeah.

A

That would be kind of the way out of this yes, but um is there a way to avoid getting into this to begin with.

E

Probably um I mean I would assume you can just discard change key operations from a receipt um yeah.

A

That would like make sense, probably.

E

Well! No, because if you're doing send-w, that does matter doesn't it, but uh it's probably just not applying it at the right level. Yeah.

A

A

So when you do the change key I'm, trying to remember how how this all works, it's changing the yeah. It obviously isn't changing all the data. That's already there.

A

um It's changing the wrapping key right, yeah.

E

What happens here is it says it has changed the wrapping key, but doesn't.

E

um Well, specifically, it updates the parents wrapping key to have a different ID, but doesn't actually re-encrypt it or something to that effect, all right so forcibly resetting the metadata allows you to get back at your data.

A

E

So what it should be doing is I think breaking because this happens. If you break the encryption relationship on the child and then incrementally send it back um and then it rewrites the parent yeah.

A

I see yes, I mean it sounds like this is just like some kind of logic bug with you know, receiving an incremental on a child where the change she has happened and the change key is currently like inherited from the parent. It needs to realize that, oh now, the child has a different key.

A

We were putting that new. You know wrapped key in the child, but not changing the parent.

E

The part of the problem might be that you might not necessarily know that the you might not be have enough data to distinguish the two to.

A

Know that's happening. Yeah yeah.

E

um Like the send stream might not have an updated to tell you that it's just the child, whose key changed.

A

E

Something like.

A

E

But if you're just receiving the child, you can presumably ignore it. If it tells you to re-key the parent, because that would be bad.

C

E

No because you could have repeat the parent on the other side, damn it wait! No, because you don't want to yeah, you can do it. That's fine.

A

I think: um well, you know if you or anybody else wants to work on this and wants to like uh I'm I'm available to talk through, like you know how the encryption or sorry how the Center receive stuff um works and like how we might need to how we could change the format of that to include additional information if we need to um so.

C

A

Me offline, if.

E

You want to see if I can carve out some time for it. I've had trouble carving out time to work on anything recently, but we'll see this should be hopefully fairly simple to resolve this one at least.

E

um I think there are other cases that trigger something similar, but this is the one I have with a ready, reproducer.

H

I am still invested in the encryption and I use it quite heavily, but uh and I'm willing to help. But every time I look at uh issue tickets, they're, always raw sand and not encryption. So I, don't every attack every all the problems. You guys talk about how encryption isn't encryption? It seems to be raw, send which.

E

A

Anything about.

H

Sending is you know, uh ugly Cody, for me be so blunt, whereas encryption are considered to be quite nice.

E

You can trigger at least some of them without doing a raw sand, If you're, receiving to an encrypted data set.

H

Yeah I would still classify that as standard receive, as opposed to encryption.

B

A

The the one I linked to kind of encryption and send receive.

H

A

B

Like the issue, 12775 is just reading data from an encrypted data set.

H

And no send received with it correct.

B

No send receive yeah there's some cloning going on, but that's yeah.

H

I'd like to help with that yeah.

B

B

um But yeah they have I, think 15 different stack, traces and that have come up on that one. It's a bit weird.

E

I think I would guess that one is probably the same. Well, you know it's the same thing right, you're, counting it wrong, and then it just depends on who dereferences the.

B

Notebook or who ends up trying to poke the null and where it explodes yeah.

A

All right uh there's a couple more items on the agenda. uh What other folk? What other thoughts do folks have on encryption.

A

Right well, if, if not, then we'll leave it there for now. um The next issue on the agenda is an M map issue. uh Yes, um that uh mark the Ubuntu guy uh filed I, don't know if he's yeah.

B

And uh it's a customer investigating the encryption problems for ran into this, uh where I think they're, running Google, Chrome in a Docker container, so docker's doing a clone and then firing up Google, Chrome and then Chrome's panicking trying to read the executable uh when it tries to end map it and the page. Up-To-Date macro returns false and somehow, basically, the MF copy and ZFS copy of the data. Don't think they're the same.

A

Yeah, that um sounds bad. Obviously,.

H

A

So the issue: it's.

A

Is the issue I see, there's like the last comment that you noted was a panic, but.

B

Basically, when the.

C

B

Debug is enabled, and we actually have the assertions then it goes off.

A

B

A

um We're doing we're doing like a read syscall is that right, yeah. We do reads this call and then it thinks that the you know the reasons call normally. It should be um getting the if the. If the data is nmapped, then that might have been modified. So it needs to read the data from there instead of from like from The Arc. But um this assertion is is saying: like uh you know, page up to date is saying that the page is the same like be you.

B

Know it's basically, it's not dirty the page. Isn't.

A

A

Yeah got it, um but we had I, guess, presumably determined that it was I'm not familiar with this code. We must have already thought that it we didn't need to do this and that's why we had that verify.

B

Yeah, it looks from the initial report. It looks like if uh debug wasn't on, then you could. It would not detect this and just return. The incorrect data yeah.

A

um But with debug on it's like the page up to date is out of sync with something else. We're kind of disagree is with something else, though we made the check based on I, don't have the code handy.

A

And just bring up the code to see all right, so it's from so mapped read does find Lock find lock page and then, if it finds it, then it says assert page up to date. Right.

A

Find lock page that must be something from Linux kernel.

B

And looks like in uh update underscore Pages, we set page up to date, flag.

B

I, don't know if we ever unset it how that works.

B

Yeah, when it Files Memory map, we must keep the I o data synchronized between the dmu cache and the memory map pages. So on right. We if we find a memory Mac Page, we write to both the pages and dmu buffer.

A

Yeah and then, when we read it should be reading from the page if it's a map, because it because there could have been stores to the page that we don't know about yet.

A

um I, don't even I'm not sure what this means, what what the page up to date actually means in this case.

A

It's like some, you.

F

A

Some bit associated with the page but I, don't know what the meaning is and why this decision is valid here.

A

Because, like you know, we do the find block page. Presumably that's like fine, the page pointer if this is mapped and then we're reading from there, which is like okay, that sounds right, I'm, not sure what the page up to date means like if, if the page is not up to date, what does that mean.

E

uh The Linux documentation seems to suggest it means it's dirty specifically that every byte in the page is at least as up to date or sorry uh uh when everybody in the page is at least as new as the corresponding one on disk. So sorry, it specifically means that we.

F

E

Think had an update elsewhere and not filled it in.

A

Yeah, so it's not that it's dirty like there have been stores that haven't been written to the file system, but rather there might have been like rights to the file that aren't reflected in the page, which shouldn't normally be the case. Because, like you were saying, when we do the right, then we always you know, write to the page as well and.

B

Then they only the only place in ZFS would recall. Clear page up to date is from zpl read, page common, which is only used to support nmap. There will be identical copies of data in the arc which is kept up to date via right and right, page.

B

Yeah, this one's very Linux specific, so I'm not much help.

A

C

A

There's some more digging to do here to understand like what what exactly all these terms mean and do in terms of the interaction with the Linux kernel- and you know- hopefully this page up to date thing is uh that page, up-to-date Panic is the same as the issue with returning the wrong data, um since obviously, that's like the page up to date. Thing is a lot more of a clue as to what's going on.

B

Then it looks like they have a reproducer with Docker and some set of instructions for even doing it without docker.

C

A

Well, I mean I agree. Did you put this on the agenda? Yeah I.

B

Just I put the link in there because we were talking about it so that people watching the video later be able to find the link.

A

Yeah um well, in any case, I agree. um This is a an interesting and important bug.

A

uh Any volunteers to continue taking a look at this.

A

Right, we will hope that one turns up before the next meeting.

A

um The last thing on the agenda is a question about the on disk format document. um If there's been any effort to update it, um I don't know of any efforts to update it. I um posted it on GitHub, including the source, um so that's available for folks who want to work on updating it, but I don't know if anyone's taking that up.

I

Okay, I think I, don't remember where it came up. I, remember recall: there was some effort or some interest in updating it and uh I was always curious about the results of that, because uh I've used it to write a little bit of software to kind of examine. Zfs volume I had trouble with and Uh I. That document is really useful, but I didn't know whether or not it was current or if uh you know could use some anyway.

C

I

I was I was always just curious about what happened with that. So can you point me to where it is and GitHub.

A

Yeah I'll put it I'll post it on the chat.

B

Yeah I imagine uh that might have been useful when Mark Johnson from FreeBSD created ZFS support in our makefs Tool uh yeah. Also wonder if that'd be interesting to anybody here, uh you can take a directory full of files uh and, as a non-user, be able to create a z-pool image uh where you can lay out the data sets and it'll copy all the files and basically write a whole ZFS file system. But uh you know without the ZFS kernel module or anything.

I

I've done, maybe not something exactly like that, but I was working on something similar using go to create and read CFS volumes, and so uh that uh the document is really really helpful.

A

Yeah, of course, the link in the chat. um It's on my GitHub uh yeah cool thanks the doc um I mean it should be accurate, but not complete, um because it's you know 15 plus years out of date. So you know it should get you somewhere. But um if you're using newer on disk formats, then there's additional stuff- that's not in the doc in the document is.

I

It possible to like do we know what sort of things are you know what kind of gaps there are in it and what things that need to be updated or.

A

Yeah I mean I think that this is written very early on so maybe like version 1. version, one or two or three on this format, um so you can look at like what all the changes were since then, um and you know see kind of which ones are relevant to whatever you're trying to do with this. So you know like, if you are not, you know, some of the stuff is like we added d-rade okay. Well, if you're not reading derayed, then you don't need to worry about that.

A

um We added you know a bunch of metadata that kind of describes additional things if you're just trying to get to the user data. You may not care about that.

A

um So you know as a kind of academic or completeness exercise, you could go through every one of the undis format changes and then you know figure out what needs to be added to the uh document based on that or if you're just interested in, like in my use case, I want to be able to do X.

A

You could probably quickly go through each of the changes and see if it's relevant to you know the kinds of pools that you're trying to access, or um you know if you're just doing read only then there's a lot of stuff that you wouldn't care about. For example, yeah, okay,.

I

Cool. Thank you.

A

um And did is anybody aware, uh I mean I, remember people asking about and talking about updating it, but um I don't does anybody know if anyone's followed up on that.

B

Not that I know yeah.

A

Not that I know of either I just wanted to make sure that we got the question out there in case. So.

B

I think the biggest differences will be stuff like the space map log and so on. Right.

A

Yeah that was so long ago that forgot about it but yeah yeah. So it's not blog and other stuff and again that that kind of stuff is not necessarily needed. If you just want to like read an existing pool, read user data from an existing pool, because that's kind of like metadata that you would need to understand to be able to make changes um or if you wanted to like kind of print everything out for completeness sake,.

I

So my instinct is to want to volunteer for such a thing, but I think what I'd like to do is uh actually see if I can do anything before before I commit to doing doing it. Yeah so yeah.

H

A

uh Okay, I think that's the end of the items on the agenda. uh Anything any other topics before we call it a meeting.

F

Very briefly, the make FSD ZFS that Alan mentioned is indeed working and it allows one to cut out a whole bunch of steps when quickly, hopefully, testing, say native encryption, so I've been using it pretty heavily this last week and I'm impressed.

H

B

Yeah, it's very useful for things like I want to make a bootable VM that has a ZFS root file system uh or whatever, and be able to just take all these files and check them in a an image file.

F

And those images are Hardware bootable because previously is quite flexible on, be it ufi or bios and I'm finding it very useful for testing the current branch in the previous.

I

What is it written in see? Oh okay, yeah makes sense.

B

Yeah I think originally was going to try to use libz pool and so on, but uh Mark quickly. Just did the bare minimum to be able to create a a readable pool.

A

All right thanks, everyone um next meeting will be in four weeks, uh which is actually January 31st, so we'll have two meetings this month um and it'll be at the same time, 1 pm Pacific see y'all later all right.

F

Thank you, bye.

H

F