OpenZFS Leadership, 15 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: September 2021 OpenZFS Leadership Meeting

Description

Agenda: OpenZFS DevSummit talks; zfs list performance; only allow encrypted send; stalebot / issue triage
https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#

A

All right: well, um thanks for bearing with me everyone and uh welcome to the september 2021. Oh, uh opens the events leadership meeting, um let's get started. uh Let me check and see if anybody else added anything to the agenda.

A

um I was just going to talk about the developer summit, uh all right so um yeah the open, zfs developer summit. We posted the schedule. um I will.

B

Share my screen.

A

So you can see um the talks that we've selected. We have uh eight great talks from community members um and if you want to see more about the talks, you can click through here and then uh these are the texts that people submitted about the talks.

A

um So the conference will be november 8th to 9th you can register now. uh The format will be pretty much similar to last year in terms of um you'll need to register, for registration is free, but you'll need to register to get access to the zoom um to post questions and answers. um Otherwise you can join on on youtube streaming, live on youtube um cool. So thanks to everybody who submitted talks, I think it's a great lineup this year uh and I'm excited for it. uh Any questions about the developer summit.

A

We'll have a hackathon second day. um I'd love to see some of the folks here, uh step up and like leave some hackathon sessions. um One of the requests that we got after last year was to try to organize the hackathon a little better uh meaning to like direct newcomers, a little bit more as to what to do and how to participate.

A

So um I think we had. Maybe one um group that was going to set up that way last year, but it would be great if we had more people that wanted to help newcomers and get help from newcomers um with their projects.

A

So we can try to set that up beforehand.

A

Questions about opensuse best developers.

A

Right, um the other topic that I saw on the agenda. Oh great, more topics, I'm speeding up zfs lists ready for review. What was that alan? Do you want to say yeah.

C

uh So this is an expansion of some work. uh Paul doydek did for freebsd like five or more years ago, uh where, if you're doing a zfs list of- uh and you only ask for fields like the name and the grid that don't require actually uh opening the the data set itself, um then you can do the list a lot faster because you don't have to actually go and read all the object data.

C

So there's a certain limited amount of metadata that you have about the data set that you can get quickly without having to actually go and open the whole object set, and that was already there, but I expanded it to incorporate some of the other fields that are readable that way, including the the new one, the create transaction group.

C

uh So what transaction group this data set was created at and so, if you list and sort only by that subset of fields, uh then it the list returns a lot faster because you don't have to go through and actually you know open every one of the snapshots. If you have, you know a thousand snapshots or something, um and so the speed up we saw was something like 300 to 400 on something like 20 000 snapshots across or 2000 snapshots. Each on 10 data sets or the same thing.

C

For you know, the same number of snapshots spread across different ratios of snapshots to data sets.

D

And alan, what is the the sort of range of you said that name good txg.

C

D

C

Create transaction group the number of clones, whether it's inconsistent, if it's redacted and the origin, uh you.

E

C

From the like, the fizz t, instead of having to actually open the object, set and and read the zap of properties, okay,.

D

And then that there is, you know, there's enough use cases around just wanting that information that it makes sense to to do this work. It's not like it's gonna, be rare. Someone would just want that information. It seems I mean it seems like it's reasonable, but I was just curious. Yeah.

C

So uh the reason I did the extra work was pavel's version was mostly, you could get the name in the grid right.

E

C

Did zfslist.oneamed s name, uh you could get all the snapshots sorted by name, but that might not be in the order.

C

You actually want them in, because you need them in uh chronological order to be able to do things like replication and the creation time is only in the zap, but the create txg is available without having to read this app, and so now, if you can sort by the create transaction group, you can get them in chronological order quickly, which means the script that you know compares the list of snapshots on two different machines to decide.

C

What to replicate can now get that three to four hundred percent speed up, especially on cases where you have a long tail of snapshots right.

D

That makes that that makes sense as a use case, yeah right.

A

So is this essentially it's loading, the dsl data set t, but then not going to the object set t. I think.

C

I wrote this a while ago, but I refreshed it recently um sounds.

F

C

Yeah, so when you call um the ioctal to get the stats.

A

There should be like a bunch of stuff in the data set too that we could provide like sp. You know not everything, but a bunch of space, accounting information and things like that. Possibly.

C

I'm trying to find it in the diff here, but yeah there's a a version to get the stats called underscore fast that only doesn't open as much and it's much faster and I just changed the kind of subset of cases where you could use that and made it broader.

C

I think I got everything that was easily available, but there might be more stats so that you can get uh because I think like by default when you're looking at bookmarks, it looks at the logically referenced or something, and if we can get that quickly, then that means more things can be fast and there's some places in this where, for example, if we can get some of that.

C

Space accounting stuff faster, we could make zfs destroy with dash v for a range of snapshots, much faster because right now it's doing a lot of math. um It's a separate issue. I'd like to look at maybe is breaking up uh listing the snapshots that you're going to delete from listing how much space that will actually free because calculating the space makes it take a really long time.

C

When you have you know, 10 000 snapshots you're trying to delete, because somebody forgot to make a rotation policy, um and if you do dash v to see the list, it also calculates how much space that will take, and that means it takes a long time and sometimes you want. You know I wanna. I want that v to see the list of what this would do, because I'm using no op flag or whatever, but I don't actually want the space calculation done.

A

Yeah, that makes sense, um it sounds like maybe it's using dsl data set fast stat. Yes, internally,.

C

Yeah, that sounds yeah, so uh it's deciding when it can uh broadening the cases where you can use dsl data set fast stat instead of the full dsl data set stat.

A

All right sounds good uh sounds like we should review that and get it in.

A

Cool it looks like there's. uh One more thing was added about: zfs allow raw sends only.

C

Yeah, so this one is a question that came into our podcast last week uh someone was setting up automated backups with zfsend, and uh you know: they've created an unprivileged user that could do the sends of their data sets, but they want to only allow them to send the encrypted version so that they can't see the the plaintext versions of files.

C

But currently the allow system only allows send or not send you can't say, send, but only raw sends, and so that means anyone who can do a replication can get the plaintext version that maybe you know for the use case of I want to back up to a remote system, or I want a remote system to come to my system and pull it back up. I want to only allow it to get a raw version that that doesn't actually give them the plain text.

A

Yeah that um that might be a little tricky. I get the use case where it's like you want somebody. You want essentially somebody who's kind of like assist admin, but not uh you know you want them to be able to do it.

D

Yeah privileges model, yeah, yeah, yeah,.

A

um The tricky thing is that, like the raw send even like the raw scent command, is you can do a raw send on a non-encrypted data set? And then you get you know just like the compressed version of it. um So.

F

A

Really designed with security from.

G

Like necessarily the person on the command line.

A

G

A

um So you need to have like a few different checks. There um I'm sure it's doable, though you know.

C

If the data sets not encrypted, then you know getting it. Raw is still something but yeah. uh It came up and I actually was thinking that might be an interesting project. That's bounded enough to be able to do at the hackathon, although if it's too.

A

Complicated, it might not be.

C

A

I would describe it as I think that what you want is not raw send only, but rather encrypted sentences. Yeah yeah.

C

Yeah, making that extra distinction there makes sense.

A

Cool um other questions about that.

A

All right, then, let's open the floor for any unscheduled uh questions, comments, topics.

G

Yes, I have some several questions about: zfs tests, configuration environment under linux and probably freebsd.

G

You are using not a zfs root and using zfs as a model, what you can load and unload and use used by zfs tests and the local created files and provided by device and the device via we have external provided, drives to vm as a full emulated how we can work with physical drives.

G

We try to tests and the different virtualization, such as vmware qvm behave, and we have found interest issues where we try to create the talk label with slices, for example, for cfs, create the create tests where we can create zfs and the slice after that. If we try create a label where we can use.

G

G

We have found some unknown issues where unknown errors, where some tests not pass again with the next circle.

G

In the fs tests, you try to use only one circle and automation system, but if you try to rerun your tests on the same devices, next try or several tries you can see. Some tests have a different results.

G

We try to use it for stability testing and right now I try to find out what the issue we have with drives, what we are using for the gfs tests as a physical devices, not by clearing files as devices for one circle of tests, it's magic, probably for stability, where we try to run cfs tests in loop, but some tests failed with unknown reason.

G

I have no answer for now, but try to find what the issue I have ported all the fs features to deals from cfs and right now we can create a pool on the linux and import studios and put it back without issues, but some the first properties hidden properties should be updated for moccas consistency.

G

And what is the question now.

G

What is a reason? Try to use a local drives local files and provided by devices for the fs test instead of provides drives to vm as emulated physical device.

A

um Let me make sure I understand your question you're, saying um you're, you have a problem where, when you run the zfs test suite the first time it runs okay, but then the second time uh it doesn't run it. It fails and it sounds like the problem is related to the physical disks on the machine. Having been already labeled.

A

G

Yeah, yes, okay, I I I tried to explain some unknown issues on my side where we can see some tests fail and the same drives would have been provided to me.

G

But if we clean up it before jfs tests, many tests pass, but with next trial to run the same list of tests, several tests can defend.

G

If we are using, not client drives, got.

A

It um yeah, I don't uh I'm not sure. What's up with that, um the uh I mean sounds like there's some cleanup that needs to be done to get it back to the original state. That's not happening um seems like something that we could probably investigate and fix, um and then you you're kind of had a bigger question about. Why do we use? Why do we use those disks rather than creating files on on the root disk? Is that right and using those.

G

Files as as v devs- yes, I try to find answer or question about probably for better testing will be much more better. Try to use the same drives for every zfs tests where we can try emulate works with physical devices. For example, you can get one drive from one system and put it to another system, and it is the same physical device if we create every time clean files and provide it with it by devices.

G

We do not know that we are using the same physical device for tests, yeah.

A

um I think that, well, there are a bunch of cases where we do use um file v, devs, uh where you know like the configuration doesn't match. You know you need a bunch of disks to test raid z or whatever it'll use file.

A

If you devs um I'll I'll, ask john kennedy to weigh in since I know he originally wrote a bunch of that, but my guess is that we wanted to test the actual um disk v dev type um rather than the file view dev type, although I mean I'm sure you could argue that there's a lot of tests that don't really care about the v type, that's underneath it and should those be using the raw disks versus using you know, file-based pools. I don't know, I think you'd probably argue.

G

A

um But john, what were your thoughts on that.

B

The the cicd app for opencfs actually uses uh file-based loopback devices, um but at delfix we use uh disks that have been provisioned to the vm.

B

um We in our automation, we don't have anything that uh uses the same disks for more than one test run in a row, but I know that uh from time to time I do that manually and I haven't seen what uh you're describing igor. I I wonder if um probably the best thing to do would be to to raise an issue sort of describing what new failures you see, what the errors are and that sort of thing so we can investigate it.

G

Okay, I tried to find the steps for reproduce this issue with and try to minimize a list of tests. What can impact of physical devices for next testing, where I have found corresponding tests? I will try describe it on issue and will ask for reproduce on next environment.

G

We try a transcript. Okay thanks john.

E

And there's also.

G

Some another question about configuration of zfs tests on devas. We are using temporary files and zfs and for some reason, for some tests we try to change.

G

Global variables for next tests, and if you have a root file system, where you try to create a local files for next tests, they can be impacts, gfs itself, because, for example, if you try.

G

To use corruption of file where we tried to create a tool and did not remove it before next iteration. Your test will be fair because it it corrupts by tests but did not remove after tests.

A

Yeah, there's a lot of tests that um I think don't properly clean up the state back to how they were before. Yes, yes and we tried to find what.

G

Tests should remove artifacts after tests, what we are creating for testing only because it is the next iteration. If we have previous data, some tests failed. This all the data.

A

Yeah, I think that um for the most part, what I've observed is that the cleanup stuff works right when the tests pass and um but then, when the tests fail, then there's a lot of then you know a lot of them do have code to that's supposed to clean up, even when they fail, but there's some where that is not complete um yeah. I don't know if you have thoughts on that. Well,.

D

I I just also want to inject that you know it's been my experience that you know. I think the testing cfs test infrastructure in general has atrophied a little bit with respect to sort of live, drive, uh test machine type test environments, because.

E

Almost all of it's.

D

Run now in vms and where things you can clean up by just simply throwing away the vms and throwing and putting a new one up, and so I think, there's been a lot less effort to make sure everything is cleaned up properly these days.

D

G

Fs tests um are using one iteration. If you try to run cfs tests in loop for several iterations, you can find different tests can be failed.

G

I think design of the fs tests was only for one iteration, but for stability will be much more interest. Try to run cfs tests for long time for 10 iterations or many other situation iterations, and I think we have to clean up of some issues for run it in a long time absolutely go for it after after some fixes.

G

I will try to upstream of our findings and we try to complete of mergers with open cfs, but it is a slow process on different platform.

C

Just for clarity is this the z-test or the zfs like functional tests functional. I.

G

C

I've used the wrapper script in the scripts directory to run multiple iterations, although I think that was mostly of a small subset of the test. Not all of them definitely said you know, use that script and tell it to run 10 iterations or whatever.

G

uh We can run gfs tests in several iterations. Okay, runner have been fixed for this for minus I option, but you completely have a summarize of all tests of all iterations, not only for every iteration.

G

That's all on my side, thanks.

A

A

um It looked like there was one more uh bonus, content, special content.

H

Yeah, um yes, so uh you know, I'd occur here with josh and uh josh have more to say on this subject, but we um bumped into josh's former colleague alex, had hit this issue and illumis with some data corruption in um with uh encrypted gfs in illumos, um and then we we started stumbling into a bunch of issues related to it in github that were of the form yeah. I posted an example there of the form. Like someone hit a panic, often data corruption.

H

Maybe they followed up, no additional follow-up. Stale bot closes, and I don't know I mean that seemed discouraging, especially like. If someone had gone to the trouble of doing- and you know making a bug report um and in this particular case it seems like there's still some outstanding data corruption. So I think that you know we're we're looking at the this particular data corruption issue, because we've we've gotten interested in it.

H

But my my my meta question is about stale bot and um you know how how I understand the need to to kind of keep issues under some manageable content, but if their stuff's being filed and never looked at, if that's the like, if that's how um you know open zfs, is managing bugs right now,.

A

um Yeah I mean I'm I'm happy to share my thoughts on that. I don't know uh I'd not set up the stylebot, um so uh maybe I'm not sure if anything.

G

A

Here so I'll just say like I'll, give my thoughts and then, if somebody wants to like correct me or share kind of more the back story, there I'd love to hear that as well. um I mean uh it sounds like uh there's kind of two problems that you're talking about. One is um like there's issues filed that seem important, but, like seems like nobody's, really looked at them and the other is um then the issue is moved from open state to close state automatically.

A

um It seems like, in my opinion, the former problem is much more acute, but but like the closing, the bug seems like maybe it's sending the wrong message, or maybe it's sending the right message um or maybe saying an accurate message, which is like not a good one. um Where did you uh did you have? Did you run into problems because it was in the closed state or is it just?

A

Was it more like you ran into you found the bug, but then you saw that it was closed and then like it felt like the like. The project didn't care about it because it was closed. Yeah.

H

I guess I guess you're right, that there are two problems: the the like there's, there's issues that people file and they like, apparently nobody looks at them, and I and and um and then the like stale block, closing it out, and I guess the problem you know to your specific question was the fact that it was closed additionally problematic or was it merely concerning?

H

uh I think, maybe both I think, maybe both in that um you know like. I think we gravitated more towards open issues than to closed ones, but that may also just be like um pilot error in that you know you knowing that there there are potentially things in the closed state that have not been attended to, but merely are stale. Maybe that's like a sufficient. You know data.

E

H

To point people in the right direction for like don't just scoop through open issues, but also scoop through closed issues because they may be indistinguishable modulo. You know some amount of time. Yeah yeah.

A

H

And also to be clear, matt just responding to one thing. You said this is not to like accuse or to suggest that setting up stalebot was inappropriate or anything but more to ask the the disposition and to see if it's concerning- and it sounds like on its face. At least you share the concerns that we do.

A

Yeah I mean my my biggest concern is like that. There's issues that are not being addressed.

E

A

I I like, in terms of stillbot specifically like in in my opinion, I think that it's valuable for the project to be uh to be forthright about, like where resources are being spent and um like the the fact that she wasn't receiving attention is like yeah and then was closed, is like that's the honest representation of where the, where the, where people who are contributing to the project and where they're not um so. I think that, uh with respect to stellbot and issues being closed or not like, let's say, sterobot didn't exist.

A

I don't think that you would have been in any better shape because, like yeah, you would search for open issues and you'd be like great there's, like 10 000, open issues you know like um versus, like at least with stalebot. It's like the open issues are the ones that have received activity in the last uh five quarters.

A

um So you know there's a better chance that it's like on some news radar um but like. I can definitely see differences of opinion on on all that stillbot stuff. I think the biggest problem is: how do we raise visibility of issues that are important and I think, usually it happens by like the squeaky wheel, getting the oil, um which also tends, which also will keep still by at bay.

A

You know um that's one of the reasons that, like still about like yeah, like it posts on the issue, and then you still got 90 days before it's going to close it if nobody's.

A

If, if nobody, you know who was on the issue tracker, including the original person, pipes up and says, like oh yeah, I'm still hitting this or still seems like a problem, um then it's gonna close it out. I think uh yeah go ahead.

E

I was gonna say: add to that matt. I think that one of the things um that stale about will probably change kind of our behavior is that if there are issues that somebody has filed, but other people are hitting it that should drive kind of like which issues we need to go address now we still have to figure out how we, you know, go around addressing and making sure that they're there, but um oftentimes like it might be an issue on a specific configuration. It's like it can't be reproduced it's like.

E

Is this happening all the time? I don't really know like um versus one that, like 15 people have commented on and you're like? Okay, that's the one that's going to be the priority, so maybe this kind of changes the way that we look at some of these bugs um and force people to like if you're hitting it, you know, make sure you're like adding more content or trying to like you know, introduce like new new information and that might help um the project kind of like address it sooner.

H

That makes sense. George, it does seem to like have a single access of sort of popularity, whereas there's another access of severity. I agree um where you know lots of people hitting some usability issues say in a cli may attract a lot of comments where a few people hitting data corruption may not but kind of. In terms of I mean even reputationally for open zfs as a project like the latter may be more significant or maybe the harbinger of something more significant.

A

H

A

Supposed to um not uh not close or mention issues that have been marked like um uh like status like type defect or something like that so basically like if somebody has triaged it and and then like yep, that's a that's a defect, that's the actual problem, then, I think still about leaves it alone or is supposed to. I think that may have been recently fixed.

G

I think we need to triage between platform guys because one issue can be reproducible on one platform, but do not reproduce on another platform. Also open, cfs supported some features to lumos and not features all features available on the rooms right now, as I know, I have ported much more features to deals. Elimos based distribution, but gfs on nadimos and deals are very different.

A

Yeah I mean in terms of different platforms. You know the the open, zfs github issue. Tracker is like primarily for you know it's it's for all of the platforms that are supported by that code base. So uh you know linux and freebsd and and maybe soon, mac os um for bugs that are encountered on other platforms. You know they they have their own issue trackers. But um you know if it seems like it's something that would affect the common code base, then uh for sure file, those bugs against opencfs uh github as well.

G

I tried to do it right. I.

A

G

Open issues what I found under my platform at and we tried to reproduce it from another and take a look if it is, reproducible, try to fix it. If not, please, let me know that's probably some platform specific issue or something else.

H

Yeah thanks for doing that and matt what what is the triage process such as for in for new sorry guys? This is smirk coming up. um Sorry.

G

H

Can't have this? No! No! No! I'm sorry for for asking a, I guess, stupid questions.

A

No, uh it's not a stupid question at all. I.

H

A

As far as I know, the triage process is uh essentially non-existent and um it's it's kind of uh so.

G

It's a volunteer.

A

Basis, you know in terms of issues we we've you. We, I think that the the project resources uh such as they are, are, um are buried under prs and um we're. You know we're trying to focus on being responsive to pr's um in terms of like the the leadership right um and and kind of who, folks that we can ask to do stuff.

A

That uh is not maybe what they would volunteer for, um and I think that the issue triaging is is, at this point, a purely volunteer effort with little to no process that I'm aware of.

H

So so I mean I get it, I get how I understand why that would be the case, but it sounds like the priority is features that are driving the participants of the project and and like the unfortunate but sort of necessary work of of bug, fixing, as it may not pertain directly to the products that folks are building.

H

You know falls to the to a is more of a volunteer basis.

A

I would I would just qualify that slightly by saying that, like it, it's not, you said like it's, it's the features that people care about that are getting. You know attention from the project and I would say you know it's. It's not just features, I mean, there's there's plenty of pr's that are bug fixes um and you know we for sure care about those getting those in and those are oftentimes easier to get in. That.

D

That could be the trick to getting attention to your issue is create a pr for it with some random incorrect fix and say this is a review.

H

Okay, uh thanks for that pro.

A

A

Well, I agree, I will sign this to you mark and then you can um find code reviewers. That's.

H

Right, you can code review those fixes into existence. Mike is now responsible for triage.

H

Okay, cool! Well I I thank thanks for answering the questions mark, so I mean uh matt sounds like and mark. It sounds like there isn't any particularly concrete action here. um It sounds like you are thinking about content for the developer summit. I don't know if there there might be something here, whether it's quantitative or qualitative, to consider to present to the group, but just throwing it out there.

A

Yeah I mean I hopefully I've like you know at least given information about the current state of the world. um I would love to see. I would love to see more attention on bugs, um and uh I think that this is an area where the um you know the volunteers could be very helpful uh with in terms of triaging um and and raising the attention of these bugs, like the ones that you mentioned, that slips through the cracks that do seem to be.

A

You know very impactful, um raising the attention of those two developers and other folks who can you know, take the next steps to actually investigate it, um because there is like a large volume of issues I mean. There's just you know now, there's 872 issues open, um you know, even uh after stillbot has had its way.

A

um So you know that's like a lot of work to ask somebody who could otherwise be fixing bugs or or implementing new stuff to go, spend like a week doing that. um But it's something that you know. People who want to contribute to the project, who might not be like c coders, um could really help out by looking at these issues and then raising you know, raising attention to them, making sure sterling doesn't close them, marking them, and you know bringing them up at meetings like this.

C

Well, uh something we maybe have a small group do at the hackathon. Is you know like we've done, freebsd4 have a bug squash of just get a bunch of people go through look at it and especially if they can notice hey, there's these four different bugs filed and it all looks to actually be the same crash.

C

uh So it's maybe it's more widespread than we think, and we should aggregate in one issue and get attention to it, and even maybe, at the end of uh the hackathon in their presentation, is here's the five or ten bugs that we think are in most need of attention uh and can do that and yeah that might help a couple. People get started with the process uh and be comfortable with doing that on a more ongoing basis as volunteers to keep doing it. You know when you're new, you kind of feel weird about doing it.

H

That's a great idea, alan in terms of like you know, people who are interested in doing something useful, giving them something very useful to do and concrete and like that they can make progress on and give them a little guidance on something that would have a big impact to the project.

A

Yep yeah uh alan is that something that you would be willing to lead at the hackathon. I was.

C

Trying to do the encrypted one.

H

Even saying the parameters of it would be really helpful, I think you don't have to write the code, but even telling people like this is a great end state to get to in a day to achievable and like you don't need to do it yourself. I don't think yep.

C

But yeah, uh I think, that's an interesting idea. You know we've done a couple like that in freebsd. You know the day before a conference, even just a bunch of developers.

C

You know you take the first 50 and the next 50 and just chunk it up, and then we presented the most interesting ones to each other and looked at you know are these some that we may actually be able to get closed uh or fixed or whatever.

C

Although I think the outcome from the hackathon, if we picked out the ones we wanted to fix, it would be a bunch of the little ui things that don't matter and and the the nastier ones would get left in the pile, but just having a list of the you know out of the 800 here, the 10 that actually need the most attention would be a useful outcome.

H

Or even doing the sort of deduping work of saying, hey, we've got these 20 bugs, but they all have very similar stack traces and it's worth at least you know guilty improve until proven innocent.

C

Yeah and also just making sure the tagging is right, it looks like uh the user who creates the issue, can put the type defect flag on it and maybe there's some that shouldn't have that flag or whatever, but making sure they flag right, so that the rules we have in the um stale bot uh apply correctly and so on.

D

So I hear that you, instead of having a hackathon, what's called a bug-a-thon yeah.

E

And nobody tries to quickly fix as many bugs as they can.

D

At the end, you get points for how difficult the bug was that you fixed, and the winner has- has fixed the more difficult bugs yeah.

A

That's a totally fair idea that we make that the focus of the hackathon, and maybe the you know, I think traditionally, we've had like three prizes, maybe like two out of the three prizes, go to um bug work.

A

A

Thank you adam for that reminder about hackathons in the chat. um uh I think that the pr to be fair to the project and and those contributing, I think that there has been like there is quite a bit of effort being spent on um fixing bugs and addressing bugs, even those that maybe haven't been introduced by the people doing that work um but like it is not enough effort to keep up with the incoming rate um and, and things have slipped through the cracks.

A

um So you know I definitely appreciate the work that is being done um while still acknowledging that like it is not enough to get us to where we want to be um in terms of uh both, like most importantly, keeping bugs out of the system and and then also being responsive to uh people who are filing, but who are doing the work to file bugs, um which is also necessary and appreciated.

A

H

And again, uh I really appreciate this discussion and hope that didn't come across as accusatory, and I also appreciate that the the values of the project um you know are aligned around this principle of quality.

A

Yeah, no thanks thanks a lot for bringing it up. I think that um I mean I I've noticed all the same things that you have so I I appreciate like bringing it up and having a discussion.

H

Well, and it sounds like I'm really encouraged that we even came up with some real actionable things. um You know bug fixing some hackathon projects and maybe some you know another hackathon project around new metrics to look at uh because, like the qualitative sense of like things, may be trending in the wrong direction. Isn't that helpful, but like turning that into something you can measure seems like it would be tremendously helpful.

A

Yeah, it seems like one of those things that probably other people have um invested in like tools that are on github to like measure. You know bug rates, bug open, close rates, um categorization like making some pretty graphs to kind of like take the tags and show us summaries of stuff. um That would be really cool if we could find those tools and apply them to our um to our project.

A

And uh just to close out since we're almost out of time, um adam did, uh I know, like you're, mostly concerned about like the the big picture in the state of the project, but is this bug in particular um one that you think we should uh reopen? I didn't really have time to read over it during the meeting.

H

uh Josh, do you have do you? Do you want to comment on that, since you spent some time trying to reproduce it on our systems.

A

H

Just to repeat the question: should we re should we reopen this github issue or.

F

I mean I, I think a reference feeling is that the one with the similar stack, yeah yeah.

C

The one that alex did mention.

H

F

That bug report, I think, that's probably a good idea. I don't think it's fixed. It's pretty rare that.

F

I think that it requires importing an older zfs stream that would require certain upgrades to occur and then doing a bunch of clones concurrently, I think okay, it seems like without doing all of those things it doesn't at least doesn't occur um in a way. That's noticeable might still be happening, but uh certainly no one has to pay for it uh immediately. Unless you do, I think several times at once, there's some kind of interaction in the arc that isn't quite right.

F

uh Yes, I think it seems to be what uh reopened.

A

Cool thanks and if you run into other bugs like that, then feel free to reopen them or ping me or being focused on slack.

C

Yeah, I think adam.

A

C

More than one uh old bug that was very similar, I imagine this one might have got less attention because it was this happens when I have a raid zed one with two dead drives uh and the long story at the beginning, but uh yeah.

A

Cool um well, thanks to everyone who participated. um The next meeting will be in four weeks um october. 12Th.

B

A

Yes, october 12th um at the later time, 1pm pacific uh and I think that'll be the last meeting before the conference, um so that'll be exciting uh and oh, I found what the weird what the fun music was that was playing. um Someone had left minecraft running in the background on my machine. Apparently it has like a really soothing soundtrack and the icon for it on the mac at least, is like it's like a little terminal window or something you know so.

E

I I say: oh it's like.

A

One of the million terminal emulators running yes java, exactly so mystery solved all right thanks. Everyone see you at the next meeting. Thank you. Thanks, bye.

G