OpenZFS Leadership, 21 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: May 2022 OpenZFS Leadership Meeting

Description

Agenda: Block Reference Table, FIEMAP; Blake3; Write Throttle Smoothing

Details: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit

A

All right welcome everyone to the uh may 2022 open, zfs leadership meeting uh months, keep flying by and uh we're getting closer to the next open zfs conference. um I'll have a few notes on that, but uh we'll take a look at the agenda. There's not too much on it. um So lots of time for other topics that folks would like to discuss today.

A

I'll start with the somebody put this on the agenda, I'm not sure who, but it was a link to the phi map.

A

um Pr, I saw some discussion on that recently. um Were there particular questions, I'm not sure who added it so.

B

I think there was just some recent interest in getting that change, cleaned up and merged, um which I agree with, and I think um I think we might have some interested folks picking up that work. um But we'll see cool.

A

Yeah, I I think that I um spurred that on years ago, with my uh with a proposal to do kind of a subset of that with like a custom myoctal uh we're still using that in our product by the way- and I think you probably saw I opened a pr to like make some minor improvements to that uh performance. Wise yeah,.

B

I wish I should finish up yeah. I think the fine map stuff could be wrapped up. I guess just to summarize there was just some outstanding review comments there about things. We could do better. I know you made a passive initial review and then I never got back to cleaning it up um and you know getting it all tested. There's still some work to be done there. I think it's all, you know very doable. It just needs to get wrapped up.

A

Cool any other questions on that.

A

All right, uh that was the only thing that was in the agenda doc, so um other topics that folks would like to cover today.

B

I just make a call for uh additional reviewers of the blake 3 changes, so I made a pass over that, and things are looking really good to me. I think they're in good shape, I'd like to go ahead and merge them this week this week, if possible. So if people have additional feedback or comments or concerns, please take a look at the pull request and post anything. You can post your thoughts.

A

Can your mind folks: is there anything um in that, uh so it's adding a new, a new checksum algorithm? um Is there anything kind of unique or um or special about how that's implemented compared to the existing checksum algorithms.

B

uh No, not really it's implemented the same way as the eden, r stuff and the sky and stuff it's plumbed in exactly the same way. In all the same places, it's got good test cases. It's got test vectors um for user space and the test suite. um There's a couple assembly implementations for non x86 architectures, but those are all pulled from upstream and like if you've got a power pc or an arm box, and you want to test those out. That would be really good.

B

um We've done some of that, um but any additional testing would be welcome. I put it through the paces locally myself, at least in x86, and it held up quite well. So it looks like uh a hands-down win or better check some.

C

The assembly implementations aren't just from upstream or not, and I know because they came from me- I've misunderstood that sorry, no, um I should be more precise, uh a bunch of them like the x86 ones, all came from upstream, um but I believe the arch 64 one and the powerpc one both got programmatically generated from the x86 intrinsics, because there wasn't an upstream one that was faster and it works fine. Mostly, um I submitted a fix for that.

C

um No, I tested the ppc 64 one on little endian, at least and after the fix that I made it works and the air 64 one I originally did because upstream has in native air 64 one, but it's much slower so, okay, but no those didn't all come from upstream. So if you're just trusting them. For that reason just be aware.

B

Good to know cool.

A

I mean are the: um is it using the salted checksum stuff that we introduced with the those other algorithms?

A

Yes, I believe so.

A

I have to go back.

B

And double check.

A

But that's fine! This sounds good out of curiosity. Are, um are we doing automated tests with uh on arm?

A

B

Not we should we're we're really close, because we've got our we've got the graviton stuff and aws, um but we're not at the moment we're building there we're just not running the full test. Suite.

A

Okay, so we would do a build on arm 64, but um we're not running the test, speed yeah. That would be cool. um I think you know I'm specifically interested in the graviton processors, so you know doing the. If the project did the test feed there. That would be interesting.

B

It should be an easy thing to turn on. We just haven't done it yet, and then, who knows if we'll hit some odd test case failures or something when you enable it right, there might be little obstacles.

C

Cool one thing I would mention the blake 3pr ads is: it adds a generic interface for um check summing benchmarks, so, like the fletcher forward breakdown in um proc on linux, uh it has one of those, but for all the implementations.

C

Like skying shot by 12, and it has a breakdown of that.

A

Cool that's great yeah.

A

All right anything else on blake.

A

uh All right other, what other topics would folks like to cover today.

D

I'm looking for reviewers for the brt work alexander already did some initial review and found some interesting issues. I guess so. It would be great if this could move forward.

A

Yeah, I know you you review from me. um I I'm not sure how much time I'll have to spend on it. I would like to spend some time on at least the kind of high level aspect of it, but I'm not sure which time I'll have.

B

I know I volunteered to review it too, and it's it's still on my list, but I'll get to it cry this week at least make a pass.

D

I know it's it's hard for people to find time. uh That's why I also propose like uh to to meet for some time or even meet after the call. So I can uh walk anyone interested uh through the code, at least to get you some like high level overview of what's going on there. So maybe it will be easier to figure it out uh alan already volunteered to to stay after this call.

D

A

Yeah, I think that's a good idea.

D

Yeah, it would be great. I would just like to push this forward. Finally,.

A

Yeah um sounds good.

A

uh The only other topic I had was the uh onesie fits developer summit um so based on the feedback that we had from this group last month, um we're we're going ahead with trying to plan an in-person event, um that'll be in the bay area. um We would love to have some help with the organizational aspects of it, as I mentioned last time.

A

So if, uh if you or if you know someone uh your company or in your community, that um could help with organizing in terms of the logistics, you know uh interfacing with the venue- and um you know maybe helping on the day of uh with um you know all the vendors and uh registration and things like that. um We could really use some help with it, um but yeah well uh we're planning to go ahead with uh in person um we'll investigate, like the venues that we've used before.

A

And we'll keep folks.

A

Keep folks attuned to that, hopefully we'll have hopefully in the next month or two we'll have the details. Knelt down matt did denise come up with a clear position from that company. um Not really okay, we're um we're talking with her, but haven't really heard back on any specific, like commitments of what they can do understood.

A

A

So yeah um any thoughts on that uh we'd love to we'll have a call for presentations soon. You know once we have the date and everything um and have details that speakers can know like when they would need to be available.

A

um I'd love to hear about brt, uh hopefully, which will hopefully be closer to uh integration or integrated by that time, um but yeah. I think, there's been a lot of other interesting work that that would also be great to hear about the conference.

A

um Any uh other topics for today.

E

So uh hello, everyone, so I believe, two months ago, allan raised some issue with this evil performance and the pr that we proposed to smooth the throating mechanism mechanism. Maybe you recall this case.

E

I can put the pr into the chart, but everyone will be on the same page.

F

A

E

Yeah, so does the issue in general: is that we what we can see that when we are using visible?

E

We are writing a lot of data and the data aren't flash to some level and then suddenly they are flash at once, and what we can see is that the flashing is like done in one let's say, step and that non additional data is accepted until the flash is finished. Because of that, we can see that the bandwidth is dropping to zero megabytes per second, and we had um a little bit hackish way of handling that to add additional uh delay uh to every right after some uh some uh level uh of duty duty data.

E

uh But after during the call, I believe it was two months ago or something there was a couple of ideas which we can look into it.

E

One of them was to add additional accounting and to add this additional size of the dirty, the urgent buffer um to the delay, so actually how it works right now is that we first uh decide how long we should delay uh for the for the transaction group, and then uh we add the dirty data to the the to the counter, and but because of that, we we assume that uh we weren't able to um delay the rides enough to to fill up the duration buffer.

E

uh We test that and it turns out that it doesn't help it's still behaving the same in the same manner. We also try to just add the accounting much earlier, so before actually uh doing the delay, but this didn't work as well. We looked into the accounting of the dirty data and it seems to work fine. I mean we couldn't find any place that actually uh was missing some uh this, where the substitution wasn't done actually um yeah and basically we are a little bit out of ideas. What we can do else.

E

We also I I have seen that enough uh proposed some touch and uh some um other pr that he pointed out. That may be the cause of the issue, but unfortunately uh nor the patch, nor the reverting, the the the uh comic that he pointed out, helped in this case as well. So we still see the same behavior.

A

All right, yeah see the last couple comments are um are from alexander kind of saying like oh, you know, we should do it this other way, um but it sounds like you're saying that didn't work.

E

Unfortunately, now.

A

All right um yeah, it would probably be good to you know mention that in the pr so that you know alexander understands oh yeah, we.

E

Have an also discussion on offline. You know I was thinking that this is a good idea to rise. This is through here as well, but yeah. We, we probably should also do some comments in the pr as well.

A

um Yeah, I kind of assumed that it was like oh okay, well that folks were gonna. Take this other approach that alexander had proposed, but it sounds like that's. That's not gonna work entirely, at least.

A

um Yeah, I I don't have a any specific ideas um based on that. I know uh last time when we discussed it, you know there was concern about kind of the heuristics in here.

E

It was also one more thing about uh barrier that we uh allowed too many threads to write at the same time, but we also limited the threats actually to one thread and it still behave the same so.

B

E

It's not the the truth. The amount of the threats isn't the issue here.

A

Got it okay? So so um so then what uh do we know what the problem is like? What's causing the problem.

E

So that's so that's the so that's the main issue. We don't know the root cause of it. So what we can observe is that the dirty buffer is filling up and suddenly it's just flashed.

E

We I trace it also with evpf, and it seems that you know that the substitutions from delta is done like in smaller steps, not like flashing, all the data at once, but in smaller steps. But still we don't know what the root cause is, and we are a little bit out of idea what to look for.

A

All right, yeah, that's a little unsatisfying.

A

um Because I think the the just to summarize for other folks following along, I think the um the observed behavior, is that um the system gets to having like 100 dirty data, and then the delays is like super duper long and then you're kind of stuck in that state until the txt finishes syncing and um the mystery is like it's not like.

A

How did it get 200 dirty data, because you know, as you can see from the graph here, maybe I'll share this share my screen, so you can see the graph that you put in there.

A

um You know the this is showing like, as the dirty data increases, the delay increases, and this is like a log graph log scale of the delay. So you know when you're getting like close to having all the dirty data. You know the delay here is 10 milliseconds, um which is a long time.

A

You know, especially if you're coming in, like with just one thread issuing right after right after right, like in one second, it could only issue 100, writes when you're at this point and somehow those 100 rights like filled us up all the way to 100 or or maybe the delay was not as big as it should have been, or something something happened wrong that allowed us to get all the way to 100 dirty data and a huge, huge delay.

A

um Is that an accurate summary of your understanding.

E

Yes, that that's correct.

A

And so like the is, is it true that this smooth delay? This is really the only impact of this pr is like you know, when the dirty data is low and it has been high in the past, then we do this additional delay.

E

Yes, we do this, so the idea is that, after reaching the the maximum level uh from after some period, we still add some delay- and this is this yellow delay that we actually um still adding after after reaching it. It doesn't solve the issue completely, but it's still like smoothing the the uh the bandwidth actually.

F

Yeah, it's almost like the delay is coming in too late in the in the in the current algorithm, somehow and then being more aggressive.

E

I tried also to add the uh that the um those uh count of bytes before actually uh doing the delay, so we shouldn't like. If we have a big bunch, then you know we still should have something like right, because right now the accounting is done after uh delay, but I also tried uh to do it uh at the beginning.

E

So at the start of the algorithm.

A

And it still didn't work that would matter if your rights are very large compared to you know the whole dirty data limit right like right. This is four gigabytes. If you're like oh I'm, coming in with a one gigabyte right, then you know it would really matter whether you're doing the accounting before or after, but if you're coming in with like a a kilobyte right, then it doesn't really matter.

A

Do you know what size the rights are um in the test that you're doing.

E

There should be a comment in this pr. If I recall correctly, I don't remember from the top of my head. No, it doesn't contain. Let's let me check.

A

That, then I mean it can't be more than like six. uh I think it's like 16 megabytes, yeah a little bit more than that um yeah. It's very, it's very surprising, especially with just one thread that you would that this you know, yellow part, would matter because it's like okay yeah, like maybe it's smoother to to do this yellow part, but it shouldn't it should, like.

A

You know: you're gonna get to this middle part where the behavior is the same and then like what you know. Why would that matter um like you're still in here in the middle you're, doing a little bit of delay? And then you get to close to here and then doing a bunch of delay and then like something either goes wrong or does it wrong doesn't go wrong, causing us to get up to 100?

A

So it's it's very. It's very confusing.

E

And I can share one more uh graph, actually, which maybe is will be more illustrative here. So I will try to share my screen. Then.

E

So this is the graph how the dirty buffer is changing. We just we just attached to the delta function and we observe how it's changed during the time. So we can see that it's accepting the data and suddenly it's flashing, all the at the same time.

A

Yeah and so the scale here, the y-axis is bytes, I'm guessing.

E

Why is the bias on x is the time.

A

And the top here is like 800 800 megabytes, probably.

E

uh Yes, something like that: it's something around like uh the maximum of dirty data. I believe we have one gigabyte of buffer right now.

A

Yeah I mean it's a little. This is showing, like you, know many txgs.

A

A

Yeah I mean I would expect this makes sense. Yeah.

F

This is sort of common after you.

A

Run into the like, if you, when you run into the bug of like getting to 100 dirty data, then the delay is so great that you know it's not until it drops weight. You know once you hit that like 100 second delay, you're kind of stuck there for 100 seconds and then it can all drain out and then after 100 seconds you can come back in.

A

um The question is like: why did it get so high to begin with and like it's not all I mean this is not one gigabyte, it's like seven to eight hundred megabytes, so you know.

A

Why are we even getting into this bug like? If you only get to 750 megabytes of dirty data, that's 75! I mean that that should be still in the kind of smooth part of the curve it shouldn't be at 100. Second delay and everything should be okay,.

E

A

That's absolutely.

E

Yeah, you are right, like we shouldn't even enter to the smoking mechanics right, yeah.

A

So, are you sure that in this workload we are getting to the 100 second delay, or is it possible that, like there's some other problem, that's causing this sawtooth.

E

Pattern, I would need to double check that to be 100 sure if we are, uh if we are actually inserting the delays, because I did so many tests that I don't remember actually.

F

E

But I I remember there was. uh We was also surprised about the amount of these delays that are that we actually insert.

A

Yeah, it might be interesting to look at, like you know, just one or just a couple txgs worth of time here and then have this amount of dirty data and the amount of the delay kind of overlaid on top of it. um That might give us a little bit more of a clue of. What's going on here,.

E

A

Cool um yeah, it's definitely an interesting mystery here and do you have um steps to reproduce this in the.

E

Yes, it's it's 100 reproducible. uh I can also put it into the pr. I guess that will be the best place.

A

Yeah, that would be helpful in case other folks are also curious about what's going on here and want to try it try it themselves sure.

A

Thanks, thank you. Yeah um all right. Other topics.

A

um If there's no other topics for today, then I'm happy to turn over the meeting to powell and his code review or code walk through.

A

um Although I don't see alan here, he was the one that wanted to work on it with you right.

E

So alan actually mix up the hours and he was thinking that the meeting is one hour later time zones. I guess so he will be probably this half hour late.

A

Okay, then um I guess you can meet up with him in a half an hour um and uh do you. um I don't think this zoom will work for um like once we close it out, then you, I don't know if you can rejoin it. um Do you want to like send out a zoom link for folks that are interested in joining in a half an hour to do that.

D

This google meet works for people.

D

A

um I I have a meeting then, but um uh google me would work for me if I were available.

D

I'll just paste the link to slack okay for google me in half an hour.

A

Cool any other topics for today.

G

F

G

My questions were answered in in slack already, but I'm looking at implementing the multiple encryption keys as a undergrad research project.

G

um So I think it sounds like it's gonna be pretty straightforward, um there's already uh a wrapped master key, and so it's just implementing multiple ways to wrap that in the interface for that and where to store that and all that stuff. um It doesn't sound like there's anything too surprising.

F

G

um But if anybody knows, if anything, I am all ears and I make no guarantees on producing a usable artifact by the end of the summer, but yeah anything anybody has to say, but I would love to hear um like I said I think my answers were my questions were already answered in slack. I think.

A

Cool uh well welcome to the community, and, uh if you um maybe, if you come up with like a design in terms of what the user interface would look like, um maybe that would be uh you share that with the team on um either on slack or uh like in a um issue on on github, um but also, if you want to like, make a little presentation about it here next month.

A

I think that that would be a good way to get feedback on the user interface. Okay, thank.

G

A

All right, then, uh we'll see you all in four weeks uh when we'll have the me at the later time uh that will be the 21st of june um and I'll see you all then bye.