OpenZFS 2017 OpenZFS Developer Summit, 25 Oct 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RAIDZ Expansion by Matt Ahrens

Description

Design update to address the issue raised around 18:50: https://docs.google.com/presentation/d/1KwLhoWly-wSpfrdyr16EyWr8KCybyB_n0_Eq_DKWcWA/edit#slide=id.g291811f78a_0_501

A

A

This one I mean it went all the way so we're in succession on this. You have crazy cool before just in it, and you want to add.

A

B

A

Lots of people also reflected in United it up all the punks like.

A

A

Like overwhelming so that's a great problem.

A

Well so first thing I figured it out. I thought about like how does great in traditional rating solve this problem and I live in a box, and this is my expectations. So what we're seeing here is on each column prisms on that this each row is, you know one parity group and also anything to parity groups by color, which.

A

So that is, um you dismiss block one if we're going to refloat the data process. So therefore gets moved from the second rope up to the end of the first road, and you could like be positive parity for that new road with one pp for and it's like, five and six gets moved for the beginning of that and 70 issues. But there is kind of like you. Can you text editor and you like change the number of columns, new text, editor and interview, folks, all the text across that?

A

Okay. So how good is the great seat with Grady? You don't know what it is depending on you. So what we're going to the crazy is reflow all the data by essentially like yeah, we're gonna reflow, all the data, ignoring where the parity is, because it doesn't actually matter so like. In this example, we take burnish walks to parity, but as webinar, so we shoot the new discs and six seven eight and shifted over nine ten get over there.

A

So this is almost like rewriting all the dating you go, but at the end of the day, you don't need to go where the parity is, you don't know, you know where the block boundaries are.

A

We do all this using the decisions based on information, so this was how I did in what Street, so we actually have to move it as allocated. So this example like 11 through 16, are so we don't have to copy them, but we're still that logically, every numbering all the space on all the guys, because this is country are there any questions.

A

A

Pulls it in wait for them, but if you didn't get what I said, then questions about this okay, great!

A

This is just what I said grace if he think that makes this possible is that each logical spring, whichever you think of the colors, is independent so like if I move this person here with five six is like one. A lot of a lot is appearances in the data.

A

We can move that up here and then Mike doesn't matter what's surrounding it, because imparity is part of the very the data all comes together in crazy and oh and redundancy is slow, preserves, so I kind of like reshuffle all the stuff, but the key thing is that every every logical straight, each of its law business sectors ends up on a different disc. They started a different.

A

That's like five six here, they're a fun different stewardess, but you never end up. You never end up where like I took. This is this box. Seven, eight nine ten is one line of the block and it's up on four different discs. If you get right up.

A

Because if anyone else is only taking out one sector from each logical straight, so you wanted to do that contraction. There's a lot of contractually paid work this way, because they didn't even have to put two sectors on the same.

A

What about the newly written data so in this example same example, but now I've been I'm going to consume that free space by writing this green block. Well, now we know that the physical strength is five. This applies so I can write using that new strength. So here, like I, have four bucks.

A

Four more seconds of data, one sector parity I mean spread across all by business here, which is you know different than these other bases, so the new after, if you can get here, is physical strength of their agency and the logical strength with which is now specific to each block. So you know here everything is visible start with four and logical straight report, but here we're physical spec with five. We have five physical blocks here.

A

Five physical disks here, but the old blocks, like the red one, are a lodging strength with before, but the new blocks that do green one are about this right foot of five and we need to know that way needed. We are read to block that noon.

A

So consequences. Are you know the after you do this transition? You refill all the data the old blocks still have. The hold is very Bishop, so this case is it's three to one and but the newly big boxes have the new pair. The new did earlier issue a green one, which is portable. So what book should know like where the perspective they're given, which are very, we can do that using. We couldn't do that.

A

Why she need the block, would you but is actually just going to me if I'm ready to death, so instead, what we did was we you we're actually to do something that we up for a long time in geometry that is base under the first time. The physical is something profound alarm.

A

A

Cool so I showed this cool picture. Is this one where like before and after it's great? But how do we get from view force after so you know when you spread out, I have an empty tips. Okay, so you sort out, looks like yeah the disc there's five minutes now, but all it is so where it was.

A

You know we have to actually do and the problem is you can't overwrite any data, that's being used so like I, can't just like read this memory and then dump it all out up here, because if I comes from the middle, that will be impossible because I was you all all 20 of these rights in the same time like this and then like.

A

A

A

A

A

But now sizes too, because I have two unused seconds next, another one that has the old version of that it will care about, and then this empty one.

A

A

So yeah, so we need to track the progress so that, like, if I.

A

Crash then I need to know if something I want to read the block that starts at 1720. I need to go. I need to know that I should get from here and not from like the new look around here.

A

Okay, so why don't we go? Do some tricks with the zoom, because I should be looking for spot? You know first 7jc geez, but this actually exponentially you talking the start of that at the end, because if I have time to make a beer right here, you can see that, like eventually each chunk size will become larger than this very quick and then like every time. I add a whole bunch. More juice is exponentially.

A

So you talking about like you, go turbo being Ruffo, you know it's gonna be 6 G's, but like reading reading, carbon is gonna, take a while as well so in practice. I'm pretty sure this is not gonna be like.

B

You'll see that it starts off slow, but.

A

You know after you've got to be like 100, 6, G's and you're definitely to be Gothel by the discs rather than juice.

A

Ok, so I showed you one example, but this appendage generalizes kind of absolutely funny the work to work with ready, 1, 2, & 3. You expand multiple times so new you have for wide and had one big, 5 vine add one b6y.

A

Like when you made those transitions, so you know what the actual day out is as a mission. The old day that has the bold is issue and India has been, do get it very ratio, the one other trick. One other requirement here is that everything has to be healthy when this is in progress. So if we.

A

But how can I mean all the dishes have to be there? If one of the sales, then it's okay, we can do this over. We still crippled know about like will do about the progress, and this is.

A

So we can do the reconstruction, but but we can't continue reflow until then you do the reconstruction, because if we continue then definitely case where we need like we need to leave something from the discussed on and you also need plate any given strength need to read something from the disinfect stone and need to write something else, something different to straight. That's gone so in one strike like Indies for like we may get many really score that we're good. We may end up losing to underperform because, like I need.

A

Like if we use what this one, what is this one and it's like money, then I can't be 15 and I can't break 14, Anna Kendrick 16 so actually lose well. I guess 60 was already already there, but if I lose 16 and I do supports me because I hit right. Of course you do that this, but I just work. You know the roof level will just pause. The device is not healthy and then, after complete standard flow versus the background, you know works with you.

A

So we don't need I thought this, so this is sponsored by the BBC foundation systems in other donors, so I wanted to jump in make them for maybe priority and forgive collaborating between Community Foundation and semi systems.

A

So this this is the sneak preview, but I do love designs, 50. Also, the toxicology is like that's solid detail and we're gonna be done. Design is a bunch of like beauty greedy like blocking. You tell us, because you know, while you, while you're in the middle of this people, leave me like reading and writing and all that stuff.

A

So you know we need to have walking to ensure that, like moving the roof of progress moving forward, doesn't comment like why, during the middle finger bead from something that's whose state is changing as a result of a group photo progress, so everybody relies code, but this is just last month so, based on this, you see, and hopefully we'll have.

A

A

You don't have to do this in context. Yes, this connection from open context.

A

A

Time so it doesn't matter but like whiskey, it possible then, like the amount that you're allowed to copy in chief she's. Like you know, you do magic, kerbecs, so working on reading, writing all the data and seeing honest doesn't have to care about. Basically, we have this huge region, huge here's, this three sectors over amethyst huge, like white unused.

A

You know News region and we didn't do whatever we wanted that so open context is kids got like tons of hours to read all the data and bring it doesn't really matter what we're against in there and then like once. That's done whatever its use to you happens to be there. We can say right. Just do a seat has to change the roof of progress from like a million to 1 billion, to like one way too late.

A

And what's used that way, you can also to staging summer. So it's like okay, they said doing I, don't have to kick off like the bold you're, a bit of IOU, not good I'll. Just you got the beginning of like the first. You know, gigabyte when I can finish the intro size and that is happening. Asynchronously, like once you get going nothing's actually waiting for anything. Even many perceive Jesus. If you continue on.

A

A

B

Like the Cynthia.

A

Rate was 7 to 10 on June, 7 and 10 are on the same just yeah.

A

So probably I don't have to think some work up. The design in Axum are special cases, thankfully, that of it. I should have mentioned I guess one. The reason I wanted to do this talk was to get feedback from folks on both my mic. Does this do something useful, like there's other additional things?

A

Thank you very much.

A

C

B

A

The question was like: couldn't we just leave it like this and then, as we read it as we write stuff, then there's some free space for it. This would increase space here, basically loved the routine, our creeks. Why don't we just make this thing, but my buddy's leave stuff as it is originally and then write stuff to the nude poles face.

A

The problem is, you might be able to name some very, very special cases, a lot more ease, logic, the but like in the general case, like everything can be allocated right then you have a situation like this looks like every sector is how I did it and then you add a bunch of sectors here.

A

But you in the number way to do a few things like we need nobody. The sector is one two, three four five, six, seven, eight nine ten so like now, I have something to this block is five six, seven, eight nine ten, but now I have like three sector to build it like you did like good. If anything, Eric's we keep track of that. I can't use it right because all this is on the same disc.

A

So if I raced up there, this is no redundancy and I wanted to come up with something that would work. I mean eight like if you're, adding space you're going this just probably because you're using all the space that you have already so.

A

Work is probably.

B

A

Particularly wouldn't be clear doing this any longer to reason about, like the available space, so yeah.

A

So you know that padding is that padding is well, so the timing is due to round up so keep us 1. So in Gracie you have to round up to your allocation has to be rounded up to your mole slope. The number 13 is fun so like in this crazy one example. You we'll go to the existing blocks.

A

They're still allocated the same size, so that would be changed when your reflow, you can't change that new blocks will be allocated like new blocks also have to conform to this Department of multiple of P plus one, but like the logical strength, is different so like which size is required. Hiding will change that thing baby. So did you death? Well, I'll, see you. Somebody asked a question which leads me to ask words to that.

D

A

Thank you for asking baby. Can you convert crazy one, twenty two or three like adding more parity? So that's not part of the scope of this project, but he is a very natural extension of this, however, is to count yes, one minor one, which is the out of a multiple of P plus one. Now it has to go out the window and find some other way of like trading for this space. I think this is totally doable. We just have to do some measures. It's not actually nurse like make.

B

A

Amount of space that we're talking about isn't so much that actually, but the other big caveat is just like this: when we reflow we're not changing any of these listing blocks like you can see, plexus still have the same good. It's still have everything the same. That would be the same if you switch from raising one crazy to where the existing data would still only have one parroting sector and that's a lot harder to swallow when you're adding we don't see, you've seen great I'm. Gonna.

A

Add it this so now, if the two digits fails: I'm, okay, but it really not okay, because my whole data jalisa right Mundus going so we need some mechanism to essentially they could be read multiple data which I think would be possible, but it can't do that through this public view of things actually go through the D mu and like reread all dirty all the and but how they work with snapshots, etc, etc, which I think they do. You read me possible, and maybe the subject of like future extensions on this. Every teacher.

A

Who reads me: I think that that would be theoretically possible because you could essentially treat the mirror as like: hey, let's just pretend yeah yeah, you look like you probably could, but it's gonna come with us. This.

C

A

Not gonna like.

D

A

The question was how about a most Lotus at once, rather than like adding one roof of another one yeah that's so that is also very naturally in the design of his project. I think it would totally work we just to keep the scope of the project as small as possible.

A

It's not going to be part of the like minimum viable project, but I think that would be really straightforward.

E

A

So the question is, could you use this for changing the rector size or compression.

A

No, this doesn't. This does not in any way address any of those things, because this is work. This is the reason this works. The member so well is that we.

B

Don't know when the.

D

A

Are we don't care where the blocks are and we're not changing to what winters so the way? This is like we're cheating the way that the sectors are numbered. So when you go to say like read block they read from offset 11/7 through ten in a little late and then you know be 78 esport in a new way. It means reduce for which, how you have the same data after I looked back for a way to be routed, but the size of that block is still poor. Great and I.

A

Didn't change the compression and into the block order of Engineers, for the record size like all that stuff is, would require much much work and intensive surgery.

A

Yeah sure, if somebody wants to do that, that's great is one liter bottles of this actually.

A

Yes, the the we was this problem where that seven in ten are both on the same. This can occur even when the chunk size is much much larger, because you don't know what the alignment is, and you don't know where the blocks are they you know like I,.

A

Didn't start here, but we don't actually know where they are when we're doing the moving stuff.

A

The that's true, but.

A

Once you get further along like so, for example, here even here, so once you get further along, then we do actually have all the copies that we need. We just have it like I think about her brilliant so like here. If we lose this, this weed is 14 to 16. We would actually have another copy of fortunate over here and that's going to be like more true as.

A

Well, you need to take that you're right that it could still been the same in the same login. We would need to take that.

A

Yeah so like they got that essentially the trunk sighs we'll have to do and we'll have to do subtraction from the trunk size that I've shown here we'll have to subtract like one with you know, Hubbard media star to make sure that we don't overwrite something that we're actually going to need back here, whereas I hope you might need 14. It takes to do this. We need to make sure that we go over in 14 until we're sure that presentation.

C

C

A

The Commissioner was like on a whim, we're in the middle of the reflow. Are you out getting with the older the.

C

A

It's logical and.

A

So, in order to avoid complications and make the time-dependent rework I'm just going to keep out of eating with the old logical, separate while the repose of the markets, otherwise we could do. We could stay like oh you're, absolutely reload and use the new levels right with, but we need to use the space in the black winter. Tell that much more complicated people that puzzle site based on not just the roofline but also the runtime and the offset was funny whether it's in what might be an actual love affair. With this.

C

A

Yeah, so um this is a great CG place that must be my cup, so we keep track of like the expected ratio. They expected it Ernie, Bishop, essentially and like in ZFS.

A

It factor that in so it tells you like what you reading like you, have a terrific file, it's using it care of it, even though it's actually we've allocated more space in that expected ratio is gonna change when we change the straight with that's I, didn't say, because I'm still working on that, we might have to live with stuff using less space in your sex. According to save us this, but I need to see if I can do better than kind of have that else to be like I'm, tiny.

E

A

A

So the question will say, as as we progress the shoe size grows exponentially like which leads come in there's like multiple bases that we could read it from. Like other complex.

A

A

As a Turkish, I suppose we're gonna issue, I think where we need to know like how much Vargas has to be, and we wouldn't know how to progress has been made like in memory as what's been, actually sinks to those gates. Gene, yes, so, yes, there are definitely potential conflict in those need to use off with proper blocking. That is, I thought it.

B

A

The last part of the design that I need to finalize, but also consider this while we're in the middle movie and.

A

That's it sorry for already go great, not ask your questions during the talk, but hopefully the next.