Ceph Developer Monthly, 5 Oct 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2016-OCT-05 :: Ceph Developer Monthly

Description

Monthly developer meeting for the coordination of Ceph project development.

http://wiki.ceph.com/Planning

A

Alright, welcome back to the SEF developer monthly, everyone, I've added it to a few calendars now, so hopefully it will be a bit more pervasive and easy to remember for everyone I if you have not yet added blueprints. Even if it's just a single comment, please do so to the page we have a record of even after we discuss things. Otherwise, I think this should be relatively short, so sage, I guess we might as well start with your allocator information.

B

A

B

Yep, okay, so I don't have done a whole lot, except that there's the discussion we kicked off in the last couple days ago about doing alligator accounting. So we can tell how many of each type of object are being allocated for doing cash back pressure stuff, like that we had a discussion.

B

Alen went off and implemented it proposal and we just pushed a working version of the branch this morning or last night I, just rebase it and clean it up a bit and pushed a pull request, which is put it in that I didn't they eat their pad um and I've run the test and it works. So that's as far as I've gotten so far.

B

But that fact the the short version, basically is that um find the code in each each class that you're going to put into a memory pool do something like this: the etherpad you just call a macro and that basically declares the new and delete operators um so that, when you allocator tlk that object, it'll put it in the right memory pool and then somewhere in a dot CC files. You actually declare you implement those operators, because they're not done in line, and then nice find that one or so what it looks like.

B

Tomorrow's and the implementation, there is pretty straightforward. If you look at the macro number cool, does this and the defining nimble as this? This could influence them. I drop that assert, and here just for performance, because the hot path- and it calls the appropriate factory thing and then do you basically declare the mem pools that are you want to do. You want to exist somewhere else, um I care. What the macro macro looks like um here at the top.

B

Let's see, and the interesting thing is that it creates a namespace or looking at the header file. It creates a namespace for each memory pool and it declares all the container the common container types versions of that in that namespace that are automatically set up to allocate in that namespace. So I can change my standard map to be my name, space map or my mem pool man, and it will all the internal allocations. There will go in that in a non pool, so that makes it pretty easy to use it's based on his.

B

He took the code that he did for the slab, allocator container, stuff and rejiggered it, and to do this and there's a normally it'll just um it has like a minimal amount of accounting. For how many for the pool you can ask how many bytes have been allocated in that pool? That's most mostly, it I think, but there's also a debug mode that you can enable with the compiler, define whatever a macro definition that will also count how many of each type of object are allocated in that pool.

B

So our debug builds can have a whole bunch more education. We can figure out. You know the cash is two gigabytes exactly how much of that was consumed by every different type.

B

Which sounds pretty cool, so I think the next step is to actually convert a bunch of types over to use this and then do some profiling and make sure it does it have any overhead. That's going to kill us.

B

But this this I think satisfies all the everything that I that I wanted.

B

I was a little bit worried because I thought we were seeing a whole bunch of sheep fragmentation the other day and that probably we would want to not just this just passed a search of a regular alligator once it actually does the allocation it does the accounting and then allocates with whatever your alligators I was thinking that we wanted to set up, pools or slabs or whatever for all their front types so that they would they would pack more efficiently memory, but once we actually looked at the types that we're allocating and the beep sighs actually looked, pretty accurate, so I think actually we're not fragmenting that bad, but we might maybe in the future.

B

We want to change that I'm, not sure um anyway. That's it so far. Have any any comments. Questions concerns yes about this path. Yeah.

C

One can we use this to specify a maximum memory. We want for the 0 node cash rather than having user-specified your opponents. Yep.

B

That's that's, that's basically, a point: yep domen pool won't have a max, but the blue store or whatever will periodically go and look. How much memory is being used in this memory pool and they'll say, oh or close to or above our threshold I'm going to exert some memory pressure and start trimming some stuff. This is.

D

B

Memory management in the kernel works: it's how how it needs to work, otherwise you're very difficult to tune all this stuff up. That's that's, then. That's the big, the big takeaway here so we'll definitely get.

C

B

Store but I think it's also going to be really useful in even just for live instrumentation profiling purposes at the OSD. So in during recovery, for example, when there's lots of memory were like where's, this memory coming front will actually be able to look and see. It might be that we could dynamically. Do our PG log trimming based on how much memory we're consuming or I, don't know whatever um all these things become possible watching their.

C

Comp your come about fragmentation too. I think I think that's still good. I mean, especially with all these lists, structures that we have. We've got points going into, like everywhere, all over the place.

B

Yeah yeah. Well I say it I think it's going to be I mean in theory. If the alligator is doing a good job, then fragmentation will be penalized. I think if it turns out that it's not doing a job, then we can revisit that. um It was a problem a long long time ago in the MDS and we switched to boost pools and it got a lot better. um But that was like years and years ago remember how long I was so.

B

It might be that it doesn't um I'm guessing the day that it's better now than it was um and that that spreadsheet that you looked at was estimating, like 500 bytes per blob and ferg spent extent, and when I wouldn't added up the actual sizes of the start. It was like within five percent, so um I think we're in pretty good shape there. But Wow. Let me come back that if we need to.

B

Damn, okay, any other comments, questions concerns on this.

E

Yeah, it seems it seems like a good idea, there's the only other, the other other intersecting information I have is a bit but relates to similar types of techniques that might be necessary to interact with things an EBR or QSB our strategy in a lock, lock, free strategies and I. Don't think that some ISIL and then I think that's heading in the same direction: commercial direction, even though its complexity understand all they converge, but they deploy do yeah, yeah it mad yeah.

B

I think that the big question I think is is this: is the this wrapper around the alligator going to be have some sort of overhead and that's because it might? The statistics are our shared, so that's a there could be some contention there to mitigate that. It's just sprayed across a vector, like you know, tens of slots based on your thread, I. You know hoping that it's not going to ping pong between cores too much but like any notice.

E

The eggs, the other that's imported with yeah, but there's almost look at it for comparison, there's, there's, there's the mcat system has a generational alligator that has some similar properties and then, additionally, as support from because it's generational has supported his support for conducting algorithms.

E

That prove that you know they're all accesses to an object where in the generation that has it all accesses authors, it may have seen this hot object have proceeded to the new epoch and never weekend, GC the other ones and- and things like that, I'm but most respected son similar to me.

E

So it's a thing. We were experimenting with and so code bases, but but don't have anything that any work in progress to you tell you things with it right now: yeah I.

B

Think the only other thing that might be concerning is it's: it's it's built on the same slab allocator that he was using for the inline plop container types, and so it does allocate memory it's allocating in chunks and then.

B

Hoping to reduce or conventionally on the other guards office I'm, not certain that we want that in this case, like we kind of, we might just want the accounting and then push everything off to the lover alligator, but I don't know, unfortunately, but when I've been doing perf profiles on my machine, it's like none of the allocation stuff is really showing up on my plots, no I! Why that's the case um so I don't know that I can really measure the impact this in both the.

E

Size, aren't you unreasonably large they're, probably a good idea.

B

We have intuition yeah this just there's going to be some walking overhead related to the to the slab, location, stuff and I.

B

Don't want to imagine that the walking is going to be more sophisticated in the generic alligators with you know, Ric you type data structures and so on. That's interesting point yeah, all that stuff yep.

E

That's further, that's that's a fair point, but.

B

We'll see I have another box, that's lower, maybe the profile will be more than meaningful there. The.

E

Challenges of it well, that statement is true: they're not presenting right now. The internet, familiar with very many alligators, represent an interface for this type of usage. Yeah I was yeah.

B

Yeah data, fine, definitely, mmm actually that reminds me I think there was something in oh I'm, not lazy, set up separate, pull something off to go Google that cuz I thought there was actually something maybe is JD Malik had a whole concept. Remember.

B

B

um Yeah surprised when I looked at like boost, pool that there wasn't a like how big is it type query, just sort of annoying.

C

How they're waiting for you to edit page.

B

B

uh Okay, well, that's all! That's all I add it there. Any other question turned or anything.

C

With with things like extents in blue store is, would it be worth doing a bunch of upfront allocations of those since they're, probably gonna, grow to a certain extent, if any one.

B

May be platinum, um I.

B

Don't know maybe again I think the the generic alligators are going to be have much more sophisticated, locking and more better performance, tuned, Fast, Pass, hot pants or whatever, then even something that we put together. That's special casing it, but it's unclear, maybe the slab stuff, that this is implementing well, that will have to have to contract it market and see.

F

Nearly improve the performance of shuttle shuttle out did did her not long doubt it true, um because for long now the structure, I guess the multi-threaded like a briquette, you like what is America gmail and they may matter for this.

B

That's a good question, I'm not really sure I think most of the I mean if, if you ignore the slab piece, the idea is that we just are passing through the actual allocations to the underlying alligator. Oh it shouldn't change, but I noticed it also is doing this by locating them and slabs.

B

So I would expect it to be the same as the alligator, except that the slabs will sort of extend the lifetime of those allocations somewhat so that things that are allocated close together, we'll be stuck on the same slab and if UD allocate one but not the other, it's going to be pinned, but I, don't really know. No. No, it's probably pretty complicated um behavior.

B

So we'll see it's probably I think you can also. It would be nice actually if we could just disable the flat piece and make it a press ups as a one and have it go away, um but I'm I'll check in without about that might be you just feed both ways.

B

F

Back back to the original go I see Adam. The proposal for this PR is merely to search through which problem is.

B

I remind that my goal is to is just a trend accounting, that's the main thing. I'm worried about I want to be able to put like 10 different data types and all the containers that use them in a pool. So I can say how much total memory is allocated by all of these things and put memory pressure on the cache. Based on that?

B

Yes, that's what I want to do. um I just think that yellowness I think he's also trying to just reduce allocations with the slab thing at the same time um and I'm not certain whether that's gonna be a good idea or not. Maybe.

F

Yes, yes, if only a very small set structure, I think this way, maybe to compress to service promo yeah. Maybe we only do so very simple, a fixed size structure to throw the very small detour us most map problem.

B

F

Yes, I think this is baby well, I, I, hope late, maybe 22 harder to make it a perfect.

B

You're saying trying to solve both problems at once, you mean yeah.

F

I, maybe each dose the horsey yeah, okay.

B

Yep yep I'll talk to him. He couldn't make it. Unfortunately, um yes.

F

If you want to build a very customer member memory allocation, maybe it's a good away, but it's a hosting I see. But if you only wanted to solace, oh, where is ma see as easy to be yeah yeah make.

B

Sense: okay, um yeah. I agree, see ya.

C

Adduci calls to New showing up in the perfect profiles of these days. I've been doing it's, it's not huge huge, but it's not in substantial either said.

B

Is it like one percent five percent like.

C

Five percent for different person: okay,.

B

Yeah I've never seen a more than like one and a half sorta. Oh, oh I'll! Try it again.

B

Yep, okay, um that's all I had there I guess. The only other topic is on the there's. The boost stuff is basically ready to merge as soon as we get the tests going again and the slop containers also are ready to merge that are mostly they're, also using a variation of this slab alligator, but they put they lie to specify that some number of those slots are in line as part of the container. So when you have only a few elements, then you don't do any additional allocations.

B

I think once both of those merge, we need to go through and figure out where we want to use them, because they are like a bazillion opportunities where, if we use we'll just mend more memory on the stack say or somewhere doesn't matter that much and mobile hitting the alligator at all um the only.

B

So the slop containers have to have like list and map and all that stuff and then boost has small vector, and it also has flat map. Although I think flat map doesn't have an inline, so it's still an allocation, but it puts all the keys and like one allocation.

B

Yeah, so I think just be aware that that's that's coming, I did it. I do have a pull request that tried changing buffer list, the list of pointers in buffer list to be a slab list with three per allocated slots, and it makes my it makes the unit tests for buffaloes run like three percent faster. But when I tried like a I did a pork, a random right work, club I'm it didn't it wasn't noticeable um I couldn't see any difference. Yeah.

F

If it's only a little, you promote, isolate, not was torture, you know, maybe yes, it is very simple in the test. Yeah.

B

um Yeah, it's I mean I, thought I. Think that the problem with all these is that it's um there aren't there isn't going to be any single container. We're switching one container over to use the inline types is going to make a significant difference, but if we changed all of them, then it would probably significant, but that's a lot of work. So it's hard to tell up front whether it's going to be worthwhile to change all of them or not.

B

Yeah we might need to go through and like well carefully, look at a profile that find like the top 20 or something that come up and changed all of those and then see if we can see an impactor again, I think we need good high quality, Bravo rusted guide.

E

That yeah I sauce was working about was about flat map and it was very workload specific, but it has very different Phil properties. And if your workload is right, it was beating really really beating Matt other maps. But but if you.

D

But if you feel.

E

It wrong that our search at your and.

B

E

Perverse yeah yeah exactly.

B

Yep, okay, apply F for.

A

A couple of late additions: here we have nitin for the adding I, skazhi and query interface to our BD devices and josh, with a qos update. So I guess we'll start with the guzzi inquiry interface to our BD and are you on and miked?

A

Yes? Can you hear me? Yes,.

G

Yeah, just a hi: this is the first time I'm attending the safe monthly call. So I'm not sure what is the overall goal of this meeting. But I thought I can add this because we would like to see this in the rbd interface, and so, if there is any update or anything, we can discuss. I means most of the stuff is captured in the mailing list and also on the tracker which is listed on the wiki page.

G

So I don't know if there's any update, because I haven't heard anything or is any plan to include it in a particularly zar NE.

G

D

Yeah I mean that's. Basically you can treat that as it's like. A backlog item abut nice to have, and we have no active like plan to incorporate it now, but contributions are are, are welcome, but yet right now, there's no one actually working on it. I.

G

See so this would go in the kernel soup well,.

D

So it's half of it all the preliminary stuff needs to go into like lib are because it would be responsible for when an image is created or a snapshot is created, that it would generate all those unique IDs that would be stored with the image permanently. But yeah then, after that, back support is including all the changes that are necessary.

D

Even on the OSD side, with like the rbd class methods, then the colonel would get updated to be able to then query the OS DS for that data and then, if it exists and be able to then map it down, you know at the K every level and then someone else that would have to plummet in you know at the scuzzy level or whatever.

D

However, else it needs to get plumbed in.

G

Ok, so thanks for the bit I think yeah we'll see how to take it from here for there. But if there's any update means that's what I was looking for and seems like. There is not much update right now and there's no plan, so it's just so nice to have features of her.

D

Yes, so far, we're treating answer. Is it nice to have, as opposed to like a capability blocker or something like that? That.

D

Is blocking other efforts.

G

Ok, thanks for the update.

G

So yeah I'm done from my side, then oh.

A

Ok, I take myself off me that works better Josh. You want to give the qos update, yeah, sure and.

H

So, basically, free and us we've been working on this for a while, now and trying to add in port for and using the DM block algorithm to be able to eventually have quality of service that, like reservations but among different clients and but start off with we're. Just focusing on getting the algorithm, correct and erika von said she's on. The call now has been implementing that and working with some other folks from south korea, telecom and CTE to fixing bugs and get that working correctly and now it seems to be passing the rate of sweets.

G

We lost your voice between.

H

Shoot my mic back or is it gone again? Yes, I think yeah you're good now so I'm headset comes at night, sometimes so the initial version that experiment is just trying to distinguish between background I, oh and the cluster, like recovery scrub bands, not trimming vs. client, I Oh, since that cadets been a problem for users in the past, so that branch that eric has been I undergoing a bunch of their performance testing lately and I think we should have some results to share soon Eric through. When I add and thing to that.

I

Not really so I've been running it in various ways and generating some graphs so I'll share those relatively soon now.

H

F

H

Anyone has any questions we happy to talk more about that. What we're doing he.

G

Is this knitting again I? Would we would like to have quality of service like guarantees if we can provide like if you're, especially in the siphon rom and if there are different services, are using the same cluster like a rbd, I use case versus ffs and radars and all that easier wins?

G

This is what we would like to see if we can provide a quality of service to us, a particular service, so that if something goes bad, that service gets least impacted, and we have like priority up services- is that what are you trying to do? I, don't know much about it, and I would like to get more information or, if there's a page or any design document, or something we like to know more about this yeah.

H

I think there's some past pages from pass against that and with some design documents. But the idea is that and eventually you'll be able to assign different kinds of reservations to different client kinds of clients, though you might have you could assign all I call your buddy clients the same.

H

Perhaps them internal clients, reservation and all your Vitas clients, some other one and all yours have those clients different different reservation, that they are guaranteed a certain level service and it's some kind of some share proportional share based on what's available after that point, but it's it's still a fair ways away from and being implemented. So far. So, but that's on that's that. That's that plan now pick up there.

G

Instead of target release.

H

Just curious, and not precisely, hopefully, the the limit more limited form with the distinction between background Rio and client AO will get into luminous and I. Think the client side and stations are further further off, since the requires a bit more work on the configuration side and you kind of management of the different limits.

G

Okay, so yeah means I would like to get the detailed save may be offline. You can send me some links or whatever to understand we're safe word going on yeah.

H

I can find the link to previous too bad in that person put it on a wiki page there, okay.

G

A

Think it was probably the shortest set developer, update on record so awesome 30 minutes and give you some time back to your day, so alright. Well, thank you. Everybody for coming. The next planning session will be on November. Second, that will be our a pack friendly time, so 9 p.m. eastern, please I'm going to be pushing a lot harder to get some blueprint. Discussions on the wiki beforehand, I want to make sure we're making good use of everyone's time for a meeting like this. You know if you have thoughts one way or the other.

A

If it's not useful, we can discontinue it and figure out a better way. If we are going to do it, I want to make sure we do it correctly. So, if you have any questions hit me on the list or directly and I'm happy to answer questions, otherwise we'll see you on November second or on IRC thanks everybody.