Ceph Performance Weekly, 17 Jan 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph Performance meeting 2019-01-17

Description

https://pad.ceph.com/p/performance_weekly

A

Good morning, Casey yo, how are you mark I'm doing pretty good ooh.

A

B

Fighting the good fighter against the rgw multi-site bugs I.

A

Actually had someone mention that last night, I was at this like Minneapolis F meetup, and they had mentioned that there was some issue with distributed: replication and OMAP data not being properly replicated or something in like silently failing.

B

Uh-Huh that doesn't mean about okay.

A

Yeah I think it was like rgw multi-site type, F, I guess, but anyway, that was that was all I had really heard about. As far as technical details go.

B

Yeah I don't think that there's anything in home app that we really replicate between between clusters, yeah I.

A

Not not an area I'm very familiar with so no idea all.

B

Right well, I'll keep an eye out for bug. Reports.

A

All right, hopefully we get sage. The core meeting was going over a little bit.

A

Give him another minute or two.

A

Hey Casey in the meantime, I was beast going.

B

By I mean it seems to be stable, it's uh let's see it was marked, not experimental and mimic, so we haven't heard a lot of feedback. Hopefully that means no bugs there's a lot of follow-up work to actually take advantage of the asynchrony and rgw that work is progressing but kind of slowly but I'm, hoping that we can get kind of the important pieces and for Nautilus is.

A

Is that gonna be like made default or is it made default now? I actually don't know.

B

No, it's not I. Guess I haven't thought about that part. Yeah.

A

I was just curious if, like there was a plan for for doing that or not.

B

Eventually, yeah once we started seeing a lot of wins from from boost Ezio and being able to cut down the number of threads, we definitely want to phase outs of it. Web yeah, yeah.

A

Well, I was thinking I should I should probably try to start moving over to that for any I. Don't do rgw testing that often but I've been doing some for the memory auto-tuning work as just kind of like another another aspect of it, since that great sharing tends to create much heavier OMAP workloads, though maybe I should switch over to beast for that now.

C

A

Cool it's it is it just like as easy as swapping out the civic web, like our GW back-end or whatever it is yeah.

B

The rgw front-ends config yeah the web okay, neat: there's tax bore I like the extra options and stuff, but basically exactly the same.

A

All right good deal.

A

Let's see if I just ping say: John IRC exercise he's responding to stuff.

A

All right, well, I, haven't heard from sage, so I'll just get started. Let's see.

A

All right, not too much new stuff for pull requests this week eager had modification for the auto tuning stuff, I, think he's just specific to on startup, making it a little bit more aggressive about tuning immediately. I need to look at that. It also changes the frequency of of tuning, which may be a good idea, but it may have effects as well. That aren't good.

D

A

At the very least, we need to test that because that will affect more things than just our uptime.

A

Let's see, there were two that closed this one from Keith who I think or optimizing stack, string stream and then also a one here for from sage for the the Numa affinity and pinning stuff Haeju is just starting to go through the requests.

A

Do you want to talk about your new, my stuff at all, your silent.

D

It merged um it's a new command that lets you easily show the Numa node for the storage and the network for each OSD, and you can easily pin it, but the stuff can fix it. Though.

D

Let's go see it. It also tries to pin itself if it happens, to notice that the network and the storage on the same note that I'm not sure how many real machines that actually applies to you. So who knows what that's really gonna be useful enough, we'll find out as people to play now to uh say: ok,.

A

This libero reading multiple cache right back threads actually looks kind of interesting he's getting numbers really close to having cache disabled.

D

D

D

That's like Jason's re-reviewing.

D

Okay, I, don't yeah I, don't know if any this know this other stuff that looks worth commenting on right. You know.

D

With the Elise is just to clean up kryptos change: let's go there. Yeah.

A

It was, it was silent for a long time and then it looks like Peter. Maybe he pushed something I didn't look at what it was looks like it's. Those I weigh first push force.

D

A

I didn't I didn't pay closer attention to it, but guys I can't look at what is less I love a copy of it from before. Yep.

D

All right, I, don't know. That's early, all I have I'm I'm, mostly focusing on the messenger, do stuff for novice.

A

Cool all right, the only the only thing I was thinking for discussion topics that I was gonna bring up. Was that I heard a rumor that the sea star OST is booting.

A

E

Yep, it's leaving.

A

Cool does it do anything.

F

It was a de that's all, and the kid did it could be therefore visible. Bye, bye, monitor.

D

D

A

So what is that now, in terms of like oh go ahead? Sage is.

D

That what that poor across was that you were, you pointed to.

D

Right I mean I can just click review. If you want just merge it, I, don't I, don't actually read you anyway, yeah.

E

F

E

Very, very small minor think we can address later after the things like consuming consuming messengers and embedding a great aggregating them inside tight OSD.

F

I could do them tomorrow because my goal is to carry the merchants but another week: okay,.

D

A

You in in Radek, so what's what's next, like how close are we to having like a mmm store thing in there, that we can test.

F

Nothing we are being able to pretend random pitch for Ellen will require certain from readers crank and the next thing to do to get get, get the repairs, correct and reply with some some some payload, that the current is asking for me.

F

Put it another way to connect to I customize the version of a memory store. We still read and read and request sure.

A

That's exciting.

A

All right, I, don't have anything else. Anyone else fire chime in or bring up something I.

E

C

E

E

Well, I guess that from s P D K, we will be. We will want to to grab the low level and VM e driver, maybe not the nvme over fabric, which looks like networking V me, which task of natural I mean networking a storage.

G

So Radek my suggestion would be to to actually get the B dev driver. If you can take a look at the B day of layers right, if still can use it yeah using the NDB, you know in India me dry, wood directly. You can use the beater that'll actually give us a lot of flexibility. I see.

E

G

Supporting our BG insight corrector so underneath be there you can, you can actually help I mean not have it inside, but but.

E

G

Can you can have nvme ad in your fabrics? You can? You can hide under you know under there like an L wall kind of layer, I cannot logically.

E

Yeah is be deaf dependent on the SPD K, even framework.

E

G

Okay, it is independent, it is just an abstraction layer that was created. You know just a level above. Let me give.

H

G

Yeah so I'm actually trying to redo the the blue store back in, but you know that be half or India and I chatted with Kapoor just a little bit about that, but I'm trying to redo. That would be there right now. So I can actually, you know, send you that that edge or ideas, the reviewer, just just what you know what I'm talking about.

G

But but it's essentially it's a it's a you know it is it blocked abstraction which, which basically hides like implementation of whether it's an off the device it's over there, it's a network device or a local device or a or a logical volume kind of implementation. Actually, we are actually even building of, like you know. Caching, like with cash kind of a block device as well so yeah I mean a whole bunch of things are can be, can be abstracted out would be do with minimal overhead, but.

F

Spin would speak of a bidder, I think the progress is more attracting because it we can. Even you can even even support the facility we qualify on those TV right. Yes,.

G

If you can do that yeah, we.

F

Can we can even field built to store over were loves, BBB and which we will build on top of our progress.

E

D

D

G

Yeah, it's just a non perfect. You know file file system, it's just a block alligator, it's a pretty bad boys thing, I. Think I think we had talked about it like initially to house.

D

G

Course something yeah yeah, the right one booster was coming, yeah Louis was coming about here and and by the way stage. Actually, the you know, I told you last year that Cass was going to open source I, know really hard this year to convince the management it.

E

G

Fact, open-source now we, the actual quince, was just last month and I mean again. This was kind of fun on the side, but just because I portrayed the world Cass I just wanted to put it out there that it's, it's actually open source and we can use it like if you need to in some shape or form. Oh cool yeah, okay,.

G

We are other ways actually I like if you're done talking with Peter stuff I I just want to you know, you know, I recognize that so so say, I think your patches or for the new my stuff will will be extremely helpful.

G

You know we throw the you know like beneath. We have been like putting together these reference architectures for with with our field guides and so on. You know, the Numa affinity was always kind of you know different hand-wired yeah. It was.

D

G

To kind of the week we so Orlando is actually on the call right now I mean he. He knows what what pain it is to. Actually, you know, use new, more control, and you know, please always be strategically so this was really useful and so so we'll we actually have some nodes where we can do.

E

G

Of you know maybe at least true and mamie's pull any one node and a balanced, we'll see if we can give this a try and bring bring some data back for the fish yeah. Okay,.

D

I had a question um my my intuition was that, if um the most the way that you would ideally want to structure, your hardware is that if you have to nuuma nodes, you would have like two in VMI's on each Numa node and then one Nick on each node, and you would ideally have the OSD on Numan of zero use. The Nick attached to new node zero and the ones on one user Nick on one. Is that actually the case is that how it works so far like all my boxes have like the Indy.

D

These are a new node one, and the network card is on new. My node zero, just like new.

G

You know that's how it should work, but but actually all the Dell- and you know, HP boxes that specifically there they're not built for you, know these high-density storage kind of application, so they always so to save and also to save cost. You know they always end up putting a by 16 expander and get four enemies that way on on one one side.

G

So so they just don't bring out it in a PCIe out of the out of the process or even if it's available like, for example, the skylake processors have 48 PCI lanes each they. You have 96 lanes, but if you look at CP 0, typically, nobody brings out more than and maybe half the lanes. So so the only only box, that's out there actually I worked. I was working with quanta to fix this problem, and I did a toggle also at a set day on this.

G

So they they quanta is the only vendor that has a system which has perfectly balanced, 12 nvme system in one you six enemies on each node and they have a by 16 slot on each node as works, where you can place up to 100 Gig Nick, so some can get some of those systems, but but we do have some internal, like you know like, like initial, you know, validation, kind of systems which are which have the ability so that we can cable stuff around and you know get get the perfectly balanced, so so in production, it's kind of hard to find these systems yeah.

G

So so for now we'll we'll see. If again, we can help you know, but in the past. Actually there are two data points. I can actually share. So in the past like when we had done some testing with pre, luminous, I, remember and Orlando can keep me honest here, but but I think the the benefit to going to fill. The new balance was about. You know 10 to 15 percent, but then we did some recent analysis where we saw going from singles pocket to dual socket. I mean again, you know called of newer balance.

G

We were able to extract only 25%, more I/o, so so we can do two more like you know, once we once we see how this how your patch does, but but we don't you're, not able to scale the two sockets. Yet it's only 25% more okay.

D

I guess that's sort of a question they had following question: I had was assuming.

E

D

E

D

All the nicks on node zero and some of the empty knees are in node one. Is it better to pin the OST closer to the storage or closer to the network, or neither and just let the kernel sort it out, yeah I know Linda can.

G

C

Sorry guys I was needed. What was the question again if.

D

You, if you have an unbalanced system where the network is on one node and India knees, are on a different node. Is it better to pin the OSD close to the network or closer to the storage, or neither, and just let that yeah.

C

That's a that's a good from actually I, don't know which one is better. Usually we pin it to the nvme device, so whatever that's located on but yeah, and then recently we just let the the scheduler figure that out okay.

D

Totally a lot easier to play with that with the new code, because you can.

G

You should be able to experiment a bit more in this in this area for sure, because maybe we'd be almost like the only kind of the two sides you know let the scheduler figured I would or can a completely partition the two, and we literally had two subnets essentially on to two.

E

G

E

G

Those, but we haven't looked at the hybrid really where, where should the OS please be? We may have just been running OS T's I mean on Windows, been using all course, so then, to the payment to one side, to be honest with you so yeah, it's.

A

D

A

I suspect that just pinning it somewhere is the the real benefit rather than having the kernel like try to move things around right. That's really what you pay for is. If you're there.

E

A

You know moving it back and forth, because it's thinking that it's you know to do something clever, that's not really clever yeah, so.

D

That the other, the other part, that's in that pull request- is that if.

D

If, if there are multiple Numa nodes and the storage is on one particular Numa node and you configure the OSD such that, you specify the public network.

D

Well then, when it searches the local machine for which IP to bind to for the network, it'll try to choose an IP, that's on a NIC! That's on the same note as the storage and the my expectation was that then, when you have a Nick on both Numa nodes, you could configure them with different IPS but on the same network and then the Oh Steve's would bind to the one whichever one was on the right, humanoid and so they'd end up on the right on the right. Nick I, don't know how!

D

Again, since these systems, there aren't really any mail systems out there, many of them just a Kelowna one and I guess that doesn't really gonna help anyone I'm yet but I'm, hoping that eventually, it will and I think that what I'm I have a feeling. What we should do is basically sort of get rid of all the old guidance around creating a frontside Network in the backside, Network roll that out and instead suggest that people have all their nicks on the same network and hopefully, eventually then Beth will be as smart as possible.

D

That sort of auto balancing across the well.

C

Is that is that common, though, to see multiple IPS on the same node, I I.

D

Don't know probably not what people do is they would they would bond them I think so. The old guidance was have one NIC on the front side network, one NIC on the back side network and then sort of the newer guidance is more. Although that documentation doesn't say this, but the newer guidance is really to bond them, and so you have some redundancy.

D

If one NIC fails, then it'll use the other one and use whatever some bonding protocol to Nick, but I have a feeling that for high performance situations the bonding isn't really going to help, because it's basically, especially if the nicks are on Turpin, then you're like I'm, assuming that it's not going to be smart, it's just going to slow it down. Instead of speeding it up and so I'm.

A

Wondering if yeah a lot of the the problems that we've run into with partners in their labs, when we try to do testing is due to bonding. They have been like messed up, bonding setup that breaks and irritating ways yeah actually.

G

Challenged with the network config, usually the IP address belongs to so usually it doesn't allow same subnet to type it to mix on the same. Subnet may not work because their routing, you know the way the current without being set up. You know you will like all your packets will be forwarded to essentially the first. You know the interface that appears in the first routing entry. So that's.

D

Maybe an issue.

G

D

G

D

You, if you it's a socket, is if you bind the socket, then you can control which network it goes out and in fact there's something in stuff. Where even on outgoing client connections, you can bind it to a particular key to force it to go out and the particular network I assume.

G

G

Is for send or receive is where I was I mean we have to do some IP routing, IP policy, routing magic. You have to do that. It is possible to do, but, but you only receive it, it will be automatically there'll, be some additional, but it's possible to do work around that for sure yeah.

A

H

So I was just commenting on the issue with multiple NIC ports on the same subnet. You know with different addresses. If you do that, you've got to be careful about art. Because what can happen is it can start sending traffic to the other one?

H

It's something we before there's a way to fix it, which is to set our filter to one, though that it will actually go to the interfaces that you wanted to go to I'll try to send a link about this, but it's a I think it's a fairly well there's anybody else heard of this I mean I've. It's been around here, hear yours, yeah,.

G

Yeah there are out there and the choices are you know, so those two settings need to need to play with and and at four l3 actually I was describing the more of the challenge with layer three, because when the packet is received for the subnet, you know it's. The route lookup is only going to happen on the subnet address right.

G

It's not the specific IP address, so it may not be delivered to the to the right interface, because the kernel routing entry tells it to go to let's say zero, but the IP address actually belongs to each one right. So there may be some something on there, but but with IP policy routing you can actually set some rules and I can I can actually try some examples and sends in the cross. But if you can share that link that you're talking about that would be great as well. Yeah.

A

The idea of having a separate network card on each Numa node is that it's so much cheaper to buy a single NIC that has like two two ports on it than it is to buy two separate Nick's at one port.

A

But if, if you're talking about like dual two port, 10 gig or 25 gig, you know, depending on how much you need, maybe we could still can't justify it. You.

G

Know the cost of the NIC in these dense nvme systems. It's going to be, you know so small, like as a percentage of total, because most of the cost is actually likely.

D

G

Kind of what you've seen and you know so so like we just didn't being exercised with OVH and it's likely that they, you know we did some TCO analysis. Basically on these nodes, so I would say the NIC cost is probably you know, ten five percent, two or three percent of the total cost. Oh yeah only made me okay, I, think yeah cool.

D

I wondered to Sheriff um separately we're talking about the insert a refresh. If, if this quantity of machines might be a possibility, it could be possible to get a balanced set of boxes.

G

Yeah so I I'm trying to see if we can, if we can get yeah get kit. You know basically the latest system that that held the right expanders. So for the skylake systems, the white boxes, saintil white boxes, they they come with different expander options, so we have figured out how to how to actually configure those new balanced. We may not be able to get like everything you about where it means the nvme we can get knew about so I'll say I will try to make it.

G

You know as close to yeah, don't think as possible and if, like I'm, also trying to get some quotes from quanta to see, if they can, they can give us systems with good dental. If you can get those qct systems, they're one you and perfectly balanced I mean that that would be actually the ideal ideal case. I'll I'll see what I can do.

G

I I'm just just being asked: we can we actually lost the opportunity for the budget last quarter, just trying to reassess things this quarter to see how quickly we can get budget, and so the idea is was actually to to announce something like this at work on that we have a new cluster in.

F

This community.

G

Coaster and then just make some announcement, so we I'm so so the marketing is definitely trying to get get this done this quarter so that we can. We have some time to you know get it installed: potential cool, okay, okay, yeah.

A

This this machine is not totally balanced. It's like six and four, but it's it's. You know maybe better than some hey I remember seeing this from.

G

Yeah, the 1028 right yeah.

A

G

1029 skylake here that's right, so that that one is six-four six-four yep, but it's it does. But you don't have the by 16. You know we don't you don't have the make like balance, they capability on both nodes. Typically on don't.

D

G

Though service the way they bring it out, yeah.

G

C

Yeah with that close actually.

G

Even if I can get 10 20 lines, we can do for and game each node and that's how we we did it actually last time and I, know and I think those 10 28 or 29 teased for the balanced config.

C

That's an eighth yeah.

G

That was the broader bill: okay, yeah, okay, yeah.

E

G

I'll I'll look at the options is for Marquis.

G

D

All right anything else.

D

All right, I'll get back to the Nautilus stuff, yeah yeah, all right thanks! Thank you guys.

G

D

Everyone take care thanks.