Ceph Ceph Month 2021, 21 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph Month 2021: Qemu: librbd vs krbd performance

Description

Presented by: Wout van Heeswijk
Full schedule: https://pad.ceph.com/p/ceph-month-june-2021

A

Yeah we set out I'm going to try and really talk like lightning, because I uh have a bunch of information, but we have customers coming in and they they have questions about the performance of krbd or librbd or krbd versus lip rbd, and this is mostly related to kimu workloads.

A

So several customers have asked these questions and it a lot of times it is formulated as a feeling of slowness. So I feel like lip. Rbd is slower than krbd, or I feel like this workload is slower than that workload.

A

So we set out basically to see how we can show these differences.

A

What we, what we set out to do is we wanted to make sure that we gathered data um that makes us able to compare the performance characteristics of the two major ceph clients, the major rbd clients, so the lib rbd user space client and the krbd kernel client.

A

We wanted to make sure that our results are comparable and that we could present them not only here but also to our customers, and we want to make sure that we are not talking about our feelings about performance.

A

um So we set out with a clear goal: we wanted to define clear scenarios to test, uh define clear tests that are comparable or the results are comparable and then present these compared results.

A

What we often find is that somebody will think something or feel something about the performance, but not be able to describe what the exact test was or what the exact scenario was. So we feel it's very important still I'm here talking about my feelings, we feel like it's important, so there are four major scenarios that we have defined for this particular test. It's chemo using librbd and we will be doing the test inside of the virtual machine using a lib io engine kimo using krbd.

A

Also for your lipvert same I o engine, then we will be testing the host lib rbd.

A

So that's bare metal on the same host that the virtual machine exists on and we will be using the lib rbdio engine of vo io and we will be testing the hostg performance on bare metal with the krbd mapping and the io engine will be lib io here now the tests are for us. What what was important is to see different types of results, so we vito did a talk yesterday about the importance of testing with q depth of one.

A

So we have that as part of it, but we also test with q depth of 32 and 128, because it amplifies problems and it makes it much more visible and it can also sometimes reveal different types of problems.

A

We did five for each scenario and each test combination. We did five runs.

A

Non-Sequential, so we did, uh we did one run for each of the scenarios and then the second run for each of the scenarios. So they are not con consecutive um we did a random, read random, write profile, clock sizes, 4k, 64k, 4mbs and the I o engines, like I said before, and we are only looking at iops right now, just because it's a handy number that is easily compared.

A

We try to avoid any optimizations or any caches or stuff like that. So it's all direct and we try to test the actual client performance and not the local caching, of the clients.

A

So the setup is not very interesting to me because it's what we are trying to achieve here is not trying to get the highest speeds or anything. We are just trying to get results that are comparable and that that was our goal not do the best, have the best chef cluster or have the best setup. This is a practical setup that is running production for this customer.

A

So here we have our results of the kfd versus liberty and the vm, and this is not really the vastly different performance results that that were felt by our customer and they're, not really the what we see in the marketplace. But but this is what we saw in this test environment.

A

So we really tried to focus on the comparability of these numbers. So in the bottom you can see the increasing q depths and the increasing block sizes.

A

And I think it's quite interesting, because if we compare the numbers here between lib rbd and krbd and we do an average of the average of the average in percentages, then they are a hundred percent, almost a hundred percent the same so the krbd performs at on average at a hundred percent of the library dvm, and this is vo, running inside the virtual machine using libio.

A

Now this is, um it almost looks like the performance goes half, but it's not. If you look at the left, you can see that the scale changes.

A

So if we look at the kbd versus lip rbd on the host there, we can see massive differences, and here we can see that the krbd had acute with block size of 4k and a q depth of 128.

A

It tops out for this cluster at 45 000 almost while the lib rbd tops out at nearly 20 000 for just about everything. um You see that if you increase the block sizes, uh then performance does take a dip a little bit, but that's to be expected.

A

So this is, um the cavity host is tested with live io of nvo and the live rbd host is tested with live rbd in fio.

A

Here the we have a little bit more comparison between the q depths.

A

um What you see here is that the the lib rbd vm and the krbv vm are are roughly the same again. So lib rbd, host and kfd host are variously different, but the vms are generally the same. So there's another ceiling there that we are encountering.

A

Here this is the block sizes, small versus large, and there was a bit of a surprise here to us that the krbd performs a little bit worse than live rbd when the block sizes are 464 case.

A

I don't have an explanation there. We just have the results and we haven't delved into why this is, but I'm very keen to see why.

A

Our notes of interest for these tests are that lib rbd has a performance ceiling in this particular setup. It's about 20, 000 iops. um Last monday, ilya had.

B

A

Talk of the rbd update and he talked about this same ceiling. I think, but maybe I can confirm this.

A

The lip rd and cabba d and keemu are about equal the performance. The average of the averages, like I said, is about 100, so krbd performs at 100 percent of lip rbd. In this case, um when we compare lip rbd versus le bio on krbd, we do get very different uh results, so we get an average of uh 150 percent, so krbd performs at a 150 percent of um lib rbd and at 64k chemo krbd performs uh a bit worse when compared to live rbd with 4k and 4mbs.

A

It performs better, which was interesting, but we haven't had the chance yet to to look into it in detail. So for us the next steps are. Oh, almost.

A

The next step is basically test with pacific, especially uh based on the on the torque of ilya, and then complete. Compare the results again.

A

Are there any questions? Thank you for your time, but.

B

Yeah, this is exactly the thing that I talked about uh and it's interesting that uh an independent test, sort of confirmed it uh right down to the same number, because uh what we saw was uh uh somewhere between 20, uh a thousand and thirty thousand. So some you know with some configurations. He could push it a bit higher, but uh it's again you know the fact that you also arrived at the same 20. 000 number is uh just another confirmation.

A

Yeah, so I think it's the 20 000 is probably based on the system that I was using and not not on a really practical limit of lip rpd. I think we did not set out to do everything the fastest. We just set out to test this system, and this is what this system did, but yeah the 20 000 seems to be the upper limit of lip rbd in any situation, and the same limit seems to be applicable to kimu.

A

Actually, so I'm not sure why, but it seems to have the same results, but now, I'm going over time. Sorry.

A

Couldn't be some throttling, not disabled,.

A

It could be- maybe I don't know where, but we we we our, we will publish our uh our fio configuration so that everybody can uh also test that with uh on their own systems. um I don't know of any throttling that we encountered. I do think I don't know if you have seen the talk of. I highly recommend it, because it's a good update on where rbd stands and where rbd is going, and there are some clear explanations in why the ceiling exists and what they are doing about it.