Ceph Cephalocon Barcelona 2019, 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: I Need More Space, It's Not You It's BlueStore - Mohamad Gebai, SUSE

Description

I Need More Space, It's Not You It's BlueStore - Mohamad Gebai, SUSE

Users are sometimes confused when it comes to the used space reported by Ceph. This talk will help users understand the difference between expected, perceived and actually used space for an RBD on BlueStore use case. The examples shown in this talk are inspired by frequently asked questions on the Ceph mailing list by new users.

About Mohamad Gebai
SUSE
Software Engineer
Montreal
Websitelinuxmogeb.blogspot.com
I am a software engineer at SUSE, working on the SUSE Enterprise Storage product which is based on Ceph. I was previously part of the Azure Storage team at Microsoft.

My main area of focus is performance I have a background in tracing and monitoring tools on Linux, both in kernel and user spaces.

A

Good afternoon, everyone welcome to my talk. Titled I need more space. It's not you! It's blue story, I'm Mohammed, Dubai I'm, a software engineer at Sousa. The outline for today is pretty easy. It's just a couple of examples that I've stolen from the safe users mailing list I talked about questions that users have had about space usage and safe.

A

You might be familiar with the FDF command. It's a command that shows, gives the statistics about SEF and the space basically tells you the capacity of all of your cluster. How much data has been used? How much is available? You have the row storage section, which is really in terms of plain bytes, and you have the pool section, which is one at a higher level and takes into consideration. You know logic rate or subjects, pools replicas. You know the smart about replication and etc.

A

So the post section is really what you would expect from a user standpoint. There are the roles, section is really bits and bytes.

A

If we look at the first example, the max available number in the pool section, I haven't this cluster 76 GB and total size and I've used 17 GB of it. Yet the used percentage is about 92%, so 17 GB out of 76 GB. How does it make sense that it's 92% used, yeah and I have only 1 point 5 GB left how's? How does that make any sense?

A

Well, if you look at the data distribution, you know the Aussies are unbalanced, to say the least. All of them have 5% used and one of them has 94% used. So we discover here that max avail was not really how much data is free on the your cluster is just how much data you can still write before any one of the OS DS becomes full, so it's maybe a little bit misleading, but that's the case.

A

So the solution here is very simple, as you just have to rebalance data across your cluster, you can use the up map balancer and you'll be good to go.

A

This is the second example that I have I create a RBD that is a one terabyte in size. I map it to a client node make XFS first I'm on it run. 7Tf I see that 500 megabytes are used, I have one terabyte I'm good. With that it's set in provisioned. The 500 megabytes are probably the overhead of XFS. All it looks good I mount the RPG create a bunch of files. I have here 500 files that are 6 megabytes each for a total of 3G I.

A

Look at 70, F 3 point 4 GB used I'm fine with that 3 GB for the data point, 4 GB for XFS metadata, all is good, I'd go and then delete half of the files, so I have 1.5 gb. I used, as you can see at the last time, run 7 TF. It's still 3 point 4 GB, so I've freed basically space inside my orbitty, yes, f, DF still sees it more or less and more data is being used and that's really a problem that happens as well, and you know, wherever there's thin provisioning.

A

You might see it in malloc, for instance in VMs. Whenever you have thin provisioning and you create data, and then you let it go. The underlying system doesn't necessarily know that you let let it go so. The solution here is to run FS trim on a file system, so you will deallocate all the unused blocks and we see that 7 TF goes back to what is expected, so 1.5 gb for the data 0.5 for the XFS overhead. That looks good.

A

Another example that you might not join it run into if, but that's still interesting, to mention, I use radios put to push a hundred objects that are 100 bytes, each so 100 small objects, I, look at the 70th and I see that stored is 10 kilobytes, which is what I expected 100 times. 100 bytes, but they used is 6.2 megabytes, so a pretty big overhead, and that is you to the Grenadiers granularity of blue store. So on spinners the smallest allocation I'll unit is 64 kilobytes.

A

So if we multiply 64 kilobytes and 100 objects, we get 6 point 2 megabytes, you know effectively used, which is really not very efficient. The solution here is mostly to write, bigger objects or to decrease the minimum allocation size. That's not really a good idea. You probably better try to optimize your application either use a higher level like a or GW or RBD, or basically just write, bigger objects. And here, if I, look at 100, large objects, stored and used is identical.

A

One last example similar to what I just saw with the compression if I enable compression on blue store and I look at the perf Dom. There are three compressed numbers you have blue store, compressed original, which is the data pushed by the user so coming into the compression algorithm, you have blue store compressed, which is what is coming out of the compression and blue store compressed allocated, which is again what is used on disk. So you might have a difference between those two.

A

If T your objects are too small, then you will have a lot of overhead, basically identical to the example that I just saw so again the solution right, large objects, and there is it for me.