Ceph Ceph Virtual 2022, 10 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Optimizing RGW Object Storage Mixed Media through Storage Classes and Lua Scripting

Description

Presented by: Curt Burns and Anthony D'Atri

Optimizing RGW Object Storage Mixed Media through Storage Classes and Lua Scripting

Ceph enables flexible and scalable object storage of unstructured data for a wide variety of workloads. RGW (RADOS GateWay) deployments experience a wide variety of object sizes and must balance workload, cost, and performance requirements. S3 storage classes are an established way to steer data onto underlying media that meet specific resilience, cost, and performance requirements. One might for example define RGW back end storage classes for SSD or HDD media, non-redundant vs replicated vs erasure coding pools, etc. Diversion of individual objects or entire buckets into a non-default storage class usually requires specific client action. Compliance however can be awkward to request and impossible to enforce, especially in multi-tenant deployments that may include paying customers as well as internal users. This work enables the RGW back end to enforce storage class on uploaded objects based on specific criteria without requiring client actions. For example one might define a default storage class on performance TLC or Optane media for resource-intensive small S3 objects while assigning larger objects to Matt Vandermeulendense and cost-effective QLC SSD media.

A

um So today's talk is about optimizing: rados Gateway, object, storage on mixed media through storage classes and Lewis scripting, so my name is Kurt Bruns um I'm I did this work with Anthony diatri and uh we basically did some some work that we put into Quincy to show how to basically optimize your object, storage and object. Storage in this case is rados, Gateway objects, not rados objects and through the storage classes and Lua scripting that was put into Pacific get to the next slide here.

A

So my name is Kurt Bruns I'm, a Staff software engineer at solidime I also contribute to the spdk project and Anthony's. My colleague are you online Anthony, introduce yourself.

A

I know he had a he had some things happening this morning, so um I wasn't sure if he was going to be able to join right away, but uh anyway, Anthony works at index exchange. He's a principal engineer. There he's written a book on Seth learning. Seth second edition he's also published some papers that are listed here and he contributes to the stuff project in the documentation. Sections.

A

So we're going to start at a high level talking about what are the characteristics of a typical sephratos Gateway deployment um and any of you that run a radius Gateway know you get a wide variety of object. Sizes being stored in the cluster uh workload is very read intensive, mostly the 80. If it's a data center workloads, probably 80 reads: 20 writes if it's an AI workload, you're, probably even more reads than writes, probably I've, seen numbers up to like five thousand to one reads: to writes on a on a AI intensive workload.

A

Once a bottle is trained and you're just doing inference. So that's very small rights like .01 Drive rights per day, type of activity uh into your back end, um but when you're sending these rights, these large objects demand. You know a high read throughput um and they want a lot of bandwidth and they don't cash very well.

A

um Small object operations are kind of at the other end of our four corners right. So you want iops, you want slow latency. You want to be able to get those objects in and out of the cluster quickly. um So you kind of have these competing demands and they conflict right. So you don't care about latency, so much for your large objects, but you do care about that, for your small objects, um so typically hard drives are the traditional choice.

A

You have great dollar per terabyte, but when you start getting these hard drives that have all these objects on them and their interfaces. Legacy you're talking SATA you're, not you're, talking 100 to 180 iops.

A

um The bandwidth isn't, isn't fantastic on those as well. So then you start getting these bottlenecks in your cluster. um It's exacerbated with these ultra dense servers. You can get 90 drives per chassis and some of those super super micro ones. The storage admins were all under pressure to minimize capex, um and so the dollar per terabyte is great, but then you could suffer in performance. So it's kind of this uh whole Balancing, Act Right of iops bandwidth, latency resiliency, and how much everything costs ssds are great, um but sometimes they've been prohibitive in cost.

A

uh I think they're they're getting much better now, but how to use them efficiently is important.

A

And so, if you're, balancing these iops you're balancing bandwidth, one way to do it is to do multiple pools right. So you can kind of put these large objects in this kind of cheap pool of, say, hard drives or high capacity ssds and then put uh create us another Pool um like eraser Well, you would do a ratio coded HDD pool for cold data or large objects. That aren't touched very often, um but then you use for these. For these small objects that require quick access, you could do a replicated.

A

Ssd pool um call that hot data, if you will um so yeah, you kind of have this in RBD layer already right. Well, there's this concept of putting these multiple storage pools um inside inside Seth um radio's Gateway can support this as well. The way they do that is with the S3 protocol feature called storage classes. You set up a storage class and then you can do basically steering these storage classes to the different pools.

A

So you create these large, cheap, deep, slow storage classes, hard drives or bulk capacity ssds for these huge objects, and then you use your TLC type ssds, which are, which is a faster SSD for small objects and high iops. So uh what the kind of issue is or what has to happen? Is you need the user then to basically attach a storage class as an HTTP request to tell it what storage class you want to store this object into?

A

So once you start having the end users make these decisions on what storage classes they need to use. You can get compliance, problems, copy and paste scripts right of storing, uh storing an object into a certain storage class could be could be invalid.

A

It could be wrong, and so um there could be a problem of basically not getting the storage into the right pool, and so some more background around Blue, Store stuff allocates something called Min Alex size of underlying storage, so for every right into the cluster, CEF blue store is going to allocate Min Alex size of underlying storage. So even if you write one byte, the Min Alex size is 4K. So that's what will be allocated um now.

A

Ssds uh also have kind of this underlying uh men alloc size, if you will, um but that's called um indirection unit.

A

Historically ssds have kind of matched the 4K indirection unit, um but if you don't match indirection unit with the actual right size, you can get imbalance in your in your drives and you'll end up um be an inefficient use of your drive.

A

um So uh the latest qlc drives uh are great for hard disk drive Replacements. um They have faster rights, fast reads, um and they have densities high as one petabyte per rack unit. um So that's in kind of the ruler form factor you can get a lot of storage in a small amount of space.

A

um The downside to the qlc is: is the fact that the IU, because of the cost you want to make? uh If you want to eliminate some of the ram that's on the SSD, so you create this indirection unit as 64k, which is now 16 times higher than that 4K. So, every time you do a write, you're going to allocate 64k on that SSD to create that right, um which can end up making this overhead worse, and it can even be where it's really bad around small objects.

A

And then, if you use Erasure coating on top of that and you're doing small objects, you're going to have issues with basically uh what we're going to call allocation amplification. So this is classic space amplification. Where you basically are allocating more in the SSD, then you actually need to use for your data um and so there's some example down there for large objects.

A

You'd want to put them into qlc, pools and small objects, you'd put them into TLC pools and so qlc's quad level cell, newer technology, where you can store four bits per cell in a in a nan device. And then TLC is three bits per cell and, like I mentioned, I use indirection unit, which is kind of the abstraction of uh of how the SSD allocates its uh storage.

A

So we talked about a little bit about space amplification um in Seth, there's two areas that I call replication and allocation space amplification um triple replication, which is the default for pools created in SEF. That always means you're at least 200 percent uh replication space amplification, because you have your data plus two copies of it, so you're, basically allocating two times more um or two additional copies for every one data copy.

A

Now a problem you can have I mentioned is writing less than that Min Alex eyes. We just talked about that causes allocation space. Amplification um may not be that big of an issue with 4K. You have that kind of high granularity indirection unit, but once you start getting into these larger indirection units 64k every time you do a write, you're going to allocate at least 64k.

A

um So this example down here shows you. So if we just had a single 4K right, we're going to have our normal replication amplification right, we're going to make two copies of it. So there's our 200 percent.

A

um But if we're writing that into a 64k indirection unit, you basically have your data, but then it's going to have 60k of just kind of allocated space. That's not been written to so you're really losing a lot of space in your drive. This is obviously a contrived example. That's worst case, but uh you're going to get that across your three copies as well. When you do your rights into the cluster, that's going to allocate three different areas to write and all of them will be 64k each.

A

So a single 4K object allocates 192 total and if you do just the 64k of data plus 128k of the replica, that's 47 times more data than you needed to. Actually you allocated 47 times more than you actually needed again.

A

This is not common because you probably won't be writing just small 4K objects, but it is a potential problem with these larger IU drives and so just to show this and data out of a ceph cluster I set up a pool, a triple replication, qlc pool and basically that that's just associated with the device class of qlc set up the pool and then did a single 4K write into that pool, um and this could be example of say, like I had mentioned before, having the user control, where these rights go, if they mess it up and they send qlc class and the HTTP request header as a storage class, um it's going to go into that qlc class and uh if it's just a 4K object, you're going to end up getting kind of this extra bytes written allocated into the cluster um and then space amp for allocation is 15 times, but truly the total space amp uh that includes allocation plus replication is 4 700.

A

So 47 times this exactly what we showed on the last slide. But this is actually using ceftools uh pool stats, OSD replication stats, those things to pull that data out of the cluster um I wrote kind of the these scripts here to basically pull out space amp um from a cluster, and it can work on. You know a whole cluster, but I was just doing this simple kind of one rados object stored in the cluster.

A

So what do we do? We need to simplify storing objects into the correct pools. Don't rely on the user to make the the correct choices all the time on where they want the data go in the pools. We can do things smarter in storage admins.

A

So for those of you that are familiar with it, the Pacific release brought the ability to have Lewis scripts, uh basically get hooks into different rados Gateway events that happen um and if we use these Lua hooks to basically look at the put objects, and so when objects get put into the cluster, we can look at the storage class, and then we can do something with it. We can make decisions on how to how to handle that we can overwrite it we can. We can, if they didn't, have a storage class on it.

A

We can add one the example that I did in Quincy. That's in there now is basically look at the size of the object, that's being written. If it's over a certain threshold, then we put it into the qlc class. If it's less than that, then we put it into the TLC class, so create two pools and and steer the objects accordingly.

A

And so, if we had a TLC pool with 4K IUS, we would create that for small objects, and then we would have this qlc pool with 64k IUS um that you only want to store, save objects greater than 64., so you wouldn't be over allocating.

A

So, uh just a little more background on the qlc and TLC devices, so you get qlc ssds, which are kind of these large up to 30 terabyte drives. They have nvme performance or nvme ssds.

A

um They have competitive reads with even TLC devices um and a total cost of ownership competitive with hard disk drives um because they're large bulk devices they can store up to 30 terabytes. They can do about. You know uh just under half a drive right per day, but you won't really get near that typically in your cluster- and we were talking, you know 0.01 Drive rights per day, so their endurance is great.

A

Ssds rarely run out of endurance nowadays, uh they're so large that it's hard to fill up an SSD every day, a drive right per day um now TLC ssds or fast rights and 4kiu I talked about that. Can minimize your space amplification um and like I, just talked about in the last slide, so we're looking at this storage class header when S3 objects get uploaded into the cluster, then we direct them to the proper soft pool in the back end.

A

The example I've talked about we're going to steer it by size. Other things are available. You can look at the rados Gateway Lua page on the ceph site, and you can see all the different headers and and different HTTP methods and things that you can actually hook into with Lua. So yeah there's tons of different ways. You could kind of slice this and and handle your front end of radius Gateway.

A

um One thing that was that we had to do as well is you know we talked about this 64k IU in the qlc devices um when you create an OSD, that's when it decides what metallic size to use- and so it's nice to know uh before you create the SSD I'm. Sorry before you create the OSD. Obviously, what the metallic size of that SSD what's its optimal right size is what it's called so there's an optimal. I o size available in Linux. That's uh that's on inside the sys FS.

A

So you can query that look at what that drives optimal. I o sizes and then set your metallic size accordingly and then you spin up your OSD and its metallic size will match that.

A

So um these are some of the changes that went into Quincy. So I put this in where you automate them in alexize, OSD composition. 4K has always been the default. Well, not always, but it's been the default for the last couple of releases. We use optimal, I o size and it's like I, said it's in sys FS.

A

um This is false by default, so your OSD will default to 4K metallic size. But if you turn this on, uh when when the OSD is spinning up, it'll actually look and see when it when it queries the drive that it's going to create an OSD on um that it'll use optimal, I o size. So, for example, in my in my system, there's two nvme drives one of them's, a qlc device. One of them is a TLC and if you just search it it'll return the optimal. I o size. Sometimes it's zero.

A

If it doesn't support that, and so, if it's zero, we'll just do 4K and Alex eyes.

A

Foreign Gateway Lua changes that had to be made to support this uh support this case here, so we created uh you had to make storage class header writable, it was readable in the past, and so, if you make it read write, then we can actually modify these storage class headers as the objects are getting put. That's how we're doing it we're just looking at it on put you could change it um on.

A

You know on gets, but that wouldn't do much for you um anyway, uh the luo class or the Lewis script here, modified storage class based on size. It's pretty simple! uh We look at this radios, Gateway opina put, and if it's a put then we're going to do something with it. We say: if the storage class is empty, then we're going to look at the content length, and so, if it's greater than 64k and there's no storage class, we'll we'll just change it we'll make it a qlc class.

A

Obviously, all your pools and your device class and your storage class uh handlers on the back need to be set up right so that these pools are created and that they know how to set these into the proper pool. But that's the um you know, that's right!

A

That's the Zone setup and things like that, uh and if there is no, if it's not greater than 64k, then it's going to attach standard to it, um standards, the default storage class and so that'll just get put into whatever your standard pool is set up for in my case it was TLC and I did uh in this example. uh We only do it if there's no. If there is no storage class, if a user did attach a storage class, then it's just going to let it fall through and not change it at all.

A

That was just my example of how to do it. Obviously you could as an admin. You could change it to. However, you want, you could force it and just change, no matter what the user put in on the storage class. You could change it with different. You know controls and things like that. But simple example just showing how to steer things.

A

And so, finally, once we've done that we go back to our example of a user, putting a 4K object and a 64k object. This time they don't have a storage header attached to it, just showing that the rados Gateway Lua script is working, checking the object, size and routing it to the proper pool. When we come down here, we see we do have a four 4K object stored in our default, which was built on a ql or sorry a TLC pool, and we get our 200 space amp uh like I mentioned before.

A

That's just what you get with ceph, with triple replication you're going to get 200 space amp um and space amp totals 200 percent, uh makes sense on this 4K object because it hit men, Alex eyes, and so it allocated 4K copied. It twice makes sense.

A

A 64k object was stored um and it allocated uh exactly um what allocated 192k, which makes sense, um and it wrote uh it wrote 64k and two copies, and so it also has a space sample of two so 200, and so we simplified the routing of objects to the optimal pool with the rados Gateway Lua um inside the um inside the inside. The data path as objects get put into the cluster.

A

Here's some references on uh some papers and presentations. These are the the pull requests. That put you know the OSD uh query for optimal metallic size for an SSD. What it says for its optimal. I o size, along with the radius Gateway Lua updates and the example script.

A

uh Here's some radio, Skateway luo I was talked about earlier, where you can look at the different HTTP pieces that you can look at with Lua. It says: what's read, write iterable, so those things are listed there. So you can see how to maybe do some fancy things with your rados gateway data path.

A

uh This is our email here, I. Don't know why mine went.com twice, but uh you don't need to put.com twice. uh This is a great space amplification table uh that Mark wrote and then there's also a TCO calculator. That's pretty neat! uh Putting in like hard drive or SSD capacities. How many you know how big your storage is, and things like that you can look at what is the total cost of ownership?

A

um That's need is calculated for you.

A

So that's the presentation, that's what we did with Quincy an example we put in place on basically modifying and optimizing your storage, especially having mixed media and this qlc devices that have a larger IU, how to basically steer things to the correct pools and make your storage efficient.

A

Any questions comments. Anything.

A

All right, well, that was the rule of 15 seconds, I hope everyone has a great day good evening, wherever you're at I appreciate your time. Thank you.