Ceph CDS Hammer, 28 Oct 2014

Previous Meeting

⏯

youtube image

►

From YouTube: CDS Hammer (Day 1) - Towards Ceph Cold Storage

Description

http://goo.gl/U4b70r

28 October 2014
Ceph Developer Summit: Hammer
Day 1
Towards Ceph Cold Storage
Matthias Grawinkel, Marcel Lauhoff

A

All right here we are on to the next session.

A

uh The next session here we have is discussing cold storage um as it applies to seph, and it looks like we have a couple of a couple of guys here that propose the blueprints, matthias and marcel, and so one of you guys wants to give us a little bit of an overview of what you're thinking here and we can take it from there.

B

Sure hello, so I hope you hear me this is.

B

First, hello from germany, um so I'm matthias gravenka, I'm from the university of mainz so and my colleague marcelo from my former university of palabra, and we are still looking at from more or less scientific perspective, but um right now for also with a natural use case, so we would like to see how far we can bend seph to into a cold storage or as into an archival store and um yeah. For this we actually defined a master thesis. So marcelo is right. Now writing a master thesis with the topic.

B

Basically, how can we cool down seth, and we have some ideas, how to do this? What could be done there and yeah? So, basically, we would like to see what you think of the ideas, if you think some of them could be doable if they are interesting. Maybe you also have some interesting questions that could go into this master thesis and um yeah. Basically, this is what this talk will be about cool okay.

B

um So basically, this is more or less a continuation. So, in the last um development subject there was from the guys from hgst. There was this first um approach they discussed. um How can we redefine a tier to make it um more silent? So basically, the biggest problem is that a lot of data is moving around, which is nice. If things are getting rebalanced, we get new posts in there.

B

Everything is rebalanced, but the question for us would be like: how can we do or create create a new tier or something that basically, the least amount of data is moved so put a file on an osd once and then in the best case, it will never ever move again until it is deleted.

B

So there we have some ideas and maybe marcel if you would like to continue here from what you think.

C

Okay, um well, um we defined some couple, some ideas. um We thought we need for cold storage and stuff. um The first is um we always came back to. um We need some kind of um placement that is energy aware and um we thought of. um Can we add, like a bucket type, that knows about how osds are powered up or powered down and maybe based on a discussion I had with luig at the seven days in paris. um Why not? Maybe.

C

Power power down based on the time, for example, if you have like three osd, some assign a weight uh to one of them. That is uh that basically prevents data from um getting there um and do that like for an hour and then switch to the next, so that the usd has like a time time to cool down or something.

C

Well, I I think it is quite doable, but I'm not sure how to get some like this uh side effect data um into the bucket. um uh The bucket choose functions.

D

Right so, okay, so I think in in that kind of situation, where you're you're, basically cycling through you, have like the three osds that have the data and you're just like or, however many and you're cycling, which ones are turned off and which ones are turned on.

D

Your idea would be that you would have like one off out of two or like two off out of three something like that.

D

I think I think it's going to be simpler if you do it not in not in the crush layer, but just above that in the osd map.

D

So the way that mapping currently works is um crush spits out the the set of ods that should store the data and then we filter out all the osds that are currently down and we get the so-called acting set, which are the ones that are currently sort of acting ones that are actually doing the work. And that's then that's those are the ones up here and actually do the rights, um and so the the down filtering it happens after crush.

D

um So I think, in this situation we could do. We could do the same sort of thing like in fact, I'm not I'm not sure in the simple case of just powering, powering down osds that it's not really that different than the oc just being down like you literally could just turn off the rack.

B

I have one more question here, so this would mean um if we power up and down disks a lot and for every disk we have one osd. This would mean for every power up and power down. We would change the crash map and push it out to basically the whole cluster would.

D

You change the osd image overhead. You would change the osd map, which is that you know like 100k structure that describes who's up and who's down, um but that that that's going to have to change no matter what so that the clients know who to who to talk to and who's up and who's down. It's a that's.

B

Like 10 000 discs, which are going up and down again, this is basically a lot of osd traffic with the map change traffic, isn't it.

D

Yeah yeah and you'll have a like a very small trickle of I o, that's sort of tracking that state okay, um but I don't. I don't know that we can get around that, because you need to know how to you need to know who has the data right and the ost map is the way that you represent that.

D

um So I think that I think the the the interesting part for me is that it's not you want to distinguish between an osd, that's deliberately powered down and one that is actually just down, because you don't know why, like it failed or because it's turned off or something like that right, yeah.

B

Basically, we talked a little bit about this sorry, but you basically to exclude them from being an active target for brights.

D

D

Yeah I mean, I guess I guess you could have it be a function of time if you had a stable clock source, but I'm not sure that you do so. I think you have to actually publish osu map updates that say that I think that's going to be the simplest. The simplest model.

E

So I've been listening this and I'm a little confused. Is there some reason that it actually needs to change the crush algorithms at all? Or couldn't these just be that we have like osc's that are down for six hours and then they.

D

Come back! Oh! Okay! Sorry! That's what I'm suggesting yeah! Sorry.

E

I guess I missed it so.

B

That um yeah yeah, the next question would be um in the pad. We have something like 1.3 like um the crush rules. There are some like this sort of a linear fill or this distribution strategies um how complex so some time ago, I read something about um making them more flexible or adding some plug-in functionality into it. Is there some some possibility to get more?

B

You know functions into these crash rules. Maybe this.

D

Yeah I mean we can yes, I mean yeah, we it definitely can I mean it's sort of designed to be um functionally modular. I guess so there. The rules have steps that you can sort of add new operation types, and we do that periodically.

D

We have four bucket types that we define right now, but you could easily define new ones, um there's a little bit of a compatibility thing where you have to make sure that the whole cluster understands the new new features that you add to it. But that's we've done that several times now, so it's pretty well worn territory.

D

um So I think if there are, if there are alternative mapping schemes that we want to pursue, then then definitely um but that that being the case, I think that the stuff, where you're you're not actually changing your notion of where the data should be stored but sort of modulating that with whether it's up or down and whether you're writing to it at this particular moment that stuff is those types of filtering. Steps can be done, one step above and without actually modifying chord brush.

B

The next question would be: how much intelligence can you get into these rules so um in in the pad? We have a point, five, which is called marked data, so basically, if as a client. So um if I would like to build an archive, sometimes I exactly know how hot or cold the data that I put into the cell cluster are, and maybe I can somehow tag them. Maybe I have blue data and red data and yellow data or something so is there any chance to get those tags?

B

Basically, maybe, as a system attribute or of object attribute that is written by the client, um can those somehow go into such a decision where, like all the the the blue uh tagged files to object stores with, I don't know, ids one to ten.

D

Something so um so you can certainly.

B

D

You can you can name them, you can tag them. You can do all those things um it that doesn't propagate all the way to crush, though, because crush isn't actually placing individual objects, it's placing placement groups, so you have a logical pool, that's the sort of logical collection of objects. That's sharded randomly basically spraying all the objects into different placement groups using a hash function, and then it's the placement group that crush is deciding where to store.

D

So I think that the way that you would approach this generally would be that if you have a blue object, you put it in the blue pool and if you have a red object, you put it in the red pool or you feed that information. You put it all in one pool and you give that information to rados and rados uses that to make a tiering decision- and it says I'm gonna actually just make this object.

D

A pointer that points off to the blue pool for this particular object, because I think it's going to be cold.

D

Which would rely- and I think that that type of redirect would rely on the um the some version of the redirect design that we kicked around. I think in the emperor time frame, but we didn't ever implement, um we've only implemented the cache steering so far, um but I think, probably having having sort of a generic way to indicate that this object is cold and is unlikely to be modified ever again or is warm. I know I do expect further updates having a generic way to communicate that down.

D

The stack would sort of inform help inform all of those turning decisions, regardless of whether it's this or something else.

B

Okay, um it would bring us to like a point to the object, steps and links to external uh storage. So I think you talked about this earlier today. I missed that one. um What is this current state? Is it like implemented? Will it be implemented? Could you give me the stock.

D

Yeah, that was that that was what I was just talking about, the redirects.

D

um We talked about it during that um summit and we haven't done anything with it since so we haven't, we haven't implemented it, so the original idea was to have have two different, um two different types of tiering, one where you have your your base, pool where the data is stored and you put a cache above it that sort of has a subset of the data that happens to be hot and your io hits the cache um and that's that's the one we implemented and then that blueprint is about having a second tier, that's colder than the base tier, where you just have a pointer where the object would normally be stored.

D

That points off somewhere else and that one is yeah. It's it's not implemented. We haven't gotten there yet. um I think there are a host of applications for it. um So one would be something that you mentioned a couple um using uh your cold raid scheme or could point off to a different raido's pool. It could point to you know the cloud it could point anywhere right as long as there's a way that the radios can sort of get input from that from that other object store.

D

um It's not currently a priority. I think um because I think the the cash steering is capturing sort of most of the tiering use cases that we're looking at right now, um I think, but but I think we definitely like to see something like that, eventually, because it's gonna open up a lot of possibilities.

B

Okay, um then we have, I think myself, would you maybe introduce this archiving demon idea.

C

Just back to the object you redirect some, I watched the um emperor video today. um Do you think it's um there you there? You assume that uh that it is uh that it's links from sev to ceph. I get. I think um this thing is hard to generalize this, to something safe to something else like http server. I don't know.

D

um Right so I took a quick look at that blueprint yesterday and I don't think it's it's hard to generalize that so the um I think the way that we originally wrote it up. I was assuming except to seth, but the only thing that's stored on the um on the cold object.

D

Is uh it's just a flag that says I am linked from something else and a back pointer to like who linked to me and that's just for like scrubbing consistency stuff, so it's not actually necessary for the um the correct operation of the system, so so yeah, I don't think it'll be. It would be hard to generalize. I think the only the only requirement would be that in order for it to work. Obviously the osd has to be able to do gets inputs to whatever it is. That's on the on the back side.

C

Yeah, of course, it is.

D

Probably tag it with some basic attributes that says, by the way, I'm a link.

C

Yeah, okay, yeah, it's it's kind of the key key. Indeed in ingredient, um we sort of for cold storage and um the next point about the archiving daemon is actually we need um these links. For that. um That thing is actually pretty pretty short. um I put some like pseudo python code in in the pad. I mean um basically iterate over all stuff and um decide on something and um um whether it's um like cold, I don't know which metadata is actually in there. I um I didn't know if there's an a-time or something.

D

This is almost exactly what the um the current tiering agent does. So, with the cast hearing there's a background thread, that's called the tiering agent. That does something like this. It iterates over objects and decides whether to demote them. It operates only on the cast here. So it goes through the cached objects and decides whether to evict them.

D

We don't explicitly store anytime, because that would mean doing a write on, um but we approximate it using bloom filters and it just so this whole infrastructure to get it to basically guess what the a time is with some with some sort of semi-certainty.

C

uh Sorry, uh it's basically already here uh there so yeah.

D

Right, okay, so, basically, if you, if you made the base, if the if the cold tier is the base tier just a regular, maybe it's a racer coated radio spool and the the cash tier is like a hot replicated tier. Then the cashier is doing exactly this there's something very close to it.

D

um So simply simply um making a rados pool that sort of satisfies your requirements for what a cold tier would do would would um get you most of this. So if you were, we were able to make a radius pool behave with powered down osds, for example, um on a schedule. Then I think it would. It would work which I think I think is sort of the.

D

Yeah, so, okay, so it sounds. I think, there's sort of two directions that we can go with this.

D

Directions um so the first is um the first direction would be um making the external redirects work, and in that case, then, um you can push cold data off to anything else. It could be. You know, amazon glacier, it could be.

D

You know your your power aware raid thing it could be, it could be anything um and that would be sort of one enabling piece um and by the way we made it when we did the tiering stuff, we exposed um all that all the information about um getting the bloom filters to tell how hot or cold an object is and controlling them promote demote decision stuff. We exposed all that via liberatos, so you could actually write your tiering agent in python.

D

um It turns out, we embedded it in the osd because it was faster and more efficient and simpler to do that, but but those raido's calls exist. So you could you could write something else externally.

D

You happen to know what should happen, um but then the second direction is, I think, dealing with uh the power down situation like how you turn up those ds, and I think we talked a little bit about this in the session that you watched earlier last time.

D

But this is the one that I'm actually most interested in in a shorter term is figuring out what what would make sense, um because I think, there's sort of two ways to do it, um and the thing I have in mind here is the actually the hgst use case where you actually have these disks that you literally want to power down individually, so the osd would be turned off um and they're are two ways to do it.

D

um One would be where the um the you spin down the disc um but the, but the processor host is still alive, so the osd is still there, but it says I'm I'm in a power down state, so don't send me any I o, or else I have to power up, um and in that case you would have to make.

D

Or and then alternatively, you would you would actually literally turn the thing down and then you'd have to have some integration where you know how to send a special packet over the internet, eighth amendment or whatever, to power back up, which is that whole extra level of integration? um But I think in both cases um the the key thing would be giving ceph the an additional osd state where it's aware that the osd exists and it's presumed to be healthy.

D

But it's um turned off, and so it should avoid it unless it needs to, in which case it should turn it. On.

D

And I think that would be.

D

It would be great to have something like that, but it um it sort of depends on what the what the expectations are and what the what the usage pattern is going to be, as you can imagine a case where you have like three replicas and you just power down two of the replicas or one of the replicas just to incrementally cut down your power or you could imagine a state where um you turn off the entire cluster for 20 hours of the day, and then you turn it on for four hours and like ingest all the new data and do the reads and then power it off again.

D

That's something closer to what, like the the cold storage systems like glacier, are presumably doing. um But do you what do you guys? What are you guys sort of targeting? I guess. What's your use case, you have in mind.

B

Yeah, I actually have a very concrete use case. um This is still, as you said, this um I've developed this um two-dimensional rate, so basically a lot of disk drives in one enclosure and um on top I have in the end file system, which is somewhat smart, where to put data, and this would be like point four.

B

um What would be the the an interesting way how to integrate an osd on top of either 120 individual disks that are smart in the way that they are always on, but um some caching schema and so on and so forth? Basically, this system is always there it's always available, um but in the worst case it will give you 10. Second latency until you get the data, this is the the actual use case there, but nevertheless, once data is written to the osd, it should not be moved to the next osd.

B

Even if I don't know the assume, the system has 100 disks, one server, 100 osds or one osd, which somehow distributes the data to the 100 disks.

B

um So basically, this additional state if an osd is on or off, is maybe not that important. For me right now, though, for these um htc disks, it's very important. I see that for me, it would be more important to have more influence on how to put this mapping from a file from an object to an osd that they will stick, that I have some control there, how to put them yeah. All blue files go to osd all the yellow files go to osd2, some something like that or to some groups.

B

um This would be very and then, if I add another server, that nothing gets rebalanced because rebalancing is good for not for cold storage.

D

Right right, okay, um so sorry in the in the context of your um your two-dimensional rate, are you assuming that you would, on top of that, you label the file system and on top of that file system, then you would have a single osd that sort of owns this big. This.

B

Is not good exactly. This might be one implementation. The other implementation would be that I have 120 directories which are internally more less mapped to the disk drives, so this would be also a possibility. Okay, okay,.

D

um So- and I think in that case I guess the third- the third direction to go and just figure out how to how to avoid migrating data, um and then I think there you just have this, this fundamental tradeoff between um having a metadataless ability to find where your object is stored versus having to store some metadata about where it goes and then then having a lookup of some sort before you find it. So.

A

D

In the in the simplest case yeah, so I I mean the way that the the current radio stuff is structured. You can break things down by pool and each pool can have its own scheme for mapping onto placing groups and placing groups onto posties. So right now, that's all done through hashing a hash function and crush, but you could presumably plug in something else, but the key. The key thing is that right now, it's all sort of defined in the osd map. So this is a compact structure.

D

That is enough information to tell where the data is.

D

So if you can come up with a way where um you know knowing.

D

Where that's enough information to find out where your object goes without having to move stuff, then then that that would work, but I think it's tricky, because if you want to arbitrarily name your objects, then you can fill in like parts of the namespace that have already written to a particular osd, and you want to make them not go there. You want to make them go somewhere else like you have to sort of like name them based on when they were written, perhaps, and that sort of makes things different right.

D

So I mean, are you? Are you thinking that you'd want to find a way to to sort of solve that problem, to make it time-based? Or are you thinking that you would actually have an external metadata server? That's sort of tracking where, where objects go,.

B

I had the hope that this could be somehow solved with the crush mathematics, some some magic in there, so having an external metadata server. I think this is not an option.

D

Yeah yeah! um Well, if you, if you name things based on when they were written with some stability, then then you could say that the you know objects written in this time. Then you could have a compact representation.

D

That says, like this slice of time is written to this device, and this slice of time is written to that device, and that could be relatively compact in that it's um it's like the size is on the order of the number of devices, and so it's not going to sort of expand the the size of the osd map by a order of magnitude or anything like that, um and that would work right um or you could be clever where you have like this this longer time period, um I was hashing over.

D

You know these 12 devices that happened to be in my cluster at that time until they filled up and then you sort of checkpoint it, and then you have some other thing you know. Does that make sense.

B

I have the feeling that I understand what you're talking about yeah.

D

Okay, there's um there is a.

B

But one one question to that: this would mean that I have to have the name of an object. This is the one that goes into it. So if I have the the um power of naming the object right, this is fine, but um if you don't plug it into a ffs, maybe this would be problematic. I think.

D

Right, but um if this is intended for cold storage, then you will be um probably not putting things directly in this pool, but you'll be putting it in a different, rados tier and then using the redirects to point off into this other thing, and so in that situation you can sort of arbitrarily name your object at that point right, because you.

A

D

You have to have a pointer that you can control the thing that the name of the thing that points to, um but if you have that pointer in where the object would normally have been stored, if it were hot, then then you can sort of.

B

Get that when exactly did you want to implement these stops.

D

Okay, yeah. Well I mean you could sort of you can sort of um for, for the purposes of like figuring out how this would work. You can just assume that you have control over naming the object or less control over naming the object right. So there's a portion of the object that you don't control or or whatever yeah. However, you want to structure it like.

D

Maybe you ask the system, give me an object name, um or maybe you say, maybe it's a combination of the part that you don't control, that the system provides and the part that you do control, but I think the thing that you might look at um that sounds like it might line up with what you're, what you're trying to do, the original the crush list buckets are based off of the original rush algorithm, which was based on um this idea that, um when you're deploying a big cluster, you sort of deploy it in stages.

D

So I would first deploy my first 100 disks and I would hash across them and then those at some point those would fill up and I'd deploy. My second chunk and so the way that the hashing algorithm- it was basically an exception that says um this percentage of the objects are stored in the new stuff and if they weren't sorted there, then they go to the old stuff. And then you have the third thing, and so it's basically a list. Is it in the newest category, if not as in the second newest category?

D

If not, is it in the third news category and I think that that type of scheme could be used with this sort of time, slicing idea where you say during this period of time I had these devices and they were sort of randomly scattered across them and then later they filled up or I deployed whatever some event happened. And then I had sort of the new view of things where I store them in the new place um and then and and so on.

D

um But you know eventually you're going to have the situation where you want to decommission the old thing and so you'll have to say like 10 of what used to be. There is now you know, got shoveled back into the new thing or something I think you could.

D

You could have some sort of layered exception type approach to defining how there that makes I'm sort of waving my hands here, but um I think that general idea could probably be applied that, in context with with the idea of slicing things up over time periods, I think would get you your compact representation.

B

um For the time slicing, um when I, when I have control, assuming that we have these stubs, um would it be possible to also integrate object metadata there? So again, this uh as a user, I mark my file as blue with a user type blue and also put the blue into the hash function.

D

I think I think that you would, I mean I think you could do that. um You could just make it part of the name even right, um but I think it would be um probably more.

D

My gut tells me it'd be more useful to um put that sort of at the next layer up, and so, if you have, if you consist exterior cluster, has a bunch of different logical pools and one pool is this, like ultra cold and one's warm and one's hot or whatever, um but the application is sort of talking to the overall data collection and marking your object, and then the system is deciding where to put it, which pool to put it in based on the temperature.

D

um I think, then, then, that that gives the system more flexibility, because you might have a really cool thing that decides to cast because for some reason, you're reading it a lot or you might have a data set, I think, maybe more importantly, you might have it.

D

You might have a situation where you have this data set where you're just ingesting a bunch of data, and you want to just mark it with you know: hot medium cold, but how it's stored on the back end might change over time, and so initially it might all be stored on disk.

D

But maybe you buy this fancy new cold storage thing and then you look at the existing metadata and you can sort of how you map that, basically onto the low level storage back end, you could you could define later at a different point in time. Okay, if that makes sense, I think it. I think it'd be a little bit more flexible to to do it. One layer up.

B

Yeah, it sounds good okay, so I think we have a lot of work and, for the master see this um yeah okay, but you think that might come out of that. Yeah, okay, okay! So, but do we have.

A

Time left or are we finished, I'm not exactly sure yeah we're just about at the end of the time slot here. But if you had a few other things to wrap up.

D

A

D

It'll probably be a little short yeah, you can um yeah, you can go if you want. If you have more more questions or whatever.

B

So actually I learned a lot. Thank you. Thank you for that. I don't know marcel. Maybe you have some some more.

C

um No, not really, I'm I mean it all comes down to. We need object, redirects to do anything. I guess yeah.

D

But I think um I think the case for having object redirects will be um bolstered by having a good thing to redirect to um so I think you could for the for the purposes, at least of your master's thing, you could probably just assume that a capability like that exists and then um and figure out how you would leverage that, like how you would structure a a logical pool that that sort of does that.

C

The thing that you can, um what do you think how much work is it to actually implement that.

D

This I'm always the wrong person to ask about that. So we had a. um We had sort of two two pieces of the the tiering picture that we originally proposed. One was the the cash tier that sort of logically sits in front um and then the the cold tier that we redirect to that sits below and the cold tier was definitely more complicated um in terms of all the interactions that would happen and promoting and handling the races and so forth.

D

So we implemented the cashier one and it took like at least twice as long as we thought it would um to be generous, um and so uh I think the the demotion one would be more more complicated, it's possible that we could simplify it somewhat from what the we originally proposed. um But I have to look at the blueprint in more detail.

D

I think it I think it's it's not trivial. Basically,.

E

On the upside, a lot of it is well specced out, and we have sort of this prototype redirect functionality built into the messenger or not in the messenger. But in terms of like the messages and the client stuff is sorted out. Yeah yeah.

D

A lot of the a lot of the pieces that it fits into have already been sorted out, so it's like it's just the sort of the core of redirect stuff. That needs to be done.

D

um So I would, let's see if you look at the if you look at the blueprint um or if I can find it the original one that you linked to actually.

D

See we actually have persistent states for indicating whether we were promoting.

D

I don't actually know if we need all of that. Do we.

E

Well, this is how we were dealing with races.

D

Yeah, I have to re-read this whole thing to see if it's actually all all that's necessary or not, because we.

D

Because we don't allow rights on the back end, similar to the cash tier.

E

uh Well, no, that was gonna, be an option.

D

Oh, I see so originally we were imagining that you could actually write to the back end directly yeah if we change this so that the back, the really cold here was only read only which I'm actually I'm guessing is probably what you would expect is that right, you probably wouldn't expect to modify this data once it's in your cold layer yeah, so that might that would probably simplify it.

D

Yeah because we were yeah, I think that was simplified quite a bit actually because we were trying to deal with.

D

Yeah being able to write the back end, yeah, okay,.

D

Yeah, all right so we'll have to revisit this, but I think it's simpler than we thought.

B

Okay or maybe you, you think it could be possible to implement it in a prototypical way for the caesars. Maybe.

D

Yeah, it depends on how much it's a it's a non-trivial amount of coding, um so it depends on how sort of deep into the stuff you want to get um again. I would I think that you would have more um get more mileage out of focusing just on the cold tier and assume that the redirect stuff exists and then implement that, because that's probably where most of the interesting stuff happens- and this is just sort of mechanics- of integrating everything into the current current structure.

B

D

B

Your end result.

D

Is your end goal is to write a paper and a thesis and not necessarily to get something that you can deploy, so they don't necessarily have to have that all. But although it's certainly great to have code at the end, but but probably not what I should be saying as the extreme person but yeah okay.

B

A

So wrap that up.

D

A