Ceph Ceph on OpenStack, 9 Nov 2013

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph The De Facto Storage Backend for OpenStack

Description

For more than a year, Ceph has become increasingly popular and saw several deployments inside and outside OpenStack. The community and Ceph itself has greatly matured. Ceph is a fully open source distributed object store, network block device, and file system designed for reliability, performance, and scalability from terabytes to exabytes. Ceph utilizes a novel placement algorithm (CRUSH), active storage nodes, and peer-to-peer gossip protocols to avoid the scalability and reliability problems associated with centralized controllers and lookup tables. Since Grizzly, the Ceph integration gained some good additions: Havana definitely brought tons of awesome features. It also made the integration easier and definitely removed all the tiny hacks. All these things, will certainly encourage people to use Ceph in OpenStack. Ceph is excellent to back OpenStack platforms, no matter how big and complex the platform.. The main goal of the talk is to convince those of you who aren't already using Ceph as a storage backend for OpenStack to do so. I consider the Ceph technology to be the de facto storage backend for OpenStack for a lot of good reasons that I'll expose during the talk. In this session, Sebastien Han from eNovance will go through several subjects such as (depends on the time that I'll get): Quick Ceph overview (for those of you who are not familiar with it) Quick state of the integration with OpenStack (general state and Havana's best additions) Quick cinder drivers overview and comparisons Building a Ceph cluster - general considerations Use cases and design examples (hardware harware hardware) Achieve HA with Ceph Operations: backups, monitoring, upgrades Tips and best practices

A

Hello: everyone thanks for being here to real pledging to to be here today for the last day of the OpenStack summit. I know it has been a really tough week and all the parties and everything. So thank you very much for being here.

A

I'm Sebastian and I work as a cloud engineer at in events in events is a multi cloud provider, so we basically run design and build cloud platforms. We have several domain of expertise, among others, OpenStack and SEF.

A

My daily job is mainly focused on OpenStack and SEF, and I also rotate between the operational development and the Freestyle Team. Apart aside, fungicide afford per ton of my foot part of my time to blogging. So that's here are the details, my personal blog and company vlog. So don't hesitate to have a look at them.

A

During the next 30 minutes. We will be discussing the safe integration into OpenStack, so I'm, going to briefly introduce staff or disagree you're not familiar with it, except is a unified, distribute storage system that started in 2006, doing H PhD PhD it's open source under a GPL license so well vendor lock-in fully oppressors.

A

It's mainly written in C++ and well, it's basically building the future of storage on commodity hardware, which is quite good actually because we don't have any restrictions, so we can just choose them all all were really diverse out where to build your first cluster and well. It evils well, according to you to your own needs, and it's also fairly easy to to run a park and do tests. So that's that's quite good, safe as numerous key features such as Seth managing and self-healing. The main point is it's a really dynamic cluster.

A

So if something goes wrong, if you lose a note or if you lose a disk, the self will just trigger covery process, because there they are like tons of else check between each component. So as soon as close to detect that something is wrong. Well, it will just eating it up. It's self balancing, which means that as soon as you had a new disc or add a new node cluster is dynamically load balance. So all the data are just moving.

A

It's really punished for scaling because it's fairly easy to add a new disk and to add on just at a new node thanks to the tremendous puppet modules, chef modules and save deploy. So now it's really easy to deploy safe and also to do to scale sep sep as a well chef is really unique because it has a really really cool feature called crush, and it's just a data placement crush is well stands for control, replication and scalable hashing.

A

It's a letter to the random placement algorithm, which means that we don't know any well, it's just fast calculation. So, every time we want to store an object into the the cluster we have to compute the location, so we don't store anything into a hash table or something we just always calculate the location, so it makes it deterministic and it's. This is what this is pretty good. It's statistically uniform distribution.

A

Well, as mentioned earlier as soon as you add, a new node, the were clustered gets with balanced, so that's fairly easy to to take advantage of the full order, and it's a rule-based configuration. The really cool thing with crushes that you can reflect your physical while we can logically logically reflect your physical infrastructure. So basically you have a while closest is a map and within the map you have liked your auto topology and just apology of your physical infrastructure.

A

So I have nodes, you have your disks and it's rule-based configuration and what I just said its topology. Where and then you can build rules and within these rules you can just specify replication count and things like that.

A

So that's that's something really unique, and the really cool thing is that you can just specify that you have several other words if the divers already have SSD based systems and you have SATA disk system, so you can just say: okay with in this pool, stole all the objects on SSDs or store on the objects on the SATA. So that's that's quite useful to have a really good placement algorithm.

A

Just to give you well the final big picture upset. This is how Seth looks like so everything is bet is built upon the Redis object stores, so everything is stored as an object, and just on top of this, we built several several components. So you have several ways to access your data into store data, but first you have the Leigh brothers, it's just a library to access the Redis cluster, so you can basically build your own application and then from this you can just write and access all the objects.

A

So it's very easy to plug with the liberators because it has several language bindings, just by thong, C++, Ruby, Java and well a lot of languages finding what the first component of step is called well as gateway. Is it small? It's just a restful api them, just a really pivotal test of what Amazon s3 does and what OpenStack Swift. Does they?

A

It's just a restful api? So you, oh sorry, it's it has me to tenants capabilities and it does good as well. You can do it supports, supports zero application and also disaster recovery features.

A

Then the second component is called rbd. H stands for rattles block device and it's divided into two pieces. The first one is a kernel, mujo part of the colonel, so you can basically create a device and then map it on your machine. So you have an uog call our drive disk. It's well quite useful. It's just same way as ice QC does. If you want in the second piece, is just the queue kvm driver, so you can create images. There are Finn provisioned and they support snapshotting and Ian Wright clones full or incremental.

A

So that's really useful and it's well integrated into presenter and kvm. Then the last part of self is the distributed file system. It's called CFS and it's a the POSIX compliant file system that supports snap shooting as well or worth mentioning that all the pieces are really rubbish, well accepts ffs, which is well. Not really.

A

Do you say this? It's not production ready! That's sorry! Yeah almost I, think that says yes, mentioning always yeah almost awesome, because everything is really robust and already really good. So that's we are almost there to have a fully unified storage. Now it's some of the first consideration. While building your first set cluster, it's really performance oriented, but it's just the general just like a methodology. When you want to build your first step twister, the thing is how to start.

A

First of all, you need a use case and while we need within this use case, you have to establish several rules like a well Italy, you might be able to tell ok, I'm more doing I up so I'm more doing bandwidth or perhaps it's mixed, but this will ridiculous the way that you build your cluster. If you want more iOS, you won't just fascist disk with more I ups, and if you want been with, you might want a really large network bandwidth, for example, you might want to also establish sort of granted.

A

I hopes for this, so really you would like to be able to tell ok I want to deliver this amount of I hopes and this amount of bandwidth for all of my customers individually. Obviously, it's really her difficult to tell us, but well, if you can, let's just do it.

A

You definitely want to know if you use self as a standalone solution or if you self, combined with the software solution, if you use with OpenStack or or the cloud solution for example, because it's at some point, if you have like performance issues, you must really want to know that this is implemented. That way, but first of all, you know how safe forex, basically and you know how its implemented software solution. So if something goes wrong, you you know that weekly. How do you look?

A

So that's something that you want to consider as well, then you need to establish well, what's the amount of data that you want to start with and usable data? No troop, then you need to tell ok. Sep does the replication, so just you can specify replica count. It should just start to decide if you want to start with two three four or more really. You would like to establish also a failure ratio which means that when you build a cluster, you don't really want to get high density nodes.

A

If you have 100 terabytes, for example, you don't really want to build 3d. Well, each node with 33 terabytes each because, if you loo to know their well, you have a lot of data to rebalance. So you need to just establish, like a percentage ratio of the data that you're willing to load balance if something goes wrong according to performance and because as soon as you as soon as you recover well, you have to write a little bit more and client keep sweating too. So that's kind of something that you want definitely want to.

A

Consider too, he really would like to have also a data growth planning. If you know that I don't know, maybe every six months you are getting 10 more or 100 more terabytes. This will definitely change the way that you build your initial cluster. So maybe you will spend a little bit more of money, but every six months that's going to be well way easier to to scale and well obviously you need a budget, so how I won't go through any consideration about the budget but yeah?

A

Definitely it's something that packed all your requirements, things that you must not do, but I just want to highlight that don't get me wrong. This is really performance oriented, so obviously everything is doable there. But if you, if you want to avoid like unnecessarily troubleshooting, you might want to follow this considerations.

A

Usually you don't want to put a red and then, if you always DS, but you always d- is the object. Storage demon and the general recommendation is just to use one this for a 10 SD step already. Does the replication, so it's quite well justjust less to to do more application with that, so you lose space and yet not quite efficient. Just also think that, because you don't necessarily only want to do, read one or raise, you can also do read zero.

A

If you really want to burst your performances, but the graded read breaks the performance and then, if you don't have the right tool to monitor everything you might get into trouble because usually that well, what we tend to do is that the speed up your cluster is the speed of your well slowest disk within the entire cluster. So if you don't want to drop down all the performances or have like spikes, well, just don't do undo this.

A

As mentioned earlier, you don't really want to build high density nodes with the tiny cluster, because you might have a lot of things to load balance and then potentially get it close to you. If you have too many data to do load balance, we could argue on the last one that don't run step on your I providers. As mentioned.

A

This is doable obviously, but and at some point you might think that you could get like way more performance, because if you, if you have your storage layer and your I, provides layer on the hypervisor, then you can just directly access your cluster. So the first it is just really fast because is look. It locally and the second one is a little bit less, but what that's my main concern there is about memory in a but also and also about consistency on the platform. Usually, storage servers do only storage and I professor.

A

They only do memory step needs memory as well, because wallah, the more memory you have, the better filesystem caching you can get so in this case. Both of them require memory so step once memory and obviously the hypervisor wants memories. So at the end, you just end up with a really huge battle with memory, but what this it's mainly an assumption now, let's dive into the state of the integration into a vana.

A

So basically what step is so good with OpenStack because it unifies all the components. Originally, it was present in glance and then in cinder and recently in a vana. So it if eyes all the components or you just have this really single layer of storage. And then all the components are plugged into this storage layer, which is quite good because you don't need to have like a diverse storage solution for solutions for one component or another. You just have the same abstraction for storage and that's that's quite good, a valise best edition.

A

Well, first of all there there were a complete refactor of the senior driver so now which it uses the lib rattles and lip rbd. This is really cool because we can get a better error handling for that, and but that's something that we had to do thanks just for doing that.

A

By the way we have new features like flattened volumes, while creating a snapshot, because what happened on the back run is that if the sips in the detects that the that sap is also the back in storage for glance, when you create a new volume, then it creates a clone from this.

A

So if you want, if you don't really want to have like too much dependency on the chaining of the snapshots and the clone, you might want to just flatten the snapshot every time you create the volume and we have also a new policy about chrome death. So that's also what I just said. As soon as you create more vol, more clones, you at some point you just want to say, stop and then flatten the original image and then continue cloning. Everything then cinder backup was already present in grizzly, but the only back end was Swift.

A

So now we can, we can do backups from cell to cell. This is well the way we can do. It is just you can you can back up within the same poop, which is not recommended, because it's just the same machine, so you don't easily anything between domains failures, but if you, if a different post, is definitely points to different machines.

A

So while you can highlight that well, ideally you do, dr with that, with this this feature, so you have one location and you have another safe cluster when on another data center it supports our pedis tribes and the really important thing is that differential. So we do, we already do actually incremental backups when backup, except from Seth with another cluster I, know that yesterday we had a discussion, we've seen the guys for implementing an incremental API for for backups, but it's just already there.

A

If you use Seth and one of wild kids for me, one of the biggest tradition for a van around save is the Nova elaborate image type.

A

Originally, this this flag is set to file, which means that every time you create a new virtual machine, you get a filing system under well live Nova instances, in instance, UID, and this file is just the root the root disk of your virtual machine. You also have a second implementation with lvm, so you specify your volume group and then every time you put a machine, it creates a new lb and then it attaches to the kvm process.

A

Now what it does is you specify a new pool, observe a new staff pool, and then you create a new rbd image and you just connect the pointer to the kvm process, so you just directly boot all the VMS within Seth. This is definitely in a better decision, so the client or the user doesn't know anything about this.

A

This was a really huge requirement for from the community and from our customers to to just can I just boot, everything within self, instead of always doing bootiful volumes, for example, so kind of hard to automate boot from volumes into it--and everything. So now we can just directly put everything himself, which makes operation like a live migration way easier. Yes, it's only forgave him yeah yeah. The question was it: is it only with? Is it also compatible of exam and yeah? Sorry, but okay, it's not another!

A

Well, it's a part of the Nova and a cinder addition to so now. Tech support qos, which is quite good, because staff doesn't do any quest at the moment. So every I/o request are just too restricted from the hypervisor itself. So this is quite useful to allocate certain amount of vibes or bandwidth for from your hypervisor itself, it's bound with cinder volume type. So that's that's good!

A

That's the big picture of the two days of an integration. So what we do is we can just boot a vm, so it goes into SEF and then we can attach a volume. So it's also calling cinder for doing that and we can also do in a viable code yeah. That was the point before the question live. Migration is made easier when you have everything into sep so because you just have to move the kvm process and then you just reconnect the link on the on the rbd image.

A

So it's like just like really really fast know that it's also fairly easy to trigger on a vibe accurate. If you lose a compute node, you can do either novi evacuate or else to evacuate, and it's already on the safe cluster. But if you have this on the topic, Leon the hypervisor, it's quite hard to reboot the virtual machine, and it's just like the workflow, as I just explained it earlier. We have the multi back and capabilities.

A

So as soon as we create a volume, we just do a copy on white clone and we do just robt incremental backups on the second location. But the question is: is a valid? The perfect stack? Well, unfortunately, it's oh, we are almost there. I would say we are missing some some mini tiny features.

A

The problem is that we we were about to submit a new pet and then the patch cuadras district rejected, because we were just after feature freeze. So now what it does, it's you just create a new vm, the compute, well Glenn's downloads, the image and stream into the stream, the image into the compute node. So you have to download the image on the computer through a glance. Then you have to import it into itself with it, which is quite an efficient, but yours are josh, has a video patch and it's ready on the pipe.

A

So the idea, the idea here is just to do the same thing as we do already with cinder. So when we create a new vm and if the glens, if the image is already present in glens, then we just do a copy on write clone. So it's just really fast to to put a new vm. So that's probably before the bug we release for for Ravana, one of the thing that is not implemented. It's the the staffs nation snapshotting.

A

So now, every time you want to, even if you put a vm into safe and if you take the snapshot of the instance, what happened is that? Well it's just really common. Will you come and snapshot with Kim you, so you have to download the image? Well, you just absurd the instant it goes locally on the compute node, and then it goes into Glenn's, but in the future we could just call a staff snapshot.

A

So the the operation is quite in what you can do this instantly, if you're in a hurry to go into to go into production and that and if you really want to do batch, everything I think they are just only three bugs and George already built a new brand. For that. So did you well, if you really want to fix everything already a little bit about the roadmap, I sauce burn up and and beyond. This is at least we made.

A

This is really pursue know once again, but this could be the safe integration for ourselves and maybe for Jay.

A

What's missing is something that you might want to do is to have the ability to store snapshots and images into different pools, because at some point you just want a replica count of two, for example, for the images but potentially snapshot contains contain customers data, so you might want to be higher replica level like three, for example. This is something that we just want to trick whether kyouko implementation is already on the road to reach on the pub. So it's not worth mentioning it for the ISOs roadmap.

A

Things that well, we would like to see is the migration support, because currently septuagint support the volume migration understand ur. Basically, the volume migration is when you want to migrate from one black n to another and FS back n into whatever other back end, but it's not supported when you use sap, something that we could easily implement is the Nova ber middle function. The barometer is basically when you want to put a new vm, it's actually not really a vm, it's more a compute node.

A

So it's a dedicated type of well bare metal machine for your customer. Thanks to the kernel module. We could just easily enable the kernel module and then create a new orbital device and just map it to to the physical oast, so that could be really easy to do.

A

There is also this lfs implementation going on, which is just just a restful, willya, well agnostic, restful api, which can call talk with rattles and with a swift, swift cluster. So it really. You will have this lfs api that talks to yourself twister. So it's it's not really a replacement source for swift or things like that. But it's just that. Well, basically, for the OpenStack, you could also use the object store from from staff from the dashboard or whatever.

A

So you will get a complete, unified storage solution because that's going to be just like everywhere and while potentially the Manila support. Analyst report is an initiative from netapp I guess in its the distributed file system as a service solution. So we could also, just just add a new driver for force ffs and just create a new distributed file system for one of our customer, because it's also really huge requirement for legacy applications to do use a distributed file system.

A

This is while this is the high selves roadmap, but it's just basically what I just said earlier with the picture, but it's just for you guys to have the this kind of reminder there what's coming up with incest or for the next release. We don't have that much like really new fancy features for Emperor's or with just directly jump for a firefly which should be landing february2014.

A

We have the cheering functionality, which is the basically you have this notion of cold and hard storage, so you can have like a pool that has a bunch of SSDs and then everything goes into this pool and then well politically. We just flush everything on the on the back end with SATA disk, when the data is less requested, we have the erasure coding with it, which is just well more or less like a red 5 on the software-defined storage fashion.

A

So you can just like have a really large compression of your data ZFS report for the for the file system of do SD, it's quite good, because we, but it's really really really deep it'll for this. So it's just good thing because we can use red parallel mode with the jomo, so we just do the same thing as we are supposed to do with bare FS, but it's instead production-ready. We can't use it.

A

So that's that's definitely good that we can have the ZFS report and obviously we will do all the efforts that we can to fully support the OpenStack Icehouse release, it's it's both in tank and community roadmap. So that's that's it that's it for me so like to thank you for your kind intention and if you have any questions it's time for our questions may be pretty fast here. So we we have like 15 minutes for questions. So yes,.

A

Yes, it's more Nova, but yeah.

A

Yeah yeah, but well, actually it doesn't support cuoco to every time we do this. You must ensure that the image is no roof format. So if the image welcome, the image is already in glance and it's already a roof format, so yeah. But the question is yeah. It does cook out because we do a clone. So it's copy-on-write cloning.

A

Yeah, because what we do is as soon as we store the image into glance, these mitch is naturally and protected and then from the specific snapshot, we just run a bunch of clones as soon as we create a new virtual machine.

A

A

A

A

Yes, I'm not really familiar with edifice but as far as I know and a if someone knows more about this feel free to jump in and explain us what lfs is. Basically it's just a restful api agnostic wistfully appear that came talks to several. Whatever object: storage back in so you just just do a request on lfs and then you can have a either Swift packin object, store or aratus objects to work.

A

It's just a way to unify all the aps and just to have a single abstraction layer and no matter which objects to which you have underneath.

A

Yeah yeah I really have no idea, I, don't think so, and I don't really know the status or nor the progress of the implementation. So uh you should have the grocery fest guys because they they initiated the initiative. Okay, yes yeah. The second is you and then yeah? Yes, encryption.

A

I know that there is something, but it's a safe specific and there is also something on encryption, but I barely looked at it, so I I, don't have the answer. Sorry about the encryption. I know that Ritter ism there is a module that may have landed for vana, but I'm not sure if it does inning. ah I just don't know.

A

There is always really complex because you don't even know where to store the key and everything, although there is a project in this like barbican or something that does the key management system.

A

A

That was you and then okay.

A

That's a good question: yeah I I mainly work on debian based systems. I I don't know about there any Colonel recommendation. Maybe you know sage.

A

A

Well, thank I. Think sage is the definitely the person to ask.

A

Any more questions, yes,.

A

What is the singing for.

A

Know that they are there, there is no single point of failure because it doesn't work like Swift, for example, where you have like this proxy. That request the object stories. You directly talk to the object servers, so you don't have any single entry point to recruit review data. You just TT talk with the object service. Ok, the question was: if there were any support from on sep, yeah largest elation. Well, as far as I know, it's 5 petabytes dream, mousse or more or less yeah, 5 petabytes.

A

Any more questions.

A

A

What is coming wait, what is more, coming.

A

A

No, no, it's like it's really flexible. So no, there is no specific use cases for that. Yeah.

A

Yeah, so what's the amount of how many discs should I put into a single machine? Why do you have into you have to take into consideration many things? What's your network bandwidth? What do you want to achieve? Also, if you're like really high-ups well from iOS perspective, you can pack a lot of thigh ops on a gigabit link, so you can do like Halfin, SSD and then you're a gigabit network. Bandwidth is full if you are 10 gig.

A

Obviously you are like unlimited honest, but in terms of bandwidth and if you, if you are, if you want to be, if you want to achieve performance in terms of bandwidth, just keep in mind that a single set up enterprise or drive disk can just fulfill your Gigabit bandwidth. Then, if you go with the 10 gigabit network, you have to think about. So, while you can mostly the river well one dot to ok and the staff has a really specific design, which means that first, you eat a journal and then you float the data.

A

So basically, this more or less split into two, the iOS and then the band wave. So general recommendation is 12 disks permission, but you can go to 24 if well, that's only the theory on the bandwidth. If you go with 24, you just can just fulfill the entire bandwidth and the wall. Server will just make sure that the red controller supports it, but yeah.

A

That's if you want to use an SSD, for example, just to burst the performance and don't have any impact, because as soon as you hit the disk, you do that it at 50 megabytes, for example, and a new flash and 50. But if you use an SSD, the f is d: just absorbs everything, but one more time, if you don't don't put too many. Oh, is this on a single SSD, so based on your calculation, you have like what an enterprise SSD can deliver. 500 makes sequential, writes per second, because the runner is only sequential.

A

So if you just put like 40 s, DS, you go around a little bit more than one Android, so you need to establish you on a ratio for that, but traditionally are becoming like 60 s, DS for a single drawer, which leads to weld 12 disks. In the end, and then you already fulfilled your Gigabit bandwidth, I, don't know that much about InfiniBand but I've been doing that so they just this will change everything if you go to infiniband, but we've gigabit the gigabit link. Here, that's the the answer.

A

That's going to be useful on the Internet yeah, I'm going to post it on SlideShare and that's going to be treated by you know vents. Definitely so you you will find it anywhere. I'll grab you an email. Ok, that's yeah going to be soon on the Internet.

A

A

The region's report ratio yeah; okay, oh the universe, recording yeah, that's what you're asking what okay and I'm sorry what's the way? What's the question, how do you guys mean I'm not doing anything for it? I'm, not the main developer of the erasure, coding and I'm, not sure Isla, it does is look there. I, don't know, but I mean yea. Once again, sage knows about it.

A

A

A

We have two more minutes, otherwise we are. If you have any more questions,.

A

Okay, thank you very much. Everyone.