Ceph Cephalocon Barcelona 2019, 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rook - Running Ceph Using Kubernetes - Alexander Trost & Kim-Norman Sahm, Cloudibility UG

Description

Rook - Running Ceph Using Kubernetes - Alexander Trost & Kim-Norman Sahm, Cloudibility UG

We are going to show how easy it is to use Rook to quickly provision a new Ceph cluster and do second day operations like updating it and using the RBD mirroring feature.
This talk will especially focus on the stabilization for the Rook Ceph part in the v0.9 release.

About Kim-Norman Sahm
Head of Cloud Technology & Executive DevOps Architect,, Cloudibility GmbH
Currently Kim is working as DevOps Architect at Cloudibility in Berlin, formerly as OpenStack Cloud Architect at T-Systems (operational services GmbH) and noris network AG. His favorite technologies are OpenStack, Ceph and K8s.

About Alexander Trost
Cloudibility
DevOps Engineer
Karlsruhe, Germany
Websitehttps://edenmal.moe/
Twitter Tweet
Currently Alexander is working for Cloudibility UG as a DevOps Engineer mostly focused on containerization and the Rook project.

He is a Rook maintainer and works on several smaller Golang projects, such as the Dell Hardware Exporter for Prometheus (galexrt/dellhw_exporter).
He spoke at meetups in Germany, KubeCon NA 2017 and ContainerDays 2018.

A

And I was working overseas so now welcome to the route talk about running safe in kubernetes, I'm Alexander, my well, my colleague isn't here to know him some. He has well he's getting old. His back is hurting already, but he's sending his greetings yeah, but there's one more well, not technically a person but Sir, sir redundancy and yes well. The sir is redundant he's the mascot of the root project and he will kind of also follow us through the presentation today for our agenda.

A

We have what is Rukh yeah I should take chance, though not only. You know that it's doing something we've searched also something we've said we have as the most important part the kuba nice integration. Otherwise well, the group wouldn't exist if it wouldn't be about companies there again kind of we have a special part for safe with the configuration.

A

How do we tell rube there to, for example, use specific discs or so on, and we also have the point: that's just for a root project in itself: they're, smart and just safe in it, and at the end, a demo about deploying yourself class in Cuba nails and yeah. Let's get right into it, so what is ruk ruk is well. This is for companies, it should probably be cloud native, so yeah it is cloud native.

A

It's is if it is a framework to well run storage pocket backends, not only like something like safe, which will we come back later to, but also things like Cassandra like in general persistence. So it's rügen itself is not only about just running big storage software like safe, it's in general, about providing a persistence layer, but as we are cephalic on surface the most interesting part here, besides that for cuban inside we utilize what kubernetes api is giving us as well even for an infrastructure, especially it is cool to have simply an API.

A

We can talk to you and, in the end, get an end result we'll go into that later on in the kubernetes native integration part there. We also have the part where, by running the whole components of a safe cluster inside of kubernetes s, containers, we can simply go ahead and update them change them as we need and spawn new ones. If you need.

A

Another part, is that s already set with ruk, not only being for safe, rügen itself tries to be a framework. For example, maybe you have heard about a company in Ex enter their half the project HFS, which is well kind of also like safe in that regard. It's also a storage back-end, where we also have block, storage and also file system search, and, to point there being that we have some common types, that you have a well that you have it easier to move from a one to the other.

A

If well, if you wanted to, when general, that there's a one common type where you which you can use to specify a disk to be used with, say if HFS or well any other, coming storage, back-end more or less on a path with, does it cost anything? No, it is open source, it is licensed under the Apache 2.0 and well, it's go for it users.

A

Besides it being open sister. We are thankful to the scene CF for hosting the project right now as great no, not incubator, incubator, yeah and yeah. So you will, if you should be at cubic and will probably also that here, one or two talks about Rukh tail.

A

This point: it's not a list as it is more or less well throw in maybe as a question to you guys who has served about CSI like well, not the police force but like CSI container storage interface. Okay, yeah CSI is to keep it simple for those who don't know yet is well an interface / API. If you will so we, where it's kind of a try to well, not a try.

A

It's pretty successful from my perspective, to have one way to request storage to get just your application goes to it or especially in that regard. Kubernetes will talk with these CSI drivers in the end and be able to no matter what storage areas be itself filled, I, don't know like NetApp staff or an NFS server, it has to say P I implemented.

A

You could just go to it and say: hey I, want storage or some well, obviously, some more parameters in the end, but you will get storage in the entire, so this makes it pretty easy to have storage for your applications and one point to note for the CGI Potter. It is not just like for kubernetes good.

A

It's more or less focused for containerization because of the container storage, interface or well the container in the name, but well you can in en probably use it for anything you want, but you would need to request Sarge another C size, future GRC flex volume still exists.

A

Before CSI they going making history dia, the Flex William is kind of the I would say temporary way of having something like CSI like well, it's basically at almost the same thing but from perspective or from feature and usage field more or less focused on cue Benitez.

A

So why am I putting this up here right now, Luke can, or by default, will use flex volume, but will already have a phone that zero has the option to deploy with CSI that will more and more be moved, I. Think in like one or two versions, it should be planned that we move CSI to the default way and slowly fade out. The Flex volume way to have one interface there and well, as kind of the work for the seaside driver that a dust mounting and everything is moved to well to start projects themselves.

A

It's kind of a bit less work on our side and SDC's I Drive a said it's not only for like group in the end, it's for everyone that would also normally just want to mount some Serge from self, so double the advantage there.

A

The architecture.

A

For dental I have my pointer from a Cuban East Side's, especially here like I'm, not showing too much off the Seth classic kimono, sir, but we'll get there at the demons part there. It may look a bit complicated it. Also it's a bit complicated, it's not just looking, but just from an outsider's perspective, there use half your cube panel tool, which well simply allows you to interact with the kubernetes api and etrog brings in custom types just me. Instead, we do ask the user.

A

If you want a new storage pool in in your self cluster, it's not a problem. You create in companies, you create a new, safe block volume or plot pool object, and then a new pool spawned is created into in the safe cluster. If you want the volume for your application, we have a small example of that later on. You would normally just use the process of how cubanía stuff sort of so called persistent volume claims, but again they're.

A

The input path is basic through the cube cobble and a client tool there, which does well, which talks which the kubernetes api, then we also have to cuba. Nice API, consisting of a bit more than one component, if you want to know it's for exact, but in the end, moving on to the more right part there we have our nodes, they are running cubelet, which is the node component, and you probably also have dist in the service and right now as it is, we flex volume, but also we've CSI.

A

You have well the root flex, volume or ACEF sees iPod there on the nodes from roof side to make it easily possible for us to kind of via orchestrate the whole thing, as you may remember, from the P nodes from seiche in the beginning, talking about the manager with the orchestration there- and you may have already heard rube there- you can from a technical standpoint already right now.

A

Oh well, we're still working on it, making it really like this from the feeling right now it would be like you create a cluster I'm going into that later. How we do that and it route will take care of creating everything, but in the end of its more fart I'd like with you, create such a cluster. But you don't get any OSD sorcerer, you just get your Mons, your manager as you need, and then you go to the dashboard.

A

Well, if you want, you go to the dashboard end and can then click click by click say: hey, please, create no Steve and then Rukh, because with running in Cuban, is, will just wanna containers on the right machines there and use those discs for the cluster right now it's well! You can have it with this too, but there are some restrictions right now, but we're working on getting the integration there, even fairer and even more improved.

A

The Roop discover pots, which should run on every node, where you have discs that you want to use it's. The part adds well, as dynamic name implies. This covers the discs on the machines and the information on what disks are in each nodes is used by well in the future, also completely in the part that you can see what is going on what disks are available, but this just what I do I want to use, but you also have the possibility.

A

You wish the information from that the operator can specifically based on that create us these already, as it is right now and then in the end, I said, 50 manager to the Rukh agent is right now the flex volume partner. This would any future be just a CSI component which takes care of the well mounting mapping and so on off the storage that you want to use from self, but also on the nodes. You will obviously will observe, like maybe a Mon, or do so you can have.

A

You can tell Luke wedge, replace those components, but for these it wouldn't make most sense. If you put it there well whether whether it is otherwise well, you won't have too much fun yeah, so dance I should check Shahrukh in itself.

A

Pretty Cuban is heavy there. But if you have someone to put a like that, if you have someone that can run you, the Cuban ease cluster.

A

It is doable, it is definitely doable.

A

Simply also because of those custom objects where you can specify what is going on or the team that runs the cluster for you gets where we come to. The Cuban is native integration, starting error with the with getting a block, storage, pool or well a pool and safe created. Yes, is what you create in Cuban?

A

Is you create a well eight line, object in kubernetes and based on that the root operator which is taking watching for those objects also for changes on those objects goes ahead, runs say what was its efforts to create, pool or something I I've done it too many times using real tight, so it under the command, if it felt like it right now, but the Ruge operator talks was to surf cluster in the kubernetes cluster, we're walking on heaven it's with an external cluster to thanks to Travis, but you create a SUP.

A

This object with that we would get a replicated pool by size free. Some are probably saying: yeah where's like stuff like min size and so on. We're working on getting more parameters available. There I think device class has just been merged, also in master yeah, so we have already knew stuff coming up after the 1.0 release.

A

But even if you want to be racially coded, you could just say racial coded and then data and department, their parameters fired continuing on with this example, for in cominius to be able to consume the storage I need to create a storage class Sturge classes. If you look at it a bit, you may already gonna see what it's basically, it's kind of just a hold of parameters.

A

It's just like information which block pool to use which cluster a namespace which will result space again ent cluster name, which well depends on the memphis you're deploying the cluster in you have the file system time. We can say: do it x, FS or x, fear or x4 on the block devices that are getting created there and well in the end unmounted, and this contains the information the operator then again also needs to create block devices in this case a.m.

A

in this effort, also because if I'm a user in the end or well yeah I'm, creating a so-called persistent volume claim object, you see it here. We even tell it like which storage class it should use. You can set them as default as a hint there, but in itself we set it here we tell what access modes we want. That is important as block search mounting it at multiple location.

A

Readwrite is well, we have sacrifice, am I right and we say, hey make it 20 gigabytes big and if we would create all those free objects and we well, we already have a safe class or existing in a kubernetes. What would resolve all of that storage, 40 application of size, 20, gigabytes and bam?

A

The application has the storage now and well to come back. Sir-Sir redundancy he slashed through the integration part for the application here.

A

Yeah coming to configuring safe for safe.

A

For safe, there is so-called safe class object that the ruk operator brings with itself, and this is just this is just a snippet of what you have as options. But part of those options are yeah. Well use all nodes.

A

You can even then, above that say which nodes of the kubernetes cluster should use like some, which have a cool label like big storage is true as a label, for example, and you would say in the top of this object, only use nodes false these, which have to label big disk, for example, or even something like use all devices where it would go through the list of devices on the server would use all devices, but only devices which are empty. So don't worry your SDA with your OS or something on it.

A

1P deleted you can even have a device filter for that again, you can say something like s DB and say what SDC to live with a red X. Basically, we have location which was the crushed key location at the cross stream, a map, well the crush map, location and well.

A

After that, we even have some Bob options which we're diving into directly. Now we have well it's already the default to use blue store at least four devices, but you can say well. I need files over, for whatever reason so well set it to false. So if you want, you couldn't specify a metadata device, you can specify how big database journalize and so honest, for example, for env meal. You want more than one OSD.

A

You can also specify that you can also say that it should be encrypted using DM, crib foster drug do trip yeah, so it's they're, basically an important part about those configuration options is that if I go back once later, I well, the intention is when you look at the config should be one more to the left, but you have this part. This is a cluster wide setting.

A

So if you said something there, it's cluster wide, but you can also, if you're going further now, you can also specify a detailed list per node, basically overriding these cluster wide settings so that, for example, you have normally the same nodes in a cluster. But then someone well has some special nodes and you could just do Adam tears in a node list as a separate part as we are in containers, we can limit and well request memory and CPU dates. We want to have for the components.

A

Noise wouldn't be too good of well use depending on the size of the disks, but yeah. So you have two settings on a cluster Y, but also on a node level. In the end.

A

Here, for example, for a node with like nvme device, where you want to have five of these for this device, but also like a device field away. You want to use all ste devices and well it's it's Yambol in the end, so it's at least a bit hume readable in the end.

A

So again with these channels that we have there, we kind of can go to the treasurer there. We just create them and we already have safe. Doesn't this? Isn't it cool? So we have a treasure chest already waiting after to store the data directory of applications so coming up, the debt is demo now.

B

A

Happy to trust double-check perfect, so what we? What I'm going to show now is I, have a free notes. Kuben is cluster which all of the nodes have one two hundred geek disk attached and well, let's create a safe class or and well besides, creating the operator in a cluster which have I done that I hope. So let us check now. I have not done that.

A

She has the circle. Would.

A

So what I'm doing right now is navigating to the repository of ruke and cutting to create in well. It's free steps, I think, actually I. Think someone created in PR to update a number of steps.

A

We had it at like two steps, but after some refactoring to make the examples easier, it was then well free, so we create a first one which what you're, seeing here right now for the people that haven't worked with Khun nai displays, be just me telling, with cube Carol to texture, yeah, no I pointed it to and run it against the company's API and create objects in this memo, and we do that for the second one as well. So.

A

To give you an insight on what we have right now, let me make it a bit smaller hope you can see it well.

A

What we have right now, we just have a roof safe operator, just a component which takes care of well if it sees those objects which we talked about earlier, do something create objects and companies like create, monitor or pods or well monitor, containers and so on, and now it has even created those other components that, if you remember back at the architecture, we'd like discover an agent. So it's also going ahead and creating those objects, and we can already go ahead.

A

Go back to the actual example file, folder and.

A

What we have here is a class IMO if now to just again what we saw previously, it's a bit more than that because well there's a bit more than just telling Seth where what distance should use from Cuba near side. You have also stuff like what network it should use in Cuba news if you want to specially put some resource limits, also again cluster white, on the on the components created by the rook operator, but I also stuff like, should there would be enabled there's also an option to disable the SSL off the dashboard.

A

And again it's it's a pretty simple one. It's just for the nodes perfectly fine right now. So what we're going to do is just double check again. Everything's running that is good and we're going to deploy the class or.

A

And you can see, but well besides the other objects previously for the other two Hamill's we created to create a safe cluster, even with a sophisticated configuration, we have multiple nodes of 10th, 20 of something or even 100,000 nodes, or so it would be just one object in the end for cuba news. So as we see it, we have created a safe class are called rube F.

A

So if we go ahead and look into that now,.

B

A

Shortly, while it's taking a bit of time and service that I'm using are not having a best, well reception, the best band wife to the to the doc up registry, what is basically happening when you create such a cluster object, the operator will spawn a container which starts the the image that is specified in a surface IMO.

A

If we, if you look here, you could specify one set first, and you want that's especially important as we decoupled that with zero nine zero nine yeah we're serious, you can specify whatever self image you want in the end, but well. Ruk can only support what it supports, as there are sometimes some slight changes of the safe click command.

A

But where is it.

A

Yeah, so it's taking this image that you specified and start one container to just make sure what version it is. This is kind of just like a well safety measure there. It's not all images at the at the end of the that you specify there would have assemble some antic version pattern like vivre, 13.2 and so on and after that Wow perfect, we have months now.

A

What the operated in does is create. The first monitor wait for it to start well to have form quorum start. The next monitor wait for them to have formed well for them again. You need free to have it in a good way.

A

Now it's basically initializing the third monitor, which is the Mon, see right now it is created a third one.

A

Now, well, it takes a bit of time till a little, not much but a few seconds for the monitors to form the actual Iranian, or also for the operator to do certain things after that, which we, for example, are now already at a stage at having a self manager running which, while it's still creating a container in the kubernetes cluster over the pot. But now we have it, as we saw any cluster yamo like those dashboard options, we only saw like the is it enabled or not.

A

Those options are, for example, now done by the operator land and how it's waiting for the manager to be ready to take those options configured with the self-clean and after it's perfect, we are you now want further and after it has set up the manager and with that it will right now, as it is, create so called safe, OST prepare pots.

A

What does do well, they prepare is these. They take the information that has been get out but by the roof, discover and which has orchestrated I, have been kind of mate well into a map. If you will sell by the operator and based on that, these jobs run, maybe we up that spits yeah. We already have some of these right. We have to free all this yep, perfect and now, after those prepared shops is run, they take two discs or what?

A

What the directors that you want to use prepare them to be a notes a first year and well then we have it. We have three of these, but to make this more visible, as we all love to see it. Also from a safe s perspective, let me quickly jump into the toolbox. This is running already. It is running already there.

B

A

We have free burger notes in this case over free servers. We have to set it's healthy. Well, we haven't created a pool yet, but we can do it in a second and suppose the it owes the tree output. It's also looking good.

A

We have to free all these up and running, and even now, if we just go to where it's the file to this wonderful llamó, we have looking at above the first part at the top part, we have again a self block pool object which we shortly had in a business storage example and well basically, those two examples we had to recombine. We have, we say: please create a pool names, replica pool with a size of free and D storage class again, where it references this block, pool and the name space of the cluster.

A

So if we go ahead and create that to then go into the tool box as we call it, SEF toolbox run stuff s and there we go. It's well still creating partially. Let's see yes, perfect next different clean. We now have created one pool with well, one object in Cuba needs and to go further on that.

A

We headed about those persistent volume claim objects in Cuba, news, I hope it's visible enough for you guys, but this is just a grand example from previously we have. We tell it what storage classes should use. We say: hey, we want 20, gigs of storage, so that's Creator.

A

And if I'm now hopping into the the tool box, again a really cool that we have it, the application could now go abs in cubanía, say: hey I, want you to use this claim of storage and it would start and get the storage mounted.

B

A

So let me switch back.

A

So maybe just woman out there is service are from in the Hetzer cloud, using their volumes storage there. It was three nodes in the end and from server type 631 from the cloud offer.

A

Yeah, so we we have mostly heard about chef right now, but Rukh is more than self just more or less as a side note, there.

A

Rooks has, for example, as we had like the roof chef operator, but if you wanted to run a GFS from the next Center guys, you would have a roof HFS operator. If you want to run Cassandra or me now, you would have will then roof Cassandra operate and so on. So it's not like you deploy one, you deploy ruk and you have everything. It's splitted that exactly the people that wants f can have their self stuff as they wanted, and people that would want to run his center can run.

A

They are Cassandra operator or even combine it and with a roof safe operator to have a cluster for storage. There yeah.

A

And they're a big mansion to the community behind route to that thanks to them, even for also for a safe part, not only for the safe part, with the great work going on with us, we had it like the diverse narrative device class. Finally coming to those block pool objects.

A

That was done thanks to the community, so we can only do it together and to get involved. If you want to check it out, el salud yeah go to to get up. We had. We just had an awesome site, redesign.

A

You Lola you'll love it. The Cypress is reason if you want to contribute data go to get up. We have also have a slack, especially for conferences, lady hashtag conferences, channel there we our on Twitter, we have a mailing list too, and we have like be weekly community meeting. If you want to well bring something up to the community there and yeah again, a big thanks to the community as well yeah we're for the community. We wouldn't be as far as we are right now,.

A

Already Christians does anyone have half a question.

A

Do we have do we have a microphone.

A

Migrating existing surf class at Uruk.

A

You can you can I have played around if you can do it. It's.

A

More or less due to the fact it's the rooks have operated inertia, inertia initializes everything for you. If it would have like an option to like the monitors, not initialized and completely, then you could start the first one. Add it to your existing class start the next one, but rooks F isn't too much into that direction. But.

A

Any other questions.

B

Thank you for your presentation. I have a question. Do we need org agents if we are using safe only to provision storage and consumers are coming outside of Koopalings cold.

A

Can the consumers of storage of a rug, safe trust, the outside right, yeah yeah?

A

Yes, it's your kind of well with that obviously kind of losing out on the fact that if a monitor fails in companies like the note on which the monitor fails right now, what you could do is spin up a new monitor on a new note, basically, which they'd have a failover. If you will, sir, if you, if you're able to keep up, denotes amount of storage, if they would at one point it really completely an amount of search, updated, roosters list, so you some way with load balancing or so there then yeah.

A

It is possible. There are people that use it especially for surface, but I think there are even people for block search, I, think yeah outside yeah yeah. We we hear about people using that already like that, but there's one specific class you would need to watch out for is. You would need to use the host network option true, because otherwise, as it's in Cuban is like capsuled into a Sdn, well, your you wouldn't be able to reach yourself components, but we've hosted rogue. You would then just use the host networks like yes,.

B

We are using host networking so yeah we and our consumers outside of Burnett's, Custer and T. It is from my not sure that why we need to run a safe for a rogue agent inside, so it's kind of icon and looks like not needed onion. We can't consuming box torch outside well right.

A

Now you can disable where they run, so you would just keep them running. It shouldn't be any harm done. If you just keep running so well, no.

B

Okay, what gives thank you at.

A

The back there was a question good, do you do you want to sprint? I can sprint.

C

Do you support any kind of auto-scaling of MDS azure rattles gateways and what are the metrics like? Can you put custom set metrics there.

A

Right now not, but you could trust from company's perspective, create a so-called HPA, a horizontal pot autoscaler, which would then based on a metric scale it up, but I, think more and more looking into like the chef manager orchestrate apart there. This would basically the way to go that he managed I would see that and then orchestrate as it's a generic interface there, that we, that is a need for more MDS, so should be possible. Yeah. Other questions.

A

If not I'm I'm here the whole day July well, even then the cube comes. Oh yeah.

D

That maybe I'll make a comment to about my related features coming up so in the 1.1 released for rook looking at connecting external SEF clusters. So, for example, if you've got kubernetes running here and you've already got an existing SEF cluster over there, you want to connect your kubernetes pods to that storage. That already exists, so we want to make that simplified.

D

So it doesn't require extra manual steps to get that working so X connecting to external storage clusters as part of that, for example, getting the CSI driver to be aware of the external cluster connect to that that storage.

A

And well, if you, after to talk now, have any questions, I think you you can. You should be able to find me and well if you want some, stick us at the upon root. Booth, there's also the flag on the table there. So if you want some, stick us or even have some question, I'm, probably being around dead poof, so yeah.