Red Hat OpenShift KubeCon 2020 Office Hours, 18 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: KubeCon Office Hours: Cloud Native Storage with Rook

Description

KubeCon NA 2020 Office Hours with Knative contributors Matthias Wessendorf and Roland Huss

A

Good morning, good evening good afternoon and welcome to a very special uh kubecon office hours, the cloud native storage, with rook edition, uh run by the wonderful josh burkus from red hat josh. Please take it away.

B

Hey there, thank you, chris um welcome everybody, um and this is our second office hours, concurrent with kubecon, um in order to give you more time to actually ask your questions from the many maintainers who are here at kubecon, so that you can find out what you need to know about using various cloud native projects. uh This office hours is going to be about rook and also ceph, so cloud native storage, um and we have blaine and travis with you so guys. Do you want to introduce yourselves.

B

C

Sorry I lost the mute button for a minute.

B

C

um Yeah, I am one of the maintainers of of rook upstream. I also work for red hat as a storage engineer, uh which I think that's a pretty good, pretty good intro, okay,.

D

So I'm one of the original rook maintainers started on the project over four years ago. Now, actually just had our four year anniversary, uh it's been, it's been a great ride. I've been with red hat for the last couple of years and yep happy to be here with rook.

B

Awesome um and um as we get started, I uh you're gonna have a short presentation um to kick things off.

D

Yep we'll do I'll share my screen and we have a few slides just to give an intro and see if I can get that yep and I'll turn it over to blaine.

C

um Yeah travis and I are going to talk a little bit about uh rock and ceph ceph is rook's oldest and, like original back end, it's the most mature and the one that gets the most the most attention.

C

I think the next slide jumps into what what ceph is, which is an open source.

C

Distributed object, storage solution and then that can, with rook, be be used to give your kubernetes cluster access to block file and object access to to your data. So if you want block storage stuff can give you that, if you need file system, storage stuff can give you that s3 access also.

C

um There are three sort of core architectural layers that we kind of break things up into uh being: rook, ceph, csi and ceph itself. Ceph itself is just the data layer. I tend to kind of think of these. In, like a bottom-up level, the csi driver will provision and mount uh storage into pods for your applications and then rook is in charge of administrating and managing ceph and also can set up the cfcsi components for you and so there's a deeper architectural look that travis is going to go into.

D

Thanks blaine um yeah, so we have a few different views of what these these layers look like, and you know ultimately, rook is the you know it's the management layer for bringing your storage to to kubernetes the rook being the management layer. What does that mean? So that means that rook will communicate with kubernetes and tell how to deploy all the various pods and services and different resources that need to be created so that you can have storage in your local kubernetes cluster.

D

Now you know at a higher level, we typically think of storage, or it's been typically treated. This way where it's an external entity to kubernetes, but now rook really brings storage inside your kubernetes cluster makes it just like another application that you're managing, except, of course it's stateful. It's going to take care of your data, keep it safe. So so how does rook do this? So this diagram is kind of showing a conceptual cluster of three nodes where each of those black boxes is a is a node with various pods running on it.

D

So on this center node right here we see the rook operator. Pod is running, and this is in a fully configured cluster that rook has created, and it's done that by creating all the different pods that are needed for for staff and the csi driver. So it's kind of color coded here. You see the book. Sorry, the blue pods, like the discovery pod, it's kind of related to the rook operator. It helps discover the devices on the on the nodes.

D

The green pods are for the csi driver. That's what helps attach your volumes to your application, pods and then the red pods are the ones that are actually the cef demons that are managing this. The local storage on on your cluster, so kind of you see this there's a ceph monitor which the monitors are kind of the brains of the cef cluster.

D

They have some persistent storage to keep some metadata and then the other critical stateful daemon is the set. Osd stands for object, storage, daemon and that that's what holds the core data in the system. So you see this disk backing this osd, where this disk can be a local disk. It can be.

D

You know a pvc itself that comes from a cloud provider if you're running in the cloud but yeah, so those are the pods that are basically running to get your ceph cluster running so rook deploys all the staff demons all the set configuration that you need to to hide all of the complexities of stuff in your cluster and give you storage in kubernetes all right, so that's kind of layer. One management that we're talking about now, layer, two says: okay, you've, let's say: we've set up. Brooke you've got the storage available in the system.

D

Now, how do you provision it? How do you connect your application to it right so kind of in the spirit of application pods, so you've got an application pod that needs block storage. Let's start with block storage here on the left.

D

What you'll do typically is well you'll, create a volume claim or a pvc with read, write wants access. That claim, will request the storage from a storage class and then that go. That request goes to the ceph cpsi driver and says hey provision me this volume and and attach it to my pod.

D

So that's! Oh, that, that's in a nutshell, how you provision block storage, similar thing if you need shared file storage. So if you're, uh you have two applications uh that need to share the same storage, so each of them.

D

Well, you create a single pvc, actually not well represented by this picture, but you got a pvc that then goes to the storage class which goes to the csi driver again, so you get shared storage between two applications and then finally, the third type of storage that that needs to attach or connect to is object, storage, which uses an s3 interface and to mimic what we do with volume claims we've created.

D

This concept called bucket claims where you've got a special type of storage class for object, storage that provisions a bucket for you, so you can have s3 an s3 endpoint in your cluster to give you that object, storage, all right! So that's how you would basically connect your storage to cell, so then layer. Three. Now we get down to the data layer once the data is once it's been set up at layer, one and the csi driver has provisioned the storage.

D

The volumes in layer two now at the end, your application just wants to write and read data right, so this diagram tries to illustrate how your application now will connect to ceph as the storage provider, and there is actually no management layer in this picture.

D

So rook is out of the you know, conceptually out of the picture at this point for writing and reading data, because it all goes through ceph, which has been a stable data platform for many years longer than or longer than kubernetes exists all right, so your block storage will be mounted with the and connect well, it's already been mounted with the csi driver, but now it connects through the seth rbd kernel driver, which knows how to take care of the complexities of connecting to all the different ceph demons, the mons and osds in the cluster for those rights and without going into all the details.

D

Basically, the shared file system does the same where it connects to a ceph daemon to manage that shared file system or the s3 client. Then just talks to the rgw endpoint, which is the theft service, fraud.

D

I know that's really quick, really quick overview, but that's basically what we wanted to share to get this discussion going so yep. Where should we go from here? Any questions.

B

Not oops not seeing any questions in the chat yet um usually takes a little while to get started. um So the um did you want to do the getting started or do you want to uh go straight into talking about 1.5.

D

Why don't I yeah, maybe I'll talk about this couple more slides here, just to give a little bit more idea? Well, how would you get started with rook say you need storage in your cluster? What does it take?

D

Ultimately, we've tried to make rook as simple as possible, so you can deploy stuff and there's basically just a few manifest files that you need to get up and running and I just realized. I need to update this slide because there's one more, but we we have some resources which gives us the r back or privileges in common.yaml.

D

We have a new one that creates crds so crds.yaml, which is not shown here and then you create your operator and then you create a cluster manifest, so the cluster manifest tells rook, basically how or what settings? How do you want to deploy your cluster on the right here?

D

You can just see a couple of those settings where, basically, we say well how many mods do you want well give me three months, that's kind of the typical deployment.

D

How do you want to consume storage on all the nodes? Do you want to consume? It from pvcs do you want to consume it, just let rook look for all of the local devices on the nodes, and so this shows the simplest way which is just lets. Rook go for it and create storage on all of the nodes, whatever free device it finds and and that's basically a little simplified, of course, for this discussion, but that's pretty much it.

D

It gives you a rook cluster with just a few simple, manifest to create, and also again so that was kind of layer. One back to the layering thing now to request storage from your application. Just a simple example: the admin is going to create a storage class he's going to or they're going to create a persistent volume claim and then the application pod. So on the right here, we've got, you know what the application pod might look like in this case the simple nginx image, which is requesting storage.

D

With this persistent volume claim for the rbd pvc, which which again points us back to the storage class that they haven't created so is that it yeah. I think that that's about it so 1.5 features you say.

B

Oh, you want to stop sharing and we'll.

D

Yeah: okay: there we go.

B

Okay, um pardon me for a moment. I seem to have uh gotten a cat here. You can just see the tail. The.

D

It's like your microphone.

B

Yep yeah, if only that worked that way, the um so um our uh my first question here is 1.5 came out on friday. I think so what what did? What are the big things in 1.5.

D

Yep so 1.5, it adds a number of new features. I guess just to go through them. The I'd invite anyone to look at our blog to read through it in more detail.

D

uh Just I'm glancing through that right now, honestly, just to remind myself of all the features, the the first one, we've got so with with the storage configuration we want to encrypt the data, that's at rest and we want to do it in a way, that's as secure as possible, so we'll use a key management system like vault, so in this in 1.5 now we support vault as a kms that lets you store your keys in a secure way.

D

Rick will request the keys from ball and then encrypt the data that at rest with those keys.

D

That's one feature uh next one you support.

B

uh Do you support key cloak as well.

D

Not yet no, we, okay yeah with with the vault support we kind of put some framework in place, so we could more easily add others in the future. But yeah vault is just the one right now.

D

Another one is what we call mirroring or rbd mirroring where so you've got two clusters and you want to replicate the data across kubernetes clusters. They might be in different geographical areas or they might just be just independent clusters, where you want multiple want the data replicated across for failover or dr scenarios.

D

So the we made it more configurable through crds now to set up that replication across clusters, so there's kind of some manual setup across clusters, but go ahead.

B

Hey, so is the the mirroring is being carried out by seth, though yeah.

D

That's right so rook just helps set it up to get it configured.

B

The um so does that work across clusters. Don't you have to have like three clusters for like a tiebreaker or something.

D

um That that's a little different I'll get into that. That's actually a new feature now too, but this one says you know you want two clusters running where you've got your application running in two different places, for if one data center goes down, you can run your application in the other data center and but totally independent kubernetes clusters as far as rook is concerned and and seth, so the they need to be connected if you're within a kubernetes cluster.

D

You don't need this, this level of replication, because seth is already replicating it within a single kubernetes cluster. This is really cross. Cluster replication.

D

Now and yeah. Well, I guess, while we're on that related topic, then so this you mentioned the stretch stretching a cluster or having an arbiter, so seth just um added support for this and well actually it's still, it's still an experimental mode, because this the ceph release that includes it, won't be for a few more months in the seth pacific release.

D

So the with stretch cluster.

D

You have two data centers that participate in the same kubernetes cluster, so you've got some form of failure domain within there, where but they're all in the same kubernetes cluster, and only two of the data centers can store data, but you need a third or tiebreaker zone so that the seth monitors can have that. Third, third, one to break the tie. If one data center goes down, you always need two ceph monitors up so that this you know the core of the set will keep running.

D

Kind of yeah that.

E

D

Not hard to describe in a nutshell.

B

Yeah so wait so we've got mirroring which can mirror between clusters, and then we have stretch which just which actually spreads your clustered storage across three or more clusters.

D

Well, stretch clusters: it's really one kubernetes cluster with the two with two data centers or two failure, domains managed in a special way, so that if one data center goes down, the other one is fully functional and still has like two replicas of two copies of the data.

D

So with stretch clusters, you actually have four copies of the data to do in each data center and the tiebreaker is just helping manage. If one of those data centers go down, it goes down.

B

Okay, so at this time the stretch cluster does not integrate with any of the sort of incarnations of kubernetes multi-cluster.

D

That's right, yeah, it's just a purely rook and stuff feature.

B

Okay, the, but I can do mirroring between different clusters and that's kind of the difference between the two features.

D

That's right: okay, yeah yeah, you with one with stretch cluster you have to be within the same kubernetes cluster mirroring, is across separate, completely separate kubernetes clusters, with their only relationship being that networking is allowed between.

E

B

uh Looks like you also got multi-support.

D

That's right so in let's see 1.3 last spring, we initially had experimental support for multis we've been working for it through a few details and fixes, and we just got the last fix merged yesterday, where it's looking like that, we really feel good about multis being able to be used as that network provider and also that also helps with communication across clusters too. If you've set it up.

D

It can help with internal the way ceph works internally too, so you can tell or tell stuff which cluster to use for the back-end network.

D

Yeah, so it's just all around good for networking and communication across clusters.

B

Okay, see yeah is rook ceph already working with dual stack support, that's kind of in beta and kubernetes, so.

D

No we've we've got single stack support for ipv6, but but right now we just assume we're in in single stack mode and we've been kind of waiting for kubernetes to settle on on the dual stack approach.

D

Well, last, I checked had been still changing a bit.

D

D

Playing any 1.5 features that I missed that you'd like to bring up.

C

I I don't think so. I think that was all the all the really big ones.

D

Yeah lots of other internal improvements, uh there's one like api extensions. We finally moved that to v1 from the v1 beta one.

D

So it's going to be gone soon,.

B

But that does mean that users may have to update their deployment files.

D

That's right and with our up each upgrade with each release that we have. We have an upgrade guide that says: here's here's how you deal with whatever might have changed and the yeah the api extensions change was a fairly big internal change. But our upgrade guide basically says here: just go apply these new crds and you'll be good.

B

Okay, do I need to apply those for an already running rook cluster rook ceph cluster, or do I only need to apply those when I'm going to deploy a new one.

D

So you have to have to create the crds anyway for new clusters, but upgraded clusters uh with with every release. Really we whatever changed in their release. We have to update them. In this case the crds all changed, and so we say, hey in the upgrade go go apply these new crds, which I mean internally kubernetes, was already converting them to v1, and so we don't expect existing crds during the upgrade to be impacted.

D

So it's more of a, I don't know almost a detail that you have to go upgrade them. It gives better schema, validation and things, and also, if for users who are using our helm, charts to deploy the helm chart just takes care of all that in the upgrade. Okay.

B

B

It's everything in 1.5! So what's next, what what did? What are you all working on um for rook, 1.6,.

D

1.6, that's a great question. You know with each major release I feel like okay, we got out the door and there's always more features from seth coming in I'd, say. The first thing on my mind next is that south pacific will be out in the springtime about the time we release 1.6, so pacific will be become fully supported, where the staff releases, of course, supported until they're out.

D

There will be new features, I'm going to forget about them on on the spot, but um yeah yeah blaine. What are your thoughts on other things? Coming.

C

um I mean this is something we don't usually talk about a whole lot, since it's kind of a boring non-feature detail. But I I'm excited about some of the ci changes that we I mean have had in 1.5 and also are, are continuing to to continue forward with in 1.6 we're hoping to be able to iterate more quickly and just be able to write like more and more and better tests. That way when we don't have to wait like an hour between ci runs, but can instead wait like 20 minutes instead and wait.

C

So what changes you making um the the ci has moved to uh like most of it runs using like github actions to run individual tests uh sort of one at a time, but in parallel, which has been has sped up our rci quite quite a lot and we're also looking to start doing ci for like nightly builds of both rook and ceph, um either on like a daily or or weekly basis. Just to get like good um integration.

C

Validation like between those two pieces make sure things are just like very like tightly integrated and that there aren't uh accidental gotchas that we end up figuring out like a week or two down the road.

B

Yeah, I I often wonder in kubernetes, if github actions had existed three years ago, whether we would have built all of the things that we.

D

Built yeah we're we're finding they make the developer life much much better in rookie at least a couple other feature areas that come to mind as well. Going forward is the you know the dr disaster recovery scenarios. Are, you know, they're complex scenarios? How do you get your application to fail over in a reliable way or testable way so that you know a lot on improving the dr automation around that, and so you know rook likewise, is you know that's one of our big areas of focus?

D

How can we make the mirroring and cross clusters better? How can we improve stretch clusters to make them even simpler or even more reliable, um so yeah, so the anything across clusters there's rgw, multi multi-site, which means you've, got object. Your object, storage that you want to replicate across clusters. Not just the you know. The block with rbd mirroring these ffs with the shared file system also will have a mirroring technology coming soon. I don't recall honestly, if that's specific or beyond,.

D

Yep and then continuing features also around just working well with kubernetes one one that we're almost done with is working with pod disruption budgets in a better way where you need to upgrade kubernetes. But you don't want to disrupt storage in the while you're doing your kubernetes upgrade.

D

So the pod disruption budgets will allow rook to control how those upgrades work. So if you upgrade kubernetes and the storage just working, because it'll work well with the failure domains that you've set up before.

B

Okay, I've. I have more questions about future work, but before we get to that, um barnes um who's just getting started with rook and ceph wants to know. If, if after deployment, only a single monitor pod starts and then the cluster stops there, what's the next step for troubleshooting.

D

Okay, if one pod one mon pod starts so yeah, because you need three to get going usually what happens is either there's a networking challenge that um well, let me back up so what the operator is trying to do is it start starts one month and then it tries to connect to it and validate it, and then it adds the second and the third one at each one.

D

So the operator apparently isn't able to connect to that first one, and so it just keeps retrying, so I'd say check for networking. If you're like, if you're in a bare metal cluster, there could be some firewall in a way.

D

Our commonly asked questions in our documentation that I'd suggest to look there or ask us in slack. That is definitely a common challenge to get past.

C

um Yeah on on the networking um there was uh there there's a kubernetes app just to kind of kind of smoke test. Your your networking and your cluster called cooper ring, which is also noted in the ceph channel of our slack of the rook slack.

C

And I've yeah I've found that useful for spinning up just to sanity check whether the kubernetes cluster is actually behaving as it's intended to do.

E

B

Okay, um the um thank you for that um we'll see if he has follow-up questions or or if they have follow questions. um The um one question for me um is going to be volume topology um and a rook ceph stack. That is, if I'm asking stuff of storage um the. um Ideally, I would want the application pod and the storage pod to be running as close together as possible um in in a in a network sense. You know in in the best of all possible worlds.

B

You know, particularly for applications that make a lot of storage, calls like databases um to actually have them on the same physical machine or the same vm. um Is there any planned work with volume, topology manager, etc to to make that a potential reality.

D

um There's not I'd, say we so steph is fundamentally based on distributing the storage across your cluster. That breaks it up and you know just basically shards it in in a lot of ways. They don't, except doesn't college charting, but it breaks up the data and puts it everywhere, and so, if, if you fundamentally, if applications need store, need the local data and they're doing their own level of replication, like many data databases like cassandra, then we'd recommend that you use a local pv. It's just a local volume and they manage it there.

D

You wouldn't want ceph in that case, to manage it because it it just fundamentally architecturally does not give you data affinity.

B

I I was actually thinking at a higher level, because the thing is, I don't run. I don't necessarily have mon pods running on all my machines.

B

If you follow me, and so, if I deploy a high storage requirement app, I kind of want that to land on a machine that does have a mon pod on it. Now I can do that right now, using affinity, but it feels like there should be something involving volume. Topology like this feels like a case that volume topology ought to address in some way, um but I don't have any way now to do that.

E

C

Tend to oh think of that, a little bit more with like networking, because stuff is so distributed. It's my it's my understanding.

C

It's really about access to the osds, especially for like block and file access that, um while I I think you could you know currently, we might use labels and use like uh pod affinities to be like ah put put this application on the same node or with affinity for a ceph like metadata server for the file system, or um you know, if I'm, if I'm doing block there, is no real metadata metadata server for it.

C

So as long as I have a like a big networking pipe to get to the osds, I I don't know if that's very different from just putting an application on up on a node with an osd on it anyway,.

D

And a couple, a couple more thoughts about stuff performance in general, so steph was really designed to work best when you have lots of requests in a distributed nature.

D

So if you have lots of clients accessing the storage, that's when you're going to maximize the throughput, really because there's no central point of failure, everything's distributed even the clients, but also even for high performance workloads like databases that performs really well when you just need to back stuff by high performance devices like nvme or at least ssd, so that whenever the request comes in, it's really quick to respond with the data from disk.

D

That's why I'd say if you need high performance, you back staff by by those high performance devices, yeah.

B

Okay, I I have more thoughts, questions on that, but we have questions from the audience um which I want to take first, so easy steezy wants to know what are some catalysts that you've seen for people migrating to a rook, ceph, kubernetes storage, solution from using regular storage classes and pvs in kubernetes?

B

This is primarily for high scale or are there benefits in smaller environments,.

D

Yeah, um let's have some some thoughts there. So if the primary use case where we created originally was really for bare metal, because when you're not in a cloud provider, there was no option for block object or file storage, and so this rook gives you that storage when you're in your own data center, that's the first and primary use case, but we're also finding a lot of people moving to rook storage in cloud providers because of some benefits like oh, we can put large volumes behind rook and seth to so.

D

You get the performance characteristics of large volumes in the cloud provider, but then you can create hundreds or thousands of small pv's for your applications.

D

So you get yeah good performance by basing on on top of rook in that sense, um and there yeah there's some other other things for different failover characteristics.

D

Like some cloud providers, don't always have fast failover times where, where seth really and rook have fast failover times to reattach your your volume somewhere else, if your pod fails um yeah. So that's that's. What comes to mind.

C

I uh one of the things that I I think about a lot.

C

There also is like I just from people that I have asked you know when I've met them at kubecon, and some of the responses are seem to kind of be around more of a multi-cloud sentiment where you know they might be using a cloud provider or multiple cloud providers, but they want to still like write their applications to like access the storage, the same way for simplicity um or some, some even the same cloud provider might not actually provide all types of storage in all regions.

C

uh So if you actually want to have you know, block storage or uh you know- object, storage in all regions, their their solution has been to use work for that for some users.

D

Right yeah that reminds me even had some users come to us and say: hey this cloud provider told us to come use rook because they don't provide that functionality like oh cool, yeah, user. It's consistent, it works everywhere everywhere. Kubernetes runs, work runs with storage.

B

Well, as the as the databases on kubernetes geek also mention one other benefit for small clusters, which is, if you're going to do storage based failover rather than database based failover for your database. Redundancy failover between ceph nodes for storage is way faster than failover for ebs.

D

Yeah, exactly that's that's what I was trying to mention before as well.

B

um Okay, so another question from claudio, which is a similar comparison question in this case, he wants to look at the comparison between running your storage inside the kubernetes cluster via rook and seth, versus having dedicated storage appliances like if you're buying a brand new cluster um uh and you're kidding it out right now, when would you decide to do choose one of those paths versus the other path.

D

So dedicated hardware versus use the same hardware for kubernetes and storage right, yeah.

D

Right, that's a good question. uh Blaine. Do you have any thoughts to start off.

C

I mean I, I guess my starting thought is that there there are so many little and big things that go into answering that um I don't know if I have any real rules of thumb, uh but I I definitely think that, like ceph uh and seth shines in like really having really large amounts of storage and in being able to scale out whenever you need it to, and so I think those are the cases when I would be thinking about rook versus if I'm like.

C

Well, I'm probably not going to need more than a few terabytes which I could just have in like a nest somewhere and have a like raid array on, uh but there there are also a lot of other considerations where, maybe maybe you want to use rook for for the kind of flexibility that it allows or just the ease of kind of the ease of deployment and installation.

D

Mode, we have also called external mode where, if you have multiple kubernetes clusters, you can manage the storage all in one of those clusters and then use our rooks external mode to for all the clusters to connect to that one storage, cluster, meaning you.

E

D

Have to manage storage in one place and everybody else consumes. It.

D

Yeah, the way that works is basically with this, the csi driver and each of the the clusters connects to that that one cluster, so that that could be an important consideration for centralized storage management.

B

So basically take kubernetes, rook and ceph and make your own dedicated storage appliance using those.

D

Right that that's one thing that you could do if, if honestly, if you just had two clusters where you want to dedicate one to rook and the other one to the rest of kubernetes I'd say I wouldn't make them separate kubernetes clusters, I might create one kubernetes cluster and then you can use node, selectors and other things in rook, so that you say, oh only run on these nodes. If you really want on certain nodes for high performance or other reasons, so I'd say, keep it with one cluster. Unless you really need.

D

B

Okay, the um I there's I've kind of seen arguments the other way, just because I'll say once again come from the database realm. You sometimes do wonky things with your kubernetes configuration in order to support storage and some of those wonky things will go away, though, right because once you guys support pod disruption budgets, then I no longer have to have a workaround for saying: hey, don't evict my storage pods, um the um um the um so in some ways you know the whole business of hey.

B

If you have a high storage cluster, it needs to be configured differently. That's going to go away in the future through improvements in kubernetes and improvements in in rook and the operator itself hasn't gone away. Quite yet so I could almost see having your high storage applications in you know, sort of a dedicated cluster of high storage applications and storage cluster and then another cluster. That's running, say your ephemeral applications there.

B

There could be real benefits to that um from an administrative point of view, rather than a performance necessarily point of view.

D

um Yeah and that's very fair people do have multiple rook clusters for reasons like that, it really comes down to you know application architecture, or how do you want to administer your storage as.

B

Well, yeah, so until we get another audience question one thing's going to follow up on so as you know, I've been doing some work on testing performance of database applications on top of cepheroc. The performance has actually been excellent, um the um even despite the fact that we are not hitting what should be sweet sweet spot right.

B

We have a single application running in, say three pods that are making all of the storage requests, um but I'm still actually able to get somewhere around 50 of the throughput that I would get on a single local pv, um which is you know, I'll. Tell you having had work with various distributed storage systems for the past 20 years is excellent.

B

um The um so, um but, like I said you know, this is a question of if my nodes are not homogeneous.

B

That is, if some have storage and some don't, um I kind of want to target those which was my whole question about eventually looking at integrating with volume topology.

B

Not so much for you know where we put rook and ceph, but more for where do we put the application that is heavily reliant on the storage um versus the other applications that maybe don't need any storage.

D

Yeah and rook does some things around topology already for sure like we, you know if you tell us, look for the local storage automatically or whatever device they're available. Well, we'll basically skip nodes where we don't see the storage and we'll just create it on the other nodes or we'll look for topology labels and set up your failure, domains based on the node topology labels.

B

B

B

Let's see other future work questions so there's various projects that are aiming to stretch your kubernetes network across, like say: multiple physical data, centers um submariner, for example, because I work with the submariner folks. Some.

B

um If you had one of those up and running, would I actually use that as part of my stretch configuration or would I be better off really focusing on stuff, that's native.

D

That's a good question, so you're asking: should you create one kubernetes cluster across the whole whole thing or create separate clusters with.

B

Well, assume assume I've already done that right, assume I've already used submariner in order to stretch my kubernetes cluster across multiple physical locations.

B

So the question is, if I want to um oh, I have an audience question just a minute after this, but let me finish this one up. um If I want to also effectively stretch my storage there um for redundancy reasons for availability reasons, should I then just tell rook to treat this as one big kubernetes cluster, or should I do something else, yeah.

D

If, if you've stretched kubernetes across the different domains, I'd say just have brooke take advantage of those domains that you'd you know, you'd need to label your nodes for different zones or regions or or other topologies.

D

But as long as you have three data centers that simple rook already works in that mode. Where you have three replicas say one in each of your different zones.

D

Our stretch mode now just assumes you have two different failure, domains where you'd want to stretch across. So I guess it depends on how many data centers effectively you have do you have research across two or three and yeah rook will and zepp will work with either of those configurations.

D

It keeps it definitely keeps administration or configuration simpler. If you can do that, instead of trying to use mirroring across independent kubernetes clusters,.

D

Yeah, so does that answer the question.

E

B

Sorry hold on, let me unmute myself. Yes, it does the um so I was trying to figure out this. um We have an audience question from uh debt. Conan who was asking? Is it possible to easily do a hyper-converged setup with rook I'm trying to find out what he means by or she what they mean by hyper-converged, um um because people use that in different contexts.

D

It's definitely overlooked, I would say: hyperconverge just means. Would you want to put storage inside your same kubernetes cluster on the same nodes with your applications and yeah? You absolutely can and lots of people are doing that and if you have specific performance requirements, that's when people tend to do something more custom, where you select certain nodes for for rook and keep your applications on different nodes, but overall there's nothing that that prevents it.

D

The other there is actually let's see. Is it xt4 I forget which file system you just can't use certain file systems for your your volumes, or else they don't work well with with ceph, if they're on the same node, but our default examples make sure we avoid that.

B

Cool okay yeah- that was that was their question. um In fact, that's how I personally run rooksef in production is on the on the same application nodes. Because- and this is one of your other sort of benefits for small clusters- is- I have a bunch of blades each of those blades comes factory supplied with half a terabyte of storage. If I don't store things on there, it's just wasted space.

D

Yeah, just as well use it yeah.

C

D

C

A benefit to uh to a situation like that also is to, to some degree like steph, does uh step? Does work really well, when it's a bigger cluster, like it's to my understanding, it's generally better to create a bigger cluster with smaller amounts of storage on each node than it is like to have a small number of nodes with a larger amount of storage.

C

Just because then, uh like you, you know when you're grabbing, like the pieces of files from all over, like you can make use of like the like large amounts of network bandwidth on each node.

B

Yes, have you seen the um have you seen the performance testing that sergey and I have been doing um I even with just five or six nodes for a large, sequential read. There is notable benefit.

D

For the six nodes, as opposed to three nodes, you're saying.

B

Six nodes, as opposed to local storage, all right as in as an even with a cluster as small as six nodes and a single client doing a large, sequential read, you still see benefit over having that read access distributed um across across multiple machines.

B

It's not a huge benefit, but you know it's like 25 faster, which considering you're getting redundancy in the bargain is pretty nice.

D

Absolutely yeah.

B

B

Okay, um well, we give people a chance to think of any last questions, um we're we're almost at the end of our time here. So any last thoughts, anything's coming up other things that are happening during kubecon that you want to mention. um Please share them now,.

D

We have a couple of rook talks in kubecon. Let me just remind myself what the schedule looks like, but friday afternoon we have our our diet talk for the maintainer track, um so it's at. Let's see where'd. He go friday at 3. 10 eastern time is that rook, deep dive and there's another one that involves rook.

D

That's tomorrow at 4, 50 eastern time, it's called whatever can go wrong, will go wrong related to storage and how stuff handles that that failure.

D

And otherwise, yeah for questions feel free to join the rook slack. We try to answer questions and be active there and we look forward to the feedback as.

B

B

Okay, well, um I'm just putting something in chat there. For our four question askers, I have just sent a link out of a chat um uh for you to request your complimentary rook shirts, um the um you know for participating in our office hours today.

B

um uh Please go ahead and follow that link um uh and you can go ahead and request that um if you are watching this later because you cut out in the middle, um you can ask for that link on kubecon slack and thanks everybody for watching and participating, and thank you most of all to our two rook maintainers, not only for answering questions participating today, but for um the awesome tool that is rook.

D

Thanks for having us, I'm happy to talk about rook anytime, all.

E

E