Ceph Ceph Virtual 2022, 23 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: NVMe-over-Fabrics support for Ceph

Description

Presented by: Jonas Pfefferle
NVMe-over-Fabrics support for Ceph

NVMe-over-Fabrics (NVMeoF) is an open, widely adopted, defacto standard in high performance remote block storage access. More and more storage vendors are introducing NVMeoF target support, with hardware offloads both for NVMeoF targets and initiators. Ceph does not support the NVMeoF protocol for block storage access; its clients use the Ceph RADOS protocol to access RBD images for good reason: RADOS is a distributed m-to-n protocol that provides reliable object access to sharded and replicated (or erasure coded) Ceph storage. However, there are good reasons to enable NVMeoF for Ceph: to enable its use in datacenters that are already utilizing storage hardware that offload NVMeoF capabilities, and to allow existing NVMeoF storage users to easily migrate to Ceph. In this talk we present our effort to integrate a native NVMeoF target for Ceph RBD. We discuss some of the challenges of implementing such a support for Ceph including subsystem/namespace discovery, multi-pathing for fault tolerance and performance, authentication and access control (e.g., namespace masking). Furthermore, we describe how the NVMeoF target design can be extended to allow reducing additional network hops by leveraging the Ceph CRUSH algorithm (ADNN).

Event link: https://ceph.io/en/community/events/2022/ceph-virtual/

A

So yeah, as Mike said, I'm going to talk about nvme over fabric support for Seth and, as you can see, there are tons of people uh on this slide. Obviously, there's a community effort. uh Multiple companies are involved in making this happen and the two people that are on the line- uh I'm, not special in any way. It's just me and Scott who are presenting this talk all right, then, let's Dive In, okay, so why do you want to want Envy me over fabric support? In the first place?

A

Right I mean we are talking uh block uh device support here, so you have RBD right. You can access it through either user space slip, RBD or in kernel krbd. You can just Mount your block device from self and you get all these nice features right in the background. It's the routers protocol, it's reliable, it charted replicated. You can even do Erasure coding right all these kind of fancy features.

A

So why would you want to replace this with something else right so, um first of all, enemy over Fabrics is an industry standard right, that's widely used anywhere, and that might be the main reason you want to use it because maybe part of your ecosystem is already using nvme over Fabrics or you have clients that only support and removal Fabrics or you want to take advantage of offloading in uh like dpus that are available from various vendors right that typically have enemy over fabric offloading support which might not have support for offloading rather's protocol or anything in in that respect.

A

Okay. So how do we do this? So there's a very high level picture so on the very left you have a bunch of nvme initiators or clients, then they are talking. Envy mirror fabric, one of the uh different protocols that are supported right, so uh TCP, RDMA, fiber channel, you name it and then in the middle. We have this component that we are building which we call the Seth Envy new over fabric Gateway, and then we talk routers to the Seth cluster, which has a bunch of monitors, managers or SDS, and so on.

A

Right and obviously you can either run this Gateway on a separate node or you can run it co-located to one of the osds and we are, we will get into how efficient this is later in the talk. Scott is going to talk about this okay, so short, nbme introduction. So we are all set on the naming convention right. So if you look at again from left, we have the initiators.

A

Then we have the Gateway and now zoomed in a bit on the Gateway, and they gave you essentially on the one side is an Envy new fabric Target that exports uh subsystems and subsystems is just a way to group namespaces, which are volumes in nvme terms and in our Gateway.

A

Those namespaces has have a one-to-one mapping to a image in Seth to an RBD image instead right and then, when an initiator like, for instance, here initiator a connects to the subsystem, it gets a controller and a controller in the in the Target is essentially responsible for issuing commands against the namespace, which are then issued against the RBD image and and USF cluster right and by the way.

A

um If you have any questions, just put them in the chat, I have the chat, open, so I hope I can just answer them on the Fly, all right so jumping in how do we do this? How how can we configure this whole thing right? What we do is um we have a control demon running um as part of the Gateway is a little python process that exports a grpc API, which you can secure via mtns, and you either can write your own RPC client.

A

So we give you all the bindings for grpc or we also provide a CLI if you quickly just want to configure something um um to grow, and so how does this work? Essentially you, you issue a command via RPC by the grpc, and then um the data path is is set up in this way. So you you basically what you do is you say: I want this RBD image to be exported under this subsystem and under this name, space, right and and what these folks have access to it, and this is the IP port.

A

I want them to have access to to this subsystem.

A

All right so then one other thing that we do is um obviously we want the conflict to be persistent. So whenever you change a conflict via the trpc API, the um toward the config in uh Seth, so what we do is uh for those of you familiar. We store them in a in a theft, object or map, um and that way, whenever a Gateway is shut down and restarted, we can restore the conflict right.

A

So you yeah, all your configuration is saved and you can specify- um and basically you can form groups of gateways I come to this later and that determines the name of this config object and you can determine which pool you want. This config object to live in right. Then, let's go one further. So now, how do we do the data pass part right? So that's important stuff.

A

um So when you issue, uh when you configure via the trpc to the control Daemon, we actually issue there's another process running which takes care of the data path and we use spdk for that. This is the storage platform development kit. It's a widely used open source project. It has been around for years. It has been proven it's used in production, basically elsewhere right, so that was kind of the choice we take.

A

So we didn't want to obviously write something like this from scratch um and spdk has a subtraction so for what you can you can create and in your fabric targets and on the other side they have what they call a block device abstraction where you can plug various block devices underneath it and one one of the plug device abstractions they have, which we extended a bit, is the RBD block device, abstraction and essentially, once the spdk process is running, you can do RPC requests against it, which the control Daemon in this case does and configures it in a way um that I said before right, basically saying hey: this is a substances I want to create.

A

This is the RBD image I want to link um two to a namespace all right, then, let's do a little demo. So what I showed you uh is basically already available. So you can, you can go to this link clone it follow the readme and you should be able to set up um a nvme over fabric set Gateway and we are going to quickly do this to. Let me see all right so so what do we have here? So on the very top we have the nvme or app initiator.

A

um So this is just a machine called flexo3, nothing special right! um Then we have I have a window where we will issue the CLI commands. um Then I have a window where we will run the actual Gateway. So the Gateway here will basically start the python process at the python process and we'll start the spdk process in the background right so um and then on at the very right bottom. Here we have the the safe cluster. So let's just see this is free. Note little self cluster. It had two pools. We already have a.

A

uh We already have some images set up in this RPD pool, which is which I call test image 0 to 11. all right. So then, let's start starting the the Gateway, that's simple! It's just starting it um all right that was created and then um we can start issuing commands again. So this is the the CLI that we have so we have a help. These are the various commands we currently support.

A

um So, let's start with, what we want to do is create a block device where they told you. This is the abstraction that is used internally in spdk, so we basically tell it.

A

um We want to block device of test image 0 that is already in set um in the RBD pool, and we want the name of this block device. B70.

A

Okay, let's create this, has been created down here. You are, as you see, that there was some action. um Okay, then, what we want to do next is: let's create a subsystem.

A

So this creates a subsystem and then you need a subsystem, uh nvb qualified name which, which is this one, and then you also can specify a serial number which I just put as Seth Gateway One. Okay done, then, let's add a names. So now we are adding the namespace um and to add a namespace. Now is essentially where you do the mapping right, so we're telling it. We want the new namespace to a subsystem, and this is the block device that we created earlier, that you should map to right.

A

Okay done so the only thing left is um we need to allow who can have access and for now, let's just say: okay, the star here means everyone can have access to this uh to the subsystem. Let's do this on here. It says to allow any host access right, it's good and then last thing we need to do.

A

We need to say where you can access this namespace, so we create a listener um on on this port and there we go so now. Let's see now, we should be able to connect from here.

A

There's the stop system, and here and missing.

A

Yep, okay, I need to use sudo all right and then, let's see if everything worked out. Yes, so we see here this this, the first one is the local, and then you realize this, the me device which is connected to- and this is the serial number I said right- set Gateway One uh is spdk PDF control and this can also be changed um and um yeah. This is the image that we want, so we can also check here um the info of this image right.

A

uh This is the image here all right and then we can do. Are you against it? Let's just do some.

A

Oh before we do it, let's, let's uh the IELTS- that on the on the safe mode, so we can actually see that we are actually doing IO against.

A

Okay, it takes some time to be clean.

A

And maybe I'm wondering.

A

This is the joy of the devilities how it works. That's just yeah, okay, I assume I'm on the wrong.

A

Yes, okay, okay, but you've seen that we actually did IO against um this. The image here from here right through the nvme mapping that that we set up. So that's basically all I wanted to show you. This is already there um in in the rainbow. So you can download this today and try it out.

A

Okay, let's go back to the presentation, so I also put the commands in the demo, and so maybe upload the slides. Really, you can also try it out for yourself, okay, so this would already, what's already there now um I'm going to talk a bit about um what we want to do or what we already started. Doing specifically, this one is a big item um which is Gateway groups or multi-parting support, and um here we already have a PR um in in the repository that implements this there.

A

There needs to be some details, um figured out and and more testing needs to happen, um but we are already on the way to to implement this, and essentially the idea here is obviously first thing you want. Is you get reliability from your uh rados protocol? Typically right, when you do RBD, you have faults or it's all. This sort of thing right, but if you then switch to Envy, mirror Fabrics. This is um by by definition, point to point protocol. You you, basically you lose this um ability, so you need to reintroduce it right.

A

So what do we do? We want multi-parting support to be fault, tolerant, and how do we do this? We have a concept called a Gateway grouping and a Gateway group is essentially just multiple gateways that share the same configuration. So they export the same subsystem. They export the same mappings of namespaces and RBD image, and here we still have our RPC client and the idea here is not that you have to do that. You have to configure each of them individually.

A

I just draw these slides to each of them, because the ideas you can do you can do rpcs to any of the Gateway in a group, and it will automatically also configure all the others in in the group right.

A

um So that means, if you don't have there's no leader or anything like that in a Gateway group, um everything is basically wherever you configure something the other nodes will be respectively, updated and and then, of course, you can also add new gateways to a group. It will just pick up the config and configure itself the same way um as the as the other node.

A

So one thing we we would also support, is more advanced. uh Nvme over fabric features is what, for instance, what is called here: um asymmetric namespace access. That means you can basically tell the initiator here which path it should be fair, uh which path. It should not use things like that right and the idea is that you basically can tell through the RPC client um when you, when you configure, you can say, hey I want Gateway One to be the optimized path.

A

So all the initiators should prepare this over all the other gateways right and once this fails, they will just pick one of the other parts. um So so this is also what we are working on.

A

um Spdk already has support for this asymmetric namespace access, we kind of have to put it into the configuration, make sure that it works accordingly right. So this is the whole multi-passing concept and then the next thing we want to add is Discovery, um because I mean, of course you can just tell your initiator um here's this IP to pay path.

A

A I have to path B right whenever you're at the Gateway you you manually, go to every one of them and tell them, but instead what what is part of the nvme spec is that you can actually um have an nvme standard, Discovery servers which basically an initiator can connect to, and then an initiator can get a discovery lock, and the discovery lock here shown in yellow basically contains all the paths to a particular subsystem, and then it can connect to it. And, um moreover, is that there's also an event mechanism.

A

So whenever A New Path is added or path is removed, it will actually tell the initiator um that that that happened right. Basically, you get.

B

A

Pass notification change or Discovery lock change, it's called, and then you can get Discovery lock again and see what what changed in in in path right and um so as.

B

A

On the left side, this is the standard mechanism and we were very Discovery service. um We envisioned this to be an extra component. It's not part of the Gateway itself, because we want to be able to independently scale those Discovery services, and the idea is that you can, for instance, have one Discovery service per a Gateway group, or you can have multiple Discovery service per Gateway group.

A

If you want fault to runs or you can have a discovery service that um does discovery for multiple Gateway groups right, so you can independently scale them and essentially what the discovery service does. Is it just again reads this config, which is synced between all of these entities and make sure that um and basically checks where New Paths are added to the config and then a non-system accordingly, so that that's for Discovery, then a next big topic is obviously okay.

A

How do you do authentication and encryption so in the initial Target is for us to support something like ipsec tunnels and I'm, going to explain a bit. Why I say initial? So what you've seen before in the demo is I told you that I had this.

A

uh We allow any host to connect, but that's that's just basically a very simple string check that is done in spdk in whenever host connects in nvme or fabric, it has to provide a a host, mvme, qualified name and basically, what you can do is you can limit access on this whole enemy, qualified name? The problem is that, obviously, that is not secure right, there's no crypto in it.

A

So what you really want is you want to authenticate on something um best case scenario, you'd identicate or dedicate on nvme credentials, and that's why I said initially, we only do ipsec, because there is something in the end we meet specification that was recently added called nvme embed identification which allows you to do exactly what I just said: connect against or dedicate against, and we meet credentials like a host and qn and a subsystem NQ, and these kind of things.

A

Unfortunately, at least to my knowledge, there's no implementation of this out there, yet neither in the kernel nor in SDK or any library that I know of at least so as an interim solution. What what with the Envision? What you can do is essentially just have a means through the rpcs setup, um basically tell it when you create a listener. A I also want this listener to set up ipsec, and these are the credentials that that are needed. I don't know certificates, you name it whatever.

A

You need to set up IPC, and then um this is not all spec out, but something like we have like a plugin architecture. For instance, there could be something like a transform plugin behind it, which then would set up uh the all the necessary things for ipsec such that initiated. They can can connect via IPC by an ipsec encrypted and authenticated tunnel um to to a gateway.

A

Okay, there were some questions in the in the chat, so one question was about Discovery. Maybe let's quickly.

A

Talk about this so, um to be honest, I'm, not sure about multi-pass D, um so I just know that when you, when you do when you connect, when you connect to multiple paths um with the Linux color right, it automatically sets it up for uh for multi-pathing um to be honest, I have not looked into how how Discovery Works um with Linux, but by assumption it it it. It works because I mean there's a standard mechanism right. That is not something we invented, but that's something we would need to double check right.

B

A

Then there was another question: what about wireguards is that uh besides IPC yeah, definitely it would be an option, as I said what I Envision is that we have kind of a plug-in architecture.

A

um So when you create a listener, you can basically say I want this to be secured via X, and then this particular plugin um sets up whatever is needed to secure that that particular Channel or creates a VPN or or whatever you name it right, so so, certainly that that could be an option.

A

Hope this kind of answers, the question all right, then I'll hand it over to Scott to talk about adnn.

B

Thanks Jonas, um so so one of the next things we want to this it'll um have to be down the uh down the road because at 80 and then support in the Gateway requires of several of the other features you saw that are that are in development.

B

um The basic idea of adnn- and this is the thing that Intel's been developing for a while and is now um working to standardize with some of our partners. It is a uh it's an extension to Envy member fabrics for distributed storage systems and ceph is one of the example.

B

um You know distributed short systems that it's designed for actually so um so.

A

B

Here is you can see we've um in adnn we will will have a a Gateway group. uh You know we've in this diagram. We have. We have four OSD nodes and- and we have four gateways in the Gateway group. So that's the first thing there is. um The notion of a Gateway group doesn't have to be just h a pairs they could ultimately they'll be able to be larger.

B

um So for adnn we uh we we eliminate that forwarding, hop and in fact we eliminate a dedicated Gateway node, at least for at least for those hosts by by moving those Gateway um demons into the OSD nodes, and the this is the extension part um we enable each host to so hosts will now see. They'll have you know four paths to the subsystem that each path will go to it to directly to an OSD node?

B

um We we give post what we call a hint that enables them to uh choose the controller to send e each of their Fabrics iOS to such that. That controller is in the same node as the OSD that contains the lbas that are addressed by that nvmika. Now that's a mouthful, but essentially um it does uh in the ceph case, 80 never enable the host to do just the bare minimum top level. Part of of the ceph crush, just just the passion, mod and and look it up in a in a hash table.

B

It doesn't do the rest of crush so there's. So this is a lot lighter weight in hosts um and they get away without doing the rest of crush, because the I o gets to to a Gateway node and it will almost always be in the right node and then we still use the normal live RBD data path there um to make sure that was actually the right node for it um and usually it will be, um and um so you might wonder at this point, how can you do that? I?

B

Don't believe this works, those it'll be reasonable things I would refer you to this Earl at the bottom of the page where, because it takes another hour to explain exactly how it works and how we prove that it worked, but it's there in that presentation from a couple of years ago.

B

um So so the gist of this there is is that uh you would then, with a lot of hosts, at least the adnn capable hosts you.

B

Would um you get your iOS to your storage nodes in one hop all that work of that Gateway uh handling the Gateway stuff would be distributed over a lot of those OSD nodes, and a lot of the work would just be avoided because the rados traffic is all local you're, not sending it back out again, um um let's see so this also, we should point out um uh we'll leverage a property of of the discovery service, um which um kind of going off script here a little bit, but um a cluster.

B

A soft cluster can have more than one Gateway group, because not all of your hosts may have adnn support, so you could conceivably deploy a Gateway group like this embedded in the OSD nodes for your adnn hosts, and if you have a lot of hosts that don't support adnn and you don't want to handle the traffic inside these OSD nodes of the Legacy hosts that are going to send the I o to the wrong place.

B

You can deploy a second Gateway group for them and then your Discovery service um can uh this is this: is the attractiveness of part of the attractiveness of the in-band authentication so that these Envy mirror Fabrics entities can know the identities and uh and trust each other? So no I'm, you know find the subsystem on. This is definitely the host I'm talking to. uh That applies to the Discovery controller too, which can customize what it sends back based on who's asking.

B

So if, uh if on the list of hosts that are supposed to see certain volumes, the configuration says you should these. These hosts should use 80 and then the discovery service would reveal this set of gateways to them. If they were on the other list, they would reveal the gateways in the other, um in the dedicated um Gateway group, just as an example scenario um and I mentioned uh uh intelling, some Partners, including IBM, um are interested in in getting this standardized.

B

um If you work for a company, that's interested in um Canadian and.

A

B

Happens to be an NVM, Express, Consortium member, then you can learn more.

A

B

Going to your technical work group um website there, but the nature of NVM Express work is that uh you know we don't it's not discussed in public until it's ratified, so you won't hear us say anymore about that process unless you're in the Consortium, that's it. For that slide. We have the status. One am I doing this. One I think that was what we decided.

B

um Jonah showed you some of it works now um are um the goal we set out to have uh was um to stand up a minimum viable product as soon as we could and and so that will be um happening soon, coincident with the The Reef release.

B

um So you know air quotes are on product right, so uh this uh this Gateway um you can, you can deploy it. You can send traffic to it. um It's uh it won't initially um support necessarily support Gateway grouping. This is that's not a thing for sure we can get done for that, um but you will be able to configure it shut it down, bring it up. Have it have the same configuration um and uh you know and see how and it's it's performance.

B

So this is one of the things that consumed a lot of time. um Its performance should be um decent. It should you should get.

B

um You know close to the uh to the level of performance we set out to deliver, which is you know, comparable to what you would have gotten from live RBD in the same um configuration uh the long pull there is um automated integration testing, which is one of the reasons that the other things are um listed here is uncertain, because these, um even if they, if we didn't, have a any uh implementation issues with Gateway grouping or Discovery service, um we still have to figure out how to automate the integration testing of it in there a little bit complicated.

B

So um so that would be the next bundle of things.

A

B

If, if grouping and in a discovery service can be completed, and we can prove they work by the first out, then we would likely you know we would like to roll them in, because um uh you know this thing is probably not uh suitable for production use until it has um certainly some kind of failover and um and authentication and um encryption of data in Flight, somehow, whether that's with the future in band off and TLS um on the nvme or TCP connection, or whether that's through an encrypted tunnel of some kind.

B

um Certain transports uh RDMA, if you're, going to encrypt if you're going to secure those in encrypted tunnel is apparently the only way. So so we will have support.

B

You know long term for some um some at least one encrypted tunnel mechanism, probably ipsec, because that's where the demand has gone um and of course adnn is on the future list. Here. This.

A

B

Definitely past the first release, because uh for a number of reasons one is the stacks. On top of all these other features you have to have Gateway group, you have to have a a probably have to have a a working, um and the reason for that is so that support for legacy clients is with is more seamless that way for for adnn and of course, adnn is not an nvme standard at this point, so we couldn't um call that a production feature unless it was standard.

B

So uh that's uh so that's the basic plan so fuzziness there, and these are um the code and the uh and the central ether pad for for the project, which has our our ongoing status. uh Links to I think you'll find there, interestingly, links to the the goals that the group set for this project.

B

um um Initially um these, uh it's probably about time that we revised these and refine them based on what will in what order we'll start doing these things um and some design notes for some of the things um I think Jonas. Anything I left out there.

A

No I just wanted to say that here in the etherpet there's also a date and time of the weekly Community Meetup, where we discuss um and all these things.

B

A

There's a typo in the link because I did a replace all.

A

Yeah, thank you for pointing it out in the comment. So uh actually the link is actually without. This uh should be the actual link. No sorry about that.

B

That was the global replace to be consistent with how mg.

A

B

Spelled in the specs and all.

A

Right right so um Scott, there was a question uh regarding 80 and then um in in the chat from Matthew it was. It would be nice to remove lip RBD Raiders from at least the fast path cases with the adna set up. We did not it.

B

Would um what you mean you mean in the Gateway and today in public? That's all I'm gonna say about that. um So stay tuned I mean uh Oh. You mean Cinder, though The Block store for uh um no specific plans.

B

um Yeah, that's a different project right. So so that would be uh right. So that would be an open source effort in in Cinder. So if it uses nvme or Fabrics um the initiator, then that's plausible and the question would be well. What initiator does it use and I?

B

Don't know the answer that question so if it happens to be spdk, um you'd be golden, because the first public reference implementation of of adnn for hosts will be in spek, in fact, there's an old reference implementation to that now, if you follow that link back there and that um that's sort of what it will end up, looking like that reference implementation basically informed us about how it should really work in terms of being standardizable and working with um within vme. But uh as far as uh you know, Linux support uh hypothetical Linux support.

B

I have to say at this point: everyone thinks it would be a great idea if Linux the Linux host supported adnn, and um that, of course, will also depend on standardization. And um uh what's this question about desegregating osds using nvm4, that's a totally different topic. I would refer you to uh yeah. You see yeah the 2018.

B

um I work with the one of those Engineers that uh that gave that presentation and uh again a separate thing. It's uh architecturally. It would be separate from for from this. You, you know this.

B

um um What you see here would would not change, um except in some hypothetical future, where the Gateway was more deeply integrated with the OSD process, but that's out of scope for for now we're just concentrating on getting the thing available as a package delivering uh services.

A

B

A level that you could use in production- and uh you know that we could prove it works and it would be full featured in terms of Discovery and All Transport Security. um Those would be follow on things.

B

um Anything else are we for time.

A

I think we are right on time. Yep.

B

All right! Well with that and hand it back to you Jonas, for the wrap.

A

Yeah I mean I have nothing more to say other than if there's any other questions.

A

I think there's there was one common I find it as a limitation for one RBD image. I think that's related to main thread for one RBD, um I um I think we discussed this in the GitHub issue. So there were some issues in the beginning with performance because of the way that uh all all RBD uh requests were funded through a single threat in in spdk.

A

um This has been fixed to some extent and I I, actually.

B

A

Some performance numbers um that actually showed that, for instance, we have 60 kilobytes uh iops performance. We can reach 100 Gig line speed, so we can. We can drive like uh six, seven hundred thousand uh ifs or something like that, um so I think performance should be decent and this particular test was with 128 volumes, uh each of them pushing. I o to the Gateway. Obviously there's much more testing to be done, and if you find any other bottlenecks in spdk, we will try to solve them and make sure um that we get away with.

B

This obviously.

A

um It doesn't come for free right, the Gateway needs resources. So this translation layer needs a CPU power so that you have to deal with and also need some memory, of course, that that is just the truth of the whole story. But if you have, if you spend the CPU Cycles I think you can get fairly decent performance out of it.

B

That resource utilization is interesting um uh in the adnn case that we have to admit that, um depending on yours, because ceft clusters are all you know, snowflakes right there, everybody builds them differently. They all have different requirements, different needs. So if your OSD nodes are were built with just enough CPU to do their OSD work, then you might not have enough space or compute left over to put an embedded Gateway in there.

B

So dedicated gateways may be your only choice to add to some existing large clusters, but we're hoping that there's enough medium-sized ones that have more compute than they really need and they're OSD nodes and um and that it makes sense for those users to basically shift that load out of the host into those nodes and not deploy separate gateways. So but it's a science project right. We, uh we won't know how it works in the fields until it can be used in the field.

B

So that's why that's the emphasis on getting something stable and available and completing the features later.

B

We use several spdk threads for one RBD image, you just sort. um That could mean several things.

B

um Yes, and uh if you look at the uh the details of the in the project, we have a we sort of maintain a a set of spdk patches that we have to have to uh for for the spdk platform that we base the Gateway on, and one of them is related specifically to that.

B

um It's um that patch needs a little bit of work right now, but um it's it's it's basic uh job is to is to spread the um live, RBD work across all the reactor threads, so you don't get a hot spot.

B

So I think we could just summarize that answer as yes, so all right now we're definitely over time.

A

All right, thanks again.