Ceph CDS Quincy, 12 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph Developer Summit Quincy: RBD

Description

00:00 - Dashboard and RBD
9:38 - NVME Status
12:25 - RBD Mirror Monitoring
28:48 - Volume groups for snap mirroring

35:24 - Promoethus Scale-out

Full agenda: https://pad.ceph.com/p/cds-quincy

A

So, let's see, we've got a number of topics and I know a little bit about some of them, so we're going to go on an adventure today. um It looks like the first thing on the agenda. Is the dashboard as with the other sessions, so are they here and want to talk about that see? Jeff.

B

A

ah There's, though, hi ernesto.

B

Here I am um today, I'm going to one is going to run the uh race and deliberation of uh dashboard demo so uh actually. uh Well, I I think rbd is probably the most mature component in in the dashboard, but currently has the three major pillars: the rvd images, the rv mirroring and also the iscsi.

B

So I will start showing arbite mirroring ice cast is hard to demo. I don't have the infrared to do that, so I will focus on rbd and reading mirroring.

B

B

Please let me know if you my screen when I start sharing it: okay,.

B

B

uh Did you see my screen.

A

B

Okay, perfect: uh do you need to increase the size, or is that okay.

A

A little bigger would probably help. Okay do that again.

B

Okay, I'm not sure how well uh that's what will behave with these changes in size, but that's right. So I'm not sure how familiar you are with the dashboard. But basically this is the uh rb section. We decided to go with the uh block uh a name instead of drbd, because it's more generic and less selfish, so so well, it was a popular vote in this case rather than the rpg enemy. So in this case we can see here the images namespaces. We also have the brush.

B

So basically you can create an rbd image like rbd.

B

You can also select. You have multiple rbd pools and also you have select a separate data pool. In this case I don't have any other posts, so I don't have the choice to do that. So um then you have all the different options if we want to use uh mirror- and I think I have to enable journalists- so I will do that right now and we'll have the stripping options and the qs options uh alphonse already shared, because the qos are available in different layers at full level, also on uh rbd image level.

B

I think it's also a snapshot level, so so there are multiple places where these settings can be configured. So in this case I can create the image so there it is. uh We have all the metadata of the image and also the configuration settings. These are the qs ones and the performance which is profile, but currently I don't have this enabled.

B

So if you have interest on have a look at that, I can mean that you know different media. So then you have. We have the options of for creating snapshots and well uh provides a predefined snapshot name. So that's it. Then we can have the specific operations for that. So we can protect that and then, when it's protected we can clone it. So that's it. uh We have namespace support as well. I remember jason mentioned that, but it was much demanded.

B

Is this kind of support? It's it's there just for compatibility and also, I think we support both b1 and v3 images. If we want legacy, support was recently added all right. It's there. So uh we found it was necessary for my gradients and some uh users had the v1 images and they were facing issues when migrating to e2.

B

So it's there. So that's mostly it about the um image section. If we go to the rv marine, I hope this is working. I just set up the. uh Let me check if it's I think it's not working now it should okay.

B

So uh there is this way for configuring, the uh rv mirroring with the uh bootstrap, so we can create the bootstrap basically and use that for importing that in the other uh in the other cluster. I think this is a really nice uh improvement and probably is the way that we should do also for other, I'm not sure if uh chef's mirroring is also supporting this or going to support this all right. I think it's a very nice way of doing that.

B

So now you don't see my other page with the other dashboard, but uh well, I'm basically pasting that token. uh Let me complete that, so that should be there well.

B

Okay, I have to enable this before.

B

And I'm using the pool because I don't have the images enabled yet so I guess that should start working now.

B

Well, that's basically it so once well, we have here the image. I don't have any image yet syncing that again.

B

And I think now it should be working well. There are some connectivities between those I'm I'm running um docker compose, so I have in these two chef clusters in uh docker images. um So basically this this will be the um look and feel of the rv mirroring section.

B

So uh yeah, I'm not sure what else is here we can configure we. We can also check uh in this case we have any images thinking there is no info here and all that same here so that could be mostly about the demo has been very short and then there is the iscsi section in this case. Well, you have to set up a an rgw, uh sorry and cephas cassie um gateway and all this stuff. I think cepheidian supports that right.

B

So it could be really easy to do that with the favn, and I guess that's mostly it so I I remember this one opened a few tracker issues for dashboard to expose rbd mirroring for snapshots and some other things is there anything else that we should start. I mean taking care of for rdd anything that you might hear missing.

A

uh That is what I'm familiar with. Is the snapshot mirroring it's kind of the big feature. That's gone in for pacific that I'm aware of illya. Do you know anything else.

C

um I was curious to see the uh performance uh tab, um wow yeah, because.

B

I can I can do that. I don't want to. I mean uh take too much time.

C

Yeah, no, no need to to do it here, just that. I anticipate some improvements uh on that front simply because we will be focusing on exposing uh performance metrics uh for capture based mirroring um in uh in quincy, and as part of that uh we should you know I should probably take a look at the uh look and feel of the uh like general performance, matrix performance tab and to sort of make them look the same uh for the mirrored images as well.

C

But that's yeah! That's that's in the future! So no no need to bother now.

B

Okay, okay, okay, perfect yeah, the performance is basically the grafana uh dashboard and we're also using the information from the red stats. So if you have that enabled it shows, it will show here in the in the overall performance and also in the rbd image. So here it will also show the specific stats for this already.

B

Okay, so if there are no other questions, I think well we're done with the demo.

B

Thank you very much for the feedback.

D

A

um Okay, so the next topic on the agenda is nvme over fabric status and next steps, uh which I think sage put on here.

A

um Oh yeah, not.

E

Expecting not expecting to present it, though.

A

E

I mean my my limited understanding here is that um we have like a skeleton repository or a.

E

F

E

The stuff I suggested, there's like a rest api.

A

For configuring.

E

A

Yes, that's about what exists, I think jason jason put it in, and I would have liked to look at it a little more before now or next week, but I haven't because I've been on vacation um so um for the rest of you, if you haven't, heard we're working on or there's a set nvme over fabric working group um which meets on, I don't remember, which day on tuesdays in the morning pacific time, um it's on the community calendar that we that we have published um and at the moment there the early thing is just sort of a we're setting up a gateway that um reads: live already on one side and speaks nvme over fabric out the other side.

A

Bottom on the ice cause. You want a bit.

C

A

Let's see, there's the repository or the, I guess, the branch that jason put together. um So that's the thing that will be seeing some work over the next several months. Probably.

G

Yeah, someone for my band was supposed also to join, and it's also we have contributing from intel and ibm, but also looking at the fabric, and we plan to use the spdk so we'll need some enhancements to the spdk implementation and probably play bd as a skill issue that we need to fix and maybe improve it in general yep, and we also need a discovery service.

G

That's because, underneath your fabric has a discovery, I put part of protocol discovery, so you can actually find out which targets you have in the cluster in which volumes name space it has or subsystems.

G

So that's something we'll need to implement as.

A

Well, um so next up on the agenda is rv, mirroring monitoring. I think this is ilias.

A

C

um Yeah, I guess I can uh talk about it briefly, uh so the uh the goal for uh quincy uh is to uh improve or basically establish uh some sort of unified way uh for uh for uh pushing metrics uh for our various mirroring solutions, and the primary targets are rvd snapshot based, mirroring and set of s also snapchat based mirroring.

C

uh These can probably be unified uh to to fairly large degree, uh and the set of uh you know high level metrics. As far as this thing is being mirrored. uh You know this thing is up to date or this much behind. uh You know how many snapshots are in flight. How many are still going to be sent?

C

uh Some kind of you know, general general bandwidth estimation, uh uh so things like that uh probably apply to uh both our b and c have a fast show, so we can uh come up with a hopefully unified with metadata and data uh scheme uh and uh present that uh as a set of uh permits, uh metrics that can then discrete uh by the dashboard, whether it's sub dashboard or something on the kubernetes side. uh The.

D

C

Manager can also hook into this. uh So uh that's that's the goal. uh The stretch goal is to also um incorporate uh rjw into this scheme. The uh rgw multi-site is obviously a very different beast uh and um uh you know to start with it's it's it's multi-way, so.

C

Sort of shuttling it into this unified um scheme that we can come up fairly easily for uh for rbds ffs rgw would probably not fit into it, but uh we should at least try and uh that's the overall direction that that that we're being uh that, basically people want uh on the kubernetes side, because the log file and object tvs are supposed to look the same and kind of being mirrored the same way at least again from the high level point of view.

C

But for for rbd and southwest it should be doable and that's the goal for pacific now. As far as this presents some challenges, particularly on the scalability side and on the manager side as well. uh Currently, the what we have is a manager module which exposes prometheus metrics that are being thrown at it through the uh osd performance, counters interface or say, would be.

C

The staff performance counters, get translated uh into uh permethries metrics uh and exposed uh uh through a single endpoint and uh at the uh at the scale requirements that we're looking at uh and also availability requirements that we're looking at. uh That is not going to work, uh we're already having issues with the uh with the manager uh basically being overloaded, and what once was uh a solution to uh you know solve the uh monitor, overload uh issues uh we now need to solve the manager overload issues.

C

Basically it's uh it's kind of a reincarnation of that, and what we're looking at is to have each mirroring demon uh present its own uh from your first endpoint and uh adopting uh all the cons all of the consumers, the dashboard uh and the kubernetes side of the house uh to scrape uh those multiple endpoints and collate them together and have that be have that happen on the on the blind side uh that, as far as far as implementation uh on the rbd side, uh we uh we only collect the necessary metrics or pretty much all of them.

C

There might be a couple that we need to be added, but uh the basic stuff, such as again how many snapshots are implied, how many uh snapshots going to be synced. Things like that um that that is already there, as well as some basic bandwidth estimation. So we can. We can take the difference between uh between the two snapshot counters instead of uh gauge the the the bandwidth from that.

C

um uh So uh on the rbd side, uh uh the what needs to be done is just you just needs to be exposed uh on the civilized mirror inside.

C

uh There are no metrics that that I'm aware of, um though uh uh comparable set of metrics uh would need to be implemented in the middle end first and then exposed- and I haven't looked at the rgw at all.

C

As far as the exposure part, uh the high level plan is come up with uh some sort of convenience, helper library, which could be linked to uh to each of the demons.

C

You know have a few methods that would send up the web server and present those metrics in in the perimeters format.

C

uh One thing that we gain from this apart from uh apart from solving the scalability problem, is that uh the current gateway from uh staff performance counters uh to communicate performance counters is limited uh in the sense that uh we cannot expose. uh We don't have support for exposing um histograms, and uh you know those types of more complicated metrics is basically just the simplest kind, which is a metric in the counter. uh That's that's what we do uh once we move away from the uh you know, performance counter bridge.

C

uh We can, and you know, have the raw data. We can actually expose those uh as well.

C

And uh yeah, then the goal is to handle that uh in that convenience library, um the web server part is uh um I'm thinking.

C

We should probably do it uh with beast uh simply because uh there is a uh library out there, a fairly established uh c plus plus library for exposing uh from his uh metrics, uh but uh it basically includes the vivid web web server as a sub module, and since we spend we've spent quite a long time, you know getting getting rid of it uh and in south uh bringing it back is uh probably uh probably not a good idea.

C

So the goal is to hopefully settle on beast, um but uh civic web and the existing libraries obviously option on how to.

C

I think that's about all. I have to say there are any questions. Please.

C

E

That a quick question on the um the prometheus endpoint part is that something that's going to be reusable across lots of different components.

C

Yes, that's the goal of the convenience library so, like I said it's something that can be linked uh into the rbd mirroring demon into this episode in daemon and hopefully rgw. If we, if we manage to uh get something done on that side, maybe not influence it, but in the future again the goal is to have a fairly minimal interface.

C

That would support all four types of uh you know, types of types of whatever they're called and uh be able to expose histograms and whatever else that we want to do.

H

Yeah earlier this week, we also talked about by trying to move the monitor, bothering a general out of the manager and source that, from all the uh versus directly to medius itself, they don't have everything funneling through the manager module. um Would this library be a good way to do that.

C

I'm sorry I'm actually on pto, so I wasn't in that discussion, but uh I imagine yes, if we're, if we're doing, uh if we're going to do a library that would basically be a web server that that that takes whatever is throwing at it on one side and presents uh a premier standpoint on the other side, this would obviously be a natural fit because we don't we don't want to do it twice.

H

Okay yeah- maybe maybe we can discuss more about this and later then um I imagine that we will need to go through the existing module and figure out what kinds of metrics are there that mean and where they need to be sourced from.

C

Yeah, currently it's uh like, I said it's very simple: it just takes. uh It just takes the simplest kind of osd performance counters and translates them to the simplest kind of you know prometheus again, I I'm going blank and what they're called, but there are four of them and we support only two and it's like one and a half to represents uh yeah so well.

C

The goal is to support all four uh and have it happen in a single uh in a single place that can be linked to um you know whatever it is that it wants to have an endpoint.

H

Cool yeah that makes.

A

Sense, I guess my my main question is: are we sure that the southwestern rvd mirroring is similar enough? That this will be convenient? um For instance, there was a discussion I like rdw.

A

I know you said it's different, but they were talking about like whether they wanted to prioritize hot or cold buckets for syncing, and I think rvd and some of us are both just sending the oldest snapshot that they have whenever they finish sending one snapshot, but I'm not sure if there are other issues like that that make it just a translation from oh, like snapshot status, more complicated to share.

C

Well, I I think these are sort of two orthogonal problems. uh One is uh the the library that presents an end point. That's obviously I mean we don't there. There is not. There is not a degree of freedom there, because the interface would have to mirror what from the page, does right. Okay, so it's it's a basic performance, counter interface. You want.

E

C

Yeah yeah, it's a basic performance, counter interface and you encode, whatever you want to encode right, so the library is obviously going to be reusable. There's no question about that whether or not uh coming like and now onto the onto the uh data and metadata scheme. uh For these things.

C

I I think that the file system and block metrics, given that both are based on snapshots and the ffs mirror demon, was uh the the implementation was uh from what I understand uh the uh the uh it was. It is basically based on the rbd implementation, uh to some extent at least, and the high level idea is very similar.

C

I mean we take a snapshot and you know we either are sink it or uh uh push it, uh or rather, in the our ugly case, we're actually pulling them, uh but the uh the again the uh the the set of counters, uh I imagine I imagine, would be pretty much the same, and so uh a unified data scheme makes a lot of sense to me there on the rgw side uh again with the buckets that that can actually be mirrored.

C

uh So it's basically a mesh as opposed to.

C

To to one-way street uh in the uh block and file cases um so yeah uh something doing something uh something coherent. uh There is probably a challenge and may not make sense in the end. um But that's like I said it's a strange goal and probably not for.

F

F

I think that the most interesting metric would be how much we are logging. Yes, yes,.

F

Is it possible to calculate ford's ffs but for urban mirror it's possible and probably it's already.

C

Yeah uh for for, I believe it's it's it's already there, it can be, uh it can be calculated.

C

I uh again, there are no metrics that uh there are no metrics for the supervised mirror that I'm aware of uh today, but without that metric uh exposing any you know, any other metrics doesn't make much sense, because that's the most important one that uh that that that will need to be there, and so we will need to uh implement it. Obviously, um but it should be.

C

It should be possible because again there's we just we don't have uh the mirroring solution that does not know how far behind it is, is to some extent worthless, because you cannot tell whether you have an enduring solution at that point or not.

A

A

All right there's no further questions about monitoring. Then michael, I think you're up next for volume groups on snap mirroring.

F

uh Actually, uh I I don't know who created that document.

A

F

G

A

G

Yeah so basically, as we snapchat best mirroring with like uh volume groups like we have for snapshots, so we can take the snapshot for a group. We'll have some a consistently group when we mirror- and I think that's it- I don't know if it's possible, I think it's possible. That is it's a snapshot.

F

Yes, actually, there was some decentralization here, probably just not aware, because I'm actually working on this project, so I'm just.

F

Added a link to the chat to my current uh vr, which is still working progress, but basically it's already provide something like described here.

F

It still needs much work, but it was reviewed by and discussed with jason and he was positive about it. I need to update it according to our discussion, not missing features and the difference with what this interface you provided in this document that you have commands for creating mirror group.

F

Yes, removing mirror groups. Actually we already have this conception of groups implemented and just they called consistency groups and already used for creating consistency, snapshots or group of images. So we just extending this functionality.

F

So, instead of adding mirror group, we enable or disable mirroring for or already existing group, and when you add so it's and after the mathematically enables mirroring for.

E

F

In the group and her belly mirror.

F

Mirrors this is, and currently it's going to support a snapshot based mirroring mode only probably in future. Someone will want what journal.

F

F

So yes and yes, the goal is that you can either so actually interface is the same that what we currently have per image, but we will be able to have the second commands, but for the group of image and you can create snapshot either.

F

The command line, interface manually or.

I

F

The smg module will.

F

Support this in in the future, it's not implemented yet, my pr, when you just add some scheduler all snapshots and they will be created automatically and rbiro, is mirroring.

F

Them basically, it's it. If you want to know some details of.

F

E

This is great news. I um this just came up a couple days ago in a meeting I think, related to um rook and kubernetes, I'm guessing. That was what prompted um orita at as well. So yes, a timely, timely edition.

F

Oh, it's doable for quince. I think if I have time to open this, hopefully.

E

I I guess a question for you are the: are the equivalent um uh concepts already present on the kubernetes side to do something like this.

G

uh There's uh there's a cap and I think that spec is alpha, which means it takes at least two kubernetes version to be finalized. If, but there are it's basically, there will be a volume group that you can take a snapshot and basic mirroring is not quite so they are talking about, but for mirroring for every listener.

E

I guess, presumably in that case it would, it would build on the existing group volume group function in the same way that we're already doing here.

E

E

A

All right so somebody's added a discussion about prometheus scale out. I don't know what this is.

H

Yeah, that was me, it was the same topic we just mentioned a little earlier, just trying to avoid spelling everything through the banjo module and recording everything directly yeah. So I.

E

Imagine this may require some.

H

Extra um work on the rvd client side.

A

A

Well, so sorry, what if this isn't the mirroring thing then? So what are you actually looking for here.

H

Not the mirroring, but the the uh whatever metrics are currently gathering and funneling through the prometheus module, um sending those directly from where they're being generated. I think for like for some things like a kernel rbd. This would be it could be handled by the generic um prometheus module for scraping, like metrics for block devices like like reads and writes. um I I have I o stats, um but for the rbd. I imagine that we might want to integrate an exporter directly. There.

C

Yeah this has come up in the uh in the context of kubernetes, uh but but not uh which, which is kind of which currently uses uh kernel rbd, uh at least for. uh If you're using sub csi, uh there is an option to use mbd, but then you call it is.

C

uh uh We obviously cannot just tie this to play my video because megabit needs to like exist somewhere, uh but uh in the case of nbd connect it can be tight assumed to be to the end of the uh to the rvdnb, uh since the instance of the demon um yeah.

A

Like like, we need a target, we can't have monitoring, scraping every client machine in the cluster directly.

C

Well, no, that's.

A

The networking being okay.

C

That's actually what we uh the the note level like taking care of uh kernel, ibd uh metrics via the uh node exporter, some kind of node exporter, whether it's the existing one. uh That would just be uh pointed at the uh you know that flash hybrid equal devices or something that we need to then we'll need to come up uh ourselves, uh whether on the step side of uh on the ocs side.

C

uh That's that's irrelevant, but uh that needs to be done on the client side, uh because the current metrics uh that are uh that are based on the um like randomized, sampling of or bloom filter, based sampling of uh uh of leds, uh that that does not present uh the uh full picture. uh You you don't the latency that you get from. It is not the way you can see that the client sees.

A

Sorry, I'm not saying the clients can't be involved, I'm saying that we can't be. I don't think we can be scraping directly from them. Like that's that's the network security problem. They they need to send it somewhere on connections that they open.

H

Isn't that exactly what the media's model is sending to the previous server.

E

It seems like it depends on what you're, what you're.

A

Right, but we need to create this endpoint like. Are we saying that we're going to have a generic set prometheus end point.

C

No, no so in the in the case of, for example, like taking the narrow case of our uh block, uh tvs, which occur in the kernel rve in kubernetes, which what I imagine you would end up with, is uh either an existing node exporter, which is uh which is already there and running on each node. uh Also taking care of uh rvd blob devices or uh something that we write, so it won't be.

C

An existing exported will be a new exporter, but it would still it would still run on each client node, that has both devices mapped uh and present the endpoint and then the dashboard or whatever it is. I can query that endpoint and uh so that they don't know that all nicely fits into the existing uh permitted model and the existing kubernetes model.

C

uh What we, what we're currently doing, uh is, or I'm not sure, if we're actually doing it, but what we could be doing with the existing bits is going to the manager and clearing the uh querying the stats that are uh being uh generated by the manager module uh which the way it does it. Is it installs a query on each of the usbs uh to send.

C

uh You know periodic uh that you know to send like badges, like snapshots of calendars, for uh objects uh based on prefixes uh based on object, prefixes for for each image and then those get uh collated in the manager module and present it. uh That's what the dashboard uh uh my nestor ernesto talked about: uh the performance tab and the uh hana uh thing that wasn't being displayed there. That's how it currently works. uh It uses that that manager module and uh to do.

D

C

Towards uh resolving on the manager uh scalability issues, we need to walk away from that as well, so one thing would be to not introduce new uh new metrics uh for mirroring and give it on the side and then, as a second step, we would need to do this, for you know in in the non-mirroring just general case. If uh here is an image, and uh I want to see.

C

It's you know the the the rbd top uh functionality, which would sort of.

E

C

To be switched away from the manager and into the.

A

Right, I'm not sure if we can do it before.

C

In the in the short term, but we can definitely do it in the kubernetes case, because rbd top is on is kind of you know the the user just executes a that's only by the instance. It just goes and talks to the manager so uh figuring out where the endpoint is is kind of an issue uh because right it doesn't need to the rbd talk command is, is not necessarily going to be run on the same note as the uh as you know, where the image is mapped or where the other dnb demon uh is.

C

This is being is being stood up, whereas in the kubernetes case, uh specifically in the kernel mbd case, I imagine that can be fairly easily taken care of by the existing node exporter, which can look at the io stat data, and it only does that for all kinds of web devices and uh the benefit of the kernel. One device that we get here is that it's only they are totally integrated. uh We don't need to do anything uh special like we would for for nbd.

C

At least if we want to make it work uh for rbd top as well, obviously the existing node exporter can just as well look at the you know mbd devices and uh do it in the kubernetes case. uh But again, looking at looking at the overall picture, uh we would need to uh eventually we would need to come up with something uh in the in the end. For this.

A

Okay, yeah, so like like talking to node exporter, makes sense. I just want to make sure that we're not saying we want to have random incoming connections querying library about its state, because I don't want library to turn into a server.

A

And then, like once, we do that. I'm also worried about the manager going out and directly talking to every server that has a rbd client of some kind on it like there are users of tens of thousands of those and if the manager's a scalability problem now I I don't see how this, how that works, we're gonna need intermediate like processes whose job it is is to do this. Metrics collection, right or not, collection, but collation.

H

So it doesn't go to the manager. That's that's the whole idea. You don't turn the stats to the manager. You send it directly to prometheus, which is designed to handle large-scale ingest of entries from many many different icons.

A

Right so I'm saying we're going to have a separate like prometheus collection of demons that run somewhere right, that handle pollation.

E

I I think it depends. I think there are different types of deployments that have different security models and requirements like, and I think in the situation that you're referring to greg, you might have like a centralized storage cluster with a bazillion clients right across an organization, um and in that case um you might not be have prometheus scraping, like all the client machines like across an entire campus or something like that. um But in the kubernetes case it certainly makes.

D

E

Because you already have prometheus and points on all the same nodes that are the clients like it's sort of an enclosed cluster, but it seems to me like sort of the distinction is here, is that seth isn't responsible for um necessarily ensuring that all of the node exporters are defined or deployed on all the same nodes like if you have a model where it makes sense, then you're already going to have node exporters on all the right machines, so you're going to get all these krbd bats already, um and if you don't, then, like you, just wouldn't get the benefit, or maybe we have a two mo bimodal thing.

E

I'm not really sure. If bilia are you suggesting that we would have both ways would work or that you would, these features would only be available if you have prometheus creeping.

C

And I think the first, the first step would be to do the primitive scraping part or for the kubernetes case basically and handle that, because currently uh I mean uh that's obviously the easiest case, uh and also on that. That would uh that would that would allow us to sort of offload that and then concentrate on solving the rbd uh top problem.

D

C

Is uh which is like rbd top is sort of designed to go to the single manager instance and get the metrics for all the images in the cluster or the top 50 or whatever right, uh and uh that that, like the way it's currently architected, uh is that it sort of means that it needs that central point to go to uh because uh it it it does.

C

It does lie in terms of the cleavancy, uh because that's not the latency that the that the actual client sees. um But uh it's still like it's still a useful thing for the administrator to go.

C

uh You know type a few characters and see what are the top 20 images uh on this cluster in this pool that are that are maybe causing issues or overloading or disease or whatever, uh and that use case uh I imagine would uh be based uh uh would remain uh tied to the manager uh to some degree unless uh at least for the christian future, because it really is like the the whole point of it was that uh we have this central uh point that we can talk to, that.

C

We can query and, uh following that is.

D

C

Going to be, you know, that's a much larger.

C

C

Perhaps, as a first step, what we can do there is just drastically uh drastically decrease uh the uh uh the frequency with which the metrics are being uh sent to the manager and collated. uh So we can make it like.

C

Even you know, you know we can go to something as as long as uh you know, two or five minute controls, uh because uh on a large scale, with the cluster seven issues waiting for five minutes to see the latest uh snapshot of uh harming the top uh data.

C

It's not going to be a problem, um so maybe, if the manager, if we don't come up with anything, better and manager, continues to be a problem, then uh as a kind of like shortage term solution, what we can do is once we once the kubernetes and other uh client-side.

C

You know consumers don't depend on the manager anymore. For the administrator use case, uh we can just have it happen once every five minutes or something like.

C

H

uh Yeah definitely sounds like a good term stuff. I think we can maybe think more in the future about how to how to do the rfd top piece. um I think that's good for now.

A

All right um somebody wanted to talk about exposing snapshots via rtw.

A

Nobody wants to claim credit for that. Okay. Well, uh there's a feature already where you can start um where you can clone an rvd image off of an rgw based image, um our base image, that's stored in rtw, and we want to go the other way.

A

I haven't looked a lot at the integration involved here, but I think it's probably just going to be like writing some rtw code that goes off and speaks rvd and exposes it as a bucket of some sort.

G

H

Yeah, I think it requires the zipper back end, essentially based on the discussion rjw used as yesterday. I'm not sure if it's in a stable enough position, where it's feasible to do this frequency, uh but that'd, be where we had it, how uh the way to run.

A

It sorry I kind of lost you there after you said you're, not sure it's stable enough for quincy.

H

The rgw zipper um framework yeah.

A

Yeah all right! Well, you know that'll be a nice thing to do when we can.

H

A

Okay, someone asked that we should move rvd suite to sep adm based and it's almost done. There's a pr apparently.

A

And then the last thing on the list was image, encryption, which I believe is from danny and orr, who are on the call.

I

All right, can you hear me.

A

I

Okay, great uh so we've been working um for pacific uh to add encryption to librbd, and so jason was our mentor. He was doing the code review and we actually got it in to pacific.

I

So what we have today in pacific uh is a lux based, deluxe format, encryption for rbd images, and so basically the user needs to set a password. If you want to use encryption, it has a password for this image and then, whenever he opens the image you need to apply the encryption load function to to have the actual load.

I

The encryption decryption to the I o I o, and so this is what we have for pacific, uh but we still are missing uh a relatively small uh piece of code and that we have a pr for and- and this will make this encryption feature much more useful- it will make it a differentiator uh uh feature so uh right now, what we have is uh when you create a volume and you encrypt it, then all clones.

I

So you, if you clone, uh do rbd clone on an encrypted image. It will have the same encryption on the on the child, and now we want to enable the user to have different different passphrases.

I

For the child and for the parent- and this is something that we already have uh pr and actually jason reviewed this, uh but it was his last day uh and we didn't manage to complete this code review.

I

So we are waiting to see who will take on uh the review for this uh pr, and so, and I think this is what I had to say, danny if you want to add anything.

D

uh Yeah, I am maybe just a few words as I say there is value in this, as is because a we have better performance. I think we cut down on one copy.

D

So just if you use chemo encryption versus lib rbd, encryption you're gaining like 20 on performance and if you move to lux 2, which is not supported by kimu, but that's the default today uh in dmcrypt, uh so just to get compatibility between images used with csi, which are doing uh dm clipped and locked lux2, and you get an extra 10 or 20 on performance above the luxon. So you get some benefits just for by using the rbd level encryption, but really to drive it through this clone uh feature.

D

We see this as if you have a lot of stock images uh running lying around and you want to clone them, but each one needs a new key, so the the parent can be unencrypted, but the child has to be encrypted, so every every bit of new data gets encrypted. So so that's kind of the use case and and as all that the state is, is pretty advanced with this.

A

That sounds awesome.

A

I'm I'm backfilling for jason, so I'll try and get this car soon. um I see that it touches qa task, so it but doesn't add any any into the suites anywhere, but maybe maybe it's already there uh that'll be my first question, for you is where it's tested um but yeah. This sounds like a really useful feature: client security.

E

I have one really quick question just to make sure I understand, because this is using the existing luxury format. Does that mean you can go back and forth between mapping the image unencrypted and then using dmcrypt on top of it and then or mapping it with the encryption in library and it'll work both ways.

I

So it it is compatible with the with dmcrypt. But if you with this new feature of different keys per clone, uh it's not. It's not looks compatible.

E

D

That makes sense. Okay, so.

E

D

Than that it's it's gonna be compatible, but we're working on adding some hacks. So at least theme clip will tell you, oh I I can't open this because it's it's been used differently, but other than that. If it's just flat images, then yeah you can move back and forth. That's super.

E

Slick cool, okay, thanks.

A

Well, that was the last topic on our agenda and I think we were scheduled to end in two minutes, but any other topics. People wanted to discuss for rbd in the upcoming.

F

Moments, I have the question for the this last topic. We have about this pr. Actually, I'm going to review it to some extent, at least, but as far as I see you had discussion with json.

F

Did you address alrighty because I don't see rapush.

I

So it's still not, uh there are still some uh points that jason raised that we want to discuss, uh and uh this is what we we said. We wanted to discuss it and perhaps uh best of luck.

F

Okay, I will look at.

F

A

All right, if there's nothing else, it sounds like. Maybe we have a couple hours off until uh stuff. This starts. Oh, wait. No sofa starts right now. Doesn't it no that's tomorrow? I can read time.

H

Yep, this will start tomorrow, 24 hours.

A

That's all I got thanks guys, uh josh any closing words. First stage.

H

Nope nope, thank you.

E

H

Thanks everybody yeah see you tomorrow.

F

G

G