Ceph Developer Monthly, 5 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph Developer Monthly 2022-10-05

Description

Join us on the first Wednesday of every month for the Ceph Developer Monthly meeting

https://tracker.ceph.com/projects/ceph/wiki/Planning

A

All right, it looks like we've got a good crowd here. I'd say we can get started so hi. Everyone welcome to the October stuff developer monthly.

A

um My name is Laura I'm on the rados team, at red hat and I'll be leading the meeting today. um We've got a a nice selection of topics today, so uh the the agenda. We did this a couple cdms ago, but uh we've divided the agenda into different categories uh depending on what the presentation will be about. So this first category is about uh usability and enhancing the stuff user experience um and uh I hope. The the speaker who put this first topic on is here today.

A

um The first topic is a blanket Mount and stuff containers is uh the person here who put this topic on.

B

um It was me who, who added that topic, but.

A

B

I, don't actually have much to uh kind of say on that. uh It was um mostly to um get Adam and perhaps good Elmer to um kind of go over this and the reasons for uh having uh this. This.

B

uh What I call a blanket I'll go to fast mount for safe containers because, as uh as the link like there's a link to the sub develop thread uh there uh and uh I've I've linked the the exact message that uh kind of fully describes the the problem, uh the fact that uh the self-container project and basically all self-container self-containers, do this. uh This includes both the like actual containers for actual demons, so, for example, a container for the USD or for the monitor, as well as the safadm shell container.

B

uh The fact that uh that the fact that all of those um uh have a bind mount for the entire uh root file system means that uh uh there are certain behaviors that are pretty hard to debug. In particular, uh this user ran into an issue with unmapping uh kernel RBD devices. uh The kernel driver was denying the unmapped due to the device being in use, and that was due to the amount being pinned uh by by the running container uh and it's kind of subtle, because uh it's only a few.

B

uh It's only if you start the container uh after you've mounted something on the host and then later, if you unmounted that, but the container is still running you run into this this issue, and it took uh uh you know quite some time to debug and the uh the original poster can do it on their own. uh So this is something that I think we should uh address, not just from the usability perspective, but also uh there. There are I believe security aspects uh tied to this uh they're.

B

Just uh I like I, don't see a good reason for mounting uh the entire root file system uh into these containers and I would be you know very much in favor of uh uh making that you know a set of um uh small targeted amounts if needed, or even just uh and just uh propagating uh the individual files required, uh as opposed to uh you know doing what we're doing today.

B

um That's that's pretty much all I have to say in this topic. I see Adam is on the call uh so, uh uh but I don't see. uh Unfortunately, uh Adam do you have uh do you have any thoughts on this um I know there are a couple of cases that uh that are actually like uh that are actually documented in the code for why that was done. uh There was uh uh one was um I believe who work around for for just uh lvm Behavior.

B

uh Another uh was related to uh exposing um some of the locations on the host. uh For you know, to make to make monitoring um uh working like the the node exporter, in particular. I believe, was the issue, uh but again I. Don't think it needs access to the entire um root file system on the host.

B

um Do you do you have any uh like.

C

B

Other um cases- and you know, do you remember like in particular Chef ADM shell does this and there doesn't seem to be a reason for that uh other than to provide users with uh easy access to uh perhaps some of the configuration files that they have that they might have on the host. uh Was that the reason or is there something else.

D

Yeah so I think the only ones that have a legitimate case are maybe Seth volume and node exporter. um We were I, haven't figured everything out yet we're trying to look at the node exporter and what was the minimum amount of things? We could not not deal with the problems we had before um and I know. Guillaume looked a little bit at this F volume, one but I, don't think he's figured that out yet either that one's a bit tricky because the uh lvms is not supposed to be run inside of containers.

D

If you already have to do some weird NS enter like hackery to get all that working um and so I think the mount is going to do with that. I. Don't think, we've cleaned up what we can do yet around that, but if we want to get rid of that one, um the ones that aren't those two I think are just maybe the mistakes I, don't think Michelle should be mounting that I. Don't remember why that was added.

D

I'll have to go back and look at the history, but I think we can just get rid of that. One um I actually didn't even know it mounted the root file system. I I missed that um so I think that one can just go Um and any other I don't remember the whole list of ones you gave, but I think I said I. Think only node, exporter and stuff volume have any legitimate case uh for me to do that. I think the other ones they should just get removed.

B

Okay yeah, so they said, volume keys and the note exporter are the only cases that I was able to find. uh You know with it, with a uh with a with a quick uh grab uh through the code and through the history um uh yeah and the shell I was actually surprised to uh you, know, see and do that, but uh the the uh original poster on the thread they actually reproduced. The issue with uh with step ADM shell, uh the minimal reproducer um uh in both the uh separating shell container.

D

Yeah yeah so that one's probably just a bug or something something that got left over somewhere. um The idea with that one is just uh we have like the mount the headline options, that's supposed to be how you include a file you want to include in there I'm, not supposed to just be mounting everything. I.

C

D

I didn't even know that was the case. I had never excused it in that way before, so they can. Definitely, let's get rid of that one, and they said: don't leave node exporter and set volume where we're going to have to figure out something um a bit more of what we want to do to make sure that it still works.

B

Okay, that that sounds great, actually much better than I expected. um If there's nothing else, I think we can. uh We can conclude this topic.

A

Awesome I, just shared um a little late. I should have shared this earlier, but I created a ether pad for notes that we want to take anything that was said. Of course, this meeting will be recorded, um but just in case anyone wants to add anything so I jotted down a few things Ilia in case you want to add more for Adam, but uh it's there. If you need sounds like we, we've got some good discussion going.

A

Does anybody have anything else to say on this topic uh before we move to the next one.

A

All right, so our next uh topic is also in the usability usability enhancing the stuff user experience uh category and it's about accessibility in the dashboard Cedric. Do you want to take? Take it away.

E

Hello, everyone.

E

Thanks for having me here, my name is Google. Some of Court participant for itself. I worked on improving the usability of safe dashboard, bye, improving the accessibility standard of the dashboard for Blind and Visually Impaired safe users.

E

So uh you may ask: why did we decide to take on that project? uh To answer that? We first of all need to understand what accessibility is in order to better understand the problem. So when we refer to accessibility in terms of web applications engineering, we mean these applications are developed in such a way that every user's needs are taken into account so that they can also use that piece of software or technology comfortably.

E

So a problem statement, though, is uh accessibility in self-dashboard self dashboard is currently built with best practices in mind and meets some accessibility criteria, but since we are not certain as to how how much conformance it is to the accessibility standards, we cannot see for sure if self-dashboard is fully compliant, which was why we decided to undertake this project so by ensuring that we are able to meet the accessibility standards by the accessibility Consortium, which is uh level a.

E

We can say for sure that Blind and Visually Impaired users can also use the save dashboard comfortably.

E

So uh for us to really grasp most of what I'll be sharing, I would like to Define some key terms, which will we're hearing a lot going forward, so the first one is accessibility which are for the run over, but I'll. Take it one more time. So when we say accessible, when we send applications accessible, we mean everyone.

E

Everyone's needs was considered, When developing it, so they can also use the program or the software comfortably like everyone else, and uh we also need to understand who a blind person is or who a visually impaired person is in this case. So when we say someone's visually impaired, we mean they rely on a form of assistive technology in order to view or look at things I for one good example here so I can be considered visually impaired, because I need classes in order to interact with a computer system.

E

Same goes for people who are colorblind and people who are deaf on the people who are fully blind.

E

Also, we need to understand what color contrast is when we refer to contrast in general. We are referring to the context, so an item is usually fixed, but you could see it from both the foreground and the background perspective now.

E

Looking at that item, if we can see that item clearly on this background we can see it has good color contrast, but if we can't see that item clearly it means the contrast, this low or bad so bringing that in relation to computers, text, visuals, good color, contrast means that is legible and the colors are actually they strike recognize balance.

E

So good color contrast is great for accessibility, but color contrast is bad for accessibility and finally, in order to achieve the accessibility compliance, we have to follow a few standards. Now the World Wide Web Consortium has a good set of rules to follow in order to achieve accessibility compliance.

E

So we have three levels: self-dashboard we strive to achieve level a which is the basic, that's level double A and level Triple A.

E

And uh finally, one more standard we have to follow is on the Aria standard. So area stands for accessible, Rich, internet applications. This is a means by which When developing user facing applications, mostly in HTML, for instance, if we have an API where we could describe the content for certain Elements, which are not having text so by adding area labels screen, readers can pick up text and the accessibility API also gets that information. So by developing the area standard, we are taking care of those who cannot see the text visually but can use assistive Technologies.

E

That was a lot, but please bear with me while we move forward, are there.

C

Any questions at this.

E

E

Okay, thank you, so I'm going to proceed to show us some of the improvements that we actually made. So we began by going through the dashboard breaking down these sections and taking a closer look at some of the elements and components that could benefit from an accessibility Improvement.

E

So with that in mind, we focused on a few areas, starting with the dashboard login page, then the dashboard landing page for stages, health and summarize, and then we went down now to also look at the sub components, mostly the data tables and the other building blocks that were used in the dashboard.

E

Okay, so I'm going to share my screen right about now, so for comparison, I'll have two versions of self running: Queen C, which is the current stable version and the code and the main branch, which is the most recent with the accessibility improvements. I'll also be using a screen reader, which I mean uh sorry I forgot to know um to see what that was a screen. Reader is an assistive technology which helps read out the content on the page, so that someone who's blind or visually impaired would move along.

E

It reads out the text, so they can hear and also helps them in navigating the elements based on how accessible they are.

E

Okay, I'm going to share my screen now, so uh this is uh Quincy I have on this stuff. Quincy and I have on this tab. Sorry about that on this tab, we go in the main branch, so the first set of accessibility improvements were applied to the login page. To give us an idea of what those issues here, where I'm going to run on accessibility, checker, which will help us highlight the accessibility issues on this page.

E

F

E

I'm, using here is called accessibility, insights forward, and it gives us this really nice highlight of the accessibility evaluations, so I'll just move us through them one at a time. So here we can see, we have from that one. So three and four are the top.

E

Looking at this, we can see um it's been these labels here, the influx for color contrast, so they are not visible in terms of the accessibility standards by the Web Consortium on accessibility so for level. A criteria. The color contrast here needs to be better and also for this password icon. Here it says the person does not have a label, so what that means is if we turn on a screen reader. Let me do that real, quick.

E

But it doesn't tell us what that button does, so we need to tell the screen reader what that means, and.

F

E

And finally, for the language double up here, it's been flapped because the element being used there is not the appropriate element for these four. So this is supposed to be a button, but we are using a link. That's.

G

E

That's the summary of the issues on this page I'm, going to take us over to the fixed version and explain what we did and how it looks.

E

Okay, so here we are at first glance. You immediately see there's a change in colors, so we change the colors of the dashboard so that we can improve the color contrast- and we can see here that the labels use for the input fields are a lot brighter. On this background, so we improve the color contrast as well and finally, we changed the this item to a bottom.

E

So, according to the web accessibility guidelines for level a an item which is a menu supposed to be a button, not a link and finally, we added an accessible name here so screen reader can pick it up I'm, going to run our accessibility Checker one more time and then turn on the screen reader. To give us my idea of what this looks like okay, so we can see, there's no evaluations on this page. Nothing is flat and I click on our screen. Reader.

H

G

Right, foreign.

H

Expensively expanded.

E

Sorry about that screen readers can be weird, but they do help. So that was the improvements for the login page.

E

Thank you. So we'll move over to the next set of improvements which happened on the dashboard landing page itself. So after login.

F

E

H

E

Going to log in real, quick, okay, so this is the dashboard help page here. I'll have to turn on the accessibility Checker once more, so that we see the violations to begin with there. We have a few at a high level. The dashboard landing page has some color contrast issues. We can see, for example, the health label. Here it's not really visible. On the white background and according to the web accessibility guidelines, we will need to make it stand out more and also we have the labels for the cluster cards here.

E

The accessibility here says it's a color contrast issue as well, so the color contrast needs to be better. The text needs to be a lot more visible on the white background and finally, the helper icons by the side of each label here, which is there to say to indicate more text in a drop down menu. So here for text that is not really needed, but in a drop down, so you could access it if you need to learn more and finally, we have these violations on the side here which are for the navigation.

E

We will talk about this small data, so improvements we made through this page were to change the colors here, make them a little bit. Darker are accessible labels to the icons so screen. Readers can pick them up and finally, also change the labels to be more concise so for a screen reader to actually call out Health one is actually longer, but if we make it something like warning or error, it's easily readable and someone could pick it up faster.

E

So uh let me just show us what we changed here. Oh, they obviously made sorry I'll log out and head over to Main no login.

E

Okay, you can immediately see the label is changed so 43 cluster statuses, which are okay, uh one in an error, we've replaced them here. We need more concise text and uh We've also changed. The color contrast made it a lot darker. You can see here and then also increase the color contrast for the labels as well and also with added levels for the buttons. So the screen reader could call them out and also increase their color contrast, so they are easily visible. You will not miss them anymore.

E

To give us an idea of what that looks and sounds like or turn on the screen. Reader.

H

E

Okay, one more thing I forgot to mention was the addition of the skip to content link. Oh this keeps the content link just gives you a link to the main.

H

E

So, by clicking on enter once you're here it descriptive content. Then it gives you it takes you to where the main items are that you need to focus on. So by clicking on that we've been dropped in the main area of the dashboard. That's the stage. You start.

H

E

Your icons, it just got announced then.

C

E

C

E

Combination of The Mouse and the keyboard to simulate how uh visually impaired user will normally go about it so I'm using a Chromebox here as my screen reader and for navigation. My keyboard across.

H

E

G

H

G

C

G

G

H

E

So, let's just give us an idea of how a typical visually impaired person would navigate to dashboard with an assistive technology and how the of these have been made actually help out.

E

Okay, thank you and on to the next improvements were quite substance. Sign.

H

F

E

Sorry screen readers can also be very fast because uh well, depending on how fast you can transition between different items.

E

Okay, so the next item we took a close look at was the dashboard navigation login. Here.

E

So the result- navigation area is this area here, the site, navigation and the top navigation here.

E

So let me under accessibility Checker to give us context, but while we are waiting your navigation here, the main issues, mostly where we were using items, we were using links to represent buttons basically and according to the web accessibility guidelines, items or elements should represent their functions and for a drop down here, for example, it should not tell us it's a link. It should rather tell us it's a button that can be clicked and expanded same goes for the drop downs at the top here and the other menu items.

E

So we had to change the items here. It's actually buttons, so for them web accessibility guidelines. We could achieve that by just simply adding rules here to make them buttons on. That was fine. Well, we were also add it. So accessibility is also an opportunity to improve user experience, so we also increased the font sizes. As you can see here, it's a bit tiny, at least when we look at the other version, we'll see the updates. Let me just jump over to you real, quick, sorry.

E

And give us more context.

E

Here we are it's one as well. You can see the labels are a lot bigger, so we borrowed a bit from pattern. Fly here, person fly series to make navigation items a bit bigger, give them font size or 16 pixels, so they stand out data and also make the contrast for them visible. So the color contrast when a link is over, for example, here, stands out more because the background is a lot darker, like the rest of the the levels.

E

Thank you and finally, the last component I received major accessibility. Improvements. Was the data table on the dashboard we use the data table for so many different sets of information, so the data was used to display the host configuration items, rgw users, so many so uh I'll just use this one. Let me run now Checker once more to give us context.

E

Yeah, so I just ignored enough about here, I'm going to focus on the table for now.

G

Let's use one with.

E

Navigation, nice, okay, so taking a look at the table, we see several things being flat. We have the filter, bars the number of items selector the pagination components on the table, action buttons. So what's going on here, is this uh inline form elements? For example, we cannot attach labels to them like standard labels. We have C search bar and then search. So this is being flap, because a screen reader, for instance, will not know what this is.

E

So we have to make sure he can and one way to do that is by using the accessible which internal applications are real API, so attaching labels to them to label them and also the navigation here is uh not easy to navigate through them with a keyboard, for instance, because the elements are nested, they are not being read out and finally, here, for instance, we can look at this- it says it's probably color contrast. Yes, it is because you can see the text is white.

E

The background is CNN, and it's not really great to read that out on that background. Same goes for the action version of here. It looks a bit gray out. The text is not really visible, it will focus it. Some more make sure it's active, it's red. The text is white, not really great, so we have to improve the color contrast for these buttons labels and also attach the levels to reform elements so that they can be picked up by screen with us.

E

And finally, one more thing we could easily pick up here is on the way we navigate the table. If we have to go through the table to jump to a certain page, it's let's see which two advanced for the filters, and we see we have 876 item on this page divided into a set of pages to navigate to maybe the last page is going to be tedious. That will mean we have to click on so many buttons to get there so solution for this was just to replace the navigation here. Make sure it's something easier.

E

I will enable quick access to all the pages I'm going to head over to main show us what that looks like.

E

Configurations page area around the accessibility Checker once more, okay- oh this! This is still flat.

E

B

Oh, oh, oh yeah,.

E

Sorry about this, we will have to resolve this, but it's fine. We focused mostly on the very accessibility but uh color contrast is actually level wa and we are working on that multiple.

E

So if I'm very focused here, I do what's uh the buttons and the forms labels here, the form search fields. So if we take a look at this now, they are not flat because we've attached labels to them, we meet them in the screen, mirror. Let's see what that is.

H

Okay, surf cluster configuration tab search so.

G

Finish text search.

C

E

A search input because of the label that's been attached, then goes for the weather buttons refresh.

H

E

Thank you, Mr screen, reader, okay and finally, the pagination component here. So if we had to jump to page seven for instance, so we already on page seven, let's see page one- we are on page one on page five, we on page five, we no longer need to click on a thousand buttons to get to where we need to uh also some other improvements of image. Here were mostly to the pools.

E

Let's see, the research bar research bar was being flapped because yeah, it didn't have a title attribute, and it's required to attach titles and progress, regions for bars which are not visible or eligible in text form, so we added those and finally, some tables actually have.

E

um Let's say this selectors, you could select some rows from tables where you could actually do multiple selections, but this will be in flap because they were not accessible and you could not tell what state they were in. That is for a screen reader. Of course, let me turn that on.

H

Self-Object Gateway users.

E

So it now reports whether the row is selected and if someone was navigating.

H

E

H

Collapse right.

E

H

E

Going to be easier for them.

E

We select that you can also see.

I

B

It's just checked.

H

Checkbox not checked.

E

Okay, so that's a summary of most of the accessibility related updates. One more change that was done in terms of the navigation was.

H

Visibility: two.

E

G

C

G

Navigation tab.

H

Selected one or two.

E

It tells us if it.

H

H

H

Moving on performance, I have not selected two or two posts list: okay, tab selected, one or two.

E

So what that tells us is, uh we can now navigate the tabs with a screen reader to you know, basically.

E

That gives us an idea of why we need to make things accessible because visually impaired users really struggle okay and uh that's a summary of most of the accessibility improvements and one other thing which we also included was end-to-end accessibility testing.

E

So after doing all of this, including the labels making sure certain sections are conformant, we have to make sure that there is no regression on that. So we resorted to add in end-to-end accessibility testing in the purpose. Yes this week. So when we deploy this, it goes through the testing pipeline. If any error, Circle or in accessibility evaluations occur, you will know about them and also fix them immediately and uh also uh what's next. So some of the accessibility improvements are still ongoing.

E

We've mostly focused on attaining level E, which means is the most basic level so that users could navigate the dashboard comfortably, but we could do more. We could do better, so we could strive for level double a level Triple A. So we are currently working on that going through the dashboard finding sets on areas that we might have missed and then working on that and uh finally one more thing: that's coming for accessibility will be a dark mode theme we can see here uh on the login page. The login page has this dark theme.

E

The colors are not really shouting out, you can look at them and your eyes won't hurt much. So we plan on extending this functionality to the rest of the dashboard so that you could also navigate it in the dark mode. On my stream stress.

B

Your eyes quite.

E

Much and also we've included documentation. So if we take a look at the dashboard development guidelines, there's a new section for accessibility, which will give us some tools and some practices to follow, to develop with accessibility in mind, and thank you very much for your time. uh I'm gonna send one off any questions.

A

Really great job Cedric, that was a very nice very nice demo.

E

And, thank you all.

A

You'll have to check the chat, Cedric you're, getting some uh comments from people.

E

Oh sorry, let me let me stop sharing articulous.

A

Thank you they're, just saying they're saying that you did a good job.

E

Oh thank you very much, uh I'm still struggling with blue jeans, but I'll figure my way around thanks. Everyone.

A

Thanks so much Cedric and Cedric was uh just part of our Google summer of code program, so um he's a model intern.

A

Awesome, does anybody have any questions for Cedric.

A

I think you covered many aspects: Cedric good job.

A

F

C

A

Yeah, of course, and if anybody needs something to reference, it there's an ether pad associated with this presentation. um That's on the agenda and I shared it in the chat so um yeah. If you have any, have any questions afterwards, you can reference that awesome.

A

So our next topic, uh still under the usability enhancing the self-experience category, is uh the dashboard landing page update.

A

So um you guys can take that one away.

J

Yeah I'm thinking of Hawaii, uh hello, everyone, my name is Pedro Gonzalez and I am currently working with a theft, dashboard team, so, as Laura mentioned I'll, be showing you guys what we have been working this past few weeks. Let me share my screen so yeah.

J

As mentioned uh this past few weeks, we have been working on revamping the landing page for fifth dashboard, so today, I'll be showing you the the work we have done so far, as this is still a work in progress, but I'll rarely show you guys what we are doing and how and yeah so that's it.

J

So, first of all, why are we doing this right? uh The the goal would be to to show meaningful data on the landing page of the dashboard and in a way that it's is Evol, is very easy and accessible for the user to see what's going on on their cluster on their system under this dashboard right. That's everything you see when you enter the dashboard, the landing page, and we want to have a clear. We want to give a clear feedback there.

J

So let's go with some history of the landing page. We have got so far. This is the first version that came out uh and 2018, and you can see that it will. We will be changing this a lot. This was the second version that we got. It is a.

J

It looks more like we are what we got so far, and this is our current version right of the save dashboard. This is what we are working with at the moment, and this is what we are trying to achieve at the moment. This is our plan, and this is what we are working currently on to try and to achieve.

J

So this is what we planned and, even though we are still working currently on it, we are making really great progress and we want to show you guys uh what we got. So we can get some feedback as well, so yeah as I mentioned, this is on still a working progress right.

J

So what we have left to do I'll be showing you right right now in a couple of minutes: uh I'll first look into it, Adam of it, but let me tell you that first, we need to still uh finish some of the functionality of the dashboard display. All the data that we want. We want to I Really, get what it is, the most meaningful data in the in safe and displayed during the landing page for the save, dashboard, right and obviously style.

J

Everything through hopefully get a look similar to this or what we want or at the release, something we are happy with that follows our guidelines.

J

So, let's see what we got, this is the current uh work. We got the work we have done for the landing page, as you can see, there's a lot of styling to do as we have finished workers on the functionality of the page right. So we have done almost all the functionality for the new landing page and I'll be explaining you guys a little bit of each of one of the cards, what we can see on each one of them and yeah. Hopefully you guys can understand why?

J

Where do we need and uh give some great feedback on it at? First of all, we will have a details card to see a different data to display of our current cluster. That's a version for example right which orchestrator are we running at the moment? I'm not running any any of them? That's why we are not showing any.

J

uh We are also on the next car showing our status of the cluster. We can see that we get uh the whole cluster and some alerts that we can pop up and see what is going on inside with our cluster right. They are filtered by their current warning or danger, or success alert types to clarify. What's going on and be notified at every time.

J

Next up we got a capacity card to display uh the data we are currently managing using and hover uh our clusters at the moment, which is very very similar uh to what we have at the moment, but doesn't mean that it is not useful right. So we want to keep that in there. Next up, we have an inventory card to display our fifth current demons and different things that we want to show the user what we got in the cluster right.

J

uh We might also add a couple of different things like the PG status later on, as we are still working on them. Next up we got the cluster utilization, uh which are planning showing different data or diff with different graphs. It's one of them being a ffecting.

J

The data from Prometheus as we as grafana is currently doing, but we will be able to see everything here on the first page of the dashboard right, so we can manage all the data here and monetary size everything in there just looking into the first page of the dashboard, uh we are still planning on how to work with the events card, as I showed you briefly earlier, but since we want to get in the landing page, the most meaningful data for the user and what's what we might think that it is the most meaningful uh we want to avoid, having the repetitive data and that's why we are still analyzing and considering different options here, but this will be mostly it for for my site as well.

J

You can already tell that it is still work to be done here, but I think we have done a great progress this past few weeks and hopefully die some next week. We will be styling everything uh to make it look as good as possible. I have a new page for the dashboard, so that's it from from me. So if anyone has any question or anything that wants to share or feel free to do so,.

A

Plan for a reef.

A

For uh what what upcoming releases, this kind of timeline, uh wise plan for.

J

I mean I'm not entirely sure as uh I think it's planning to go into 6.1 at the moment, uh if I'm not mistaken, yeah.

K

But better but Upstream wise is it's going to be a brief.

J

A

Nice and and I'm, but from my understanding the goal is to display more information that uh or better information on the the dashboard landing page in general.

J

Yeah I can give you a quick example. If we go to the thermal landing page, oh, we can see that a lot of space is being occupied, but with simple cards that only show one uh dimple information like your host. One total right. We don't get really a lot of data from here and it occupies a lot of place. So we might maximize the data we're displaying just the most money in full data and keeping what we think the user or we would like to see into the dashboard right.

J

So we can access more data in this space right.

A

Yeah, it makes sense.

A

J

Well, thank you if anyone any questions uh anyways, if someone wants to get in touch with me, feel free to do so uh to get to give me some feedback on what they might, they think of will be a great upgrade to to have in the new landing page, but thank you very much.

A

It's the magic Pedro I have one more follow-up question.

A

Showed is that kind of like approaching what it will look like, or is it more um like a proof of concept currently.

J

uh Both I would say right, so this is what we first planned, but this is what we are going for right. At the same time, it's uh still up to change, obviously, but this will be mostly what we, what you will see in some weeks.

H

A

Thanks Paige Pedro, very nice work. Is there anybody else? um Oh thank you. Does anybody else have any questions or Pedro.

A

All right great work, Pedro thanks, so much for sharing with us today.

J

Thank you for having me.

A

All right um so this next uh category uh there aren't any topics in the quality, improving testing and releases process, um so I'll skip it for now, but I can come back to it in case anybody wants to uh or forgot anything to add under there, um but I'll just skip it for now.

A

The the next category that we'll talk about today is a performance boosting stuff speed and memory usage. The first uh item on the schedule here is manager, architecture proposal and Perry I'll. Let you take that away hello.

L

Been working out hasn't.

L

But receives someone to reach me offline to countries mean to permanent 34th, actually, so fifty to those who I take the capacity anyways. Let's start the presentation, we are going to look at the content. First, how the manager is currently working, then we'll move on to the issues with this architecture.

L

To develop and last.

L

We'll look at this pretty simple clusters that we have maps and we have mass in its host and this pair with the monitor and we because architecture were on one deductive and if we're not completely understand what it does to produce data from the meter and it syntheses the data from different places.

L

H

M

Me just just sorry to interrupt I think your audio is pretty choppy.

F

And there's some discussion about it in chat.

F

L

A

I think I think.

L

A

Network problem for you, I see your connection slow.

A

A

Maybe a good solution would be um if you call into the meeting and then, if you want to share the slides with somebody, we can flip through your slides, and we can also.

H

Come back to you.

A

If you want to try that.

F

H

A

You can uh share the slides with you, can give uh or I'll put my email in the chat or yeah also says just try rejoining, um but uh you can share the slides with their nest over me and we can uh you know you can narrate your presentation through your phone.

L

A

Can somebody let me know if Perry is on the call yet.

L

uh Can you hear me now.

A

Yes, I can hear you I think that's much better.

L

A

So let me know if I'm on the right, slider and just say next slide when, when you want me to Advance.

L

Perfect internet so I'll start from the awesome. So you can look at the monitor active more closely and you can see that we have the data that we centralize on the left in the monitor map FS map, all of those Maps and also at the right inside the monitor active process. And even you can see that we have an embedded, C python interpreter and they see python interpreters also have sub interpreters and different rates and those threads also for the the modules of all the python modules. So if you can go to the next flashlight.

L

And since we are working with C python interpreters, we also have the global interpreter log and This Global interpreter. Look what it basically does is it you can move to an externally and well what it does is it blocks?

L

Well, only one super interpreter can work on the global interpreter. Look each time it wants to perform some actions in the in Python objects, foreign.

L

Can work with the lower interpreter.

H

L

Meaning that if you are performing something on one module, you might block other modules also move to this next Outlet.

L

Yeah, so you can also request the data through python bindings. You can look there and if you want to request something from the the merger demand, you go through python bindings and request it and we perform some serialization onto python objects and we send it back to the super interpreter and move to the next slide. Please.

L

And well, we are going to look at some of the issues with this architecture and you can see that there are we've listed both of them, but there are a lot more than with anything but as meaningful to the pleasure. But if you go to the United States Lite, we are going to see some issues with the super interpreter.

L

L

Taxes through multi-threading because of the global interpreter log, so that's a big issue for performance that makes the monitor easily uh well. It becomes a bottleneck for different modules. Then some of most of the dependencies are shared. Sorry previously,.

L

And now the the so some of the dependencies are.

C

L

Of the dependencies- and maybe all of them are shared between modules, meaning that, for example, Cherry pipe. The cherry pie version is shared between different modules and we probably want to move to different dependencies in different modules and, for example, if you want one dependency one model, if we can just keep it in the other module.

L

Also, there are some packages that have embedded the state in The Interpreter, for example, the numpy package. That means that if you import numpy in two different super interpreters, it may cause some unknown issues, uh security problem that we can also have if there are some non-core modules that could claim privileges through the monitor. You could also call the orchestrator module through a new module. If you wanted to uh move to the next slide, please.

L

uh Another issue: this is mostly on the developer side, when we are trying to debug the the Monitor and its modules uh because of the way it works. We only have one entry point for all the modules and the demon itself. So when you start the monitor, it will start The Interpreter inside, so you cannot call start a python module by yourself, and that also means that you cannot call the debugger for its python debug and to its python interpreter and move to the next one.

L

Well, I think there are going to be some extra slides there that those are I posted some logs, but how it looks at the end. Our looks you end up having this bring the login sessions. He needs be sure that you want to develop in the monitor all in the module that you are developing, so it then that's been uh and well. It makes us less productive and next slide. Please.

L

Also, whenever you try to enable or disable a single module, it will end up restarting the whole interpreter and also it will respond the monitor, uh yeah. You cannot simply enable or disable one module at a time you have to enable and disable every one of them. If you want to restrictions and enable something decide or something.

L

Next, one is Monitor Primitives. If you can move the next light, and here I'll only listed the CF Porter demon, because the it is a work in progress to fix the problem that we have with Perth counters and R1 is going to present that in next uh in this presentation. So it will look at that more closely later, but anyways the from misuse module. What it does is it passes all the pretty much all the data that we have in the manager.

L

So it means that it have to realize all the data to python objects and yeah yeah when you do that it uses the global interpreter. Log, Street and I've been adding about the neck and next slide. Please.

L

So now we are going to look at the the new.

H

L

That we want to implement so going back to the to the previous architecture. You can look at the same one as before, but what we are going to try to do here is first of all, remove the python interpreter. So.

I

L

Get rid of the global interpreter log and you can go to the next headlight.

L

So there we are only missing removing the super interpreters and moving out the dashboard, the modules. And if you go to this message light, we will see that we move every module to a different process and we connect those processes through a grpc server. And we are going to talk more about the messaging.

I

L

That we want to use, but for now, let's think about the RPC server, so we have a drpc service that connects to other modules, and if you go to the next slide, we we see that we will also have to add health checks to the modules. If we want to restart some of them, because there was a failover situation also, you can still request data.

L

If you go to the next line and instead this time around, instead of serializing to python, we will serialize to the skma of the message in protocol, so we will get rid of the global interpreter log.

L

If you go to the next slide. Please.

L

Yeah with regards to the Backward Compatible changes: well, we will try to make this as Backward Compatible as possible, and for that to be possible we will only make changes to the API side of the the demon and also the modules and yeah. We will have to implement a new interface to deploy its module and its processes.

L

Yep. Can you move to Industry.

L

So, coming back again to the same list of before of the issues that we listed, we are going to look at the improvements over the previous architecture, with a newer one and starting with the soap interpreter. If we go to the next slide and this time around, we will have a module to Running In Parallel with.

N

L

Multi-Threading, so we have nothing novel as you can see, but it this there will be an improvement of the bottleneck because of the global interpreter. Look if you move to the next flashlight and we move from having only one entry point to having one entry point for the demon and one for its module, and this will basically improve the possibility of its module.

L

Also if we move on to the restart single modules, since we don't deal with interpreters this time around embedded interpreter- and this means that we we should be able to restart, enable disabled modules without modifying the other ones and on the monitor, Prometheus site. uh I haven't listed most of them since one is going to present in the next presentation, but basically by removing the global interpreter log and by serializing to something different uh python objects, and this will help by reducing a lot of load from other modules.

L

Yeah- and we also got some master improvements over here with one great Improvement I- think it could be. Nice is having different modules written in different languages, instead of all all of them, written in Python, and also something great for us developers would be to run python by the modules by yourself. Instead of having to run them through the monetary API and move to the industry.

L

Yeah we are going to look now up more about the messaging protocols. We've had a lot of discussions over here, so we're going to list some of the the features it's protocol we've listed first so with we've divided the TCP based protocols and the memory based once because some of us one are thinking about TCP based because of the the per host communication and also the memory based one because of the performance boost that we get from those.

L

So if you go to the next slide, we will look at the feature about what is that the API latency, so with TCP based protocols, we will all we will obviously have more latency because of the network obstructions and in the memory based one. It will.

C

L

Will have less latency? Obviously we move to the next one and we're going to look at the performance and yeah because we are using TCP in the tcp-based protocols and because of the instructions most of the code will be more expensive.

L

Probably obviously we have to measure all of this and a nice addition, I would say, is grpc supports bidirectional streaming, so you can allevit some of those problems and then on the memory based protocols. We have to expect login because we have to store most of the centralized data in in memory, so we have to keep track of those.

L

If we move to the next flashlight.

L

Of the these protocols with grpc, we got a full-blown library with protocol first and flat buffers.

J

Everything is implemented.

L

In every language that you can think of so we don't have to do any work there. That's why I'm.

I

L

Biased with grpc I guess, then we have the messenger tool that come from theft and tutorialize these messages we have right now we can only use C plus classes serialized with buffer list, so we will have to implement those in every language then goes for memories based ones with shared memory and device.

L

We have to implement the serialization zone for each one of them, so there is a lot of work to be done. If we want to go with those- and next is late, is.

L

Also, one great feature with grpc I guess is that we have one tool to technical for every language and yeah I. Think that's a great Improvement, so we don't have to leave it with implementing every API call for every uh endpoint that we had, and this is an existing, the messenger one, the third memory one and the post there seems to be some packages for its language, so we would have to look at those and next one. Please.

L

And also with, if we wanted to go with host per host communication uh with memory based one, you would have to go with distributed shared memory which obviously take more time and yeah. It needs a special software hardware for that, but with TCP base, one.

B

L

The rpcm message: there's two: we don't have to do any work there. So that's a thing to take into account and move on to the next one. Please yeah.

C

And at last we are going to look more.

L

Or less of the development plan at this plan and performance plan, but as you can might guess, this will obviously change and the development plan relation plan. We basically will rewriting the manager module interactions with the monitor Daemon through grpc, and for that we will remove all reference to python interpreters and the global interpreter log and finally get rid of those. Then we will also need process of pounding for its module and the language it's implemented.

L

Also. We need health checks so that we can keep track of failovers and we might need to change the manager monitor because, right now the options and commands from modules are only loaded when the the monitor is pumped, so you cannot modify them dynamically while the monitor is running, um maybe maybe in the future. This is this.

L

We will not work on this one for at least now, but maybe we will work on scaling out the monitor load between hosts but yeah. We still have to look at that. So it's not we're not sure about that one yet and yeah, and about testing it's very much, always technology. We respect to pass the totality the dashboard and windows, because this runs on VMS and we test a lot of features about a per module communication, inter module connections and in the performance side. We have to do a lot of work there.

L

Obviously we have to make a first measurement of every what the previous architecture and then compared to the new one. So we will have to disable all the modules and create a benchmark module that runs parallel, but yeah more or less. This is the work that we plan on doing and yeah if I'm open to questions. So if everyone has a question, go for it and that's it.

A

Thanks so much Perry.

A

Does anybody have any questions.

A

I guess I have one about the testing you mentioned. So do you plan to um add any more or additional testing around uh this architecture proposal, or are we thinking more about uh using existing uh testing infrastructure to to validate that? It's working.

L

I say: I want to well I've already written some of the code for a PLC and that is implemented in RPG. So one thing that I had to do was Implement a new interface for the monitor modules. So a great start would be to test that new interface and yeah place that interface and also probably add more unit tests to the monitor, because it I think it doesn't happen that many and yeah in the filter we'll see we already have to test more things. But for now this is obviously stealing work in progress.

A

Okay, that's good to hear, because I was thinking about um uh the manager tests and the radio suite for uh technology and um I wouldn't say it's like too much coverage, um so it'd be good to have. uh If, if this is like, this seems, like you know, a rather large change to have um any extra tests that could be targeting it directly um as unit tests or in tautology.

A

um That would be good to have.

L

Yeah I agree: uh Walter I'll try to make us Master as new test as possible.

L

Yeah, for now the the grpc server would be in the the manager demon on each demon. I guess the Distance by one the active one. Each one of them will have a gtrpc server and it connects to the modules in the same host not on different hosts.

L

Oops I, what I don't know.

A

Very um I was adding something to the agenda. Did we get through all the questions here in this? In the chat, there were two from um Matt Benjamin.

L

I thought they answered it.

F

M

um But I was confused by with the slide that talked about new new serial and sterilization and deserialization for Messenger. Two, that's good! That would be expensive if we were going to do that differently than we've ever had our native implementation. Is that actually a thing or is that just was that just a part of a matrix, that of all the verticals.

L

I'm sorry, but can you repeat again uh it was hard to hear from the my mobile phone.

M

Sorry I'm sorry I apologize for not knowing which slide it was on, but he had a matrix of all the protocols and it seemed like you were saying that there's a need to do new to serialization and deserialization too, from streams of serialized buffer list output. At the case, my misunder. What was I misery in the slot.

L

L

Well, the in the entire PC one we have the protograph and then the messenger 2 we had I presented the.

M

Does everything moved it does all the energy and it's all the communication between the marketing components? Now the of the of the distributed manager now become grpc? If so, that answers my question.

L

And not only the communication between module, the modules and the monetary itself. The communication in the well different demons on the monitor it will prevent the same.

M

Okay, but this is the one, the one module that already existed.

A

I'm not sure if this is the slide you were mentioning, but.

M

Yeah so I'm just finished, so is there anything new that gets done on the right hand, side of messenger, 2 or not I think it stays the same, but I'm I'll get my best I'll give up asking it anymore.

L

I mean if we were to use the messenger protocol for communicating modules and the monitor we would have to do more work to serialize to python. What do we see realize from in Python and all different languages.

M

A

I have this uh ether pad up of uh for any notes that we want to add about this um pair I I added your presentation to it, um but yeah. If there are any uh details, you want to add to it, uh feel free.

L

Cool. Thank you.

A

Thanks so much does anybody have any more questions or comments about this topic before we go to the next one.

A

All right, oh Sam, go ahead.

N

Can you guys hear me okay, cool? um My question was there? Are there is communication between manager modules? uh How does that get addressed? That is, there are shared memory structures that multiple manager, modules, access.

L

F

L

Plans for it the first one that I tried at least in my previous year, was the RPC server footage mode. So it looks like more peer-to-peer, but we didn't like that and without us we thought of having the communication done through the monitor and basically I mean a poop soup server in the active monitor, so that you get you send a request and you have an interface implemented in a different module or you expose the methods that you want to expose.

L

Basically, so, for example, the dashboard makes use of the the orchestrator a lot, so we would have to have an interface in the orchestrator that it poses all the methods that we want. So we would try to make this as easy as possible, so, for example, having a decorator for its class. Well, it's method that you want to post, so it everything is posed with that decorator on it. Something like that. That is the plan for now, but.

N

Yeah, we are open.

L

To to feedback on that side, I.

N

Mean bloodly that seems like the most complicated piece here, much more complicated than the um interface forwarding from the manager. uh Do you have a slide on that architecture?.

L

The you mean the communication between modeled. That's right for that. Yes, I don't have one currently, but I cannot try to make sure I'd.

N

Encourage you to focus on on that sooner rather than later, I think you're going to find that that's an important piece. My other question was this is strictly single host right. Yes,.

L

B

uh I I think just to uh just to uh kind of expand on Sam's comment that uh we need, uh like the inter module. uh Communication channel uh needs to be uh uh very well thought out. um uh This was raised in the in the internal presentation that Pierre did on this topic uh and the the concern, like the other concern uh Beyond just the mechanism is also uh that uh the current plan seems to be uh that you, basically just uh you, know, make it uh like the backwards.

B

Compatibility bullet on the slide is all about um substituting uh the existing. uh You know python interface uh with uh uh with something that looks pretty much the same, but speaks uh grpc on the back end, and uh that would work. uh But uh if you, you know more or less extend that to the uh to the uh inter module communication, uh I'm I'm, pretty sure you would run into cases where uh the global interpreter lock that you that you're getting rid of uh was actually um uh sequencing.

B

uh The modules uh between themselves and uh also ensuring that the data that is being shared uh is stable.

J

B

In in in the pub sub approach uh that that you mentioned, uh this would no longer be the case, and uh it's not just a mechanical transformation of uh okay. uh We used to call this API, so how about we expose it uh through Pub sub or any other means uh you would need to.

I

B

Every module in in each Direction, uh uh so consumer producer and the other way around, uh for uh you know whether the data uh that is shared- and you know the responses that have been uh sent, uh whether those, uh whether that's immutable data or something that needs to be protected uh across the across this API calls potentially from multiple modules.

B

Because, if you uh imagine you know, the orchestrator module might actually serve, not just the dashboard, but also you know someone else, uh and uh there are all kinds of uh consistency, questions that that come up here at scale. uh There may not be much of this uh there today, uh but uh with this the the uh the uh architecture that you're proposing it it encounter.

B

It basically encourages uh decoupling everything as much as possible and I'm afraid that you might find that, uh especially between the dashboard and the and the orchestrator, or between uh the uh like, even between how the different modules are exposed uh to the cluster State uh to those shared Maps.

I

B

Might be some assumptions uh that uh what one module sees uh the other you know pretty much sees the same thing, and that would no longer be true uh in the proposed architecture.

N

C

N

The larger distributed maps are basically free to access now, placing it with this system means every time you try to access it. You're going to incur marshalling and demarceling costs, you'll need to factor that in as well um just piling onto Italia said I'd really encourage you to audit literally every such shared memory. Access before you start on this, for real makes sense, and you don't like make a list and send it out acceptable.

K

Yeah I would like just to um make a comment on that, because currently, the only way, the official way to uh communicate across modules is via the remote code, I, don't think that's heavily used actually I think it's only used between occasionally between dashboard orchestrator, also the NFS module I. Think. That's also does that and in my opinion that's always been a not a unoptimal pattern, because basically it makes every piece of code in a module uh like uh an external interface right.

K

So basically every model could consume other modules, functions or methods, and that's not ideal. So in this way I think we, the idea, would be to force the modules to indicate which methods are available for a public consumption by other methods and actually enforcing a public API for each module rather than the current status, which is that you can call and I think that's also happens in Telemetry, right or I. Think Telemetry consumes via remote uh call uh info from other uh from other modules or the dashboard itself does that with telemetry.

N

So I think having well-defined interfaces is great, but that's not what I'm talking about.

K

um Yeah, for instance, that goes through the grpc server instead of the current approach. I think that I mean would be ideal, and also we have face issues in the past with this, because you basically have access to the memory Arena of other modules. So technically you could corrupt that it's I think it's been an anti-pattern iterative from the sub interpreter things yeah.

N

Those things are all true. What I mean is that when you need to Marshall data like a big chunk, for instance, a piece of the PG map, or something from one piece to another, for real reasons, because of an external interface, it's still more expensive in this scheme. That may well be fine, but you won't know until you audit the code.

K

Yeah yeah. Definitely this should be Benchmark. That's one of the key things that we have to do that in terms of latency, rather than throughput, probably because I don't think that we are actually moving that big amount of data. But probably the throughput might be an issue here with a networked interface.

B

Yeah, it's just just based on some some of the kind of past experience. I, remember us running into performance issues uh with just uh repeated uh Deca like encoding and decoding of those Maps, uh and uh so uh that suggests that uh this this code path, or at least in in some of the in at least in some of the cases.

B

uh uh That's that's a you know uh very performance, sensitive um uh path uh because uh there we were talking about you know shared memory, but just CPU overhead from uh from decoding it from a uh you know, from from a from a buffer list and back essentially uh and uh or between the python object and the the C plus representation.

B

uh In this case, uh we are also adding a hop over grvc. So uh some some of those uh uh some of those call sites, uh might might be a problem.

N

And I'm not calling this out as a blocker for doing this. What I'm saying is that it's worth doing the work right now to identify any of those problems, because, as Matt's pointing out there are other techniques for sharing well well distributed structures like that across multiple processes, and that's what you'd have to do. It's not that it would make something like this impossible. It's that knowing about it now, will allow you to make better decisions.

N

So I'm asking you not to be surprised by it. Basically,.

L

Yeah so I said we have to.

J

Measure everything.

L

Right now and yeah I'll.

H

L

Into account, the comment for sure I didn't think of that, so I'll be looking at it.

A

And it sounds like you have, a proof of groundwork. Can.

A

um Deciding all and measuring everything and yeah it seems like that, can be worked in.

L

Yeah right now, I have implemented the enough code so that we have only one manager active running with.

F

L

Most of the modules are working right now. The only missing part is inter module communication, so yeah I'll have to look at it right now so and then the performance side, the shared infrastructure I, have to look at them. Also right now, and then the Third, but we have to look at, is the failure. Will recovery in this module.

A

That says, I think introducing distribution now is good, but mostly because it repairs for scaling Beyond single process later. That said, just translate to golang or something doesn't sound crazy to me.

A

Well, good, it's it sounds like we've got some good feedback here, Perry and I've written down um a lot of what people have said. Also we can have this uh video to refer back to, but um I've got some notes written down so um good job on your uh presentation and you're I'm glad to see that you're working on a proof of concept. So you can address uh all these um concerns early on and provide like points of measurement.

A

um To answer some of these questions.

F

A

And kudos to you for uh having presented this uh now three times very good job.

A

Thanks so much does anybody have any final uh comments, questions or concerns for Perry about this presentation.

A

All right thanks everybody for your feedback. That was a really good and productive discussion, appreciate it awesome. So the next uh topic on here is uh still under performance, and it's about the suffix borders latest status with the demo.

A

I

Yeah, so basically I will just go with the brief of this. Have exporter like why you will ask like that is already a self-explorer, what this new self-exporter? So basically, this news of exporter is to goal is to basically fetch the perf counters from each demon.

I

And then expose it to the HTTP metrics I know that Prometheus, so why it's doing the same job as what we have already for Prometheus module basically is like at scale what was tested with, like thousands of osds. Maybe um has performance issues as Prometheus uh just use one in a single exporter which tries to get all these people of counters from all this demon through the MGR call. So that's a huge load and it becomes particularly unreachable at that scale. So they um solution to introduce a new export, which is a per node exporter.

I

Basically, so it will get deployed on each source and it will fetch the perf counters of all the demons deployed on that host. I can briefly show the design as well we'll share my screen yeah. So that's why I was saying that the um the self-exporter is. This is a per node design.

I

uh So basically we have the circuit files dumped uh in the weapons, at least in the fdm you can say in in the Vista we have ISO folder. Where do we turn those files? So using this admin circuit client, we can talk to those files and basically run some commas like perf dump and perf schema.

I

That's how we can get the contracts for different events, so suffix, basically a collector which has different methods like update sockets, which is responsible for going through these um of these shortcut files and extracting the uh yeah, basically collecting that, uh like what all uh uh files we need to read and uh the other is the dumpster but Matrix which we basically uses the socket client and then runs the pulse, schema and thumbs and yeah that's how it gets the perform counters. Then we do some formatting.

I

According to the Prometheus format like adding some help type or whatever labels which we may need to add, and we are also entering the metrics. We are basically sorting um based on the demon names and there we have the HTTP server, which is based on the Boost based Library.

I

So just a simple HTTP server, exposing a H, the metrics endpoint, where we have all this um metrics.

I

Implementation for this going on basically yeah defining a new container.

I

Thank you, yeah for this self-explorer demon, okay. So next I will show like how you can actually.

I

For the statement of help helper for it, you can see my terminal right.

A

I

Okay, so yeah. Basically, uh there are like four to five new, uh so it has the options it provides. So the socket directory is basically what's the path for the socket files, and then we have the address. Basically, that is the one word host you want to deploy this.

I

The port varies by default, which is a 99 into six which was available from the Prometheus, so yeah we choose it, and private limit is basically, you can filter out uh which part of counters I need. uh I know the default is five which what the MGR does as well uh or at least I, mean Prometheus module. Another Elementary modules as well, that they.

I

Basically, that means that, uh after how much time it will refresh the data or update, sockets and again run those Buffs game and dump commands. So, yes, and else is the same- you can still run with the default. If you want to run, you can run in The Hyphen, the foreground. So right.

D

I

Want to run this limit because I already have a container running with the safarium cluster, which I will show.

I

Okay, see the self Network.

H

I

Yeah so right now there are like Two Hosts I have deployed, and for each there are for each host. There is a soft Flex full of demon running you can see, and we can even see here that this this is basically the um this will be so which is running an electrical 10.6, posing all these metrics, which Prometheus can grip.

I

If we look at the Prometheus targets, we can see that uh so there may be something wrong with this, but yeah, but it basically is, has was able to scrape the metrics from the from both of this exporters that there is this one.

I

This perf counter and I will show that. Okay, we can, we can see, we still see the data in the grafana dashboards which may use this counter. Yeah show The Matrix.

I

So right now, if you see this people OSD, it basically uses this uh hdmpg to get the number of features. Histories yeah, so maybe last one hour make make sense.

I

Yeah, so it has um some data coming, which is basically was coming from this exporter.

I

Okay and now, I'll also show that uh this old, but this is the what the exporter which uh Prometheus model is running, will not have this.

I

Basically, the.

I

Numpy, so num PJ doesn't have it because uh uh I have removed the perf call from the Prometheus model, because we no more need it. As exporter is dealing with so but yeah now, the question is like why we still need this exporter, but uh the thing is there are some cluster by metrics which are still exposed like self-health.

I

There are some arbit steps to read and write Ops metrics, which are exposed so and for that like um OSD map is used and some other maps are used, so they are more cluster-wide and this is Upon Our export or so so. We still need this. um The old Explorer uh for the customer matrix, but yeah the uh still exploring what can be done or uh if exponent can do something here, but I guess yeah, that's, basically it um and okay I will show this as well.

I

This was the last test result, basically that with the 100 osds I was able to test at least uh the script. I was extremely concerned with the two yeah a bit improver, but yeah. We still need to actually test it with it like how it looks actually with the thousand noises so yeah. That's the next uh I can say but um yeah I guess: let's go sleep.

A

Much Perry um I have one question for you. So when you plan to test on the thousand OSD uh configuration, how do you plan to do that, like with um simulated tests or like on the giva cluster, or what's your plan for that.

I

Well, testing yeah I'm, not judging because we still I go to this level. 200.

I

uh Maybe uh yeah the plan was to maybe somehow get with the safarium Box. uh uh But yes, there is still some working progress, I'm, not sure.

A

Okay, because yeah um I I'm not sure uh how, if you could for sure, go through this path, but something I had in mind was our give a test cluster um that it's a thousand osds are around that number um and we've uh often tested uh upgrades or features on the gipa cluster. Even before we upgrade the long running cluster um and the self-exporter may be a good candidate for uh testing it on the gibba cluster.

A

Just a thought.

I

Yeah yeah: that's a good idea.

A

No I really like the live demo. You did a couple of cdms ago. We had a discussion about the load on the manager with the perf counters so, um and we had mentioned several different Avenues. One of them was uh the stuff exporter. So I'm really excited to see all this work that has been put into it since then, and um how polished it seems.

A

Does anybody have any uh questions or comments or concerns for Evan.

B

um Yeah I had a question, maybe I just misunderstood. um You mentioned something about uh kind of cluster-wide, metrics uh and I'll. You know.

F

B

Doesn't mean that the self-exported demon at least has currently implemented um doesn't mean that, apart from scraping the admin sockets on a particular node, that it's running, uh that it also connects to the monitors uh to get the uh to get the cluster Maps. uh Is that the case or have I misunderstood.

I

Yeah right now it's not haven't planned yet because um what I thought is this? The exporter basically, is now on each host, so I don't want uh like each exporter, one uh fetch that metrics also so but yeah. If we are looking. Maybe export can help here for cluster, where maybe we can do some load. Balance thing also like there is just one exporter may be responsible for getting all these maps and exposing similar way to get metrics uh and yeah, and we can maybe load management with them.

I

Is that goes down then some other Explorer demand comes up uh just a job so but yeah, it's totally open for thought. So I would say it's still not been yet planned fully that what could be the case for the rest of the metrics, which still Prometheus model, is exposing so.

B

Well, I I think for for cluster-wide metrics. uh The manager might actually be.

B

uh uh You know good enough, so the the existing uh Prometheus module in the manager like one self-exporter, is uh uh kind of fully fully implemented and uh and fully integrated are the existing uh Prometheus module in the manager can be repurposed uh to just deal with this uh kind of cluster-wide metrics of which there shouldn't be many, uh and uh that that wouldn't be a scaling issue, because uh you know it's it's it shouldn't depend on the number of osds or anything of that sort.

B

uh So uh you could kind of uh you could did it did it move that role to the manager- uh and you know- and you know the self-exporter demons- can then be completely local, uh never connect to the cluster um uh itself uh and uh you know be more or less. uh uh You know completely optional uh component uh for the uh just just for Prometheus.

I

B

And uh actually kind of uh one more uh related comment is uh uh when you showed uh the self-exported demon uh help uh in your terminal. uh There was, apart from the options that actually made sense uh for for self-exporter. uh There was a bunch of uh kind of standard options uh that I think uh on the air.

B

Just because uh you've, uh you know you've used uh like you, basically cut and pasted uh it from from some other demon, uh and so things like, for example, the debug underscore Ms, uh that's an option that uh is specific to Seth messenger, and uh you know that's a debug level for the shaft messenger and if the tough exported demon is uh you know completely local and uh interacting with this ceph cluster or interacting with the seven demons only uh via admin sockets, uh it shouldn't be necessary and the same goes for uh you know the the minus minus ID and the minus minus name.

B

The minus minus cluster Etc, et cetera. All of those options uh should not be there because um uh self-exporter should not need them.

I

Not there, but at least there is a config option, debug address which lets you uh yeah. That's delivered level to get the basically the logs.

I

Or like a log level yeah, this makes sense to not having.

B

Well, actually, I see that uh Casey uh mentioned in the in in the comment uh self-exporter does connect uh to the cluster uh in order to get the config.

B

uh Is it like I I, wonder if that's really necessary, though, because uh on the.

B

Well, I guess it may be necessary for the uh for to to get the uh desired, um uh like Turf counter dump level, uh which currently defaults to to five, uh and uh is that the reason or I'm, just not following uh the the like, what what would self-exported demon actually need there? Apart from uh this particular setting, Maybe.

I

Yeah, so basically, it's being implemented as a self-cleaned even and um to expose these options with this uh having a demon, config so I'm, not sure it's uh having a global config. So it's not uh so yeah in this F1 I would say you would have to set for that. uh For that particular demon, you will have to set that config instead of the monconfig.

B

Yes, but the question is: is access to that config even necessary? You know with a global or local. uh That's uh you know, that's uh that's a stuff, yeah I guess further. The question is: is it even needed in the first place.

I

Yeah, so that's how it's uh I'm installing this like. If you want to get this prior limit, for example, so or someone you can even set it, uh this self config option um and change the priority so yeah, so so next in the next Loop or and when this once that period is complete, then it will yeah yeah. It will, if it.

B

Okay, so so you're, basically seeing that current leads done uh for to be able to get uh the prior limit value uh from the centralized uh config store on the monitors. And that's the only thing that the self-exporter Daemon is uh interested in.

B

Or is there something else like that's basically my question because I'm not clear on what else may be needed uh uh-huh if it's just the prior limit, like the the the quote-unquote log level uh that that's fine I was just wondering if you were looking.

F

B

Know some other configuration options uh because uh uh you know and what the use case for that was uh if, if any.

I

Yeah so yeah during.

O

Review I had requested that most of the command line options were added as config variables so that they could be overwritten by the man database and updated at runtime. Some things like the addresses don't really apply to that, but.

O

Being able to configure log levels and the prior limit I think is useful.

I

B

That makes sense in that case, those options uh should should, of course, be there. uh The the the ones that I picked on in the health in the health output.

A

Apologies I had to drop for a moment because I had never because she's did I miss something in the last two seconds.

B

uh No I don't think so. uh My my question was uh resolved and I just commented that the the options in the uh in the help output that I picked on uh there is like they should be there, because uh the self-exporter is actually connected to the monitors. After all, you know, with the exception of debug underscore Ms, maybe everything else makes sense.

A

About the suffix Porter demo, um again I've I've written down some of the questions and answers that were given um on either pad. So, uh if you have any more questions for Avan um feel free to add them to that ether PAD as well, and he can pick it up later, and that goes for anybody who presented today.

A

All right um we're a little bit over time, but we still have one more topic, so I I think we can get through this last topic. um If you have to go, that's totally fine, um but for those who'd like to stay, this last topic is um from Peter and uh it's about an rbd-like virtual disk.

P

Okay, so I'll make this really quick, um so I'll copy three things in the uh yeah Three Links into the chat. This is a paper we presented at um at Euros. Is this spring we've been working with red hat on a um you know, an alternate um log structure, virtual volume, and it's it's basically.

P

You know it's it, it stores on. So the idea is that, rather than splitting it into a bunch of modifiable objects, we actually do essentially an object, translation layer where we have a series of much like a flash translation layer. We've got a series of objects that we write and we don't modify them and then, sometime later, as data gets, it gets outdated. We garbage collect and delete so we're batching rights and and writing them out in something similar to the X4 Journal format.

P

You know, basically a header saying what the lbas in the following blocks are, and we use a map from LBA to a location in this stream to to handle reads, and then we also have a local cache where we Journal data, um you know locally using a similar format, and so the key thing is that we're able to keep the um the cache and the back end uh consistent, even across cap, even across crashes, because we were keeping rights ordered in both cases. So at this point we have a proof of concept. That's uh well.

P

We had the proof of concept for the for the paper, but you know grad student codes for paper. Often is you know a long ways from something that you can get? uh You know that you might be able to use in real life. We have a aversion replicating the RBD live RBD interface. uh That's close to the point of being testable. You know it runs bio, it's so on so we're hoping to you know to move forward with uh with with showing uh you know with with getting that code out there and showing it.

P

G

Know the the key.

P

Thing about it is that by batching rights, since rights are basically the most expensive operation on the back end devices, um you know disc greatly reduces the iops workload on the devices underneath the Seth cluster. So you know if your cluster is all.

P

Okay yeah, so there's a couple of issues here. Yes, the LBA map is periodically um checkpointed, so we only have to you.

I

Know serialize.

P

And save so rebuilding it. Isn't um that big a deal sequential read performance? Is um you know right now we don't have we have some? We don't have a wide enough range of workload tests to tell whether it ever runs into significant trouble due to read fragmentation. We do have um a defragmentation mechanism of the garbage collector, which seems to perform pretty well on the real world trade system.

P

That's, that's the you know three-minute overview and, uh given the time I think I should probably uh so garbage collection strategy is well. One is that we garbage collect slowly since um and essentially allow the pool, allow a volume under heavy right load to use.

P

You know more storage than um you know to to basically be over Provisions by quite a lot and then catch up we'll point out. Yes, we.

P

One thing about this is that it works well with Erasure coated pools which actually give us higher batch rights throughput, so we're saving. You know factors to right there um and then it's it stops collecting in large batches uh and it's a generational collector which, which tends to give a better right amplification because it keeps it avoids polluting old data with recently written data.

P

P

um And finally, uh the because of the cash, a lot of the garbage collection can be done without having to to without having to fetch data.

P

um The in-memory state is actually proportional to the fragmentation of the map, because it's an effect map so far, we've seen that you know, with with virtual machine block traces, uh the the map Size Doesn't grow. Excessively, it's in the you know, 100, you know 100 megabyte range, for you know for a terabyte volume.

A

Awesome thanks so much for that rundown um I uh and thanks Sam for your questions. So it looks really interesting questions.

A

Does anybody yeah? Does anybody have any questions or comments for Peter.

P

And feel free to contact me if you have further questions. I'll put my email, my I'm at red hat for the fall, but that address may not last past the end of the semester. Faculty address.

A

Thanks Peter I'll, add this um to The Ether pad under your topic and also for anybody that has questions like after the meeting um feel free to add it to the ether pad for Peter. Thank you so much.

B

Yeah and just to note that uh Peter Peter um uh and uh you know, has been in touch with the RBD team uh on this, uh and um uh we are really excited to kind of see, move forward and actually kind of graduate from the from the grad student code to actually something that is now. You know resembling a shared library that can be uh dropped into uh things like fio and qmu for some actual workload testing, then that has that has come a long way uh and uh yep.

B

uh If you know, feel free to reach out to me as well and uh I I can forward um uh anything. You know um I I'm, not aware of I can't answer uh to Peter.

N

Probably running over I have a couple of longer term questions.

A

Do you plan on uh adding do you want to ask them here or do you want to add this and.

N

Well, one of them is this would be an interesting or colder lbas. You could write them to a different pool with the strategy.

P

G

N

Would be an interesting adaptation of the future, that's something that would be very difficult to do with RBD in its conventional form.

N

um The other thing is that uh if you had a direct channel to the osds in question, this would be a metabol to a much weaker pool consistency model.

N

For instance, if you had a direct nvme or fabric channel to the backing disks, you could write independent logs to the Target osds, and the osds could be responsible for checking consistency if the client dies.

N

Oh, that that was the end of my observations. I think those two possibilities are exciting.

P

Right this also has a clone mechanism. Where you could do the same thing. You could put the Clone base in a different pool and we actually have. We haven't implemented any code, but it should be possible to share the read cache between multiple virtual disks, which you know in most of the. If they're independent wouldn't be useful, but would be useful if they're, if they're.

P

It's um in fact the the original the original uh you know goal of this was a virtual disk over S3, so it is in fact um you.

G

Know it treats the objects.

P

As purely right, once um you know for using it in for using it in Seth uh the S3, you know read overhead: is uh it's a bit high, so it's much more effective to do it just straight over radius.

A

Awesome Sam: did you have any more oh, go ahead.

A

With somebody trying to talk.

A

Okay, um awesome uh did anybody else, have any more follow-up questions for Peter.

A

All right awesome, it seems like people are really excited, I'm excited. So uh thank you so much for sharing today, Peter.

A

Also, I'm really glad to see the communication. That's already been ongoing between uh the RBD team and uh and you that's really great- to hear.

P

A

A lot thank you and with that uh that concludes uh the October CDM, so I'd say it was pretty successful thanks everybody uh for who attended and presented uh great job and I'll, see you guys next month.

A

Bye, everyone have a great rest of your day.

E

Thanks everyone. Thank you.

F