GitLab Scalability Team Demos, 7 Jan 2020

Previous Meeting

⏯

youtube image

►

From YouTube: 2020-01-07 Background processing kick-off demo

Description

Part of https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/96

A

All right, well hi, everyone to the first inaugural demo for this team. Let's see how its go, but I am going to give you a short intro on what is expected here. So basically, the expectation here is low level technical discussion that is developing around the work that was accomplished throughout the week. Whoever has something to show can put themselves on there and we can go and discuss what value does it have and if it has any value how to increase it or if it doesn't.

A

Why did why was it done, then, more importantly, with whatever demo we do here, it is also important to just go through the motion of. If this is something that you're showing is hard, you should do it regardless of how hard it is. So if something is not working properly, that is the point to highlight, and we want to see like whether there are better ideas or whether we can do something a bit better as a whole team.

A

This has been successful in the two other teams. I was running and at the beginning it is a bit awkward until you get into it and then, when you get into it, there is a lot of value. You extract from just understanding where others are in their work and what kind of work can be improved. I might be jumping in to ask questions. Some of them are gonna be well. Actually, most of them are going to be very dumb, but I'm hoping I'm gonna squeeze in a nice question directly as well.

A

Everyone is encouraged to stop folks and ask questions and do the same thing over and over again until we get to the bottom of it for now, once a week, we'll do Fridays most likely from from next week. Let's see how this goes first, so with that andrew has a first demo. He wants to share so Andrew share your screen and take it away.

B

So it's actually pretty old thing that I started working on probably beginning in December, and it got delayed because we realized that the Prometheus metrics for sidekick were basically terrible and pretty much random.

B

So you might find metrics for one psychic instance or one psychic process inside cluster, but none of the others, and so at that point it kind of didn't really make any sense to continue with it, but then think Mattias in the memory team fixed that and so so you now have metrics and sidekick, and so I can continue with us since I started finishing off today, and so basically all it is was it's kind of like a first step in the direction of error.

B

Budgeting, so I think that one of the things that's going to be really important with this team is trying to kind of push, lift and and push actually, you know basically figure out attribution like who can we who can help us with with particular problems that we see in the code and which teams do?

B

We need to speak to you, and so with that in mind, who started attributing all of the sidekick workers, and instead of attributing them with teams, we decided to actually eat them with the teacher category because features sort of sidekick workers are pretty well sort of tied in his features for teens tend to move around a lot and different teams together different features, so it seemed like a better fit.

B

Instead of saying that this team is responsible for this sidekick worker, we can say this feature is associated with the psychic worker and then, as the team's change around we can go. Look at this stage is nothing file, the stages document in the handbook and that helps us map. It's a team so in here.

C

B

Say you know if I've taken words that you know if I say: oh, we've got a problem with an incident management feature. Actually, let's take her I'll, explain it better in a second. Let me say: are the pages feature 70% of the time is giving us errors. We can go back in here and we will see that the pages feature person to speak to is probably Davi frame, so he's the person that we need to go and speak to about that, but so giving a quick.

A

Demo, actually it's not go back yeah. So if you go it's groups and then its release management, section.

B

Yeah John John Hanson, you thank you yeah. So that's that's Tory! Yes, good! Yes,.

D

B

But we can we we can totally sort of you know. We've got this look up now, so we can basically say we have this. Each worker has a feature category and then we can map effector team and in future we could start sending alerts to channel specific to teams where we could say hey like.

B

Could you guys start going to call with us because we're seeing a problem with this feature so going back here, I guess may be a good place to start is by kind of explaining the attributes that we've got on every worker, and so could you.

D

Show me cool how you get them into graph Anna, because I know how they live on the workers and how they're added there now. But how does that get here?.

B

But that's very lucky: I actually happen to be in exactly the right place. I think.

B

B

Ladies, so, basically on you know, we added the the attributes and then we added some functions to to the classes like get feature. Category latency-sensitive worker question mark how rebus specifically say the question mark at the end, when you, when you're talking about a method, that's got a question mark on it. Would you just ask say it in a question: II kind of way go.

D

For the question you kind of wait: I want I care, okay,.

B

So working class has external dependencies. Work, has external dependencies but yeah, so so, basically, the the classes have these methods on them. So we know that the classes you know whatever Mahler or whatever, and so we can. We can pull these off. Obviously it's only if the class includes work, attributes which we test and funny story, but the first version and let's go all the way into staging without that and didn't work very well, but we fix it but anyway, so we take those labels and we apply them on to.

B

Joe Sakic jobs compute in seconds. So if you go take a look at this in for me. Yes, he looks. These have now got a lot of labels on them. It's it's. It doesn't look great, but it doesn't add to the cardinality, because each worker only has one specific label, so we're not kind of getting this hot nasty explosion. But it's kind of.

D

Because the class was already there anyway.

B

Yeah, the class is already there, so we're not multiplying it's not like each one is like got a yes and a node for latency-sensitive, so kind of.

D

Why do you use this one enough? The palace was.

B

D

Dennis today, but.

B

I'm gonna go to the famous one now well.

D

No I was asking because it was very slow for me just a few seconds ago. Yeah.

B

Yeah and- and that is pretty much why I spend quite a lot of time just going to the individual ones. Sadly, um and then so you can see here, we get you know this particular one is the new epic you that we're talking about cheers so I, guess that's what, when you create a new epic I'll, get lab it's falling, something on the new epic. You and you can see here- we've got all these attributes no like doesn't have external dependencies.

B

No, what's the feature, category, agile portfolio management and then obviously, we've got like whether the job succeeded or failed, whether it's and it's not latency sensitive and then the priority and and all the other attributes that we use out before. But we get these new attributes, and so we can aggregate everything up according to those attributes, and that gives us this kind of first version of this dashboard and so technically what we can I haven't been able to I'd like to sort this every time by, like worst at the top.

B

For some reason, I can't get tables in refinement to do that. um So obviously, the two sort of columns in this in this dashboard, the first one- is apdex, which is your what percent effectively what percentage of your requests complete within an acceptable amount of time? So we want that to be a hundred percent and the lower it is the worse it is, and then this is what percentage of your requests will. Workers in this case end in inner error and obviously for that we want lower, is better so the to a slightly different.

B

But if we roll it up by feature category, we can see here that's five percent of of things that are not owned or don't need their latency requirements and that's kind of not surprising, like obviously that the stuff that nobody owns is always stuff. That's gonna perform the worst, and when we did all that work attribution, one of the things we said was we should really aim to to to reduce that down to zero or close to zero overtime.

B

But it's kind of interesting that as soon as we put it on those are the things that that aren't making there.

B

The other thing to point out is that the values that I've been used are kind of, like I just did a thumb suck on them and we should probably discuss them with the wider audience, and so what I'm saying is that, if something, if we declare that something's latency-sensitive, we say that it needs to finish within ten seconds. So we this was a discussion. I was having with Sean on Friday I. Think we're saying the infrastructure team will will make sure that that your job starts on time.

B

If it's a latency, sensitive job, we'll make sure it starts on time. As long as the engineering teams that develop that job make sure that it finishes quickly like we can't have things that take ten minutes to run, and we it will have something takes ten minutes to run. We can't guarantee that we can start it. You know in five milliseconds, so so what we're saying here is that for things that are latency sensitive, they need to finish within 10 seconds and things that are not really sensitive.

B

They need to finish within five minutes on average and if we, if we aggregate that up, you can see here that you know other than our own. The next slowest thing is container registry and one of the cool things you can do define an hour is you can actually click on that and it will filter- and you can see here that container registry has put these two jobs, and this is the worst one. Performance-Wise soda needs contain a repository whatever that that's so we go. Take a look at that we're going to hear well.

B

Actually, no is it latency-sensitive? No, so that actually tells us that that's there's three percent of cases that's taking longer than five minutes which, like to delete, contain a repository. That seems like quite a long time.

B

So maybe that is something that's worth: investigating, I, don't I, don't know, I mean I know everything around the registry is really slow, so maybe that's something you've just gotta kind of except for now, but anyway, that's what that is, and then I still haven't quite figured out how to remove filters with this for fun and this new fire engine face so I just remove it from the gyro which isn't very good drink.

A

So really funny Andrew, because you just mentioned registry and I searched to my inbox and I, find a three months old issue raised by guess who Andrew saying that the same thing it is performing really badly and last update was last night from one of the directors saying we need to schedule this investigation, so yeh.

B

In like everything with the registry, everything that took Stu docker, slash distribution seems to be really slow. So if you look at the endpoints on the API that talk to it, it's the same so.

D

Obviously the next.

B

Step with this is obviously to do the same thing with routes or controllers and actions, perhaps, and then once we've got that we can, we can have a much more overall view because obviously sidekick is only part of the system. Once we have everything, then we can start prioritizing and maybe using this as a as a narrow budget, and so saying you know the container.

E

B

Like that team perhaps needs to consider spending a little bit more time dressing technical debt over the next month or two, because you know they're the team that that has consistently the worse tactics cause.

B

Obviously, the one thing that this doesn't take into account at the moment is that we not weighing this by the number of requests, so it might be that this you know, I had that issue with the unit's is go back to this delete container repository. Now we saying that in three percent of the case took more than five minutes. Maybe they were only like seven requests in the last 24 hours and it doesn't make sense for that team to kind of think in the next month's. Focusing on this, because it's you know.

C

B

Happens every now and again, but it's not that bad and so before this becomes like a proper Eric budgeting tool for us to like help drive those discussions. We need to include some sort of waiting so that we don't kind of like say: oh there's this thing that happens once a week and every time it runs its slow hundred percent of the time it doesn't achieve its applix cool. You know it probably doesn't matter, there's another case of that. Where there's a feature, category called license management and like 50% at the time that it runs.

B

It ends in error, but it only runs like twice a day. So is that a problem or not I, don't know, but it's it's not a high priority right like what's what's more interesting, it's interesting because obviously it's run since then, but you know pages, you know, look at this and there's a thing called yeah. This pages domain, SSL renewal and also the other thing you can do is you can actually click through? So it's quite nicely you can click through from that table. It'll. Take you here.

D

Doesn't have external dependencies but I'm pretty sure that it should have oh okay, yeah.

B

That that's that makes sense.

D

Because I guess.

B

It's what's it talking to for the the SSL renewals, it's talking to. Let's.

A

Encrypt most likely yeah.

D

B

But I mean the error like 33 percent of those engineers right and so that's.

A

B

A

Not shocked because let's encrypt is very specific about what can pass in cans. So if you don't pay special attention to that renewal, 90% of the time it's going to end in error, if, if the user doesn't take an action, so the.

B

Other thing I've learned with, let's encrypt, is, if you get things wrong too often you get blacklisted very quickly, yeah and then you've got problems right, yep.

A

Let's examine one of my pet peeves that I had when we built in the Omnibus support. Yes,.

B

I've made that mistake: it's like use the staging server until you absolutely sure you know anything. It's like I know what I'm doing I'm bad now, but you know so. This is kind of maybe help drive that conversation with the team and sort of like help us define error budgets is very very early at the moment, but it kind of feels like a first step and I. Guess, that's the end of my demo. Does anyone have any questions? Other questions.

C

When you had this unfiltered a second ago, there was feature category not owned, but there was also feature category blank. Yes,.

B

I looked at that that is because of the active job : :, oh.

C

The mailers mailers yeah.

B

It's that one and I don't really know what to do about that. At the moment, I was I was kind of deciding whether yeah I think.

D

That I think we could emerge that with not own, because yeah.

B

D

Belong to somebody, but all mailers is one of these things that yeah yeah.

B

D

I'll just do a label.

B

Replace for the moments on the on the client side and just kind of treats it as a yeah.

C

I think it could also apply to anything that sort of you know done with without an explicit worker class, but I think mailers is the only place. We use that right now. Yeah.

B

I I went looking, and that was the only thing.

B

A

Do we know whether there is any validation inside of gitlab like true danger or whatever else that specifies what is like? If you create a new worker? What is the structure you have? We have tests.

D

We have tests for now.

A

D

This kind of worker is like within email, so it's not the source code. That's why my.

B

First, my first thing was like: oh, it's somebody's created a new worker and they haven't added attributes and then I remembered that me specifically added a test for that which, which is yeah. The filter, doesn't seem to work very well.

B

It is attractive in its manners. Yeah.

C

And how do we keep track if a featured category changes light, not changes? Ownership I mean just changes. Like you know, say we changed importers to something else, I think they're supposed to be pretty stable, but yeah.

B

C

When I start mr. memory loss, yeah.

B

There isn't anything I mean if you were to look over depending on the time frame that you used. It would probably well. What would happen is that for the time that it was you know where prometheus would we'll handle. It is that during the time that it was belong to one feature, category would be attributed there and then we'd switch over. So you know 50 percent.

B

At the time of this 24 hours, it was owned by you know continuous integration, and then another 50% of the Sun was learned by Cuban eighties configuration the attribution would sort of be 50/50, but there's nothing really, oh, and you get the other things shown. Is that you get double you'd, see it twice in this list, you'd see a dip because sorry it would be yeah. Sorry.

C

My question was slightly different, like okay, if we, if we change the importers, the category to be just like named something else in that website repo, yes, do we know to update that automatically.

B

Yes, so what would happen? Is that there's a it's not totally automatic? What happens is that we import the stages, the mo file, occasionally not on every single build, but there's a there's a script on the get lab just shots.

B

You think I'm answering the wrong question again, but in the github repository there's a script which will take the stage Azam will file and generate a kind of intermediate form off, basically, which is effectively enlisted future categories, and then the CI script will validate the CIA job will validate that the future categories match up with that and kind of the idea is that every now and again you go and run the script and get a new list of future categories. And then you check that in a menu.

E

And you go alternative.

B

Is is, we could do it on every build, but it kind of chose I didn't want to break the get our build. Sorry, the get leveled when the handbook page was broken and, for example, can.

A

You go and show that right now you know github repo, so we know where it is so.

B

In the it looks like a conflict, conflict feature categories and then at the top it's got this header. This file contains a list of all future calories in gitlab.

B

It is generated from the stages file at the location. You would like to updated, peas run, and so the alternative is that we could just generate this on every CI build um I, just kind of felt at the time that it made sense to not like break the gitlab build if something was wrong with the handbook and it kind of gives you a bit of a buffer between you know, because, obviously the whole world and his wife are working on me on the future categories.

B

I file, you know that gets changed every day and something could break in there and we don't want that to break the gitlab build.

B

So you know the idea is that this will get run I'm. You know be interesting to run it now and see if it's has changed. Those peach academies are so stable from the last five minutes. Yeah.

C

Well, I think that answers my question anyway. So like yeah, thank you so.

B

You can see that since the last time we ran it, we've got a whole look at that is actually epics future category now, and so, ideally, that one we were looking at, which was under agile project management should probably go under epics yeah.

C

I didn't actually know that was gonna change. I was just curious, yeah.

B

No, it's it's and agile project management no longer exists. So if I check this file in now the build would break. So that's been kind of interesting.

A

How do we ensure that we stay is updated and then periodically should.

D

A

D

That runs, and shout some slack, so we can take a look at it.

A

D

B

That's not a bad idea, something.

D

Like that, because, like in this case, a human will need to take a look at it and fix yeah.

B

And fix the yeah, because that's another thing is: if we don't want to get that, build to break just because products have made a product decision: yeah. Okay, so that's that's, probably a takeaways. We should.

A

Create a nation.

B

B

B

Yeah I know how yeah we can just add a like a schedule. Well, presumably, there's a schedule build in get lab project.

A

There are a bunch as far as I know, cool anything else, you'd like to show Andrew no.

B

That's great sharing.

A

Time, well, we are almost at allocated time. I just wanted to check with you all. Are you okay, with going a bit over like 15 minutes over.

D

Okay, kinda part, with a with what I was going to show you right after this anyway, so we're all.

A

Right, let's go for it.

D

Share my screen.

D

Okay, so the thing that I've been working on Sean and Andrew, probably seen is adding kind of metadata tags to our psychic logs. We did that because, right now, the only way Sree could know which projects job was running for which used their triggered a job or anything like that was to know kind of what the arguments like they need to know the code and the order of the arguments to know which IDs were being used and so on. So to get around that we added metadata tags on top of the jobs.

D

When we look into the Cabana. Those are added.

D

To the sidekick log already, we could add them to the the West node and so on as well, but right now they're only here and they're all these that start with meta. So these percolates project root, namespace and user. It's going to add these I.

C

D

Know we're gonna.

C

Make this public on YouTube, but we can't now just point out.

A

Nowhere don't worry about it. I handled that by scrubbing things out when I upload.

D

So now you scrub things out manually, yeah.

A

I think the video and I'll use this soap and I'm joking okay,.

D

So like this allows us to choose to see which things are running, which user triggered them inside cake, which queue that is and so forth. So that's visible here and that allows us to and build like some visualizations, the one that I had here. But it's gone now so.

D

What do I have.

D

I'm always confused by sidekick terms.

B

And terms yeah.

D

So if you want to see how many jobs are running for that project, you could filter for that.

D

And then yeah we can see, wait, I'm going to add some more because, like now we will have double information. So what is it what's? The field named where the state is a stasis, no papers, I.

B

D

So this will show us like which projects are like in the last 50 minutes have scheduled a lot of jobs. For example, we could add.

D

The few into that.

D

To see if something weird comes out and I expect something weird to come out, for example, the.

D

It's too bad. It doesn't show up right now the logical beam clear it because we had a bargain there, as you probably know already, but perhaps it was fixed, like Jim a recent deploy. Now you take longer.

D

B

Just already so incredibly useful yeah.

C

So much I, don't I, don't know if we actually told anybody about this, but Stan already used this in something that he shared so I think Chris having it having it there, people see it and they're like oh that's handy, they don't I mean it's always a great to tell people about it, but even just having it. There means that you know you can just yum.

B

C

I find it yeah.

D

Let us, as you know, like the job we had yesterday and that we worked around quite quickly was that you would attribute mirrors to a project that would have triggered an update, which is not my type of project receiver Webb, who that would prioritize its mirror, update then yeah, the updates that get scheduled, because they were even more urgent and others would be attributed to that project, which is obviously incorrect them yeah. So we fix that yesterday, IFIF.

E

We having a video like showing how to create futures visualizations on Gabbana. Maybe we could even create like a five-minute video showing how to filter using those three new attributes, nd.

D

You know what would be valuable like what would be a thing that you look up like that before here I before this call, I was like making a visualization to see which are the top root namespaces, that yeah.

B

D

Stuff gitlab like creating stuff that was left remote mirrors because that's invalid, so philipose was a lot there and then the thing that you were complaining about the daily statistics. Something is like way up there. So yeah, that's the kind of stuff that could drop out of that.

B

And this, and as well though there is this I think somebody is a Java had a issue.

C

D

We can now see the bug that I was talking about I added the root namespace here, and we can see that it has a whole bunch of repository update, mirror jobs that are attributed to something in this namespace. But that's why accident, probably because yeah.

D

Web group from github and then trigger to sink figure. This thing for all other projects like for now I just fill them out, but.

B

As soon as you bug fixes in production, yakking kind of this isn't gonna be.

B

Seeing a RIT namespace there I'm wondering whether they should be Maki customer women's brand.

D

So yeah now we can. This is more valuable. So, for example, now we see that this namespace that we saw before the garbage collect is also going to be a consequence of that.

B

C

D

B

Do you mean no there's that I mean you can also click on the on the Anton's like in the down side like that, and then do you like the negates on it? Yeah.

B

D

Right now, this is more in line with what we expect get my Borg, as I could biggest one with most keep with the most workers and mailers is always present days. I've noticed, and then this one is the one that Andrew was talking about before, which is stated, statistics which now ran a thousand something times in the past. 15 minutes.

B

Yeah, just on that one namespace, just on that.

D

One namespace, yes, pretty active queue. This also like shows that all this really curse for the for the all the namespace, the top five main spaces we run around so I. Think, though, elevating a pretty cool yeah should I show you. Maybe that's mostly everything for us. Well go, but I could show how how we add this, how we add this metadata to the jobs, which would also explain why the bug was there before.

A

You good and do that, which you think we should Bob I. Think I would like you to take an action item to record one five-minute, video for like just focusing on this, for something that we could use to present to a series like in the infra coal I could link that and then like really this information further so like asthma, take it around books also.

B

The DS team are gonna love this stuff yeah.

D

I also thought that once we have it like a little bit more confirmed like with the bug fixed and all that, and it would be good to mention it in the background channel, because today also care which I like how their jobs are doing. Yeah yeah.

A

Just record one five minute: video tomorrow, for example, because right now the your change is being deployed, so theoretically should be working right, and if it does, then you can just focus on on real, quick here's how you do this.

D

So the code will.

E

Be interesting to understand like if someone wants to add a new metadata to the context, how we could do that, like.

E

D

And get lap rails, we have this application context, file and.

D

Currently, these are the attributes that are known there, so use a project namespace, and these things are.

D

Good lad rail specific, so they expect object of these types. If you want to add something there, and you would also have to add at the lab tips, which has a complex not I'll, be with. If.

E

You make kind of a type checked, quote-unquote, yeah.

D

So it's not kind of a pipe like it is a bike check is.

D

The the file man on the left here is like the way the complex looks like for rails, that knows about namespaces the objects and so on. While lab kid only knows about primitive, like the strings that are going to be written down into logs and yeah, which you.

D

So, if you add a new value, you will need to add it in these two file and one of the values that you are going to be adding this probably likes the plan, so that would be like of the type plan here, but here that's just a string that make sense if the way the these things get pushed is kind of connected to the correlation ID.

D

So when a request comes in, we can record the correlation ID and we generate a context like the one here and let me create with one of the attributes of which is a correlation ID.

D

You can keep nesting these complex to like specify things or remove things, and so on. There's a rack middleware that make sure that the complex gets thrown away after the request is done.

D

The way we add this information right now. This happens in only two places. This happens in the application controller.

E

D

So it just sets the the namespaces that the attributes based on what we think that the variable name inside the controller should see so.

E

This this piece yeah so.

D

Inside application complex, for example, if we don't have a group variable, then we are going to fall back like a root. Namespace is then going to be the truth. Namespace of the project and I selected fixed like that, so we don't always need those variables to be set for the plan. I think you can do something like that.

D

We also set this from the grape endpoints in a similar way.

D

Think I'll it's here, yeah.

D

So that's where it gets pushed it's the complex gets pushed off and the API doesn't get popped. We count on a middleware to clean up after so there. The way it gets into the logs is in the sidekick.

D

There's a flight middleware that will add the current context to the job. This is this job object. Here is a hash that gets written out to psychic, to Redis for the psychic server to pick it up and then all the server we reinstated here with them. So everything that happens within this job would have the same complex like that. Yep.

D

If anything more work.

A

So there is one actionable item that I would like to suggest from from this section feel free to reject it. Obviously, I don't recall, seeing in the merge request that you created earlier right, the the main one I, don't recall, seeing development documentation.

D

Because right now there is nothing to be done. Okay,.

A

D

A

D

You trying to.

A

Suggest here is that as well. The goes to it goes in to add new items here. It would be great if we document this in our development documentation on how to add.

E

Stuff, like we have neither words, we have a few components, separated companies so maybe describing what each company? What do you want the process? What.

D

Do we want to document here?

D

Do we want to document how people can change these values like if they know they're going to be working on a different context within a request, for whatever reason we can document how they should be using what we built to to articulate the context, or should we be documenting what the context looks like, which is what is always going to change the line, which is what is going to change I, think the latter is more important to us and I think that's only going to be changed and so on by us. I think.

C

So that's as important for developers to know like how this all might they don't need to know exactly how it works, but they need to know like you know, this is how fields get added here. This is how, like you know, this is how you could add a field if you wanted to add one stuff like that, so I think and then.

A

Exactly and then for each and every one of the places which you showed right now, I would like to see that doc linked in the comments, because there are already comments in there so just add another line that links to the documentation once we have it, so that we can make sure that whoever reads this again, it doesn't have to look through the video but can also find the documentation and as well, though this would be your task because you're already going to be working on this right now.

A

So you can also document this while you're working.

D

Like while you're doing it, you're probably going to encounter things that I forgot to mention now, so that's probably good idea should I take the second one or I.

A

Would I would give us all to both at the moment, because he's dealing with the whole thing and you can be the reviewer go awesome.

D

Anything more who wanted to know around this. The next thing that I'm going to do where, like we're, still lacking a bit like the thing that I mentioned the bug with the mirrors. That thing like that happened because I assume that update all mirrors would only happen from like or cron job and would therefore not have metadata. But it also got triggered from requests within a project and that's why it does have metadata. And so one of the next steps that I'm going to do is add metadata for cron jobs. Based on my arguments and.

E

Yeah we have an issue for that right, yeah.

D

E

Jobs, starting on prone to keep the context at the end.

D

A

Time so I would I don't want to take more of your time because we already went over well, maybe write down your thoughts for the next demo and I know that there's some items that Shawn edits and Andrew. So, let's, let's cover those next next time.

D

A