GitLab Manage:Workspace Group, 24 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Workspace weekly sync 2021-11-24

Description

2021-11-24

A

A

Oh I just this was the plan, sorry about the uh the problem of the date in the agenda. I just fix it.

A

Okay, so maybe I'll start real, quick, there's, not much update for me. I just added a couple of issues from from workspaces to from workspace to the planning issue. So please, let me know if the scope is too big or too small. uh There are things that we are currently working on so backfilling and other things that are, as I saw, are in in in the development right now and three issues from um phase two which are around routing.

A

If you think that this is not a good place to start with the phase two, let me know but uh yeah this is what's uh so. Please give me some feedback on hispanic issue. If there's any anything that I can do to make it better.

A

The other thing for me is that I have a question around issue about unifying policies, because I saw that charlie did some research on that and I was asking like if it's ready to make issues out of it or we need more more time to to research. Some of these problems.

A

So please, let me know.

B

Yeah regarding the the police.

C

Can you hear me.

A

C

I'm sure I can hear you again.

C

Not exactly it was okay, it's just very.

B

Regarding the policy breakdown, I think that the charles approach makes more sense, but I did some comment yesterday, but some more feedback would be really cool.

B

So if you have time, please take a look at it.

A

Okay, I will take a look at this and then create issues. I already took the routing issue and create the issues out of the.

C

A

That alexandria created before so I just like copy pasted it into the issues. So we have uh it's like easier to assign assign work. Those issues.

A

So I will do the same thing with the with the one from from charlie, around policies.

B

B

B

Maybe alexander my descriptor, but yesterday he he did some performance testing on the hardware and it seems that we are hitting the maximums for the hardware there.

B

So I wonder if it just makes sense to try again with bigger setup or if we have even something else available. Instead of this testing environment.

A

This is the question to send out. Probably I don't know.

C

Yeah we spoke to grant and they said that even the I think he said that even the 50k architecture that is being used for for performance testing would not exactly handle the amount of groups and projects that we created. So we created a data set of a million groups with five projects in each group.

C

So it's five million projects and obviously that, like we've, we've seen some some slowness to say the least, and I asked if he couldn't recommend one of the um architectures that we already have in qa and he said like we will probably need something closer to production kind of environment.

C

I don't know how the qa architectures for 50k or 10k are constructed. What kind of machines we are using there, what resources they have, but it sounded like those will not exactly be able to to handle that as well. So what we are doing right now is we I'm going to run actually running right now, a performance cap on a smaller data set 10 000.

C

uh Ten thousand groups and about 50k projects and we'll see what we get out of that and then slowly increase to 100k and so on and see if we can get more data. um I also have figured we've been able to track slow queries and we'll see if we get anything out of of the slow queries there.

C

We may not, because just the data set is so small that everything may just run smoothly, but we'll see I don't I don't. No, I don't have any predictions basically.

C

One other thing is that the performance tests don't actually test any post requests. Most of it is doing guests so yeah I don't. We will have to analyze what we get and then decide if we need to do additional testing on the creation of resources. Basically,.

A

So we don't have any performances on just we just have different tests on, gets nothing on.

C

Yeah, that seems what we are seems to be what we are using to do the performance tests that that we do for the 3k 10k like 50k architectures, we mostly do api gets and web gets, but yeah I didn't see- maybe I'm maybe I'm just missing it, but I didn't see any any posts in those tests.

A

Okay, okay, can we ask like sanat to if there's a possibility to have like better hardware like better hardware for the machines, so we can actually like get this speed test on the bigger sets like no one millionaires.

C

Don't think about how closely we can go right. We cannot spin up another gitlab environment because that'll that'll be too expensive, so I I think, like we can go to a certain limit, but not something beyond that. So yeah, I don't know, I don't even know if 50k is something that we something we usually do right. I, I guess that's something fairly expensive already to run, but I don't have the data it's better to ask yes sonata and brandon. Whoever is uh managing those. Is that cost effective?

C

Is it okay to use it to test where uh or not.

B

Yeah, I suspect that even if we would use the biggest setup, we won't be anywhere close to the dot-com-like deployment.

B

So I wonder if this is acceptable or something like blocker or if that's the only option we have just use whatever we have and test on it.

A

Okay, there's two things we can do. We can like ask the database team. What are the opinion on this yeah.

B

We already asked: yes, there was transfer that they don't have data for us or they haven't done it. So we may try infrastructure team, but they pinged us back to.

A

Database teams, okay, the other thing that we can do is um okay, it's something that I can do. I can ask around because we are like. I know that they're currently working on new staging environment- and I know how and which is supposed to be like closer to production.

A

So maybe they are already thinking about like big data sets there, and I can like see how how advanced are the works on that? But I don't believe, like I haven't heard it- it's like very close to being done. So uh maybe we cannot wait for that.

B

One more thought: uh I'm a little bit worried of.

B

Basically all this performance testing is for the only purpose, and it's for migrating production. I mean.com.

B

So I wonder how much time we should spend on this, whether it's next x weeks or I don't know how much or how much we want to risk or how perfect these tests are expected to be, because, because I can imagine, we can spend another many man many weeks on this.

B

If and we won't have production like this think anyway,.

A

But like have anyone like convert with database team about like what is acceptable for them like when they are like ready to sign off this work because, like like, because like they are responsible for like database and the database load so like um yeah like because maybe they are okay with like performances we have now.

A

But it's a question for them, and I would like to like understand how how they are approaching this like because, like I believe that I'm sure there are companies that you know are dealing with largely data sets that we are, but I just want to make sure that they are like accepting the risks plus.

A

The other thing is, like you said that you want to first uh make the migration only for like one group, for example, like github group, which is also quite big and then uh which is, of course not perfect, because it won't uh mimic the load that we are going to create in the final run. But at least uh this is something that we can try.

B

Yeah yeah, that's a that's a good point to check with database team cool. uh Do you plan to check with them or or should we should we uh ping them in the testing environment issue.

A

Please ping them on the testing environment issue, okay, so yeah. They need to know what's going on there and like in the issue. There are all the information that we that we have right.

C

Yeah, I think, after this round of testing, especially with the slow queries enabled I've set a threshold of one second, so we should be catching any queries that are slower than one. Second, I don't know that's a good threshold, but that's what I started with um and yeah I'll try to gather as much information as I get from that one. Other idea that I've just thought of is that qa is using this data generator.

C

I think maybe we can use that to kind of test uh the posts, but it doesn't do a a great throughput like it doesn't create a lot of data. So we may not see any slow queries using that. So I I don't know if we can test it in a great way.

B

What do you think about end-to-end tests? I think this should cover the wall stories like also creating resources. Would it make sense to run the end-to-end automated tests on back field, environment and measure.

C

That that doesn't create any performance testing right like I think we do run those anyway in some environments, so we should be seeing failures. I think those are very useful to find um to find scenarios where we missed something or where we have bugs, like. I don't know when creating a project. In some specific scenario, something is not working right right, we're getting exception just because I don't know things uh but like for performance. I don't think those are helpful. We need something that we will do multiple requests.

C

At the same time, um I think I'm not sure I think the generation one, the one that generates data does multiple requests, but I don't I don't know for sure. I.

A

C

Noticed something yesterday but I'll need to recheck and make sure that's true, um so yeah I'll try to ask also grant and see. Maybe maybe there is a setting to say, create like multiple threads or something like that, and then we can. We can see if we hit any kind of performance issue with that um and with the sql log enabled, then we can basically because they don't, like the generator, doesn't give us any results on the performance testing.

C

We can use the fact that we track the slow queries and and try to analyze it that way and kind of make sure that we're not we're not seeing any slowness. Basically.

C

But yeah we can, we can spend like months doing this testing. We will never get to 100 confidence. I don't think um especially just because we cannot run it on a production-like environment with the traffic and all that stuff so like we should. We should decide what's uh a good point, we we need to stop at right and and and say. This is good enough.

C

Let's, let's move now to production and again on production, we'll also do it in in parts we'll pick some group migrate um have a a period to to analyze and and then move from there.

A

Like I would because we have some results now, so I would do the first round of like uh first round of this, of assessing that that performance that we have right now from uh from the database team because, like they need to understand, like what's going on, if they're going to sign up this work, we need to like check with them as soon as possible.

A

Even though, like we, we know that it's not perfect, and it's not even close to our production, uh a lot, but that's all we can get and like. They also need to assess the risk right now.

C

Okay, I'll prepare after because I'm running the tests now and every performance test is running like one and a half hours or something like that. So then I only need some time to compose everything to just to do a reporting and then I'll bring database team on the issue um for them to just for, for the for the heads up and for feedback.

C

um So that will take me it's still running, I think tomorrow. Basically, I guess by the end of the day or or maybe the day after tomorrow, I should have the results and then being the database team, because the results I did post a set of results, but those are not.

C

Those are basically dirty results because we just overloaded the machine with too many groups too many projects it could not handle it. So it's just showing a lot of failures, so those are not usable. We kind of went a bit too aggressive to say so.

A

Okay, so the plan is that I will check the new staging environment if it's ready or not, I would say, probably not, and we will ping the database team on the testing issue while when the our current performances are ready, so they can, they can say like what's what is the next step from their perspective.

B

And it's one more unrelated note, so I got paint about the the resource about group level, resource query, which I worked a month ago for epics, and there was a performance improvement on traversal id. So I plan it to get back to this. If this is fine.

A

While we working on performance review, there's no like other work to do, that's it's fine, but once we are like cleared from database team- and I hope it will be very soon. uh I know that uh we're all like waiting for the phase one to end, so we can move to phase two. So it's also important.

A

Is there anything else we want to discuss right now.

A

Okay, so thanks everybody for joining and for the discussion about conformance tests thanks bye.