GitLab Delivery: GitLab.com migration to k8s demos, 10 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-12-10 GitLab.com k8s migration EMEA

Description

Discussing the progress of current blockers and planning next steps for the GitLab.com Kubernetes migration

A

B

Good morning, sir, how are you.

A

I'm good, how are you.

B

I'm ready for the weekend, otherwise, I'm okay.

A

It's uh it's a whole another day still, but it's coming it's coming soon.

B

I don't, I know.

A

What's going on with aj proxy.

B

What's up with aj proxy.

A

There's a bunch of uh noisy.

A

Well, it's only warning, I think it's. I think we should remove this.

A

Let's take my hats off.

A

Because otherwise, he sees my.

A

C

Oh hello, hello, sorry, my still loving uh zoom, so.

C

D

Awesome thanks for adding in all the updates, comic um super helpful, so.

D

uh First up with reviewing the blockers so up. First, two: three: three: four: um let's go back your comment. uh It's closed, which is great.

D

So next up, um we've got the logging across cloud native uh gitlab installations, so um the way I've uh interpreted this is.

D

That only one there's three issues.

D

C

B

It sounds like because we decided to revert this. um It looks like distribution needs to revisit how they want to perform. The logging work as a whole.

B

If you go to the thread that is linked or at the very bottom of that issue, robert marshall is proposing a few items.

B

We should figure out what we want to accomplish here and help drive their direction.

B

I personally don't want them to stop working on this, because I think it'd be highly valuable if we could at least contribute to the logging data that comes out from them. So if they can continue working on this, at least for sidekiq, I think that would be beneficial and we could just enable and disable as needed that allows us as they make the changes we can test it out and see if it meets our needs, make sure we don't blow up our inelastic search cluster, for example.

B

I know there's other product updates that need to happen in order to get logging to work as they are envisioning, so I think they themselves are still blocked in.

E

B

Certain case, but I think, if we could at least get the logging mechanism in place, that they're trying to, I think that'd be beneficial. But.

E

What do you suggest, okay,.

B

I suggest that they continue the work and maybe see he's got two options. You know which.

E

Is dumping, let me rephrase what will you write to them in a comment.

B

I would suggest that they.

B

Choose option two that he's provided, which is the dropping the dropping everything of the four except for the one. I think it was sidekick.

B

um They want to pick one uh they've they've already started one, so it would be great if we could get that push through so choose option two, preferably if we could add it behind a feature flag would be my suggestion and that's how it would respond.

E

So what I usually do when I don't like any of the options I put up another option so.

B

Which technically is a third option, because they don't have this behind a feature flag at all, and I think that would be helpful that way, we don't have to revert things causing problems with our clusters.

E

Swag, I would call it something else, because it shouldn't be a future flag. um We, I think we talked about it when we were reverting. This whole thing was the problem. Was there was no control over what is happening now, whether it's a feature flag or not?

E

I guess it's a philosophical discussion more than anything, but what I would expect is to have a chart configuration, because that is where you can actually configure things, um because you can't control anything inside of the image, otherwise right, so whatever they end up choosing, however, they want to call it. Yes, I think that would be a good option to go with.

B

E

Having having an option where you can roll it out, yes,.

B

Precisely yeah yeah all right I'll I'll word it as friendly as I can there, but I'll respond to this thread after this meeting.

D

Awesome, I also wasn't from just scanning through I'm, also not super clear. So robert has a comment um which about the starting point, potential starting points and then he's got another comment further down which goes on about the four approaches which I thought were about the four issues that are scheduled in 13.7, but I'm not 100 sure, but that was how I interpreted it.

E

Yeah I'm trying to understand as well.

E

Yeah there from the four options listed there, there are actually a couple of different things completed here. um I wouldn't confuse the things further. I would just repeat the request. This carvick just made yeah.

B

I don't want to muddy the waters. It sounds like he needs to do some more evaluation that I think is outside of our realm.

E

Yeah there is like a larger story here if I understand you're right amy. If I understand correctly, this is where distribution actually needs to work with the application teams and figure out what is the actual general story? This is not up to us an infrastructure department. We we should participate, but on the same level as everyone else, because we are adding feedback, but actually someone needs to be operating. The the story across all of the groups, projects, services.

D

Cool that makes sense.

D

Relate to this one, I don't believe this is the case. We've got a comment that this is a blocker for moving past canary, forget ssh. I hope that's not true.

B

I think it would be because api and web they're going to have.

B

I don't like our logs, I think we're going to really blow up our elastic search clusters if we attempt to move api and web traffic over into kubernetes, as is- and I don't want to cause pain on the people that manage our elastic search clusters, because that will cause a lot of pain in other areas as well, especially incident management.

D

Okay, but it's not a blocker, forget ssh, though right it, but it is a good.

B

Ssh just done the api, it's not going to be a blocker for websockets because we receive so little traffic for that. So I think for the time being we're okay, but we need to make sure that we, it is still a blocker for web and api. In my.

E

E

So I guess it's not a specific blocker for a specific thing, but it's a general blocker that we are now choosing to ignore. So we shouldn't be dropping this whole story anyway, right.

E

It's just gonna, get progressively worse as more. We migrate.

E

But I think um there is a lot of requests we are putting forward with uh with them.

E

It would be good if you and jarv or or one of the two of you interact with with people in the observability team, to figure out what actually structure we expect want to see to provide them a bit of a feedback from our side before they start talking with other teams, because I'm suspecting there is going to be either a lot of opinions or no opinions at all, because not many developers actually do care about how the log looks.

B

I agree, um maybe I'll try to send up a meeting with a few people, if possible, to see if we can't at least provide some guidance or pointers.

E

Or or just like continue, there is a discussion in one of those uh issues that are linked in the epic um I invited mikael and igor just maybe coordinate a bit there.

E

Try to do async a bit sounds.

B

D

B

I think we can remove bullet point four right. There.

D

Oh, uh oh, I know I didn't remove it already. uh Do you happen to know if, uh as probably for you andrew these comments that you added in your requests, uh do you know if they're somewhere other than this stock.

A

Sorry I haven't been paying attention because I was trying to help um uh hendrick with the production incident. That's.

D

Right, I'm looking right now, yeah, it's just! I.

A

D

Deleted earlier- and I thought maybe this was like the single source of truth.

A

um Let me try to find it quickly. We are so.

E

Yeah one a four and then bullet points. There is.

A

A4, oh yeah request and for changes to gitlab logger date.

A

Oh yes, some points from my side.

A

No, I don't think I put that in anywhere else. I will. I will do that now and and uh and put something in place right now, because yes, it needs. It needs a better home. Thank you. Sorry.

D

About that no problem.

B

Some of those notes are kind of logged in the issue. I just dropped right above that section, not not everything, though.

D

A

Add it to that cool.

D

Great- um and uh this is why we've derailed you last time as well, andrew, because we've got a request for you straight after that, one as well, which is about the um labels.

A

Yes, so I did some work this morning and jeff put the mailroom um and the plant uml labels on, and I I basically tried to bring them in line like especially plant your l, because it's not even it's not neither of them were really the matrix catalog and plant ul plant uml was not in the service catalog either. So I did some work around that um and when I did that, I could see that the plant uml stuff was not working still, so something needs to be done there.

A

But I pinged java on that, um and the most important thing for me is still the deployment labels on the um hyperscale horizontal scaler, um and so I don't know if there's been any work on that, because that will help us quite a lot.

A

D

Look like you know,.

C

A

Like when we gave graham that demo the other day, the first thing he said was yeah we're really good to get the hpa, and uh we can't do it until we got the labels and there was also a minor incident the other day where it would have helped um so yeah. That would be those would be great.

D

Hey jeff we're just going through uh the labels so actually uh andrew the mailroom wasn't done. uh I think 1375.

A

Okay, yeah mailroom mailroom is good um plant. Your ml is done, but not good from from what I could see, but it wasn't. We didn't actually have an issue for blanchard uml, so I think that was kind of a bonus from jav, but um it doesn't seem to be working.

A

um And then that's okay, what's what's this 524? Is this still open?

A

um Alright, merge, okay, so this adds okay. So maybe this has been done. If this is done, then I will start working on those on those things. So I will um open an issue for that. I'll put that in here.

D

D

Cool and then uh pages is progressing along quite nicely. Actually.

D

So there's plans for.

D

D

Testing and preparing um next steps in the migration are being planned out. So looking looking hopeful.

D

Cool anything else, and it wants to cover on.

D

Blockers awesome, okay, so uh going down to discussion, uh starbuck.

B

Okay, so I've got two epics that are, I think, populated with at least the start of what we need for these one is the helm3 upgrade and the other is the work for websockets.

B

If I'm missing an issue, I know the helm upgrade is definitely missing issues because we, I know we need to test things before we develop a full plan, but for the websockets. Hopefully, everything is in place that we have everything. We need to start that work.

B

So I guess the main question is: what do we want to start picking on next, because currently, I'm kind of sifting through some of the lower hanging items that we've got in our tech, dad epic, and also just pulling some of the p1's that have come across us in the last few weeks. Due to some incidents and such um I know, helm, 3, helmed version 2 is no longer supported. We've surpassed end of life as of november, so be wise to pull that one.

B

We could also potentially do this work in parallel, but just looking for opinions, I guess.

D

Yeah, that makes sense so uh the old, the overall goal for the quarter is the apis. So um what do we like? What's what would be blocking the apis today?.

B

B

I think websockets would be a relatively quicker one to work on historically compared to everything else, historically, just because it's already running in kubernetes, it's just a matter of separating that traffic out into its own configuration inside of rha proxies and then making sure we capture the metrics dedicated for websockets and creating the dashboards for such.

B

Lower blast radius because it receives such little traffic in comparison as well.

D

With websockets, is that a um we need to do it in terms of? Does it? uh Does the api have dependencies of some kind? That would help us, or is it a? It is small enough that we think we could just get through it.

B

Sure I've corrected me if I'm wrong here, but I'll go with the lighter it's small enough that we could just get through it and rock it out.

F

I mean it's small enough if we just keep with websockets as they are now, but the idea here was also to enable action cable and um that we need to go through a readiness review talk to developers to see whether we think we're ready um but yeah.

B

F

In general, like there's, nothing preventing us right now from creating the websocket service and getting it like getting a websockets dashboard ready for it, and you know, and just getting the website construct traffic routed properly.

D

What do we gain by doing this ahead of the apis.

F

As far as where I stand right now, I feel like api is probably should be blocked by the nginx upgrade, as well as the helm upgrade, and I think also possibly the websockets work should be blocked by the helm upgrade. I think we should just focus on that. Get that out done get that finished before we do anything else, because um this is going to be like the traffic split, probably where we're going to have to do a coordinated cluster upgrade, and since it's going to touch the regional cluster, it might be even.

D

Messier so.

F

uh Sorry but scarborough correct me: if I'm wrong, we don't really know the full extent of the upgrade impact right.

B

Not yet we need to test it. That's the first issue created on that epic.

B

C

B

We hit a blocker.

B

Part of that was just due to not testing thoroughly enough. Distribution has documented what the blockers are. So I think it'd be wise if we retest with a gke cluster um versus our previous testing method was which was just using mini cube. It wasn't an accurate representation of what we were running at the time.

F

Are we I saw you had an issue to create a new pre-prod cluster? Is there any reason why we can't just apply this to the existing pre-prod cluster to see.

B

I don't want to break pre-prod. We could potentially use that cluster. um I just didn't want to break that cluster entirely. If something went wrong, I guess it's not hard to rebuild it, though so maybe that's sufficient enough. I.

F

I would just like use pre-brought as it is. um Okay, I think we can easily. I mean worst case we're down for like an hour or two, uh and I don't think that's a big deal for pre-prod and it's only down for good https, and you know sidekick.

F

We could even actually we could even actually like divert traffic to the vms. um We may even I think we still have the git vm. There was like one git vm for pre-prod and then for a sidekick. um We could even it's. Let me just say it's probably faster for us to spin up a sidekick vm than it is to spin up a new cluster project and deploy to it right.

B

Yeah you're right, that's easier.

D

Okay, so um what I I'm hearing is, so we got the api work which we believe is blocked by an nginx upgrade. We think we should do web sockets before that and that's blocked.

D

uh We need a ready notice of you and we also have the helm3 work yeah right, okay, so um helm3, um I believe uh marion correct me. If I'm wrong, you want this completed before we do the next service migration.

E

D

E

Because I want to get it over and done with all of the breakage that we get to see um before we move, I mean I'm not saying the traffic that we have right now is not impactful, but it's going to be less impactful now. Another thing that worried me in jar's statement is um dimension of original cluster and how that's going to get complicated with an upgrade.

E

Do we need to think about helm4 upgrades, I'm just giving an example right, like the next foundational upgrade that needs to happen? Do we want to figure out? Is there a way for us to do this now? If that means the additional cluster that has no traffic in general, or I don't.

C

E

I'm just spitballing here, but basically the point I'm trying to make is: how do we make sure that we do it as painless as possible now so that we don't have to do it as painful as possible later.

E

Because I'm not really sure I follow exactly what the pain is because zonal clusters, I get it right, like it's simple for us to just take zonal clusters out of rotation, redirect the traffic upgrade move back and so on, but the re, the the regional cluster is always taking traffic or what.

F

The regional cluster doesn't have an nginx ingress at all, um except for uh canary, but in the gitlab namespace and canary. We can just drain and forget about so for the main namespace.

F

um We don't have any ingress at all, since we moved registry to the zonal clusters, um nothing, so it's all sidekick and sidekick is tricky because um we have like it's not. We could spin up a new cluster and then we could activate side sidekick jobs, but then you would have jobs running on both and some of our cues are throttled. So we don't want to run them like you know, into clusters simultaneously.

F

We would have to think about how that would be done.

E

So that that would actually get my vote like helm tree. Obviously, right like we want to get that over and done with, but it's obvious that there isn't some preparation work that needs to be done because karbek did that what was it a year ago skarvik or something like that, so things have moved on it's. It would be good to actually start on that and then also figure.

E

Well, while we wait for the nginx upgrades to get merged chart upgrade to get merged, we can then roll that out separately and then also at the same time, we can maybe start thinking about what do we do with the original cluster and at the same point we can actually ask developers of action cable stuff, to write up the readiness review, so we kind of parallelize all of that work in one go while we wait for the login work be completed for the api, so we're talking about like five different parallel things, not all about all of it on us, but some of it.

E

Yeah with what 10 days to go until everyone is gone.

E

F

I I'm in favor of just focusing on the hell of three upgrades, 100 yeah.

D

The only thing I was wondering um do you know: jeff, have you already spoken to developers about the actual cable readiness review.

F

No, I have not talked to them about it.

D

It might be worth getting them started on it and they could be working like whilst we're doing helm three and then, when we're done with helm. Three, hopefully we'll have something to to build on.

F

um Yeah that that sounds fine, I don't know like yeah I'll talk to them to see what they see, what they say.

D

Cool um the only other thing I was uh going to uh also ask about was the only other thing, but one other thing I was going to ask about: was the um we've got this uh observational uh sorry observability, uh epic I'll just show my screen, and what I would like is to try and give that a bit of a scope so that at some point we can close that off, rather than it just being the epic that's going to be open forever.

D

D

So, uh within this epic, how much of this stuff do we need to get completed before we do the api, because what I think is we could maybe frame this around cluster observability and troubleshooting to support api traffic, and then we could have.

F

D

In the future,.

F

I I added those items in the description um under exit criteria, and this is what I was thinking at a minimum we should complete before we do the next migration. um We should probably be linked to specific issues, but this was just like off the top of my head.

F

What in the epic, we should definitely focus on before we do api.

F

I don't know like: is there anything else? We should add to that list.

B

Personally, I think a lot of these really should be on that list, like all these are a decent priority in my opinion, but.

F

Well, that's that's okay, but I mean we need to. We might need to work on some of this stuff in parallel. Is there anything that, like isn't covered in that list of extra criteria that we should do before we work on api.

B

No, that's probably fine.

D

Could we get that list then? So? Could we get these issues um to be there the ones we want to match that that list um yeah in some ways awesome cool and then uh what I'm hoping is that will mean we can get on our epic, we'll be able to have observability and troubleshooting and we'll get that we can get that wrapped up for the api stuff got help three on there as well and.

D

Then, oh, I need to add in the api one that you created uh skybeck, so we're looking at roughly this sorry.

B

I believe the api one is mixed in with the web traffic number 272.

D

Okay: okay, oh yeah, that's right! Yep, okay, cool.

E

You should consider splitting that off because.

D

E

Discrete amount.

D

Yeah, that makes sense cool.

E

Okay, good um skarbic, a question for you uh before we continue further is nothing is preventing us from no. This is not a question.

B

E

Is a statement, um so let me ask a question: are we even able to upgrade to helm 3.

E

And run both 2 and 3 at the same time, across clusters,.

B

um We could run our tooling needs to support that it currently does not so no, but we also need to take care of that. We both have our own helm, repo for the gitlab com stuff, and we also have the components for gitlab helm files, so the tooling upgrade is going to enable us to run both home versions, but once we upgrade one cluster, we need some way to tell our tooling to use helm3. For that.

E

Cluster, my god that sounds.

B

Miserable yeah, I think, if I was more keen on software development, it wouldn't be a terrible thing to accomplish, but it.

E

Has nothing to do with software development.

B

Well, all of our scripting is bash. So from a hypothetical standpoint you know we should be able to pull that kind of thing off, but um I think it would be.

B

I think, when we perform the testing, we'll know where our blockers are going to be, and hopefully we can develop a plan of action based on that information.

E

Okay, the reason why I ask is um there is a natural pause point in this um effort, which would be upgrading the zonal clusters and having them run on tree and leaving the regional clusters cluster running on two to use it as a feeding point for the making it simpler to upgrade regional cluster as a exercise.

B

It would be fun to find out that upgrading the regional cluster is a breeze just because there's nothing that blocks us and then it's just arizona clusters that are questionable, because those are the ones that contain the nginx ingress.

C

B

But up to you, okay,.

E

We'll find out thanks.

D

Cool um and then for the um epics that we have in progress uh I'll, consider help three in progress uh for this testing one. um Could you get the? Could you both get the issues that you like your next round of issues ready and on to the billboard.

B

Yeah, I could do it awesome.

D

Cool um is there anything we need to do to help.

E

I do want to ask: do we want to see if we can involve graham um into this effort, but like from z from day zero.

D

He's out until uh late december, he's on pto.

F

E

Of course, it comes from you.

F

um Starbuck, how about um we should decide like who's, gonna who's, gonna, dri, the helm, 3 upgrade um you're release management, you're, doing release management as well.

B

Yeah, hopefully thank.

F

B

Come december 22nd, though,.

F

Sure, but we're going to crank this out like before new year's right yeah, maybe well, okay, how about? um I think there are two major things that we need to focus on: what is the home three upgrade and one is the logger stuff just because, like I feel like we're stalled right now with our feedback saying that we are like you know, we definitely can't turn this on. So I think we need a dri for the logging stuff and for the helm3.

F

um Maybe I should start with the helm3 since you're busy, with rmming and um and for the logging stuff. Did we discuss? I'm sorry, I missed it earlier in the meeting. Did we discuss what the next steps are there I saw that andrew is giving them their feedback and um it, but it looks like it's completely stalled.

B

uh The proof the feedback we're going to provide is something in the lines of having them do a different option than what robert has proposed, which is continue.

F

B

As necessary for at least one component but they're going to push behind a configurable option to enable or disable logging as a whole um that way well logging using the new mechanism that way, we could turn it on and off and test as needed to help them and provide feedback as they continue to work on it.

F

Okay, so this is realistically not gonna happen until january best case best case, probably yeah. Okay.

F

So then it sounds like we should just focus on home.

E

And observability.

F

Yeah observability yeah.

E

If you're going to tell me now, there is not enough work.

E

B

Plenty of work.

E

There's plenty of work.

F

I'm good at making work for myself. That's not a problem.

D

Okay, um okay, is that a plum.

F

That's a plan, I'm gonna start flashing out the helm out pick then and um scrub back. I think what I'm gonna do is like move traffic over to vms in pre-prod, so that we can mess around with that cluster um and, let's just see, if maybe it'll, be a breeze, maybe it'll just work.

F

B

F

D

Okay, is there anything else anyone wants to bring up or cover today.

B

I can't think of.

D

Anything fantastic, okay, well enjoy the rest of your dates. Thanks very.

B