GitLab Delivery: GitLab.com migration to k8s demos, 21 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-08-21 GitLab.com k8s migration EMEA

Description

Reviewing migration blockers and production readiness plan for WebSocket and Git https nodes on Kubernetes

A

B

A

It just the two of us or are there more people.

B

Yes, I think it's just two of us.

A

Cool, so maybe we can just take this time to go through blockers and hopefully it'll be a short one.

B

Looks like we've had some good progress, um so the first attempt at the live traces on production has been completed. uh Just added the note there. So it looks like it was fine, um but there's some follow-up. So there's a couple of follow-up issues for this one.

A

Yeah I haven't been following it closely: um what's the follow-up issue or what's the what's, the dldr of this.

B

So um I think the structure.

A

Okay, I'm reading it now so next steps.

A

I see so next step is to kind of do a slow roll out.

A

Okay, that's that's good, so probably like over the next four weeks, or so you think.

B

Yeah, hopefully yeah they both got 13.4 on them so yeah. Hopefully that will just go ahead nicely yeah at what point do we um we're not like immediately blocked for those? Are we that's as we move more towards the web stuff? Is that right.

A

Yeah, it's not immediately blocked. We're really aren't going to be blocked by that until we migrate web and api, which won't happen until after both get https and get ssh are complete. So I think if this flight lines up with like the next four weeks, that puts us, I think, we'll be in good shape.

B

Great okay, nice, that's good, good, start um cool and then the support for dependency proxy is closed. It's got a 13.3 milestone on it.

A

B

That's good um the remove nfs dependency for pages. I've just dropped in an update uh here for the sidekick stuff. That's from uh what rachel had yesterday in the sas meeting.

B

So it's all moving.

A

um I have a bunch of new ones that we need to add, so let's just go ahead and start adding them. So first.

A

Oh, I'm sorry, I thought we were done, but we have more to go through so mapping. Services to web api get that's unknown.

B

Yes, there's a comment from yesterday: I think jesse put it on uh it. Doesn't it still doesn't look yeah from this day, it still doesn't look like it's a good uh idea for how that would be solved, um but it's still being reviewed. At least uh cool logging is great and I've added on the catch-all, email, config stuff as well so.

A

ah Perfect, oh.

B

And then the other one that you have from today right the.

A

Costume, never ending right. So first is um proxy request: buffering blocker for web api.

A

This is something.

A

That kind of ties into mapping services where we have no way to have specific nginx configuration for different services for it. It's all all the https services are bundled together. So I think we'll need we'll need a solution for that, uh but it doesn't block, get https or get ssh.

A

So um that's that's good.

B

Cool, so that's probably gonna be like a like a 13.54, that's probably yeah like I. I guess it will block us from late september. I'd guess like mid to late september,.

A

Yeah um trying to look at what other things I'm sure my screen.

A

So this is related to live traces. I guess for camille proxy requests. Buffering is what we just talked about service desk. You have that incoming mail, you have it. um This is the nginx uh one of the issues, but um I think uh this is this is what I'm proposing for reducing the amount of crosshairs across daisy traffic.

A

So we'll see what happens to it. I'll just add.

A

A

A

I don't know if this, um so we can link to this one too.

A

I hate it when we have group labels, because then we have like I'll just pick the first one. I hope that's right. Who knows right, I'm going to add this one as the migration blocker to be on the safe side.

A

um So you saw this like this. One came out of that network diagram, which shows that, like we're not using the local unix socket between nginx and workhorse, we're now talking over the.

A

A

Maybe this is not a big issue, I don't know.

B

Yeah, I was going to say like who do we need?

B

I guess for both of these right like who do we need to have um help us.

A

B

A

I think whether or not like, even if we create the sidecar, we still have this problem unless, if we use a local like a unix socket or something- and um we just need- probably I don't know should we get secured, I'm afraid to get security involved. You know, but maybe that's what we need to do.

B

I think we probably should yeah.

A

A

I think we need to understand the risks a bit better.

A

I assume that if you get root on the web service pod, even with an unprivileged container, you can still sniff traffic and then in that case, maybe you could see other people's traffic, but uh I'm not 100 sure of that. So I think, like.

A

We'll have to want to confirm with security.

B

Yeah, I think we should probably should do you know uh if, assuming if they come back and say yes, it should be encrypted. um Does that cause us big problems like do? We know how.

A

God I mean like what workhorse doesn't support, tls so um outside of adding dls support to workhorse, which is not going to go over well, um it might be using like a local socket instead, but.

B

A

Like you still have the probably the same problem, um maybe we just need to like document this and move on. I don't know um like uh it's. It's I think it's a bit.

A

You know it's like it's like a bit worse. I think, because nginx and you know, nginx and web service are running on two different nodes and we have a small number. I don't know it seems. Like I mean we have a small number of nginx pods that are servicing all the traffic and then they're forwarding to they're fanning out these web service.

A

Pods right, I I I don't know how big of a issue like if you were to get root on the engine x-pod I mean it's kind of like you would have access to a lot of traffic. Then.

B

Okay, yeah. I think we should definitely check this one um yeah.

A

We, I think we just need to check to see how much we're exposed and then move on I'm much more concerned about the cross hazy network traffic, because we spent a lot of effort to reduce that and it seems like we're. There's a there's, a 3x multiplier. By going to kubernetes, it seems so.

B

A

That's like a big issue.

B

Yeah definitely um like how do we move that one forward, as long as we can see uh igor, has added some comments and uh yeah back to graham but like who? Who does this fall to like? Is this a like infrastructure as a whole type of question like or who should make a call on this.

A

I think um I think, for one it's it's probably not gonna block canary. Well, it shouldn't block canary. I don't. I don't see it by causing that much a problem for that, but yeah. We need to decide whether we're we're okay taking on the extra cost or whether we want to my proposal was to run nginx as a side car and that would at least keep nginx close closer to the.

B

A

Pod, um that means we go from 3x to 2x, which is a bit better and.

B

A

Don't see any way for us to reduce it much further than that.

B

A

So yeah, um let's see where this uh side conversation, how you know, let's see.

B

How it goes? Okay, yeah, I was gonna, say like what's the downside of running it as a sidecar.

A

The downside is that we already have the configuration to run the nginx ingress controller, so you would have to keep those configurations in sync.

A

um It might be a bit hairy in the charts, but I don't know like I see. No other downside. I don't see anything like um I'm, not sure why I was looking through the charts to figure out why they decided to do it this way.

A

I am not sure why so I need to.

B

I'm sure there's a good reason.

A

To talk to the distribution team.

B

Okay, cool all right yeah: let's keep an eye on that. One.

B

Cool so as it stands right now, we have um the web service. Stuff can go to canary, but we need to then decide on this stuff catch all. We need to wait for the email, config stuff right.

A

B

A

um Let's take a quick look at the epic because I have been linking everything to this so.

A

This is actually.

A

A

um Configuration audit what I'm doing here is. I uh I spun out the nginx configuration audit into a separate issue. As you see here, um I think we're pretty much done with this. I did make some changes out of it, which was to.

A

Basically, turn off all this proxy request buffering.

A

I think I think that's okay, but it's good enough for canary for sure and we'll see.

A

I finished this so this is done. I just need to confirm it's working, um so so really we're actually like. I know I keep on saying this, but we're ready to go to canary it's just that. We had a whole bunch of little small issues yesterday and this morning that I had to work through. So I guess like to my current knowledge in this moment in time, there's nothing that prevents us from preparing the change issue and getting to canary.

A

But it's actually this this uh production reviews review. I was a little hesitant to do it at first, but it actually really helped and.

B

We're able to yeah so.

A

We really were able to figure out some things like this nginx stuff came out of it. This network stuff came out of it, so um I think uh in hindsight I think uh it was good that we spent a little bit of extra time.

B

A

To do it, yeah.

B

I think it was really useful.

A

Like I don't know like, I think, we'll probably I think we should probably do one for all of the major services I think for sidekick. It's a bit different because we're doing these incremental rollouts but uh for, like uh you, know, forget ssh. I think we should also do this and uh web and api.

B

Yeah, I totally agree yeah. I I think it was it feel like it felt like a big enough thing to uh justify like the time going into it. Yeah I thought of just having everything like that in one place was quite useful as well.

A

Cool okay, so um I think that's it sadly uh no canary today, but uh it sounds like uh we'll be in we'll be positioned to do this monday. When is.

B

A

Group conversation.

B

It's next wednesday, so um so I was gonna I'll link the slides into the to delivery, but uh they'll start being shared around the end of monday. So.

A

B

If you can get canary before, then I can update the fight.

A

I will like at the last moment right now, we'll see it's possible, it's possible, but um yeah. I was thinking like I really like this. uh Maybe you saw it.

A

I really like this diagram because it shows the the green um for for migrating https. First, it gives you a big picture of the cluster and everything we have running in it and um the things that are colored green are the new things for git https, as well as the new dependencies outside of the cluster for get https.

A

So the ones highlighted in blue are things that exist in virtual machines, but new things that we depend on. For example, we depend on the console cluster. Now um we depend on the replicas, which is new and- um and we now depend on, like the load balancer the aj proxy, for these back ends. um So I don't know we could possibly incorporate that into the uh group. Conversation um slide uh or at least point to the readiness talk.

B

A

Mean I wonder if it should be.

B

On the handbook, you know the um kind of clustering.

A

So, unfortunately, you know my love affair with plants. Uml is sort of like ends at the handbook, because we don't support it there we would have to uh add. I could. I could save it as an image and then like this is basically just an image right, um but uh yeah. I could yeah.

B

It would be nice.

A

If he had planned two ml diagrams in my handbook, I'd have to see about um adding that.

B

A

It would be cool, though yeah, um but anyway, a diagram like this would be would be nice. I have something similar in the handbook, but it doesn't go into this level of detail, although this is very specific to the https git migration, but it's just nice to see like what we're adding.

B

Yeah, okay, cool I'll, see if I can fit that in yeah, it's nice um cool, so in terms of um pushing blockers and things through I'll leave you to do the um sidecar uh engine uh stuff.

A

B

And what else do we need to be pushing on a lot of this stuff's just moving itself, which is great? uh Oh, I suppose, there's this unencrypted stuff.

A

Yeah, um do you want to try talking to security about that, or at least at least bring it up to them? And uh I can I can be part of the conversation as well. Maybe maybe we should talk to distribution. First, okay, get their feedback, and then we can move to security if they're, if they can confirm what I'm saying.

B

Okay, that makes a lot of.

A

Sense, so is the release. um Well, I guess we're done with the demo right. Is there anything else.

B

Are we done uh yeah? I think that's, probably everything from my side.

A