GitLab Delivery: GitLab.com migration to k8s demos, 17 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-04-17 Sidekiq migration

Description

https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/89

A

Good morning, good morning,.

B

C

Good afternoon for me, but.

B

A

A

D

D

D

So I don't see much else on the agenda except for the demo. Does anyone have anything they want to talk about before we start I'd.

A

Say let's start off with poster boy, planchet pageant patches and then maybe talk about what we don't want to do with sidekick afterwards.

D

Sounds good okay,.

D

D

Okay, so let's meet a hot hatch on staging, so if you've never done this before the first thing, you need to do when you create a hot bat, you need to figure out what's running on staging the way we do. This is that we go to slack and we do all of the place status. Let me just see if there's any deployments in progress I'll just do it again, just in case I think it hasn't changed so last time this was done. I know you can't see my screen, but I'm.

D

Basically just doing that chat, ops run out of the play status in the release 10 1210 channel. This will give us the revision of what's running on staging.

D

D

Go to get lab ee, whoops ee, let's create a branch, let's call it patch demo one. Actually, let's do it from the ref, so call patch demo, two.

D

And let's make an edit so.

D

Let's go to how to comment to this file.

D

I'm not going to push anything upstream I'm just doing this locally, like you would, if you're recording, Apache and like it left us down.

D

So there's the diff, so when you're creating the patch you need to go, you need to have a check out of the patcher.

D

And we'll work on my branch here so in the patch, where there's a patch directory. This has a bunch of directories that mapped to the package. That's installed on the environment, and this corresponds to the omnibus package and the way that we have patching working for kubernetes is that it will like work for both. So we were basically, we create a patch as if we're gonna be patching v ends, and it's also going to go to Cooper Denny's to get the package I just also looked at chat ups, so I have it here.

D

I'm just gonna copy it I, don't think I have one so I, don't have a directory here, so I'm going to create this directory and then I'm going to copy this patch to that directory.

D

Or a cuss patch got punch.

B

You might not matter that small typo in the file extension I'm, not sure if that matters on all this matter,.

D

Thank you yeah, because we only look for things that have like a patch.

D

So so here's our patch, you can actually the way that post-deployment patch works is that you can stick in multiple patches in and it'll just apply them one after each other and it sorts it alphabetically. So you know if you're patching different files. This works fine. If you're patching the same file, you need to have patches that can be applied on top of patches. Super super like I, would say, hacky this whole process, but it's what we do. So this is what we have so I've created the patch.

D

What I'm going to do now is commit this to the patcher.

D

D

Let me just share my screen again.

D

So now we have the pipeline. You can see like if you're creating a branch on the patcher, but we have our. We basically dry run patches on the fleet in the case of kubernetes. What this is going to do, hopefully it's going to trigger a dry run or a like the pipeline that runs in check mode. This already completed, so that's good, so aha, okay, so looks like so first thing we did is I went to the kubernetes cluster and it looked for what's running it found this time, but we don't have this time again.

D

Maybe this is just. ah This is a formatting difference between the the kubernetes tag and the VM, so I think I'm gonna have to do a little bit more fuzzy matching here to get to get it right. It was looking for a directory named. This actually was looking specifically. I was looking for a doctor, I'd like named this, and that doesn't exist because we created a directory. We.

A

Still see your browser, John yeah,.

D

D

So he was looking for a directory named bat which doesn't exist. We created.

D

We created that you can see like and also the Shaw is different. Let me double check to see.

D

Okay, I actually copied the wrong package. I copied the get led comm package. It should be here's the package name.

D

But so so you can see like what's running in the kubernetes cluster. Is this tag, and this is the VM package? The difference here is that the VM package has the EE sha here and then the Omnibus sha here, but we don't care with the omnibus since it's scoober Nettie's, so so I think I'm gonna have to improve this fuzzy, like fuzzy matching in the meantime, what we can do is I can justice is looking for a directory with this name, I'm just going to create it, and we just don't see the patchwork.

D

So let's just create this and.

D

Let's create, let's just move the test patch over to it.

D

Okay, now, let's see if it works better, this time.

D

We could do what we could potentially do here is just make it so that in the patches directory you just create a directory that that's just the shop in that way. You'll just match both the VMS and the kubernetes bye shop. So now it's working a bit better. So the first thing it does is: it goes out to kubernetes to get the version it got. This is the version and then say so: I found a directory here, that's the directory that we just created. Ok, so it logs into the dev registry.

D

This is a dry run and what it does is it actually creates this docker file here from the image that's running in the kubernetes cluster. It copies the patch file to a temp directory, a working directory, and then what it does is. It applies the patch to the image and then it builds it again. So we're basically just taking the same patch file that we slam onto the VMS. We are, you know, creating a patch image, so the first thing it does is it pulls the image from dev.

D

This is the base image after this is done. Then it executes the then it builds the docker file you can see here. This is where the patch is applied, and then after this, then it uploads the image when it builds it. It tags it with.

D

Is the image tag, so you can see like it basically takes the base tag and it adds this patched and then a date string, and then it kicked off a terminating pipeline.

D

Which is just try run so we should see here is a diff of the old image. The past image.

D

And here this so before I continue, does anyone have questions about this.

A

What's the granularity on it looks like a timestamp at the end of that yeah.

D

A

The granularity of that timestamp seconds.

D

So one thing that I had to do here was okay, so let's say we deploy dispatched image, and now we want to apply some more patches to the same version. Now it's gonna go to the kubernetes cluster and it's gonna get this tag. It's gonna say like, and this tag is like the patch tag right.

D

So actually what I do is I like chop off this patched when it goes and fetches the image so that if we run this again from the patcher, it's still gonna look for a directory that has this like 10 tag name and it will just reapply the same patches. I think. The next thing we can do is we can actually apply the patch. This is just a dry run.

D

So if we to the pipeline the way the patcher work is that you, you kind of like you, staged the patch on the branch and it shows you a dry run of so you can make sure that the patch can apply successfully on staging and then often you just merge it, and then it gets applied to staging and production. You also have these manual jobs that allow you to apply it on staging.

D

If you want to from the branch we don't normally do this, but sometimes it's necessary just to kind of like see how it'll apply it before you merge. So we have a manual job for kubernetes here, just like the other ones. If I play this, what it's going to do is it's going to do the same thing again, but this time it's going to trigger a regular kubernetes deploy so it'll actually deploy the patch.

E

Well, we are looking at that. We have a pretty good audience today. I'm curious really to hear questions, remarks not necessarily only questions but also remarks, because this is an interesting way of doing things. I.

D

Think merits being nice.

E

A

E

Very creative but I know exactly why we are doing it like this and I fully support the fact that you did this, but we have people in this call who might not know.

D

Yeah I kind of put in the dock. If you pull up the agenda, I I kind of I put in there like what the ideal process would be and and why we can't do that now, but I think eventually, like maybe we'll make this process a bit better.

B

So sorry, latency government I think.

A

As it stands like, this process makes the most sense because it's the least amount of change required for people to learn how to do patches if needed in the future. Like this is the same process. It's just a we're, adding a different service to it, which is perfectly fine, I, don't know how else we would do this without waiting for like your ideal process, but we would have to wait for things to build and that's excruciating, and we can't wait for that in times of an incident.

D

You were gonna say something like now.

B

Forget it it was, it was half-baked, I'm, not particularly familiar with our patching or deployment workflow at the moment. So I'll come back to you after some reading. If they still have the question. Okay,.

D

Sure yeah, so so now we did the actual patch deployment. So it's triggering the Kate's workloads pipeline without the dry run flag set, and this time we do the dry run, we're doing a dry run, and then we actually do the deploy which is being done here.

A

B

Many times per month or week or whatever unit time, would you say that we end up patching production as opposed to the one releases, the normal way, yeah.

D

Used to be pretty frequently but I would say lately twice a month. You know not that often.

D

I think it's interesting I think what what changed is that we we now have the continuous deploy pipeline, and it's well understood that if we patch production, that pipeline is halted. So there's like a lot of incentive to not patch and you know, fix it and deploy normally or instead of patching, will maybe do some other infrastructure workarounds, like a.j proxy blocks for security issues. Things like that I think before Auto deploy before we had this continuous deploy pipeline.

D

We were sitting at patch releases on like production for weeks, and we would just like patch as a quick thing, because it was just like there was so much work to deploy. I mean now it still takes a while out, but it's a lot easier just to get changes to production. So we do that more often now I think.

E

So what we're seeing noted actually create an image like the logical thing you would want to see now that we actually deploy from from a docker image. Right, like the logical thing you want to see, is that someone creates a docker image, real, quick and then pushes that and then we use that to go and do things. The problem is our pipeline that creates these images has stages that are very much tied together. It has six stages.

E

If you go to the components, images or four charts, it has 18 six stages and each of them takes some time by the time you actually get to the point where it's gonna take the longest to build an image you are already able to patch. So we are talking here about jarv, correct me if I'm wrong, but I think last time I checked, it was something like three times slower to actually create an image.

E

Then patch manually like this, so do you wanna in the middle of an incident, wait for a pipeline to build images for 45 minutes. I, don't know when I was on call I didn't want to wait that long. So I'm kind of guessing that you guys as well don't want that. So this is like the most shortcut, the shortcut that we can take.

C

So related to what I just said, just two things came to my mind. So one is: they did that long term? We will bring down the time it takes to build the image, or are we happy with with this process and the second one is: if we're patching, is there any sort of tests conducted so like I'm? Not talking about you know like a full QA case or information test or anything, but is there any form of validation that the Patrick Lloyd is? Actually you know it's not? Actually you can make things worse, I think.

D

It does have I guess that if the thing that we do is we do give out QA on staging before we move to production, so we have that other than that we have the liveness in readiness checks right and that basically just tells you what the things are kind of functioning in the minimum level. Those are the only two checks that we have for post deployment patches usually, but though this is like, we don't typically want to wait for a full battery aspect test pass and you know so so normally like we don't we don't.

D

We don't want to wait for those things, but we do do them and at a minimum we push it through staging first and then do QA. Not always, though it really depends. If it's a super urgent thing, we may actually skip QA.

D

C

If someone, you know, came up with the patch employed, the trigger just postures and it was built- and you know community started deploying it. Would that mean that it would fail on the I'm guessing on the first thought and then stop I'm not allowed to do to the other yeah.

D

I mean I could just demo that, if you want to see, we can definitely create a patch that I'll, just you know, destroy the application.

D

Yeah I mean like yeah, but I know this works because it's just a typical it's a normal kubernetes deployment right. So it's actually, it's pods aren't going to be healthy until the lightness and readiness checks pass.

B

And, unlike the VM deployment, I'm pretty sure we don't even have reduced capacity during that time, because kubernetes surges, rather than takes.

D

B

Pods first yeah.

D

We had like a 20 percent surge that we allow, for. We actually have an issue. Skarbek and I were talking about this earlier- that you need to make this bigger for sidekick I like to surge more so that we can speed up the deploys, but yeah we have like a 25%, you know surge on deployments and yeah I mean we wouldn't lose any, and we also have the pod disruption budget as well. So it's much proud of the kubernetes, the equivalent on the V ends, is like the H, a proxy checks.

D

So as we as we roll through, we would take out three instances out of the fleet. You would upgrade them and then they wouldn't pass the health check for H a proxy, so we would abort yeah cool. Is there anything else they won't like to see before we stop this.

E

I'd like to cover Mikhail's first question: we answered only the second one pause.

D

The second option, the.

E

First question was whether we are planning to make a change on this, make this faster image, building faster, it's a tricky one. Okay, it's a tricky one, mostly because with whenever we apply something temporarily temporary like this, it becomes permanent and the next set of super urgent priorities that need to land yesterday takeover, like you know, so, ultimately, it's gonna happen with the same way. I think is.

E

Gonna have is happening with the omnibus package, which is you have a tolerance of package building taking 35 45 minutes max, so everything gets delayed by an hour and you're kind of weighing whether that one hour is acceptable delay for your workloads.

E

If it is, you don't take any additional action and you're going and optimizing tooling in other places, but when that start sliding down like it has couple of times with with the package build times, we go to distribution, team and say this is like something is wrong like we are now taking an hour to build a package right, so it's trade-offs like everything in life. It's traders here as well. So, while the image building is very complicated right now and janky and I, don't I, don't think anyone in distribution likes it.

E

Actually, no one actually had the time to look into that, because that's considered technical debt that doesn't bring any additional value right. So until we get to the point where the deployments are going to be slow, regardless of whether they're patched or not, I don't see. This is becoming a priority right like we have some basic things missing from from the charts and from the images right now that we are discovering, as we are migrating so I, think those take. We higher priority.

C

I was I was just wondering you know. I was just curious. If, if it's because I don't know, we've had some hard limitations or, as you said, are we just for now prioritizing something else which is you.

E

Know a very fair.

C

E

It's it's a trade-off right like even when we build the original pipeline back in what was it 2017 I think the the CNG pipeline I, don't think anyone with the team liked it's done as well, but it was the quickest way to get us to point where we now have images that we can leverage right. So it's always that trade-off that you keep going and forward and forward until it starts being more expensive right like when it takes too long to build something, and that affects your availability or productivity of larger parts of the company.

E

Then you get some priority. It's unfortunate, that it is what it is.

E

Jeff, this is awesome, I know how you feel about it, but I also have to say this is awesome. The fact that you managed to get it done and that we now in production have a way to to you know, fix problems on both platforms very quickly, just heads down free for you right, awesome, job.

D

Okay, well, I, guess that there are no other questions. We can end the meeting.

D

Okay thanks. Everyone.