GitLab Scalability Team Demos, 24 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-07-24 Creating a change issue

Description

Craig M. is running through creation of a change issues

A

Hi, I'm going to be recording creating a new change control. um This is just part of the closing deck between dibs and zary's. Moving the line uh with the scalability team.

A

It's gonna be stream of consciousness, not well edited. I will see where this goes. I may delete this and try again, uh but we'll see what happens, and um hopefully something of this becomes useful shows you just a little bit about what I'm thinking. When I'm creating these things um sort of issues, I need to think about, etc.

A

Sharing my screen, so we start from the production tracker. um This is where we create the the change issues, not necessarily change controls, um stroke sense we're using them, sometimes just for recording the fact that something's happened uh other times. It's it's to ensure that we um think, through all the issues that we need to think through. uh You know how we're gonna do this, how we're going to monitor it, have we automated it and so on and so forth.

A

So I'm going to create a new issue. um This is for staging, so it's pretty simple um good title is obviously pretty.

A

A

Let's double check max is used to confirm station.

A

That's the word, um so we've got a fairly well described set of criticalities here. um C1S are rare, very, very rare. um High impact high risk, you know downtime will always be a c1, um otherwise there's a few ones. um Failover, there's a few things that are really really scary, that just not sure.

A

C2S uh yeah no downtime, but it's a bit risky. um If something unexpected happens during it, it could be bad. uh That's the sort of thing I think you know, um c3s yeah not really expecting anything, but we're gonna do a little bit by hand and c4 stuff. We've done a hundred times before.

A

C4S are very much the recording just so that someone knows it happened, and you have something to do so. I think for this one here: it's staging which definitely reduces the risk or the criticality. It is not ch we can. We can do this without any downtime, so it's down to a c2 or c3, obviously very, very low criticality. For this one here, negative impact is pretty minimal.

A

We're adding connections from rails to pg bouncer, which is precisely designed to do this and connections in a pool, so yeah c3 for staging might make it a c2 for production. But I don't think I will do that now.

A

We have done this before similar basic process, which I will grab from another one. It seems like it's cheating to go and copy change controls from somewhere else, but I think it's valid because it's meant to be a collection of tools, not collection of tools, a collection of well defined processes that we've done and that's exactly what we should be doing. We should be reusing these things, cutting and pasting yeah.

A

If we do something 10 times sure we'll rub it up and automate it to the to the nth degree, but reusing the change, control and adjusting it slightly. That's perfect exactly what we should be doing so again. We need to do that. Often I just copy that title straight into the change objective, because the title really should describe it recently. Simply um a lot of the modus configuration changes.

A

Now we do have to list the services and because we are changing every single one. um If we have rails.

A

We need to divide.

A

And I think that if I just have a quick look, uh what's worth noting, is let's bring this across? This is the note? That's not the one for these this one here.

A

um This is the chef change um and one of the handy things is danger, but will tell you which nodes will be affected and because we've changed that on the base, role applies to water view stage. This one is basically every energy stage, um we're looking really caring about the ones that have rails on them.

A

So I'm looking for apis fasting's done blackbox. No, no, no, no, no front ends, that's electric proxies, they run italy, don't care about those they get service. So they do the good over registration get over hdp. They do run rails as well. So that would be relevant maturity. Direct pay, just doesn't pg balance, it doesn't and first queue grade prefixes.

A

It's definitely not pops up, no written snow, sidekick and web. I don't think web pages does.

A

Sidekick get and api, so there's four classes of machines that we need to think about pop that back over there. um That danger bot is really really handy for finding that so I've listed this for I've got psychic web api and get and it tangentially affects postgres. I don't think we have a pg bouncer. We do.

A

Actually, I might add, that one as well.

A

I personally don't actually use these terribly much, but it would, in a correctness, perspective, that's necessary change members, people who have been involved with uh critically c3. This is the testimony staging.

A

Yeah there'll be no dry, run output due date. So I'm looking to do this next week, which will be the 27th.

A

27Th and uh just for the record, 1 30 is about my favorite time for doing these things. I've come back from lunch. We have a change lock that goes over the weekend. We want to avoid making changes during the weekend unless they're absolutely necessary, not that it's inherently more dangerous, just there's less people around, so that actually goes up to midnight utc uh some sunday night monday morning, which is my noon.

A

Usually uh one o'clock during my daylight savings can be a little bit annoying, but the logic is that monday morning, when I'm here by myself and everyone else is still on there someday- you know- maybe graham and australia is waking up in the mid morning, not the best time for it.

A

For someone to be making production changes, maybe they're safe, maybe they're, not it's just nicer to hold back from that a little bit so so we have the change lock and that'll actually prevent certain mr's from being merged and checked and all the pipelines from running. um You know chef and terraform related ones.

A

So if it's 1 30 there's some time to have lunch, come back from lunch woke up, be ready. Caffeinated as necessary, um so originally we put staging in production in the same step, but we'll do staging this. One is just staging one day and then I think production the next. So we'll do that yeah. So because we are changing this, we want to stop shift first, because we want to do this.

A

Control rollout now shift runs every 30 minutes, so it takes a couple minutes to run when it completes it schedules itself for 30 minutes later, so it's 30 more minutes and it skews over time displays over time across the fleet, but it doesn't mean we don't have precise control if the chef damon is running as to when it will actually run on any other node. So for a lot of these changes, we're running we're chef running we'll, deploy the change to get lab.rb, which will then trigger an omnibus reconfiguration and then restart rails.

A

If we just let shift do it, we might take out all of our web nodes at once or all of our api nodes at once, particularly on staging we've only got a few, a little less likely on production. But even then you still want to control this roll-up. So we're going to stop shift across everything.

A

And that looks almost right, except we haven't, got the backing servers. We've got web api and shift, but we've applied okay.

A

Oh, they found.

A

A

And that correlates with what we're expecting? No, we still haven't found those. Why not um stage base if you get oops quotes.

A

uh Said jewels are important. There we go. Those are the servers we're expecting to see in g-stage cool, so we will find copy and paste that into here.

A

So we will stop chef. We will merge the change that the mr.

A

Won't show you here exactly, but um the in the chef request when you make changes to roles these staging ones. Also, the non-production ones apply immediately when you merge, the production ones have an additional manual step. So you can merge the thing now. We've left this, mr and work in progress state um so that someone doesn't come along and merge it without actually thinking that you know without doing it properly um and I'm going to note saying it's worked, because it requires a managed rollout, um not that anyone would come along and move someone else's.

A

Merch request like that randomly but yeah accidents happen. um Someone for someone to do that. They'd have to take several buttons to to bypass that. um So the next thing.

A

So we will unwrap and merge that and wait for the appliance station to complete. At that stage the roles will have been updated on the shift server via the ci job. And now, when we run chef, the roles will be picked up with the changes to the roles, which is our new environment variables.

A

We want to run chef on the psychic nodes, there's only one of them. There were the three apis three goods and three webs, so what we'll say is uh actually we might just do that first, because that's nice. Yes, I could go down staging's, not the end of the world and just verify.

A

And we'll get into a little bit more detail about that shortly.

A

And then for the other three, so we've got these other three sets of nodes. We have three apis. Three gets three webs. We might look at this in parallel dash, so what we'll do um is we'll copy that.

A

Red notes, but what we'll do is we'll add one c1, so this will run.

A

So knife ssh will go around this thing, but only the c1 will only run it concurrently on one of the time. So with three nodes, one of the times, it's reasonable yeah one, two three um for production, yeah with 12 nodes, 18 nodes on the right on the web fleet. I think it's 12 at the moment.

A

uh Yeah we'll probably do two or three at a time, especially given that it's 1 30 in the morning utc, which is the real quiet time, yeah half the capacity if we had to and that just keeps it keeps it ticking along without taking too long I mean each of these shift runs takes you know two or three minutes. It will take plus restarts time for um for rails to restart. You know you, you don't want to be sitting there for an hour watching the stuff watching paints pro so the same for.

A

Api and get, I think so what I will do when I run this is, I will have three terminals and I'll run this in each of the terminals.

A

um Yeah and now one of the important things when creating change control is to think about how you're going to roll this back now, a lot of them are revealed to roll back. They are revert them. I do the same thing again. Some of them are not, but even the fact of stating we're just going to revert, the change in the mr and rollback is good, because then you've thought about it.

A

um Monitoring production just yet um so actually yeah.

A

So I can probably define that there.

A

So I'll probably do that is.

A

Cool, so what are we going to do? How do we know this has worked we're going to check the view stage like it logs.

A

I know you can save logs sub, save queries and cabana. I'm not convinced they last forever. These short links, but we shall find out.

A

Oh there we go. Thank you oh indeed, and we can leave seafood and cpu around there. It doesn't matter terribly much, um it's not in vms anymore. We've already got catch all. Basically, what I'll be doing here is looking to make sure that something is still occurring um and that they uh yeah starting done they're not failed.

A

Logs are a good start. Graphs are even better.

A

Let's see if this still works, rails database connections, there we go, and that is like a catch-all cool. So we can see where we've got this. So what I think we will be expecting to see is that we are bumping that number a little bit. I think um so we're expecting to see that um this effective percentage drop a little bit uh reduce saturation.

A

So that's what we're saying uh and then what we will say is.

A

We'll, probably just so, actually how would we do whip? I would probably.

A

Get some grass, perhaps the easy one. I think um there we go just pop that in there.

A

um Also, we heavily use these check boxes, so we can check them as we go. um Don't always get it at the right time, but it can be quite handy, um especially because you actually do get along in the issue as to when it was clicked. So let's get that weapon get. Let's just check that that.

A

Does show something cool that does work.

A

A

Cool and honestly, I think the logs are not going to be that interesting they're going to be. I guess it's a start. It's probably a bit easier to see for sure.

A

Honestly, I think we're looking for rails logs yeah. I don't have an error message: no q and status.

A

um What's this look like.

A

And that right yeah, so we're looking for. What's this one.

A

Here, yeah, don't know some medical on earth.

A

I guess it's not an info, I guess we're looking for what we'll do is we'll drop out. Brac attack.

A

All right, let's course with that. ah Actually, so I guess it's actually status that we care about there we go um so we add status.

A

uh Did I just pop out in the photo level.

A

A

Okay, this might actually be hard to. I guess we're looking for normal looking traffic. uh What I might do is just drop out a few obvious.

A

A

Wait what the shard go. Oh of course, we've got rails on there too. Don't we that's interesting.

A

A

So what I might do is we'll just sort of save that one and call it good enough. I mean it's there's a little degree of errors that happen.

A

Let's just note that we do see some of.

A

A

A

That's part of the background noise so we'll just accept that, there's that what we're saying is and what I'm trying to do here is get it to a point where I could theoretically hand this over to someone else and they'd be able to reasonably successfully navigate this um run it through and do it themselves yeah. You don't always expect every person on this very team have exactly the same capabilities or knowledge of the system that you're working on um in this case, you know we're just making a more affordable change to the whole yeah.

A

Everyone in the sra team should vaguely know how rails works, so they really should be able to run this through. Look at that and say: hey yeah, we're seeing about 80 errors to set um errors for whatever just letters every 30 seconds. That's normal! That's fine! We'll we'll run with that. You'll! Look at double, we'll look at heart, whatever cool, um so I've actually described the key metrics.

A

A

A

A

A

Staging I'd still probably let this uh the encore, know and again staging isn't super critical, but people do rely on it. You know qa relies on it releases. You know we should let people know that we're doing something a little bit unusual here and we check that there's no active incidents. Otherwise we can confuse measures immensely, thankfully, when their warnings need to be pretty.

A

A

So what I might actually.

A

A

So that's something like station.com would probably do.

A

The window, what can I say.

A

A

That's good enough for that, because we are, we want to have a very quick look at what's going on there. I've just kept the keyboard out of the thing on my key monitor.

A

um I think that's about it, though, um so I'm going to assign that to myself. No other details needed here at the moment. There's a bunch of labels in there that will be created.

A

Cool that's about it.

A

Hopefully that's been a little bit enlightening. um This is a simpler one. We can put some very complicated ones, I'll, try and get some of those sometime ones that are a little bit less obvious. What's going on more at a time, but for now I hope that's helpful.

A