GitLab Delivery Team, 11 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-05-11 Delivery team weekly rollbacks demo

Description

Testing a rollback on Production

A

B

A

Oh before we start I'm going to transfer this over to you're the host alessio, I'm going to have to drop in a bit for an interview. Yeah.

C

But this is this is your account because if you have to drop, but you have to start something on your account, it will not work because it's just one meeting for the recording. Do you mean no just for so it is. um You can have one only one meeting per account running at the same time. So if the next thing you have to do.

A

C

A

uh That's a good point because.

C

This is if this is a problem we can use. Let.

A

Me try, let me just drop and no no, it's fine. Let me just drop out and.

C

We also have the ama, so this can still be but yeah. Let's, let's try it.

A

Yeah, that's a really good point. Let me I'll try.

A

Okay, so let me see if I can join my other one.

D

Okay: okay, oh good! Yes! Well,.

E

C

C

F

I'm having a little fun.

C

Strobo party.

F

I'm here I'm just I'm still like catching up on my day, so I'm just kind of swamped and lots of little tiny red dots next to a lot of larger icons that are just everywhere.

F

That's why I don't use notification, you know if I could get rid of notifications. I could, but you know gitlab kind of enforces it. uh Linux and various applications enforce it, etc. So you know it's it's kind of everywhere.

F

You can't escape it man, you can disable on mac os 6.

F

yeah. I bet you can.

F

Are we uh at the change steps now.

C

Yeah, I just give you a quick recap because you joined this is just the beginning of your morning, so uh we entirely skipped seven utc package. I mean we didn't promote it because it was too close to the the time of this. So we decided to just skip it. So three a.m to utc am was rolled out to production, post deployment, job is cancelled, seven rolled out up to canary and and then it's just dead there and 11 utc is ready, is rolling out in to staging right now and I canceled cannery and on so.

F

It will not progress any further, so our goal after we're done, is to promote the 11 into production. Well, after canary.

C

Or seven I mean what one production yeah it can even be seven. It really depends on what.

F

C

We are when we are complete right.

F

Gotcha all right all right so I'll begin uh make sure there's no ongoing deployments. So that's easy. I just checked the announcements page and look for anything. That's currently running only staging is reading canary is finished and the last production job looks like it was finished as well. So let's see.

F

Oh check that box, if I can't.

A

See you want to share your screen.

F

A

Do do we uh have a step early on about uh notifying engineer on call, or is that.

C

It's after we, we do chat before.

A

We can run.

C

It and then we just communication and things and things like that. I'm.

A

Just going to post our zoom link, then in uh in for a lounge, so if people want to join us, they can but keep going on that yeah. It's one of the steps, also that one no, but uh we should yeah like people mentioned they'd like to join right, so have we done it already.

C

No, I'm just saying that, as part of this step, that skyward.

G

A

G

One of this.

D

C

Posting the link, okay.

D

Yeah rehab was interested in joining, maybe to also look into this because she was doing the um uh uh hot patch uh fire drill also today, so we can look into the new world idea with that one. So maybe she's training.

D

D

F

Okay, hi maren. How are.

F

G

Doing well, how are you scarbeck.

F

I'm doing okay, I rescheduled our meeting.

E

Oh cool. I was looking forward to that meeting though wow tomorrow.

F

All right, okay, so I am on the step to find the packages to roll back to um I'm going to follow our documentation.

F

Let's see so open the link to our documentation.

F

Do the rollback check command should provide me the deploy command that we need to run. So, if I go back to where I ran that which is inside of slack and it's the last message, I posted in f upcoming release.

F

It currently tells me the robot command is to run the. uh We want the 2021 511 package.

F

Unless you- because this is an experiment, can you confirm that is the correct package? I believe.

B

C

I couldn't remember.

F

Which package you said.

C

The one in the rollback command, or it.

F

Wants to roll back to 13 12 20 21, 05, 11, 0, 320.

B

So this is the the time stump. Oh okay, I don't know if we fixed this, so we should check.

F

That's why I'm asking.

B

So timestamp here is that doesn't so.

C

The current current, which is 24 to a so okay as we know the command, is not right, as we know, because it's uh we cancel the ongoing deployment. So this is a this is a known bug, so the the correct pack. So there's a you. You should follow the manual step. The wonder are collapsed.

F

Very well, I will follow the manual steps because.

C

With this one, with the sha of the current, you can get the correct package version.

F

Oops so the sha of the current, which is 242 a.

F

F

So this will be the package that we roll back to yep. Now, where did my documentation go.

F

I did and we want to replace the plus with the minus so perfect, so I basically want to do that same command, but we want to roll back.

C

Yeah, if you go back to the incident, it's not sorry to the issue. We have some place where we have to supposed to drop the package version. Yeah.

E

F

C

F

I'm just building the command that I need to perform the rollback. With that way, I could just hit enter.

F

All right, so that's the command. I'm gonna run in here.

E

Let's see find the.

E

F

Oh, we put the command in here guard I'm over here. Building it.

C

Yeah mara did a great job in expanding the steps again here.

F

All right, so we found the package. I wrote down the package per our run book right because our runbook says to drain canary it does, although we should probably make that step more clear, but regardless we know to drink canary.

F

F

F

F

Just out of curiosity, I'm sharing my screen when I'm flipping back between two different screens. What do you guys still see.

C

We only see your browser okay,.

F

I just ran the chat ups commander, disabled canary in the production channel, where you afford to return results to me.

C

Yes, I'm I'm boosting your message so marin, you posted the message in the thread, but I'm going to send it again with uh also in what's happening at gitlab, so that I don't think people would just see that in yesterday make.

G

Sense, I missed the button and I.

C

Usually, do this yeah just and then cancel and write it.

F

Again, all right there we go canary is now drained, so we can now perform a.

F

F

Over in this command and f upcoming release actually know what I'll run this in production.

F

That way, it's clear to, because not everyone is an upcoming release.

G

What dashboards are we looking.

F

F

We're going to be looking at the metric slas and the platform metrics, I should already. I already had those tabs open.

F

Pipeline is started.

F

So if I recall correctly write down the link.

F

F

Written down, let's see, we want the gillian prefect as manual steps. If I recall we're not going to see that the actual manual steps until we get closer to the end, so I'll check back yeah.

D

Yeah, the preparation you need to reach the preparer.

F

Yeah there's no assets job. That was part of. I forget the name of that stage, but that stage doesn't even show up no warm-up job, no warm-up job.

F

Monitor the prepared job which this is where we get hung up on that's here: oh checks.

F

C

There is the deployment is still ongoing because we canceled.

F

I was not expecting this one to fail, though right this was gprd checks.

C

Because we, we don't have it in staging and we introduced the deployment locked check because of the.

B

So if you oh just be sure what it is.

C

The downstream pipeline should have the the failure, and the flare should be that there's an ongoing deployment. I.

H

C

H

Pipeline link on the chart.

D

Also, when you did the production check checkups, you got that we are not ready for rolling back right, because exactly of this.

D

F

Let's say: production error: what's the unsafe error.

B

Though I can show you, if you.

B

C

So I'm running in the f upcoming release chat ups around algebra blockers just to show the result. The result will be that the the production environment is locked for deployment because of the ongoing deployment, which is the pos deployment, the um post deployment job yeah, the the my personal migration job that we cancelled exactly.

C

B

Never saw this in our.

C

Testing, because when it runs in dry, run mode, we ignore. Oh, this is skipped yeah, that's it I mean it gets triggered, but he just he can. He can never fail.

F

So I think I corrected you when we created our documentation that we need to look for this job. So I guess we need to modify our documentation to switch it back to that job right.

F

B

Can you repeat again.

F

I think I recall when we were discussing yes.

B

C

So this is the same problem yeah, it's the same problem so, but it also happens uh one step ahead and in this.

B

Case the failure.

C

B

So not the failure just skip the way we skip the failure.

F

Yeah, it's ignore production checks.

D

F

Is it a variable that I could set or do I need.

E

F

Something different with the rollback command because we do this, but.

E

F

If I roll back, I mean it's going to accept that right.

B

You are right, this is always said this ignore production checks there, but where is it coming from.

H

Do we have post deployment migrations like on the previous? Yes,.

C

H

C

Otherwise we could just move forward, but now we have them.

B

B

B

So this is the deployer, so let's just check in the deployer how we skip this because it's in our documentation.

C

It's handled by there's a variable.

F

Should be ignored, production checks and I should be able to set that to true correct.

F

F

Yep variables, goodness.

C

Yeah, let's ignore production checks.

G

Please be careful uh with revealing values in a recorded call.

H

Skip deployment check.

F

Skip deployment check is for the other job. I think we have two right.

H

I'm just okay, just double.

H

F

Skip owner's roll check is the one that we documented that bypass the omnis updates, and then we've got this other one.

C

Yes, basically in deploy tooling being promote there is this ignore deployment checks which defaults to false, but if it's provided it will just take the value yeah.

F

All right so I'll set this to true variable, add variable and then theoretically, I could retry that job.

B

Take note of what we did.

F

Yep, so this should succeed.

F

F

F

And what's the type of the variable name.

B

And what no it's a trigger variable.

B

So I think that we have a problem with variable priority because it was provided from the outside.

B

Then he has priority over variables, defined at job level.

F

Okay, so the other option is that we could use chat ops, but does chad ops send the appropriate flag over in the deploy command? Is that repeated yeah.

C

I mean yeah, it should so. We.

F

Are rolling back with.

C

So we're rolling back with a chat, ops deploy command so that one has the trigger variable for doing this. So let's double check.

F

You're checking the chat out source code yeah, I'm checking because it's just.

F

B

So ignore production checks is a valid option.

F

C

It should be I'm checking if we are making.

C

C

Yeah, it should yeah it's fine.

F

C

Every variables is composed together and it will provide, it will trigger.

B

F

Before I run it, I want to remove that variable that I just set.

B

F

Let's do it just fine check, so I'm going to leave that in here, just prefix, it like we've, been doing with everything else: okay, so I'll close that ignore production checks. Hopefully I spilled it.

F

Correctly just ran the command of the production.

F

C

We can check the trigger variable to see if we have the right variables.

F

B

False this is not good.

B

C

Okay, oh right, sorry, this is a string. It's supposed to be a string, because it's the reason why you you're skipping it so we're using ignore production checks during regular roll forward deployment, and so we provide something like um sri on call allowed. This deployment or the blocker is not relevant to the ongoing deployment.

C

That's right, so we had no parameters in what we sent now. So we just said you just left.

F

The default value of.

C

F

Chat up should really complain if we do.

F

That all right, I will provide it a string and we run.

F

It three is on its.

F

F

F

We got set to a string which is good.

F

So this job should now pass.

G

G

Let's stick for a sec here. If we are rolling back our progression, production checks are going to fail because we have an incident ongoing. It's active so just check. What does it do then, in case of rollback.

B

C

Useful right there you go cool.

F

B

C

B

F

Just it really should just ignore or maybe set that value for us yeah.

C

I mean it leaves it leaves a note in the monthly issue. That's the only thing that is doing, but maybe because it's a rollback we can just set in no production check. This is a rollback so that it logs the rules back.

F

In the monthly.

C

Issue with just a message that we are rolling back.

F

Yep does the robot command also drop a comment in the uh release issue.

C

It's a promotion check for uh regular deployment in that regard,.

E

C

So I'm going to check if we have a message it should be there.

F

All right so we're now at the gprd prepared job and we're expecting this one to fail for a very similar reason, because I've not set the skip on this roll check. Yet.

C

Okay, so we have messages for all the failure and the final one where you say this is a test for and the link. So it's actually logging rollback attempt in the monthly issue.

C

Okay, I don't know I wasn't following the screen, because I was checking this. I don't know if you changed the link to pipeline because we rerun so pipeline links in the change requests are, are the old one and not no.

F

C

F

I need to do that, but in the meantime we did hit the retry loop, so I will cancel this job.

F

And I will go here copy that settings ci.

G

Why did we hit this loop.

F

G

In this case, I know, but what I'm asking why.

F

Unless you want to explain that real quick.

C

Yeah sure, because we cancelled possibly migration to not have to deal with a package that we know that has no positive migration. So the the simulation allows us not to have to deal with timing in creating a fake package with the extra complexity that we are running this with an ongoing deployment that is cancelled. So for from the chef point of view, we are in the middle of a deployment.

G

Can you please write it down in the notes yeah? I.

C

Think it's already in the change.

G

Yeah right, but I'm the.

I

Reason why I'm seeing.

G

This the reason why I'm asking this is because this is something we would probably want to address, given that um in in the moment where we need to do a rollback, there is a chance that there has been an ongoing deployment somewhere and so on. So we'll need to either write down the directions on what to do here or improve the situation. Yeah.

I

F

G

I

In in general, our check for ongoing deployment uh will change soon right. We're, like I know in other places, we're using the release api to see like whether there's a deployment in progress. If we were to change it, to use that this problem would go away correct.

I

uh No, because.

C

I

C

Is still running, environment is still markets.

I

C

Because we canceled positive limit migration, so it's still marked because we.

I

C

I

C

Will be failed, it's a different stages, but we.

I

C

I

I see and there's really no way for us to market as failed during the for cancellation.

C

No, we need a manual job or something like this, and we have to remember of doing it.

G

Right, but you can imagine a situation where you know a production uh deployment failed, mid deployment for whatever reason we're in an inconsistent state. We want to roll back because something has happened, which means that our previous deployment actually failed again, which means that our rollback will run into this same same situation. So it's not.

I

No, no. What I was saying is that if it was failed instead of running, then we could in that situation, then we would just continue right, because the whole purpose of this check is to make sure we don't have two deployments that step on each other.

G

I

G

I

That makes sense.

G

Also, sir karbic, are you sure this is the correct dashboard? We need to look at. I see only web.

F

I think I'll, probably just look at our home page dashboard. Instead of this one.

G

I'm not looking at this one I'll link it in delivery channel.

F

C

Yeah, I think that one that is linked here is just uh there's a drop down where you move from service to service. There's not a big overview of all these yeah.

F

But I don't want to be doing that. I don't have time for that. I'm touching all the services, not one.

I

Yeah, I would use the one that merit like two. That's the one. I use interesting to see this rps jump. That happened right after the failure uh for web kind of strange.

F

I

Yeah, I know I know it's definitely not related to this. It's just.

I

F

All right, so the rollback has begun.

F

I'm just gonna click on two of these and swap back between the two just to see how they're.

F

F

F

F

Is there anything we should chat about while we wait.

H

Why are we keeping an eye on the kubernetes pipeline.

F

I just picked two random jobs that I just wanted to watch. That way. I had something to watch outside of dashboards.

F

Just something I kind of sometimes do when I watch deploys happen too, but.

H

But it is not related to this rollback or is it? What is it.

F

It is, this is from the.

H

F

H

H

G

Is the dashboard that I'm a link to you loading for you, because it stopped loading for me.

C

Yeah I've got it it loads for me as well.

F

C

F

It's stuck in a loop but yeah. I've got it refreshing every five seconds.

G

There you go that it's not.

C

Good it's there, but it's not refreshing yeah. I just opened when one second before you asked so in that case it just loaded yeah.

F

All of us watching the same dashboard bringing graffana down.

G

It also for me, it hanged in the worst possible moment where, as I was looking at api aptx going down in saturation going up so that doesn't look.

G

F

The single node puma worker component.

F

C

Yeah, we have drained cannery and this is taking machine out of rotation.

C

So the question is: does the fleet handle this.

F

That's a good point. I did a web.

G

Saturation is also high by the way and the web is.

D

F

Let's see, I did the math yesterday and canary actually takes a decent amount of traffic.

I

It's about five percent, so.

F

You're concerned.

I

So your concern is that we like we're overloading the main stage.

F

I

But we so we do canary deploys all the time and we train are. We is canary drained right now because of the rollback yeah yeah. That's the two.

C

I

I see I see so it's kind of drained for an extended period. Yep um yeah I mean we are.

I

This is why this is why the rps went up. I bet yeah, that makes that makes sense yeah. I should have thought of that.

C

Is this a situation we can sustain, or is it.

I

We haven't tested, are we? Are we in peak or not? That's a question.

D

um Still, okay, I mean we are not below our.

I

D

Latencies but you need.

I

To watch it, I um I thought an api was a little bit worse than red, for uh you know capacity right now. We might want to take a look to see if we should expand those fleets.

F

Web, yes, api, I'm hoping to start rolling kubernetes traffic soon.

C

A little bit so what I'm thinking now is that so we do the fleet with uh in batches, so the number of machines that are out of rotation is constant over the the deployment. This is correct right because we remove.

E

The cannon so instead.

C

Of uh complete fleet, we remove ten percent five percent, whatever it is, but it's still the same amount. So if we can sustain it during the first wave, we can sustain up until the end of the deployment, because.

I

The amount of machine.

C

Of the rotation would always be the same.

I

Yeah, it's just with canary, drained right, you're dropping not only two nodes for the deployment, but all the canary nodes, which was taking five percent of the traffic. uh It's not ideal, but.

G

Considering seeing this web uh web drop at the moment- and uh I just.

E

G

Yeah there you go sorry igor.

I

J

Yeah yeah I got paged for web yeah makes sense.

I

Are we are we okay turning canary back on, or was that be a? Would that be a problem.

C

That's one way we can kind of help because we have no, I mean there's no real, so we we removed cannery because in theory you are simulating that you have something broken right. So you you want to remove.

E

C

But we have nothing broken on cannery, so we can put it back online. I I would say, let's add canary back and yeah. I hope.

I

That I could do.

C

Yeah point taken: we have to consider this because in case of a real incident, we are not able to sustain uh our rollback with our traffic.

G

Iso, can you write it down please in the notes, yeah, I'm going to write it because I think we might even need to think about having some reserve just standing.

E

G

And um when we drain, we put those.

I

Yeah we're at unfortunately, we're at peak right now. This is the hour where we hit the top like of our rps, so that's playing a part for sure.

G

But you know murphy how it works.

I

E

G

We need to kind of.

E

G

Out how to do this? Because we need to do these practices more often, and they need to be without any impact for the uncle yeah, fair.

G

F

All right canary is now back.

F

G

um You see we're recovering a bit, so that's good.

G

On the web service.

G

Eager, how are things looking at the pager at the.

J

J

Yeah seems to be slowly recovering, but let's give it another minute or two. Oh.

J

J

It's definitely interesting to see how the saturation metrics respond, because it it appears that the main saturation point is actually cpu on the nodes, and once that hits a certain level, we see a very sharp increase in the the puma.

J

Puma workers, so I think we need to definitely pay more attention to the cpu as kind of a leading.

J

G

I'm really curious, like I can't wait to see how this is going to change when uh these two services run in kubernetes.

G

When it comes to rollback.

G

C

In terms of timing, I I think that in kubernetes right now we are something like four times faster than what we have with vms, because we are already on the zone c. Then we have zone d and we are done it's how long 20 minutes since we entered the fleet upgrade so it's impressing compared to I mean api will take one hour even one hour and.

C

G

So one way we can think about this lack of capacity uh when we drink canary is to not drink canary, so figure out a way to deploy both canary and production, or rather roll back both canarian production in one go. When we start is it one.

F

Way, I think I would rather not do that. I would rather see us expand our fleets.

F

Me too, I think doing I was doing that, would eliminate just the fact that we need to solve that problem for one and two um I kind of want canary to be a representation of what we can enable after we get our next deploy rolled out. So theoretically, when we come back from a fix canary could then be enabled after the defects has been deployed, and then we simply re-enable canary send traffic to it as validation.

F

I guess that's kind of the same thing, regardless of the fact, but.

C

But then the problem here is made worse by the fact that these are vms right. So we don't have the ability to just sk scale up in terms of minutes, I'm asking because I'm not familiar. How long will it take for us to say had five more machines to the fleet.

F

Hours it's a multi-hour project.

F

After the evaluation that we are within quota limits and making sure terraform still works,.

D

Fiddling with aj proxy roles and chef and stuff yeah, so that's the nice thing about kubernetes right. You can easily just add containers instead of setting up a full machine which makes it slow and also that fee and kunis can scale up and then, after that, take away some pots right, the old parts. So we don't go down some some number of machines and doing this in the via m fleet would just be too slow or we would need to start way before we do the deployment, so how they have uh reserve would be better.

D

G

Maybe we should start the kubernetes migration of some sorts.

F

I'll try to start that soon. In the meantime, jarv and igor might be worth having an issue to up the size of her web fleet. They're out is it oh. They left us.

G

They said this was below their level, so they left yeah. This is the boring part.

F

F

C

Is that saturation going up again, I saw a spike in the.

F

We're going to see it bounce around for quite a while, since we're taking servers out of rotation during the deploy okay. So it's going to bounce until the deploy completes essentially a little worried about the api being so high in general. But I think, if we're at capacity for the api, that's going to be kind of risky.

C

I don't know if you I don't know if you saw the message: jarv posted the link in the on slack, we're basically showing this. We are a traffic peak right now, so there's things showing the traffic and basically, we.

C

Are yep that one.

G

I thought you selected this time specifically because of that you wanted a harder challenge.

C

Well, you never know when this happens right, so it has to work on every situation, so in certain sense, I'm glad that we picked the wrong time so that we know now that everything is safe and there's no real incident that this is a problem. No.

G

You said it: alessio come on you've been in infrastructure for long enough to know that you shouldn't mention the e-word, because it will happen. Man.

G

G

Now I'm nervous, I wasn't nervous up until now. I'm.

G

C

So the api fleet is 41 percent was lucky enough to open the job. At the same time when we, when it was outputting the percentage.

H

You can follow the downstream jobs to check the percentage. It's not the friendliest way, but you can also do.

H

G

So what happens with.

G

Get ssh traffic so get traffic in general and container registered traffic in general.

G

Do you see anything for container registry specifically, given that it's in kubernetes.

G

We didn't roll back because it's the same version right.

F

Yeah, the register is not being rolled back right now.

E

All right, because it's completely separate right precisely.

F

E

F

G

And get ssh or you get in general.

F

I get ssh the gitlab shell that is being rolled back.

G

Okay and that's because we use the shots there right, so the shots are different.

E

G

F

I could probably prove that to you.

G

I do trust you but show me anyway.

F

F

A little bit easier, if I look at the def.

F

Job, it's not user! If I look at the diff job, uh web circuits get and get web shell, so they were all.

F

F

F

It's the web service.

F

It's also the web service.

F

So we get lab shell deployment and modified its image from 8186 to 24280.

F

So they get rolled backwards, went from 5 12 to 5 11.

G

Okay, cool thanks for.

G

C

Who was checking the help page when we see the version and if you keep refreshing, just flips from old machine, a new machine? So we can see both versions coming back depending on when you refresh. So we are because we are around yeah a bit more than 50 percent of the fleet.

F

So alessio we've got manual, jobs for getaway and prairie effect is the expectation that we perform those today because that's not in our procedure.

C

F

Okay, good because I don't want to wait three hours for getaway to deploy.

I

Three hours really, but how long it takes now.

F

It's close to two uh there's a bug in ansible, where it's not deploying serialization or calculating sterilization properly open an issue for it. I just have a.

I

F

To really dive deeply into it,.

I

Yeah, could you send me that I'm curious and because we're we're doing a hundred percent of the fleet simultaneously right.

F

E

F

It's configured to do 50, but it's not calculating the roll properly. So it's defaulting it's falling back to our default of only 10 of the nodes at the time. Oh yeah. I can take a look at that. That sounds like a bug.

I

Somewhere, um I think we should consider doing 100.

I

I see no reason to do 50, except for maybe a little bit of safety, but it would be kind of I. I don't see us recognizing a problem between the first batch and the second batch right, because they would happen right after each other.

F

I would be mildly worried at 100, because we'd be opening up connections to 72 file servers at the same time, and I'm.

I

F

Sure what ansible will do with that.

I

F

To be able to handle.

I

That come on must be. I.

F

I

F

You know I'm just playing a little cautious here. Yeah, like my worry, is that we have not been deploying to 50 percent of the fleet. I don't think we've.

E

Ever done that properly.

F

Based on how often this has been happening and um robert showed a chart where employed times have always been excruciatingly high yeah, um so.

I

Let me see, let's fix.

F

The bug first, I just tagged you on it: let's fix the.

I

F

And then, let's think about ramping it up.

I

Yeah I'll take a look at it. I'll say it to myself.

I

Oh you, just you literally, like just noticed this 16 hours old. I thought it was a long-standing issue. I noticed it.

F

Last week I just took forever to open an issue about it: cool.

I

G

If you take a look at the web saturation, you can actually see how we are putting vms in and out of rotation.

G

This integration graph is like so, although I see something in uh git service as well so like.

I

Interesting um during the deploy, so you think it's like all the fleets. Maybe we should just add capacity.

G

Well, it's a bit surprising to see this in git, because git should be completely independent. Now from vms.

G

Oh right yeah, of course oh, but it says railsdb connection, pool component 80, that's odd. Why would that be a problem.

F

I suspect it's when the new pods are coming up, they're going to be opened up for new connections prior to the old old pots being turned down, ah so we're doubling. Basically right not this is doubling but like maybe 25 percent, more being added at a time during the deployment process.

G

hmm I wonder whether that's going to become a problem at some point.

D

Yeah, we have to be careful with these numbers, so if we fiddle around with our scaling strategies like how many at the same time will be added, we obviously need to think about um how how that works with the database connection pooling and things like that.

D

Same as we saw with um pre when we added real load to it and overloaded the databases there, because they are configured for only 25 connections for registry and perfect and the main database in pre is only taking 100 max connections. So if you really run qa jobs there, you just run out of db connections.

G

I'm gonna add a note to our doc and let's, let's create an issue for this and talk with scalability team about it. A.

G

J

How are things looking seems like we're still dancing around saturation point a little bit on web.

C

The web fleet is at 83 percent now, so we should be close to complete the draw.

C

Out so if I'm just thinking so if adding cannery is enough to say, keep things uh working during a rollout means that we are out of capacity for five percent for peak, because now it's peak, so we have the highest amount of traffic coming in and I think we mentioned that uh cannery is taking five percent of traffic yeah. This can't be true because we.

D

C

Everything github.com: this is.

D

Always changing all the time right I mean vv, we sometimes adjust api or web feeds for my traffic. Then after via traffic grows up, then we need to adjust again, so it totally depends on where we are with scaling and adjusting to the traffic that we have. There are weeks where we are good with that, and then it's slowly starting to get bad again during deployments, and you see how we reach saturation.

D

So I mean once you end coordinators. That should be better because we can automatically scale but on our vm fleet, we manually scale, and then we end up with this one or you need to really look close at saturation metrics and then adjust beforehand, which is more or less what we do right. We look during deployments or when canary is joined at our situation matrix and see. Oh, we are getting close and then we need to adjust again.

D

So that's how it works until.

D

J

Hello- and I guess- and I guess we generally also use tamland for more for forecasting on cpu utilization right. We've used that in the past to inform the decision to scale and add more api or web notes.

C

Yeah yeah, my point is more that if, when we are at peak right now, we need cannery to handle the load. Then we are kind of running really.

E

We're under provisioned, yes, yes,.

C

Actually, okay, that's the point right and yeah. That's the fact right, because you I mean we are, I mean canary, can be drained at any moment in time because of various reasons. So if you rely on cannery to handle the normal production load, then you're under provision.

C

A

How are the robots going aside from that and just having a quick scan of the document.

C

Yeah we are at 80 something percent of the main fleet for web and last time last thing we checked as well and yeah. We had some hitchcock in the beginning and then now is, is rolling out. It's rolling back, but yeah.

A

Was that just at the beginning was that just about getting the right override variable.

C

Yeah we had two main two main issues: one was the override, something that we overlooked, because in dry run mode, those check are always skipped. So basically um there was we had ongoing um deployment. We want. We canceled.

E

C

This was preventing us from starting the rollback and yeah we we we haven't this documented, so we had to figure out how to handle this, and then we were able to start and then the second problem was that uh is this thing about um being able to handle traffic so because we drained canary as soon as machine got out of the load balancer uh the saturation started skyrocketing and we had a paging event, basically in the web fleet.

C

So we decided to put cannery back online because in this case their cannery is healthy. So it is just we're just removing it, because it's what we would do in case of um production incident, and so that's what we are. And now we are looking at this and checking the status.

G

um Amy now that you're back, I think I'm going to drop off for my next meeting. This looks good so far, so fingers crossed everything ends up fine as well.

A

Thank you thanks.

A

C

So web is 91 percent rolled out. Api is 83 percent. Everything else is.

A

A

A

Is that a little faster than we expected less than an hour suppose it depends how long the web takes.

C

Yeah, we are yeah 15 minutes now and it's not completed so yeah.

D

Who's, the host.

B

I think it's me.

F

I just got a message saying you should unmute.

F

I have nothing to say.

A

It's like a passive aggressive, like you need to contribute yeah.

A

You can have the alternative right like where it just meets people when they've been talking. If you've been talking for like more than five minutes, you get muted.

H

So web just finished, we are just waiting for the api now.

D

And I need to drop off now for half an hour or a little bit more see you later.

A

See you later see.

D

F

And we've got six more nodes left in the api fleet.

A

So that's quite exciting right, because that means that once we've got api running in kubernetes, the rollback should actually be quite a lot faster than we expected originally right, like we're, probably looking at around an hour or so no we started. Do we start like what time to start we started like no, we started the.

C

Recording is one hour and a half, so we chatted a bit at the beginning. Then we had some hitchcock been starting. So yeah probably is one hour an hour.

A

That's great: how long does a hot patch take? Does anyone not roughly know.

C

It takes less 20 30 minutes, but there is a lot of pipeline timing ahead, so you still have to prepare. The merge requests have a full pipeline on it and reviewed. So this is more than two hours I mean you also have to find. Where is the problem.

A

Of course yeah yeah, but I mean that's, we are.

C

In two hour two hour and a half, then you prepare the patch. So when you have the patch prepared it's kind of five minutes for the pipeline in the merge requests review and then to rule out, it takes 20.

F

40 60 minutes.

C

F

Validating that it works in the lower environments as well prior to rolling it all right.

A

Right, okay, nice, so robots would be considerably faster. I actually wasn't expecting that. I was thinking that rollbacks would um be slower. I was expected to take the like time of our production deployment so like two hours um but be safer, that was kind of in my head. What the benefit of rollbacks was so great that they are safer and faster.

A

F

Let's, let's um the word safer, let's tread carefully on that since well.

A

I mean safer in terms of that we are reverting back to a package, that's fully tested and has been through all of the all of the.

E

Lower environments.

A

It's not new code, I mean sure the way it behaves like across the servers is different. Maybe, but I mean in terms of like we're not putting new code on having cut corners.

C

It's a new it's an old stage that we already had in production and we are fairly confident.

A

Yeah, like the code, has been created and merged and passed through to production in the ways that it normally has aside from the rollback so.

A

One thing I realized um the other day is: what do you think about uh the rollback pipeline posting a message on the release issue? Do you think it'd be worth like useful to actually it's.

C

Posting the message: oh, is it that's the problem why he didn't start it this morning, because he was checking.

F

C

The message there's an ongoing production. This starbuck is trying to do something wrong here and just writing.

E

C

So we have, we had a note in the agenda we are going to I mean the proposal is that if there is a minus minus rollback parameters, then we will, by default, put skip production checks. This is a rollback because yeah, if you're rolling back at least we have an incident, so it will never succeed. So yes,.

A

Okay, yeah yeah. It makes sense.

C

Yeah and we have a nice track of deployments of every of every kind, either a rollback or a regular deployment.

A

Exactly yeah great okay, I see.

H

So pizza party to celebrate anyone.

C

I had pizza for lunch.

A

But yes, absolutely yeah.

E

Let's do it when we.

A

Get the uh yeah: when we get this epic closed out, we should absolutely we should do something, because this is a huge milestone.

F

In order to close out this milestone, I would expect for us to be able to follow a procedure from start to finish. Without any issues, do we want to try to test this again.

A

I think it might be worthwhile right like um it would be. Certainly it would be good to not have the questions at the beginning like for us to know like have that documented and have the like. We should consider whether we do want to do that improvement to the rollback command so that we don't have that. The question about, like is a deployment. I I almost wonder if there's a step in our process, which just says, run the um the deployment blockers command.

A

So we have that information there. It feels like there's, definitely a gap in like what we need to do uh so that we have all that information, and we know like when it says this. If that happens, ignore that package and take this package could just be a doc update.

F

I think, in that case, in my opinion, I would also try. I would encourage us to try to get infrastructure to bump up the fleet sizing of api and web as well prior to our next test. That way, we could do the full motion leave canary drain like we want.

A

F

For our procedure, because right now we can't do that.

A

Yeah, I think we should do.

F

A

F

This during apec time- oh yes, I.

C

Was going to suggest, we can do an epic time.

A

Well, actually, do you know what you joke alessio, but actually it wouldn't be impossible. I mean, when I say apac time I mean kind of like late in apac time, but like we have a with the time with the clocks being as they are, we do actually have a time in the day where me you, graham and henry, are actually all quite frequently online and traffic, presumably is quite low at that time. So we actually could do one.

C

We can do one starting early emir morning, which is yeah.

A

Exactly I mean traffic.

C

From emea is not the same as traffic from the u.s.

A

Yeah exactly we could do that and see if we could get graham involved as well.

A

Yeah sounds like a plan because actually I was chatting to graham this morning, and I realized that when we get him into release management, he's potentially going to have quite um busy shifts, because I think he gets quite a lot of stuff merged in that we'll deploy in that first deployment anyway. I stopped for it. Exciting did.

C

It succeeded sorry amy, but.

A

This is much more exciting, yeah.

C

Finishing touches.

F

So we're on the finish task. So, while we are here I'll, go ahead, go through this step, so it'll pass here in a second, but just going to the help page, I do confirm that we are running 242, which is what we wrote back to so we're good on that part.

F

Our next step is to check, execute qa, so I've got a pipeline here, I'm just going to hit run pipeline and pipeline. I don't know how long this takes.

A

I think it's probably.

C

This will run our production.

A

20 20 30 minutes.

F

I don't know, I think it's longer than that.

H

We can check the previous one and try to see if there is some time.

F

Two stages yeah, so this said run time of five minutes. I don't believe that 52 minutes for this one. Four minutes: nine minutes, five minutes, fifteen. It's all over the place.

C

So because um it depends on how you trigger them, so I know for sure that they have scheduled pipelines that are running on the production check. So I don't even know if the thing that you started will run a real qa test on on master and sorry on production. So because, if you go on.

F

C

I'm under the production, uh yeah yeah.

F

C

I'm just sitting there if you go on schedule, and this is b howardly smoke test the second one. Second one. I think this one is the is the right type of pipeline.

F

In that ten.

B

F

Too short yeah: this is not the pipeline that I ran. uh Maybe.

D

There's three girls, something.

F

Here: let's what if I do this, what if I go to the pipeline, I started: cancel it go to that job and tell that job to execute how's that yeah. This is what I was thinking, but 10 minutes. Isn't it a.

C

F

Let's just do that.

E

F

Okay, it stopped okay, so schedules.

F

By early smoke test, let's just play play please.

F

Yeah see even that pipeline looks different than the one I created.

F

Okay, so we have qa has started, execute qa.

E

F

I want to see all green check marks.

F

All green so we're good closing that tab.

F

uh We want to roll this package out for at least 30 minutes. Qa will provide us at least 10 of those minutes or so, and then we can just continue forward with our to deploy stuff.

A

Yeah very nice awesome, hey work well done.

F

F

A

That's fine. How did that compare like to what we were expecting for a production rollback?

A

I was expecting something much longer um same. Actually, that was a lot quicker than I was expecting. Yeah.

F

E

F

um I think there's room for improvement as far as node saturation goes yeah and I think the steps that start us off on this process need a little bit of refinement just to make it a little bit easier. I think dry run was a hindrance, but that still enables us to fully test other things. We just found new, not necessarily blockers, just new obstacles that we need to refine a little bit.

A

Yeah, I agree the saturation one's really interesting, though, isn't it because um there must be times in our normal day where we get really close to that, because we don't, we don't spend a lot of time. Looking at the difference, like you know, we just let canary in production, roll kind of on separate schedules right, so it's unlikely that you would have those things running together or impacting each other, but certainly we could but yeah. I agree on the.

A

I think it's just that very first step when you get presented with the rollbacks checks, it's interpreting those.

A

And knowing like you know, really knowing like this was the situation, and so therefore these are the packages. I think that's going to be the the trickiest bit that we all have to master.

C

Yeah, I think we started with. um We started working a lot on that one, and then we reached the point that we were just refining that script and and not having progress on. Do we really know how to run the the pipeline? Do we really know how to handle problems, and then we moved on that aspect and we never looked back because we were kind of okay.

C

We we know how to fix problems or quickness of that one, let's see if it really works so that we can go back and publish and make this yeah say stress safe, because in a stressful situation, maybe you overlooked some of the that information. You roll back to the wrong package or things like that right, exactly yeah. So you don't want this to happen.

A

And we have a, um we have got an issue that we opened.

A

So does this issue capture everything we want it to have so improvements to the rollback uh check, command, uh scumbag opened last time we saw this so.

A

I don't know if we've got a.

A

Do we have a way forwards on this, like? How could we make this more clear, because we I we are definitely going to hit this right, and it's almost like.

A

The it's almost like in this instance, the rollback command is, is misleading right because that's.

E

A

Not going to be the package you actually want, I almost think that's. Maybe the thing that like doesn't, I don't know how much extra complexity, but what do we have something like if the if the deployment checks fail um or if they were going to fail like if there is an up coming, that's a new package, it's called now, then, should we maybe not have the rollback command or something and actually be like use? The run book.

F

G

F

Love to be able to fill out the robot command here, just to make it easier on us if possible. But if it's always going to be wrong, then it should not be included.

C

Yeah, if it has to be wrong, I'd rather prefer having package name next to every show, so that it's up, because the the things that I don't like is that if you go, we go back to manual. Then there's this extra problem that you have to figure out the package name which, because it's something that is automated, usually then you maybe.

A

Don't know how to do it.

C

Properly so yeah.

A

This is the problem right, so potentially we're going to roll back, infrequently and so based on release management schedules. We're probably not going to be individually doing this very often so.

C

We will schedule file, drills and things like that, so that we keep.

A

Doing this, but yeah like we should it's. That is those extra steps where it's unusual on some of the times. If we have so just to get logic on this one right that I'm going to call it upcoming, just because that's what's visible, um that upcoming line will only display if there's an incomplete deployment. Is that correct.

B

A

Would it be hard, then, in that case, if we have upcoming, would it be possible to make that rollback command? Also aware of that, so that it gives the right package.

C

Yeah, the there's, I think, there's an extra thing here that we have to double check. What happens if it failed so because we, I think we show up the upcoming, which is the new package. If something is running or maybe also if it's cancelled it fails or cancel, is not a right status now, if it's running it may be because it's really running or because we cancelled, usually we don't cancel, but you know right. You.

A

Definitely could yeah for sure, okay and if it's cancelled, it's completed right and there wouldn't be an upcoming.

C

Yeah I mean if it's cancelled, it's not completed, but.

E

It's not even running so we can go back to.

C

Current, but if it's running we we are asking for troubles here because we were discussing about, we know there will be an incident, so we want to skip by default. uh Promotion checks, fine, but it has. I mean if it's running. This is a problem.

C

If it's running for real.

A

But, uh but just to like, rather than skipping on I'd, rather than sort of talk about skipping on the checks like um is there ever a case where we would well, I guess, let me phrase this differently. How would we determine on that rollback command? How do we determine what the um previous, whether we should be using the previous or the upcoming package? Is that just literally a case of us knowing did the fleet complete running or not.

C

Yeah, because so one step of the of the production rule of the change request was uh make sure there are no ongoing production deployment.

C

Now we still live in a world where we release manager manually play promotions, so release managers are aware if there is an ongoing deployment or not, but that tool right now is not able to handle something like this yeah.

C

And so that's that's the point right. So I I don't know I mean is maybe I can let me try showing something. So if you don't mind, I can share my screen. Oh so I'm gonna quickly show something that I was playing with. It's it's local script, so it's not, but yeah just should give you an idea. So let me close this and this and all those things.

C

So here we go so I was playing on this two scripts here.

C

So this one check for uh pipeline in deployers that are so basically no just this checks for uh release tools, pipelines that are running on tags and out of that it just goes to metadata omnibus whatever and get out information. So it tells you this is red. So it's a production deployment for this thing. This is still running here, because I I can't is the one that we canceled. So is the 320 a.m, but disregard the fact that this is still market here. But the point is that we can start we can.

C

What is this one just open this up? So basically, with an api call, we can get something like this, which are pipeline for tags and pipel, and release tools. Python for tags are deployment and navigating the output of this we can try to figure out if something is running as a deployment, but but the things that I like more is this one here.

C

So this one shows you every tag package with time stamp and gives you links to outside branch omnibus whatever- and there is this, for instance, this one is wait for you to play it in the uh the promotion production promotion. Now this is because we cancelled, and so it's market has failed if it's running it shows you what is running so this gives you idea that, starting from um the tags on release tools, we can navigate a lot of information.

C

So maybe we can build on top of this, making it part of our checks to actually get useful information like? Is it really running or what's the difference? What's the real difference between this package and the next one, because this this does not rely on environments apis and things like just go from from releasing tags.

A

Yeah, that sounds like something: a good, a good thing.

C

Because there are insights there right so if we have a running deployment, then we can check this type of information because we have running deployment. So we have a tag or whatever- and we say is this really running and we can make it we can make the check smarter or even just bailing out and say something is running. So could you please check and links to where we can check.

A

Exactly, I think, that's the thing absolutely particularly kind of um when it's a stressful environment. I think as much we can just say like hey, go check. This thing make a decision all based on this type of.

E

C

B

C

How far are we from being able to continue this.

F

Q, a is completed.

C

F

So we're good to go in that front. So at this point we just need to finish waiting for the 30 minutes to be up which is about another 10 minutes or so, and then we could roll forward.

H

H

This is good news.

C

So do we want to roll forward with the 11 package? I mean 11 package still needs to roll out to cannery.

C

So maybe we want to roll forward with seven package and and in the meantime let the 11 package go to cannery and then we have baking time and so on and so on.

F

Why not just leave it as is, and let canary run the 11 package.

C

Yeah yeah, I'm saying do we want to I mean we are waiting, 30 minutes which looks like after 30 minutes. We want to roll forward, but right now we can only roll forward with the 7 and 20 package and not with the 11 one, because the 11 one is still locked at staging.

F

Let's unlock it.

C

So you are going to we unlock it, we let it go. We wait back in time and we and you deploy later on your day, yeah. Okay. I can't unlock it. Do that.

F

No reason for us to hold canary out.

A

No sounds good.

F

So I think, because we're going to do that method, I'm going to go ahead and just click the boxing that we waited 30 minutes, because it's going to take us uh roughly two hours before bill to do our next promotion and we're done with this uh change request.

F

F

Which is one issue for my to-do list, which is wonderful.

A

Fantastic nice work. It's exciting, uh see how to get closed off.

F

Big thanks to alessio myra. All I did was follow the procedure.

A

Follow them very competently, so we got done. You did.

G

A

Rollback so.

A

Fantastic, um what do we want to do about um planning next steps, then, um for for sort of future future improvements.

F

Okay, this is a good question, so, if we don't already, we need to have issues for expanding our web and api fleets. I know the api kubernetes migration is happening, but it's still in the process of happening.

E

F

An issue should be open just in case we don't complete that and they have time to pick it up. Yeah.

A

I can open that if you'd like.

F

Sure and then alessio we should have an improvement on just the starting portion. um We we could.

E

F

Reuse, the one that amy was showing earlier, uh but I think we do need a minor documentation change to help us um one. I think disabling canary is not prominent in our docs.

E

F

It's in there it's just not a prominent step, you know and then the other thing was um determining whether or not a production deploy is happening. I think that determining that is missing from our documentation. We.

E

F

We're release majors, but we need to be able to make sure other people outside of our team knows how to do that as well. Yeah and there's.

C

And yeah still in the aspect of of making the the first step better, we need to skip the check when we trigger our rollback. There's no way those checks are going to be green, so.

F

Yeah, so we need to bake that into the deploy or the rollback command in chat ops. So I'd like to see that done as well, so.

C

I can open an issue for this one yep.

F

Yeah, so that gives us at least three action items that we should accomplish prior to our next test.

A

Yes, that sense.

A

Cool so um alessia, can I let you do something with the issue 1699 and work out like what is feasible or we want to do there like if we split it up or what we can do and then.

E

Let's go back, are you.

A

Okay, to do a bit of a documentation, update and see what you want to put in there, based on your experience.

F

Yeah, I could certainly do that.

J

A

Stuff excellent. Thank you very much. Everyone, um a successful, a successful test, huge milestone.

C

F

With that, can we before we end the call, can we just stop the recording sure.

B

I think that I have.

F