GitLab Infrastructure Group, 24 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Firedrill determine deployment version

Description

There was a recent problem with background job processing that we think may be due to a deployment. Help figure out exactly what version is running and the changes since the last deploy.

A

Hello, everyone we're going to start the kubernetes service fire drill scenario number five. This is a scenario about determining what version of the software is running in the cluster and the changes since the last deployed version. The scenario reads like this I'll go ahead and share my screen.

A

There was a recent problem with background job processing that we think may be due to a deployment, help figure out exactly what version is running and the changes since the last deploy joining us are henry and miquel.

A

It's a light crew today. So um I'm sorry to put you guys on the spot, but do either of you have any suggestions about where to look to find out what version is running on.

A

Well, I guess um because it's background job processing, that's sidekick. So what version of sidekick is running in the kubernetes cluster.

B

What I would do is, I would um use cube ctl to get the version of the actual docker.

C

Image that's running.

B

In production and then work my way from there so try to figure out how the image was built, which deployment in which deployment the image was generated and then find the relevant gitlab match, requests.

C

And then we are done yay.

A

Sounds okay but it sounds. It sounds pretty complicated to me like first you would. um I guess you would first have to establish a tunnel to the cluster, so you would have to be set up for that right.

B

Well, it's these.

A

B

Yeah so well, so, first of all, I would like I've already done that in the past and we've got a landlord for it and we suggest to do it before people go on call so um and and the way I would do it is, I would rather than establish a channel, because I think that correct me if I'm wrong, but I think that we're not supposed to establish channels to production.

B

I would uh ssh to the console and um I've got um cube ctl config there to to access different production clusters, so I would ssh to the console um use.

C

B

To get the the exact version of the image.

C

I have one question there because I think I looked into the documentation for settling up connections via cubecontrol and I think we have at least three different methods now, so one would be to go to the consoler server and have your setup there right. Then people are suggesting s shuttle and then there's this other thing to use, uh set up an sh tunnel and use aj proxy, the http proxy, with a websockets connection, socks5 connection and and method to do.

A

That, as far as I know, um I know that devin has been working on these uh dedicated developer, uh console machines, and my previous thought was that we're waiting until that work is complete and then once that work is complete, we'll all have we'll all be connecting the same way to um the cube api could be wrong.

A

I think that's, I think. That's like the medium term plan for.

C

Now you're right, I think, like I use that okay.

A

Yeah, I think uh I think one of the problems with the shared console right is that uh it's a shared vm that multiple sres are on so there's potential security concerns around that. um I'm not saying tunneling is much better, but I think you're right that many different people use different methods, but let's say um for, for whatever reason we can't use cube. Ctl uh does anyone know other ways we can determine without using cube ctl? What version is running in the cluster.

C

I mean what what I would try to do. First always is look into the announcement select channel to see what is supposed to have been deployed um and which environment and then for figuring out versions. I would try to look into grafana first there, I'm not very sure if we have this information right there, so I would probably go to the graphana overview thing for sidekick and see if we get some version information there, but I'm not sure if you have that there I mean that would be the way I would probably go in in.

B

C

First, because I wouldn't be sure how to do the cube control way that mikhail mentioned right now out of the box.

A

Sure um so the first thing I'd like to highlight is the new event log that we have. um We have a link. Can you guys still see my screen or not? Sometimes it doesn't work? Can you see my screen yeah.

B

Yes, I see it cool.

A

So there's this link on every incident to recent production of deployment configuration events. If you click this.

A

This will show you um for all of like the deploy events, and you can see like here's an example of a gates deployment here. If we, uh it tells you the version, so this is like the image tag, so this would be one way you could.

A

A

This in the agenda.

A

A

I think I think we can improve, obviously like the best way would be just to go to grafana to see um you have annotations, but they're not really easy to look at. um If you want to look at sidekick specifically and just let me know if my screen is still sharing, sometimes.

A

I'm amazed, okay, um we haven't incorporated these into the general service dashboards, but we have pod info dashboards and maybe you've seen this before. So if you go to sidekick pod info.

A

You wait yeah, I think, there's like a really busy graph here on usage, which is why it takes takes a long time um as soon as this loads uh yeah this this, like.

A

This is this graph is not meant to have so many pods like it's really too busy it's having trouble, but we can just look at the active version and then we'll we'll turn off all these like annotations, and then we can go back like two days.

A

So here is like, if you just want to look at a very specific deployment and want to see what versions are running on the cluster. This is where we have it graphed in grafana. You can also look at thanos to get this as well. I'll, put a link to.

A

C

I just noticed when going to the um sidekick overview dashboard, you have links to cuban emitters, details like containers, detail and deployment detail which.

A

C

But it doesn't show the version right, I think yeah. So if you would have a link there also for the pod info, that would be a good improvement. I think so that you have one place from a service to get to this kind of information right, because I need to remember that I that there's a pod info dashboard that I could look into.

A

Yeah, the idea here is that the pod info dashboard came first and I think we're planning to deprecate it. What we probably probably better to do is just to add the version info to the deployment dashboard since it's useful, so maybe.

C

A

B

I really like the the grafana approach and I'm curious about the events log in elastic. Like I it's fairly new and I haven't used it a lot and I'm curious like how does it expose things like failed deployments, or how do we see if a deployment actually contained a change to the sidekick.

A

A

No, it doesn't, uh it doesn't track failed uh deployments, nor does it. Nor can you be absolutely certain that the code so yeah, maybe event log is not the best tool to use. I think um we have to decide whether we want to ex like I could. Probably we could probably.

A

This is hard to do in ci, but we could probably figure out a way to.

A

Capture the return code of the helm, command and then post that to the as an event and then fail the job or something like that.

B

I I think that the event look is still useful, just to get a high level overview of.

A

B

Changes like I'm not dismissing its value, I'm just like trying to you know, learn it to get the maximum amount of body out of it.

A

Yeah, we should probably make sure that's clear. The docs, for this is a link to them.

B

And what about figuring out what changes were included in the deployment? Would I have to go to um to each individual deployment to figure that out was that already exposed in the event log.

A

um It's it was thought that we would um expose that eventually, in the event, log.

A

So yeah, I guess that's part of this question: does anyone have ideas on how we would get a list of changes or what? What would you do now to find a list of changes for, given that you have like an image.

B

I would I would take the the version, the the hash from the from the image and look at the announcement channel um to find the actual deployment or, in case of the event log. I would go to.

B

There so that would get me to the to the pipeline um and work my way from there. I think, there's a link in the description that links to all the changes in the description of the mr or the pipeline generated by the deployer. I think.

A

um So, unfortunately, it's not that straightforward, because what you're seeing in the pipeline you're seeing like the changes to the deployer project, not the changes to the gitlab or gitlab project um henry. Do you have an idea of how you would see the list of changes.

C

um Not right now, that's exactly the thing that I can't remember always sure.

A

Yeah, it's not it's not straightforward. I think um what I would do personally would be. um I would I would go to this grafana dashboard and I would take a look at the the sha and then I would just do a diff of the gitlab or gitlab project. So I would do like a compare um if you, uh you may see like these links in the auto deploy status like something like this.

C

Didn't you also have something in the announcement or the um uh other deployment um delivery channels that we are using? Wasn't there somewhere, I linked to to all changes in a release. I think there was something like that and.

A

Yeah, so that's right so in whenever we do um an auto deploy status, which happens in the upcoming release channel. Quite often uh we generate a summary of what's deployed to each environment and then there are diffs between the environments.

A

So maybe that's what you're thinking of um yeah.

C

Right that was the one I was looking for. Yeah.

A

So so, let's say: let's use this as an example, let's say we're going from this blue deployment to this green blue green deployment. No, it's not, but um the colors is like from blue to green. So.

A

A

um Whoops reversed it.

A

So we're going from image to this image.

A

What we should be able to do is just copy the shot.

A

Space in there.

C

A

A

B

A

Why this is not.

A

C

One question between mention that you would use queue control to look at the deployment versions right. So what command would that be exactly.

B

I guess it's it's sidekick, so I think it runs in both the regional and zonal clusters. If I remember correctly so the first thing that I would do is, I would figure out if psychic is actually running in the cluster that I'm connecting to so it would probably be cube. Ctl um select the github namespace, so keep ctrl and gitlab and then git deployments, and that will give me a list of full deployments in the gitlab namespace. So then I would find the sidekick one and then do cdl dash and gitlab get deployment.

B

The name of the cyclic deployment dash o uh yaml, and then I would probably just because it's it's it's easier to then do some automation around it like like on jq, etc, um yeah and or describe deployment to to see. If there's, perhaps an ongoing rolling update.

C

A

So, um going back to the example, if we're going from this image to this image, um here's the compare link.

A

um What does anyone notice like something a little bit off about the compare link you might like see this security? Could you think of why we have to use the security.

A

C

Because it was a security release.

A

um Yeah exactly it could be a security release. I guess is the is the answer, because um we always create the auto. We create the auto deploy branches in the security group and um you're guaranteed to have the changes in the security mirror. um There are three mirrors of the gitlab project. You have there's three projects for gitlab, there's the canonical, which is uh gitlab or gitlab, which gets mirrored to security.

A

Security gitlab and then that gets mirrored to where we do the package. So um I guess you could say the security. The gitlab project under the security group is where you typically want to look for revisions, because it's guaranteed to be there, it'll, probably be in just regular gitlab or execute gitlab as well, but not always depending on whether, if it's, for example, if it's a pick into an auto deploy, because we only have the auto deploy branches on the security group, then it's only going to be there. So that's that's the place.

A

You always want to look. um I definitely think this could be improved, because I think, if you um there's there's all sorts of uh like opportunities like to mess this up like you. If you didn't know that you might be looking at the wrong project and you might not see the shaw. So if we look at that, compare link like these are the changes that went into the sidekick, deploy from from this blue version to the screen version.

A

Any questions about this.

B

Is is there any other way of getting those changes.

B

um That you can think of like it's fine. If there isn't I'm just like curious, you know what other options we've got.

A

As far as uh yeah, the other options would be if you're, if you're, promoting. If it was a recent.

A

um I would say: there's no better way right now, I was going to say like if it was a recent promotion from staging to production.

A

You could look in the back scroll and see if anyone did an auto deploy status, which would then tell you like, which would just give you the differences between the stages in the production environment before it was promoted, but I think, by the time it's promoted, you really are just going to have to construct this link yourself, so you need to first determine the version that was running before the current version, and then you have to remember like okay. What project do I need to look at and, like you have to construct this compare link?

A

That's a lot to remember at 2am in the morning, in mikkel's time, on a saturday.

C

But but couldn't we couldn't we not run uh how to deploy a status command right now to figure this out.

A

You could, but it wouldn't show you um it wouldn't give you the right to compare link, because it would show you a link. The difference between staging and production staging is like the newest version right like the or the what is about to be promoted, not the old version, so.

C

Yeah that doesn't really help an enhancement to this command, like um comparing the last two production deployments, or something like that. I mean if that is the way where you, where we have automation, could construct something like that to make it easier already then yeah, maybe because I I already can see that that people who are not always doing deployments, will have a hard time to remember this information. I think.

C

So maybe that would be a nice issue to create for the delivery team to get this into this um uh chat command with some, I don't know enhancement, maybe to compare the.

B

Environment, I guess maybe even um a link in the event log to that is your this. This exact link to the security project, with uh with uh with the different shots, would be extremely helpful.

A

Yeah, the the the tricky part of this is like knowing what version you're coming from and what version you're coming to we're going to. uh We would need to add some extra logic to before we do the upgrade. We would need to inspect what version is currently running and then, um but then.

C

Then the announcements channel would be better even because there we see for each deployment right the information that we started.

B

C

Pipeline and everything so there we have the information from where we are coming and where we are going to right. So I guess there we could get this internal information into then and then you would see it without needing to run a command to get this information. Maybe.

A

Yeah, um I think we would need to. We need to figure out when, like maybe.

A

Maybe we could do it on the leader of the first node in every fleet.

A

um Yeah, I don't know, I don't know whether it's better to do this in announcements. I think the problem with using announcements is that we would end up spamming the channel because you have you have a version upgrade for every fleet api web um and then git, kubernetes, sidekick, kubernetes, uh gitlab, shell, but they're all the same version change.

A

So we could either just pick one say like okay, we pick web or we just pick the kubernetes cluster and then generate a link from that or we spam the event or we add messages to the event log.

A

I think it could go either way on that. um Maybe the event log starts to be our ssot for for this sort of thing, instead of the announcements channel, because I think that is a little bit easier. One thing I like about the event log is that the times are utc, whereas in slack I don't. I don't think it's possible to change your times to utc.

C

Good point: yeah.

A

But um I think I think I like the idea of having to the auto deploy status.

A

It's a bit tricky, I'm not sure how we would do that like we could source the version data from prometheus, maybe to kind of see what was the previous version and just generate a link from that.

A

C

And the events that we see in kilafana do we also see the um from there to where we are going like version two, no.

A

C

A

We don't no, you mean the annotations annotations, yeah yeah. No, you don't.

C

Okay, that's that's a nice issue to govern. I guess yeah.

A

Cool are there any other questions or comments about this.

B

This has been really useful.

A

Glad I'm glad you found it useful and hopefully some improvements will come out of it.

A

Okay thanks, everyone.