GitLab Delivery Team, 12 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-05-12 AMA about GitLab releases

Description

AMA with the Delivery Team

A

Okay, I'll get started so hello. Everyone welcome um to the may um delivery teams, ama um so feel free, please, to like add any um questions you have on the agenda um and we'll also have time at the end. If you want to just verbalize so uh nick you have the first question.

B

Yeah um we're so we're working on a lightweight training for development engineers, to uh increase awareness of version compatibility issues and and how they affect upgrades for comm and self-hosted.

B

um So yeah, aside from our our docs on multi-version compatibility and zero downtime upgrades which I've linked, uh would you recommend any any other resources um for this training or are there any gaps in the existing documentation?

B

uh That you'd like to see addressed.

C

I can take this one amy, so I just started writing an answer in the agenda. So we recently created an end book page, which is name, is coding at scale. There's a link in the agenda right now. It's just a description of the problems and a collection of resources. So there are more resources on top of the one that you linked. We have something about user, uploads, sidekick, multiversion compatibility and migration.

C

Now uh this is coding at scale, so it covers both delivery and scalability type. But I think these are both important things to create awareness around.

B

Yeah, this is great thanks for for pointing to this and yeah. Maybe it's as simple as pointing people to this page and saying and saying read: read all of the resources collected um as a single source of truth on the subject. So thank you, yeah, but oh.

C

There's also a follow up on this, which is, I wrote some of those pages, not all of them, but I think we, we are lacking the kind of a feedback loop from developer in this area because oftentimes, we are aware of this problem. Some of them are very high level or eye complexity. We try to write something, but then usually only engineers that ends up helping in an incident related to this one, read it and then start to understanding it.

C

So if something isn't clear, just open a merge request bring us let's discuss this, because maybe some of the content is outdated or is not as easy as we thought to understand. So, just if you can help us to get this documentation better.

A

And I just want to add in there I'm really happy to see this um so yeah thanks so much nick for creating the issue and uh pushing this forwards. One thing we um have been talking about quite a lot around rollbacks is: is this sort of um version compatibility and how it all fits in with post-deployment migration?

A

So it's such a relevant topic for us right now, um and actually I think, if there's, if there's things we can review in the training and uh documentation as well like we'll, certainly read through these things, but if there's anything we would be useful for us to add in. um Let us know we can we can help out there.

A

Do you want to verbalize.

D

Yeah just saying, I think this is great um one problem I do find working with self-managed customers is their upgrade cadence and the way they run, gitlab, the environments or setups are actually quite different than what we use for sas or test environments. So I think this is super helpful and thank you so much for creating.

A

This awesome so on to the next question: uh pilar.

E

You have this wow, um so we are looking to stand up a monthly user group virtually and given that we have a wonderful monthly release, we were thinking. We absolutely want to make sure it's part of the content that we're talking about. um So I was looking to see if there was a good place that I should be keeping an eye on as we build out the monthly agendas. I saw a blog post from march. What I'm looking for is something that speaks to the value, so we've got this new monthly release.

E

So what does that really mean to you, mr user? um Mr maybe potential prospects, upsell customer um to help them understand what it really entails, and then I did find the site that looks like it updates monthly on the 22nd. So I was just looking to see if there are any other resources that maybe I should be keeping an eye on, and I see that nick dropped in a very handy tool. So thank you for that.

B

Yeah, I'm not sure if it addresses the um the the part of your question about speaking to the value but uh yeah. I just wanted to link link that for anyone who wasn't aware of it.

E

Yeah, that's very cool. I I hadn't seen that before, but do we normally do monthly blogs around the releases? Okay, cool, so I'll keep an eye out on the blog post as well. That's.

A

Gonna say yes, and there is, uh there should be a section in there as well about what's new um and sharing that, but um yeah the blog post is the main way we we generally just uh enable these things. So we, the product team, um create the blog post um yeah, create the release post uh each month, um and then we just publish that on release day.

A

E

Sound sounds great, um any other training resources or get lab team members, you think, would be interested in you know this would also be a platform for anybody who wants to talk about the the monthly um updates.

C

So maybe something worth mentioning is that um the next milestone will be 14.0, so measure release breaking changes, and things like that. So we are uh putting some effort in communicating up front breaking changes or new yeah breaking changes mostly so because things are gonna change uh daily for our sas user and just on the 22nd with the new package for the on-premise installation. So we I think you should you could reach out to orit because she's trying to organize communication around this, and maybe it's working the more we can amplify the message.

C

The better is because some of these changes may require user to actively change something in their workflow. So we want them to be prepared as soon as possible, because yeah one day we break and we want them to know as soon as possible.

E

That's great, oh, and thank you amy for putting in the correct I'll.

A

Put it in here there's also an issue I'll add in I'll find an link hidden um afterwards for you uh that I could ping uh but yeah just on a kind of a little bit of of why the breaking changes will be coming uh through throughout the month is um the breaking changes will all be targeting the 14.0 release on the 22nd of june, but as they go into gitlab.com on our daily deployments, those are as soon as we can make them, so the changes drop in on gitlab.com around our deployment cadence um and then make it out into self-managed on the 22nd, but yeah eric is leading the way on coordinating this stuff.

A

So I'll I'll link you on the issue, so you have all that stuff.

E

Sounds great, thank you. So much really appreciate it. You're welcome.

A

um Stan over ta.

F

I had two questions. First, one was, I noticed the rollback happened today. How did that? Go any surprises, anything that you want to share here.

C

I can take this one as well, so surprise, yes, a big one but yeah. Let's start with things that so the deployment the rollback went. Well, we had some quickness with our tooling at the beginning to yeah.

A

C

Basically, we were always running dry runs in production and then today we're running a real one and the checks were more restrictive than we expected. So we had troubles in starting the process other than that. The real surprise was that if we we removed cannery from the fleet to just have a simulation closer as possible as a real incident and production wasn't able to handle the load of the deployment without cannery so likely we are running under provision it. So this is the big finding.

C

So we because there was no real incident, we put cannery back online so that we could handle the deployment but yeah. This was the big finding.

F

That's really interesting is that majority of that traffic and canary is still us, or is it actually like a certain percentage of the of the whole gitlab.com? Now.

A

It's it's both yeah, I mean, I think a lot of it is us, um but yeah. There are also some people who have opted in on canary.

A

uh So there is some traffic, but I would assume the majority of it is of the load, at least is internal yeah, um so yeah overall, like the rollback, went really successfully. We as elections, said we we had some. It was interesting at the start. We had to set this one up as a kind of test case, um so what that meant was because, in a real in a real world, uh we'll be rolling back.

A

uh When we have an incident, so a deployment will have completed, we have discovered something uh has gone wrong and then we would initiate a rollback uh to the previous uh package. In this case we didn't have an incident. It was a test scenario, so what that meant? Was we cancelled the previous deployment to allow us to roll back? um So it's a little bit of a different one which made it start a little bit trickier, which was the diploma, was technically still ongoing because it hadn't completed.

A

So we've got some improvements there on documentation to make that a bit easier and then yeah. We will look at how we can also increase capacity across the fleet so that we can actually run these things at the same time, but it was great great test and the rollback was successful and it did what we expected it to do.

A

uh We ran qa tests afterwards and they all passed so overall, we're really happy and we'll be making some changes and then looking to run another test before um next step will be unleash on incidents so we'll be starting to check out incidents um rollback's worth noting. We can't roll back every type of incident.

A

So there's a couple of constraints: we only roll back to a previous package, so it does rely on us uh detecting a problem before we do a follow-up deployment um and then that gives us an opportunity to roll back and we also can't roll back post deployment migrations. So those are the things that we are we'll be sort of starting to monitor and work out like. How can we? When can we roll back and also? How can we have more opportunities to roll back.

F

Great thanks for that summary. That's it's always good to do those kind of tests before you actually have to like. Do it right, because then you're surprised.

A

Hey, we don't have.

F

We don't have servers to us support gitlab.com without canary. So thanks for that.

A

That's right absolutely yeah um and liz. Do you like to verbalize.

G

I yeah oh uh yeah. I was just curious if you had a retrospective or if there was something that would be created, that we could look and review async.

A

That's a really good question. We haven't got one um at the moment um we have got. uh We have got our agenda document, which has quite thorough notes uh where we sort of debriefed in uh so we probably won't create a separate ah thanks henry uh yes, we probably won't create a separate one. We kind of discussed in real time and put in some sort of follow-up actions, but please feel free to add comments uh in that document as well. If, if you have any other questions, perfect awesome, thank you thank you uh and stan.

A

You have next one.

F

Yeah, so in our staff meeting we talked about how to improve the time to production, and I know there's a lot of epics and discussions around that, and I I noticed that epic I linked it said 86 percent of the time it's just waiting for that thing to be picked into the branch, but I think there are cases when the merchant quiz has been picked and now we're just sitting and waiting for a deployment, and sometimes it takes 24 hours or so, and this seems like a really long time for people waiting for like a critical fix to go in.

F

So I'm wondering if we're sort of measuring thinking about that- and you know not thinking about the aggregate but like those kind of situations where we need to speed it up. What can we do to kind of improve those things.

A

Yeah, absolutely, and certainly certainly, we track so we track on the mean time to production, um and uh at the moment, oh, we haven't, got metrics they're running a bit delayed. I think at the moment we are our target's 24 hours.

A

I think we're a little bit below that at the moment we raised it up um in at the end of march, so there are two things that really impact this, and one of them is incidents, so it if we have incidents, we often are blocked on deployments, which is why we've raised up the target at the moment to 24 hours, um we're seeing uh like over the last, as we all know, like first few months of the year like increased number of incidents, and that just has a big knock-on effect to deployments um and then the other one we see particularly around when we're sort of maybe responding to incidents is actually it's.

A

It seems to be very much around the kind of uh approvals on anyone feel free to dive in, but my perception seems very much around like actual the time to get something approved and merged and run through those pipelines. Before we even begin the deployment process.

C

Yeah something more more details on this, so, basically from. If there's no incident, you can think that your change will be in github.com in less than 24 hours. 12 hours is a good average, but as soon as something blocks us, it could be whatever uh delayed post deployment. My positive immigration taking too much or any type of incident or near miss incident, because we block deployment also, if something starts going wrong, then basically to get out of this.

C

It takes around eight hour, which is a lot because, um basically everything is blocked, so we need to identify the problem fix it. Then we have a pipeline running because it's a merge request. It has to be reviewed in a complete pipeline, so it takes roughly from one to two hours.

C

On top of that, he has to be picked on how to deploy. There is another pipeline running roughly 40-50 minutes tagging and then regular reload, so from tagging to deploying to gitlab.com is roughly six hours.

C

So you add everything before that pipelines review, and this explains you why? If something goes wrong, basically, you have a at least eight hour gap.

A

There's a couple of things that are taking place to improve on this, so quality are doing a super job of like working through ways to test run faster and uh reducing flaky tests and um having it so that we can actually like well, we we, as get up, can actually run the relevant tests against a change before we uh merge things in. um So those are all super helpful.

A

We also um just recently had graham joining the team he's based in apac, so over the next few months, we'll be training graham up into release management duties and that expands our window, we'll suddenly be uh available to deploy potentially every like all the all hours. At the moment, we're limited to deployments in emea and america's, which is another reason if you merge sort of uh you know, like eight utc you're, probably going to have to wait 12 or 14 hours to our next deployment window.

A

So we'll see these numbers start to come down, um but yeah. There's, there's, certainly lots of um lots of improvements. We would love to make here.

F

Yeah the apac deploy time window would be huge. I think that would be amazing.

A

Yeah exactly so that will that will have a massive impact. Yeah do.

F

We have anyone planning on joining from aipac on your team. I think grandma's in mia still right so.

A

We have graham in uh in australia, so it's just him, so we won't have a full uh cover because uh he's also on call, so we will have a sort of skeleton cover over there, but um but yeah as we as we sort of look to um sort of expand things over the next year or so I'm sure we'll we'll look to expand this out.

A

It's also very helpful for incidents as well to have a release manager online. Who can help with these things.

A

Fantastic so um greg. Let's get down to yours on number, seven.

D

All right, thank you, um so just raising awareness, nothing really actionable, but support is using. We have an epic setup to prepare for 14.0 deprecations braking changes, upgrade problems and I just think it's a great opportunity to collaborate. We'll get customers creating tickets, we'll surface issues that affect the customers, and then we can all work together, find solutions, document, workarounds or mitigations.

D

Basically, yeah don't be a stranger. If you have any questions about edge cases or how things might affect self-managed customers feel free to reach out and if you have anything to add in that epic um see a feature deprecation there. Yeah, please, please add to those issues.

A

That's great thanks. So much for sharing.

A

Fantastic has anybody else, uh got any questions, any follow-up comments or anything else, I'd like to bring.

A

Up no okay, in that case, thank you so much uh once again, like really appreciate questions and discussions and all the links and things everyone shared. So thanks so much for uh joining us today um and I hope you'll have a great rest of your day.

A

Okay, bye thank.

A