GitLab Delivery Team, 13 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-01-13 AMA about GitLab releases

Description

AMA with the Delivery team discussing GitLab releases and deployments

A

Hey everyone I'll just give a few more, maybe a minute or so, and just uh see if there's any more people joining in but good to see. You.

A

A

Fantastic right I get started so uh welcome uh to this ama with the delivery team. um For those of you who don't know us, um I'm amy, I'm the engineer manager for the team and we've also got alessio and myra on the call um who are in the team as well and uh delivery are responsible for releasing all features and bug fixes out to get lab.com and also to our self-managed users.

A

So this means day-to-day we're coordinating on releases, but then we're also building up the actual scalable release tools that we use as well and then, alongside this, we have a big project where we are migrating gitlab.com over to kubernetes.

A

So uh this will be um helpful in improving site reliability, but it also gives us kind of additional options on deployments as well.

A

So at the top of the agenda item uh just highlighted here, we've got an issue um which you can kind of review, which is our 2020 in numbers. So it looks back over all the things that changed in the last 12 months. um Quite a lot actually.

A

um So hopefully that gives you a little bit of an idea of the sort of things we're working on a bit of a direction for us next year as well. Well, this year, sorry as well!

A

So as we go through, if people could help out uh writing notes, that would be very helpful. Does anybody have a question they'd like to start us off with.

A

Or is there something um like, maybe what's the um perhaps, what's the like least clear thing about this team, that would be uh helpful. We can maybe try and uh clear up a little bit.

A

So how about I tell you a little bit about this uh 2020 numbers. um So last year we made some big changes to uh the way we were doing daily deployments, and you can really see that reflected in the mttp. So early in the year we were around 100 hours of mttp. So that means that code that is effectively ready to be used by users um is taking 100 hours to actually reach those users and through the work the team did over the last year. We've got that right down.

A

We now have a target of 12 hours, um so that's quite a big big difference, and that means like I say not only. That features are reaching users much earlier, but also it means that if we have bug fixes and things like that, they can also go out much more quickly.

A

So that was quite a big defining change, and that was one that also then led us to our next big significant project, which um was around removing the blocking nature of security releases. So security releases are super important and have a very special uh process to to make sure we don't release anything before we should, but by their nature it meant that they also impacted on our daily gitlab.com deployments much more than we wanted them to.

A

So we had a big project uh running for quite a lot of months, maybe the whole of last year, actually um but quite a lot of last year to remove those- and that was the other big step change for us on mttp.

A

So those were two quite uh exciting changes and as we go through this year, the um big exciting changes will be around um progress through the kubernetes migration, so we're seeing the further through the migration we go, the more um so the faster the deployments to gitlab.com are running. It's just it's faster to deploy to kubernetes, so we'll see continued improvements there and then also the next big goal for us on our mttp uh journey is enabling rollbacks. So that would be a big goal that we'll be working towards as well. Throughout this year,.

A

Is there anything else I can helpfully uh explain or answer or give any more sort of color around? That would be interesting for people.

B

So I do have a question just to keep the conversation going. What do you think the delivery team could do this year that could completely exceed your expectations.

A

That's a great question: he wants to go.

B

B

um So to answer my own question, I think it could be deploying directly from master. So as soon as I merge a merge request, it would be deployed to master so one hour. Mttp sounds great.

B

Having security releases on demand not no longer need to wait at the end of the month to actually schedule a security release could be. I like to dream.

A

Yeah, both great suggestions, um yeah, I think mine- is um sort of a linkedin into those as well like. um I think it's going to be the auto rollbacks, because I think once we have that uh safety net, then it opens up quite a lot of opportunities for increasing the deployment window.

A

um I kind of imagine in my mind as a as a time where um I can come online and actually the release tools tell me like there was a deploy and the new stuff happened. It went out. There was a problem detected before any users saw it and it rolled back to the safe thing. So you kind of come online and immediately have the here's the task I need to pick up, because um I mean it just means that not only will deployments be able to go through more frequently.

A

But you know it's uh it's much closer to the kind of fully automated approach. So I think for me, that would be the really exciting.

A

One, what about anyone else like is anyone? Are there any um like changes that would be helpful for uh engineers or um like design or anyone else who's on the call like? What's uh what are the kind of um sort of dream items for.

A

A

Cool. Okay- that's great hello! I think uh I suppose we also, interestingly, are going to um expect. I think that mttp stays around 12 hours for some time. um We have it in our on our handbook page that the next time we focus on um changing the target time will be once we have three months continuous achievement of the 12 hour goal, whilst um also having roll backs in place.

A

Is there is that a general view like that 12 hours is a reasonable like a reasonably suitable time for a new, say, new feature or a new bug fix, or you know, a new design to actually reach users.

A

A

Cool darwin, nice. Thank you for your question. Comment. Go for.

C

It, I'm sorry, did you want me to verbalize it yeah.

A

Yeah go for it yeah.

C

Yeah, just uh looking at to see if there'd be a way to get amazon linux, to support on um our main install page and then possibly we could do the um amazon ami on amazon, linux 2., it's technically centos binary compatible sent us 7. So I think we have that on the page. So I'm not sure if we actually um prepare that or not um or do we do, we do we test on the centos 7. I guess that's. I don't know the answer to that.

A

um I think distribution do uh we do release it. um I believe, um but uh distribution actually provide us with these and uh we deploy to them, so we may have to defer to them, but we can certainly follow up for you on that one um and find out what, if there are plans uh to do that as well.

C

Cool yeah we're getting more and more demand um from the amazon community and amazon continues to improve a lot of optimizations in in their version of uh centos. Seven.

A

Okay, yeah, that's interesting, well, yeah, I'll uh I'll, ask over in distribution and see if we can get you an answer for that.

A

Cool um excellent: are there any other questions.

D

uh Yeah, I actually have a question uh sorry about the background noise. uh It's my time here um so with frequent deployments to production, how is delivery ensuring that production stays stable because the more frequent we deploy, the more we'll probably be, maybe requiring rollbacks or we're seeing issues that we would see less frequently for deploying this less frequently.

A

Yeah someone you.

E

Want me to take this yeah sure, yeah, so great question thanks for asking, so there is um first of all, there's also the cons. We also have to consider that when we release um with more frequents like like we're doing now, we release less changes, so it's easier to test them to detect, identify a problem and eventually roll back or roll forward, just removing that. So what is actually what we are actually doing to prevent this is is this so we added new metrics, so we have this.

E

We call them deployment metrics, which are based on error rate and um the our slo, so the abducts in our application, so, instead of using the same um accuracy that we are using for paging um on calls, we we are our metrics are more sensible sensitive. So what happens is that we deploy to to staging first, then we run a qa on it and when qa completes we start rolling out to canary on canary, we run qa again and we keep checking those numbers.

E

So after that point in time, we have a baking time of one hour where we keep monitoring those numbers. And if everything is fine and there are no active incident, then a risk manager promotes the build to production and yeah. I can tell you that it's actually working well and having smaller change set gives us more confidence in promoting things, and we we ended up having less incident, because there's also a kind of psychological side effect here.

E

Is that now that engineers know that if they merge something now, it could be online in production in a matter of hours, because we are speaking of hours, half a day worst case half a day because now, in other words, cases average half a day, you are, you tend to do more iterative work, because you know that if something doesn't get shipped today, it gets ships tomorrow and so you're, taking less chances and less risks, and the end result is that yeah deployments are more stable and we don't have big problems with production.

E

Does it answer your question.

D

Yes, thank you that great answer, especially the list changes part. I didn't think about that.

D

A

Cool are there any other questions.

D

I have one more question: um does the delivery team um ensure that um production is um identical to canary and uh staging and development and the dev environment? Because if these environments are not identical, then the qa would pass and in one of the environments but then wouldn't pass on the other one? So who is who is doing that delivery? Or um what's the deal with the difference between the different environments.

A

Yeah, that's a fantastic question, so um infrastructure as a whole owns this really um as we're all kind of responsible. On the whole, I mean they're, never fully identical. I suppose one of the biggest changes is as we as we deploy new services or you know, continue through the kubernetes migration. They begin on the staging environments and we sort of gradually roll them out through the canary environment through to production. So those points, then the environments are quite different.

A

So I suppose it's us knowing what the differences are and kind of making changes to the environments carefully. But um it's also, I suppose, a key part of when we have when we do have incidents or near-misses, where we actually kind of consider was there something here.

A

So I think um probably the biggest change is not just infrastructure related, but we've over the last few months there have been quite a lot of improvements to the data that exists in the databases on say, staging um to enable us database uh migrations to be tested more effectively, and things like that will often be quite like corrective actions that come out of incidents so that we can actually bring these things closer together.

A

Cool, thank you cool. Thank you for asking um ian. Do you want to verbalize.

F

Sure you just mentioned in the goals for 2021 option three was to change the direction towards shipping git lab features instead of the manual tooling. uh That's kind of interesting, so I'll just provide some more details on what the the goal is with that one.

A

Yes, thank you for asking. um So yes, um through necessity, um at the moment um we run most of our deployments and uh release tooling, is custom, built and sits outside of the actual gitlab product.

A

So now we have a lot more features available to us in the product that we can be starting to move over to and also we've simplified, a lot of the uh deployment and release process. So actually it's a little bit more standardized than it used to be so the goal is the end goal is to have our deployments entirely running from within the gitlab product.

A

There are a few things that are very unique to gitlab, so we'll like be moving towards those incrementally, particularly the way we handle security releases, but for this year, what we're starting to do what we started last year, we're continuing through this year is identifying features that exist already that we can be moving over to so one example is um last year we did a big piece of work to rewrite our release code to use the api um which got us a bit closer and we're also currently doing work to change the way we generate change logs.

A

So that it sits more closely within the product as well, so we're looking through this year to actually identify additional features that either exist, or you know we could be close to getting so we're working very closely with releases at the moment on. How could we move over to the deploy freeze feature or how could we um adopt the rollback feature that already exists and how we can, actually, you know, take a bigger part in those things as well.

F

Thank you. That sounds pretty interesting.

A

Cool thanks for asking.

A

Are there any other.

A

A

No, if not well, thank you so much for everyone for coming along and uh for asking questions and yeah we'll look forward to seeing you all in a bunch of time, cool thanks for everyone. Bye.