GitLab AMA & Q&A's, 9 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-12-09 AMA about GitLab releases

Description

AMA with the Delivery team about how we deploy GitLab.com and release GitLab to self-managed users.

A

So many ding dongs.

B

Is that a statement about the sound of the people on the call.

A

I would only refer to myself as a dingo. I would suggest that about any of my team members who are excellent.

B

Absolutely absolutely.

C

Fantastic um thanks so much for joining. um So this is the uh ask me anything about get lab releases and deployments with the delivery team, so uh I'm amy, phillips, uh engineer manager for delivery. We're joined by quite a few people from delivery as well. To answer your questions to make this slightly better, uh trying to explain that this is as delivery team. We are focusing on deploying and releasing gitlab.

C

um So all the code to get up, uh we are moving through to production, so we will run through some of these questions. We may not be the experts, because I think some of these may cover the uh features of deployments and releases for github users, but we shall see if we have someone on the call who may be able to help out so um john. Do you want to verbalize your question? Let's see if we have someone who can help you out.

B

Yeah, so so you're right, uh I did have a bit of a misunderstanding as to what the meeting entailed. So um my questions are more around the features of get loud releases and deployments. So um forgive me if uh yeah these questions aren't necessarily appropriate, but I'm actually to start with jamie's question down to number three um just to confirm how we count deployments.

B

We use usage paying a lot in customer success and deployments is a great metric that we like to show customers and how they use it and we're just not sure what ticks the box or increments that deployment counter. Is it the presence of an environment and a job definition.

D

I think I can help here. Okay, so great.

C

D

Okay, again, as hammy said, this is something that we are user as well, because it's something that developed by the release group, but we know we contributed some part of it. So we we may help answering these questions. So, yes, if there is an environment label in your ci job, this will be recorded as a as a as a deployment and we increase the number, but we also implemented api support for it.

D

So if you have special requirements, you can do api call to gitlab and record your own deployment, which means you can you have also more granularity. You can record the deployment started and then maybe it failed later on or it got completed, or you can just say I did this. It's a deployment is completed and you provide all the information.

D

Okay, that's cool.

B

Good um yeah, I do think probably my next. My first two questions are probably I don't say inappropriate for this group that aren't necessarily relevant to this group, so mine are around sort of just tracking deployments to production, metrics um and that kind of thing so uh alessio. I see you answered my second question around the environment feature being flexible enough for some of this stuff. So um I don't think this is the right venue for these questions, but thank you.

E

It's as a follow-on to that answer about the api. Is there a way using the api to decrement that count.

D

uh No, I'm quite sure the deployment is just you can add, but not remove.

F

E

C

And for your other questions, john we'll um make sure uh the releases stage group know about them so that you can get an actual answer on this because they will certainly have the answer for you.

B

Yes, thank you very much.

D

C

D

Want to discuss the first one amy actually is well the fact that we are tracking mttp as a feature as a key metrics for our team, and we are doing this in in periscope. It's definitely yeah. It shows a lack of the feature right, so it could be useful for us as well. Yes,.

C

Yeah, so I know it's in that roadmap yeah. Please go ahead.

G

uh I'm actually working on a frequency of deployment api for for metrics right now. It's a gitlab ultimate only, but it should be in this milestone, and that is for a project scope. So you could get.

H

G

Of deployments that have been deployed in that project over a period of time limited to a date range of a quarter just because of the performance of the query there can you.

B

Also specify the branch- no not yet absolutely yeah, because that's really why questions related to is we record deployments and deployments over time, um but those deployments could be just a review app in a dev branch. um I think most people want to know their deployment to reduction velocity, so measuring yeah, so being able to restrict that count to the master branch. I think, um which is where most people deploy to production from is probably a more important metric than overall deployments.

G

Since I'm yeah I'm actually working on that right now, it's in review. um I can talk to my product manager r8 and see if we can get that added, because I don't think that would be too difficult to add right now.

B

Cool that'd be awesome. Thank you.

C

Fantastic um thanks for that amy glad you're. Here I was hoping there'd be somebody from releases here, fantastic, uh so um dad. Would you like to go ahead.

A

Hey everybody, um so I'm just thinking about deterministically! uh Well, I said deterministically determining that's pretty bad. um I apologize um uh what was released in a particular deployment. uh It's something that's come up a while ago and I have not seen any updates here. So I apologize if I missed a really cool announcement here that uh we've been proceeding here, but um it's really this idea of, like figuring out what will be in what release so that we can be explicit about when something is available or not and to whom it becomes available.

A

Self-Managed, customers um and and and all of those other groups that may or may not be interested in in releases.

C

Okay, yeah over to you eric.

I

uh Yeah I was looking up the link to it, so I can add it to the doc, but uh there we go. We added in 13.6 the ability to filter.

A

I

Requests that are deployed to a current environment.

H

I

You can say staging production, etc. um The api does have functionality so that, given a deployment, you can get the merge requests. That itself is not visualized. So there's no page where you can go and click on the deployments and then see there requests you have to go to the um merge requests index page basically and then say give me all the merch question: production staging or whatever or merge requests that have been deployed in this time window, but it's all focused on which quests so not uh individual commits.

A

Yeah, that's that's sorry. I don't know. Sometimes I've been like reverse meeting myself recently where I'm unmuted and then I mute myself and I start talking, which is probably valuable for some people but um yeah. That's that's! That's good. I think what I'm thinking a little bit in terms of is the release post and how uh ems can determine something's, released and sort of you know really be able to be sure about it.

A

I mean so actually that works better for mrs than commits um uh well that, I believe, would work better with mrs than just having a commit uh being part of a deploy, so that's cool. Thank you very much.

C

Fantastic thanks for those questions.

C

Does anybody else? Have any questions they'd like us to answer.

A

I have another one. If no one else has questions, I didn't get a chance to write it down. That's why I apologize um what what's the uh the uh uh what's? The team really focus the delivery team. What's the next big exciting thing that the delivery team is working on to release.

C

That's a great question, so kind of the overarching big, exciting thing is always making deployments and releases faster and safer. um Well, I think we actually have all the dri so dris. Would you like to give a bit of an overview of what you're currently working through so uh alessio? Do you want to kick us off sure.

D

So there is this well one of the goal of the delivery team is automating ourselves out of our work. So basically, there are some parts of the deployment process there are still manual and we take rotation as release managers. Basically every two months, so one of the direction is trying to remove all the out the works.

D

It requires us to act manually, so automating rollbacks and completely automating deployment up to production and another thing we are working now in this regard is we are going to simplify some of our pipelines because they were built over time and they we accumulate too much complexity. So now we are focusing on moving all of our deployments back to a single pipeline in release tools, so that we can actually focus more also on dog footing, our own features, because, right now we have too many pipelines around in several projects in several instances.

D

So it's really hard to uh get to use our own features that are designed usually for a project level and not that much on multiple projects, so that that's for, for my side,.

H

I

Yeah, no I'm I'm working currently on uh automating change, but change lock generation uh within gitlab itself, because we have our custom tooling for it. I like so many projects out there. Everybody has their own thing so we're building that into kit lab which uh touches upon you know, parts and ghillie parts in gitlab.

I

Also now I want to release tooling with ultimately the idea being that you just you know, send an api call. You say hey, you know this is my new version. This is sort of the the range of commits that went into that version or will go on that version actually and then get that takes care of updating the changelog for you and all that stuff.

C

Awesome over to you, jeff.

H

um For uh for like the uh kubernetes immigration work, which is what uh myself and uh john starbuck is working on, the next big thing for us is probably just going to be moving the next service to kubernetes. We just finished, get https and get ssh following that. We're going to be creating a new service to uh to for websockets to segment websockets from the rest of our fleet, which is currently mixed in with the get traffic following that, then we're going to be creating a dedicated service for api and web.

H

Those are currently running in vms, so we're looking forward to getting those over to kubernetes once that's done, then our entire front end will be serviced by kubernetes, pods and also currently right now. Sidekick is also in kubernetes and uh except for the jobs that have to do with gitlab pages, which still relies on nfs, but that should be changing soon. So I think that's it for me. Does anyone else hear in delivery want to speak.

C

Just to tie all those three things together, so we kind of have these three focuses at the moment. One is around the pipelines themselves and reducing complexity, one is about dog fooding and how we can actually bring more of our deployment process and our release process into actually things that get up features either that we exist that we start using or uh expanding them. So they also work for our use case and then, finally, with kubernetes.

C

All of these things we get kind of out of the box deployment uh capabilities once that's completed as well. So at some point these three all kind of come together and will give us a much more powerful deployment process by powerful. I mean fast.

C

Cool are there any other questions.

C

One question I had uh for people who are in some way linked to deployments and releases: either you benefit from them or you're, doing them or or somehow like you're sort of linked to somehow is I'm curious to know what people think would be. If there was one thing you could magically change about the way we deploy or release with gitlab. What would that thing be.

A

uh Yeah, I'm always what I'd I'd love to see uh like real time, but that's like the definitely the big magic wand right, there's so many factors, not least of which is you know, access permissions impact measuring having good. You know, metrics, to determine how things are going um so I'd love to see an engineer, merge something into a master and and that's just let's live right away, that'd be really really cool and particularly at the scale we're running at.

A

But I recognize what a big ask that is, so I'm not not sitting here, stamping my feet expecting that to be done or anything so or even possible, necessarily depending on how the organization looks at it. Overall.

C

I think we'd love to see the same thing as we track on mttp. Like that's the ultimate goal, is it's like five minutes?

C

I think from my side um I would love to have a way to eject a uh mr out of uh out of one of our packages without having to run it right through all of our environments, or maybe I'm thinking like have the pipeline so fast that it doesn't matter. um Okay, I think it's. uh We quite often take the cautious approach of reverting an mr that we believe may be causing problems.

C

Excuse me um and the time that takes to actually revert it and rebuild go through our staging and canary environments is quite long, um so I'd love to see a way to just magically eject something and be able to continue things forwards.

E

I'd say for my customers: it would probably be you know that one magical thing would be to just have full aha deployments in kubernetes. um I understand the challenges. I explain them all the time, but you know, since you ask about magic.

E

I'll throw that one out there.

C

Yeah, that's a great one. What sort of things are people uh struggling with on the kind of uh with their kubernetes deployments?.

E

So the probably it's the when they when they try to really go highly available with kubernetes at least the the recommendations. I've heard recently have been that the back end services, postgres, redis and giddily- still have to be outside the cluster and customers would just like the whole thing to magically be in the cluster, and I understand, there's not an h, a cluster ready, postgres um and- and I don't think it's really our place to write one. But um but but that's certainly the request that I that I hear frequently from clients.

C

Yeah that makes.

C

Sense cool: are there any other, either comments, questions or anything else. Anyone would like to.

C

C

In which case, I have to say thank you so much to everyone, who's asked questions and a huge. Thank you dan for writing. All the notes, like amazing, um do you want to verbalize this stuff jason.

F

I actually was avoiding it because it's more on the distribution side than delivery, please um so when it comes to the, where does the state live, and why does it not live in kubernetes? um This is really more about how we're doing the charts and what our recommendations are based on our actual understanding of the applications.

F

When it comes to postgres. We actually currently recommend that you just run a postgres external to our chart, because we don't ship a easily flippable single postgrads to multi postgres.

F

It's it's not that easy to do, and even in the omnibus it's not easy. You can easily make use of another provider's chart in kubernetes to do postgresha. Just we recommend you not use ours when it comes to redis we're basically in the same boat.

F

In that doing it well is not easy and we don't want to reinvent the wheel so we're relying on bitnami's redis chart and right now. They too actually have a split between the individual, redis and then doing redis in ha with multi-redis.

F

That's something that we continue to rely on upstream. For the time being, we actually have a poorly, admittedly documented methodology to flip redis to be sentinel-ready within our chart, but we need to extend and expand on the documentation of how to do that using bitnami's chart.

F

The big one is giddily, that's the one everybody wants to see in there, I'm not arguing with it in any way. The reason we recommend it be outside is it's so easy to get wrong.

F

Giddaly has high high, I o rates at large scale. If you are hammering the poor thing with get pulls and all this you can actually crush the individual kubernetes node, on which the getaway service is running. If you don't understand how the application behaves and the implications of having 100 nodes do get pull at the top of the hour every hour.

F

This is much easier to handle when you don't have. Who knows what competing for cpu network disk we've had customers in the past, who had 100 percent kubernetes native deployments, and we have had some very unfortunate experiences for those users, because, despite having a dedicated node for getaway and giving it the same amount of traffic, they gave it elsewhere.

F

The constraints that were put on it resulted in getting literally just being kicked because it used more memory or cpu than it was supposed to, and it's really bad experience when your stateful service. That's that critical just goes from your services list and disappears off the cluster. It comes back really fast, but you shouldn't have to deal with.

F

C

Fantastic thanks for that.

F

And dan, thank you for verbatim writing that down.

A

If you're trying to keep up with it, like you can't actually summarize otherwise, you just lose track of it. So sometimes it's a little bit too wordy. I apologize for that, but you're welcome.

C

It's fantastic, it's great to go fantastic! Is there anything else anybody would like to bring up today.

C

Are there any um updates or progress things on any of the work we have going on, that we've covered that people would like us to share more widely like? Are you getting enough visibility of? What's going.

A

On I guess, I wonder, did the did that that release went out in the release post the one, the filtering, the mr's right, that that's actually in a deployment. Presumably it went out the release posted. We also put that in engineering we can review, because I have been reading that I'm not sure if I saw it, I just missed it, because it was something I skimmed over.

I

uh Yeah, it was in there, like, I think, a week earlier before the release, something like that.

A

Okay, so I just missed it. Oh thank you for that.

C

Awesome, thank you so much everyone for joining us today uh for the questions and say once again, dan super typing very, uh very impressed. So thanks for doing that, brilliant enjoy the rest of your days. Everyone we'll see you all uh next month, sometime take.

C