GitLab Incubation Engineering, 16 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Incubation Engineering 5mp - Weekly Update - March 16th 2022

Description

5mp is now Cloud Seed ⛅🌱 https://hello.cloudseed.app

Cloud Seed is an open-source program lead by GitLab Incubation Engineering in collaboration with Google Cloud.

Deploying web application (and related workloads) from GitLab to major cloud providers should be trivial.

Cloud Seed makes it ridiculously simple and intuitive to consume appropriate Google Cloud services within GitLab.

00:00 Intro
00:18 Agenda
00:31 List of MRs
00:45 MR !78875
01:51 MR !78712
02:42 MR !79487
05:22 MR !80895
06:54 MR !81890
08:16 MR !81069
10:03 Phases & Testers
12:18 Live on dev.gitlab.org
14:40 End-to-end demo part 1
15:25 Last ever 5mp video?
16:12 End-to-end demo part 2

A

This is shree and welcome to the last ever five minute production video update.

A

It's been several weeks since my previous update and I have a lot of features to share with you, a lot of updates to share and demos to present.

A

While alternating between travel and pto for the last several weeks, six important merge requests have been worked upon and merged.

A

Let's jump right in the first merge request: I'd like to share is 78875, which adds a confirmation checkbox for the service account creation phase. As you know, service account keys are sensitive and we want to validate and verify that the user knows what they are doing.

A

We also introduced a little bit of an alert which I will demo as well. The reviewers for this were anabcenko.

A

I hope I'm pronouncing the names properly, I'm pretty sure I'm not, but apologies for that and thank you for the excellent review, let's jump right into the demo, so this is the little alert over here. It says: enhance security by storing service account keys and secret managers, and then we link them to the relevant documentation page around that.

A

So this is one thing that was added. The second thing was, uh if you go into the create service account form, there is a mandatory check box which doesn't let you proceed until you check that in so these were the two changes that were introduced.

A

The next one is 78712 titled grant permissions to the generated gcp service accounts, so we are creating these service accounts, but we need to authorize these service accounts to act.

A

So this merge request introduced changes which add specific gcp roles so that uh the service accounts can be used by the user for the tasks they were created for here's the list of roles I'll not get into them in detail, but just uh just highlight that we can discuss this offline uh if you want to understand a bit more, why certain roles were added and certain rules were not the reviewers for this, mr, were mehmet amin and peter leitzer. So thank you to both of you for your support.

A

Next, we have 79487 titled, configure gcp regions from projects in front of google cloud page. This is a big one, as you can see from the list of reviewers. It had six reviewers. So thank you to patrick bear nicola milo milovic.

A

Apologies again for messing up the names, but thank you for your support and review. So the idea is that google cloud has got a bunch of regions and we want to allow our users to simply configure where certain deployment environments uh should actually get deployed to so I'll. Just start off with a quick demo. Here you can see if you go to project infra google cloud there's this section out here, you'll see saying no regions configured. So let's jump right in and configure the region I'll say.

A

Okay, I want my master branch to be deployed to uh north europe. North 1 press the button, and this creates a ci variable which gets used in the pipeline and selects that as the region again, the refs can be all as well. So let's say everything should be deployed to us central and, for example, I'm going to select another branch which must be in asia.

A

The way this works is that now you have three ci variables: let's confirm that we have three region: ci variables uh targeted for specific environments and uh based on that, let me see if I can reveal all the values should be safe.

A

So, based on that, uh if a pipeline is executing for the master branch, the value europe north takes effect and if and asia is for deploy and us enter for everything else- that's pretty standard how it's going to work, and now the user has got this little bit of interface for them to easily configure their regions.

A

You might have also noticed that this is actually a subset of all the available regions, so we want to select a few regions which have all the features that we need either right now or in the future. So this is a subset uh over time. This can be added, but personally, based on the little bit of research that I did, I decided the best way forward is to reduce it to the subset and over time we can increase it.

A

You have basically asia, europe and us, so that's already quite broad, but specific zones can be added over time as well. That's that.

A

The next one is awake sounding uh mr it's 80895 and it simply says: refactor google cloud service account creation, workflow. Some specific things were done over here, uh which are quite relevant. Let me first thank the reviewers bom bob fan nand out uh miguel rincon alex spouse and tom crook. Thank you all of you for your support.

A

Okay. So what does this? Mr? Do you already noticed? Git refs during the service account creation or the region configuration workflows. This used to be environments. We changed that to git ref, so select your git ref and from there we'll generate the environment. For you. uh This is consistent with the way we do auto devops or the original aws five minute production workflows, and the added benefit is that now there is no reliance on gitlab enterprise only features. This is 100 open source.

A

Apart from that, there were little tweaks made in the front end table component uh of some optimization on how we update the ci virus in the background, some serialization being done for efficiency and so on. So minor changes nothing to demo, except the fact that you are now selecting a ref and not not a pre-created environment. This is true for service accounts as well, so you're, selecting a ref and not an environment.

A

A

Eight one eight nine, zero track project, google cloud events with slow cloud, so snoop love is a little bit of a framework that our team is set up. um It's excellent everybody in gitlab uses it for tracking events. So, okay, let let's have five minute production use it too.

A

So all kinds of uh events, for example the user landing on the google cloud page within the gitlab project or the user, creating a service account or failing to create a service account or configuring a region or setting up the mr all of these are being tracked, and this will be useful because we plan to invite some external testers soon, so it'll be nice to see what they're doing so that we can debug and find issues and try to understand the usage. So we are at the moment only tracking events.

A

There is the next logical step where you have to make some kind of a dashboard or reporting tool so that we can understand, what's going on, that's a different story for a different time for now just emit event from the source code from the application when certain things happen.

A

The reviewers for this nikola milojevic uh excel garcia and maxorface again excellent support. Thank you. So much.

A

And finally, revoke authorizations81069, so we'll give user explicit control to rework to revoke the granted oauth token. This happens right now um automatically. I think it times out after 10 minutes and then the token is not valid anymore, but this gives the user explicit control to do so. Let's have a look at the demo, so uh so, let's revoke this authorization, because I'm seeing the button okay. So now I'm not authorized and if I was to create a service account or if I was to do something else.

A

Google would ask me to grant permission to this gitlab instance right. So you are giving this gitlab instance. Some control over your google account and this token expires in 10 minutes, but you can explicitly revoke it if you like. So if the token is active, you see this little widget out here with a button and you can revoke the authorization, keep in mind uh that when you revoke the authorization it doesn't actually nullify the service account that is created.

A

So there's a little bit of text here that says that this does not invalidate service accounts, so the service account which is used by the deployment pipeline that still has permissions, but the gitlab application gets its permission revoked on press of this button.

A

Let's thank the reviewers. uh We have ezekiel kickball great support from him albert salim, doug, stull and justin ho tuan dwong great support from everybody. Thank you.

A

In january I showed you this. I showed you this timeline, it's pardon the the graphic, it's a very poorly done timeline, but I gave you an update for fi 22q3. This is it f522, probably uh sorry, I get confused uh f522 q3 yeah and I explained the progress made in that quarter.

A

Then I laid out the plan for q4 fy22q4. That was the last quarter that ended january 31st, and then I set a goal for myself that in this quarter in fy23 q1 we should have a preview rollout made for early testers. That was the target I set myself. Let's have a look where we are now we are done with the development phase. uh Famous last words, you're never really done with the development phase, but we've developed enough that we can actually move on to the testing phase.

A

So the target for april, which is the last month of q1, is to invite external testers. We've been promised external testers from several uh partners. Who've uh who've been very vocal in their support, so we want to invite them and the mechanism for that will be rolled out in the coming weeks, but the idea is to have them sign up on gitlab.com.

A

We invite them to a very specific trusted, testers group on gitlab.com and in that group the feature flag will be enabled and they may create their projects and test out this feature. So this is scheduled for april 22 and we are on track for that. uh What I will initiate this week and I'll give you a detail about that in my next week in my next week's video. But what I will initiate this week is the absence review based on that we'll understand what is the?

A

What is the fallout like what changes we have to make and how quickly we can roll it out, but we have uh one and a half months to go before the end of the quarter. So that's plenty of time to get an initial reaction for the absent review and to privately invite trusted testers.

A

The focus for the next quarter, of course, is to launch it and then launch it together with our partners.

A

But what? If you are a gitlab team member I'd like to draw your attention to the url out here? The url is dev.gitlab.com on staging and dev. This has been enabled, so I invite you to try this out right now.

A

If you want, of course, uh no pressure, the only requirement of you is that you have a gcp account and you have the ability uh in or the entitlements in that gcp account to create a project or you have access to cloud run on an existing project.

A

So uh typically, based on the reviews that I received in all the mrs many gitlab engineers do not have the ability to create new gcp projects because of the limits that that there that that exists. um This is something I cannot help you with, uh because I don't have the permissions to grant you those uh those rights to create a gcp project. But I do invite you to try out with your own personal gcp account, because cloud run as such should not should not cost you anything so running.

A

A small application on cloud run will be very much in their free tier, so it's not going to land a big bill, and if there is a tiny bill here or there, you can of course expense it. But that's data. You are invited to test this out on dev.getlab.org.

A

It's live it's running and if you are, if you're, not a gitlab team member, then uh I will set up a mechanism in the coming weeks. I'll set up a mechanism where you can sign up on a google form and uh based on some very basic wedding of the candidate. We will invite you to a private testers group uh in that group, we'll enable this feature and you will be able to try out this amazing uh infrastructure, google cloud page.

A

You will have access to this page and you'll be able to uh deploy your web applications via cloud.

A

Finally, let's see an end-to-end demo, while I talk a bit, let me go into the deployments tab and let me hit this button. It's the cloud run button configure via merge request, and this created a branch. For me the branch introduced a pipeline and let me just create the mr, so now I've merged the changes uh in the gitlab ciaml file. This has triggered a pipeline execution.

A

Let's have a look at the pipeline execution. It's got one job, very basic, deploy to cloud run and.

A

While this job executes it's going to take a few minutes for it to execute while this executes, let me let me talk a bit about why this is the last five minute production video.

A

It is the last five minute production video because as of next week or I'm hoping as of next week or a week later, this project will be renamed. So we are no longer five minute production, we'll have a public facing uh name which is catchy which is representative of what we're trying to do and which will be marketed right through. So I hope I was able to build some suspense around why this is the last five minute production video, because the next video will be titled, something else.

A

That's that here's the pipeline being executed, you can see it's fairly straightforward, pulls a darker image which contains the deployment scripts and everything else and it executes. You can see that it's passing the region based on the configuration. Let's verify that, shall we, the region, is u.s central one, and did we configure us central one?

A

Oh, we did so again, it's applying the all wild card, so this new branch is called deployed, cloud run and it's using us central one. This is my project id. This is potentially going to be the url bam there you go, the job is fully executed, which is great, and let me close this tab and close this tab and take you back to the mr.

A

This is the mr right and let me refresh- and you will see that we have a button out here which invites me to view the application and the application is the one that we just deployed. This is an end-to-end demo. Hopefully this long video update was exciting enough for you uh and, I hope, you're looking forward to the new 500 production name next week, see you later.

A