GitLab GitLab Hackathon, 31 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Q1 2021 GitLab Hackathon: Runner Office Hour

Description

https://about.gitlab.com/community/hackathon/

A

Yes, hello, everyone uh good morning evening afternoon, wherever you are um welcome to uh the runner office hour uh before I introduce steve, let me just spend a few minutes just to remind everyone, a few things. uh First of all, it's really important. I want to remind everyone, the community code of contact here at gitlab.

A

We strive to have a safe and inclusive environment for everyone, so please be aware that any unrespectful disrespectful and offensive behaviors will be addressed, and if you it's any point either through a comment or through a merge request or through any of our community channels, uh feel unsafe or threaten please reach out to us, and we will be there to respond to responding support as soon as possible.

A

um Again, this is the time we are almost halfway there. The hackathon begins on the 31st of march at your local time zone and ends on the end of the day of 1st of april at your local time zone, so we're already there halfway there. We have a lot a lot of emerge requests so far and I'm really happy. Thank you all for your contributions. It's really nice. We've passed the 120 something merged requests already when in during our last hackathon, the overall hack member mars were 167.

A

So that's really exciting um in case you need support or help, or just you want to reach out to the community. You can find us at guitar. Guitar.I am slash gitlab contributors. If you need anything specifically feel free to ping me through here or through any of her, mr or mars or issues, and with no further due.

A

I would like to introduce to steve who is going to walk us through one of my favorite and most successful I'll, say our office hour that we've been going on at gitlab for quite some time now uh for the runner. Let me stop setting my screen and the floor is yours.

B

Hey everyone, um so uh my name is steve. I'm located in malta and I've been working at catlett for around two and a half years, and I'm one of the project maintainers on gitlab runner, which is the agent that runs dci jobs for gitlab um the usual format of this office hour. Is we either have like participants during the call we can like answer any questions they have or review to help them contribute something to get lebron or if we have no questions or no other items on the agenda.

B

I can do some live code reviews on community contributions that opened in the past and that might give some folks some ideas what goes into the process of reviewing a a marriage request and trying to get that merchant to our code base.

B

um I see one person so far, so if you have any questions or any merge requests that you want me to go through, I'm more than happy to chat with you about that.

C

Currently, I don't have anyone uh that needs to be reviewed, I mean reviewers are already assigned to the mrs. So that's it awesome.

B

uh Do you have any questions uh in mind that you wanted to ask, or um is it just listening so.

C

I was just listening, but uh I mean: is the git lab runner, the one that is actually running the pipeline? I mean uh the integration tests that we that we write. That's where the runner is there.

B

Yeah, so that's a great question, because it's not very um uh if you're not familiar with the gitlab stack, it's not very clear what gitlab runner is doing right, because when, when I first looked at gitlab runner, I thought was also parsing, the gitlab ciaml and creating the pipelines and things like that. But that is not the case.

B

So uh to give like a quick overview of the architecture, we have gitlab the application, which is a ruby on rails application, and that is where all the data is stored, and that is what we call the single sources route. So that is where the gitlab crm parsing happens. That is where the jobs are created, that is where the pipelines are created and then, when everything is created, um it will assign this specific job to gitlab runner and basically, what the club runner is going to do.

B

Is it's going to get a specification for that specific job? So it's completely ignorant about pipeline scheduler. It only knows about one single job. It will get a specification of that job uh and little browner will tell it hey. I need you to run this script and with this infrastructure and so on and super and then kit labrunner.

B

Does it like it just parses that json json string and creates like a docker container with all the specifications and then just runs the script for the user in one container, for example, if we're talking about docker or it can be kubernetes or it can be anything else um so gitlab runner is mostly somewhat of a low level thing where the execution of the user script actually happens. uh The creation of the pipeline, the job and all the magic that is gitlab cima file.

B

That is all done inside of the rubio and race application, which is our monument.

B

Is that a bit more clear or do you have any follower questions understood perfect?

B

So let me share my screen actually, and I saw some community contributions coming in during the hackathon, so we have two so far um and it's always nice to see documentation changes. Those are like one of my favorite. It was mostly because they're super quick to merge and they improve the the quality of life for everyone. They improve the quality of life.

B

For me, when I go reference, my our own documentation, when I'm trying to figure out a problem and they help uh increase adoption, so we actually we're very lucky at gitlab that we have a technical writing team where they spend like enormous amount of uh hard work, making our documentation look consistent and uh with proper grammar and so on and so forth, which is something most developers don't always have time to do so.

B

So we have this process where uh engineers usually vet, that everything looks okay from a technical level and then we usually assign it to our technical writer, uh which will look at the consistency and the flow of the documentation.

B

So we can go through these two and I'll do that right now, so this one is updating uh reference for minion and one who's not familiar. We use uh manio for caching, so uh if you're not familiar with what caching is for gitlab runner, uh imagine that you have, for example, let's stick with a nodejs application right most of the time you have to run an npm install before you run any tests or any builds right, and that means you're downloading a bunch of javascript packages from the internet.

B

And if you do that for every single job for every comment, you push that's going to take a really long time right, because you have to do an npm download everything and over and over. So we have this caching feature or you can cache a specific directory. So you can cache the node modules, for example, and doing so you don't have to download everything over the internet and one of the backends. We call it is using minio.

B

B

Let's see what this document, what this is doing.

B

B

So, like the first thing I always end up doing is looking at the description and see if there's any related issues to it,.

B

B

Okay, so basically, what this is saying is when you start a menu uh contain docker container. uh It gives you like it, uses these specific set of default credentials and um for simplicity, simplicity reasons. We always uh use those uh default credentials, uh but this one, what it seems to be doing is it's um updating uh the menu, uh username and password, which I agree with.

B

We should try and have uh the most secure uh copy, based uh kind of commands for users, because um me personally, I do it all the time, I'm not too familiar with these technology sometimes so I just hope the base and then, like I kind of start to, I forget to reset it or something like that. So it's always good to make it clear to our users. They need to do something to make it secure by default.

B

uh So let's see what this is doing, so just updating the docker to start it in background mode in demon mode. So if we look at.

B

B

B

So it's a it's gonna run the uh container in the background, so I think that is fine uh before we were running it into the foreground. So when the user starts to continue, it spits out the logs and they can't really exit, but without terminating the container.

B

So I think that's good, because uh the way this minion container is gonna, be it's running in the background um in the server somewhere. So I think, uh passing in the minus d is good, um so this is setting the menu root, pass user and password.

B

uh If we look at d minus e, that is setting an environment variable.

B

B

B

Perfect, so what I usually do in this case this seems to be correct and I agree with the change. So what I'm gonna do is.

B

Go into the markdown file and just act as if I am a user. So I'm just going to copy paste this.

B

I have a docker for mac running locally on my laptop, so I can just do this and I'm gonna pass in um just let's do it to my temporary director, so it's removed after I restart my computer and it's starting a minial container right and I'll just pass in the username, which is.

D

B

So this command seems to execute as expected, so that is good. Give it a second to download the container.

B

So if we do a wps, we should expect a.

B

There is no monster night.

D

B

It might be because I passed the wrong thing, so let's die.

D

D

Let's try it again.

B

So now I have the menu container running, so if you do adopter logs and search for video, I see that it actually started. So if we go to.

B

Localhostport9000.

B

That does not seem to be working because.

B

We are um forwarding port 99005.

B

So if I zoom in now, I see the secrets, the indexes key and if I look at it should be hello, gitlab and secure password. So let's write that out.

B

And that worked so I like this, so to see what we had before. I always uh try to like enjoy going on our master branch, which is our uh main branch that we consider as a stable branch and, let's just.

B

Run it again to.

B

D

That's the first of the mania.

D

B

B

Oh, wonderful, so, okay, so it's this command used to fail as well, so that that is good. So it's fixing the commanders at the same time. Instead of using the default credentials.

B

Awesome, so I think this looks good, um given that this is just a updating, the docker command and one sentence. I feel comfortable merging it myself. So uh there's always somewhat a judgment. Call right. If you want to.

B

um If it's a large change, you should always go through the technical documentation. If it's a small change me myself, I can merge, so I feel comfortable matching this one, so I'm gonna say.

B

I'm just gonna merge it myself, so thank you so.

B

B

So um yeah that should be added to the merge string. It won't be merged uh immediately. We can see a new pipeline started, so we have a feature at get lab called merge range where we run the pipeline on the merge result, meaning the current branch and the master branch merged to see if it will cause any failures, and this way we can prevent any broken master pipelines, so that will take a while to run so we can move on to the next merge first that was opened as well, and that was this one.

D

B

Update top security index, spelling it grammatically, so let's see what it is.

B

Yeah this is like the perfect merge request as well to get started with right, like it's just updating a few um grammar mistakes, so I'm just gonna, approve it and add to the train as well.

B

Just because it's small enough- and we don't really require a technical writer to check that.

B

D

B

That's it: um let's see if we have any more merchandise that came in through the hackathon, so um we usually use labels for the hypertension, so that is quite useful, so communist contribution.

B

And there is the two that we have currently open and they are approved and in the merchants. So that's good. So unless anyone has any questions or comments they want to go through.

B

I have some other merchandise that I have plans to review.

A

uh Give me one sec to check again.

A

B

Awesome, so um uh what I had planned to review is two community contributions, so the first one is around kubernetes.

B

So for folks who are not familiar, um gitlabrunner can run on multiple types of infrastructures right and we call these executors so imagine that your application is containerized.

B

uh That means you're most likely using something like docker container d or like kubernetes, for example, and like your ci infrastructure, should try and mimic as close as possible to your um production environment. So when you run the test, it's closest possible to your production environment right. So we support multiple execution environments.

B

um We, the most popular ones that we support, is uh the docker executor, so we're running everything inside of a tucker container, with the image that you specify, and we also have the kubernetes executor where we run the job inside of a kubernetes cluster by creating a port for you, and everything is isolated inside of that pod. So one job is one port.

B

And the way we do, that is talking through the kubernetes api, for example. For this specific case, and um basically, as I said, we create a pod right for each job. Now the pod can have multiple containers inside of it and with kubernetes. You get a lot of flexibility from a scheduling perspective, so with kubernetes you get multiple nodes.

B

So, for example, let's say you have a cluster of three nodes, so three compute servers I like to call them and you might want, like one one of the nodes has a faster disk or one of the nodes is a different cpu architecture, so, like r, for example, is very popular nowadays, so you might have a cluster with uh intel, cpus and then arms rmcpus, and this specific application that you're running only runs on r, for example right.

B

So you want to make sure that this job only runs on the arms uh on the arm notes, instead of on the internet, for example right on the x86, so that is where these uh powerful constructs of kubernetes come into place. Regarding scheduling, um there's a lot of them, I will not go through.

B

um I will not go through that myself, because this is not the communities tutorial, but it was mostly um runner related. So we have what we call affinities. So there are two types of affinities. There is a. Let me zoom in a bit sorry about that.

B

There is a node affinity and there's also a pod affinity.

B

I will not go into too much detail, what's the difference, but um basically one of them is selecting where the notes is gonna be scheduled and then the pods is making sure that one part is close to the other part so like, for example, if you have at the server it should be next to your application server just uh so it doesn't go over the nar, for example.

B

Those are um a few examples. The documentation here.

B

And this url is, um it goes through all the details, so I will not explain everything there, because that can be a whole to our lecture about it. So at the moment uh we do support node affinity. So if we open up our runner documentation, so we go to runner executors.

B

So if we go to this url here, we will see all the documentation related to our kubernetes, um so using affinity, so we have. And basically what is this saying? Is this gitlab runner when it creates a pod? It's gonna attach this uh affinity affinity specifications to the job board, so then it can be scheduled properly and right now we support node affinity and all our github runner configuration right now is inside um tamil, and that is like a different configuration language similar to yammer, for example, right.

B

But we have to have the tunnel structure so then we can translate that into the json api from kubernetes, so we send kubernetes say: please create a support with this affinity.

B

So right now we support the node affinity and this one is adding the port affinity and the port and the affinity.

B

So I see um config so we're updating the config and then we're updating the kubernetes um uh tests.

B

So one thing the first thing that pops up into my mind is we're adding a new feature but we're not adding documentation.

B

One thing that I really like to see is one merge request where, if it's adding a new feature, it should be adding documentation. So everything is titled like in one package kind of thing. Sometimes it happens. Sometimes you have to have documentation in a separate merger as it can be, because the merge request is too big or there is some time pressure or like um in the case of a community contribution.

B

um You don't have time to contribute to documentation so that that is something that sometimes you take on. um I I usually enjoy communicating this with the contributor because, like they might not be familiar that they have, they might not be aware that they had to add documentation. So it's always like a two-way communication like just hey, be aware that we need documentation. If you want to add it, you can, but um if you don't, we can we're more than happy to do it.

B

So let me comment on.

B

B

D

See that we are.

D

B

Let me link to the documentation, so if they want to update it, they know where to go to so that is.

B

B

So we need the part infinity.

D

D

B

So that is one of the things.

B

Let's see what else there is uh here, so this is not touching the node affinity.

B

Adding in a pert affinity and adding the part on the affinity.

B

Okay, um let me open this up in my editor as well, so you can look at that.

B

So I have an analysis in my git config, where I do get a more origin and the id of the amar, and it will automatically clone it for me, which is quite useful to um quite quite quite useful to uh for code reviews and I'm gonna try and find it as this alias right here, where it automatically automatically checks up checks out anymore for me.

B

So let me open this in my.

B

B

Let's go to the file that we were looking at here in the comments.

B

Search for quad affinity.

B

So I see that this description is missing.

B

And I feel like we should add it, because the description is somewhat important to the users that are trying to cover vertical brand right, and this is also shown in our help. Documentation.

B

And let's search, let's do.

B

Less give it a second to load.

B

There we go, and now, if we search for thank you.

D

B

We're not showing this in our command.

B

Okay, it might be because.

B

Yeah, I think that makes sense. uh It might be because um we're not specifying the av keyword and uh we have like some automatic transformation from these fields too.

B

That lets me double check something yeah, so um yeah, and that makes sense, because these uh structures are not something you can start with environment variables right, because these are somewhat complex of a structure, um because it's it's having like a bunch of this structure, we usually have like environment fibers for values like streams. So, for example, let's, uh let's look at an example- um I don't know the kubernetes source, for example, where to find the kubernetes api.

B

So if we look at the help commanders again just to show you an example.

B

um We will show you that easy amount of it, maybe uh optional, kubernetes master host url. When you register a runner, you can pass in the kubernetes host environment url and it will automatically populate the configuration.

B

But since this is a complex structure, it's not just a one field value, so we don't really specify it. I allow the user to specify it, because it's just confusing at that point, um but they still like to have a description, even though we won't be used anymore, just as a a documentation.

D

Purposes inside of the code right.

B

So let me comment on that.

D

B

So, and by the way, what I'm using here, it's called uh conventional comments and it's actually a nice way to specify uh what kind of thing you're talking about in the chord review. So, for example, if it's a suggestion, if it's a nitpick, if it's blocking or not blocking, if it's a issue that you think it needs to be resolved and if.

C

B

A question, for example, like most of the time, it's me asking questions in a code review so like hey. Why is this like this and I just prefix it with questions, so the user, like the other person and the other side of photography, knows what I'm trying to get out of this, or if it's a chore like hey, we need to update documentation or hey.

B

We need to run this script, for example, before we do that, so it's always nice to run through this, um and probably what I like the most is the praise like when you say something nice about the decoder was contributed.

B

B

B

So so sometimes what I do is like. I either give the solution to the user, to the I keep, calling them users and I shouldn't to the community contributor or to uh or just tell them like, hey uh I'll, let you figure it out. If you have any issues uh reach out to me again, um and that is mostly like a um how to say a case-by-case basis. um I've worked with this contributor before and uh we have somewhat of a relationship with him.

B

So what I usually do is just point out an issue and he usually fixes it, but if the user is struggling or it does not have the time capacity, I usually just give them a solution or, and then they just apply it, for example.

B

So it's mostly a case by these basis, so that is it on this case. One other interesting thing is uh we're updating the tests, which is perfect, but what I see is we're not actually updating.

B

The kubernetes itself, so I'm curious how it's updating the uh response, and that is because see.

B

Let's open it up in my.

B

B

Because, basically, this is just the user configuring it, but somehow we have to get all the configuration here and send it to kubernetes api. So I'm trying to see how that is done right.

D

B

And I'm guessing, it is part of the get affinity, and that is what it's doing here.

B

So api affinity is the definition from uh the kubernetes api and, as we can see, there is no dfinity pod affinity at port on the infinity and I'm guessing. The merge request is updating this um method. So, let's see where that.

B

B

Let's see oh there, we go that's just line here, so basically we have one function where it's called during the creation of the api request right here and this parses everything for us, so note affinity is something that we've already been doing. So that is something we don't care about.

B

Now we care about part affinity here. So let's go inside of this method and see what it's doing so I get part affinity. I expect this to save and the and new methods- and that is what we have here.

B

So uh let's see this is just ranging over the required and preferred.

B

So what does this mean- um and this is uh probably the trickiest part- about this merchandise right- making sure that we map our config tunnel to be the same as uh the kubernetes structure and the api, and now we had discussions with the community to try just to try and make this easier, both from our perspective and our users perspective where they just specified the part specification and we just translated and that's it, uh but at the moment we have been doing it on a case-by-case basis, but might not be ideal.

B

But that's what we have uh right now, um so uh what I like to do is look actually at the api specification here.

B

So it's under the kubernetes api reference and I just want to deploy affinity one and we have two high-level ones, so preferred steering scheduled, ignored during uh execution and required during schedule. If you are ignored during execution and if we look at our code, that is pretty much what this is right.

B

So this is the required.

B

And this is the preferred right here.

B

So those two high level, few high level structures are.

B

B

So this is the getting the and each, for example, let's start with the preferred during scheduling it has a a put affinity term, a topology key, a label selector a name space and the weight. So what I expect the user is to be able to set all these inside of the config.file.

B

So I get part affinity term: let's see what that is doing.

B

D

B

The label selector.

B

Name is based on the topology key, so topology.

B

So a few questions come into mind like where is the weight and so on and so forth. So, let's see.

B

Okay, so this is looking at the kubernetes api and it specifies a of an editor and it's a label selector namespace, anthropology. Okay, that makes sense- and we are- ah this- is about the required. Of course, that's why it's not matching so we're talking about required schedules during scheduling, sorry, so the logic key label.

D

B

Namespace, that makes some sense.

B

um So what is testing label.

D

B

Label selector, so let's see the api definition of a label selector and make sure it matches.

B

It's the key, the operator and the values, and I'm guessing. This is what we have here: uh the image labels and the expressions.

B

Now, I'm pretty sure we have a label selector somewhere else.

B

Where is this the first time we're introducing clipper.

B

B

And also feel free to jump into um any uh dude, with any questions that you have at the moment, I'm more than happy to answer.

B

D

B

So it's specifying the label selector and put infinity.

B

Key operator value, so let's see how this is.

B

B

So this is creating a requirement. It's looping through all the which expressions.

B

And it's returning deep.

B

D

B

Sense now um I will not go through all the translation field, because that was very boring for you to watch, but I'll go to some other things that I usually do when I review emergencies.

B

So uh the lucky part is we're doing a lot of uh we're doing more tests than we're doing actual code, which is nice, but the reason we want to have tests is to have coverage on all the um cases that this can go to right. So what I usually look is at the coverage report now.

B

uh Unfortunately, this community contribution is quite old, so I don't think it's gonna. Have the coverage report still available? So oh awesome, so part of our pipeline is generating the coverage report. So if you look at what we're doing here- and most of our logic is inside of the config.go file.

B

So if we look at common- and I will zoom this soon-.

B

D

B

And this gives us a nice report of, like the red part, is code that can be covered, but it's not covered degree. One is code that is not able to be covered by our tests and the green one is um tasks that are being covered and the higher intensity green. It is the more uh hits we get.

B

Basically so the more tests we have covering the score part, which is something that might be useful to say to think like if we need more tests or um if one thing like one line, is only being covered by one test and that might help you think like. Oh, should we add more tests or not?

B

So uh let's look at what we're adding here. So, um let's start with get affinity, for example, and we see we have a few coverage here, but we're.

B

We have one hit actually, so we only have if you hover on it, it will tell you how many uh coverage you have, so this is only being tested by one test, which is fine in this case, because someone wants us that we need right.

B

um Let's look at the, for example, the get.

D

Label selector.

B

I see it is also being tested and it's which.

B

And, for example, here we see that we added the uh get part affinity, for example. This is a new change. We added, but we're not really testing anything about the preferred during scheduling ignored, and this is a clear thing that hey, we should probably add tests to this, and this merger press so make sure everything is covered, and this helps me like figure out. Okay. What is uh the state of our test and should we add more coverage, um just a whole philosophy where you should like?

B

You should never aim for 100 uh code coverage, because that does not mean that all your tests and all your edge cases are are uh are tested, and that does not mean that your code quality is a good either right, because sometimes you can like you have to go like uh way, uh adding a lot more complexities to just get 100 coverage.

B

So it's like uh mostly a judgment call here as well like should we have coverage or not, um since this is just parsing fields, I would feel a lot more comfortable if we have coverage. So, let's see why this is not being covered at all.

B

Let's look at the config test here.

B

Let's open this.

B

B

B

So we have depart affinities, but what I do not see is the port and affinities being tested at all. uh If not.

D

B

We go sorry, I spoke too soon so.

B

B

B

So this should be covered because we are testing.

B

B

Sorry, this part so preferred during scheduling and the coverage is.

B

Let's try and generate the report locally. That's.

D

B

Does not seem to be.

B

B

I'm just going to check if this part actually will run from this test or, if I'm misunderstanding, something yeah.

B

B

B

Okay, this is part of the cuban lettuce. That's done not the coffee test.

B

Because the conflict is just strictly checking the marshalling of the tunnel structure.

B

I know we're at times so I'm gonna pause here to wrap up, but uh basically what I'm gonna keep looking at is if the translation from our configuration to the kubernetes api makes sense and if it's like a one-to-one relationship making sure we have the right amount of coverage and also making sure that we have uh the configuration uh the configuration documented inside of our inside of our documentation.

B

um If there are any questions, I'm more than happy to answer, we have three minutes left, so I can either continue or wrap it up.

A

uh I don't see any questions either on youtube or guitar, or here so yeah. We can wrap it up.

B

Awesome so just to give a quick overview of what we did.

B

We uh reviewed to merge requests about documentation, changes, those were pretty straightforward and then we have uh community contributions like these ones, where they're a bit more involved, just because it's either a lot of data structures involved or like a lot of um considerations to take into um a lot of factors to take into consideration. Basically, uh we have to keep in mind that if we release the configuration, uh we have to keep it backwards compatible because we can't have users updating the gitlab runner version and their old config does not work anymore.

B

So it's something that we have to be careful a lot of the time where we try to add a new configuration. We need to make sure that is something that we can extend in the future, and that is correct just because, if we want to change how that works, we need to make it in a backwards compatible way and make sure we deprecate it slowly and like inform our users, and we usually do the applications uh once every year when we create the major releases.

B

So, in this case, for example, in 14.0, that is when we can do some breaking changes, and that is always a long and heavy process, and it is by design to be long and heavy just because we value backwards. Compatibility uh a lot because, like if your code keeps breaking every upgrade. That is not a nice experience for our users.

B

So yeah, uh that's it. I think.

A

Thank you so much uh and thanks everyone for participating and watching uh see you on the next um office hour, which is in one hour from now. It's gonna be the package group office hour.

D