KubeVirt KubeVirt Summit, 24 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Avoid merging broken code with Prow

Description

This session is about creation of prow jobs in general. Prow is a Kubernetes based CI/CD system.

We cover what is a prow job, what job types are available, how to configure job triggers, testing jobs: requirements, obstacles and pitfalls configurations for merge gating other usages of jobs (publishing, bumping, ...).

Attendants will be able to know what job types are available, which one to use for a specific problem and how to verify their job works on their machine (TM) before creating a PR.

Presenter: Daniel Hiller, Senior Software Engineer OpenShift Virtualization, Red Hat, @dhill3r, @dhiller (GitHub)

A

Position and kubernetes like having said that daniel or yours, thank you.

B

Thank you too um yeah, I'm going to talk about uh prowl, um especially this is important for keyboard, because we are using prowl as our ci infrastructure. um So um at first I want going over the topics, so we are going to talk about what is brow. What is a proud job in general? What job types are there? um How do you configure jobs?

B

um How do you test jobs, especially not on the server of course but locally, um and how do you configure jobs for merge, gating and yeah the final stage? We will also discuss other usages of jobs and.

C

If there is time.

B

I'll try to do a small demo of testing pro testing a project. So, let's start what is prop browse ac icd system based on kubernetes. It listens to github events and can react to github commands.

C

And issues and pull.

B

Requests and it can attach status to github commits which is important for merge gating. It can automate the merges of pull requests, um and the most important thing, I think, is that it can run jobs when github events occur so um also. um It is, of course, the front end for the pro components for the several components that are there, for example, for tide or for prow job overview, which is what I'm going to show you now.

B

This is the proud centers page where you can see the recent proud jobs like what status they have um when they were started. How long they ran how the overall job status is. You can filter by job types repositories or by authors.

B

Sorry for that, um so what is a proud job, proud job in general is a kubernetes object, then contains a ports back a certain decoration center status, so a project was created either on certain github repository events or on regular basis.

B

So um and also a projob is created from a job definition that is located in the repository in general and then somehow loaded into the prior configuration.

B

So the job status that is derived from the exit status will then or can then get attached to github commit objects in github, which is the basis for merge gating.

B

So what types of jobs do we have? We have a pre-submit job which runs on each push to an unmerged pr. We have post-submit jobs that run on each new commit.

B

We have periodic jobs that run like, for example, a cron job or on a time duration base, and we have batch jobs. That test can test multiple unmerged prs. At the same time,.

B

Yeah use users, examples of uh jobs are, for example, for a pre-submit is in general, of course, the validation of a commit before a merge, so, for example, run unit tests for a post submit it's, for example, something like um some some update after merc. So, for example, if you have merged to a branch- and you need to build update website on that, so you do the post submit.

B

Then there is a periodic uh which, with which you can, for example, do regular testing or regular creation of resources or whatever needs to be done regularly and of course, there's the at the last music example for for batch job is that, for example, tide uses batch jobs to test multiple pr's at once, which are tested all in in one job.

B

um I only want to concentrate today on pre-submit jobs, because we just want to look at how we can disable merging broken code.

B

um So I want to talk about uh what uh what a proud definition at first looks like it's proper yaml, of course, where you can see at the top um that there is uh that there you can see the group where all pre-submit jobs, for example, are located, and the second is an org repo string, so that uh brown knows for what repo this job needs to get executed.

B

um Then you have the list of the proud job definitions and the interesting thing about this is, of course, that each job definition has a pot spec that defines what the project does.

B

So um then you have some options because you need to, for example, um you can you can uh you can tell the job definition that it only runs, for example on on a branch types like for example. In this example, we have the skip branches, so we are going to skip all release branches and just run, for example, on everything else that doesn't match this pattern.

B

So then, with always run versus run. If change you can decide whether it should run on.

C

B

Change in the repository or if it should run on specific file path, changes where with which you can do with run if changed, where you can input a regex pattern. So if it matches that expression, then it gets gets executed.

B

Then the optional flag is uh required for um telling prowl or telling um in general of the system. That um is. Is this a job? Is this job execution or this status? Is it required for a merge to be enabled or not?

B

um And at last the skip report um tells prow whether it should propagate the status from the prowl job run to github and attach it as a status to the commit and, of course, just just as a side example in the in the bottom, you can see presets that are away on proud jobs, just to generalize a general configuration so yet that you don't have to repeat on every job.

B

If you need, for example, if you need, if you need um credential information or something else or other settings that are required for this job, so that you can generalize this and externalize this so that you don't have to repeat it any everywhere.

B

Okay, so the interesting thing about proud job is so you see this this example of a small crowd job which uh tests, of course, like you, see it, calls them a test target. So what you see in this container spec there is an image qrci bootstrap. This has nothing to do in general, with our cuvette repository, where this job in general runs.

B

So the interesting thing about prow is that by decorating the project or pre-submit job, it detaches the current state of the repository that is to be tested to the container and um makes it available. So I can just call the make test target of the cubeville repository with that yeah. That's what I exactly said now, so I was a little bit too fast.

C

B

So um one thing you have to notice, of course, is that any tools that you need, while you are going to test, they need either be provided with the image from which the container is run or they need to be in a repository. They can get checked out or they need to get downloaded on the fly.

B

So, um for example, what we also have is required credentials most of the time.

B

For example, you need a github token or something so that you can fetch execute actions, and uh these are getting these get mounted in the usual way like like volume, ones and volumes, um and uh of course you also have to have um you don't have to have some automated hiding like you, have it in some other things, for example, this is one caveat that you need to be aware of minus x and scripts, which might meet might lead to leakage of tokens.

B

So, okay, how can we test the project?

B

The proud job definition in general is only active when it gets merged to the configuration branch of your pro configuration, so uh prowl then loads the configuration from the configuration repository and then it gets active, which means that as long as the product configuration is not in a merged branch, it's not active, so you can test it a way to test. It is just to run it.

C

Locally, for example, or you can also, of course,.

B

Run it against the cluster, if you don't want to run it locally, because you can't um but yeah. My point is that I want to test a proud job before get words. It gets merged so that I know that it's working.

B

So there are two steps to test power: job the one. The first um is to create the proud job yaml so uh from the job definition, and the second is to actually run it.

B

um The first one is you use mkpj, which is a tool from kubernetes test infra to create the project jammer, uh which is uh here's an example. How to do this, for, for example, for for an example, job from mine and the second step is actually run the project. Like I said in the um a example, it is just a cube cover, create minus f from the project yamu and then it gets executed on the cluster or use.

B

Final final is also a tool from test infra, and um this enables you to run prowl drops locally, so um they attach everything to a docker to a docker, a docker container, and this docker container then runs locally on your machine.

B

Okay. So um then, I'm going to show you a demo. I hope everyone can see or let me see, stop sharing so then I need to start sharing again and a window. I want to share this one.

B

C

The left you can see.

B

The definition of a project I'm going to run this is a very a relatively simple, proud job that, as an example um tests.

B

In this uh this context, I use github of the github client to check whether all members of the qubit organization have two-factor authentication enabled. So this is just an example where, in the upper body, you can see from line 466 until 478 that I'm downloading github client on the fly, because I don't have it available in the qbci bootstrap image which which I'm using currently, then I'm going to log in using this github token that I am going to attach um you see here and from the lines from 488 to 495 now or 2, 497.

B

Sorry, and then I'm going to call github api um for checking all all fetching all members who have two two-factor authentication, disabled and uh checking the length and yeah. um I think in general, you get the idea. So this is just a simple example on how to uh on how on how to create a project. You can, of course, uh do everything in this uh this script. What you want, um besides, of course, running running any binaries or whatever you have available in the image, um so, okay!

B

So I'm now I'm going to run this.

B

C

First, I'm going to.

B

B

The project ammo, so you see you have a conflict path where the configuration is, then you have the job conflict path uh where the job configuration is and by specifying the job name, which needs to be identical.

B

um I'm telling the system for or the mkpga tool for which proud job to generate the project camera so and last but not least, I'm just posting it into a temporary uh file which is called temp project yaml.

B

So I'm just going to show you the brow job, yellow so that you see that it's just a regular yaml file um where you can see, for example, that is of kind it's of kind, proud job, and you see the specification that uh it, for example, holds the uh the complete script which you can see here and um yeah. You can also see the attachments of the volume mods.

B

So now I'm going to run the project. So the first thing it is asking me, of course, from where do you want to mount your github path? So, as you remember, I guess, on the left side, you can see that the volume mount is pointing to hcc github and, of course, when I'm running it locally. I need to provide this so now I'm going to provide this my from my local uh directory, where I have stored my tokens.

B

So the next thing it is doing is it's starting to run.

B

um Maybe you we can see, make a little bit more room, um I'm going to go a little bit higher so that you can see how the docker run uh is set up like, for example, the attached volumes which are which are mounted and how the parameters are injected like so it simulates the complete uh environment that the project would run in and um yeah. This.

C

B

We can see, of course, uh this job has failed because of a token scope, which is not correct for this for this application.

B

So um I need to take another token, of course, or I need to pick another token that has the excess rights. So what I'm now doing is just running it again and applying the correct token with the correct scope, and then you can see that when it runs this time by the way, the downloads every time, of course, the github uh binary, which I, uh which I want to have and um yeah this time.

B

You see, for example, that there are no keyword or members which is two-factor authentication disabled found, so, which is what I was expecting, so this job should be ready to go okay, so this was a small demo.

B

So I'm going back to my slides, sorry for being so slow.

B

Okay, let's continue, this was the demo. um So what what does this help? Now um we want. Of course we have the. We have seen that we have a failing status of exit, one and a succeeding status of exit zero, and we have, for example, the automation component in prowl that is called tide that looks at all the prs and have either a status of success, or else that is a failure and um also looks at the configuration what labels it needs to have. For example.

B

Of course, it always needs to have the labels lgtm and the proof, and it doesn't have need to have any of the do, not merge labels.

B

um Now we would consider that the following project configuration uh would be existing.

C

We have optional.

B

False and skip report false, which, uh which effectively means that the uh results of the running power jobs were attached to the uh were attached to the job itself or to the github status itself in the github repository pr and optional folders would would make this this job required.

B

So this, what you see on the bottom left here is an example from a keyboard pr, where the uh top project execution pull keyboard has failed, and you see on the right that is, it is required, and this effectively prohibits the job from at first, for example, from from being able to get merged uh on the one hand, and also from proud to get picked up and to try to merge it automatically.

B

Of course, even if the, if tide was not configured, we still could, of course get it merged, also by an administrator. But that's not the topic here. um Okay, so the last um thing I want to mention is other use cases for prior jobs, which are, for example, um for periodic jobs. You have, for example, reportings that get updated on a regular basis. For example, we have reports for flake finder that are run hourly daily or weekly.

B

Then we create our nightly builds, for example, with periodic jobs and, of course, for post submits. We also have some a couple of examples. For example, uh we are currently working on creating releases from branches um on post submits and um uh yeah. We are, we are. uh We can build and push images, for example, to a container or image majesty uh and post submits, and also we can sorry uh build the published websites and post submits, like I said at the beginning already, I think but yeah okay, so I think that's it from me.

B

um So yeah, if you have questions I'd, be happy to answer them. So let me just unshare my screen.

B

Am I still here so okay, um let's see.

B

Coriolis cloud stack.

C

C

Hey folks, if you have questions about prow, please either post them in the chat. um If you want to ask them out loud, then say in the chat you want to ask them out loud and I will enable your mic.

B

Yeah just one one thing: I forgot, I'm sorry thanks for your time, and I hope this was useful for you.

A

Well, if there are no questions uh thanks daniel, that was a very interesting um presentation. I actually I if I may, may I ask a question um so just to clarify so the demo that you presented was about not necessarily emerging code. It was more about you know, kind of an administration task of checking the github repos. My question would be so how what else? What's the whole um spectrum of of things that you can do with power?

A

Are you doing everything, um or are there plans to do even more things or what's what's next, let's say beyond beyond merging code, which would be one thing.

B

Yeah yeah, like you said, a merging code, is still gating of merging code is just one aspect on this right, so um we are, of course, always preventing that bad code gets merged, but also uh we can of course uh have lots of other interesting um things like, for example, um just um yeah. Whenever you have a commit on this on on a certain certain pr or not on a pr on a repository branch, you can execute anything. You want on that.

B

For example, like shortly, we have created a job that um that would update um a.

C

Website, for example,.

B

After merge on a branch and yeah um or um yeah, maybe you can a little bit elaborate. I'm I'm not sure. If I completely understood the question.

A

Yeah, I mean, and let me actually also share the camera, actually, the the one. One point that I actually was uh the point that I had behind my question was to try to you know realize that proud was much more than at least what one has in mind. You know one new things testing on merging code.

A

Another aspect of this is might be. uh Maybe I would mention that there are repos in the cube.org that are not not. Everything is code, so um proud is also managing now. Things like the website, for example, or the documentation and doing things like verifying spell checks for.

B

Yeah sure, of course, you can run anything that that has some script, and that has some some image where you have the environment, where you can want to run uh your your binaries or or scripts or whatever you have.

B

Everything can can be put into a project and can be executed due by some action either be it emerge or be it a uh or be a a periodic trigger like like a crown or or like a like a like an interval like for yeah, like you said, for example, uh we are currently in the process of changing the user guide from for to mk docs, and we are currently we are nearly finishing um this migration and after that's finished, we will use mk docs to create the user guide and um just yeah update.

B

Whenever a merge to master goes in to user guide, the mk, the the user guide will automatically get updated. I mean the github pages will automatically get created and updated and yeah published so yeah cool.

A

Well, thanks. Thank you.

A

Well, um as there are no other questions, I see here in the chat, um a reminder that um in hash virtualization channel in slack for the kubernetes slack, uh feel free to reach out to daniel and ask any questions or any uh any other questions around you without the project. But there is a thread there in that channel for each session and one for pro, um so it just filled two minutes to the next session, so I think we will take a couple of minute breaks. Thank you very much. Thank you for your great presentation.

A

We will be back.