Jenkins Google Summer of Code Office Hours, 19 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Online Meetup: GitHub Apps Authentication & Checks API for Jenkins

Description

Recording of the Jenkins Online Meetup on Aug 19, 2020. Presenters: Tim Jacomb and Kezhi Xiong. GitHub Apps authentication support was released in April as a part of the GitHub Branch Source plugin. It is a huge improvement for the Jenkins community, especially for users using Multi-Branch Pipeline and GitHub Org folders. It brings us many benefits: larger rate limits, user-independent authentication, improved security and fine-grained permissions, access to GitHub Checks API.

Based on GitHub Apps authentication support, we have recently released an initial version of Checks API plugin and GitHub Checks plugin. The two plugins are about defining and implementing the checks API in Jenkins. By consuming this API, other plugins can easily create GitHub checks. Thus, any information during a Jenkins build like warnings from static analysis tools, code coverage, and test results can be directly shown on GitHub UI.

In this webinar, we will go through the benefits of GitHub Apps,how to set them up for Jenkins, how to consume the checks API in Jenkins plugins or pipelines and show you examples on how we used the API to report static analysis issues and code coverage to GitHub.

References:
* GitHub Apps Authentication Blogpost: https://www.jenkins.io/blog/2020/04/16/github-app-authentication/
* GitHub Checks API project update: https://www.jenkins.io/blog/2020/08/03/github-checks-api-plugin-coding-phase-2/
* Checks API Plugin: https://github.com/jenkinsci/checks-api-plugin
* GitHub Checks Plugin: https://github.com/jenkinsci/github-checks-plugin
* Warnings NG plugin: https://plugins.jenkins.io/warnings-ng/
* Code Coverage API plugin: https://plugins.jenkins.io/code-coverage-api

A

Hi all welcome to the jenkins online meetup uh today is august 19th and we have uh cajon team who will present a github apple identification and checks api integration uh just uh to introduce jenkins online meetup.

A

uh It's a regular online meetup organized by jenkins contributors, where we present various stories related to jenkins use cases, play games, development tools and, basically anything about the jenkins project and the jinx community, and we invite everyone to participate if you're interested to know more or if you want to join a live discussion and discuss your topics, and we are always looking for speakers.

A

If you want to know more about the online meetups yeah, there is a meetup page on meetup.com, so you can subscribe to the meetup announcements here and, if you're interested to participate, etc. We also have an online meetup page on the jenkins website. So here you can find information about what we do, how to participate, how to apply as a speaker. If you want to present something and again any topics about jenkins are welcome.

A

Today, uh as I said, we have a presentation about modern github integrations, so we'll have a presentation how you can authenticate jenkins to github, using github upon notification, um and then we will talk about github checks, api and support for reporting, static analysis and coverage reports to github checks, and if you have any questions uh during the presentation, please ask them using zoom connect. So you should see a connect button on the zoom panel.

A

You can ask question at any moment and then we will ask the presenters. You know after the presentation or maybe answer the questions it's in pronounce there.

A

Also, if you want to have a discussion about particular topics, we once we finish the recording, we will have of the record discussion uh where, basically, we will prevent everybody voice permissions. So we will have a kind of expert zone where we can discuss any questions related to the talk and maybe other jenkins use cases. uh Please remember about code of conduct and yeah quickly, just a big kind and uh again all these meetups are organized by contributors.

A

We will appreciate any feedback uh and we invite you to participate in all the discussions after the meetup. There is a guitar channel specifically for github checks, api project and we also appreciate feedback from meetup participants. So there is a feedback form. I will share link in the chat later and we will appreciate any feedback so that we could make our meetups better slides are already published. So again, I will.

A

Provide the link or in the chat okay, uh so I guess I can hand over the presentation to team and meanwhile uh yeah we'll probably launch the poll with a few questions. um They may help us uh to get more information about your big background and interests with regards to github integrations. So again we will appreciate the effect and team the floor. Is us?

A

B

Hear me: alright, cool, hey everyone, so I'm tim, um I'm a jenkins core maintainer and a lead software engineer at kanos.

B

um I've been working with jenkins for about the last eight years or so, and I've been maintaining it maintaining a couple of plugins for the last two years and doing its core maintainer for the last six months or so so today, I'm here to talk to you about github apps.

B

So this was something that I contributed to the github branch source plugin back in march or april, or so it allows you to instead of using a personal user token for a real user allows you to actually act as a machine user which the primary benefit to start with is much higher rate limits. So many people will be familiar with the 2000 rate limit and hit and hitting and having to wait for their jenkins to catch up, especially on busy um organizations.

B

So so, just as a um about the minimum gain you get here, is you go from 2000 uh calls an hour to 5 000 gates. Net thousand calls an hour uh which is really huge for a lot of places, um but there also scales with your organization as the organization grows, um so for an organization with a few hundred repositories from thousands you get up to twelve and a half thousand an hour, and if you pay, you have more money and you go into the enterprise plan.

B

You can actually go up to twenty thousand an hour as well, um so that's kind of behind the scenes um improvement, but for ease of use and onboarding you have user independent authentication. So you don't need to create an actual user. You don't need to create an email address. You don't have to have someone associated with it. You don't have to worry about. When someone leaves it's a machine user, you get fine-grained permissions management, so you don't have to just um give it right access all over the place.

B

Some people even give it owner access so automatically gets right access. You don't have to do that anymore um and a great benefit which gives will be talking to you about soon is access to the github checks api, where you can put the results from the from the job directly into the github ui.

B

You can get your static analysis results and you can rerun jobs just it integrates the github developer, experience a lot more and stops you from leaving having to leave github and go to your jenkins, um another small tip that we hit recently on the jenkins project infrastructure.

B

If you're a large organization, you should definitely change the rate limit strategy, so there's a default one which it's had for a long time just to normalize it over the hour, um the other one just the throttle at or near the rate limit, you're unlikely to ever get near the rate limit with github apps, but that the default strategy throws you way too early with github apps. um We were seeing.

B

um We were seeing throttling with 6000 requests to go um and changing this meant that we were able to complete an aux scan for 1600 plug-ins in under 50 minutes, um contrasting that to what we previously had with a personal users token it used to take about four hours to complete an org scan because of rate limiting. So that was a huge gain for us and just a very easy tick box change. If you are having any issues um on library organizations.

B

You can easily so one thing you may be concerned about is how do I use this in my current pipeline, you might call out to github to do things like add labels to pull requests. You might be using the deployments api to be registering where an app's been deployed to you may be pushing code, so you may be automatically detecting stuff in your pipeline and maybe automatically changing files or publishing releases.

B

um So good news is, it's very easy jenkins handles this, for you behind the scenes, um so forget how apple authentication, while not complicated, that's non-trivial, you need to sign requests with json web tokens. uh You need to have two different authentication tokens, a authentication, token and a an indian access token. So, while not complicated, it's something. That's abstract in a way for you and all you have to do is just act like this is a regular credential in jenkins.

B

So you see it there, that's just the standard with credentials, you even look it up as a username password and in your password variable you'll get the github access token and you can use that just as normal.

B

The example here is to making a check run if you're wanting to make a custom check run, but I'm using this to add labels to pull requests and to call the deployments api with no issue and we're also actually publishing changes to github as well, by changing the files in our pipeline and committing back and that all just works so that and there's a blog post which you afterwards, which has all this in it as well.

B

um So I'm just going to give a quick demo of how you set this up and just point you out some of the documentation and some of the resources available.

B

um So the first thing that you need is so this is using. It's called the github branch source plugin um it's installed by it's one of the plugins. That's suggested by default. So if you install your jenkins- and you just say, select and recommended plugins you'll get this automatically and there's a documentation guide here for github app authentication, it's on the front page of the readme, and if you just go through here, you get all the details that you need.

B

It tells you what permissions you need to set this up and all the permissions and all the fields that you need to fill out and just all the details and I'm just going to quit, I'm just going to relatively quickly just go through it with you. um So the first thing you need to do is you need to set up a github app oops. That's not quite the right place!

B

So I'm using this on my personal organization. It doesn't have to be an organization you can use it on your personal account as well. I'm just going to sign up a new github app going forward jenkins, um meetup demo, I'm just going to put a home page url. Just on my.

B

Github profile, I don't need to worry about any of that, so you need to set up a web hook, even if it's not activated- and I am using this locally- but so you just need to set a valid url. So again, I'm going to set this to my profile and it's not active.

B

Actually, I will just leave it active just to show you. um You need to pick the permissions that you need so there's some minimum permissions that you may need which are detailed in the guide. You may also want to add a few more on as well, so the minimum to use github apps is just commit. Statuses, read and write and pull request, read and contents. Read you'll quite likely want to increase that, though, depending on how you're using jenkins and github so and what I'm using is contents right.

B

I'm also using deployments freedom right, I'm also using pull request, freedom right and checks and for kishi's demo coming up. Is you also need reader right here as well? um So it's up to you! It just depends on how you're, using it it's easier to set up ahead of time, but you can also update it as you want uh trivially.

B

um You may want to set up some event subscriptions here. So things like pull, request, check, suite check, run push, um and you also get the choice of installing it on multiple accounts or just one. Most places only need it. They'll have it as a private github app and only install it on one organization, but you may have multiple organizations, especially if you're using gap, enterprise, so github enterprise, different teams tend to have their own organization, but with standard github.

B

You may or may not just be running on one organization and just take this any account. If you do need to install on multiple.

B

Okay, so that's the first thing here so github's now telling me that I must generate a private key. Okay, that's good! So let's do that. You can also go through here and see it like a logo and description information about it. So I've just downloaded that jenkins meetup pin key.

B

So the next step here is to just set it up in jenkins. So you go here to manage credentials and then you go here and add a credential you want to. You want to pick github app first things. Actually, so I want github app id. So this is just the jenkins internal id.

B

um The next thing is so. The next thing you need is the app id. So that's on the general page of the app called app id here and then I need to add the key, so I'm just going to get the one that was downloaded earlier.

B

um So if I just bring up the terminal here, um it's called jenkins meetup. So if I just cat, uh you can downloads jenkins, meetup him and I'm just going to put it onto my clipboard and if I just paste this in here.

B

And then I'm going to click test connection, so what I've seen here is that it didn't work so there's something wrong. If I click show details, it's telling me here that private key is not in the correct format. So there's an issue that github gives you the private key in pkcs1 format. um It's quite difficult: there's not there's no standard code in java to work with it. um There's some extra libraries, you can add and it's a little bit complicated or you can just convert it.

B

It's truly convertible, so we're just going to convert that to pkcs8 format.

B

So let me go here so my current key is this and it's going to be called new key. If I cat new key- and if I put that onto my clipboard again, it's done that and that's test connection again and there's another issue. This is a common issue.

B

You see that github doesn't actually prompt you to install it into your organization, um so couldn't authenticate. Has it been installed to your organization?

B

So you need to go to this install button here and click install and now you can choose. So this is another benefit. Is you can select here which which repositories you want it to have access to? The easiest thing is all but depends on your threat model and just how you want to see how your organization is structured and how your github is structured, but for the sake of this demo, I'm just going to click all and that's now installed. So let's try test the connection again. Okay, so that worked.

B

You'll also see this help text here. Alongside all of these, um tells you where to find it and help text here also tells you that you need to do the description, but I would just always recommend when you're setting this up just using the test connection button, there's an advanced feature here.

B

If you want to have a github app installed across multiple multiple organizations, you have to specify which one it's connected to just because of a internal implementation detail on jenkins, it's quite difficult to share that information, so it needs to be set on each credential currently, so I'm just gonna test commission one more time, it's okay! So let's save that and now for the invert, where we hope everything works.

B

So I'm going to call it tim j, org to github organization and see that was pre-populated um and I've just picked the credential and the github app has been verified. It's showing that's all authenticated. You also know this is 5000. If I was using a personal token, that would just say 2 000..

B

One thing to note is you do have to specify the owner here. It needs to be um even if you've set it to be a different one in the credential they both need to match. um But let me just save that and you'll see here, that it's connecting github app for 10. all and that's all going through quickly and you'll, see they're all starting to show up and trying to start build and that's all working. So that's um that's the end of my demo.

B

Thank you for your attention.

B

There's one thing in q: a: how can we raise pr in github using the hub progress command using github, app auth.

B

So raise pr and github.

B

um So I think I answered that um during the slides, so with with hub pr, you need to provide an api token in some way, and you just use that with credentials um that I showed to and inside of the closure, you will have a access token available to you and it should just work as normal.

C

Tim tim, I don't understand why I would want to do that. Can you I use the hub pull request for command all the time, but I don't understand why I would prefer to use the app the github app authentication token. Instead of my personal token, for a pull request. I.

B

Think there's some context of doing it inside of jenkins, so you've got an automated process that you know that changes some files or generates some files, and then you want to pull requests created for review thanks.

B

Cool, if there's no other questions, then we'll go on to kishi.

A

Yes, so just we proceed because yeah git hop up authentication is basically a foundation feature which enables many additional integrations in jenkins, and now we will see a few examples for user facing features which become available with github authentication.

B

uh We've also got one more question here: um instead of curl, is it possible to use powershell command eg invoke wrist method? Absolutely it doesn't matter. Curl is just the example um yeah, but all you need to do is just take the access token from the environment variable and pass it through to help the authenticate.

A

Okay, thank you tim. So, let's move on.

D

So I'll start sharing your screen.

D

Okay, let's see my screen.

A

Yep: okay, great.

D

So hello, everyone, I'm coach today, I'm going to share you with my gsoc project, the github checks, api plugin and I'm currently I'm an incoming graduate student at northeastern university and I'm now in the gap year, and you can find me on social media and github.

D

So first I want to introduce to the github checks api. So what is github checks? Api, as team has said before it's based on the github app and it's a highly customized way to integrate ci tools to make reports for pr and so for check. You have status, which now we are using this to indicate the current stage of a drinking spirit, and you got the conclusion and that annotations annotations are great, because you can make wordings and notices.

D

Sorry, you can make foreigns and notices on specific lines of a code and- and those annotations are just like the comments you left when you are reviewing other people's code and github, and so that is very convenient to the users. And the next thing is actions.

D

The extremes is kind of another extension point for github and since you can create many customized actions and the user, when users click on that button, a request will send you sent to the the integration, and you just implemented the logic in your integration tools and to handle that, and these are the- and these are the. uh This is a screenshot. I got from the github official document and you see the status and some other information here, and you can also say this is the action, uh the rewrite action I created earlier.

D

Okay, so next I want to explain our motivation for the project, and so first we need to take a look at how jenkins is is integrated now is through the github status. Api and you'll see a short message and the status here is it's implemented by the github branch source plugin, and so you can see it's very limited and and if you want to see more, you have to click the details and and you it will lead you to the ocean view of the build, and but we know that jenkins has many plugins.

D

For example, uh we have the warrants plugin, and so you can say many uh the wordings from different kind of static analysis tools, and you can also see the trend for those warnings. It helps you to maintain your project on code quality and next we have the coverage api, plugin, and so it gives you you can show your coverage different kinds of coverage for different files in a very straightforward way and so we're thinking. Maybe we can integrate github with them.

D

So, for example, there is a source code view from wireless plugin and we can migrate them to github, as the checker annotations, as I said earlier, just make this more convenient and straightforward for the github users, and so now we have development plugins like the general api plugin, and we developed this api plugin based on the github uh checks, api, semantic, meaning, and that we provide this general api because we want to be prepared for similar concepts in different platforms like like gitlab or bitbucket, and the next thing is: we provide an implementation for the github checks api, and so you can find those two plugins in the github repository and these two screenshots is a czech high made through our implementation.

D

You'll see the annotations this. This is just a a demo check, so maybe not no, not very meaningful. Just for sure, and we now have released both plugins. You can search them just type checks in jenkins, plugin market and you'll see both plugins and if you want to try this plugin and you just need to install both plugins iron jenkins and use the github app credentials for your multi-branch and github organization project and the support for freestyle project is on the way we have opened the pr and we will merge the features soon.

D

So after install that you will get uh the first one is, you will get this stage uh state indicators which can show you the current stage of the drinking spirit.

D

So next we use those apis in practice and first we create the varnish hex, which is powered by various next generation plugin and so in this plugin. uh If you, if you install this plugin, you can also get the wordings on from different tools to pick up and you'll get the quality gate. The quality is shown here as uh as a result of the check, and you can also get many statistics for the issues for different severities and you can also get the annotations. The annotations will just show in the along the code.

D

So it's very easy for you to say what happens to code, and so you can do your fixation very quickly and now this feature has been available down say so, if you want to take a look at the result uh before you try it, I just just go to the uh set that say that drink style and I'll show you some here.

D

I just this is just a good client plugin and it's hosted launching his organization. So you will get the wedding checks. For example, uh you'll see those warning checks and like java check style and different tools, and if you click details, you'll go to the check page for this check, and but I'm think one thing, I'm thinking is that since we just reported the result here, so I think many spaces many functions which we have just it. So I'm thinking better we can make it make greater use of this page.

D

If you click this, you link you go to the location view yeah, so it didn't a success because of the deployment. So the status here is as well and use. If you want to see the warnings here, they are just the same as those checks here.

D

So if you click this link, it also leads you to the build page.

D

And so the and at the end, but at the end of the day you may find there are many. Some may maybe many warnings that you don't care are for, for example, maybe the javadoc warning. So you can just skip.

D

Add the ski publishing checks in the pipeline and script of the world checks and you you can find more about the pipeline usage in warnings, plugins, documentation.

D

A

To so it will report the github checks by default. uh If you have a github check, cpf plugin installed and configured right. Yes, as a default feature.

D

And the next next thing is the coverage checks which is powered by the code coverage, api plugin and from the coverage checks. We get the coverage trend and the coverage healthy score and different kinds of coverages, and if I show you more about this, so this is a coverage check and here's a link to the reference view.

D

D

So it's it's my jenkins, so it just needs to the pr4. uh The last view the the last build of this pr.

D

If you go to the builder list, another view it's the sixth, it's a stake build, so it's just compared with the last build and the link will lead you to the reports directly.

D

And you can also disable those this coverage checks. Feature is also set as default feature, so you can disable them in the pipeline, and you can also find more about the coverage checks pipeline script in this link.

D

So and now we are developing more features. For example, we want to add a pipeline support directly from our plugin, so the users can publish checks directly in the python script without depending on a consumer plugin and, for example, our simple usages. You just want to create a check. You just said the name conclusion summary and but since we lack the user scenarios now, so I didn't think much usage of this, maybe maybe one one, the one I'm thinking is just to indicate the current stage of the averaging experience.

D

So if, uh if if one stage takes very long time, so maybe this is useful and you can find you can- I left your comments and give us some feedback in this feature. I I added a link here in the blue, so we are glad to receive your feedbacks.

D

And another uh feature is rerun request. For example, the users can request the rerun for field checks directly on github, and this is a github ui and if you trigger the re-run, the jenkins build will adjust. The course will show, as rewrong request by me through the github checks, api, and so the effect is.

D

So, for example, I'll show you a demo of this feature.

D

Okay, let me check the I'll, just click the re-run here and the github will give some feedbacks that you have successfully requested and let me see have we seen this I'm using a forward uh directing tool. So maybe it's very it's a little.

D

Decimal look at my.

D

Changes, okay may be disconnected or.

D

D

Okay, you can see in the console that received the rerun request. You'll get a checks, api and they say uh schedule will be wrong for for job and requested by me.

D

If you click yeah here it goes. Let's start it.

D

Because the link, if you go to jenkins yeah, I say I just figured to rerun, build here and.

D

If you can see the rerun request by me, through the checks.

D

Api, okay, so it just ends now so concluded as vineyard just as people, because I didn't change anything.

A

D

And we, this is also still a proquest, so you can give us some feedbacks in the pro request. I just dropped the link here. There we go.

D

So the last thing I want to talk about is uh to how to consume our api. So since there are so many options that you can configure for check, uh so maybe our uh the the consumation of our api is somewhat complicated, and so you just create a checks.

D

Details builder and I just add those properties you want to configure for your check and you just remember to stick to the github uh validation and validations, because uh if you don't follow that it will just fail the creating the checks so for, if you want to create a check like this, I just need to use our api in this way and you just uh here the details are just the properties of the check and at the last you just create a um just created the publisher from wrong, and you just invoke the publish method with the details behind all this progress.

D

A

Mean so this api is for plugin developers right.

D

A

So for pipeline steps, api would be different. Basically, what you presented several slides ago.

D

Yes, uh yeah the pipeline is, uh is support. Is here it's much more simpler than that.

D

A

D

And another thing is that I'll show you how to create actions. So should you make standard jenkins by adding your uh adding your customized actions uh if you want to create a button github? So when the button is clicked, a request will be sent to github app and you just uh with the actions and add two attractions.

D

The last the last parameter is identifier for the check, and so when you request, when the users click the button, the identifier, which is the last parameter of that method, will be sent back to that. Send back to jenkins.

D

For example, uh if I just listen.

A

So so yeah, obviously these buttons are part of github checks, api plug-in not of the generic checks, uh plugin.

D

Is correct? Yes, users can extend, can just create the checks and extend the function.

D

So if you click the format or.

D

And click the report, so you will so you say you see the payload here and you go to the chatroom this into here. The requested action is report, so just the identifier you said for the check, and here is because it's master branch, so the the progress is empty.

D

And so, if you want to and then you have to implement those actions, you just need to extend the github event subscriber, which is provided by the github plugin and just remember, to subscribe the check run event and event. I implemented event handling logic there. You can find more examples in the subscriber folder of github and you can also take a look at our our implementation for the rerun request.

D

Is still in, it still appear.

D

So it's all from my side ask simple questions.

A

Thanks for presentation- and if you have any questions, uh please ask in zomconi, we still have some time to discuss them more live, and after that we will switch to the discussion and yeah. If you have any feedback, please use the feedback form link in this zoom chat.

D

So one question look at that: uh there is a good nap uh and almost the same, but it's very limited. uh I guess. If you, uh the gitlab, is just the looks like the status api. So I showed you earlier. You can just leave the message and the result of the of the check and in gitlab I think it's called uh status commit commit status and I don't remember the name. This status comment.

A

Thank you and yeah regarding that, I guess this checks api. It's not limited to source control management systems and source code repositories, but basically you can integrate with other services.

A

So, for example, if you want to submit this data to data.org for notifications to pager beauty, etc, theoretically, you could use the same api if you want to notify them or maybe, if you just wanted to send a message to email again in principle, you could use checks api for that instead of existing command plugins.

A

Is it right or are there some of the any limitations.

D

uh The rerun will only show for the uh failed checks.

B

D

B

Also not merged yet so it'll be.

D

B

D

A

Okay, so I have a question for you regarding github authentication support. So in jenkins there are more than 20 plugins which integrate with github, but not all of them are supported half of our identification at this point. So if I need to add a github up, authentication support plugin. What would be the steps for me.

B

Yeah I'll answer this one: um you shouldn't need to do anything because um so those plugins will be using a username password credential in most cases, so uh it'll either be a actual username and password. Although that's getting deprecated by github in november, I think or it'll be a username and personal access token, um and so it's using their password credential, the github app credential, basically pretends to be a username password, credential and when the password is asked for it does the authentication behind the scenes, so it's transparent for both users and plugins.

B

So I'd be very surprised if any plugin actually needs to know that it's um that you're using github apps behind the scenes there is a you can specifically check if you do need to so. The github branch source plugin does need to change the behavior of a couple of cases, um but that's only due so certain apis aren't available to github apps.

B

um The get me in point, for example, um so the github branch source uses the get me endpoint to check um whether you're successfully authenticated um but get me doesn't exist um for github apps. So I think it uses the rate limit api instead.

B

But so you shouldn't need to know is the answer really: okay,.

A

Thanks and it should be a good advantage for users of existing plugins.

A

Okay, it looks like we don't have any questions left in the queue. So I suggest to thank speakers and uh to switch to the discussion. So thanks dean and yeah, we will publish a recording of this presentation uh uh later today. If you have any questions, please use uh the guitar link. Genki cs github checks, api and you can ask any questions related to what's. Authentication and checks appear there.

A

Okay, thanks all I'll, stop the recording, but if you have any questions or topics to discuss, please stay online.