GitHub GitHub Universe 2019 - GitHub Product Updates, 14 Dec 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Advanced GitHub Actions: workflows for production grade CI/CD - GitHub Universe 2019

Description

Join GitHub Product Managers, Kayla Ngan and Edward Thomson, for a demo-packed session on GitHub Actions. After a short primer on advanced features, see how to deploy to GitHub Packages, auto-merge dependabot pull requests, and deploy a web service. This talk will inspire you to get creative with how you use Actions in your daily workflows.

About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.

For more information on GitHub Universe, check the website:
https://githubuniverse.com

A

Hi folks, my name is ed. This is my colleague kala today we want to talk a little bit about github actions. I hope that you've heard a lot already between yesterday's keynote with Jeremy the session between that the Jeremy did after that and of course, today's keynote as well so I think that I hope at least that you've seen a lot about actions so far, but we want to take it just a little bit deeper.

A

We want to show you a few more advanced things, a few more advanced scenarios that you can do with github actions, real quick introduction. My name is Edward Thompson I'm, a product manager for github actions, so I'm part of the team. That's that's been working on this really exciting product, so I wish the whole team could be here. Sadly, it's just me and Kayla for now, but we'll have some more people from the team available after this.

A

If you want to hang out and talk and get your get your questions answered, and if you don't, if you're, if you're feeling a little introverted or a little shy or you're overwhelmed with the number of things that you can do at universe, that's totally cool feel free to hit me up. You can find me on twitter. I'm e Thompson. It's Thompson without a P ETH OMS, o n happy to answer any questions that you have.

B

And I'm Kayla I'm, also a product manager on github actions and, like Ed, said you can find me after the talk at the hack space down, though so diving right in I'm gonna start by covering a few concepts before we go deeper. So I'll talk about triggers matrices and caching, so here I have a node app it's for a basic calculator and let's just set up CI for it.

B

So I'll go to the actions tab to get started and github actions suggest a couple of different templates for me to use sees that I have a note app. So let's just use that and template.

B

Taking a look at this template. We can see that it's going to trigger on every push it'll run on Ubuntu latest and then it's actually going to create a matrix of jobs for three different versions of node.

B

For the steps it'll check out the repository it'll set up that version of node and then it'll do the build and test.

B

So right now, it's configured to run on every push, and that may not be something that I want. I might not want it to run on every push. Every feature- branch, for example, so I can actually set up triggers to be more specific, so I'm going to actually update this trigger to run on pushes two particular branches, specifically master and release branches, and here I can actually use wildcards. So I don't have to specify a bunch of branches. I can say anything that matches release.

B

Slash I can also set this up for pull, request, validation, so I'll, add a second trigger and I will run this on pull requests into master.

B

Great so right now it's running a matrix across three versions of node I'm, actually going to update this and run a matrix across the three operating systems. So let's replace this with the three os's. We've got: Ubuntu latest Windows latest and Mac OS latest and then I'll update the runs on above to use the matrix OS variable.

B

And then we'll also want to replace the node version variable that we had using we're using down at the bottom and let's just use twelve points for now.

B

So one other thing that I can add to my workflow that'll help speed it up in some cases is caching, so I'm going to go over to the market place here and look for the caching action so I see here is the cash action. I can go view. The marketplace listing and I can go, get some more information about it. I can see, you know, who's contributed to it. The number of stars it has and I can view some documentation.

B

So I can see the inputs. It requires the outputs, an example workflow and here at the bottom. There are some other examples. So let's click that and we can see that there's actually a node NPM example.

B

So at the top here there's a snippet for Mac, OS and Ubuntu. You can see it takes the path, then it has a key. It hashes the file. So let's copy that snippet into our workflow.

B

Now, if you remember this was just for Mac, OS and Linux, so we'd want this step to be conditional. If the runner OS is not Windows. I could add another step here to do something similar for Windows, but that's actually kind of clunky. So you might have seen that below there's a snippet for multiple systems. So let's grab that and stick that into our workflow.

B

All right, so you can see what it's doing here. It's actually setting the cache outputs directory and then using that in the next step for the path. So let's actually go ahead and commit this.

B

We can create a new branch.

B

And then we will create a PR.

B

So you can see here in the bottom, the jobs are popping in. Let me see all three OSS and then we can go to the checks, tab and view the live logs, so it sets up the job it pulls in the actions that it's going to use. It checks out the repo sets up the version of node that we specified and then does the cache steps that we had and then finally it does the build and test.

B

So if we dig into these steps here, we can see that it set the cache directory and then, as expected, in the cache step and there's actually a cache that it's not found. This is the first time we ran it and then the end it actually saves the cache successfully for future runs.

B

So that's the first example pretty quick overview. The next one I want to show is how you can integrate with github packages.

B

So again we have the basic calculator app we'll go to the actions tab again. This time we will look for a docker template.

B

It's a scrolling down.

B

And there's the docker image on the right okay, so this is actually a very basic workflow. It's triggering on every push runs on Ubuntu latest checks out the repo and builds the docker image. So let's actually update this to also publish to get home packages.

B

So let's update the name here.

B

And we will update the build step as well.

B

So we have the name of my package here, my username, the repo app and then to publish to github packages we'll need to log in so let's create a multi-line script here and we will log in so docker login.

B

And then provide my username and then for the password actually, because we're using actions and packages together I can get the github token in the secrets context.

B

So it's super easy to integrate the two and lastly, let's just push this so docker push add the image that we create in that first step. There.

B

And then, to make sure that those versions match we're actually going to create a version variable and we'll reuse. That.

B

Great, so let's go ahead and commit that create another branch.

B

Create the pull request and again we'll see the job pop in and like before. We can go to the checks, tab and view the live logs.

B

So it set up the job, checked out the repo building, the image publishing a ticket a packages.

B

So if we skip ahead, we can actually go over to the packages for this repository and we can see that it was successfully packaged or published so now. I'm gonna actually hand it off to Ed and he's gonna go into some more examples: cool.

A

Thanks Kayla thanks, so much I'm really excited about the integration that github Actions has with other parts of the github ecosystem like github packages. Another part of the github ecosystem that I think is incredibly important, as I think you saw on today's keynote.

A

Is the security aspects of github I think that we have a real responsibility at github to make sure that your code stays secure and stays safe, and so one of the ways that you can do that is by integrating github actions with the new, automated security fixes that you saw earlier today, so I'm, actually gonna use the same repository that kayla was working on this sort of node J s, calculator application, and this is one that I've had for a little while I've been using this repo for a while and sort of organically I wish.

A

You know, I had faked this, but organically over time. I've had security issues in this repository. I've had vulnerabilities crop up in my dependencies. So if we actually go up and take a look at my pull requests and go to the closed tab and then navigate to this one, there we go so you can see that this is actually a a pull request that was opened by our automated security review process.

A

What we used to call dependable if we scroll down and open up this little commits, we can actually see exactly all the things that it that it did it upgraded, Milo dependency, actually several times, conversion 4.17, not 11, to 4.17, not 15, something a little safer, something that didn't have all of the vulnerabilities right. So this is actually emerged, pull request. What I want to do is go to a different repository, one that hasn't had this merged yet, and we can see what our security alerts actually look like.

A

You can see this, this yellow bar in the middle of my repository now. This is something only I can see as a repository owner. It's not that anybody can navigate to my repository and see that there's a security vulnerability, but it gives me a lot of visibility into the problem. So I can go, I can look at that and I can come over to the to the right hand, side and click view Security Alert, and that will actually show me.

A

The information I can drill down a little deeper and it will actually tell me exactly what's going on, I can click one more time and actually get to the pull request. So, unlike my previous repository, this pull request is still open and I have github actions set up to do. Continuous integration builds on this repository, so we've detected this security vulnerability. We've opened a pull request that fixes it automatically upgrading to the latest version of lodash and we've run all of these CI checks.

A

So we know that the that everything builds and that all my tests pass and now in a perfect world what I would do is is just click, merge, pull request and I have a high degree of confidence, because I've got good test coverage to know that everything's safe, but in a more perfect world what I might actually be able to get is github to do this. For me, it's gone so far as to create this pull request.

A

What if we could actually just merge it, and in fact you can, let's take a look at what that looks like so I've got yet another repository, and at this point I don't have automated security setup. I don't have to pend a bot set up in this repository.

A

What I do want to do, though, is.

A

Sorry, let me pop back to that.

A

There we go and now I want to navigate to my workflows there. We go and open up my workflow, and this is right now, just the default nodejs workflow that you've seen now a couple of times. You've seen it in several demos over the last two days. But let's, let's change this up a little bit so I can click Edit and what I want to do now is change the trigger from on push to be on pull request and.

A

I want to scope it just to a couple of branches, we'll just select the master branch, and you can see that that that very nice yeah no editing experience because I don't know about you. Yeah Mel's, not my favorite thing to work on having this autocomplete. This wizard is a big help.

A

Ok, let's scroll down a little bit and what we want to do is add a new job that will run after our build. So we've got that build job, but I can add a new one. Let's call it Auto merge we'll give it a name and.

A

Now, what we want to do is actually add some conditionals. We don't want to always run this auto merge. We want to be very careful. We want to scope it just to things that were opened by dependable and we can actually do that really easily. We can look at the github context. This is information. This is the same sort of thing that you would see in a web hook, but we can see the github context in our actions. Workflow execution, so I can select.

A

Neads right here, first we're gonna run after the build. This is this will run us sequentially instead of in parallel. We want to make sure that our build succeeds before we go ahead and do that that merge and then we can say if and we can see all the things that we can do. We can do a little bit of advanced execution, but what I want to do is look directly at the github context.

A

So I can say: github dot is master, so if it's a going into master and I can say if github not actor equals dependable, that's the name of the dependable user. So now this particular step will only run on dependable pull requests and then I can say, steps will do the merge and there are several options here. At this point we could use run and type some, but one of my colleagues, Jake LEM, came up with something incredibly clever, that I'm really excited about.

A

It's called github script, so we can say, uses actions, slash, github script, we'll give it a version, and basically what this lets us do is it lets us run JavaScript with the octo kit context and with the github context, so basically I'm ready to go in an environment where, instead of scripting with bash, I'm already authenticated, I'm already set up and I can just write JavaScript, so I can say, with this script and I can say, github dot, dot merge.

A

This is basically octo kit, for let's merge a pull request and I'll give us some more information and I can get this out of the github context. So we need the owner. We need the repo that we're in.

A

And finally, we need the ID the pull request, ID the number of the pull request. We want to merge. So that's the current one and.

A

Again, we get all this information, we get the basically the payload of a web hook, so we've got all sorts of information about the the action that happened. Finally, I need to add a github token and that's in our secrets.

A

That's github.com and so Kayla used this earlier. This is basically a very short-lived token that allows us to work on our repository. It's repository scoped when you fork it it's read-only, so it's it's very safe, very limited amount of time and very limited amount of privileges that this token has, but enough that we can in fact merge this pull request.

A

So we've got this red squiggle here where so. This is another nice thing about our gamal editor. It will actually do some linting until you when it's wrong I've missed a required key. So let's add runs on here. We're gonna run this on just a Linux image. There we go and now we can check that in so we'll, go ahead and check it right into the master branch.

A

There we go and if we go up to the actions tab. Oh sorry, let's go to the security tab. First.

A

And let's do that again there we go we'll turn on automated security updates. This turns on depend amat. So now, when I do a push or when I, actually just by turning that on that'll that'll start the security scan. So there we go now. I've got a security update. I can click on that and what it's doing now is it's, in the background, doing a creating a pull request to do that fix. So it's analyzing the problem.

A

It's gonna try applying the newest version of lodash, then the next four newest version, until we get to the absolute latest version, trying to make sure that they all actually work with my project and then it'll open up the pull request and github actions will actually do the test to make sure that it works with my project. So there we go and we can look at this pull request.

A

Let's dig in here and if we here's our pull request, so if we scroll down, we can look into the checks area and we can see that our CI is running. Actually our CI finished. So, let's go look at show all checks over here and oh, oh, oh, it happened really quickly. Our pull request actually already got merged, which is actually exactly what we wanted to happen. We wanted our CI to run our tests to pass and then github actions to merge the pull request for us. Let's take a look at what actually happened.

A

Let's look at our actions run its just finishing up its that top one right there. Let's click into that and so yeah. So we had a matrix, build on NPM, eight, ten and twelve on no date, ten and twelve, and then we did this last up the merge pull request step. Let's dig into that and open up the merge job there we go and that's just going to show us this github action script that we ran so very straightforward.

A

Only a couple lines of code to get us from the point at which depend upon is opening pull requests for us to us validating those pull requests by running CI tests to finally just saying well, if we have good test coverage, we can trust dependable to do a good job and merge those pull requests. So not something you want to do until you do have that good test coverage, but once you're there I think that this is a great way to just automate security from end to end.

A

A

One of the other things that I'm really excited about in in github actions is the ability to use containers as part of the build, so with github actions. We give you virtual machines that you can take advantage of. We give you Windows Linux and Mac OS VMs, and we and we've got a ton of software installed on them. But it's honestly not always enough. You might need some very bespoke sort of tool as part of your build. You may have some dependency. That has a very special installation requirement, or it's very large that you don't.

A

You know, want to download and unpacked every single time. So by using containers, you can scope exactly what you want to your CI system. You don't have to rely on the things that we give you so instead of my node app, that is a very simple calculator. I've got an even more simple node AB here. This is this is something that we'll talk to a Postgres database.

A

It doesn't do anything really advanced I'm, a I'm, a product manager, not an engineer. So, let's take a look at what it does do, though, I'm gonna open that up- and you can see here up top we're gonna, basically just connect to a Postgres database, we're going to read out of the environment the database to connect to the host and the port and then just gonna connect and once we're connected we're gonna. Do a really awesome query: we're going to select the now function.

A

Yeah it'll give us the current time, like I, said I'm, not an engineer. Here's the problem with this, though we don't actually have Postgres installed on our runners that we give you. So we've got all these these virtual machines, but not a one of them has Postgres on it. The nice thing, though, is that we can solve that by actually just using containers.

A

So one of the things that we can do is we can either allow github actions to run a container by specifying that in the build steps- and so when you say run that run context will actually be executed inside a container or we can bring what we call service containers to the party and that's a this is a great use of that. What I want to do is use a Postgres database as part of my test, but I don't really need it for anything else.

A

So I can just actually spin up a a Postgres container, just exactly the ones that I'd get off docker hub and use it for my tests. So let's take a look so, like I said, here's the list of software that's installed on the github hosted runners, I can scroll through here and scroll and scratch. There's a lot of stuff. Look at all those haskell versions, but if I, if I search what I don't see, is Postgres right, so I've got all this stuff. I've got dotnet core SDKs.

A

I've got several different versions of just 2.2, but yeah, not a single instance of Postgres. So, let's fix that, so here's what I want to do. I want to go to my actions. Tab and I want to set up a new workflow.

A

And I'm not gonna, take one of these out-of-the-box ones. I'm going to set up a blank, workflow I'm gonna check the set up a workflow yourself.

A

There we go and we can set up Postgres as a service, so I'm gonna navigate here above our steps and I'm going to add services. I'm gonna give it the name Postgres, and this is arbitrary. This is this is just a key, so I I'm able to use this to refer to it later. It's just the name doesn't have to be anything. Special I could have called it foo. But what is special, then, is the image so I'm gonna specify image, Postgres, 10.8 and what's gonna happen when I specify this.

A

Is that it's going to download this container image and it's going to start it inside my execution vm so, like I, said by default, I'm running on a VM I'll. Have this container there running side by side inside my VM, so I can I can take advantage of it.

A

If I were, if I specified other services, they would also be spun up on the VM and they'd be all networked together and, finally, I could even again use a container to actually do my run, steps to run my actions workflows and then that would bring yet another container to the party all networks together. So it's very straightforward to to set up actually complex, build and test environment that I can that I can use so.

A

After that, I want to provide some environment variables again. We need Postgres.

A

Postgres user and we'll set that to Postgres Postgres password, we'll also set that to Postgres because again I'm a product manager, not a security guy and finally Postgres DB and we'll call that Postgres as well once that's there, we're gonna, add some options to the container I'm gonna paste this in what this does. Is it just ensures that the Postgres database becomes healthy before we actually start testing against it and finally, we're gonna expose a port will expose port 5, 4 3 2 on the container there we go and what that does?

A

Is it Maps, basically, a port on the VM? You know if you're familiar with docker I. This is probably something that you're used to doing we're: mapping a port inside the container to the VM.

A

We don't actually know what port, though this particular syntax lets us we'll just pick an arbitrary unused port on the virtual machine, and you might think that that's kind of weird and I concede that it kind of is, but we'll take a look at what that. How that works in just a second. So.

A

Once that's done, let's come down here: let's get rid of these dummy steps and actually do some work. So what we want to do is run npm CI and then we want to run our little application, client j s, and we will set up some environment variables again, we'll set up Postgres, host and Postgres port, and these are actually used by our application to to connect to Postgres. That's that's! Actually in the java script that you saw now what port number do we use?

A

Well, what we can actually do is query the the data that we that that github actions has created as part of this workflow. So when we set up that port, we can actually interrogate it later using that kind of dollar, curly curly syntax, that you've probably seen it's not just for for the secrets variables or for environment variables. It's also for variables that github actions will set that you can take advantage of.

A

So in this case, I want to say job dot, services, dot, Postgres, dot, ports, five, four three two and that's, I concede a little bit verbose, but what does it actually mean? Well, it's it's hierarchical, so it's actually pointing up to our jobs. Up top so up, it will actually map eventually to this ports key right here: services, Postgres ports and then it'll actually expand out to five four three, two so or I'm.

A

Sorry it'll map to the what port five four three two on the container is mapped to in the VM, so it'll actually be something random like 32, five 64 or something okay. So let's commit this.

A

There we go and let's take a look at our actions, run.

A

Okay and then come down and look at this workflow execution.

A

Cool, so right now, what we're doing is the first thing we do is initialize the containers so we're basically downloading that Postgres container image and then we'll start it up we'll map the ports so that we can actually get to port five. Four three: two on the container.

A

Once the container is actually started again, we want to make sure that Postgres starts healthy will run npm CI, and then we will run our our client, our javascript, and that will actually do the connection to Postgres as our tests. So let's actually expand that out and see what happens well, yep. Indeed, if I look there and at my results I did select now- and I got my query back and so that's the the current time. So I think this is really great.

A

It's super straightforward to set something up that allows us to use dependencies that we don't actually have on our runners, but that we might need for our our CI build. So we do offer self hosted runners in case you need even more of a spoke things. You know, I know people who work in very or who work with software that has very strict licensing requirements and actually needs a dongle like plugged into a computer in the year 2019.

A

And if that's true, you can use a self hosted runner, but if you have less complicated, if you will licensing requirements, then you can take advantage of docker containers and still run them in our in our runners.

A

Finally, I want to take just one minute to talk about creating your own action. We've we've talked a lot over the last two days about creating workflows and running workflow executions, but what we haven't talked about is actions and I. Think that's really important, because that's where github actions got its name from actions are small, reusable components that you can use within your workflow that you can share within your organization or that you can upload on to the github marketplace and share with others. So.

A

So there we go so how many people have played untitled goo scam. Yes, it's my favorite new game. If you haven't played it yet, the the conceit is that there is a lovely village, full of lovely people just trying to go about their daily basis, and you were a goose. You are an angry goose and you sneak up on them and try to be difficult.

A

So my one of my friends, one of my colleagues here at github, has created an action that can that can help with this, and so, if I come to into an issue, and it's a lovely issue and I come in and I want to add a comments and I do that he's created an action that will run on issue comments.

A

If we look at the camel, we've got a trigger that will run on issues created and when that happens, it will run his action. It's called Jeff after Punk action. If we go into his repository, we can look at what makes up a github action. It could be one of two things: it could be a docker container or it could be a JavaScript application. The nice thing about using a JavaScript application is that it's cross-platform docker containers are only gonna run on Linux.

A

If you build an action using JavaScript, it can run on Windows, it can run on Mac OS and it can run on Linux. So in this case, he's actually written typescript and let's take a look at what it does. So we've got some imports. This brings in the github actions toolkit.

A

This basically allows us to use octo kit and a number of other. You know pieces of functionality that allow you to write cross-platform actions very easily.

A

The next thing we're going to do is actually inspect that github issue payload and so we'll see what happens. Basically again, this is just like what you would see when you have a web hook execution.

A

We can get the issue information. Then we can get the comment. What was actually posted? The issue comment that was written. That's comment body right here and what we're gonna do is see if the comment body matches honk. The word honk is in the comment body. If you are, in fact an angry goose, then we'll return we'll just stop doing anything at all. If you didn't say honk in the issue, we're going to delete your comment, so this is this is an angry, unfriendly action right here.

A

So again, if we go back to our actual, oh yes, our our comment was deleted two minutes ago by github actions, and it has replaced it with a honking goose. Well, that is very unfriendly. This obviously, is in a very useful action unless you really hate your workers, but it's just an example of some of the things that you can do with github actions you can trigger on any github event. It's not just about CI CD. You can look at issue comments.

A

You can look at when releases were created and ultimately you can use this github actions toolkit as a way to to take advantage of that. So if you are interested in using pieces of reusable components within your workflows, sharing it within your organization or sharing it with the rest of github, and you can go to github marketplace to look at all of the actions that are there, that people have contributed. I.

A

Think that that's a a great piece of technology to build. It allows us to really work together to enable community powered continuous integration and repository functionality.

A

Great, so with that, I'm afraid that we are just out of time for questions, but we will be in the connect space. Kayla and I will be there as well as some other members of the github actions. Team we'd be happy to answer any questions that you have.

A

If you want to come to us with your workflow problems, we'd be happy to see if we can solve them and again, if you, if you have other stuff going on this afternoon, that's totally fine feel free to reach out on a Thompson on Twitter and if you see Kayla or I around the event feel free to ask us. So thanks so much.

A