GitHub Office Hours - Adopting GitHub Actions at scale., 11 Feb 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Customize self-hosted runners (w/security best practices) #OfficeHours

Description

Adopting GitHub Actions at scale in the Enterprise (Part 3/3)

For the full series: https://youtube.com/playlist?list=PL0lo9MOBetEEk9gIFox8EbCf1Co_4ppIO

0:00 - Start
2:03 - Intros and recap of the series
7:02 - Reference Architecture
10:01 - Enabling the web application firewall
13:18 - Testing with a simulated SQL injection
16:00 - Enabling Docker-in-Docker
22:02 - Creating a custom runner image
29:14 - Create multiple namespaces
46:11 - Configuring runner group access
50:35 - Deleting the resource group and everything in it

Demo repo: https://github.com/link-/actions-at-scale-ghoh
Contact Us: https://resources.github.com/devops-learning-journey/

A

A

A

Good morning, good afternoon and good evening welcome to the adopting github actions at scale in the enterprise office hours. If this is the first time you join us, allow me to introduce myself. My name is bassem grady and I'm a senior solution architect, part of the expert services team, my team and I work with enterprises and corporations across the globe to solve a variety of different technical problems. One such problem is figuring out how to scale github actions within the enterprise.

A

This uh series was composed of four events. During these events, we were gonna be tackling these questions of how can we deploy and auto-scale self-hosted runners? uh Can we use kubernetes for those? How can we manage our access to self-hosted runners?

A

We've gone through a big part of these or answering these questions in episode, one and two, I'm gonna tell you where we are right now in this series and uh we're gonna be answering so much more uh technical questions and inquiries and implementation details in these sessions episode one we basically set up kubernetes and we used aks for that. This is where we were gonna, be hosting our self-hosted runner setup.

A

In episode two, I showed you how you can install actions, runner controller on aks and how we can use it with webhooks and github apps and tls termination. You know to scale up and down our self-hosted runners to meet the demand of our workflows, and in this episode I'm gonna be showing you a lot more of more advanced best practices so that you can tighten harden and secure your environment while also customizing it for more specific needs that you might have previously. We had planned for this event to be four parts and episode.

A

Four was about me showing you how you can create a custom scaling solution for your for your setup. uh However, we thought that it might be better if we gather your feedback first and listen to you and understand what are the different types of problems that you might be facing, and then we can design an episode that will basically answer these specific problems, as opposed to us coming up with what we think your problems are and giving you hypothetical solutions for them. So we would love to hear from you.

A

Please make sure that you reach out to us and tell us what you would like to see in the fourth episode, and we will make sure that you have that and prepared for you today we're going to be talking about the advanced best practices and we're going to be covering. First of all, uh how do we set up and configure our raf web application firewall?

A

One of the main reasons why we adopted the application gateway is our ability to have a waf in front of our webhook endpoint, and the waffle will basically allow us to protect against a lot of a wide variety of malicious attacks that could, uh you know, hit our internet facing uh webhook's endpoint.

A

This is very important for enterprises and it's way more important if we are exposing this endpoint from our kubernetes cluster. So today I'm going to be showing you how you can protect that. The second thing is, I'm going to be showing you how you can install multiple action, runner controllers in multiple namespaces, because within enterprise environments we want to create isolation, sometimes in between teams and isolation between self-hosted runners.

A

This is a very critical requirement for a lot of corporations out there, especially the ones that are more regulated than the others, and you also want to guarantee that if you have a compromised runner that doesn't you know lead to compromising the entire cluster or the array of runners that are, you know in different areas. So namespaces will guarantee this separation and they're guaranteed this isolation. I'm going to show you today how we can leverage that for more security.

A

Next is we're going to talk about docker in docker and how we can use that, because right now, ourself also going to cannot run docker based workflow. So if you have a docker, build docker push docker whatever inside your workflow, they will fail because our runners do not have docker equipped in them.

A

I'm going to show you how you can enable docker in docker so that we can use you know, containers in our workflows and, lastly, I'm going to show you how you can create your own custom runner image, because, obviously you might want to install your own certificate bundles. You might want to install tools or compilers or, let's say, keys or secrets or whatever on the runners themselves, that you want to make available for your teams and engine and workflows.

A

So I'm going to show you how you can customize the default image and how you can embed whatever you need to in that container and, lastly, as as we usually do we're going to end up with a live q, a where I'm going to be taking your questions live and answering them to the best of my abilities.

A

Now, of course, these sessions are going to be recorded, they're going to be made available on demand, so one more time just sit back, relax uh focus on the content that I'm going to be showing you today and, of course, think about questions that you will be asking me towards the end of this session.

A

As usual, we start with the reference architecture before we dig into the implementation details and allow me to jump to my diagram. This whole setup we've already configured it. We have it in place, so nothing is going to change to it.

A

What we're going to be doing today is we're going to be basically working on the application gateway to enable the waf and add the necessary rules for us to protect our you know: webhook endpoint, that is exposed to the internet right, so we're going to be doing a lot of work on the application gateway level. Next, we have been operating within the boundaries of one namespace in our cluster and today we're going to be creating another workspace. I'm going to be showing you how you can uh install the action runner controller in multiple namespaces.

A

There are a few tweaks that you need to do before. We are able to do that and then we're going to be exposing another. You know endpoint through our english controller, for the second webhook configuration and I'm going to be showing you also how you can do the integration with a personal access token, as opposed to a github app, which we have used so far.

A

The personal access token will allow us to have runners configured on the enterprise level and then, if you have teams that are spread across multiple organizations that you want to supply the same runners for you can leverage this mechanism, uh you know to set up your self-hosted runners and have them available for uh teams across organizations without installing multiple github apps on the different uh organizations and having to manage them and uh yeah.

A

This is this is what we're gonna be doing today, so we're gonna be focusing a lot on our kubernetes cluster, as well as the application gateway to introduce these changes and without too much talking about the theoretical. Let us dig in, let's jump to our browser and as a quick reminder, there's a repository associated with this. With this video series, uh you can find it under my github account link forward, slash action, scale, dash gh office hours.

A

In this repository you will find basically all the resources- and you will see here that I have added more resources specifically for this session and I've updated the folder structure. You can find all the documentation that you need and description of what is contained in every one of them. We are gonna and, of course, the step-by-step instructions for everything that we are doing through the video.

A

It's very important to watch the videos before consuming this repository, just so that you can understand the context and what's really happening uh with every step of the way, I'm going to be adding at the bottom of this repository a set of resources for additional documentation, more tutorials, more documents that you can read for. You know further configurations, further changes that you might want to introduce down the line and for troubleshooting, I'm gonna be sharing this reference with you, so you can come back to it whenever you need to now.

A

Let us jump to our azure portal. This is where we left off. Last time we had the main resource group, github actions, runners in this resource group. We had the public ip. We had the v-net for the application gateway. We had our container registry, our application gateway, as well as our cluster and we're going to start by enabling the web application firewall on our application gateway.

A

So inside our resource group, we can just click on the application gateway over here and we're taken to the you know, landing page for this application gateway and the nice thing about it.

A

I like these visuals because they can show me, like you, know the different requests that were made, and I can start troubleshooting from here I'll- show you in a little bit how you can have more advanced views uh for you to understand, what's going on a little bit more on the application gateway level and for you to troubleshoot this at the organization or enterprise level.

A

So, first of all, we're gonna jump to the web application firewall section on the left hand side- and you can see here that we have started our application gateway with the standard. V2 sku we're gonna switch the waff on and we're gonna click on, enabling the firewall next thing we're gonna do is we're gonna change the firewall mode from detection to prevention. This will basically block http requests that are malicious or that the web application firewall things are malicious right. This is very important.

A

uh You need to be aware of the implications of this, so if you enable it in a production environment, please be aware this will block your requests.

A

Exclusions, we're not going to exclude anything and we're not going to inspect the request body, I'm going to keep it open, because also, I don't want to specify a body size. This is important because the web web hook, events might vary in payload size and they might be either small or big. It really depends on your workflow and what you're trying to do so, I prefer not to inspect the request body so that we don't block. You know false positives.

A

The next thing we're going to do is we're going to jump to the rules section and we're gonna define the rule set, and you have multiple ones, and these are continuously updated. You know with with more um rules to block or more rules to check against, and then, if you wanna, do more advanced configuration, like disable certain rules from being enforced, and this is important as you do some trials, you know back and forth checking what is really blocking the what is creating a lot of false positives versus what is really blocking.

A

You know, security threats. uh You can enable and disable them on a per item basis, and, of course this probably will enhance the performance of your web application firewall the last rules. It has to check each request against now. You can read more in the documentation for azure about what these rules are. What is the changes between os 3.1 3.2? There are some changes, but for now we're just going to stick to this and I'm going to keep everything to the default. Defaults are quite sensible and I am going to save my changes right now.

A

It might take a little bit of time for these changes to be applied, uh but once we they are, we will get a notification and I can show you right away how the web application firewall behaves. Brilliant you can see here that it has been successfully enabled if you go to our application. We can see this configuration applied right now and for us to do a quick sanity check. We can we can simulate this. We can do a fake sql injection, for example, on our url and see if the wife responds.

A

So I'm going to jump to my terminal so that we can simulate a quick spl injection attack and we are going to first pull the application gateway hostname and we are going to craft our curl command and we can do that with the following and we can append a simple. You know simple simulation of a scale injection we're going to encode the single quote with this and then we're going to say uh one four one.

A

This should trigger the buff to be to block the request you can see here we receive a 403 forbidden and for us to verify that this is just not the default behavior. I'm going to make a request to you know my test app and, as you can see here, because the request is okay, it goes through.

A

I get the proper response that I should be expecting, and this basically means that our web application firewall has been configured properly and it is uh you know, preventing or blocking attacks that are malicious, of course, in addition to the web application firewall, I really like the application gateway, because it allows us to have a lot of metrics and information about the requests that are coming in and it can prove allow us to measure so many different things.

A

So if you go to your diagnostic settings, you can add a diagnostic setting over here and you can see I've already added one and if I click on edit settings, you see that I'm already forwarding all my logs and all the metrics, and I am sending them to the log analytics workspace, um basically for storage and for you know for analysis later on. And if I go to, for example uh logs, I can run certain queries, for example, to see like the number of requests per hour, or maybe even the failed requests per hour.

A

And then these are some default questions that are granted for you and then you can also. You know craft your own uh queries based on you know different parameters, and here we're is querying for all the http requests that have a response code that is higher than 3.99, which basically 400 and above um this is a pretty cool. I highly recommend that you enable it. You can also configure alerts of course.

A

So if you, for example, you have different attacks that you want to keep an eye on, you can configure alerts for those, and then you can respond more effectively and block the attackers from you know moving forward. So this is the second benefit of using the application gateway. We are going to jump right now to talk about a different thing, which is enabling docker inductor so that we can use containers in our workflows and for that I'm gonna jump.

A

One more time to my terminal, I'm gonna clear my terminal over here and I want to show you a small. You know configuration file that I have already prepared for you, so that we can enable docker and docker we have already previously. If we basically list the folder, the files that we have, you can see in actions. Runner controller, uh if you do the following, you will see that you have the auto scale underscore web hook. This is basically what we have deployed right now or what we have in our you know default namespace.

A

Let me show you the content of that. uh Sorry, it's like! Yes on the scale of hook and we have just basically deployed the vanilla. You know deployment. We did not really change the spec or the configuration for the runner images, so we are not using docker in docker by default, it's disabled by default.

A

In order for us to do that, I have provided you with another configuration and it's in the same folder and the file is called dind underscore deployment.ammo, and if we, you know open this one, you will see here that I've added a couple of parameters. One of them is image. So now I am using the image that has docker in docker installed and set up, and the second thing I am enabling the flag docker within uh runner contrainer to be true, and I am going to deploy this as a separate deployment right. Why?

A

Because I want to give this a new label and I'm going to give this label docker, and this is another example of how you can have multiple types of deployments- different runner images. You know that are scaled differently depending on the needs of your team. So, for example, you're not all your teams might require docker and docker.

A

Then you can create a deployment with a smaller number replica. uh You know number of replicas uh so that you don't needlessly need a scale, uh docker and docker type of uh containers, so we're gonna deploy this very quickly with the following uh command. So, as usual cube cutter apply and then we provide the file and then we're gonna, deploy it to the default namespace.

A

And then, as you can see here, the runner deployments and the horizontal runner scalar have been deployed and if I do keep cuddle.pause.n default, I should be able to see here that I have the registration docker in docker runner currently being created, and now we have all the building blocks that we need.

A

So I'm gonna jump to my browser and I'm gonna go to my inner sanctum organization and then we're gonna jump to the scaling actions test repository and I have created a new workflow that I've already provided you uh with the sample workflow in the uh repository I shared with you a little bit earlier and it's called docker drop and, as you can see in this workflow, I am using the label docker and what I'm doing here is. I am asking this workflow to run inside the container right: node version 14.5 alpine right alpine version 3.12.

A

So I'm asking this job to run inside this container and then the next thing I'm going I'm doing is I'm instructing this specific sorry. I I missed one step. So the first step is, I am fetching the operating system, release right or the container image name or type over here, because I want to guarantee that I'm actually running inside the container and I'm not running you know inside some other uh inside the or the main docker container, which is the runner.

A

Basically, that's why I chose alpine because my runners are ubuntu based and they're, not alpine based. So if this returns alpine, that means that I'm actually running inside the container- and here I am testing another step- uh I'm trying to run it in another type of container which this time it's node version 12 and I really did not specify alpine and I want to see also the outcome of the operating system or the version of the the container image that I am actually using.

A

So this demonstrates that we are running inside containers and not necessarily within the main runner container, so I'm gonna trigger this job run and we're gonna see the outcome of that. I'm gonna jump to the docker job section, I'm gonna click on run, drop and run workflow and we're gonna wait for this job to be completed so that we can actually have a look at the outcome right now. You can see that this job is running and if we click uh on this one, we see that the image is currently being pulled. That's great!

A

That's exactly what you want to see! You can see here that the workflow has completed running and if we look at the step, initialize containers yeah, there's not much going on here. You can see that the images are pulled successfully run inside container, so the first job is running within alpine linux. As you can see here and then use another container, you will see that I am running another container here and it's running inside a debian uh distributional vaping image.

A

This basically is a quick demonstration of how you can use docker and docker to run docker based workflows. One thing you need to keep in mind is: whenever you use docker in docker, the pod will run with, or the container will run with in privileged mode. This means that it is running at root level. This is an acceptable security concern for certain teams or groups, but it is not acceptable for others. You should be aware of this and please read the action runner controller documentation to learn more about this at the moment.

A

There is no way around this, I'm afraid you can probably set up a docker rootless, but I'm not sure that you're going to get all the functionality that you are expecting if you run it that way. So this is the caveat that you just need to keep in mind brilliant. The next thing I want to show you is how you can create now your custom, uh you know runner image and how you can install on it certain custom tools and how you can use that to create a unique deployment with your own.

A

You know, custom runner images and we're going to do that by going to my repository here- and I already provided you with a sample docker file for how you can achieve this. So if you go to the folder custom runners, you will see that we have a docker file over here. If you click on it, I will show you what's going on so. First of all, we are starting with the base image, the summerwind actions. Runner. You can, of course, download this base image scan it for vulnerabilities for issues.

A

You can have a look at the content of this image. Also, you can rebuild it yourself feel free to do to take whichever approach you want to take and in this docker file. You know I'm just taking the easy path over here and then in this docker file. You can add. Basically, your ca bundle if you're using a custom ca. This is the perfect place to do so. If you have proxy configuration that you want to add.

A

This is probably also a good place to add this proxy configuration and then what I'm doing here, to give you an example that this is a custom runner is that I am installing go which doesn't come by default in the you know in the summer, wind or in the action runner controller base image, and I am installing go version 1.17 and then I will be running a custom workflow to compile a go, a small go program and run it inside this runner.

A

So this is how the docker file looks like what I need to do right now is. I need to basically build this image and then push it to acr and then change my deployment so that it pulls this image or create a new deployment that will utilize this image, as opposed to the default runner image and the way we're gonna do.

A

This is one more time I'm gonna jump to my terminal and because I'm already operating in a container here, I am going to jump to my regular uh machine because I would like to use uh docker, so I'm gonna, first of all, fetch the acr url and I'm gonna store it in a variable. Sorry, let me just jump to bash very quickly so that we stay consistent, because I personally use another shell. First of all, we're gonna fetch the acr url.

A

We can do it with this command az acro and it probably used it before it's very simple and then the next thing we need to do is we need to log in to our container registry. You can do that with azacr login, and then you provide the name of your container registry and azure. Let's see, azure cli will take care of the configuration on your machine. That's great and for us to do a quick check whether we are authenticated or not.

A

You can have a look at the you know: config.json file inside your dot, docker folder, and you can see here that I have the github actions- acr, sorry ohacr entry available, which means that I am properly logged in and now we are ready to build our docker file target and push it to our container registry. So, first of all, I just want to do a quick directory check.

A

This looks good and I want to make sure that I am in the root directory of the repository that I'm not in some sub directory, because otherwise the context will fail and what I'm going to do right here is I'm going to call the docker build command. So I can do it with docker build and then I'm going to tag it with the url for acr, and then I'm going to call this runner image and I'm going to give it the label sorry give it.

A

The tag, go 1.17.6 and I'm going to specify the docker file. Basically, in my sub directory custom runners, okay and we're gonna build this image right now, it shouldn't take a long time. um I already have this image built, so I have a lot of the layers cached. It will take a little bit more time on your machine and once you have this image built now, we can push it to acr by doing docker push and it might take a little bit of time to push all the different layers fantastic.

A

Now that our image has been pushed to our container registry, we can add, execute the new deployment or the new configuration for these for the set of runners- and I have already, of course created the configuration file for you. Let's have a quick look at it, so if you do cat and then the file go dash, runners dash auto scale underscore webhook.yaml.

A

You can see here that I've configured the image to point to my github actions, acr and then the proper image with the proper tag and I've also provided the image full policy to be always because, if you're trying to do tests, you would want your runners to be created with the latest image. Every time and don't worry, kubernetes caching capability does not pull the image every single time right. It just pulls it whenever there's a chain. Everything else is pretty much the same. We haven't changed that much, but we have also provided the label.

A

Go to these specific runners and we're gonna deploy this right now by again running a cube, cube cuddle, apply, dash, f, oop oops apologies. I need to run this from my uh container because I don't have the kubernetes context uh configured on my own machine. So again we're gonna, deploy the configuration from here and we're gonna. Do it as such.

A

That's great quick sanity check, get pods dash and default, and we can see now that we have the go runners, registration, registration, container or pod already being created, and now I have created also a sample workflow that we with which we can use to do a quick test on this. So I'm gonna go to my trusted repository and then we're gonna jump to actions.

A

I have a custom runner job workflow that I've created, let's jump to the workflow file very quickly, so that we can see what's going on inside of it and uh it's called custom runner yamo and inside of here you can see that I've only changed the label that I'm using and towards the bottom of this file. I am simply compiling you know, I'm sorry creating a file which is called hello.go and I am adding to this file.

A

Some go uh a small hello world go program and then, basically I am calling the go compiler to run this uh go code. That's all I'm doing! I'm gonna go to my actions, tab and I'm gonna click on this workflow and I'm gonna trigger a new workflow run and I'm going to jump to my workflow to see if it has been dispatched.

A

And, as you can see here, a runner has been assigned and our job has been completed successfully and if you see here hello world, that means that our workflow has been successful and we were able to utilize our custom runner.

A

You know to do something that is not available off the shelf, and this is how you can simply create your own custom, runner images and if you have specific, builds that you need to run or you need specific tools or you might want, maybe bigger you know, runners or install proxy configuration or custom certificates or whatever you need. This is the way to go for the last part of our episode.

A

I want to show you how you can create multiple namespaces and, in this case, we're just going to create an extra one in addition to the default namespace and we're going to install another actionrunnercontroller there and I'm going to show you how you can configure the application gateway- and this is the third nice thing about the application gateway, how you can configure it to point to the webhook.

A

You know node or runner or sorry a container of the second action runner controller in the secondary namespace, and this will allow us to have two controllers running managing the self-hosted runners within two distinct and separate and isolated namespaces, and I think this is a pretty much desired functionality that a lot of enterprises or a lot of you would want in your setup. So let's dig right in I'm gonna jump to my terminal.

A

One more time and first thing we're gonna do is we're going to create a new name space, we're going to call it alt, namespace or alternative namespace. Basically, the next thing we're going to do is we're going to do some editing to the values.yaml files and I've created a couple of you know samples for you or the samples that you can use.

A

Why? Because uh I will show you in a little bit uh when we jump to the to this configuration file. So let us jump to the actions, runner controller, multi-name spaces underscore values.yaml, and what's going on in this, I'm going to show you a little bit now what's going on in this file.

A

So I'm going to open it, and I will actually let me make this a little bit more uh distinctive by using vimbif and I'm gonna open this one and I'm gonna compare it to the values.yaml file that we have created the first action runner controller in the default namespace with, and it's going to show you basically the changes that I have introduced to make sure that we are, uh you know multi-name space compatible.

A

So the first thing that we are going to introduce as a change is, you can see here on this side the name override. So the name override is the first thing, we're gonna change and you need to append. You need to append a value to this. So when you install your action runner controller in a specific namespace, you need to put a value for this name override. Otherwise, your controllers, if you install them there, either the installation is going to fail or then again a misbehaved because they will clash in terms of resources.

A

So the name override will guarantee that we have a unique name for the. You know resources that we're going to be creating, and I'm going to call this one I'm going to I'm going to give it the name of my namespace. It's going to be default and then the full name overwrite is going to be default.

A

Actions dash runner dash controller, and that's it that's all you got to do in this section and then in the scope section you will need to change the single namespace value, and you can see here that I've edited the single namespace value to be true, because this means that the actionrunnercontroller will only listen to events in this particular or will only listen to events and create runners in this particular namespace and it will not take cover any other.

A

You know namespaces and then the second thing we're going to specify is the watch namespace, and here we are saying that this controller manages and creates runners and listens to events in the default namespace and no other namespace.

A

These are the second changes that you need to introduce in the scope section and the third change you need to introduce is in the github webhook server section, which is also introducing the name overrides so same as we did before. Just introduced the default name overwrite and in the full name of write you can give it the name default, github dash, webhook server.

A

Of course you can configure these names to whatever you like, but these are basically the only changes that you need to introduce and, as you can see, there are no other diffs between these files. Everything else is pretty much the same, so I'm going to close this and what we're going to do right now is we're going to update our helm, installation of the action runner controller with the multi-name space configuration and we're gonna.

A

Do that with the following, so home upgrade dash dash, install and then we're gonna, you know specify the default namespace and the file that you're providing this time is the multinamespace values.yamo instead of the values.yamo.

A

Please, please please make sure that you also make sure that your github app private key is properly configured along with all the configuration you've done in episode. Two don't forget those right, so this file is provided, but you cannot use it. As is you need to go and configure the different values that you configured in episode two before you can move forward, and in this episode I showed you, the extra values that you can configure.

A

You know for running a multi namespace setup, so we're gonna run this very quickly and it shouldn't take a lot of time for the action runner controller to you know, update or upgrade to the new configuration- and you can see here that we are good to go on that side and then the next thing we're going to do is we are going to install another controller and I've created another values.yaml file, but before we do that, I want to show you what's in that values diamond file.

A

So if we do a quick uh them actions, runner controller alt, namespace values.yamo and I've created a new folder called alt namespace, where I am going to include this file. Sorry, I have an example file, so let me first copy it and make it a non-example, so action running controller, alt, namespace and then values.example I'm going to remove that example from the end and then we're going to edit the non-example file.

A

Sorry values all right great, so here you will see a few changes. First of all, instead of using the github app configuration right, I'm going to be using a github token and I'm going to be using github token here, because I want to show you another form of registering runners and in this case we're going to be doing enterprise runner registration, which can be installed and configured to operate across multiple organizations.

A

So you install them once, and these runners will be available for multiple organizations that you can specify and you can create runner groups that span across orgs or you can create, add them to the default runner group, and it will be made available to all your organizations and repositories within these organizations, with the exception of public repositories, of course, and uh unfortunately, there is no way to install github apps on the enterprise level. Yet, but uh you that's why we need to use a github token.

A

You can see here, it's redacted, I'm gonna change that I'm gonna create a token right now and I'm gonna show you how you can create one and what are the privileges required and we're gonna update this value in a bit. The second thing uh we are gonna do is we're gonna, of course, make sure that our name overrides are different than the name overrides we used for the first namespace, and here we're going to choose alt and s, and then alt and s actions dash runner dash controller.

A

Everything else stays the same of course cluster ip for the service. This is the configuration we did in the episode two then.

A

Everything else stays the same. Of course. Single namespace needs to be true, watch namespace needs to be alt and s, and then, of course, the.

A

The github webhook secret token- this is also very important and then, of course, the name override and the full name. Overwrite need to be provided as the following or you can provide them with, whatever you know, name that you want and then, of course, the service for the webhook cluster ip as well. Ingress needs to be disabled because we're going to be using the api gateway as our english controller and that's basically it of course you're free to change all the other values.

A

Once you have a good grip or idea of what they are for now. This is sufficient. Let's go create the personal access token, so I'm going to jump to another account right now and I'm going to be using another organization, and this organization is called beirut and it's part of the poison inc enterprise. So I'm going to first of all go to my profile settings and then I'm going to jump to developer settings personal access token, and, as you can see here, I've already created a personal access token. I don't know why.

A

I call it inner sanctum. It should be beirut, but whatever, uh let's add our password over here and then you should land on this page and the privileges that you need to provide for. This token are as follows. So um I think I gave it a little bit more than it needs to so repo.

A

Yes, workflow, not necessarily admin org. Yes, you need to provide it admin repo hook. These are optional and then the main one is enterprise or admin enterprise right because it needs to register the self-hosted runners. And it's this one is, I don't think it's optional, it's required once you have this, I'm gonna regenerate my token and I'm gonna give it a seven day expiry period. I'm gonna also expire it manually before that, because now it's exposed and then I'm gonna edit my values.yaml file, to include this token. So we're going to jump to the.

A

Section over here and we're going to change these values and we're going to paste our new token we're going to save the file, and now we are ready. So let's install our second controller in the second namespace and it should be quite smooth as well.

A

Fantastic and how do we verify this? We can do a quick cube, cuddle get bugs dash n alt and s, and, as you can see here, we have our. You know: webhook server perfectly running. If you want to do a quick extra verification, we can look at the services and you can see here that the service for the web hook is also running perfectly over here, the webhook server, which means that we are good to go.

A

The next thing gonna do is our enterprise configuration and our enterprise webhook configuration, but first we need to also configure our ingress controller right. So for this I have uh a couple of changes that I need to make. So, first of all, let's have a quick look at the new multi namespace ingress configuration compare that with our latest english config, with the latest english configuration that we deployed, which is english tls-runners.yaml, and you can see here that I've just introduced a couple of small changes to the paths.

A

So in the default we had forward slash runners-scaler and now I want to create a distinction right, because I'm going to be introducing different rules and I'm going to have different name spaces. I would want to understand from the path from the url right. This web hook belongs to which namespace and it's probably a good practice. So, instead of having something as generic as runner's controller, now I'm going to be using stash the namespace and then slash runner scaler.

A

So in this case it's going to be slash, default, slash, runners, scaler and then the service you need to make sure now, because we are using a name override right. The name of the service obviously changes, so you need to put here the name of the service that belongs to your namespace and with this this is the initial change that we need to introduce. So let us do that. Let us apply this change in our default namespace.

A

First, we did not create the rule for our second namespace yet right, and we can do that with the following file. I'm going to show you very quickly: what's in it and same exact configuration right, nothing, nothing, changes the stack, the rules, everything is the same, and this is the nice thing about the application gateway. If you apply multiple rules to different namespaces, it will not create another controller. It will not create another application gateway.

A

It will just simply create a new rule in the same application, gateway instance from across namespaces, and this is how you do it just make sure that the annotations are proper. As you can see here, and then you can just specify the new path, and in this case it's going to be alt and s dash runner, scalar and, of course, we're going to be pointing to the service in the second namespace and deploying this is as easy as just applying the configuration file in the correct namespace right.

A

Please make sure you specify the namespace alt ns and when we do that, we can have a quick check on the ingresses to make sure that all our configuration is correct. So we can do cube. Cuddle get ingress english dash main first of all in the default namespace, and you can see here that our ingress is working. So, let's describe it this time- and we can see here that we have forward slash default. That's looking good, we have the test app, that's already running, that's great we're gonna!

A

Do a couple of curl commands right now to do a quick check and then uh we are going to now do a description on the ingress main, but this time in the alt ns namespace, and we can see that the path is properly configured and it's pointing to the appropriate service, and I really don't expect any problems over here. So let us do a quick sanity check by doing some curl commands, so we're gonna run curl, dash g and then http forward. Slash default forward, slash runner, scaler or is it runners?

A

Yes, we get the proper response and then this time let us change the default to alt ns, and we see here that we also get the appropriate response and for extra you know, for extra checks. We can go to our, of course, our application gateway. We can refresh this and we can have a look at the listeners and we should be able oops. Sorry, we can go, look at the back end pool and we should be able to see the three configurations over here.

A

So we have the alt ns github macbook server and we have the default github web book server and this looks really good. This is exactly what we want and now we need to configure the you know the hooks on the enterprise level for our poison inc enterprise.

A

So I'm going to jump to my github and I'm going to go to my org, which is called beirut, org and then from here I'm going to jump to my enterprise and from here I'm going to go to settings hooks, and here I can configure my um you know, web hooks.

A

Of course you can see that I've already you know configured one for testing, so we're gonna, delete this and we're gonna configure a new one, we're gonna click on, add webhook and then we're gonna provide the url over here, which in this case is gonna, be http.

A

I have some problems with my tls. That's why I'm not using it, but you need to use the ls, of course, because it protects your payloads you're you're, sending sensitive stuff here and then alt ns forward, slash runners, scaler right. This is the url and then we need to change the content type. We're gonna provide the secret and where do we get it from we get it from our? You know, yaml file, so we're gonna run the following.

A

A

And then we have the values.yaml file.

A

And, of course, the webhook secret is provided in the webhook section, which is going to copy it or better. Yet you need change it to something much more. You know secure than this, so I'm going to copy this, I'm going to put it here and now, instead of send me everything.

A

Don't do that because it will overload your cluster or your webhook server, we're going to select the workflow jobs and workflow runs and we're going to click on, add webhook once we do that, we can click on this one more time to see the events, and you should see the first payload delivered successfully.

A

If you don't see this as a success, that means you have a problem with your configuration, because the webhook server should respond with the appropriate message to this ping request, and now that we see this configuration done correctly, we are ready to utilize uh to create a new deployment that will create enterprise runners, but before we do that, we need to do one more thing on the enterprise level, which is to configure a little bit our policies and then go to actions, and we need to do one thing on the runner group level.

A

So under other groups, we have the default group by def that is created or that is available by default or off the shelf, and this uh default group. We need to make sure that is accessible by all organizations. This is very important. When you add enterprise runners, you need to make sure that these runners can be accessed by organizations, and you can control this by creating either new runner groups that you customize.

A

Whichever way you want and the action runner controller does not create runner groups for you, you need to create them manually or through the api first and then you can add runners to them by default. We're going to use this one and you're going to change the organization, access to all organizations or specify a small set of organizations that you'd like to allow to provide access to these run, to provide access to these runners that we're going to be creating right now.

A

This is very important right and then the next thing you're going to do is on the organization level. So I'm going to switch back to my beirut uh organization, this one we're going to go to settings and you also need to change in the actions section. If you go to runners, group or runner groups, you need to make sure that the default runner group over here has as or is allowed or is accessible by all the repositories.

A

If you don't do these two steps, you might you most likely will send the webhook events, but the runners are not going to be created and they're not going to be provisioned for your workflows or even the web book. Events are not. You know the workflows that you create and the repositories are not going to run. So these changes are necessary and once you do them, we can jump to our test repository in the beirut org. Sorry before we do this, we need to install the deployment right.

A

So, let's jump back to our terminal.

A

And let us deploy our our enterprise runway deployment. How does this deployment look like if we do a quick chat on this alt namespace and autoscale webhook.yaml file? You can see here that nothing really has changed. However, first of all, the namespace is alt ns and then the second thing is, I changed in the spec. Instead of organization, I'm specifying the organic enterprise name right now, which is poison inc and then I'm providing labels they're the same labels. Nothing really has changed and then also in the horizontal runner, auto scaler.

A

I am changing the namespace, that's it that's all you gotta do and then you can just simply execute this deployment, and when all of this is created, we can do a quick sanity check, also get pods dash, nltms and uh yeah. We should see this in our alternative, namespace provisioned, and now we are ready to do some trial runs.

A

So if I come to this workflow- and I pick single job- I just added one workflow here- we're probably not going to need more than that- it's going to scale and behave in exactly the same way and we're going to trigger this workflow run over here and now we're gonna, you know, show you a quick verification of. What's going on so watch dash, n two, you should be able to see that we're gonna have a new runner that will be provisioned in a bit.

A

Let's put this over here the first time it takes a little bit of time because it needs to pull the images and whatnot right. um If the images are not available, it might take a little bit of time, but otherwise the the runners should be hot and available. As you can see here, the runner has been added and I think the job has been completed successfully. No, not yet, let's see it's still cued.

A

Oh it's running, that's great, and you can see here that it has probably completed the one line script and now it is running the multi-line script. I have a sleep command in it, so once this is run, the workflow or the job should be completed, but already this is a good indication that our you know, trial has worked in beirut, org using enterprise runners, and that's it that's all. I wanted to show you.

A

The last thing I want to do before we wrap up is I want to nuke this whole setup, and this is the fun part of using the cloud with one command. I can delete the entire. You know resource group as well as all the configuration that is in it and then everything is gonna be back to where we started. This is everything I wanted to show you, let's jump back to the slides. If you have stuck up with me so far, you are the true heroes really like through three episodes.

A

We have gone through so much information, so much configuration so much setup that it is mind-blowing if you were able to keep up with. All of this, that's been going on. Of course, if you have prior kubernetes experience, this is all too familiar for you nothing out of not if nothing here is out of the ordinary, and now I'm gonna jump back to uh answer your questions live. Thank you very much for watching.

A

Wow, this has been a very nice series that have we that spanned across three weeks, I'm very happy that we've come to the conclusion of it. I really hope that you got the most out of these sessions and that the demonstrations were useful to you. I hope you were able to pick up new information, new insights, new knowledge. Of course, this part of the episode is dedicated for your questions, so feel free to drop them in the chat, and I will do my best to answer them.

A

I just want to mention also that I really love and appreciate your feedback. It has been amazing. Thank you very much for your honesty and, for uh you know the your kindness and your kind words, I'm really uh happy that you were able to get uh some value out of these episodes. I also want to thank uh some of you who have contributed, fixes and changes and enhancements to the repository that contains the instructions. Thank you very much for those of course, contributions from anyone are welcome.

A

This is an open source repository, it's made for you, so if you spot any problems while you're doing the setup or if you think that there's a better way of explaining or providing certain instructions, please do create a pull request and I will do my best to review it and merge it as soon as I can one last thing I want to mention before we wrap up. If there are no more questions, we at the expert services would love to take on. uh You know your questions and support you.

A

My colleagues in the solution, engineering team also would love to help you you know, set this up and also make it available for you in your own organizations. um The action runner controller is not the only way that you can scale actions in the enterprise and specifically self-hosted runners in the enterprise.

A

We do have another solution which is not made public, yet it's called elastic machines which allows us to have auto scaling, but for vm based runners and, of course, anyone who is any enterprise organization that is deploying uh you know that is doing intensive, ci, cd work or you're doing stuff beyond. Just simple automation: you're definitely gonna require beefier, bigger vms, so also this is a service that can be provided via expert services. Please make sure you reach out to us. The feedback form is now in the comment section.

A

I would really love your uh feedback on all of these episodes and please do make sure to reach out to us, for you know further instructions, further information to share with you certain best practices and knowledge that we have picked up from the field. If there are no questions for me, I'm gonna wrap up. This series uh definitely tell us what you would like to see in the future. I would love to come back with more. You know, demos or instructions or sharing with you just best practices.

A

So please do let us know what you would like to see and we will make it happen for you until then. I want to wish you a great rest of the day enjoy your weekend. My name is basim, and I will see you next time.

A

A

A

A