GitHub Demo Days, 22 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Friday Deploy! Integrating Terraform into your GitHub Actions with HashiCorp #DevOps

Description

2:11 - Program Start
3:04 - Intros
4:47 - Program for the stream
9:48 - Demo Start

Live stream replay with Kerim Satirli and Taylor Dolezal of @HashiCorp talking about Infrastructure as Code and Iterative Infrastructure with GitHub Actions.

Project link to follow along: https://github.com/ksatirli/iterative-infrastructure

Join us for the “What’s next for DevOps?” Panel on Feb 4. Experts from GitHub, Lightstep, Red Hat, and Redmonk will explore this topic! https://resources.github.com/webcasts/Whats-next-for-DevOps/

twitter: @ksatirli, @onlydole

A

A

A

Hey everyone welcome to another wonderful edition of demo days nice to see everyone up in the chat. Bonjour hello good morning, good evening, good afternoon, good night, nice to see y'all good to see everyone. I see some uh folks from russia in chat. I see folks from india in chat. I see folks from all over. Yes, it is it's definitely a fry ea live stream. Today we are, we are going to have an amazing day today.

A

uh I've actually got some really special guests for all of y'all today, so I'm going to bring in oh, I think I still hear some music in the background. Let me kill that yep. You know me always audio issues, but hopefully we're good to go. Bonsoir, hello, hello, yes, super pumped today, all right, let's bring in our special guests and I'll stop delaying all right, kareem and taylor, our friends from hashicorp, welcome, and thanks for coming on to demo days here today.

A

Thank you for having us yeah. Well, why don't you introduce yourselves and like tell tell everyone uh why we brought you over here from from hashicorp to our github demo days, absolutely.

B

So my name is kareem satirily, I'm a senior developer advocate at hashicorp, where I focus on the infrastructure line of products, so terraform packer and a lot of github repositories. We do fun stuff. I think- and I'm here with my colleague, taylor.

C

Hi everyone, my name is taylor, dolezal, and I am also a developer advocate at hashicorp. I'm also focused on infrastructure, so again, terraform packer and the like. I also focus a little bit on application delivery, as well so working with tools like waypoints and making sure that your your clusters are operational and running are some of my favorite things to do, and uh we were asked to come here today because uh sounded like somebody wanted to do a friday deployment and uh obviously cream, and I kind of jumped at that.

C

You know we had a shortage of that uh being developer advocates so uh sounds like fun to me.

B

Yeah I gotta say if the deployments are as good as the background music. This was. This was beautiful.

C

I was gonna say listening to that. I'm surprised that no one has come out with a. I know there are those several permutations of uh chill wave like chill wave listening to music with all of the different uh title cards, for those surprised that no one has made one with octocat just yet, but uh you know maybe maybe soon.

B

See you're putting ideas in people's minds. I like it so we'd like to talk to you about iterative infrastructure and specifically that really means that we want to show you a couple of slides really quickly just to get a base understanding, because we see so many people in the chat. You all look great today, um most of you anyway, some people ah nah, I'm kidding they'll, look great.

B

C

I mean it's early.

B

C

B

So thank you for having us um yeah. I think we've done intros. So let's talk about infrastructure as code and do you want to do this? One yeah absolutely also perfect. Let's do it.

C

So in today's session, uh before we start diving into some code, let's talk about you know what what are we actually going to be doing.

A

C

So we're going to be talking about infrastructure in the it sense, so servers and the applications we run on the servers and the configuration that's needed to run those applications.

C

More specifically, we're going to be approaching infrastructure from a code perspective, so simply put infrastructure as code or iac, as you might have referred to it or heard it referred to, as is the process of managing and provisioning resources with machine, readable definition files. So what does that mean? That means to that? Instead of manually going and configuring things via ui or an admin dashboard or clicking buttons, you're going to be setting that up with code.

C

So when we talk about resources, we're going to be talking about virtual machines, security groups, network interfaces and software defined storage options.

C

When you think about infrastructure as code, you can think of it as executable documentation, it's human readable, but you can execute it with an application like terraform infrastructure, as code is an enabler for many good things of which I'm very excited to show. You today provides a codified workflow so that you can commit your work and version it and then because it is code everything you love about version control also applies here as well.

C

You can collaborate with other engineers. You can roll a change back, submit a new change and easily spin up an additional set of resources by using things like terraform modules, for instance. Most importantly, because everything is written out, you can spin up those resources in a safe and predictable way, which again sounds really great for our friday. Deployment here code is obviously a very big part of infrastructure's code, it's literally in the title, and there are a handful of languages that you can use, such as json and yaml.

C

For for today's session, we're going to talk about hcl or the hashicorp configuration language. The hashicorp configuration language or hcl is a common language, supported by all of our tools, but rather than talk about hcl. Let's take a look at what it actually looks like.

C

So, as you can see here, we're going to pop up our terminal, we've got a service of type http.

C

We gave it a name of web underscore proxy we've defined a listen address for localhost and then a process called server which runs a specific command, as you can see, hcl is both human, readable and machine readable, and this syntax is something that you can pick up in a few hours and use across all of the other hashicorp products, which is really helpful. If you want to jump into nomad or packer today, we're actually going to show packer using hcl, which I'd prefer to json.

C

Personally now, I know that that was quite a lot in a very short amount of time. So here are a couple of dog pictures for you. Take a breather take in the sights of these these uh these wonderful animals. Here uh some of my favorite pictures of some of our favorite dogs.

B

I mean they're literally our dogs, which is why they are so amazing. That's.

C

Why they're the best.

B

So while I get set up for code stuff, let me quickly switch my screen off, so you can look at us if you have questions feel free to leave them in the chat.

B

If you don't feel like leaving questions in the chat or the great ideas come to you right after the session, which always happens for me, we're only dole and edcasterly on twitter and pretty much everywhere else. um So if I haven't responded to your pull request, just feel free to call me out. It's totally fine.

C

Yeah most people say don't at me, but kareem is like no go ahead. Just just do it. Oh.

B

You absolutely should so quick check. We've got our editor up and aj if you can get that into the stream. Can you see this?

B

Basically, what you should be seeing is an editor interface on the left side, with some code and a terminal interface on the right side.

C

Fantastic that looks good to me. The I think the terminal, the terminal on the right might be just a tad just a hair, too small, but uh besides that, I think everything there.

B

C

B

Updated so when we talk about infrastructure as code, we obviously talk about code and we want to show you a few cool things with terraform and since it's friday, the first thing we're going to do is create a github repository and to show you that this is all live. Let me open safari and we have the demo organization here, as you can see no repositories.

B

So, let's switch back to our editor. What you see on the left side, these eight lines of code is terraform same code type. That taylor just showed you here we're grabbing a provider specifically the github one, which makes a lot of sense right. We want to start out with a repository, maybe some team, memberships and we'll define that if you've never used, terraform totally cool you've probably used something like npm or any other package manager.

B

Think of this, as your package.json, we define the versions we define what module or provide in this case we want and we go from there, and so this is already part of the first terraform workflow. You define a provider you run init and on the right side of my screen. You'll see that terraform is actually grabbing the provider from the public registry.

B

Now, while that happens, let me switch to a different file.

B

C

You see here, and it looks like we're getting a couple comments on zooming in just a little bit more.

B

Absolutely on the code side.

C

uh uh Correct, I think both could side and awesome. How does that look for everybody.

C

It's funny that, uh with uh you know, with everything that's going on in the world and being on zoom uh when people refer say zoom in the chat, I'm not sure if they're uh on the zoom call or.

B

It's been uh yeah here we go we'll make it a little bigger. The good thing about this session is that this is actual real life code. This is not just an animated gif. uh So afterwards, we'll share the repository url with you, and you can look at it on your site, which is even better because that means you can try it out.

B

So back to our code, we've grabbed the provider which you can see on the right side of my screen. Everything worked out. Everything was downloaded, I'm going to go ahead and clear my terminal to make it easier to see. What's going to happen next and back in the code, editor we're defining our code. Here, we've got the demo organization, that's one.

B

I just showed you so here again, no repositories just tailor myself in there, and so, let's start out by making sure we actually have some admins in there taylor, you feel like being an admin today.

C

You know, uh I don't think I'm doing anything else this afternoon, so that sounds like a sounds like a good job. For me.

B

Beautiful so you're only doll on github, I'm at case of dearly on github, so I've added us to our organization admins.

B

It's a list in terraform, again really simple, syntax, so we'll grab that and we'll throw that into a membership resource which I'm going to call admins and I'm going to iterate over everyone here and just say: let's do this: let's first do a plan to show that we're actually going to be doing a little bit more.

B

But if you notice terraform already understood that it doesn't need to do the membership because we're already in there and we've obviously previously tested this.

B

So while we skip this, let's go and create a repository which really is the coolest part.

B

So, on the left side, just like we showed you before another terraform resource this one of type repository and what's really cool about this- is that in we've got it written out over 26 lines of code, but you could probably have it in about five or six lines of code. You can codify all your github repositories.

B

Now, if you run a personal github account, you could say you know what that's too much work. Five lines of code is way more time than it takes me to just click in the ui, and I get that if you're doing this for an organization, this is a lifesaver and I mean because we're live here. I just got to say: jeremy kudit who's working on this provider with a couple of other contributors.

B

Thank you. So much for work. Github is absolutely my favorite terraform provider. Until it will prove that it, it is literally. The first thing we do is go in set up the organization set up the repositories and make sure everything is nicely codified, because if you have repositories, then you can do all the good things that you can do with iac.

C

B

So it's really.

C

It's really helpful, too, being able to codify your repositories out like this. Typically, a lot of patterns that I've seen used in the community have been people creating kind of a repository to spin up their other repositories with infrastructure. As code, you do typically find yourself in a situation where you might have you know chicken and eggs situation where you have to unfortunately manually create. You know that repository uh the first time, but then, once you do you set yourself up for a lot of success in the future.

C

Being able to you know, add repositories and then configure things in in a very broad way. You know if we wanted to make all of our repositories private, we could go ahead and do that public. We could go do that and then this also allows us to check to make sure everything is all right in a pr process. You know collaborate with others make sure everything looks good and then you know recently github allowed for us to change from master to main. That makes this very easy to do with infrastructures code as well.

C

So I'm a huge fan. I can't I could probably talk all day about the github provider specifically, uh but we we have other things to talk about, but uh very helpful.

B

We definitely have other things to talk about, but um one thing that you really love: if you're, that kind of person uh you can disable and enable various merging strategies right, I'm a huge fan of sports merging, mostly because we do a lot of exploratory commits, I think, is the pc term and by squashing them nobody ever gets bothered by all the craft that happened.

B

So we've got a repository. I think all looks good on the right side we can see. Terraform on plan tells us that one research is going to be added, which is absolutely correct.

B

So let's go ahead and apply this.

B

And so just to be sure, terraform verifies once again with us. Is this really what we want now in the sense of a github repository, there's no cost attached right. Imagine you're doing other type of infrastructure like a virtual machine, or maybe you know a gpu heavy instance.

B

You might end up spending a lot of money if you confirm this and don't actually want that type of instance could happen, not what we want. So in our case we're gonna just confirm and wait for the api to do this takes about 10 seconds, which is beautiful.

B

Yes, that's five seconds longer than you could click in the ui, but I also have it as code now and.

C

Again, I think the five seconds is clicking the button and then the other five seconds in the ui is typically the waiting for your repository to load up.

B

And figuring out what to call it right. So, as you can see our first terraform deployed repository beautiful we've got one for packet images and that actually takes me to one of the questions I saw pop by in the chat. I don't know if aj can actually find this one back, but it was about code organization and how you would normally organize code exactly that one wow, that's beautiful.

B

So the reason we have all of this code in one repository is mostly to make it easy for you to download it and to make sure that we can depend on the other repository or subdirectories in here, depending on your organization. A monorepo approach might not be the best version or the best way of doing this.

B

I myself like splitting these things out because it makes the blast radius smaller, but I know everyone has different approaches to that.

B

So taylor, what do you prefer monorepo or just I.

C

I really do like breaking it out. I'd say you know working with you uh on this demo and and this code it was definitely different. uh Typically, I don't. I won't follow a monorail pro approach and it became interesting when we integrated. You know github actions and some other things around this project, but I have seen a couple: people set up their code structures such that they'll create their base modules or things that they're testing.

C

You know some modules they might be testing and then breaking out the configuration for different environments into different folders uh and then just kind of importing that in again using local files or modules. I I personally really like breaking things out, like you said, into different repositories, which allows me to then have different terraform states and not you know, break things as I as I set them up. You can get into a bit of a problem if you start to reference state of other. You know repositories things like that.

C

So it's just one good thing to be mindful of. If you're, in that kind of situation, you might be able to leverage tagging and other ways and data providers within terraform to start to reference things and look things up, which is the best way I've found around that. But typically I will like setting things up in a microservice type approach and uh and then it makes a little bit easier to kind of like focus on that specific thing. Instead of worrying about breaking some other dependency or something else within a monorepo.

B

Think that's actually beautiful right, like splitting it out like that. You just limit your blast radius and especially, if you're new to infrastructure as code in general. If you're new to a language, the less you can break the better.

B

Obviously, all these tools, all infrastructure as code tools, try to make it easy for you to not break things, but at the end of the day, if you're gonna ask your application to deploy 100 servers, it's going to confirm with you and if you hit yes, then there you are.

B

So let's see, if there's any other questions that we missed, I think not cool. So, let's move on from our terraform code for github repositories to one for azure, so we've created a repository for our pack images. Of course the logical next step would be to test it in github actions, but we first need to actually be able to have some code to run it.

B

So what we'll do is we'll use terraform once again and if you're, following along with the repository, I don't know if we can get a url for that shared in the chats or, if we'll share it later sure that there's going to be a way for you to find this.

B

We split this into a different directory again for the same reasons that taylor just mentioned.

B

So what we're doing here is we're instructing terraform to download three providers, specifically the azure provider for azure cloud local provider, because we're going to be doing some local stuff on our file system and the random provider for reasons I'll get to in a second.

B

And so, if we switch to this one.

B

On the right side, I just switched my shell to the azure directory, which has the code that you see on the left side. So specifically, these three files, and I'm just going to run a plan to show you what we're doing here and give you a basic idea and so right away. You notice that we have two random things. Popping up.

B

One is a random pet dinosaur, which is obviously the best possible pet that you can get and the other is a random string, and so you might be wondering why do we need those- and that's actually just realize- that's no longer part of the demo, because we removed that a little bit so we're leaving the dinosaur in there because it was so beautiful.

B

This is the beauty of doing things live. Sometimes you come across code, that's dead.

B

We need to up our pr game, make sure we don't have this in there.

B

So, moving on from that we're just actually going back to the one resource that we define for packer to be able to build images and for us to actually need a repository to put our packer code in and then run actions against. We first need an azure resource group, I'm in amsterdam, so I built my stuff mostly in west europe, we're calling our group packer just to keep things simple.

B

You can call it whatever you want as long as it makes sense to you. We could obviously use our beautiful random string that we just generated, so let's actually go ahead and use that and what you see here. We use this with storage accounts.

B

So what we initially did was grab a random pit and then added a random string to it to make sure our resource. Our storage account always had a unique name, because these have to be globally unique.

B

So if we go back here.

B

B

B

B

We've just updated our resource group name from packer to packer dinosaur, eh w2 and we're just going to see how packer responds to that. Sorry. How terraform responds to that, and you can see because we've deployed this group before terraform, actually knows that we are indeed creating a new resource.

B

Resource groups can't be renamed, so in this case they need to be replaced, which is what we're seeing here now for the purpose of the demo.

B

It doesn't really matter so we're not going to go ahead with that, but one of the cool things is that we also write that information from terraform in a format that packer can use and we'll get to that in a second and so essentially, what we're taking is we're telling terraform dynamically grab the reference of the resource group and then what we want is the location and a name and write those to a file and that file is terraform, underscore data in a format that packer will automatically load.

B

So let me show you how that looks like and you'll see here very simple. We have a key value format. There's nothing special about that right, except for the one thing that if I actually.

B

Apply this change, which might take a second.

B

Because everything is live, you'll see that our template file will get edited and updated with the new resource name, and the beauty of that is that we're now driving packer configuration.

B

On this approved now driving pack configuration from terraform's knowledge of our infrastructure.

B

And in the meantime, you can see on the right side. The terraform is cleaning up our old infrastructure by removing the resource group and it's going to add a new one, and sometimes that takes a while because we have a couple of resources in there.

B

But while we wait for that.

B

C

One one thing I did like too is that, so, if you, if you notice, when cream, did the plan and the apply when you got that feedback from terraform we're using terraform 014, which is a the latest generally available version that we have right now and with that came concise diff? So if you haven't seen that before it's really a great feature, it eliminates a lot of searching for what changed you know or what's going to change when you're running a plan.

C

So as you see that you can see you know, it'll say three resources that not changed have been hidden or three attributes not changed have been hidden. So if you do change, resources using terraform that have you know like a helm chart is one thing that has a lot of information.

C

um You don't want to see the things that didn't change every time. Concise diff is really helpful on that front. So, if you're not on terraform zero 14, uh that is just one of the many fantastic reasons to to bump to that version. If you can.

B

And while we've been trying to give you interesting and relevant information, we're still waiting for the resources to be destroyed and I had a quick look in another window. The reason it's taking a little while longer is that in that resource group, the packer resource group, we have a couple of resources that are now also being shut down as part of the removal.

B

We're not going to wait for that, because nobody has time for that. It's friday. I know you want to see something more interesting, we're just going to let it run in the background and in the meantime, we'll go to our packer interface and just making sure code size. Can we all see what's happening on the left side, approximately.

C

I would say bump it up just two more increments than it might be.

B

C

Much better much better.

B

There we go beautiful so as taylor mentioned hcl or the hashicorp configuration language also works with with packer.

B

So, despite being in the same repository, we've actually switched an entirely different tool where terraform deploys infrastructure for you in the broadest sense of the world word also in the broadest sense of the world. Definitely servers databases, dns records, packer focuses on building images, specifically machine images.

B

You can do that for azure, aws, google, but also for a couple of other providers, and so in today's image. What we're going to be building is what are we building today? Pretty sure it's an ubuntu base image that we are just updating with the latest changes, and let me find my code for this.

B

Yeah definitely ubuntu server. We've got all of this in the repository for you, so you can work work along. None of this ends up costing a lot, especially if you have a new azure account which comes with a couple hundred dollars in free credits.

B

So definitely give this a try and we'll start out with our image definition. So you might be wondering hey wait. I see you all define everything statically, that's not cool! It's going to take a lot of time!

B

Well, the good news is, we don't have to we're actually defining our whole build strategy here in terms of what the base image is who's making it which version and where in variable files and so what's useful about this- is that I generally build my stuff in regions that are close to me in azure, for example, that would be west europe.

B

Taylor might be wanting to build something in the u.s or specifically on the west coast. So we could just go in and find that property somewhere, which is, of course in the file that we did ignore.

B

So you can see ah beautiful terraform actually completed the long running task, so it's beautiful right friday deployments that work.

B

And there we go now the file is actually here. So you can see. We had terraform fill up this for this file for us added western europe added the new, updated resource group name and we're good to go.

B

So I wrote some packer code and I generally, when I run this locally, it's really hard to type with a microphone in front of you.

B

I'm still looking at my keyboard, I've been in it 15 years. It's fine.

B

So while we are doing this live demo over building is an image with packer ubuntu 1804, the server edition on azure and our code looks good. I just run packer format which, if you know, go, go fmt, terraform, fmt, just to be real.

B

We we live by discrete, always formally a code code. Quality is important. Then you know this works, but it's friday right. So you might go like you know what this is beautiful. My code looks good. I'm gonna skip the format for right now and we're just gonna. Do this live and I'm sure there's nothing ever to worry about.

B

So let's do it live or actually add a new branch. We're gonna call this hotfix, which you can't see in my window, but I just switched to a different github branch. Sorry git branch and in my packer file.

B

I think this one was going to be called image published and we're just going to say.

B

B

Reference so we're building infrastructure for our team to use we've got to open to server, we've already deployed it once, and my colleague just called me up and was like hey look. This repository had a little bug. I would love to see what happened, and so let me zoom in here a little as well.

B

What you could see is I just updated our base code to image published. I know that's wrong. That's okay, because hopefully our github actions will catch us and alert us.

B

Early friday, let's go, can you approve.

C

It's like somebody, somebody's clearly ready for the weekend uh and.

B

The white house was broken in the first place. I mean it looks good to me, published is a real word. My editor didn't pick it up.

C

B

C

It's fine, it's great too, because if you start to use things like github actions and you codify some of your approvals when it comes to your prs, you know doing some static checks. You can really help yourself out. You know, help future you out and you can run things like format or validate and different parts of the process so that you don't have to necessarily pull the code down check it, especially if it appears to be more of a simple change.

C

You can create some unit tests and other things that you know really help you out, um and so that's uh so that's what we've done here inside of packer is run a format to see if everything looks good.

B

The terraform one is actually also breaking, and we should get into that in a second.

B

And superlint is also still working. So while we wait for those to execute fully, you can already see at the bottom of my screen. Some chicks were not successful. um That's what we expect and that's actually what we're hoping for. So in order to show you what we can actually do with uh packer. I'm just gonna fix this once again and oops typo.

B

The cool thing is: we've got this whole um github workflow codified for you. So if you got grabbed the repository, I think taylor, you already shared the url for that you can just copy it into yours and have packer linting terraform linting linting in the form of format and validation, ready to go.

B

That's honestly, the best way to start the weekend and if terraform or packer come back with a lot of errors. It's a great way to start next week.

C

Give yourself something to do next week, yeah.

B

So pushed a new version of this and pretty sure the terraform code is still failing, but at least there we go packer by now successful in 13 seconds man. The speed of this really got a lot better, so we're grabbing a github action here, specifically one mattel and I built which also get some community contributions. So if you are one of the people that contribute to this github action, thank you so much contributing to open source is never free because it always costs somebody's time. So I definitely appreciate it.

B

I know you do too taylor, but in the meantime let me show you what we are actually doing here, maybe not the less interesting part of actually pulling the file. So we've got packer here which we're running and we're validating our file and there's a lot of code here to be seen. But the most important part is the highlighted part.

B

We did a syntax, only check we format it and validate it and the github action told us everything looks okay, which means we don't actually need to block this pull request and that's pretty much what you want.

B

I make typos all the time my editor usually captures most of them, but definitely not all, because at the end of the day, no edit will ever be as good as the tool you run it against and by having this kind of stuff codified in your github actions, you're making your life a lot easier.

B

So we'll get to the terraform part in a second, but in the interest of time, let's see if we can live build this image, I think the first one might fail because it already exists.

B

Let's give this a second, oh wait. We haven't deployed it in this region, yet awesome yeah. I really get happy and excited when we do stuff live and it works out.

B

uh Get some we're definitely not running these github actions locally, um mostly because there is no specific reason for us, and also because we like doing it remotely and making sure everyone on our team has access to them.

C

I had a good laugh at uh live. Demos are very scary. I absolutely agree with that. um That's and that's why we're so excited to be here today is so that we can kind of share some of these github actions with you. uh Github actions are really versatile because you can you, I don't know of a good way to run them locally. I know there have been some developments literally figuratively made on that front, but you can host your own github runners, there's great documentation on how to do so.

C

If there are things if you're on a private network depending on your infrastructure, you have a lot of resources available to you in that front. The github actions that we're working with today can run on any github instance. So if you have github enterprise, if you're just working with github, you can use these actions and then we in total, we'll you know we'll kind of get in more into this code.

C

The code in a little bit, but the three actions that we have specifically today, that we're running are packer, which you saw with what kareem was doing and making sure all of that syntax looked good and was okay, just really a static check, terraform, which is going to be doing the same thing. It's just going to be doing some static checks. It's not going to be handling the plans and applies specifically. You can do that, though. We do have a couple examples which you know feel free to at me.

C

I've got some examples of a pr process that will actually post back comments and let you know if the plan succeeded failed, who actually made the the code contribution and then lastly, we had super linter which will go through and handle. One of my favorites honestly provided by github is that it will go through all of the code in your repository and then, if you specify I want this to lent for bash. I want this to link for markdown terraform.

C

It will go ahead and scan for all of that, and then you can configure your own rules like cream said. I'm also a serial typoist and so I'll make mistakes all the time and that linter will help me write better code and have better code quality.

B

So, while taylor was talking, I started the kicking off of the packer, build image and we're getting there. Most of it looks green. We had a couple of errors.

B

Nothing important we're also for this build gonna, ignore those errors, because they weren't important to us also note of advice if you're building images- yes, paco will check if your exit code is good, but it can never verify if everything you try to install actually installed in the way you want to do it. There are other tools for that. I definitely recommend checking whatever you build and never just trusting one source to do it for you.

B

So in the meantime, while we wait for the packer image to build and what that involves is essentially packer starts up a vm in azure, it runs the base image that we instructed it with. It runs a couple of commands, and once it's done it captures that image and stores it it's a little bit of a time consuming process. I'm guessing will be five to six minutes before we're ready.

B

We are going to be looking at some of the code in terraform that we want to use to start up that image in whenever it's ready, and so as before, we've got.

B

Let me actually show you a very simple repository. We only have the azure provider in there and you could do this with any other provider in our case we're building the image on azure. So we need to use the azure provider we're defining a virtual network, a public ip, a couple of other things and you notice there's a local variable that we're referencing and so what you'll see these local variables actually use a remote terraform state, and so what we're doing here is we're grabbing that code.

B

So the same way, packer got a file generated from terraform we're actually using the azure subdirectory, which has its own terraform state or, in this case terraform cloud state to retrieve those values and by referencing them. In this way, I don't have to update the resource groups in three different places. I don't have to update the location, which was one of the options it's all defined in a single place and everything else just pulls from that place very, very powerful, because it means less places that need updating, and it also means less code maintenance required.

B

So we've got our network we've got. We should do some compute stuff. So what we're going to do here is first of all we're building a packer image and let's see how this is going right, it's actually pretty far along good, beautiful we're going to tell terraform to look for the packer image in our azure account and because we only have one image called packer. It should find it we're going to start up a virtual machine which we're going to start with our local ssh key and word of warning.

B

Tell an iphone! Well, didn't really consider how this would play out.

B

So while this uses our local ssh key till this local ssh key and mine are not the same, which means, if you want to do this with multiple engineers either you all have the same. Private key would not recommend- or you figure out a different way to do this.

B

It's actually a super easy exercise, but it's one that will get you thinking about how to do this, or just consider that you will never want to log into your image into your machine and, just don't add an ssh key so for the virtual machine, we're starting up we're grabbing our packet image, which will be dynamically retrieved and we're going to be outputting some things here and let's see if we can actually do that. Awesome.

B

Six minutes, 23 seconds to build a new image that my whole team can now use and start up a hundred or a thousand virtual machines from pretty good deal.

B

So in the meantime, let me update my terraform code by changing a variable that I know will break, and I know that it will break because my editor is already telling me it's incorrect, but terraform format doesn't detect it because it's not misaligned.

B

If I run terraform validate.

B

We have a reference to an undeclared resource, but it's friday right. Nobody has time to run validate over just make sure. Are we yep.

B

It's reference.

B

And if we switch to our browser.

B

Oh yeah still use the um the hotfix branch, so you can see here hoping this is big enough. We change azure rm to azure. It saves us two characters. Why would I not do this?

B

Storage is free and you can see this is running since 30 seconds and let's see what the check is saying.

B

Let's go here because that's nicer, and so this is the azure compute directory and we already see above there or warning aha, so the first one is we're trying to pull a public file that is not actually available everywhere.

B

Specifically in this case the github actions runner doesn't have that file.

B

Was this expected, of course not? It worked locally right like it worked on my machine, it's going to work on taylor's machine, but it might not work in our deployment setup which might be github actions, so maybe for this file we're going to use something different, but we already see that this is definitely a problem.

B

So let's go ahead and um can we remove the public key here? It's a good question.

B

Actually, let's look this up real.

C

Quick you'll notice that, with this with the repository that we've shared today, um you'll note that all of the terraform state is stored locally, uh and so you can use things. You know we typically recommend a remote state if you're working on things for production or environments that people might run applications on just because that's a lot safer way to do it, but for you know just for showing off a demo or a feature using local state wool. You know that's a that's a really good place to to start.

C

What's really nice is that terraform cloud is a great they've, got a free tier, it's just a fantastic option to get started and and very easy to store your state remotely and then start to audit changes. I believe it's free for about between one to five people, if I'm, if I'm not mistaken, and so you can start to scale that.

A

C

Team, if ever you want it, you start to get more people in project. You really like how that works in the workflow, and you can continue that forward, but uh really helpful here and then that's where that back end remote when you saw that remote state get referenced to that's what that was in reference to so, if you've and if you've any questions on anything today feel free to throw it in the chat, we'll be more than happy to answer on that front.

B

And so yes, we thought about this before um so to make this easier for you, we put in links to the actual resources that we're using, which allow us to do a quicker lookup.

B

I made a quip saying, storage isn't free, but realistically the more you comment, your code, the easier it gets. So we leave helpful little comments in there for you and, as it turns out, if we specify admin password, we don't have to specify admin as h key.

B

So if you could all just not locate your screen for a second I'm just going to type in my very secret password here, which is password but doesn't know smart because you know theory, guidelines and let's do a terror from validate.

B

Oh man, there's there's some rough comments in uh yeah. I can't comment on that.

B

So in the meantime, terraform is still telling us that our resource is incorrect and that's absolutely correct, but it's friday we're still going to ignore that so updated password cool thing about hard coding. The password is that everyone can see it right like. I don't need to tell my team what the password actually is.

B

And if you know jokes, sometimes don't translate well through webcams. That was absolutely sarcasm and you should never hard code. Your passwords, if you need a password manager, hashtag makes vault there's other tools that most cloud providers provide as part of your cloud.

B

Any of those is a better option than hard coding, your password, which is what we've just done and as you can see here, uh let's see we have a terraform. Well, we have an error in our github action. Terraform format is now checking out, uh probably also because I miss aligned some stuff man. It is really friday on this. Isn't it.

C

See that's what we do without our github actions.

B

And I'm kind of glad it got it caught both of those because as a human and as somebody who's been using terraform for a long time. I did not catch this, but the good thing is because this is failing.

B

Taylor won't be able to approve my pull request and that's really what fridays is about right, don't merge, pull requests that have failing github actions, because if you do your saturday becomes your friday and then your sunday is really your whole weekend and monday is not going to be fun either.

B

So, while we fix all of this- and I know nobody has been looking at my screen when I had the password up- so I'm just going to scroll that away because that's way better, um what other bugs do we have in here this one.

C

I did, I did see one question by verify uh and it was: would you agree with having a build agent like jenkins, to store the state remotely, as well as it being the sole automation user, to deploy items to different environments? Great question uh when we're?

C

Typically when you get started with terraform and when you're trying trying some things out again, you can store that state locally and kind of like get a get a handle of. What's going on and try to understand it, it's always best to understand something before you go. Do it. I know that sounds really obvious, but something that we forget a lot. A lot of people will typically jump into the code and not really give things the thought that they really need. It's.

C

Okay, to get up from your desk, take a take a walk around and just kind of like ruminate and think on things, and so I absolutely recommend doing that. I know I definitely have to practice what I'm saying as well I'll forget to do that at times too, but it's always good to have an understanding of what's going on.

C

So when you have that figured out and you're working with the team or pushing that to a production-like environment, it is best to have just one runner for that, so that's again where terraform cloud can help out or if you set up some ci cd. uh You know just having that be the thing that's running, uh the actual applies for everything. Typically, you won't want to. Let you know open up the floodgates to everybody being able to apply right, because then you can run into some issues there. So that's that's.

C

That is a really good pattern to just have that single type of apply, um and then you and then yeah like jenkins, github actions. Anything like that would would be a good uh good thing on that front. We also got one more question and answer really quickly from pablo. uh Do you recommend having both infra and deployment templates as part of the application repository?

C

Personally, my personal preference is breaking that up and so having different layers of infrastructure. Like say, if we were working in a cloud, we need to set up a virtual private cloud. You know there are a whole bunch of really great modules on the terraform registry.

C

We could take a look at and utilize I'll set that up in one layer and then tag that appropriately and then, if I'm setting up something like an aks cluster or something to deploy my applications to, I can reference the labels and the tags and use terraform's data lookups to kind of see what's going on there or data sources to see. What's going on there, so really really helpful on that front.

C

When you have a mono repo, it's easier to make a change uh somewhere and then kind of like clobber other changes or you know, have a problem with terraform state and so breaking that up can make it a little bit easier and give you a better blast radius around the changes that you.

B

Make and so, as you notice, there were a couple of other errors that crept in. As I was talking, I hit spacebar, I had some misaligned code. I know taylor takes code quality really seriously. He wants all my code to be correctly formatted.

B

I get that sometimes I forget that's why we have github actions that we can run against this, and so far 21 minutes in my change is actually starting to look good if we find the right tab. So far, uh two of the three or three of the five are green.

B

Actually, four of the five. This is beautiful yeah. I might have a saturday after all,.

C

Super linter uh does pull in all of the linters. I believe, and so that's why it can take a little bit longer a lot to pull down a lot of code to sift through. So definitely I think that once the code looks good, that's typically the github action I've seen run the most quickly.

B

Yeah and so talking about superlint and the other github actions, we've got the code in here, for you specifically yaml has all the packer stuff, as you would expect from the name, we're essentially formatting and we're validating and we're doing a syntax only check, and that's mostly because we don't want to try and test our deployment right away from github actions, because for that to work, we would need to give our azure credentials to github and if you're running github enterprise, that's a whole different world right, like you, have much more control over what happens in our case, all we want to do is make sure our open source image.

B

We're going to share with the world has good syntax, we'll also be deploying it in other ways and validating it that way, but for this action for something that you can use all the time without having to add your credentials, this will work same story pretty much for terraform we're doing a matrix, build against the three repositories that we have in here. Sorry, the three directories and that's really cool right, we're spinning up different agents making sure it all runs.

B

Super quick feedback has been great, I think, from start to finish we're at um 20 23 minutes and I managed to break and unbreak packer. I managed to break and unbreak terraform multiple times. Well, I only um broke it once but definitely broke it multiple times.

B

I've validated specifically our pull request. We format pretty much everything um and we make sure that our output reflects what's going on, and this is beautiful right. You don't need to think about this anymore, because your repository does for you and then finally, superlinter taylor already mentioned we're running it against our code base.

B

In our specific case, we validate it against terraform.

B

We also validate our yaml, which we oh, of course, the github actions, and we don't disable for errors.

C

B

Thing, that's nice.

C

One thing: that's nice, too, a really quick cream, if you wouldn't mind going to the top of that github action is so if you can see the on stanza there at that level.

C

Github actions are really helpful because you know they're made inside they're made for github, and so you have access to everything inside of your github repository you. If you wanted to run an action when somebody opens an issue- and you know you run an action to help triage that or add labels, you have the ability to do that. um If you want to do it for specific branches, you can here, you can see on pull request. We don't outline which branch we want to use, because we want all of them.

C

If people create a feature branch and then submit a pull request, we want this action to run. So there are a lot of cool things that you can do here. uh One. I made a note for myself to go check. I believe that there is the option to do something when somebody stars a repository, so it might be fun to send an email or do something like that.

C

As somebody stars, a repository maybe make an update to your to your readme and and commit that you know something something fun on that front, but lots of interesting ways that you could go about extending github actions um and then again the packer action that you saw today was made by one of the people on the stream, and uh you know thank you. Thank you for all the others that contributed, but it's fun to be talking with the maintainer and and get to use that action live for all of you today.

B

Yeah, I think, with all of those things and whenever you think about contributing to open source, think about what scratch you could, what it you could scratch with your code, because I guarantee you when I built that action I was like. You know what nobody's gonna use that, because otherwise somebody would have built it and I get pull requests. I get suggestions from people, it's super fun and it's also nice seeing people actually use it.

B

So, let's go through a couple of questions, as we have a little bit of time. Left, let me see linda should definitely be a requirement for graduating college. I absolutely agree.

C

With that I'd like to see super linter uh certification, uh that would be a phone to add your linkedin profile, um one one that I I kind of want to ask you as well. Kareem is uh by get some was, do you agree with one person as a gatekeeper uh to accept uh allow changes, or do you have an open culture where multiple people have the ability to accept changes in deploy? uh Personally, I think that it's it's always difficult.

C

When you uh have you know one person, typically it's better as a set of people or just as a team, in my opinion, to be able to prove that. I think that that creates a better culture because say you have that one person as the the person that's approving things, what about when they go on vacation? Oh, I never go on vacation. That person might say, uh do you you know, but what about?

C

If you get sick, you know if, if something happens in your life and you're, just not able to pay attention to that repository or or fulfill those duties, maybe it's saturday, uh maybe somebody's working that day- and this is something that needs to go in. This is a hot fix and it's critical, um typically giving the whole team the ability to prove is is best. You know, get your team's eyes on it uh other. You know.

C

Typically it's hard to see if you've made a mistake yourself, because you're too close to it is, is the terminology uh and but you know having other eyes and more views is always better. In my opinion, uh do you feel similarly cream or do you have anything to add on that front?.

B

C

You think about.

B

Infrastructure, as coded it's a huge part of devops right- and I think one of the core principles is comms, ca, lms and the first letter c stands for culture and to me this is really a culture question: it's not just about being able to go on vacation, it's about being able to trust your team.

B

If these are the people that built the same infrastructure, I should be able to trust them to deploy. I shouldn't need to go through a single person, because that it removes speed from the process.

B

It also means that we still keep a human element in there to say hey. This is good or not good and I think that's dangerous, because humans are faulty, um we get constantly version upgrades, but still there's bugs galore.

B

So, in that sense it makes a lot more sense for a team to be able to say. We've got these many checklists that we run through automatically on every pull request and on every merge to our main branch, and if that checks out awesome, you should be able to deploy. It doesn't mean you can't have gatekeeping in the software like say: let's not deploy friday 9pm because sometimes still things slip through and go wrong, but um yeah. You can definitely approach this from different ways and yeah.

C

One other question I saw by get some was once you commit a secret or password to github. You can't erase it right. um So one thing to be mindful of is that, while github actions do a fair, fair amount of scrubbing from the logs, that is some one thing that you definitely want to be mindful of. If you put a password in there or if you've made a repository public, the previous runs can be seen.

C

So if you do have any sensitive information in there, uh that's definitely something to be mindful of and and getting that cleared out when it comes to secrets in the repository, if you're, storing something as a secret uh that will automatically get scrubbed out, but um you know, which is great, you know, secrets dot and then, if you put like environment variables in inside of your github repositories, um one thing that's really cool.

C

Is that and another shout out to the github uh provider is that you can actually, you could set up vault and then create secrets and then push those up to github for the for your repository, which is really handy, feature for organizations. You can also push that up to the organization as a whole, so you can have that global I've got feelings about global variables, but uh if you need them, you know like if you're doing something like organization, name or something. You know some.

C

It's great there's easier ways to do that, but if there's something that makes sense to store at that level, you have the ability to do that which is really helpful.

B

And I think the vault flow you mentioned there is really really good because it essentially and it doesn't have to be evolved any tool that can manage your secrets without you having to know about them, but having the tool in a position to provide secrets to something like terraform is great, because at that point you essentially get become secretless.

B

As a team, you- and I know this sounds very marketing e, like serverless, because there's still secrets they're still servers right, but if you- and I don't know about the secrets and just our secrets manager does that's perfect, because what I know what I don't know I can't tell- and I also don't have to worry about. It- also means if we're pulling that from a secrets manager.

B

We can update it there and our terraform code will keep working and I don't need to have another piece of updating very much the same workflow that we have with having packer generate sorry. Terraform generate packer files very much the same way. We have terraform lookup dynamically, the packer image. It's all meant to make it a little easier and to have less friction between various moving parts.

B

And absolutely realize that's a very long answer to a very short question, but security is something that requires multiple layers.

C

It's really true, and it's always something to be mindful of it's best to start that process. um You know day one it's it's security's, never best, you can do it, uh but I advise uh strongly against that. You know start thinking about security from day one, don't let it you know, don't let perfect be the enemy of good things do still have to work. Sometimes there are security requirements where it's you know it would absolutely stop uh you dead in your tracks and there's a better way to go about implementing.

C

It doesn't mean that you know maybe the second pass. You can kind of implement that you can always iterate with infrastructure and infrastructure as code, but it reminds me of a great book called the phoenix project for those of you who haven't read it that absolutely recommend it and they kind of talk through some some issues there where in this fictional company there they go through a situation where uh somebody implements some encryption and then, unfortunately, all the systems stop working because everything's encrypted and they can't decrypt it.

C

So really a really good example of you know when that might not be the best thing in production, but always think about security for sure.

B

Yeah there's many moving parts and I think a couple of the comments are really reflective of stuff that you encounter with a lot of teams right. Verify windows is saying: security is a luxury to some devops teams. Just get it done yesterday, of course, yeah. We never have the time to do everything perfectly and properly and that's why we need to make the friction of doing it. The right way as low as possible.

B

For us- and I know- we've said this before in the past hour- for us github actions is part of that process. When I don't have to worry about format and validate, and it gets done for me, then that's perfect.

B

Obviously the best place would still be that we don't get a situation format and validate actually bring something up, because my code is perfect, but it's friday, my code definitely was not perfect.

B

See, I'm not the only one with really secure passwords here I like it.

C

I did want to give a uh shout out to nico uh from ohio. uh Also uh from ohio here was born in columbus. Ohio uh then moved up to cleveland ohio and lived there for a while and now I'm out in los angeles, but uh always happy to see ohio on the thread.

C

Gotta give that a shout out save.

B

uh Verify windows has a great question: would you use github actions as a deployment tool?

B

So with a lot of these kind of questions? The answer is always. It depends for anything where I'm 100, confident that the information that could leak is not important. I would say absolutely. It also depends on your organization. I know the teams running github agents, both in github e. Obviously that would be your teams and in on github.com it would be well the github sre team they're making a huge effort to make sure your information doesn't leak, but you might still be in an organization that says you can't share your cloud credentials with a provider.

B

That's not approved and that's really what it comes down to.

B

For example, the first thing we showed you today was deploying github repositories with terraform and codifying them so we're doing this in github actions against github.com we're running the actions on github.com and all we need to make that work is a github token which actions have access to anyway. So what's our blast radius here practically zero.

B

So absolutely for that no question us.

B

If we're talking about creating um a cloud hsm, so a hardware security module and making sure we can sign our pci data with that, maybe not so much mostly because pci dss the standard doesn't allow you to do things like that without a huge risk assessment.

B

So the short answer is: it depends.

C

And it does make I can I can absolutely attest to I. I missed the in the comments where it was, but um for any of you that uh don't like managing a jenkins, server or machine. uh This is a really good workflow and it does kind of improve that granted. You know it depends for your situation, of course, and it's always uh in in some cases difficult to justify moving to a new thing or when is the best time to move to a new thing.

C

You know it's tip, newer, isn't always better right and it can kind of it can really slow you down. If the, if the team doesn't say, we have a build process and everything works, it might not be the best use of our time, and you know focusing on some other things, but uh I can attest for any of you that are interested in moving from jenkins to github actions and can do so. It's it's really, it's really great for for me and some personal projects and home lab stuff.

C

It's it's made the process a lot easier and it's a lot better and more visible too. For you know, other friends and and family that I work on things.

B

With think one of the comments was pretty nice in that it's an organizational problem and that's really what it is.

C

B

C

Any other questions or thoughts and and granted you know we, we love spending time with you here live, uh but if you again, if you do have any questions or things come to you after the session uh feel free to reach out to cream or myself, that would be more than happy to have chats with you and you know, see what you're working on uh hear. You know good good, bad and ugly.

C

You know in terms of your workflows and code, that you're writing and, if there's anything that you want to see from the terraform or just infrastructure in general landscapes.

B

Yeah, absolutely I mean we're here to answer these questions. We can also answer them after a stream, and these have been really good questions. It's when you think about a lot of these problems. Talk with your counterpart from a company that you don't work with. Everyone is running into the same problems. Nobody has a perfect deployment.

B

Nobody has a super frictionless security process because none of these things are possible, but we can all learn from one another. If you need to define 18 20 layers of security, maybe somebody has already figured out six of them and you can copy that and essentially replace the private keys and you're good.

B

C

I like to follow them on twitch twitter.

B

And myspace actually gonna check my myspace really hoping for a follow there other than tom and tail, and nobody follows me. There very.

C

Sad, like you know, time too interesting. I do.

C

B

Yeah, my geo series is definitely offline, but um yeah. We've got a couple of really great links in terms of github action guides that were popping up right now.

B

All of this stuff can get complicated, because when you're building your image inside github actions, you need to think about. Okay, where am I deploying it? Do my actions have access to that? What are github's ip ranges?

B

What other things do I need to consider and then you bring in a tool like terraform, and you can retrieve a lot of this and just have it run automatically for you.

B

Good thing is: if you're just joining us you're about an hour late to the party, I'm.

C

B

Everything was recorded, so you can watch it later.

C

uh One of the things that I I see people do people people I've worked with typically have a lot of questions about terraform, and where does it fit into your your? You know the application development life cycle and it's I've seen some people kind of stick with one provider right. They might use a major cloud like aws, azure gcp and then only for that, but as you've seen today, we can do a lot with you know: github actions with terraform.

C

We set up a github repository with terraform, which is incredible. We spun up a machine. We built something with packer. We used hcl all over the place. There's there's a lot that can be done a lot of things you can string together, uh but that does make it confusing on places to start, because when you think about configuration management, I've seen some people try and you know, trend that direction and use terraform to also be configuration.

C

Management um terraform works really well in a declarative, sense right, and so, if you're doing a blue green deployment that might work better for terraform. But if you're trying to modify something in place and keep configuring it, it can get a little bit.

C

Messy terraform has really nice tooling inside of it so that you can use you know, spin up new resources as you need, but if you're just getting started with terraform, take a look at data sources, there's a lot that you can search for and pull down and use without having to you know, really do any kind of bash.

C

You know black belt bash type, scripting or anything like that. You can just use a data source. It's really really helpful. I'd say that was a more harsh learning experience for me as I got started with terraform back when it first came out. Data sources are really a wonderful thing.

B

Yeah and they've been growing a lot one of the things we just mentioned. Ip ranges when you're doing remote deployments a lot of the providers that terafrom supports actually have data sources that allow you to retrieve specific ip ranges and what's super cool about this is let's say, you're running your github actions, agent locally, because you want to utilize all the formatting and validation tools remotely, but the actual deployment needs to happen on your network cool run a local agent.

B

But now do you really want to open up your local deployment tool to the world? Of course, not so one way to get around that is to retrieve github's ip ranges through terraform and then use that to open up your firewall.

B

Taylor mentioned azure and google and amazon, but you might be running a local firewall that has an api very likely. Terraform will know a way to interface. With that.

B

And that's friday, bliss for you.

C

Well, we hope that you enjoyed the stream that you enjoyed our friday deployments.

C

Please check out the repository uh if you haven't, if you do want to follow along and check out those github actions, uh don't be stranger, feel free to reach out and talk to either of us and uh with that kareem do you have any closing words.

B

Yeah, I I noticed two really cool questions. um Get some. uh Do you guys deploy to prod on rolling bases, it's kind of personal? It's it's not personal. It's only personal when the rolling basis doesn't work out, um then you know, then it's definitely. No I'm sorry. I can't tell you uh where it makes sense. Yes, because the gatekeepers should be automatic and they should. You should have tooling in place that feels comfortable to say. Yes, this is good or no. This isn't good and also be very restrictive.

B

Not every pr has to end up being on main and being a deployable version. Sometimes you know you're just exploring, and the other closing word is literally that keep exploring it's. It's been a hard year for a lot of us with not being able to go places, but that doesn't mean that the learning has to stop. Conferences might not be here, but cool streams like this one are and you get to learn from other people. You get to see other people make mistakes.

B

Think about what you can show others. It goes multiple ways show your local community hey. I learned this cool thing and maybe you know you'll create a community of your own.

B

Think with that. Thanks for having us and we'll see you next.

C

B

B