Cloud Native Computing Foundation GitOpsCon NA 2022, 15 Nov 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: How to Achieve (Actual) GitOps with Terraform and Flux - Priyanka Ravi, Weaveworks

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How to Achieve (Actual) GitOps with Terraform and Flux - Priyanka Ravi, Weaveworks

GitOps might sound like a self-explanatory term, but it is not as easy as it sounds. Many think this just means to store your Infrastructure-as-Code in Git, then have a pipeline run the code, but it is actually much more complicated than that. True GitOps takes the deployment out of CI/CD, and the most popular solutions are using Kubernetes controllers to do all the heavy lifting. Ensure what you’ve defined in Terraform is what’s always running and available. Flux continuously looks for changes and reconciles with the desired state. Take advantage of all the benefits of GitOps: streamlined and secure deployments, quicker time to market, and more time to concentrate on app development! Pinky will provide an in-depth look at the new Flux Terraform Controller, which enables Terraform deployments to be done the GitOps Way. They’ll end with a demo of a common use case implementation.

A

um Hi, um my name is Priyanka Ravi I, also go by pinky. I am thank you. I did the same thing in Valencia, um so I'm a developer experience engineer at weaveworks um uh and uh if you're not familiar, uh oh and dog, mom, tattoo rescues and I I forced them to to pose for me. So um if you're not familiar with weaveworks, it is founded on open source.

A

um We created flux and Flagger, which our cncf donated projects, um and we have a bunch of other projects within the cncf, along with several additions that we've created for, um uh like as additions to flux for git Ops such as we've get Ops the terraform controller, which we're going to be talking about today, um vs code extension and several others.

A

Okay, so I know they already kind of went over this, but just you know for anyone that just walked in or people that are watching this later in the recording.

A

um What is get Ops did I think I might have sorry.

A

A

No, no there, it is it's back. Thank you Scott. So um I just want to highlight uh that, like what is get Ops, it's an operating model for um Cloud native applications such as kubernetes um and I, do want to mention that it's not just for kubernetes.

A

So if you are doing a multi-cloud infrastructure, you can still use git Ops, but um we're here talking about kubernetes, so we'll be focusing on um because we'll be focusing on flux, so um get Ops utilizes a version controlled system most commonly git as the single source of Truth, and it enables continuous delivery through automated deployment, monitoring and management by a version controlled system and with get UPS like they mentioned earlier.

A

You manage your infrastructure and applications declaratively, so they went over the get Ops principles earlier, so I'm not going to go over them in a lot of detail, but uh just wanted to point them out again here, um like they mentioned they, they created these. The getups working group created these get UPS principles as guidelines um by talking to several end users and coming up with these best practices.

A

um I'm briefly, you know uh they should be. Everything should be declared declarative. Everything should be versioned in immutable. Things are pulled automatically by a software agent and continuously reconciled as well. So you get away from configuration drift, everything's written in code. So it's reusable. You have an audit Trail and there, because it's versioned and immutable there's no sneaking in a change um and keep in mind that these are just a set of best practices. If you don't already meet these, you can still get started using gitups.

A

um Everyone's Journey looks different and you can always you know, add tweet tweak it to add hardening later so.

A

Why get UPS? Why should you, even you know, want to do this I'm sure if y'all are here, you'll probably already uh know that there's a lot of benefits to get Ops, but um because of get Ops tools unique way ability to treat everything as code. It creates a direct impact on security, for example. If all configuration and security policy is treated as code, then everything can be held in version control.

A

So any and all changes that are made can be reviewed and input into an automated way, and so there's no manual processes, hopefully you're less likely to be at work on a weekend. Because of some. You know mistake that someone made when they're doing a manual deployment so now we'll get into what is flux. So flux is a get-centric package manager for your applications, but git isn't the only um source that you can use with flux and it provides a set of continuous and Progressive Delivery Solutions for kubernetes it's a natural extension.

A

It was really created with kubernetes in mind, so it's a natural extension of the benefits of kubernetes and at the core of it. It is continuously monitoring your version control system and it applies the desired State. That's been expressed, declaratively there. The nice part of this is that you don't have to worry about configuration drift because it reconciles on a set schedule.

A

So if things for some reason have gotten out of sync, it will set it back to the desired State and it really reduces developer burden because, um like I said, it removes the need for manual deployment processes as well, and also the flux. Cli is a really convenient way to bootstrap the system in a cluster and to then access the custom resources that make up the API.

A

So these are some things that we like to just highlight about flux and the key things that we point out in our website about flux, flux provides git Ops for both apps and infrastructure. You just um using flux and Flagger. You can actually do Progressive delivery with canaries feature flags and a b rollouts as well. Flux can also manage any kubernetes resource infrastructure and workload. Dependency management is built in as well.

A

Also you just get push um and flux. Does the rest flux manages deployments through automatic reconciliation flux works with your existing tools uh because, like I said, it was created with kubernetes and all your like ecosystem in mind.

A

um It works with your git providers, GitHub gitlab bitbucket, and you can even use S3 compatible buckets as a source, Helm repositories, all major container Registries and all CI workflow providers.

A

It also works with any kubernetes and all common kubernetes tooling, such as customize, Helm, rbac and policy, driven validation like Opa kyverno, admission controllers. So it really just falls into place with what you have set up already. It does multi-tenancy and, as we like to say, multi everything um it uses true, kubernetes rbac via impersonation, and supports multiple git repositories, multi-cluster infrastructure and apps work out of the box with cluster API um with flux.

A

You can use one kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters itself and manage clusters, including lifecycle and fleets as well flux, also alerts and notifies you. It can provide Health assessments, alerting to external systems and external events handling you just get pushed and you can get notified on slack and other chat systems.

A

Users, trust flux, I myself am actually my experience with Lux and why I'm, even here talking about it? Besides being a developer experience engineer at weeb works now I before that was actually a get Ops platform engineer at a very large company and we set up get UPS on our kubernetes on-prem kubernetes platform there. So that's my experience with flux and I'm a huge fan of flux, and it was super simple to set up for us and there's a really awesome. Community I feel like this community is one of the best I've ever worked with.

A

We also welcome contributors of any kind. The components of flux are on kubernetes core controller runtime, so anyone can contribute and its functionality can be extended very easily as well all right. So um what are the benefits of flux? So flux is like I said it reduces the developer burden by um it removes the cube control problem. You don't have to worry about Cube control versions anymore, to be able to interact with your cluster.

A

It's also very extensible. It's versatile. It works with existing tools, it's really flexible and because of the microservice architecture which I'm about to mention in a second it. You can pick and choose what you want to tailor your own experience with flux.

A

So uh flux is a set of kubernetes controllers, and so it's, if you're, not familiar with controllers a controller handles the life cycle of objects in kubernetes like what should be done when an object is created, updated deleted, Etc, um and so these are the controllers and the main controllers.

A

So the terraform controller is actually in addition to the flux I'll get to it, but uh the the actual flux controllers are the source controller, fetches, artifacts and stores them as fetches manifests and stores them as artifacts, so that the customized controller can then apply the manifests and it runs manifest generation using customize I just want to mention real quick, too um I know the like fact that it's called the customized controller can sometimes be confusing, but it is named thusly because it's using customization in the background.

A

So if you do have a customization yaml already set in the folder that you're pointing it to it, will just take whatever's written there and apply that if you don't have a customization yaml set, it actually creates one in the background. It recursively searches for your yamls in that folder path and it will create its own customization and just generate all those the run. All those manifests and then there's the helm controller, which controls deployment of Helm charts, the notification, controller specializes in handling inbound and outbound events and the image.

A

Reflector controller reflects image metadata for the automation controller, which then updates yaml. When new container images are available, so basically they both work together to um to update a git repository when new container images are available and it works with a ton of other tools.

A

I'm not going to list them all, but these are just some that uh thought to list out and so basically, like I said, it'll work with whatever you're, using basically in your environment, and these are some reasons that um I personally as an end user and others that I've talked to love flux, it really does does make life easier.

A

um In my own personal experience, multi-tenancy is uh super simple to set up um and I personally have only ever done soft multi-tenancy, so creating name spaces within a cluster, but, like I said you can do hard. Multi-Tenancy create additional clusters as well. Using it um depends on is a cool feature. If you, if you're not familiar with it, look it up, it's really cool and then it's got fantastic. Helm integration, I mentioned notifications and alerts.

A

Bootstrap is something that you'll see um when I show the demo and then the flux CLI is really awesome as well and now the thing that y'all are here for obviously uh the terraform controller. So what is the terraform controller? The terraform controller was created by weebworks and it's a free and open source flux controller that can manage terraform resources, so it will not come when you bootstrap it when you bootstrap flux or install flux, but um it is an additional thing that you can install onto your cluster and terraform.

A

Resources that can be managed are not limited to kubernetes resources um and, if you, you know want to install it. This is the GitHub link right here and then there are some really great docs that are written as well um with this. This specifically I wanted to point out this use cases link as well, because if you go there, it's uh super detailed on like different ways that you can use the terraform controller, and then it has detailed instructions on how to actually set those specific resources up.

A

So okay. So what are the benefits of the terraform controller? uh You know, there's there's full get Ops automation um and you can use gitups for existing terraform resources as well, and you don't have to have it automatically apply your um terraform resources you can set it to actually do manual applies as well, so you can read the plan and make sure it's what you want to have applied before applying it. Also you can set it up to just do drift, detection of terraform resources, so, let's say you're already.

A

You have your terraform deployments deployed in some way already, and you don't want to switch off from that, but you want to be notified if, for some reason, things get um out of sync, you can set it up to do that as well, and I'll show that in a second as well, uh and basically it's used as a glue for terraform resources and kubernetes workloads as well.

A

So um these are some features of the terraform controller and I kid. You not like if you check the docs within like a few days, it'll- probably already change, because there's so many things being added to this controller daily, so um keep an eye out for any new features there.

A

um So the first one like I mentioned you can set it up to do manual or Auto approvals, um and then drift detection like I mentioned, is a really awesome feature you can have it do that, whether you want it to be doing applies or not as well, um and then it also accepts a list of config maps and secrets as variables.

A

Okay and the state file is stored in a secret by default. So that's the default action if you don't already have somewhere that you're storing your state file, but you can override that with your, um like, whatever backend you're, using right now. So, whatever back end, you want to use to store your secret State file. You can um and then also it has the ability to set up health checks um it you can there's a flag that you can add to tell it to destroy resources on deletion.

A

That's not the default behavior, um just like it's not in flux, and you can write outputs to a secret as well. So if you need to use it later, you can write outputs to a secret and then use them somewhere else. Also um recently, fairly recently, I would say within the last maybe month or two um concurrency was actually added so before this it would do all the jobs right in the same um controller, and it would just run them like one after the other and recently um Runner pods were added.

A

So now it can do I think they, the terraform controller team, did a test with 1500 terraform modules um successfully, so that's really exciting, um and so it's able to reconcile and provision High volumes of terraform modules, concurrently, um greatly scalable, and with that you can actually customize your Runner pod as well.

A

So if you have like a custom image or something that you need to set, you can do that um and with uh flux's new edition of um oci support, uh the terraform controller was quick to follow, so the terraform controller also can use the oci artifacts as source and create oci artifacts All That Jazz, um and there is the ability to force unlocked terraform state. If for some reason you need it and lastly, we also have it does play nicely with terraform cloud and terraform Enterprise as well.

A

There's integration with that, if you're using that for your back end or whatever, okay and so um before, I go into my demo. I just want to mention that um we have uh um you know these are. These are the events that weaveworks is uh doing at um uh kubecon get Ops con. So if you want you can scan this QR code and get more information about what we're up to and um things like that as well, so and check us out at our booth.

A

We have a flux Booth uh in the project Pavilion, and we also have a weave Works Booth as well. So please come talk to us. um If you want to chat with me about the telephone patroller as well. So now, I have a demo, but I have to tell y'all. I am terrified of live demos, so I pre-recorded it and I'm gonna talk over it because I do not want to tempt fate. So that's what's happening.

A

So all right, so in this demo, what's going on is um my co-worker Russ actually created this awesome repository called flux, Vault demo and you can go Fork it. You can go play with it. It's really awesome. If you um it just what it does is it sets a vault and it has a couple of uh it sets up uh the it bootstraps flux, hang on I'll, just I'll run the demo and then I'll talk over it and we'll see it.

A

But um so, if you run make it, there's a make file in this project that um give myself a second to catch up. Okay, so in here you can see it creates a kind cluster make sure that these environment variables are set, and then it just runs bootstrap, which bootstrap is a really awesome command um within the flux CLI to get started really quickly and easily with flux, and it tells it which path so that dash dash path is telling it to listen to. In that repository flux, Vault demo, it's saying listen to that kind.

A

Folder and the thing about bootstrap is, if you don't already have this repository created, it will create a repository for you in your GitHub and then it'll bootstrap into it, so it'll clone it down and put all the manifests that are necessary into it and then run them and then have flux listening to that Repository.

A

But if you already have that repository created in your bootstrap into it it'll um just you know, clone it and run the Manifest that already there and the nice thing about bootstrap is you can run it as many times as you want? So if you go and update the flux CLI and then you rerun, bootstrap it'll actually update to the newest version of flux, which is really cool.

A

So you can see there. Sorry really briefly, the the pods that were set up.

A

So those are the controllers, um and so in here you can see uh like I mentioned those yamls that are created by bootstrap or in this case they're already there, but so it creates this flux system namespace, which is where all of those flux components are installed, creates a gotk sync file, which is this file that has all of the um oh, no, the sorry it has all the the just, the The Source that's created, and the customization and the source is telling the source controller to go.

A

Listen to this flexible demo, repo and that interval is saying every one minute go check this repo for any changes and pull the manifests. The customization is saying: go apply what's in that flux, Vault demo, but specifically in that cluster's kind, folder path, that we told it in bootstrap and that's going to be reconciled every 10 minutes. So it's going to go for Supply, what's pulled by The Source control there, and then in here this is the apps.yaml that so this is an additional customization.

A

That's then pointing to this base apps path within this project itself, and in that base apps path. We have the yamls for the terraform controller and it's pointing to the helm chart for the terraform controller. This is my favorite way to set up the terraform controller is through the helm, repository and the helm release. So um this is kind of what I recommend to set it up, and you know you can point it to the latest version of the telephone controller chart um if you're not familiar with uh Helm Helm uh influx.

A

This is how you set it up. You tell it to point to a hum repository and then specifically to a Helm release, and this is where you give like the values that you want to override in the chart, um and you point to the Chart specs as well, and then this is saying every hour go apply this chart, so that's what's happening in here and then so also in here is Vault. So this is also setting up just the Vault instance, so the Vault namespace and then also Vault through the helm chart as well.

A

So this is using the um Helm chart for Vault and um what we are using terraform for which is not set up yet, but um so basically right now we just have a generic Vault instance. That's running, but we don't have anything. That's set up.

A

Okay, so I am actually setting up a web hook so that I can set up notifications in slack. So this is how you set up the notification controller. I, um don't make the mistake. I did and push the secret with the web hook too um get because it will delete your web hook in slack. Slack is smart enough to know that if you've pushed a secret with the web hook to get, it will delete it. So that took me a little longer than I like to admit to realize, but I applied it as a secret.

A

So then I can reference it when I'm setting up the notification controller part so I'm I had it all commented it out, but um yeah so, okay. So this part this is important because by default, the notification controller is built for flux, right, just flux. So it's not actually listening to terraform events by default.

A

um So this this, uh these crds needed to be added to tell the notification controller to also go. Listen to um terraform events, and things like that. So this is important to add. If you are trying to set up a notification controller specifically for the terraform controller um and then that's bullet pushed so me.

A

Okay yeah, so this is the terraform notification like how it's actually set up. So um you tell the notification controller, which provider you want to to connect to. In this case, it's a slack web hook.

A

um I I have that secret ref that I pushed with the actual web hook, and then the alert is how you tell the notification controller, like hey, alert me on on this thing, based on that provider, so I'm telling it to um alert me on any terraform events um and any any and all to perform events, um and so that's what's happening there.

A

A

I don't know yeah, okay, so I'm gonna reconcile it. So this is how you reconcile when you make a change in flux, um so in this case I'm reconciling the source and the reason I only have to reconcile The Source in this case is because um The Source controller, when it actually pulls in a new Shaw, will also kick off to the customization controller and tell it to go apply if there's actually a change.

A

um So if there's not, it won't do that, but so okay, so um now in here, I'm going to add this. um So this is similar to how you add, like a customization or a Helm release. It's saying um it's like I guess. Let me.

A

It's okay. Basically, what it is is it's telling the terraform controller hey, go, listen to this terraform file path and so yeah. So, okay, so you say kind terraform and then it's listening to that. K8 spell config path and it's saying every one minute go and apply it. There are a few things. I want to point out really fast. Oh, my oh yeah, okay, I came back.

A

um Is this a proof plan and it's set to an empty string, which is how you would tell it to be manual applying um if you wanted it to do auto applies, you would say Auto within that quote or I, don't think if you just don't, have it I think it also would do the same thing.

A

um Also, this store readable plan is also important because um it's saying store my plan into a config map and human is just saying you know just the way that you're used to seeing a plan with all those plus signs and changes and everything, um and then okay. So this is what the terraform is pointing to. So there is a you know: it's setting up the Vault instance with all the the back end and auth and everything, and then it also creates this Secret in here um this KV secret too.

A

It's just a demo cred secret with some. You know, exposed credentials in there just some fun stuff, um and so then, once I reconcile this. It will then have that terraform uh also added there.

A

Okay, okay, so I'm gonna skip ahead a little bit so um there so now that that's actually there um it's not ready actually to be applied. So I can do this. Cube control get config maps to Output the um plan, specifically to see it before we apply it. So this is what that would look like what, if you're used to terraform you're used to seeing things like this. This is very um familiar to you, and so oh great looks good. We love it! So I'm gonna go in here and you take that.

A

um There's a uh there's, a uh sorry.

A

It outputs a so you see here where it says, plan generated, set it to the string, so I grabbed that and I um put it in there into the um that that quotes where it said, approved, plan and I reconciled that, and so that's the way of approving the plan manually. If you want to go that route, so then um I'm going to actually set it back to auto so that we can um I can show you like what configuration drift looks like and what the notification looks like real, quick.

A

um So that's back to auto um we're going to reconcile that and now, if we go into the instance of Vault that was created, you can see that there's this demo creds that you know was created with that. Terraform I deleted it, um and if we go into um that slack Channel that's being notified, you can see that there is a drift detected and it says that that cred has been deleted and um if we give it an even I, think a second more it will actually um it's back. It stood back up.

A

So that's because we have that auto apply set, so it's continuously reconciling and that is it. I'm done, foreign.