Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps to Automate the Setup, Management and Extension a K8s Cluster - Kim Schlesinger, DigitalOcean

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

GitOps to Automate the Setup, Management and Extension a K8s Cluster - Kim Schlesinger, DigitalOcean

In this workshop, you will experience the power of Infrastructure as Code and GitOps to automate the provisioning, modification, and extension of a Kubernetes cluster. Join me to learn how to use Terraform to spin up a Kubernetes cluster and install FluxCD, which will watch a GitHub repo and automatically apply any changes made via git commit. In order to keep all of your credentials like secrets, passwords, and tokens in your GitHub repo, we will show you how to use the sealed-secrets project to enable one-way encrypted secrets that can only be decoded inside the cluster. Finally, you will install and use Crossplane to provision digital infrastructure from inside your Kubernetes cluster, including resources from different cloud providers, giving you a chance to experiment with multi-cloud infrastructure.

A

Hey good morning, everybody thank you so much for being here, so this session is called gitops to automate the setup management and extension of a kubernetes cluster, and I'm going to talk about how the session is going to be structured and how we might need to move around a little bit in the room. But first my name is kim schlesinger.

A

I am a developer advocate at digitalocean. I focus on our kubernetes education and prior to digital ocean. I was a site, reliability, engineer at a company called fairwinds and I'm actually a career changer. So before all of that I was a primary school teacher and I was an instructional coach and curriculum designer. So I like bringing tech and education together and what we're going to be doing today is uh I've created a repo for you that we're going to be working through. So let's take a look at this.

A

So this is the repo. There are five chapters in the repo and the goal of this workshop is for you to have some experience getting like an aha moment, either with infrastructure as code or with git ops. And so, if you look through the chapters you see, the first chapter is we're going to set up a cluster using terraform.

A

The next thing that we'll do is we'll install and set up flux cd for continuous delivery. The next thing is that we'll use a project called seal secrets to encrypt kubernetes secrets so that you can store your secrets in a public git repo.

A

The next thing that we'll do is we'll use a project called crossplane to make the cluster that we've spun up a universal control plane so that we can create other cloud resources from within that cluster and then finally, we'll tear down our cluster.

A

um So if you're using a cloud provider like digitalocean that you don't get charged for any usage that you want, don't don't want to be charged for, and so I know that workshops are really tricky and my goal is for you to have an experience, but the internet can be really unreliable at this venue uh and so here's how we're gonna do this. um So if you are someone who is interested in watching me, do a demo of all of this and you maybe wanna follow along or you just wanna observe.

A

Then uh in a few moments, I'm going to invite you to move toward the front of the room, so we might need to shift if you're someone who's sitting in the audience and you're thinking.

A

I already know a little bit of this, like maybe I'm already familiar with terraform, but I'm really interested in digging into flux or cross plane. You're absolutely welcome to skip ahead to those chapters. They have very good instructions in them and I think we'll see how this works.

A

What I'd like you to do if you're someone who's skipping ahead and isn't going to follow along with me, I'm going to have you move to the back of the room, and I have two colleagues here: adam wolf, gordon and wayne warren they're, both engineers at digital ocean, but they'll, be available for you to ask for help if you're someone who's working ahead and you get stuck they'll, be there to help you troubleshoot and so yeah. If you want to move lockstep with me, you'll move to the front.

A

If you want to move on your own you'll go to the back, and if you are someone who wants to watch just a little bit of this workshop and then uh step out after a little bit, that's totally fine. The goal of this workshop is for you to get what you need to out of it. So if you need to leave that is no problem, so.

A

If you are following along uh with the workshop and you want to spin up a cluster using digitalocean, we've got a promo code for you, so in this section in the readme it says promo code you'll get a hundred us dollars worth of credit and the way that you apply. The credit is that you create a digitalocean account. You go to the billings page, and then you enter this credit here, it's kim at uh kubecon eu100.

A

There are a few things, though, if you are already a digitalocean customer and you've used a promo code in the past, you can't enter one yourself, so we'll have to do that for you manually. We won't be able to do that until after the workshop. So you have a couple of options. If you want to use that credit, the first one is, you can slack me on the cncf slack and send me your name and your email address and I'll apply the credit after the workshop.

A

This is what uh this is. What I look like on the cncf slack and then the other option is you can just create a new digitalocean account with an email address that you haven't used before and then you'll be able to apply the credit that way so yeah. There are also prerequisites in terms of binaries that you want to download.

A

So for every single chapter you want a digitalocean account. You want to use doctool, which is the digitalocean command line tool that lets you communicate with our api, terraform, helm and cubecontrol. I bet a lot of you. Have some of those things installed for chapter two you'll need a github account and flux cli installed for chapter three you'll need a tool called cubeseal installed and for chapter four there's, nothing, that's not in the list above that's required and then, finally, uh so how to ask questions and troubleshooting tips.

A

So the first thing is, if you're working through the workshop- and you get stuck since there's so many of us there's a process. I'd like you to go through. So the first thing is: if something fails out, reread the instructions that you just tried and run the command again, if that doesn't work, google, the error, if that doesn't work, talk to somebody next to you or find adam or wayne and ask them for help.

A

And if you are attending virtually ask a question in our slack channel, you can also do that if you're in the room- and so that is 2-cube, con custom custom extend kate's, uh oh and it looks like there's some trouble with the video hanging for the virtual folks. So um all right so last thing before we we break and reorganize, is that I've got annie here, who's, our moderator and so I'm going to demo each chapter after each chapter that'll be the opportunity to ask questions.

A

The way that it'll work is annie will take a few questions from the virtual platform and ask those on the behalf of those attending virtually and then, if you have questions, you'll queue up at the microphone in the middle and we'll go through there. So next thing we're going to do. I'm going to put a 10 minute timer on your job is to make sure you have a digitalocean account and that you're all set up and then rearrange as needed. So moving at your own pace is going to be toward the back of the room.

A

Going step by step with me is going to be at the front of the room and then when the timer goes off I'll get started with the demos, so ready set. Go.

A

Yes, thank you.

A

And if you guys want to get settled in go, go right ahead. Thank you.

B

Okay, that's fair.

C

Oh, come on, that's not that's not what I wanted.

C

A

I'm just gonna see what people are doing.

A

um Can you put okay, you already did, um or somebody did no.

B

They're voting up so it means that likely there's at least three people. Oh.

A

B

Yeah years ago, when I entered the promo code for validation, says what do the coaches look like? Oh okay, so they're saying that it should okay that one.

D

A

B

All right there we go.

C

C

B

Can you hear me hello, hello, hello, check check? Can you hear me.

A

Hello, okay, hey everyone. It seems like there's an issue with our promo code, we're going to look into that. If it doesn't get applied. Send me a slack message and I will fix that on your digitalocean account.

A

I will come see you in just a second. We've got three minutes so you're just getting set up. uh If you haven't already, you can fork and clone that repo and then we'll get started on chapter one which is creating a cluster with terraform.

D

D

C

C

E

A

All right, I've already got some slack messages requesting me to take care of the code after the workshop. Thank you. I will definitely do that. All right. I think we're ready to get started.

A

So the first thing we're going to do is we're going to try and set up a cluster using terraform, I'm lucky as the presenter that I have uh you know I have probably better internet access than you do. So, if your internet's timing out or if we're having issues with wi-fi no problem, we'll watch the demo. So, let's uh head to chapter one in the repo.

A

And so what we're going to do is we're going to use terraform to create a digital ocean, managed kubernetes cluster, and so the reason this is part of a git ops workshop is that in git ops you have to use infrastructure as code tools, because you want to be able to define your infrastructure in files that you can commit and track with. Source control like git terraform is an infrastructure as code tool. It uses declarative configuration files that help you automate provisioning infrastructure resources like vms, managed databases, firewalls or kubernetes services.

A

So what we're going to do in this chapter is we're going to use terraform to create a managed kubernetes cluster, and it's going to be a a kind of cluster that you see a lot in tutorials. So this is a diagram from the kubernetes documentation.

A

This diagram shows on the left, a control plane and then on the right. We see three worker notes and that's what we're going to be provisioning in digital ocean with this terraform.

A

So if you're, following along or if you're watching this after the presentation, there are four tools that you need to complete. This chapter, the first, is a digitalocean account. The next is doctyl, which is the digitalocean command line tool. The next is terraform the command line tool and then cubecontrol, so that you can interact with your kubernetes cluster, all right so step one is you wanna have access to the files in this repository, so you're gonna, wanna fork and clone this repo and then you're gonna wanna change into this repo.

A

So on the command line, you see uh the name of this uh repo, which is kubecon 2022 doks workshop.

A

And you've got two sets of instructions there and then the second thing that we're going to do is we're going to configure doctor so that it can communicate with the digitalocean api.

A

We're going to do that with a token, and so I'm going to tell you the steps and then I'll show you we're going to create a token in our digitalocean account and then we're going to store that token as an environment variable and then we're going to use doc tool to authorize our account and then we're just going to make sure that doctor is actually communicating with our digitalocean account so step. One we're going to create that api token.

A

So this is the digital ocean control panel, I'm on the tab that says api and I'm going to click generate new token. You give your token a name so I'll, say kubecon workshop and then you can select when you want your token to expire. It goes all the way from 30 days to no expiration date and you're going to want, read and write permissions. For this token, and then I'm going to generate that token, I'm going to copy that to my clipboard and I won't be able to see that again.

A

So I'm going to be careful with that and then the next step is. I want to store the value of that token in an environment. Variable called do token, so I'm saying export do underscore token, and then I'm setting it to the value of that token.

A

All right, I think that looks good, and I know that different operating systems have different ways of handling environment variables. So you may just want to keep that token on your clipboard and paste it in manually throughout the tutorial.

A

All right next step is: we want to use that token to grant account access to doctyl, so I'm going to run the command doctor off init.

A

And doctor's, giving me a message that says: hey. I need your account token. In order to do this, so it's on my clipboard, I pasted it I'm getting a message. It says that I was able to validate that token and then I'm just going to double check and make sure that worked. Let me check the command, I'm going to say doctor account, get all right and we see the information about my account. My email address like what's my droplet limit, that's our vm product uh has my email been verified, and things like that.

A

So I feel good that I'm able to connect to the digitalocean account from the command line, all right and now the fun part. So in step, four. What we're gonna do is we're going to uh look at a terraform file, I'm going to move through the file really quickly and then we're going to run. Terraform apply. That'll. Take a few minutes, we'll go back to the file at that time and dig into it, then so in the repo that we forked and cloned I'm going to open it with my text, editor.

A

And I'm going to the terraform directory and I have some additional stuff in there, because I've already run a few commands, but doks.tf is the file that we want, and so we see some things here, we're using the terraform digitalocean provider.

A

You see we're going to be using the do token in terraform, and so we have some lines that point to that, and then we get to the good stuff. So here we're going to create a digital ocean, kubernetes cluster, we're going to call it the kubecon cluster, and then we get to set some arguments here and so I've already got my name set cubecon cluster. This next one is important. You want to select the region where your worker nodes are going to spin up their droplets, so what digital ocean data center?

A

Are you going to be using? If you want to see a list of those data centers from the command line? You run this command doctoral, compute region list.

A

And you see the list of all of our data centers and for this exercise you want to pick a data center that is located geographically close to you and you want to pick a data center. That says it's available that you can create resources in there. So I'm from denver colorado in the united states. I usually use the data center san francisco sf03 because it's closest to me and it's available, but now that we're in valencia, spain.

A

um I did some google map sleuthing and it seems like amsterdam, london and frankfurt are all roughly the same distance from valencia.

A

So you can pick amsterdam, london or frankfurt just make sure that in the available column that it's listed as true as available so amsterdam 2, I'm not going to use that, but I will use it looks like amsterdam. 3 is available and so right now I have the london data center specified. I'm going to change that to amsterdam. 3. again pick an available data center, that's located close to where you are, and then the next thing that we're going to do is we're going to specify which kubernetes version that's available through digitalocean kubernetes.

A

Are we going to use so again, this doctoral command will give you a list of those versions on the command line. So this is doctor kubernetes options versions.

A

And there are two kubernetes versions available for you. The earlier one is 121.11 and you need the slug that has this dash do.1 at the end and then the more recent version that we're going to use is 122.8, and so I just want to make sure I've got the right. Slug looks like version. 122.8 is set h, a is for our high availability control plane. We've got two control plane options right now.

A

This one will spin up a little bit faster and then this node pool block is where you specify information about the virtual machines that you're using for your worker nodes. So nothing that you need to change for this workshop, but you select the size of the nodes.

A

I've picked a basic amd droplet with 2 v cpus and 4 gigs of ram, and then you specify whether or not you want those nodes to auto scale, and then you say the minimum number of nodes that need to be running and then the maximum number of nodes that are running so you can have constraints on the size of your cluster.

A

All right, so I said we weren't going to go through that too in depth, but I wanted to go through the arguments. So we've got this terraform file set up and now it's time to run some terraform commands.

A

So first thing you want to do. Is you want to change into your terraform directory and you're going to run terraform init?

A

So this is like running npm, init or any other initialization commands we're just making sure that the software is ready for us to go and that if we need to download any dependencies that that's taken care of, I ran this command earlier, so you wouldn't have to see that so it went pretty fast and then my favorite command in terraform is the next command, which is terraform plan, and so what this is going to do is terraform's going to start communicating with the cloud provider api and it's going to say hey.

A

This is what I am planning on. Creating in your cloud provider. Account. Is this actually what you want me to do. So, let's take a look at that, so I've got terraform plan and then I'm passing in an argument of the variable and I'm saying what I named a lowercase do token in the terraform file. I actually want you to grab the value from my do token. Environment variable all right. So here is the plan command. Nothing has happened. Terraform is just saying: hey.

A

Is this what you want- and um I just I this makes me feel safe as someone who's creating infrastructure.

A

It's saying with these green uh plus signs, hey, I'm gonna create a digital ocean, kubernetes cluster, it's going to be called kubecon cluster, and then you see a lot of those arguments that we set, and you see some arguments that we didn't set that are going to get created in the cluster and then terraform will get that information later, like the end point like the created at time stamp uh like the cube config value, we've got a maintenance policy here we didn't specify any of that, so it's just gonna default uh to whatever the defaults are for digitalocean and then information about the worker node pool.

A

So next thing I'm going to do. I feel good about that. The plan now I'm going to run terraform apply, so this is actually going to create those digital ocean resources.

A

It gives us the same information again saying: is this what you want me to do and this time you have to type the word yes in for those uh things to be created. So now we have this. um This message saying: hey: I am creating a digital ocean. Kubernetes cluster called kubecon cluster terraform gives us information, but I want to see if that cluster is actually spinning up in my digitalocean account. So I'm going to go back to the cloud console.

A

And then, if you look at the kubernetes cluster list here, you can see kubecon cluster. It was spun up just now. All of those things are being provisioned, but terraform initiated that action. I didn't do it from the web console and so that's part of the beauty of terraform. Now I just did it as a human operator, but uh hopefully you can see like how you could make that that's an automated process where you have that infrastructure as code all right so uh just to go back to the terraform file.

A

Hopefully some of these things are looking familiar. You've seen them in a few other contexts, so we've got that variable called do token we're pulling in information from the digitalocean terraform provider, and then there are all of these uh digitalocean resources that we're creating.

A

If you go to uh digital or to the terraform uh provider page, you can see other cloud resources that you can create. So, let's see, I think I have that at the bottom here.

A

So once you get familiar with the terraform documentation, you see here on the left. There are all these things that you can create with digitalocean, so you can create a container registry using terraform a database, a firewall, a droplets, a domain. Basically anything you can create in the digitalocean cloud console or with doctor. You can create with terraform and there are providers for so many things.

A

All the major clouds aws azure and google kubernetes has lots of resources that you can create via terraform, and so that's just some of the power of what terraform is looking through their directory.

A

All right, let's check in on our cluster still creating these clusters are probably going to take between four and five minutes to create, so we've still got a little while to go. uh But let's uh take a look at where we are in the tutorial for chapter one.

A

All right, so I ran terraform apply. I entered yes, I'm waiting for that cluster to provision and I'm checking in a few places. But this message in our terminal is what's going to tell us: hey terraform is done creating that resource, while we're waiting for terraform to do that uh we can get set up to access our kubernetes cluster using cubecontrol, and so uh what we're going to do is we're going to add an auth token or certificate to our cubeconfig file and here's how you do that with digitalocean.

A

So you go to your cluster and on the getting started page. If you go to connecting to kubernetes.

A

You grab this doctoral command and I'm going to open a new tab, so I can do this.

A

And then I have information that says: hey, I added those cluster credentials to your cube, config file and I have changed your cube control context to that particular cluster. So if I say cube control, config get context, I think I'm going to have two my backup cluster and then this cluster yeah I've got both. So I have my backup cluster that I spun up prior to the workshop, and I have the cluster that's spinning up now and I should not be able to access anything in the cluster yet because it's still provisioning but I'll.

A

Try anyway, cube control, get nodes.

A

It's taken a while yep, not ready yet to access.

A

A

And then the last step is what I just did is I want to verify that the cluster is up and running and that I can connect via cube control. So that's the command cubecontrol getnodes. So this is a good opportunity to stop. Let you catch up if you're, following along and we'll take questions so annie will ask questions from the virtual attendees and then, if you have questions, if you're here in person, if you could line up at the microphone.

A

No questions all right.

A

Well, uh we'll take a few minutes we'll let my cluster get up and running, and then we will move on to chapter two.

A

And, like I said earlier, if, if you've gotten what you needed or you want to go see another talk, I will not be offended if you leave now. I want you to have the experience so.

A

Just out of curiosity raise your hand if you're, following along with me, like running the commands, all right, not a majority, but we've got a few cool, and this this repo will be available uh for you after the workshop. If you'd prefer to do it some other time, all right, we'll check see if the progress bar gives us any information.

A

All right, we'll give it one more minute if it's not done we'll switch to my other cluster and then we'll hop into chapter two chapter two is exciting: it's uh setting up flux uh to build a get ops pipeline. Oh and we do have a question. So uh thank you. Go right ahead.

F

Hey kim thanks very much. Can you hear me okay brill, um so I noticed the the api token. The scope of it was very broad.

G

F

Other plans for digitalocean to have like, uh like tighter scopes on the api token. I know it's not exactly related to this, but I kind of clocked that.

A

Sure yeah, I believe, that's on our roadmap yeah.

F

Cool. Thank you very.

A

Much thank you.

A

All right: well, let's hop into chapter two.

A

So this chapter is titled, uh build a get ops pipeline with flux, and I have found that uh the term git ops to me.

A

It feels sort of like the term agile or some other tech terms that get really overloaded with meaning from a lot of different places, and so I'll tell you my understanding of git ops, how we'll be using that term uh in this particular workshop and it's okay, if you have a different understanding of git ops, um but my understanding of git ops in is uh it's a set of practices where you make get the main source of truth for both your infrastructure and your application code.

A

There are tools for setting up git, ops pipelines, things like flux, cd, which is what we're going to install and use today and argo cd, and there are others, but those are two really popular cncf backed projects.

A

So a git ops tool is continuously like watching the state of your git repos, and so, if you make a commit and you push that commit and a change is noticed, then the get ops controller says. I I want to reconcile that. I want the the change. That's in git, I want that to be true in the kubernetes cluster, and so it's that reconciliation process, yeah and uh flux cd helps you do that. Synchronization.

A

And it makes sure that uh your state inside your cluster, whether it's your infrastructure or your application matches what's in the git repo, and so this is what we're going to be setting up in this next chapter.

A

So we've already set up our digitalocean cluster and we already have the control, plane and ncd running, and so what we're going to do is we're going to use helm to install flux, cd flux, spins up a source controller and a helm controller and it communicates with helm and manages releases and then on the right side.

A

Here you see the people, that's us, and so when we make changes to our git repository, we push those changes, we're going to push them to github and and then we, if we add them to a particular directory in our repo uh flex, flux is always watching that uh and when we push those changes, flux does what it can to make sure those changes are true inside of your cluster, so this is hopefully a git ops experience for you.

A

So the prerequisites for this chapter, you need a github account and you need the flux cli tool. So let me just double check on my cluster excellent. So this is the message from terraform: hey. I created that digitalocean kubernetes cluster. I just want to verify. I can connect uh cubecontrol get nodes.

A

Excellent looks good all right, so installing flux.

A

The first thing uh is: we are going to bootstrap flux, cd and have it installed in our cluster and what we want flux cd to be able to do is we want it to have the ability to create github repos and make changes to that github repo and so we're going to do that through a github personal access token. So, if you've never done this before you go to your oh, that's the documentation, which is also good, but I'm going to go to my github account.

A

And you go to your your icon and then you find the settings and then all the way at the bottom. It says: developer settings and there's get up, github, apps, oauth, apps and then personal access tokens. This is what we're going to create for flux to use, so I'm going to generate a new token, all right, I'm going to unplug this real fast. While I.

A

Get my github password in there.

A

A

What's this token, for we're going to say, flux, cd, uh expiration, we'll go seven days, and then there are all of these different scopes.

A

We want flux, like I said, to be able to do all the things to repositories in my account.

A

So I'm going to generate that token and I'm going to copy it on my clipboard like before, and then again we're going to store this as an environment variable in our terminal session. So I'm going to create an environment variable called github token and then give it the value of that token. I've got on my clipboard, so uh I'm gonna go up a directory and I'm gonna say export. Github underscore token.

A

All right- and next thing this is so this is so exciting. I love this um so we're going to run this flux, bootstrap command. It's going to create a repository on my github account, and I just need to make sure that anything in the angle brackets that I change the value of that.

A

So I'm going to copy this whole command.

A

I'm going to paste it here, it's got a couple lines, so, first, I'm just saying flux: bootstrap, github flux can bootstrap all sorts of different environments. It can do bit bucket, it can do git, it can do other things as well, uh but I want this to get connected with a github account. uh Next thing is the owner of the account. That is me. So this is my github username kim schles.

A

I want it to create or pull from a repository. That's called kubecon workshop, the path that flux is going to be monitoring and then doing the reconciliation process with is called clusters dev, and so I'm going to hit enter, and this takes a little while so it says connecting to github. It looks like the repository was created.

A

And it's syncing, so let's go look at my github account and see. Do I have that repo.

A

All right it created this kubecon workshop repo, nice all right and uh we've got all this information, so it says deployment is ready. All components are ready. So let's see what got created inside this kubernetes cluster, so I'm just going to say, cube control get namespaces, so I have the default namespace and the other namespaces that come with a cluster, but I also have this brand new flux system name space. So, let's just look at the pods in there cube control get pods from the namespace flux system all right.

A

So if you remember that architecture diagram, these should look familiar. So we've got the helm controller, which is running in a pod. We have the customized controller, so flux really uses customize, a notification, controller and a source controller, and so just to look at this again, you see those components now in the kubernetes cluster created through that bootstrap command.

A

And then there are some nice flux commands where you can check that things are working. So if you run flux, check, it'll check prerequisites last night, I did this and it let me know my flux. Cli was out of date, and so I updated it. You get some good information from that. If something fails, you can run flux logs to investigate and try to find out. What's going on, this looks good to go.

A

And then we can say, flux get all to inspect the uh resources that flux has created.

A

All right says: hey, I created a git repository um and I'm gonna be looking at that all right.

A

I think we're good on that. Oh yeah, and then what was built into that repository, it looks like it should have three different yaml files. Let's, let's take a look at that inside the kubecon workshop.

A

We've got this sort of long file path, but we have those three three files, we've gotk components got k-sync and customization, and these are files that are auto generated by flux, and uh some of them are long. You shouldn't have to mess with them and they have some nice uh like a nice warning at the top. This manifest was generated by flux. Do not edit, so you don't have to worry about those all right. So we have flux set up. We have a repo that flux is listening to, and so the next step is.

A

We want to clone that repository onto our local machine and we want to prepare a little bit more of the layout so that we can have the sealed secrets. uh Chapter work for us, so this command is a git clone of the repository that you created with flux. So I'm just going to grab this I'm going to step out of that directory, I'm going to say git clone.

A

And I'm going to change into that so kubecon workshop, excellent and I'll open that with my text, editor.

A

All right next up.

A

We're going to create some new directories in that and it looks like we have a question so go ahead.

E

Yes, two questions. Actually, so how did flux authenticated with github because you did not provide the github token, so I would assume it took it from your terminal and how did flux commun authenticated with kubernetes cluster before creating the namespace.

A

Excellent, so two good questions: one was how does flux authenticate to github and then how does flux authentic authenticate to kubernetes?

A

uh I believe that the yeah that um environment variable that we set the github token that flux is looking for an environment, variable name that in your terminal session and then flux also, I think, has access to your cube, config file so that it can communicate with your kubernetes cluster thanks.

A

All right, so this is where it gets a little a little bit more complicated, but we have some nice commands here for you. So I want to create uh several different new directories. So one is going to be called helm. One is going to be inside the helm, directory repositories, then releases and then secrets so copying. This command should get you there.

A

And once again, it's the clusters: dev directory, is where you have to put anything that you want flux to look at, uh because that's what we specified in the flux, bootstrap command, telling flux- hey look here, so, let's make sure that we have uh those directories. We've got excellent, so we've got flux. Okay, we've got helm releases, repository secrets, that's what I am looking for all right and then finally for flux, uh we want to add some items to our git ignore file, um and so the first thing is uh we're gonna.

A

This is for a sealed secrets. We're gonna ignore anything uh with this values, uh dash dash values, dash dot, yaml and then we're not going to ignore sealed yaml files and we'll see what that is. I think I already have that in the git ignore, but let's double check.

A

Yes, they are here, those things are ignored, all right, so that is bootstrappingflux to communicate with github and your kubernetes cluster. And now uh the flux controller is running and it's going to take care of any of that reconciliation and we'll see that in action in just a few minutes in the sealed secrets chapter.

A

But since that's the end of the chapter now is a good opportunity for questions. So we'll take questions from folks online and then, if you are in the room and have questions, if you could queue up at the mic.

B

Yes, so there's a question online. What would be your recommendation regarding git repost structure, ea, the folders for close cross-plain providers config, along with the sealed secrets, resources, benefits and so forth when working with git ops.

A

um I'm going to try and restate that question. So the question is: how do you approach structuring your git repo? Does that sound right, annie yeah? So, if you're following along, I would do exactly what's written in the uh the tutorial it's sort of a mono repo, where everything's all in one repository, but the thing that honestly that I struggled with the most was realizing that you have to specify for flux which directory that it's going to be monitoring or which directories and so being mindful of that yeah.

A

For this tutorial, we've got a terraform directory, we're going to create some sealed secrets. Files we've got cross planes. We've got a lot of stuff going on, but I would I would stick with the tutorial for now and then, once you get familiar with the tools, then you'll have more confidence to restructure it as needed. Thank you all right. We have a question.

H

Hi yeah, uh I have a question. You use personal token uh for github to connect to flux, as I think right.

C

H

Is it? Is there a way to avoid that, because personal token is connected to you know a person and if a person leaves the company, uh you have an issue. So is there a way to do work around that.

A

Yeah, that's a great question, so the question is we used a personal access token from github to give flux access to our our our github account, so it can create and make changes to a repository.

A

So you could uh do that through, like your work like the work organization and I'm sure there are other ways of handling that, but I don't know them off the top of my head. So I can do some research on that or I can hop over to the flux booth after this and talk with the flux maintainers. But I'm guessing there's a better way.

H

Okay, thank you. Yeah.

A

Good question: all right: um we've got about 40 minutes left, we've got two chapters, um oh we've got one more or we have more questions. So, uh let's, let's take that question.

C

um I've seen that you bootstrapped the the flux stuff with an imperative command. There is a way to do it in a decorative way.

A

Oh, is there a where? Oh that's a great question, so the question is we used uh like an imperative paradigm with flux uh where we ran the commands from the command line and we passed in the arguments? Is there a way to do it, declaratively so using like infrastructure as code? That is a great question.

A

I don't know the answer right now, but I'll find out and share it in the slack channel or if you do the research and find the answer, let me know and I'll share it with the group so uh excellent question yeah in the spirit of git, ops and infrastructure's code, uh not running those uh commands uh imperatively, all right. uh Well, let's take just a minute. If you need to stand up and stretch, uh you can do that um and then we'll hop into the next two chapters.

A

Oh say that again the slack oh yes, my name is kim schlesinger.

C

C

H

C

A

All right, so we've spun up a cluster with terraform we've installed, uh flux, continuous delivery, and the next thing uh that we're gonna do is look at a project called sealed secrets.

A

So in get ops and using infrastructure as code, if you want git to be your source of truth, you want to be able to publish files and not worry about it.

A

But one thing: that's really tricky is when you have confidential data like api tokens or passwords or database connection strings things that you don't want to commit to get or github or gitlab whether or not the repository is private things that are not safe to put in there, and so there are a lot of different solutions for this in kubernetes and today I'm going to show you a project called sealed secrets.

A

So let's look at the diagram, so you can see our cluster is getting a little bit more complex. So we have our digital ocean kubernetes cluster. We have flux cd inside of it. We've got our control, plane running and then on the right here you see, there's a sealed secret object and a sealed secrets controller.

A

So we're going to be installing that inside our cluster.

A

And then on the outside of the cluster like where the developer or the platform engineer is we're going to encrypt kubernetes secrets using a tool called cubeseal, and then that's going to give us an encrypted string. That's okay! To push uh to your git repo and uh inside of sealed secrets. uh There is a private key that will decrypt your secrets inside the cluster. um So that's how we're going to handle that.

A

So, let's, let's see what we're going to do so in order to complete this chapter, you need to have completed chapter two. So you'll want flux, cd set up up and running, uh using the same structure that we did uh in the tutorial and then you'll also want a command line tool uh from the sealed secrets. Project called cubeseal all right. So let's hop into setting up sealed secrets.

A

So the first thing that we're going to do is uh we're going to create a flux, custom resource called helm repository, but we're going to set up a helm repository for sealed secrets. So, let's see the instructions say change the directory where your flux, cd, git repository, was cloned. So let me make sure I'm in the right projects yep, so I'm in kubecon workshop and then we're going to use flux to create a sealed secrets. Helm repository with this command and I'm just looking through- is there anything that I need to change.

A

I just need to make sure that I have an environment variable called flux, helm manifest path. So let me check that.

A

Yes, so those are some of the directories. We set up clusters dev and helm, so I'm good to go there and then I'm going to copy this and paste it all right. So it says flux create source, helm, sealed secrets.

A

We've got some explanations for the commands and we're going to have a new file.

A

Oh wrong wrong place, and here it is so it's a yaml manifest we're using a flux, cd, custom resource, it's called helm repository and so, if you've ever run helm from your command line, where you have to install a helm repository and then specify the chart that you want to create.

A

The release from this is how flux does that declaratively in a yaml manifest, so we're saying, hey, create a helm, release called sealed secrets, put it in the flux system, namespace and then you're going to pull it from the sealed secrets directory or from the sealed secrets repo.

A

Actually, that's just the helm repository, not the helm release, but that's. Okay, all right next thing that we need to do is uh we want to have some value set for the sealed secrets, uh helm, release and so we're going to set the version of sealed secrets that we're using and we're going to pull a values file uh from a different project. So we'll just run this command and I'm expecting to see a file that starts with sealed secret values.

A

Yep in my kubecon workshop file so looks good. Not a whole lot of value set here, but we have ingress is not enabled right now, all right next thing. Now we're going to create the helm release. We have the sealed secrets repository um and so looking at this long command we're creating a helm release called sealed secrets, controller specifying the release name, the source, the chart, the chart version, the values that we're going to apply to that helm, chart and where that file is going to be created.

A

So, just in the root of the kubecon workshop directory running that command and then we should see another file. Oh it's in repository! Oh, no, that's wrong! It's in releases all right! So we have this helm release with all of that information, so declaratively, creating a helm, release.

A

And then next we want flux to see those changes and to enter into the reconciliation process, and so we're going to add all of those files as a git commit and then we're going to push that commit to github and then we're going to see helm, uh create those uh new resources. So um in this command still setting the sealed secrets, chart version and just adding the new files that we created with those flux commands.

A

All right so adding those I'm pushing that to my github repo. Let's just take a look at that and make sure those changes.

A

Okay, I was updated 10 seconds ago great and then uh let's, let's just do a let's do a flux logs and see if we can see what's going on all right. So we see flux is in that reconciliation process and we see like a helm chart is being created. Oh it says no artifact available for helm repository and then, let's take a look at what's oops cube control, get namespaces.

A

Let's see what's in the flux system, namespace, what's new, so cube control get pods from the flux system, namespace.

A

All right, so we've got the sealed secrets, controller, which was created 40 seconds ago, so that was installed because we made that change in git and github and flux noticed the change and then did that reconciliation process. So we've got the state of our cluster now matches the state of our git repo, which is uh get ops, making it the single source of truth.

A

All right um so the flux controller. I think it runs uh one minute every minute by default. So if you apply a change and you want to see it happen right away, uh you can run this command where you force a reconciliation, flux, reconcile yeah and then just to see that helm release one more time.

A

All right says it's ready and that reconciliation succeeded.

A

All right next up, we are going to export the sealed secrets public key, so a public private key pair was created inside of the sealed secrets controller and we need to have access to the public key. uh So we can use it to uh encrypt our our secrets that we don't want people to see, and so uh this happens in one of two ways. So you can try this cube seal command where you're grabbing the controller from the flux system. Namespace excuse me and you're asking for the cert and you're saving it as this file.

A

This doesn't always work for me. So, let's see if it works, that didn't work for me, and so what I'm gonna do is I'm going to port forward the sealed secrets controller and uh then I'm going to uh yeah use a curl request to get the certificate from the api endpoint. So I'm going to copy this, I'm going to open another tab.

A

All right, so it looks like that pod is port forwarding and then I'm going to run this command and I want a pub sealed secrets: dot pem file in my kubecon workshop directory.

A

Oh it looks like already. Oh maybe it did work that first time. Oh no, it created the file, but there's nothing in there all right. So did that curl request succeed. Yes, so now I have the public key on my machine.

A

I can stop that port forward process and back to the instructions. So I now have the public key and oh, it looks like it's safe to commit that to git. So I'm going to commit that and push to github.

A

Excellent, all right next thing we're going to encrypt a kubernetes secret. So this is what I think is the exciting part is we're going to have some data that we wouldn't want to have in a git repo and we're going to encrypt it using cubeseal and sealed secrets.

A

So I'm going to create a kubernetes secret and I'm going to call it your dash app dash secret.yaml.

A

So I'm going to say, touch your dash app dash secret dot, yaml.

A

And I'm going to paste this yaml manifest, so this is a secret, it's called your app and then here is the data. This data is base64 encoded, but that isn't uh strong enough to have it out publicly and so what we want to do is we want to use cubeseal to transform this string into something that can only be decrypted by the private key. That's in the sealed secrets controller.

A

So how do you do that.

A

So we're going to run a cubeseal command, we're going to ask it to give us output. That's yaml we're going to have it create that new, updated secret in the flux system, namespace we're telling it the name of the public key we're telling it the name of the file where it's going to uh encrypt the secret and then we're giving it another file name. It's called your app sealed! That's going to be the file that we commit to github.

A

So let's run this command.

A

Let's see do we have your app sealed dot yaml, so I have your app secret, that's the one I don't want to commit. I have your app sealed. ah Here's the one I want to commit, so this has been transformed from a generic kubernetes secret into a custom resource from sealed secrets, called a sealed secret.

A

It's got some of that information we passed in through the commands and then look at this encrypted data. This is safe to commit to a public github repository, so you are good to go with having safe and secure secrets.

A

So what I'm going to do is I'm going to delete that file that has the secret in it and I'm going to keep the sealed file and then I'm going to commit that to git github and then sealed secrets controller. We'll take a look at that it will decrypt the secret and then we'll take a look at that.

A

A

All right, I already got rid of that your app secret file- oh, this is beautiful. I don't even have to run a flux command. I can just use cube control, apply to create uh that sealed secret. So all right, cube control, apply your app sealed.

A

Excellent and then, if we look cube, control, get secrets from the namespace flux system. Oh there's a lot there.

A

All right, so we have uh the your app secret uh and it's opaque. It was created 10 minutes ago, so that sealed secret got turned into a regular kubernetes secret.

A

And we can inspect the secret to see if it has the decrypted data. So let's see all right so there's the base64 encoded string that sealed secrets uh decrypted. So we've got that secret in the kubernetes cluster, but it wasn't committed to git or to github and then we're not going to do this step but optionally.

A

If you would feel safer, you can create a private key backup and we've got some instructions there and then a list of some security, best practices and then some resources. So we just use sealed secrets to uh create the sealed secrets controller so that we can encrypt secrets from our command line, commit them to git github, and then they are decrypted through the sealed secrets: controller in your kubernetes cluster.

A

So that is the end of that chapter. uh We'll take questions so we'll do questions from the virtual attendees first and then I see folks in lime and then we've got one more thing which I'm super excited about, which is using cross plane to spin up some cloud resources and then we'll be done.

A

Any questions from uh virtual attendees annie. Okay, no questions from virtual attendees uh people in the workshop yeah.

G

My question is the crd, so the select secret, the file that we created you.

I

G

The cube cut will apply, but so this file isn't managed by flux.

A

Oh yeah, that was not managed by flux. We could oh did we commit it? No, we didn't, but if we committed it, I think flux would reconcile that.

G

So it's not in a specific directory is at top level right, yeah.

A

So yes, oh that's a good point. You'd have to put it in that dev clusters directory.

G

Okay, so it's not managed if it's there. Okay, that's true yeah! Thank you. Uh-Huh.

J

Hi, I would have the same question uh but another one: okay. What would you recommend to not accidentally commit your unsealed secrets file? Maybe a pre-hook pre-commit talk, something like that. That's.

A

A great question, so the the workflow that I showed you where you created the secret with the base64 encoded string and then you manually deleted it. I think, having a a guard rail like a web hook or some check on that is a really good idea, because it would be very easy to accidentally commit that so yeah putting some boundaries around that some automated processes to check that would be would be a very good idea.

A

J

Hi hi, where is the sealed secret private key stored.

A

The sealed secret private key is stored in the sealed secret controller pod. So when we did the port forwarding and we asked for the public key, the private key is also stored in that same place. Thank you, yeah all right.

A

I wasn't sure we were going to get through all the chapters, but I think that we are we've got 20 minutes left, so we will do some crossplane and then, if you're, following along I'll, show you how to tear down your cluster and clean up any resources. So you don't get charged and then we'll be on our way.

A

So this next chapter is called make your cluster a universal control plane with cross plane and so crossplane is a cncf backed tool that allows you to create cloud resources from inside a kubernetes cluster. So you can make a cluster a universal control plane. um I think crossplane's really exciting. It took me a couple of days to sort of understand uh some of the exciting parts of it.

A

But what we're going to do in this part of the workshop is we're going to install the digitalocean crossplane provider in our cluster and then we're going to spin up a digitalocean droplet, that's just totally separate from our cluster, so you can see us provision some cloud resources, so the next part which we probably won't get to is uh that you would pick a different provider. So all the major cloud services have cross plane providers. Big projects in the cloud native world have uh have have cross plane providers.

A

Cloudflare has one so hopefully, when you see the pattern of how you install the provider and create the resource with digitalocean you'll be able to apply that pattern to another provider and get up and running with crossplane.

A

So this chapter has no special prerequisites uh aside from uh the tools that you need for all the chapters and having a kubernetes cluster, and so the first thing that we're gonna do is we're gonna just create a separate name space uh where we do all of this installation, so we're going to say, cube control, create a namespace called crossplane system and I'm just going to make sure it did what I expected all right cross plane system next up.

A

So I'm doing this all via helm and, like some of you have already pointed out, you could do this all in flux. Absolutely uh you'll you'll have to use your imagination, though I'm going to do this just from the command line, so I'm going to add the crossplane uh repository and then I'm going to update it. uh I'm guessing I already have that in my helm repository so I'm just going to run the helm, repo update command.

A

Excellent and then I am going to use helm to install the cross plane helm chart in that cross, plane system name space so through this command.

A

Excellent, so getting the information that crossplane has been installed. Let's take a look we'll just look at the pods in the cross. Plane system name space all right, so I have a cross plane pod and I have a cross plane, our back manager, pod excellent. So that's the first step if you want to use cross plane for any of the providers.

A

That's the first thing you need to do is install cross plane somewhere in your cluster, now we're getting into the provider specific instructions, so we're going to install the digitalocean crossplane provider, and I have that defined in a file called install.yaml.

A

So back to the repository for the workshop going to crossplane and just looking at the installation manifest.

A

So I'm using a crossplane crd called provider and I'm creating a provider that I'm calling provider do for provider digital ocean and I'm installing it with this particular package. So I'm going to run a cube control apply command on that it's in the cross, plane directory. Oh I'm, in the wrong directory.

A

All right, let me try that again, so cube control apply the cross plane, install manifest cross plane, install.

A

All right it says that was created. So now, if we say cube control get provider, we've got one provider, it's the provider, digital ocean. If you were using multiple like if you were using a google cloud provider, you would see that one as well. If you were using cloudflare, you would see that provider installed, so we're ready to uh talk to the digitalocean uh api or almost ready uh getting there to talk to the digitalocean api from this kubernetes cluster.

A

So next step is we want to configure two different things: one's uh a secret and one is the config provider, and so I need that digitalocean token value again, let's see if I'm still in the same session excellent. So I'm gonna use that in just a minute, and uh we need to put that in this file, so we have the config.yml.

A

So here's where the token is going to go, there's a placeholder here called base64 encoded provider creds. I just have a little comment here. This is an opportunity to use sealed secrets, so I won't be able to do that right now, but this is a place where you would use sealed secrets. So you could commit this file up to a public repository, so I have to base 64 encode this value and I have some instructions if you've never done that before you can do it from the command line.

A

I'm not sure how it's done in windows, but uh and I'm not sure how much you trust duckduckgo. But if you, if you search base64 and then a string value and duckduckgo it'll, give you the base64 encoded value.

A

So I'm going to echo that.

A

All right so there's my base, 64 encoded value. You've got to have that for a kubernetes secret, all right. So here's how crossplane's going to communicate with my digitalocean account. Using that token and then the next thing that we're going to do is set up this custom resource called a provider config. It's going to be called do example, and it's going to pull the credentials from this secret that I created above the cross plane. uh This uh provider do secret, so I'm gonna apply the config file, so cube control apply cross, plane, config.

A

All right, no error messages, so it looks like it's good to go and then the next thing. This is where I think it's so cool um is we're going to create a digital ocean droplet from the cluster from inside that kubernetes cluster and so uh with crossplane uh any like resource that the provider offers you can probably create it using a yaml, manifest and so for digitalocean there is a custom resource called a droplet. You give it a name, and these things should look familiar.

A

You say which region, which data center region that droplet's going to be created in the size of the droplet, the image you want that droplet to use, and then the provider config ref it just has to be the same name as the config ref that we that we created here. So let's look at our cloud console.

A

Let's look at droplets.

A

So these are droplets for the kubernetes clusters that I have running in this account and what I want to happen is, I want a cross plane droplet to get created. So uh let's do it so cube control apply, cross, plane and then droplet.yaml.

A

It says the droplet was created and if we look at the cloud console.

A

Hey the droplet was created so that droplet was created from that kubernetes cluster. I didn't create it from my command line. uh I didn't create it. uh You know using doctor or from the web console, but just like a little a little bit of the power of what cross playing can allow you to do in terms of being a universal control plane. So that is how you create a droplet uh on using the digital ocean cross plane provider if you're working through this tutorial in the future.

A

So the next steps is that you would choose a different provider to install and the steps are very similar. You install the provider you set up, however, they like to authenticate, probably a secret with a token, and then you find a resource that you want to create from your cluster, and this is the official list of cross plane providers.

A

So you see aws gcp, azure, alibaba, you've got rook, helm, terraform sql get lab equinix metal, sivo, argo, styra, cloudflare, so many things. So, um just imagine if you want, like your kubernetes cluster, to sort of be like the mothership. It can create all of these things, and this is a way for you to implement multi-cloud infrastructure. uh Maybe you use digitalocean for something, but you also use uh google cloud uh resources for something else. uh You can set up a cluster that can handle all of that, so that is the cross plane.

A

Part of the tutorial and crossplane does so much more.

A

I wrote a blog post about it for digitalocean a few weeks ago and the power part of the power of cross plane is that if you are a platform engineer, you can create different resources that you give to application developers that maybe they don't want to know much about your infrastructure, but you want them to be able to spin up like databases or or whatever you want, but crossplane also gives you that power or your platform engineering team can define special resources and tell your application team hey.

A

If you need to do some testing or you need to spin up this or that here's how you do it. So I can't speak highly enough of crossplane. I think it's a really exciting project, so that is the end except for one thing of the workshop. The last thing is uh destroying your cluster with terraform, and so um you don't want to get charged for having resources running that you're not using, and so you can just destroy your cluster with a terraform destroy command, so I'm changing into the terraform directory and I'm destroying it.

A

Terraform is like hey. Do you want me to get rid of all of this stuff and I'm going to say yeah I do and it's destroying that and then I'm going to just manually destroy this droplet that I created.

A

Excellent, and that is that so we'll take questions um and uh if, if you haven't left already, I would love if you, if you learned something or you want to ask me- questions come up to the front. I have a few t-shirts and tote bags and a ton of digital ocean stickers. I'd love to meet you and talk about that again. Slack me if you need those credits, apologies that the credit codes didn't work, but we'll take questions from online questions from uh the folks in person and then we'll be done.

B

Perfect, um so there was two questions from the q: a box uh can sealed secrets, convert a yaml with string data, so you don't need to base 64 encode as you're, removing removing a secret file after you've, converted it to a sealed secret anyway.

A

Okay, so the question is: can sealed secrets uh does it? Do you have to base 64 encode this the secret before giving it to sealed secrets, and I believe it's a kubernetes uh constraint that the secret data must be base64 encoded, so so, yes, you have to have that there great.

B

So how should we maintain the universal control plane, which is basically another cluster in a git ops way? Is it recoverable if it goes down.

A

Great question: so how can we maintain like the universal control plane in a git ops way?

A

I guess you would want to have a very strong disaster recovery plan, so planning for let's say this cluster goes completely online, like what is the plan uh to handle that?

A

uh So that's like the scope of a different talking workshop, um but um yeah I uh like using all of the things that we were exposed to today, like terraform flux, maybe use argo cd instead like finding a way to like get all of your pieces in a git ups workflow, so that if you have a cluster that goes down and everything is defined in git in theory, you would be able to spin up that cluster using tools like terraform and flux.

A

So that's how all right a couple questions from the audience.

J

Thanks there was another question in the channel I want to highlight: it was about using flux with, for example, 100 microservice, so multi-repo. What would you recommend for these amount of repos regarding conflicts, for example,.

A

Yeah, so the question is what, if you are running 100 microservices and they you know each have their own individual, git repo. I think flux would be able to handle that you have to configure flux to. You know, listen inside of those repositories, but if you have like a good templating system for your teams to copy and use, I think I think that would that would uh help you use flux for all those micro services.

J

Different different rappers can have conflicts between each other right. How do would you resolve them? Oh.

A

That's a good question, I don't know I'll, try and find out.

J

D

Hi hi, I noticed that you uh removed the crossplane created resources manually. My question is whether crossplane has the ability you know to manage the drift or manage the lifecycle of the cloud resources.

C

D

A way like argo city flux, does it.

A

Yeah crossplane does have that capability. We've got a bug in the uh digitalocean provider. So that's why I had to do it manually, but yeah you can destroy you. Can you can use crossplane to get rid of resources as well good, good eye?

A

C

A

Right thanks so much everybody thanks for those of you who are watching online, uh I'm kim come, say: hi I'll, have a mask on get some swag reach out to me on the cncf slack workspace or send me an email, especially if you need those credits we'll get those applied as soon as possible. Thanks so much.