GitLab #EveryoneCanContribute cafe, 3 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 15. #everyonecancontribute cafe: Kubernetes deployments to Hetzner Cloud, step 2

Description

Last week's session (recording & URLs included): https://everyonecancontribute.com/post/2021-01-27-cafe-14-kubernetes-deployments-to-hetzner-cloud/

Blog post: https://everyonecancontribute.com/post/2021-02-03-cafe-15-kubernetes-deployments-to-hetzner-cloud-part-2/

Repository: https://gitlab.com/ekeih/k3s-demo

Questions? Join our Gitter channel: https://gitter.im/everyonecancontribute/community

A

And we are live yeah for the second step or the second part of our amazing workshop, how to uh deploy kubernetes uh kubernetes cluster to redstone cloud, and I think with that I just hand over to our expert max who wants to continue and dive into kubernetes, k3s and and much more.

B

All right, yeah, thanks um yeah, if you watched uh last week or if you watched the recording, then you have quite a good idea of what we already did yeah we started last week using teraform and ansible with terraform. We set up some virtual machines and networking on headset cloud, and then we started to use ansible because we want to use it to deploy things inside of our virtual machines.

B

This week is a little bit more improvised than last week, so don't be surprised if it gets chaotic, but sometimes that's half the fun to just try and look what breaks and then we will work together to fix it.

B

So after last week, I deleted everything in this project, so there are now zero servers in our everyone can contribute project. So to have a little bit of a recap of what we did last week. We are going to use terraform again, therefore go in here and we run terraform apply.

B

And if you remember from last week or in case you don't- and you are already watching here for the first time we are using the gitlab state backend for terraform, um but from last week it's already configured so terraform notices that our network configuration and all our servers are still missing.

B

So let's say yes.

B

And in a few seconds we should see those servers being created.

B

We are using one server where we will deploy our control plane and then to workers or agents. We will deploy some workload, so our machines are now already running so last week we created our virtual environment for python and installed ansible. So, let's switch to this environment.

B

I think it's called everyone can contribute, and then here in our ansible directory, we don't have much there. Yet we created this dynamic inventory to create some groups based on the labels of the virtual machines.

B

So I think last week the last thing we did was something like ansible inventory list.

B

And there we have it all our three servers and then what we did was to run ansible with a ping module which basically just verifies that the ssh connection to all machines.

B

Works- and here we see it works as expected, so the goal for today is that when we end the call, we have a running kubernetes cluster running running kubernetes cluster, for us will mean that we use cubecontrol or cubectl or whatever you want to call it. It's the cli tool to talk to your kubernetes cluster and what we want to have working there is that we get some nodes and we also want to retrieve some pots.

B

If this works, then I will be happy. So that's the goal for today and right now, our virtual machines are quite yeah, unconfigured, there's not much in it. Just running linux, ssh server, not much more before we start with k3s. As our qmanages distribution, we are going to play around a little bit with ansible.

B

So as a first thing, we will. Let me check my notes. We will create a role or we will use an existing rule to be specific to.

B

Yeah to configure unintended upgrades on our nodes, because before we start out to do more with ansible, I thought maybe, let's start using an existing role and just see how it works.

B

So last week we configured this dependency here that we wanted to use not only the hexner cloud collection, but also the devops collection, which is a collection of ansible roles, to configure some things and to set up react, common software and under debian or ubuntu. You have this package called unintended upgrades which will take care of updating your server without your interacting with it.

B

So, for example, you can configure to do all the security updates and reboot your machine during the night and because we don't want to take too much care of our server when it's running. We will just configure this to automatically apply some updates, and during this we will learn a little bit about ansible.

B

So far we have seen two ansible commands. One was ansible and the other one was ansible inventory with ansible. We called like this one module. We only wanted to run this command ping connect to each server and check if it works.

B

Next thing we will learn is the ansible playbook command, which is a command to yeah, do more things uh in the at the same time, to like to have a playbook, you run through several steps.

B

What you want to do- and we will create this so in our ansible directory, we will just call that main yammer and there we will tell ansible what to do with our servers and our first stop will be some common configuration and just a minute ago I mentioned that we are going to use this collection and I'm not sure why, but the devops collection is splitted into four collections.

B

I guess that this is due to some size limitation or something that they have so many roles that I decided to put it into three different, not entirely sure, but we are going to use this from their documentation and just accept it this way, because it doesn't matter too much for us, so, okay cool, for what hosts do we want to do it? We want to do it for all our hcloud hosts this each cloud. Where does this come from when we listed our inventory down here?

B

We had this automatically created group for us hcloud, where all those whole story.

B

All right and then we are going to start use some rules.

B

And ansible will search through those collections for this role and then with ansible. You often configure host specific settings in variables, so you can use an more general role and then apply your specific settings and in the devops project, which created the roles we are using, they yeah they did this a lot. So we can configure the whole software with variables and before I do 100 typos, I will just copy it over here.

B

It's nothing fancy! For now we are just telling the road to not how to reboot our notes and to remove unused packages.

B

Then one nice thing you have an ansible and you usually have in all config management. Software is that you can assign some tags so later on. If you want to run only those steps, then we can limit this to the stack all right.

B

So this way we already have our first set of yeah things we want to do for now. It's just apply this role to all nodes. So if you have never used and any configuration management or never have you just answered at all, then this might be a little bit abstract. So, let's look what it what it's like in action: let's call the ansible playbook command and pass the main yammer as a playbook.

B

Then we are going to use the check parameter so for now we just want to see a diff, so we don't want to apply the changes yet and it does not work.

B

There's some syntax issue.

B

Just in the beginning, it's.

B

A

I think it has to be rowitz instead.

B

Yeah yeah, thank you. You're right.

B

Yeah now it looks better so now ansible in the first step, it will usually connect to all nodes and gather some facts like what is already installed. What kind of linux is it? Is it even linux?

B

What ip address is what disk what is available and then afterwards it will continue with those steps of this unattended upgrades role, then we can see that it tries to oh. If we run it without this check mode, then it will apply those changes to some configuration files.

B

All right, no, it's not done yet.

B

And one of the nice things is that usually you can find a lot of roles or for common software and you can use them or you can get inspired by them, how they have written. And then you can decide that you want to do it a little bit in a different way. Then you can write your role based on it, but at least you have a starting point and all right, then: let's do this without the check.

B

B

Okay, so while this is running, we can look for a second into our headsnow web interface. Last week, when we wrote our terraform code, we configured a network and in there we created this sub network, called, I think, k3s exactly, and there are our three servers in it so network wise.

B

This means those nodes can, or these virtual machines can communicate with this private ip addresses, but on a when we are looking at or if we want to really have a secure connection between those nodes, then we have to realize that there is no encryption between them. There's no automatic safety, it just protects us from communicating over the internet. It's like it's heads now makes it more secure for us, but it's not really secure yet because everyone who has access access to this headset infrastructure can capture those traffic and look into it.

B

Now you could say: okay. If someone has access to this headset infrastructure, then maybe they have access to our servers anyway, and that's probably true, but for this demo we will still go the little bit advanced route and use wire guard to encrypt all traffic between those nodes and with k3s what we are going to use. This is quite easy because it's mostly a configuration setting, so our nodes will communicate with us private ip addresses, but in addition to that they will encrypt all traffic.

B

So if we switch back here, we see that ansible is done now it changed things on seven servers, and if we do it again, then we should see that it does not change anything because the whole, because the whole point of answer is that we run it several times and if there's nothing to do it won't do anything.

B

And while this is running, we can continue a little bit with this wireguard stuff. I just mentioned as a first step. We used this already existing rule, so now we want to create our own rule to setup wireguard.

B

So let's create a directory.

B

Let's call it roles because by default, ansible will look for this directory, let's create another directory and let's call it wireguard, and now we will use some yeah. The default setup of ansible and ansible will look for directory called tasks and in there it expects some younger code to tell it what it should do and we will create this. It's the directory and in there we will create a file and call it mania because ansible we use that this as a starting point.

B

If you have a more advanced rule, then you will usually start to create several files in here and include them in your main file, if necessary, but for the wire guard rule. We will keep this very, very simple.

B

So maybe, if you're wondering how do I figure out what to write here?

B

A good approach is to just go to the ansible docs, for example, look for the to install and debian or ubuntu package with the apt module, and then you can figure out what options exist and for all ansible modules. This kind of documentation exists.

B

If we go back here, we will now create our job or our task, not our job. This is now uh arbitrary name. We can do whatever we want, can call it, for example, install wireguard also here this prefix is just uh just a string, so just for us to know which package we use there and then we use this module we want to have. I want to install a ubuntu package called wireguard.

B

B

And if you already installed some packages on dbn or ubuntu, then you will know that usually you run something like apd update. First, to update your local cache of packages to your local index. What exists and we can tell ansible to do the same.

B

To make sure that it installs the current package or that it even was able to find this package, but now, if we run ansible several times, it will update this cache each time, which is a little bit of time consuming, because each time it has to contact a few servers, download things, yeah pass them and everything. So we don't want to do this every every time.

B

We only want to do this if the cache is older than 3 8 600 seconds. I think that should be an hour and then now, if you run it the first time it will do this update and afterwards it won't do it anymore. If you do it several times, but if you do it tomorrow again, then it will update the cache and that's all we need to do with wireguard.

B

We don't have to configure it ourself, but we need to install the package, so the kernel module and the command line tooling and everything is installed because later we will see that our k3s uses this installed wire guard to set up some vpn configuration.

B

All right created this rule; now all we have to do is to use it.

B

We also want to do this on all our servers, so we do it in this same same block here and we tell it to use raw wire guard.

B

And that's all because our wire guards roll is so simple. This has no configuration for to be a little bit more consistent can add a tag for this as well, and that's all so now. Ansible will look in the local roles directory for this rule and if it doesn't find it there, then it would look here as well, but it will already find this here. So that's good for us.

B

Let's run this again and this time we want to save some time. So we don't want to wait for this. This output that tells us that there's nothing to do with the unattended upgrades, so we can use the text parameter and tell it to just run wireguard and that's.

A

B

If we have time later on, we can try this with a lot more servers. When I prepared the last week, I tried this with 30 servers. Then you will notice that ansible by default is not super fast.

B

Then this step already takes a little bit of time, and when you do it several times in a row, then it's like ah waiting waiting, because each time it has to do this ssh connection to each server and do everything and there are some tricks and config options and other tools to speed this up, and then it gets a lot faster, but we are not going to dive into that. So we will just be patient and wait a moment right now.

B

It's also not taking time because answer was slow right now, it's taking time, because uh each of the servers is installing wire guard, so that needs means that it needs to download this extract package and everything.

B

I think michael is writing. We should create an image.

A

No, it's it's a fun fun reference to xkcd303.

B

A

Github gitlab, whatever is down, we could like create a different image and say: hey uh and separate is slow.

B

ah Okay yeah now I get it.

B

All right: it's done. It installed wireguard. So that's nice and yeah. Now we are ready to start with k3s and for k3s. We will also write uh our own role. It's a little bit more complex than the wire guard rule, but I guess everything is a little bit more complex than the wire got role we just created.

B

So don't be afraid it's not it's not too complex or anything. You will start slow all right. Let's create another directory here and call it k3s.

B

And in there create a directory like for the wire guard row, we created one that was called not like this dash. All right, then another one. Let's call it defaults we'll go into this in a second and another one. Let's call it templates and another one handlers, so that's all will start with the tasks for now, because task is something we already know so create our main yama file, starting with refresh and empty file going to close the terminal for now. So we have a little bit of more space.

B

So I'm not going to type any everything here, because then it will take forever. So I will copy a few things, but then we will talk about it. So first thing: let's go to the k3s website.

B

B

And they have this wonderful way of installing something by downloading some random script and then running it. um That's nice! If you just want to get started and don't care what's in there, but for now we are not doing that.

B

I'm not even sure yeah, maybe let's not search here now, because maybe it's here- the installation, documentation.

B

Yeah, anyway, their their default way of installing it is to use the script. But another way to do it is to download the binary and just run it, and the script mostly does exactly that. It does a few more things like checking what system you are running and what what k3s version you want to have, and then downloading this for you and yeah setting this up. So just if you dive into the k3s documentation, there are a few other ways to install it.

B

So all right, we I'm just going to copy this here. Our first task is to download k3s, so we are using the get url module from ansible and what it does it's doing, an http request to address, redefine, and then it puts it somewhere and off case. It should put it to this directory and yeah. The file should be owned by root.

B

We want to set some permissions and we even can define a checksum and as a parameter again we put in the address so yeah, it's a little bit of an extra step that ansible will verify that our downloaded binary matches this checksum.

B

Of course, both both queries go to the same domain. So if someone had github or someone was able to hack into the k3s account and uploaded a compromise binary, then they would also be able to switch this checksum, but at least it protects us a little bit from some random download errors. If we have uncomplete download or anything, then it will fail, so it doesn't hurt to put it here.

B

You may notice this weird thing over here and that's a variable and the brackets tell it that this should render this. It should put the right value in here. So where does this come from?

B

For one way we can go to our defaults directory, create a new file, call it main so in those directories mostly, you have a main unit and maybe some more files. We will just leave it with mania and in there we can define this variable k3s version and then maybe let's go to github k3s and see what the latest version is.

B

So yeah, let's, let's pick this one, it's 20 days old, kubernetes, 122.,.

B

That's nice. It's a very recent version of kubernetes.

B

All right and now ansible will use this variable to make a better url here and if we would download.

B

Binary here then, at the bottom of my screen, you can see the url, that's the one we are using in ansible in the corner on the left. I can't move my mouse because then it disappears all right.

B

Maybe, let's just try to run this for now, so we can all verify why we are doing it if it works. What we are doing so, let's go in our main playbook and add this row there all right.

B

Let's use this row.

B

Let's create attack for it, and the nice thing that we use this variable to define the k3s version is now. If we want to use a different k3s version, we don't need to modify the role itself.

B

We could use the variables in here, for example, and call it k3s version and then put another release in there and then it would install us a different version.

B

So this is especially helpful if you are using some upstream role but want to change only the setting just the way. We did it above here, but we are fine with the default version of the zuru, because it's the most recent one.

B

So, let's connect to one of the nodes: let's get one of the ip addresses here, for example, we want to look on the server and copy this ip address.

B

And let's go there or maybe, let's open a second terminal, we log in as a root and somewhere. I have an ssh key for this demo. I think over here.

B

Yeah, that's great just because last week we.

B

Already had this ip address, probably.

B

Let's just ignore everything, security related and log in there and look into there and that directory is empty. So now we go back to our ansible and run everything with a k3.

B

B

All right, it's done now. Let's look here, there's a new binary k3s. So, basically now we installed kubernetes, it's not running yet, but it's installed.

B

Okay, so let's go back now, we want to start it and therefore we will create.

A

B

Let's close a few things and the basic idea here is that each time we download a new version, we will restart k3s. So even if we have an existing version installed, then we update the version here, for example, tutu and then run this task again. It will download the newer version, and if this happens, we want to restart k3s. So we will use the notify option of ansible and we will tell it to restart k3s.

B

So how does it lead to ansible actually restarting k3s? Let's create a handler. We already created this directory.

B

So if you work with ansible, you will see a lot of yellow files.

A

And it's the same same with gitlab crcd, so no problem.

B

And let's call it restart k3s because it has to match the name we just used in our notify option and because we are using ubuntu, we are using the systemd module.

B

We want to restart the k3s unit.

B

Now we actually have to tell them that it should be restarted right here restarted. We also want the systemd unit to be enabled.

B

And we also want to tell systemd to do a daemon reload that tells systemd to check for new, existing systemd units or service files on the disk, because in the next step we will create a new systemd configuration file a systemd unit and to tell systemd to pick this up. We need to tell it to do a relay reload, so this doesn't mean that your system d is restarting or anything. It just means it's uh yeah reinitializing some things, and we will also do something here.

B

We will tell it to ignore errors, but not all the time only when our ansible is running in check mode, so this variable always exists if you run unanswerable and it's either true or false.

B

So if we go down here in our terminal earlier, we used the option minus check minus check and you also could use the shortcut called hc.

B

So then, this variable is true, and if this variable is true or if it's running in check mode, then it's possible for ensembl to reach this handler where it wants to simulate a restart of k3s, but because we are running in check mode and we are running for the first time there is no k3s service. Yet so ansible would fail and tell us there is no k-3s service. What should I do and then you could never succeed, running your playbook, so we will tell it to yeah the restart can fail.

B

If we want to check mode, we don't care. We will use this for a few other steps as well. You will see all right so now, when we download this. We also restart and enable k3s. That's good. Then, let's look into my notes.

A

Yeah, I think we need a system that service file now or is it automatically created.

B

uh No yeah, that's that's one thing. We will definitely do yes, we can do it in the next step. I think oh no. We can't because we we need something for this um k3s. We yeah, I think, in the documentation, there's probably some visual representation. Here we we have those three virtual machines and on our server we will install k3s and on our agents as well, and the agents have to register themselves as a server and for this they need a token, because we don't want to allow everyone to just register at our server.

B

We need some kind of authentication and for in for k3s the server if it starts for the first time and notice, it has no configuration at all. Yet it will create such a token for us- and we have to take this token and tell our kcs on the agent nodes to use this to register themselves and that's what we are going to do.

B

And so there is a wonderful ansible module called slurp, it's my favorite, because I find the name a little bit funny. It's like the sound you make when you're drinking some soup and.

B

We are going to slurp the token from the server like we are going to read this file and retrieve the value of it and.

B

Yeah, so all right, I will not type this path.

B

uh Yeah, this file is created when you start a k3s in server mode for the first time, and then we want to store this value, which we can do by telling ansible to register, and we give it a name k3s token, and this way we can use the value the content of this file or information about this file in the rest of our ncbi code.

B

We only need to do this when, when the node we are currently running ansible on is one of the agent nodes, because only for the agent nodes, we need to retrieve this for the server we don't.

B

So we check that agents is.

B

In our group names, so this may look a little bit like magic there's, also something oh, no, now the it seems to be correct. um The group names variable in the answer will run will contain all groups our host is part of. So, if we look again at our ansible inventory, then we remember that we automatically created those groups, servers and agents. That's why we are checking his agent agents in this list of group names.

B

And in this case we will again use this ignore errors option because it's pretty much the same reason as it was with a systemd unit, because we run our ansible layout for the very first time. Then this token will not yet exist on the server when it runs for the agents, so in check mode.

B

We will just ignore that it fails it's a little bit of sad because then, when it already exists and it fails for another reason- then we won't really notice because we say ignore this error, but it's a compromise and because this token is secret- and you saw earlier that, if there is are some some changes, then ansible will display this and we want to disable it for this job.

B

So please never never show us the content of this step, because if you, for example, run this in a gitlab ci pipeline and answer will print out your token, then it will be stored in the job log on your gitlab server and it will be visible for everyone who has access to your gitlab ci pipeline, and you don't want to have some secret token in there all right.

B

So one step we forgot a little bit is delegate2.

B

So what does this mean by default? This job is run for each host. We limited this to our agents so now, by default each time ansible would connect to this agent and then run the slab command there or the slab module. But in reality we don't want to do this on the agent.

B

Instead, we want to do it on the server because the token exists only on the server. So, let's start to delegate this.

B

Let's we are going to use the group here and currently we know that there's only one server in this group, we will just use the first one.

B

If we would decide to set up a k3s cluster with several masternodes or several server nodes, then it would still be okay to put zero in here, because we just needed from one of them.

A

um What happens if there is no server in this group? Is there an error on linkedin.

B

Or yeah yeah there will be an error when we run ansible because it will notice yeah. Something like index does not exist or something like this. So okay, yeah.

A

Thanks and will this token only be used for the agents connecting to the server also for the servers running in a cluster to communicate with each other.

B

Only during the registration, I think it is needed at each startup, so you can't like register it once and then throw it away. You have to use it each time. You start k3s on an agent and then connect to the server.

B

All right so now we have this token, and now we have to use it and therefore we are going to use the template module of ansible. You are using ansible. You probably will use this module a lot because it's the basic module to create some configuration files with some variables in there to yeah yeah to template, and we want to create this systemd unit file. Now.

B

And for template we have a source, so, let's give it a name for now.

B

Ansible templates by default are using jinja 2, which is a templating engine for python or written in python, and as ansible is written in python. It makes a lot of sense to use this, and we want to store this on the server as a systemd unit.

B

B

And the owner again is rude. I think it would be the case by default anyway, because we are logging in as root, but let's just make sure because it will create, contain a secret, so maybe, let's be sure about it, and then we will do the same like we did above that.

B

If this, if this file changes during an ansible run, then we want to restart k3s, because this means we changed something in the configuration of k3s, let's restart it, and the nice thing of handler's answer is that if you call it several times here in this row like buffer and down here, it will only happen once at the end of all those jobs. So you don't restart k3s once here and then you restart it again down here and if you have like 20 20 different tasks that would need k3s to restart.

B

You don't need 20 restarts you just one restart, which is usually means yeah, a shorter disruption of your servers. If you only restart at once, and then again, we are using this token in there and we don't want to expose this token. So we're going to use the diff option again, but this time we are not saying false, because we only want to hide it.

B

If, oh, no, we only want to show it if it's the server, because in the server file there is no token in there there's no secret in there. So, let's, let's display the stiff whenever we can, but not for agents. So let's only do it for servers.

B

And also for the same reason as earlier, we are going to ignore errors.

B

And so we check mode all right.

B

So now we are almost done. We are almost done, but we still need this template. We call this k3s dot, services.j2.

B

So, by default, if you use the template module, ansible will look into this template directory of our row. That's the magic matching of ansible there, so we will definitely not write this from scratch, because this would kind of assume that we already already know stuff about k3s, which we don't know yet we will get into this later on.

B

So this first part is the some basic options for systemd.

B

And then we will now use the the templating magic of ginger too, because first of all we want to check if our current server is in the group servers and then we run to define yeah a little bit of long start command. So xxstart is an option for systemd to tell it what what binary do you want to start? What command do you want to run when the service starts?

B

We tell it yeah our freshly downloaded k3s binary, and then we will use some options there. So, let's look into this a little bit, we want to run into server mode by now. By now you know we have the server mode this agent node mode, so we use the server mode.

B

Then what flannel backend do we want to use? Okay? What's flanner flannel is the networking of our cluster, so it's a sdn software defined network and k3s by default will use flannel.

B

You can run kubernetes with a lot of different network like networking plugins there there's a ton of it, and one of them is flanders, so really common one and then again, flannel has a few different options on what to do in the back end like how does your actual network happen and because earlier we installed wire guard, we tell it, please use wire guard and then flana will take care of the rest, and this is quite nice.

B

It will set up this wire guard network between our nodes and then all traffic between our nodes is encrypted.

B

K3S comes with a few services enabled by default. One of them is traffic, which is a web server, a reverse proxy, and in kubernetes it's used as an ingress controller to get http traffic into our cluster.

B

We are going to this change, something we are going to disable it because we will later on looking at the clock- probably not today, but maybe next week or the week after, we will deploy our own ingress controller to get http traffic into the cluster.

B

So, let's disable this for now, you will also disable the load balancing of k3s by default. It installs a very, very simple load balancer. But it's not cloud. Has this beautiful load balancing feature and we will use this so heads now we run our load balancer and then send the traffic to our nodes.

B

So let's disable the integrated one, we'll also disable the built in cloud controller, because also something for next time. We will install our own cloud controller and this cloud controller will be hetzner specific. It will communicate within the head with the headset api to do some nice things for us. We will see, then then we will also disable local storage because we don't want to use local storage. Instead, we want to use the storage that headsnap provides us which will also yeah.

B

We need to install some things in the cluster for this, but we will see and yeah. Then we pass an additional command to the cubelet.

B

The cubelet is the part of kubernetes, which runs on each of our servers on each of our nodes, and the cubelet is basically the primary part who executes things for us on every node like. If we want to start a container on some node, then cubelet. The cubelet will do this for us, and the cubelet is also the part that talks to our kubernetes api server when we start like it registers itself there.

B

It reports how the node is utilized what's running there what's going on, if there's problems, everything and so the cubelet also has to know on which cloud provider we are running and we for now we just tell it to yeah it's an external cloud provider, so don't use anything internal existing in k3s or kubernetes and then, as the last option, we will add a node stain to this node.

B

Let's not think too much about teins, yet basically it tells k3s kubernetes to not run anything on this node, except if it says I'm okay, with this taint, you can run me there, even if it's a master node, so we we are using server as a name for this node, because it's more inclusive, um unfortunately, the naming and kubernetes still includes master. So the taint is still called like this, but I expect this to change in one of the future releases, but not yet so this was a lot of commands.

B

Don't think too much about it. Yet.

A

B

A

Thing is where, where do we have your cheat sheet for copy pasting? This.

B

uh On my other monitor, but if if we are going to like, if you wouldn't have this cheat sheet, then you would go in the documentation of k3s.

B

And installation options I.

B

B

For example, here's the no taint.

A

Option: okay, maybe my question was not specific enough: um the the entire xx start line. uh Did you build this up yourself or did you consult the documentation or.

B

A mix of both so I started to manually, play around with it, which you just started and see what happens going through the documentation and then step by step. I figured out what options I would need for to run it on hetzner, and this took quite some time.

B

So I don't know between christmas and euros eve. I spent almost each day at least a few hours on tinkering around with it and figuring out. What's working and what's not so it's a little bit too long for this one hour slot.

B

But if you start from scratch, then yeah you kind of have to look through the documentation here and I think it's a little bit hard. If you know nothing about kubernetes yet like then you maybe don't start with all the options, then maybe, instead you would go to the node, and here we have this k3s binary and just started and with all those options enabled, then it will have this default load. Balancer, the default ingress, controller traffic and everything and to you know, to start learn about kubernetes in the beginning.

A

Yeah, okay, but um just change your template. um You can share it later on in your repository.

B

Yes, definitely uh yeah everything we are doing here. We are going to I'm going to commit this in the git repository afterwards, so you can, you can definitely copy it from there.

A

Yeah, this is what I want to try and probably others who are watching later on yeah.

B

B

Then we're using yeah. So if it's a server, we want to use this command and if it's an agent, we want to do something different.

B

It looks similar, but it's not the same again. It's k3s at this time in agent mode and now there's this little bit long command option here we want to tell it which server are you supposed to talk to? Where do you should register yourself and because we just created our virtual machines with terraform?

B

We don't know what ip address this got. It's completely random. We don't know this so yeah. We we use ansible because ansible in the beginning and beginning had this gathering facts step. It mentions it as very early here. Gathering effects and part of this is that it will figure out all our ip addresses so again for now we are sure we have only one server, so we can just look into that. One give us all ip addresses, and now the point is there will be several ip addresses.

B

There will be this external ipa we use for ssh. There will also be this internal ip address on on our yeah on our internal subnet, so because we know which subnet we created it's this one. Let's look into terraform for a second, the terraform.

B

Here from last week, we used yeah here this one, this ip range, so we know that our ap address to communicate with our server has to be in this network.

B

So you put it in here now. If we think a little bit in the future, it would be possible to template this as well to have this as a configuration option for our role and everything that would be, of course, a bit more yeah make it a little bit more universal, but we are going to hard code it here to keep it a little bit more simple.

B

So what we get back here is like take this list of ip addresses, then filter for one matching, the sub network, and then we still get a list. It could contain one or more or zero or more ap addresses we. We just assume there will be at least one. If there's none, it will fail, and then we will take the first one. That's what it's doing here and this way we get the ip address dynamically. So if we recreate the cluster the ip addresses change, then we run ansible again.

B

It will use the right one, and here then again we tell the cubelet to use another cloud provider ignoring that for now, and then we give it our talk.

B

So if you remember in our in our tasks, we used this slurp module. To get this token, we are looking into the content and then we are going to decode it because by default the slop command. We put this into a base64 encoded string, but in this command we need to need the plain text string, so we decode it and then we make sure that everything white space wise is removed like if there's a new line or space.

B

In the end of the token, we just remove it to make sure we don't have accidentally a new line in our command here yeah. So this may sound a little bit frightening if you never use sensibility4, but it's basically a chain.

B

First, we use this variable. Then we use a filter and then we put the output of it in another filter which selects the first one. If we are used to lose using linux and the command line, then you know the pipe command and it's working very similar like put the output of one part in the next one, all right, so in case it's neither a server nor an agent.

B

We will just fail so then our systemd unit should fail as well. That's something that shouldn't happen, but let's just put it in here we will create a few more systemd settings, but we'll completely ignore them.

B

It's just systemd stuff. It's not k3s specific! So let's put it in here and ignore it all right.

B

So now this templating module will now use this file to render a file, and this file will then contain this and everything at the bottom and depending on. If it's a server or an agent, it will create contain one of those commands.

B

All right, almost there one thing missing.

B

So when you bootstrap infrastructure from scratch, then you all often have this kind of chicken egg problem like you can only do something if something else exists. First, for example, we need this token to exist before we can configure our agent nodes or we need this hetzner subnet to exist before we know the ip address of our node. So it's can be kind of tricky to don't get into a race condition there.

B

If you created for the first time, often if it's already existing and then you just want to gradually add stuff, you don't have those problems, but in the beginning, when nothing is there yet, then you have this quite often so, and it took me one moment to figure this out in this case. So uh we are not.

B

uh I will let not force us to run into this issue and then debug it, and what I'm getting at is that we have those steps and one of them is to run the k3s role and then ansible will connect to our servers or to our virtual machines.

B

So server is a little bit of not not a specific term because we use it for one server of our virtual machines and then it will run all those steps, but it will run this task for all of our virtual machines and if all machines are done, it will move on and do this job of this task for each of our machines and then it will move on and do the last one and that's a problem for us, because it needs to finish this and then it for this slurp command.

B

We already need this token to exist on our server right. So on our clients, this or our agents. This command will fail because k3s on the server is not running yet because for the first time starting, we do it with this handler, but the handlers only run as the last step. If everything else is done, k3 is where we started.

B

So here we have this chicken egg problem. The token does not exist, so our agents will never continue here and because this is grouped for all the whole playbook will not continue so the server will never restart it and we will never get a tone.

B

There are a few different ways to work around this one would be to put this into different steps here, but we could tell it here to like just just install k3s on the server and then when this is done, do the same for the agents. It would be an option. It would take a little bit longer because we would do the whole playbook for the server then the whole for the agents.

B

So let's not do this. Let's do something a little bit different and yeah. I think this is a little bit of advanced ansible usage.

B

You can configure a strategy on how those knows how ansible works on your notes, and I think the default one is serial or something like that, but not sure right now what it's called, because the default one you never need to configure and we are going to call to use the strategy free and, in this case, ansible will instead start this task on all nodes, and when it's done on one of them, it will already start the next task on this node, even though this job may be still running on other nodes.

B

So that's that's kind of what we want here right. We want to the playbook to continue on our server, even though it's still running on the agents, because this way we can ensure that the server creates this token and a little bit behind it. The agents can collect this token and use it.

B

So we need one little extra step for it, because right now it still would run this on all our servers on all our virtual machines, and then this would still fail if it does not exist yet so we need a way to tell ansible hey. Wait until this file exists on the server and there's a nice and simple module for this.

B

It's called wait for so it's a very, very good name, because it's exactly tells us what it does so by now. We already know the basic syntax of those tasks we have this name and similar to the slap thing we delegate this to the server.

B

And yeah, we only do this for agents, we ignore errors if it's in check mode and what are we waiting for? We want. We are waiting for this file to exist. We are doing this for 900 seconds. I think that's 15 minutes.

B

So that's more than enough to wait for our k-3s server to come up on the server node. Usually it takes more like 20 or 30 seconds or something so. The combination of waiting for this file to exist here and our different strategy to run those tasks as fast as we can on each of the servers allows us to yeah to solve this chicken egg problem so yeah. If you are completely new to ansible, then this might be a little bit of mind. Blown don't be afraid.

B

If you try this out yourself, if you're working with it a little bit, then you will just like notice how the pieces fit together.

B

All right, I think, that's all we need to run k3s. Let's try this it's already after seven o'clock.

B

All right, let's run this most likely edit some typo, I messed something up, so maybe it just will just fail. We will see here gather an effect, so ansible is aware of all the ip addresses. For example, we need another template and you see it will. It is skipping those wait for a slap step on the server node.

B

So it's already templating the configuration file on our.

B

B

So now we are waiting here. You see how it restarted k3s on the server and then it's waiting on our agents and then it slurped this token. Now it's templating the configuration on our agents. Now it's restarting k3s over there nice.

B

So let's switch to our different terminal where we are still locked in on our server node and hopefully or maybe at first we can go to etc, systemd system and there should be k3s unit somewhere over here. K3R service.

B

And here we see our exact start command, it's exactly what we put into the template. We are on the server node and we only wanted this exec start command and if we look on what processes are running then, hopefully yeah we will notice here is running k3s, so let's go to red, and now, just in the beginning, when I told you what is our goal for today, it was like run cube. Ctl cube control, cube cuddle, whatever you want to tell it so there's this huge discussion in the kubernetes community.

B

What this binary is called, I usually call it cube control. We can also call it cube ctl. So there's a ton of options uh or cube cuddle. It goes on and on so uh later on, we will use cubecontrol on our local machine right now. We will use it on the server because it's part of our k3s.

B

Binary, you can run cubelet, get notes and hq later cube control. Sorry.

B

Cube c t, l and there we are, we now have a running kubernetes cluster. We have three nodes, we have a server, we have two agents which apparently are in a ready state. We see that the server has specific roads, control, plane and master assigned, and we can also see which kubernetes version we are running. It's exactly the same version we earlier put in this defaults file right, 122, k3, s1 and the other thing we wanted to see was.

B

Pods and okay, so I think, looking at the clock, we want to do the full dive in into kubernetes right now and talk about namespaces and everything, but let's display all pots. For now we can see that we have two parts deployed: metric server and core lns.

B

The metric server is collecting some metrics in the server and core dns is the central dns server in our cluster, so all components in our cluster later on. We use this to resolve dns names in the cluster and to the outside right now they are pending, and I think we will not look into it. Why? Right now, because it's a little bit of a bigger topic, because we still need to configure the networking of our cluster, but that's maybe a little bit better for next time.

B

So if we look at our notes again, we can use an option, output or the short version of it dash oh white and then my terminal is too small.

B

Let's look at it like this, then you have a few additional information here, like the internal ip addresses and the external ip addresses, and here you'll notice that at the internal ip column there are external ip addresses, but at the same time there are no external ip addresses.

B

So apparently, k3s still has a few things messed up there, networking wise so because those ip addresses are wrong wrong and those parts are still in a pending state. They are not running yet also because of missing networking stuff and yeah. When we look into the readme file from the last time, it's a little bit of preview for next time and there's a link cloud controller manager.

B

It goes to github and yeah, that's from upstream headset cloud again similar to our ansible terraform provider and next time we can deploy this to our cluster and then it will tell kubernetes hey. You are running on headset cloud. This means you have the following ip addresses for the following notes, and then everything will be correct, configured the right way, so the cublet will then know yeah, there's this internal ip, this internal network with those ip addresses starting 10 to 9, blah blah and then just like magic.

B

Also, our pods will start to run and everything will be fine. So that's a little bit of preview, not sure we are going to do it next week or the week afterwards. But when we are feeling like continuing with this cluster, then we can do this but yeah the goal for today was to see. We have some notes. We have some bots, they are not running it, but they exist.

B

The nodes are apparently communicating with each other, because the server is already aware of those agents. So the registration with the token worked we yeah, we played around a lot with ansible created some roles and in the same way you can write yeah quite complex ansible roles to set up everything automatically.

B

I hope you all learned something I didn't follow the chat. So please, let me know if there are any questions in the live stream, we should get into.

A

We did, we don't have any questions. As far as I can see, I wanted to say, like thanks was amazing. I learned a lot uh lots of things to unpack um and when you, when you open up the headset interface, doesn't does this look different to before.

B

No, no, not for all. No, it's still or so. Hetzner is only where we have those three machines and that we have this network next time. When we create some volumes for storage or the load balancer, then we will see this over here as well.

A

Okay, perfect because I just like I'm sorry, I'm in the learning mode, I would say- and I bet others are as well. um I would totally love if you got time next week, so we dive further into uh the the cloud host controller before. If I remember.

B

That correctly, uh the cloud controller manager.

A

The cloud controller manager, because right now I didn't understand what it means, um but I have a have a picture in my mind and um I would love to see how it works.

B

Yeah yeah, we can definitely do this. I just check my calendar and my calendar says next week. I am in this call.

A

Perfect, so I will tease, tease everyone on twitter, then, um and we can. We can also collect ideas for for other things. We want to try out in the coming sessions.

B

Yeah yeah next time we can definitely set up this networking stuff and the storage stuff, and then I would call it a yeah, a running kubernetes cluster.

B

Perfect, we can go on and on and on.

A

Oh, I have, I have a lot, a lot of ideas, what can deploy to kubernetes. um I also want to bring in the the monitoring and observability part with like prometheus and cube state metrics, and there are many things probably you want to try out by by following following our sessions and yeah. Maybe I don't know what else can we do yeah? We can create a website and like make it as complicated as possible.

B

Just before the call when, uh when the other michael uh mentioned a graphql, so so my personal block is actually using graphql and it's the prime example of do you want to run your static block in the kubernetes cluster and yeah, it's a static block using graphql running in accumulators class.

B

So if you want to go in the over engineering side, we can definitely do this with that. The most complex.

A

B

We could ever build.

A

Yeah, it's it's it's a fun example. Everyone brings up I'm thinking of like finding something which which is easy, which is aesthetic website and just like, deploying it, because we can and then we can say hey. This is like it's a different use case when you like, connect, ci cd pipelines and deploy it in different environments, and there were. I think there are many things which are of interest and I haven't tried out yet and probably others are interested.

A

In and I can see that marcel shared shared a link with us, there was an ignited osmc around this website and kubernetes and terraform and other things from. I think it was bram fogeller from inuit. um Who, who did the talk was funny, but maybe we can try it out and see. Oh, this is interesting um okidoki.

A

Yet, and thanks again for preparing everything, um yeah.

B

A

um We're looking forward to next week, then let us know on twitter um how you feel, if you have questions or not, or people probably have questions, and with that I would say, bye, bye till next week on youtube.