GitLab #EveryoneCanContribute cafe, 10 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 16. #everyonecancontribute cafe: Kubernetes deployments to Hetzner Cloud, step 3

Description

Blog: https://everyonecancontribute.com/post/2021-02-10-cafe-16-kubernetes-deployments-to-hetzner-cloud-part-3/

Step 1: https://everyonecancontribute.com/post/2021-01-27-cafe-14-kubernetes-deployments-to-hetzner-cloud/
Step 2: https://everyonecancontribute.com/post/2021-02-03-cafe-15-kubernetes-deployments-to-hetzner-cloud-part-2/

Demo repo: https://gitlab.com/ekeih/k3s-demo

Follow Max Rosin on Twitter: https://twitter.com/ekeih

A

And we are live on youtube. um Welcome everyone, I'm looking forward to the third session on uh deploying a kubernetes cluster to headset cloud and, as you can see max already prepared the demo and prepared everything.

A

I don't really remember what we talked last week, what we're going to do today, but I think we have some kubernetes ports or things running, and we want to do something about that. Is that right? Yes, it's correct.

B

All right yeah, it's the third time already and if you're wondering how many times could you possibly spend on setting up a kubernetes cluster, then I can tell you it's a thing where you can put an infinite amount of time into like this no end to it.

B

But I think today, in the end, we will have something that we can call a running cluster where we can deploy whatever we want to and the last time we had a goal. I think it was to run cuba city, get notes and get pots and see something, and I like the idea to have a goal and to achieve it in the end. So today we want to try as a goal to have a cluster with 20 nodes, running and 2 000 pots running.

B

It doesn't have to make much sense what's going on in those ports. I just want to show that it's possible to start this and yeah, hopefully we'll we will get to this so um yeah the during the first session. We set up all our notes with terraform and ansible, and then last time we installed k3s.

B

Most of the time, we spent writing an ansible role for this and yeah. Today we want to continue with k3s, because last time we ended in a situation where the ip addresses were not correct yet, and our pots were also not running yet and to speed up the beginning a little bit. I already created the virtual machines today. So if we look in the dashboard we can see, we already have those servers. They are 18 minutes old and just before we started the live stream.

B

I also started ansible and it's finished now so hopefully we can now connect to one of the.

B

B

All right yeah, it's uh I often get the same ip again and again from hetzner. So let's try this all right and last time. In the end, we had this that we had a running k3s cluster, where we could run k3s ctl, get notes and we were able to see that we have two pots, but they are not running yet they are in this pending state and today we want to explore a little bit why those are pending and solve this. We also want to yeah the reason why it's not running.

B

It is some networking issues. I think we covered that last week, because also when we look at those notes with a little bit more details.

B

Then we were able to see that our public ips are marked as an internal ip address and there is no external ip. So this also looks very wrong want to solve this, and we also want to create some persistent storage. So if we restart a pot, the data is still there before we start to look into those ip address issues on the spending pods.

B

I want to look a little bit into the cube ctrl command. We have here because, right now, it's not very comfortable to use it. I have to ssh into the server and then instead of running, cube ctl.

B

What I'm used to, I all have all the time to do like run the k3s binary and use the bundled cube, ctl command, which is yeah kind of annoying, and especially if you are used to run cubectl from your local machine and yeah, while working with your cluster, it's much more easier to just use it locally on your machine and in many cases in the in the real world, you don't even have ssh access to your kubernetes servers.

B

So your only point to talk to is the api and you're using cubectl to talk to the api, to use cubectl to talk to a server. You usually have a configuration file, it's the keep config, which includes all the details. You need to know to talk to your api like the ip address, the your client certificate or your client token to authenticate.

B

Usually you also have the server ca in there to verify that you're talking to the right api server and if we connect to our server again, then I think it's located in.

B

Ranger, because rancher is the company that is developing k3s and.

B

Yeah there is k3s yammer, it's uh add the configuration file and we can also look into it.

B

If you have this file, don't share it because there's a secret in there like here the client certificate is audio under the client key data. So all you need to connect to the api server. So that's a bit of secret, but I am positive that during this one hour nobody will type this manually from the stream.

C

And mostly, it's encoded also right. It's encoded in basic 64.

B

Yeah yeah, you have to type it, and then you have decode to have to decode it and yeah. For us uh interesting is this server, and currently the server is like localhost, because it's only meant to be used on this machine.

B

So we want to take this cubeconfigure configuration file, this cubeconfig and download it to my computer, and then we want to replace this with the right address and then we should be able to use cube ctr in the location and one way to do this would be to now manually copy.

B

This put it on my computer and edit the file, but we are trying to automate as much as possible here and if we create and delete and create and create delete this cluster again and again, we don't want to do this every time and during the last two sessions we already explored ansible quite a bit, so it's yeah. It comes natural to use ansible to do this.

B

So let's start a new role. Let's call it keep config, let's create this folder tasks again with one file, and we are not going to cover this in great detail right now, because I think last time we already looked a lot at. How does answer will work? How do we write a role? So I'm going to copy a few things in here and if you have any questions you can just ask or write in the chat, so I will go over it uh very fast and we need one other thing.

B

I've also seen this last time, additionally to the task we can create this defaults to specify some variables.

B

And we are going to put this in there, it's like the at the local path, where I want to store this file on my machine. So if you're trying this yourself, you can set whatever path you like here, and so we have four tasks in our row. One is to check if this exists locally or yeah. If this file exists, we are using the path we just said here and then we register this check in a variable and then the next three tasks we yeah check.

B

If this state is false and if it's false only then we do the rest, so we download this file and yeah from this directory. We just saw this. When we were logged into the machine, then we store it locally and then we are going to use the replace module of ansible.

B

We are going to replace the localhost ip with the ip first server in our cluster, so the service group, the first one we are going to use the ip4 address of it to make our life a little bit easier. We are also to replacing another string in there because it's called default and we want to call it k3s demo. We will see this name in a minute, so yeah, that's kind of nice to use ansible to do this.

B

Every time we create a new cluster, all right, then, last time we created this main playbook and we want to use this new role.

B

So this time we don't add it here to this to those hosts, because those hosts are our hcloud hosts and we want to do this locally on our machine. So let's create a new one here, calling it pullcubeconfig.

B

And we want to do it only for localhost. So, even though localhost is not part of our ansible inventory, because our ansible inventory only includes those hetzener servers, localhost will work, that's a magic part of ansible. Your localhost always exists, it's always the machine you're running ansible and we want to use our role, keep config and.

C

B

We did with the others. We also set attack right.

B

Let's go back to our shell and let's run our main playbook, but let's not run all only tax cube, config.

B

All right and if we check which directory we used, we wanted to put it.

B

B

There you see it's yeah just from today a few seconds ago, and if we look into the fire, then it looks yeah exactly like what we have seen on the server.

B

Now I have to reload my shell too. So it reads all those files.

B

There and now I should be able to have three cube contexts. We are not going to dive into what a cube context is. Let's imagine like one cube context. Is one cube configuration file? That's like one cluster we are talking to, and I have two other contexts or two other clusters and the new one is k3s demo and with cubecontext, which is a separate tool from cubecontrol. But it's super useful. You can switch between those.

B

So then you would see that this one is selected, but today we just want to work in k3s demo, but the other one center server k3s is set up in a very, very similar way to what we are doing in those live sessions. Here, and now that this is selected, we should be able to run cube, ctl get nodes and there it is they're, also like 14 minutes old. So this is definitely our new cluster and if we run cube ctl get parts, then it's also there.

B

So now we don't have to use ssh to connect to the server and run cube cdl. We can just do it on our local machine that will make the rest of the evening a lot easier, all right.

B

So if you're new to kubernetes, then maybe it's a little bit confusing when I type all those cube control commands. So uh this will come over time that you get used to it. So you remember the important commands and you don't have to remember all the options that there are. There are also some graphical tools, so visual tools to explore your cluster, so you don't have to do everything with cube control, but it's useful to know the basics of it.

B

So one command we already have seen. Is this cubesat, I get notes and then maybe sometimes I used get an o instead of nodes, and it's doing exactly the same. So for a lot of the default resources in your cluster. There are some shortcuts, so you don't have to write the long things for nodes and ports, there's not much of a difference. It's only a few characters, but there are a few more things, for example, uh network policies and then it's getting a lot easier.

B

If you are just have to write, cube, cta to get network- and if you have this yeah the back of your mind, knowing that this exists, then it can help. So please don't be surprised if sometimes I just write keep ctll. I get note it's that of cuba. City notes yeah. Well, so we can use this cube. Ctr get command to explore. What's going on in our cluster and cubecontrol works like this most of the time you have cubecontrol and then a verb, for example, get or delete or apply.

B

We will look into this later or create, and afterwards you take you control what kind of object you want to modify or what object you want to get or yeah. So that's why we tell it to like get our notes. We could also delete our notes. We are not going to do this now, but it's possible and then our list of notes will only contain two notes.

B

The hedsner server would still exist, but the cluster wouldn't know about it anymore, um yeah. That's like the basic way how keep control works and over the evening we will learn a few more commands. uh Sub commands of cubecontrol so there's another concept in kubernetes: it's not namespaces, so let's do this get namespaces and then we see we already have four of them and each time you run cubecontrol. You are running it in the context of one of those namespaces.

B

That's why, when we do keep control get parts, it tells us there's nothing in the default namespace, because right now we are in the default namespace.

B

But a few minutes ago I did this, which is a shortcut for all namespaces, and then it's showing me all ports from all namespaces in my cluster here in the front you can see the namespace keep system is running those two parts or it's actually not running them because because they are still pending but as an object, they exist there so yeah. Sometimes I use the word resource or object. So what is it? Actually all this data of what's running in your cluster, is stored by the api server of kubernetes.

B

Often you use etcd, that's called 8etcd, which is storing this icd can run in a cluster and it's part of a control plane and your api server talks to it all the time and stores all information in there and fetches all information from there and with k3s it's a little bit different by default.

B

We are using the built-in storage of k3s right now, so we are running this api server on our server zero node and on there it stores it in some file or some directory all this information. So if we delete this, we lose all information, but each time we use cube control.

B

It's talking to the api server of k3s k3s, looks it up in its storage and then tells us what the storage says and if we, if we want to change something, then the api server takes this change and stores it in the storage and then other parts of k3s take it out there and act on it.

B

For example, if we create a new pot, then the api server will store in the storage here there should exist a port and then the scheduler of k3s will pick this information up and say. Okay, I have to schedule this pot on some node.

B

Then it will decide where to run it, and then we have those notes running in our cluster, for example our agent, zero and agent one and those are talking to the api server and asking him hey. Should I run a pot or should I do anything and the api server says yeah, the scheduler decided. You should run this part and then on this node we have the cubelet running.

B

It's called print like this, and this is fetching this information and then it tells our docker daemon to start this container, which makes up a pot.

B

So this is a lot of stuff going on and over time you will learn a little bit about it, how all those moving parts in the cluster work together. For now, most of the time we will just use cubecontrol to talk to the api server and then look what happens.

B

So we don't have to think too much about what's going on in the background. What's doing what this the cubelet doing, what is this scheduler doing?

B

We don't have to think about it too much, but we should know that those parts exist in the background all right. So we are super curious. We finally want to learn why those uh yeah why those pots are still pending.

B

So let's switch to a different name space cube system, so we can use cuban s. It's it, ships with a cube context, tool which I used earlier to switch to it all right. So the advantage of switching to a different namespace is now we can use cubecontrol getpod and directly get those without running with all namespaces.

B

Alright, let's look into a new command cube control. Describe we want to describe this coordinate spot.

B

I'm going to make this ah now I did a mistake. I used unit verb, but I missed the kind of object I want to look at and it's the part I want to look at and there it is all right. So there's a lot of information here.

B

First of all, we have this name and we have this namespace. This plot is running on in uh then we have some labels, no annotations. We have the status pending. It's the same. We saw in the table earlier all right, then we are able to see here what containers are running in this pot right now, it's this currently in this container, and we see it's based on this image.

B

If there's no domain here, then it means that this image is pulled from a docker hub. We see that it exposes some parts, for example 53, which makes sense for a dns server.

B

So you see a lot of more things we will ignore for now and at the bottom we see a list of events and we see that it looks a lot like errors. I mean it's called warning, but it's definitely more like an error and yeah. Our scheduler is telling us something it says: uh yeah zero nodes are available and yeah. It says I'm not sure why it says different numbers here, but, for example, three nodes had taint node cloud provider. Kubernetes io initialized is true that the pod didn't tolerate.

B

Okay, so maybe.

C

C

Is it every time the same.

B

It's all the time the uninitialized change isn't.

C

It that's interesting. uh The other option could be because then, when you spin up the two with three notes, suddenly that's it's the timing of your lens, because the studio goes every time to through the loop to the um loop, and that could say some part, but the latest one is eight from 24 minutes right, so the oldest one, so yeah, that's and I'm fine, mostly in the end, it would be three notes on the first. He found the first note.

C

Then, after a few seconds, the second two teams happens and then the third thing will happen.

B

Yeah yeah yeah. That makes sense. Definitely I'm a bit surprised about the last event, which is older than the or.

C

Above you have orders currently.

A

If you're, if you're a beginner, what would you recommend when I see a warning event type? What's the next next action? Is it ignore? Is it like copy paste to google? It's.

B

Definitely copy paste to google okay, so um especially when a pot is not starting.

B

uh The first thing I do is uh describe on it and look what's going on here and if there's anything I don't recognize, then I throw it into google and see what happens and if we would do this now, then we would start our learning path on taines and tolerations.

B

So those are two keywords we can see here, it's taint and tolerate so yeah. Maybe we can throw those two words into.

B

Google and we are not going to read the full page, but so if you are new to kubernetes, you may have never seen this documentation. So that's the kubernetes documentation and it's a very good resource to learn about all the concepts and all the thousands of layers of abstraction of kubernetes and the whole idea of taints and tolerations is that you contained a node and if a node is tainted, no pot will be scheduled to it, except if the pot tolerates the taint of the node.

B

So this is a mechanism we can use to control which pots are scheduled on a node or if there are pots scheduled at all on the node.

B

So the taint is part of the node. So let's maybe not do the describe on the pot, but let's do it on a node. So we can use one of our agents here. Cuba ctl describe node.

B

Here we also see some events and.

B

Yeah those are okay, for example. Now here we have this. uh We also have a warning, but we can ignore this for now, especially during startup, it could be happening that there are some warnings and but if everything afterwards looks healthy, then maybe it's just a timing issue that something didn't exist. Something wasn't initialized during the starter, so it's a very good sign that the last few events are, for example, our agent status is now note ready, so that should usually be the state and notice in, but we are looking for tanes.

B

So, let's scroll up a bit at the top again, we have this name. We have some labels and annotations. We have seen this stuff a few minutes ago for the pot as well. So labels on annotations are very common on all the objects in the kubernetes cluster, and then we have attained here like teens, and it's exactly retained that we were able to see in our warning or our error message and attained as a key value pair very similar to the annotations and labels. There's also always a key and a value.

B

So our taint has this name and to make it unique most of the time you use a domain for it. So if you have a very custom taint for you or your company, then for you would, for example, call it like yeah foo dot. Everyone can dot com, slash, uninitialized or something like this to make it unique and make sure nobody else uses this and yeah.

B

Then we as a value we have true and no schedule so yeah true is says: just says: yes, it's uninitialized and then the part behind the colon is the effect of this taint.

B

So it's more like a key value pair with three parts: more like a tuple and yeah. It basically says: if a note has this taint, then please do not schedule things there, and so why does? Why do our notes? Have this taint right? Where is it coming from?

B

Let's take a look last time when we wrote our k3s rule and we had this template where we generated our k3s systemd unit and we disabled the built-in cloud controller, and we told the cubelet that there is an external cloud provider so based on this information k3s and the cubelet which had started on the agents realized that they have no cloud provider, that they have no working network which would be provided by this cloud provider.

B

So they add those taint that it's not initialized yet and yeah. That's the reason why our pots are not scheduled on those nodes and because all our nodes have this taint. Nothing is scheduled at all.

B

So we want to solve this. We want to install a cloud provider in our cluster and because we are using hetzner cloud, we are going to use the headset cloud provider all.

A

B

So it's not a cloud or it's actually, it's they're not called the cloud provider, it's called the cloud controller manager and if you're using a different cloud, then you would use a different cloud. Controller manager, for example, for openstack for azure for aws and other yeah club providers.

B

So, as now provides this and yeah here's for an example, I can zoom in a little bit, uh for example, that it will set the right public ip address. That's kind of the information we are currently want to fix in our cluster right. It adds a few more things, for example, labels of like what what kind of virtual machine are you running in which region region? Is it running stuff like that and then there's a example how to deploy it?

B

We are going to ignore this because down here it says if you want to use headset cloud networks head over to the deployment with network support documentation.

B

So let's look back at last week, I think at the first week we deployed a network here, it's a private network with this ip addresses and we put all our servers in there. So we want to use this cloud controller manager with support for those networks.

B

This is nice because it makes our nodes aware that it's running in this private network and it's becoming aware which ip addresses exist in this private network, all right. Let's click.

B

This a new network- we already did this with terraform, then we should download the latest deployment latest deployment file. We can do this and we are going to use a cloud controller manager with networks.

B

B

And then we close the ansible stuff for now. Instead, we are going to create a new directory, let's call it hcloud, because now we are doing some very it's not specific stuff.

B

Let's create a new file, it's our cloud controller manager.

B

So that's a lot of yammer.

B

The beginning of it are a service account and a cluster rule binding. We will ignore this. This is stuff that is used to manage permissions in your kubernetes cluster. So if you are going to run stuff in production kubernetes at some point, you will have to learn about airbag, which is yeah the technology or the yeah, the mechanism that is used to define roles and permissions in your cluster for us, it's more interesting to look at the deployment right now.

B

So, for example, down here we have our containers there we see we are pulling this cloud controller manager from headsmash docker hub account, and we are seeing that it's running a command with a bunch of options and down here, there's this network and we need to use a different one here.

B

So we, this is also in the documentation, explained a little bit back down here: change the clusters to the reflect in the deployment file to fit your pod range. So what's our pod range, we should look this up because we could at no point we configured this. So let's google, this, like k3s k3s networking.

B

And here we have this parameter cluster network and it's this one by default. So if we wanted another network, we could define this. Why starting k3s, but we have this one. So, let's put it in there.

B

So we could do this in a little bit more, even a little bit more automated, like we could build something to figure out which port network we want to use and automatically put it in here, but right now, let's put it manually in this yaml file.

B

All right, then there are a few more things, for example down here it defines some environment variables, for example the hcloud token and the hcloud network, and it uses a reference to a secret secret key reference that it expects to exist in our cluster.

B

So we never heard about kubernetes secrets before so we will look into this in a second.

B

All right so cubaneta secrets.

B

If you know, containers a little bit, then you probably have heard of the 12 factors app.

B

It's a design principle on how? How could you design- or this is this- is german now and okay- all right now, it's english um yeah, how how you should design an application. So it's easy to run it in a container to those 12 factors to it, make it easily configurable and everything and one of those is uh yeah three store config and in the environment and kubernetes kubernetes is a yeah, an orchestrator for containers. So all we do here is running containers.

B

So it's following a lot of those design principles as well, and a lot of applications that are built into containers are able to use environment variables for their configuration, and the same is true for for the hcloud cloud controller manager and the cloud controller manager needs to talk to the hetzner api with our api token.

B

So if you don't remember from the first time we set up on a security, an.

A

B

Actually, I did this before our first session, but I showed it a little bit in the first stream that we have this token, and this token is stored in this variable. So I'm able to use this without showing you the api token, which is a little bit nicer than to expose this token in the stream, all right so yeah. It needs this token to talk to the hudson api and because we are deploying it with this support for private networks.

B

We also need to tell which network to use so yeah, this concept with 12 12 factors and one of them is put stuff into the environment. How the? How is it realized in kubernetes in kubernetes? You have two two objects for this. One of it is cube through this config maps, and we can see here that there even exists a few and, for example, maybe let's look into this one coordinates. I think it's some configuration file for the core, dns pod and so far when we used cubect.

B

I get we used it like this or we used it with output, white for configuration or for conflict maps. There is no wider output, so it's kind of boring, but instead we can also tell it to use to print it in yammer and then it's a lot of stuff. So all right and there we can see the data that is stored in this config map and yeah. It's basically just a text file with a bar or not exactly it's again, some key value stuff.

B

So there's a key client ca file and then there's data in stored in this key, and this is done several times and the core dns port uses this config map to get its configuration and in the same way we want to do it for our cloud controller manager. But we are not going to do it with the config map. We are going to do it with a secret and there is a bunch of secrets in the cube cube system default in the cube system, namespace and yeah.

B

What's the difference, the idea is that what you store in a conflict map is it's? Okay, if everyone with access to your cluster can see this information and it's not just secret, it's just configuration and what you put in a secret is more like yeah secret, for example our api tone and then there's this huge coming in my sentence, because, but it's not really a secret, but in what's in there um yeah. Why not, because the whole encryption of our secret is a basephase base64 encoding.

B

So all it does is to encode it and put it in there and if you know base64, then you know it's completely reversible, so you can decode it and know the secret.

B

So if you just use a secret in kubernetes, uh don't think your data is safer than it than when it then it would be in another situation. It's the same as storing it in plain text.

B

It protects you from a few things, for example, if, if you print it- and someone is looking on your monitor and there's this huge blob of numbers and characters, then they can't directly read the secret, but if they can copy it and decode it or they have a screenshot of it and can type all the characters, they are able to figure out your secrets. So yeah, that's about it.

A

Is this something where you could like integrate hashicorp void, then, or is this a different.

B

Layer, yes, it is.

A

Yes, instead of storing.

B

Your secrets there it's a good practice to put them in hashicorp world instead.

C

But you could also use literally because obviously pictures are written into the database into etcd.

C

You can also use a mechanism that you need to support directly out of the world that you can store all the secrets directly really encrypted, so you don't need to use historical, but most of them of the most companies that I know don't use this approach mostly, but there is a support for that, so you can store secrets directly, also in the situ way.

C

The reason why it's um not secure is because, like max said, yeah it's encoded in base64, so this is only the consistency checks that you can really ensure that the data is correct, but mostly the um data are saved into the database of kubernetes encrypted. So it's like not a re-encryption, but you can attach this also. I can check for volume for the dots so that you can. If someone is interested in the stoppage um the person can read about that.

B

Right, yeah, thanks and now what we want to do is to create a secret that includes our token and our network, so it can be used here.

B

So let's use q controller for this. Let's do this in the namespace dash n is a shortcut for namespace. Let's do it in cubesystem: let's create a secret, so that's a new verb. We haven't used that before it's like you get but create something, and we want to create a secret.

B

Now a secret can have a specific kind and we want to create generic secret. We want to give it a name hcloud. Why do we want to use this name? Because in this upstream yammer example, they use this name? So let's just go with the same, so we don't have to change it here and then we want to say what secrets do we want? What information do we want to store on the secret.

B

And so it may look a little bit cryptic, but what we tell cubectl is to create a secret and, under the key token store the value which is currently in my environment, regular hcloud token.

B

So my shell will put the actual token in here and then cubecontrol will receive the output token with this string in there, and it looks a little bit simpler when we do the same again.

B

All right um and it was called network and we want to use the network a3s all right now. We as we do cube cta, get secrets again, then we can see that there's a new secret each cloud, it's seven seconds old and yeah. I'm not going to print this now because it includes my api token but yeah. It's it's there now.

B

All right, so, what's next, let's check the documentation of the cloud controller manager yeah here it also tells us to create the secret what we just did and then we are supposed to apply this yamaha file we created in our case it's not called deployment, but it doesn't really matter.

B

So, let's keep. Let's use cube control again, so maybe let's switch over into the hcloud directory, because right now we are in the.

A

B

Directory, let's see we put the right ip address in here, the file is saved, so now we use apply the difference between create and apply with create. You pass, for example, yeah what what you want to create, for example, a pod and with apply you tell cubectl, to read all this information from a file which is usually more helpful because you don't want to type all those stuff all the time. Instead, you have your configuration in a yummy file, which you are then going to apply.

B

Let's apply this.

B

All right yeah, we can ignore this warning. This is related to the example from hetzner that uses an option that will that is currently deprecated, but I will update this soon. Probably it doesn't matter more interesting for us. Is it created a service account?

B

It created the cluster role binding the stuff we wanted to ignore, and then it created created a deployment.

B

All right, so, let's do cube ctrl get pods again and we are now seeing we have three ports and none of them is pending anymore. So we have those old ports metric server and coordinates. They are now running and we have the newport hcloud cloud. Controller manager, which we just created and what's interesting here is the resource we created, isn't called pot.

B

It's called deployment. So maybe let's try to look if this is something that exists in kubernetes and it is actually for the other two. We also have a deployment so yeah. There is one abstraction layer after another. So most of the time you don't manage a pot, I'm not sure if we also looked into what a pot actually is so a pot is a group of containers.

B

The group can contain con. The group can contain only one container, but it also can contain several containers. So in our examples right now, each pot contains one container, but the pot itself can again be part of several things. In our example, it's part of deployment, so there's something else between this. For example, if we describe this pod hcloud controller manager.

B

Then at the top we can see an option controlled by and then we see it's apparently not controlled by this deployment. Instead, it's called the it's controlled by a replica set.

B

So that also exists in kubernetes, so why do we have so many layers and layers and layers? Why do we do this so to make it easier? Actually it might not appear this way that you do this to make it easier, but the idea is to have several parts and each part in your cluster takes care of one simple thing and the pot, for example, takes care of like defining what containers should run there and the replica set is taking care of how many of it should run there and the deployment yeah.

B

The deployment is basically a template, including all this information, and, if I'm not mistaken, then for example, a replica set is completely immutable, but you can modify a deployment or you create a new deployment or you modify an existing one and then kubernetes will create a new replica set with your new configuration and then kubernetes will create a new pots who match the configuration you put into the replica set.

B

So in the beginning, don't think about this too much, but just have it in your mind that usually you do not do something like cube, cta, create pot, or usually you don't put a port directly into your yaml file, but usually you will create a deployment which is then taking care of the rest for you.

B

We will see this a few times all right, so we have this cloud controller manager running so I I claimed that this would remove this taint and apparently it did because those pots are running now, but for to be a little bit more complete. We can.

B

Describe this note again,.

B

And then we see there's no taint anymore. So that's nice all right. Another issue we had was where those wrong ip addresses of our.

B

Node, how is it seven minutes before seven? We just started um all right. So here are our notes. Now we can see there are those internal ip addresses which are exactly from the network we have in our headset console like 1029.

B

You can see they are the right ip addresses and it has those external ip addresses all right. This is very nice, because now our cluster is aware of the cloud it's running in.

B

So if we add a new node, then this information will be automatically updated with the right ip addresses. This information is now correct and the cloud controller does a few more things for us, but we will check this out later on.

B

Maybe something else it does is yeah. I did the cube. Ctrl describe node and now here's all those labels in like which data center, it's running, which can be helpful when you operate a kubernetes cluster.

B

Because, based on those labels, for example, you can tell a part to only run on a node which is running in a specific data center, and for this you can use those labels all right.

B

So we have roughly six minutes left. I think we can do a little bit of overtime, but let's.

A

Not die yeah just just take your time to explain everything and round it up.

B

C

From beginner questions, mostly um because currently I was wondering how currently the pod gets another ip address so that it isn't more in the pending state.

C

So can you please describe a part with oh with option or white so that we can see which read up here it was it has yes, yes, definitely because did we already deployed and cni.

B

Yes, uh last week, I think you weren't.

A

B

But we can look into it because I think last week for everyone, it was more like a magic part of what we did. um We wrote our ansible role to.

B

Deploy k3s and as an option, so the default uh cni is flannel and.

C

B

Thing we configured for it was to use wire guard. So then flannel sets up a wire guard tunnels between all your nodes, and so you so in addition to using our local network or private network on hetzner, the traffic between those nodes is encrypted.

C

Rjs at me, okay, then I understand how it works so because, basically, the rear node has a different ip address and it's interesting how the packet flows from one node to another, typically all the nodes, um using a technology, mostly based on the ips lens. So that means that your ip package on the node will be pitched into a new pic ip package and then transfer to the next node and then will be encapsulated again. And then you can that's the reason why the plot can receive a reappear address by using this virtual period.

C

That's not free reappear, address of a node.

B

A few days ago, I tweeted something or I retweeted something, so the content is not originally from me, but it's uh yeah this one understanding, humanities networking in a single picture, and it's like this, so uh it's sums it up pretty good.

B

So yeah niklas gave a short explanation on how it works and then the more you dive into kubernetes the more things you will discover how the networking works and how it may work differently on each provider and what weird bugs can come up, but most of the time, most of the time you don't have to worry about it. So that's a good part. Yeah.

C

But definitely.

B

Oh yeah, sorry go on.

C

Yeah yeah, mostly what I found interesting about networking, because we are on the company we're using a high pass data, so something like aws or azure and when you're using their virtual needs network um you currently all your pots are reachable by their ip address.

C

So if you, for example, you have a typical normal virtual machine, you can use go into the virtual machine, then ping, the part directly or it says, report you don't have like, like the virtual network over it, so that you need to do some turning or something like that, because it is mostly requires a really flat network. If you want to do nothing in there or firewalling, you will have a lot of more fun. Mostly it's like a hell. Yeah.

B

So uh most of the networking in a default kubernetes cluster is done by itp tables and to give you a little bit of an idea, let's maybe execute iptable save on this node, but that's less than I anticipated all right. Okay. So if we deploy a few more things, then this will be a lot longer, but yeah all this or everything most of what kubernetes is doing with the network is happening here by setting up ip table rules and for each container you start off for each port.

B

You start there will be a few new rules in here.

B

Okay, so I was hoping that today we would create 20 nodes and 2000 ports. I don't think we will make this hopefully next time, but what we can do is we can create one load balancer, so we are able to reach something in our cluster from the outside. Just some basic example of running a web application.

C

But for the next session, probably we can use for auto scaler to scale up two thousand nodes, literally uh 20 nodes based on two thousand parts, so that.

B

Would be yeah yeah if we uh prepare the outer scale for it, then we can do this yeah.

A

um We we also want to add storage volumes. As far as I know,.

B

Yeah yeah, I think we will be busy on the until 2023 or something with this cluster.

A

B

If you ever want to to do something different than kubernetes as in to do something else, then just uh everyone say so right. We can always take a break from this kubernetes stuff.

A

I think so what I imagined or dreamed of was like we're building something up and we keep using it and like using it with the gitlab ci cd deployments. Then, because we have a running cluster, we can actually deploy something on top. Then we want to monitor the performance and the metrics and all this stuff. We maybe even can install grafana temple for tracing and test that, um but it would be awesome to like use what we have been building over the weeks.

A

um Definitely my dream: if, if you have time in the coming weeks max so it's it's also a resource uh thing, of course,.

B

Yeah, no, it's! uh I actually really really enjoy those wednesday evenings because they are like yeah. It's a fun time to build this.

B

I will definitely make time all right so.

A

The last exercise, okay,.

B

I guess it's about 10 minutes, yeah.

A

B

All right so yeah I'm going to do this a little bit faster and then maybe next week we can take a short look back at it and we are going to create.

B

uh Yeah to create right now we are just going to create a pot like five minutes ago. I said: usually you don't do this, you create a deployment, but now we are actually going to do exactly this because to play around to test stuff. Sometimes you actually create a port. We are going to create to use.

B

Oh sorry, first, we are going to create a namespace.

B

And let's call it k3sdemo and then let's switch to it.

B

Then we are going to start in image nginx and we are going to call this thing simple nginx.

B

Now we have this pot, it's in the status container, creating why it in the state. Let's do a group ctl describe.

B

And I always forget to put pot in there all right, so it was in the state container, creating because it had to pull this docker image and pulling a docker image usually takes a few seconds or longer depending on the size. Then it created the container and it started it.

B

So we can get parts white. Then we see it has been scheduled on our agent zero. All right, nginx is a web server. It exposes something on port 80..

B

So let's expose this plot nginx, let's export, expose part 80 and do it with the load balancer.

B

We completely ignore the fact that we don't know anything about load, balancers and servers and services. Yet we will go into a bit more detail next week, so we now have this part.

B

What we also have is a service which is also often written as sovc the same thing. So the idea of a load, balancer and kubernetes is that it exposes something to the outside of your cluster, and for this you want to have a public ip address. That does not change, and then kubernetes takes care of routing this traffic from this public. Ip address to your actual pots in the cluster, that's something we will look into a bit yeah more next week and probably each time. We then work on this cluster right now.

B

It's still in the state pending, there's no external ip. Yet.

B

So we want to know why. So, let's do cube. Ctl describe service, simple nginx, that's yeah! Also this kind of kubernetes resource we can look into. We also have those events and there we can see an error message.

B

It's coming from the cloud controller manager, the cloud controller manager noticed that there is a service of the type load balancer, and then it tried to talk to the hetzner api and get an ip address or get a load balancer from it, and it failed because our service is missing an annotation and it's even telling us which one location and next time we will put this into a little bit more nice form right now we are doing this with cubectl.

B

B

And this opens the yummy definition of this object, so yeah, that's the definition of our service and what you already saw earlier. That often an object has this kind of labels. It has a name, it has a namespace and what it often also has is an annotation and our cluster is running in feinstein, which is a town in germany, and so we are going to use a load balancer also in feinstein.

C

What happened if you use a different location, that's.

B

Absolutely crazy, that's absolutely crazy because it works ah cool yeah. I I was 100 sure that this would not work, but you can actually create a load balancer in helsinki and then get your traffic to feinstein.

C

B

A

B

I was super impressed.

C

You have uh maybe better tcp hendrick time.

B

B

Your performance will definitely be better if you put the load balancer in the same data center as your in the same region as your server, but I was super impressed that they made it work across the regions. I did not expect that all.

A

Right, a challenge for monitoring and debugging is if something doesn't work. This will be interesting to try and see how to fix the problem.

B

And now we can see some something interesting now it has an external ip address. The external ip address has two things: it's an internal one and an external run. We can talk about this another time. For us, the more interesting one is the external one. So where does this ip coming from.

B

Let's look into our cube system namespace, let's look into the logs of our cloud controller manager, so cube ctl logs super helpful to look into logs of your pods and yeah. We see a lot of text, and one thing, for example, is that the cloud controller manager ensured our load balancer.

B

So the cloud controller manager watches all services in your cluster and if a new service of the type load balancer is created or it's deleted or it's modified or anything else, then and we'll make sure to update the configuration on hetzener's site via the headset api. So apparently it did something with the headset api.

B

Let's look at our dashboard and click on load balancer, and here we see we have a load balancer. This wasn't here before it was created a minute ago. I just know it switched to healthy.

B

It has a very generated name, so not very helpful. We can change this another time, but we can see that all of balancer has now two targets.

B

They appear to be healthy, so let's click onto targets to cloud servers and it's our our agent- and this is super nice because now hetzner runs a highly available load balancer for us, and we don't have to take care of this.

B

If they receive traffic on this ip, they will send it to one of our nodes. If one of our nodes goes down, it will be marked as unhealthy and the load balancer will just send the traffic to the other node. So yeah, that's nice! Let's go to this ip address. It's welcome to nginx. So.

A

We are now able.

B

To yeah expose a website to the public running in our cluster, so there is no dns, there is no ssl, but it works and yeah. That's uh it wasn't the goal for the day, but I guess it's in it's a checkpoint where we can stop so yeah, which way is our traffic going now? That may be an interesting question, because I said it a few times today: abstraction layers, abstraction layers, abstraction layers, so you're, maybe not confused.

B

I open this ip address in my browser. How does this traffic go from my browser to the pod in the cluster? The pod doesn't even have a public ip address. How does does it work so we have the browser.

B

Then our traffic reaches the hetzner load, balancer load balancer. As a service, then hetzner randomly decides to send the traffic to one of the nodes.

B

Then we have the service of type load balancer.

B

So this service is not something that really exists. I mean it is there, but it's not like a running pot or a piece of hardware. It's only virtual. So let's put this into some yeah brackets. So it's the representation of it is in ip tables. On those notes, that's where service is created and so with each service. This iptables list from earlier gets longer and then based on those iptables rules, the traffic will be sent to the correct node in our cluster. What's the correct node, the correct node is the node.

B

Our pot is running on and then the traffic goes to our pot and then all the way back and that's how we are able to look at this wonderful example, page and yeah kevin is laughing because he probably thinks. Oh, my god, this is so complicated.

B

But yeah that's basically the start of getting traffic in our cluster and another time. Next time. The week afterwards, we will see we will put another step in between it's called an ingress controller, and this way we can use a single load balancer for several websites, which is very nice, because if you look at the load balancer, you will notice that it costs almost 6 euros per month and if you create a load balancer for each website, you are running then this gets very, very expensive, very fast.

B

So what we want to do is to use one load balancer and use it for a lot of websites and still only pay five to six euro. So yeah. I think that's about it. I think it's nice. Now we have finally used more than just a server. We now are using this load balancer to get traffic in our cluster.

A

Can you go back to the overview which looked like a monitoring overview.

A

Wanted to take a screenshot of things.

B

It also has some graphs like where we can see how many connections, our local site currently has not so many.

B

So if you want to go to the ip address, I can post it in the chat.

A

So we can let live ddos yay, who is faster with curl in an endless loop.

B

Would be a bit ridiculous if it goes offline now.

C

I could run lambda without testing if you want to.

B

Yeah go for it.

A

While true to add a slight delay, don't give.

C

C

B

And the ninth nice thing is now: each time we add a node or remove a node from our cluster, the cloud controller manager will add or remove this node from our list of targets and yeah. That's there are things we don't have to do ourselves anymore now,.

B

We are exposing one service, port 80 and it's then sending the traffic to this port in the cluster and all nodes on all nodes. Exactly.

C

That's the reason why multiple nodes in there.

B

So what is the graph doing.

C

So we have no six connections. Yay, let's drop in here.

A

Maybe I should use parallel.

C

Like I said, we could build a lambda function.

A

C

A

Curl there's a curl parameter where you can instrument it to use parallel requests. I remember that, but I think it's it's not important. It's for another time.

B

Yeah but lambda function is actually niklas. You can go for this another time. Yeah.

C

B

That would be nice, I I know a few things about it, but I never used it myself, so I would be so interested in it.

C

Yeah, that's the tool I missed, but probably the promise with load testing on this stale is that you need to inform probably the cloud provider. It's not like that. You should do in light out of the box yeah because and on all the slas. Some of the cloud providers also mentioned that if you want to have really really high traffic on that, you probably need to make them aware of so that you get better rules and so on, like nothing, you should do out of box.

B

Maybe we can do it a little bit limited to just show how it works without sending all traffic we can.

C

B

All right, yeah.

A

I think that's for today, and you should also recycle the ip address after the screen.

B

Yeah, I can do that. There's nothing running there, so yeah, except this.

A

um Just like avoiding any confusion on the appearance next week, all right.

B

ah So you mean okay. Now I understand yeah.

A

Yeah, I'm assuming positive intent everywhere, but I know that there are people outside who want to just like beat us something or do something, and I want to avoid some some troubles when we come back next week.

B

Yeah sounds reasonable.

A

Okie dokie, then thanks a lot max for the uh educating session. um Now I need to unpack everything in my mind um and hopefully next week we can jump into maybe the storage volumes. The ingress controller also sounds interesting, cicd monitoring and other things. um The only thing is next week I had to move the meeting to 7 p.m. If that's, if that works, for you.

B

Yeah, that's, I think, for me, it's better yeah.

C

I'm not sure that if I can't next week, but we will see.

A

So I'm I will be here at 7 pm and if max is here, we can do the live stream and maybe like follow up on on the youtube stream. um If we decide during the next days that we want to reschedule it in two weeks time, it's also no problem. So um just let it sink in um have a great evening and, um let's say bye on youtube. Bye, bye, bye, see you around.