GitLab #EveryoneCanContribute cafe, 17 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 17. #everyonecancontribute cafe: Kubernetes deployments to Hetzner cloud, step 4

Description

Last week: https://everyonecancontribute.com/post/2021-02-10-cafe-16-kubernetes-deployments-to-hetzner-cloud-part-3/

- Demo repository: https://gitlab.com/ekeih/k3s-demo
- Twitter thread: https://twitter.com/dnsmichi/status/1362101461337985028

A

Okay, we are live now um back to english, back to kubernetes back to head snack cloud deployments back to magic, stuff maxwell tell us today in step four or part 4 of the workshop. We are having so looking forward to learn something new today and max.

B

Over to you yeah, I just noticed that my video is disabled. I think when I start screen sharing, then my video is disabled automatically. Then I have to enable it again all right, yeah welcome everybody. Today we will continue what we started the last few sessions.

B

Last week we deployed the cloud controller manager. We if you remember this one, the cloud controller manager which is responsible to organize all the ip assigning stuff that is related to our cloud provider, like the external ips and the external ips for our nodes and the ip addresses for load balancer services.

B

Last week we created a load balancer, and this way we were able to access a website running in our cluster by we're just entering the ip address in our browser, which was quite a nice first step to expose services to the internet, and last week we moved really fast in the end, and it was a bit hacky because we needed to edit the service, the load balancer service, to add the correct, annotation and yeah this week.

B

We want to clean this up a little bit, so what I plan to do, I want to show you for a second, was to.

B

This is the documentation for the load balancer stuff, with hetzner and at the bottom.

B

There is a explanation about cluster-wide defaults, where you can set some environment variables for your load as a service, and I tried this yesterday because I wanted to do this with you today and I was confused by it wasn't working and after a while, I noticed that in the repository there is a change log and those are listed on unreleased, so we can't define any default values right now, so for the for the time being, we have to add those annotations to each load, balancer we are creating, which is not too bad.

B

We just have to know that we have to do it. Oh cause. You will say something.

C

Build the image by our own and use it.

B

That's also an option yeah.

C

Because it's only a docker image so right, it's only.

B

C

The deployment and then we can use it.

B

Yeah that would also an option yeah um yeah, so we are not going to do to use those variables now, so we are going to start to look into a bit of another direction.

B

We one thing I showed you last time was that in the end we had a load balancer in this list, and I explained if we would expose every website the way we did last week. Then we would need an additional load balancer for every website we expose and with each load balancer costing around six euros per month. This adds up really fast like for 10 websites. You suddenly pay 60 euros and that's not really nice and it's possible to do this all with one load balancer.

B

So kubernetes has a concept. It's called ingress and I'm going to open the documentation for this. So in the upstream kubernetes documentation, there's a yeah there's. This is the overview page for ingresses, but there are a lot of more stuff documented about it, and the whole idea of the ingress resource in kubernetes is that if you want to expose a website on a specific host to a specific path with some configuration, you can create an ingress resource.

B

Just like you create a pod or a deployment or a service which we did last week. You can create an ingress, and this ingress includes configuration for uh yeah for the endpoint you want to expose and then, in addition to this, you run an ingress controller in your cluster, which will which will monitor the changes in those ingress objects and, according to those configuration in those ingress objects, it will configure itself to serve those websites to the public and you are free to choose the english controller you want to so.

B

The kubernetes does not force you to use one specific ingress controller. There are a few well-known ingress controllers, but in theory you could also write your own. I think the most common one is the nginx inverse controller.

B

It's uh yeah it's using the nginx web server as a reverse proxy and yeah nginx as a reverse proxy is better tested for, I think decades now, and so it's one option to use which we will do today, but there are other options as well. For example, there's traffic there is an engine, an ingress controller for h, a proxy.

B

Even if you use a hardware appliance like f5 for load balancing, you can use an ingress controller for this, which will run in your cluster and then reconfigure the hardware appliance according to your english objects.

B

So there is a yeah wide variety variety of inverse controllers and yeah. You have to choose what suits you best for today we will use the ingress nginx controller and it's a little bit tricky, because there are two ingress nginx controllers. One is developed upstream from engine x itself and one is developed by the kubernetes project and I think the more common one is the one from kubernetes.

B

I think, especially it's the older one. So more people are using it already and yeah the same for us. We will use the ingress nginx controller, there's some documentation. How to deploy it, how to configure it and yeah? That's what we will do. We will click here deployment and we are going to use helm to install it.

B

So there's a lot of interesting documentation. So if you are new to it and you want to deploy it, it's probably yeah sensible to read this. We are going to jump ahead to help and yeah. We will talk about him in a minute like what it is and what it does and why we use it.

B

First, let's go back to our cluster and create a new namespace.

B

Today I got a new keyboard, so I will make a lot of mistakes while typing, because my muscle memory is totally off now.

B

We will call this namespace ingress. So now, if we look at our namespaces, we have those default namespaces which exist from the beginning, and then we have the new one ingress, I'm switching to it. So all my commands now default to this namespace and then yeah. Let's talk a little bit about him. Helm is a package manager for kubernetes.

B

So if we there's also a website for switch it to yeah to do the full life cycle of your containers and everything and the last few times we used for uh just yammer files, for example in here here for the cloud controller manager, we just did something like cube, ctl, apply and then provided file, and then in this fire there is plain yamada.

B

So this is okay in the beginning, but the more you work with kubernetes the more yummy code. You write and it's hard to reuse this code, because every deploy, 5 difference, you can see. There's quite a bit of boiler, helm uses the github repository check out the hem chart, parts or helm, or something like that, and here we have a hem chart for ingress engine x and a minute ago I said we are going to template jana files. So maybe let's look in the templates directory.

B

Now we see a bunch of yammer files, quite the list, and I will look for deployment controller deployment.

B

Now it looks similar to the yaml file we had last week. There's also there's a kind there's metadata labels and down here search. We also have the containers in our pot, and but in addition to this planar code, you have all those curly brackets and each time you see a double curly bracket. That's a part of the go templating language which tells help of the templating engine.

B

Please don't print this as yammer. Instead template this and there's an yeah, if condition in here, like only do this, if some value is as if some variable has the value deployment and.

B

Yeah then, for the labels there's one variable, including the labels, and then then this variable is reused across the chart and yeah. This makes it quite easy to only provide templated based on this so times values we can see, though quickly those are the settings you can apply to a hem chart available depends on the hand chart some are more common, for example, and it's because often we don't need to so we only have to change one or two variables.

B

So, let's look into this in action and I think it's easier to understand so.

B

All right, let's maybe create a new directory here, let's call it helm and we are already in our ingress namespace. There are no pop ingress.

B

Nginx so let's install this.

B

Maybe while this is running, we can open my second terminal and let's do something different. So, instead of helm, install, we make template ingress engine x, ingress.

B

And then it does not directly install it in our cluster; instead, it will print the rendered template to us and that's quite a lot. So maybe, let's pipe this into less, and then we see now that there are instead of this variable, there's a bunch of labels and they are the same across the whole uh yeah output, and here we have a bunch of rules. What our controller is allowed to do then here we have a service, that's something we already know right.

B

So apparently it's some controller service web hook. Then we have the controller deployment again with the same labels.

B

And down here we have the container.

B

All right, so that's what ham does it renders all this yummy code and then it applies this to the cluster and our install command is done by now, which is nice. It gives us some help, output which we can ignore for now, then we can see that now we have pots in our cluster right. Actually, we have one in this namespace and we can also check for services.

B

Then we can see we have two new services. One is the controller itself and one is controller admission which we will completely ignore. Maybe I don't know in a few months when we are very comfortable with kubernetes, then we can talk about admission controllers, but for now it's only for us. It's interesting this controller and yeah. That's the same kind of service which we created last week by running something like cube: ctl, xposed, simple engine x, type, load, balancer, something like this.

B

We did last week and then we got a load balancer and just in the same way the external ip stays impending.

B

So let's do a describe on the service and just like last week it tells us that there is some annotation missing so now. Last week we did cubectl edit service and then modified it on the fly. But now we are using the same chart and we don't want to change it after the installation. We want to configure it from the beginning, so what we can do is to override those values.

B

So maybe let's create a yummy file, call it ingressian. We are completely free to choose whatever we want to choose here and then look. Let's look into the chart can look into our values file and then we can search for annotations because we want to change the annotations of the service.

B

And the nginx controller is quite the complex software, so it has a lot of options so for smaller charts, often you have more like 20 options and not 500, and so in the beginning you will need a few minutes to search for the right option. You want to change, but down here in servers.

B

C

B

And but the annotation dictionary is less empty, so it's a little bit hard to see here, but the service key is actually part.

B

Of the controller dictionary, so what we will do.

B

Is I don't have to open this in vi? We have visual code here right. So let's open this here. This makes copy and pasting a little bit easier. So we define the controller key in there we have service and then we have annotations and the annotation we did last week was this one. Lo balancer had snack cloud location feinstein one because our kubernetes cluster is also running in feinstein.

B

Now this way we can change all the settings. This feels like the wrong amount of spaces here, yeah right like this and yeah, let's change a few more things.

B

So, if you remember last week when we had this low balancer, it had a very weird name here. It was like just a bunch of numbers.

B

Let's give it a better name, can use an annotation for this and to figure out what annotations we can use. You can actually go here.

D

B

Cloud controller manager, I think in docs, then log balances- and here is no- it's not it's funny. It's not listed here ah here. A list of available annotations can be found on package. Go death.

C

Obviously, they are also on the official documentation which annotations- oh okay. Yes, uh sorry, I wrong topic. Okay! No! I was thinking about for engineers, annotations.

B

ah All right, yeah, yeah and for heads now we have to look at a bit in this source code or not directly in the source code. It's a co code of the it's the documentation of the source code. That's right!.

B

And here we have an explanation: what annotations exist, and one of those, for example, is the name and yeah. What we will also set is, uh I think, disable private ingress and use private ip. So this one we will disable and this one we will enable.

B

I will explain what those options do after we created it, because it's a bit easier to see when I can show it to you. Instead of just explaining it.

C

Question is it also possible on hats now that you can set the ip directly so an ip4 address and you will get assigned to it so from the space from there so interesting, because on hyperscale, it's possible to do that.

B

I don't think so, but I'm not exactly sure looking at this here. It doesn't look like it, but it wouldn't be an annotation right. We would directly set the spec field for the ip address. No.

C

No uh sometimes oh, no you're right, it's respectful, you would set it in perspective.

B

But I'm not sure if they support this, but we can. We should maybe try this out later or another time, because this would be interesting if you don't want to lose a public ip address.

C

Interesting, why can you set a cookie because I thought it would be in hardware opens or nothing?

C

B

We will now use it as a tcp load balancer, but it's possible to use it as an http load. Balancer.

C

B

Cool, so if we create a new one, uh we can select the protocol. We can even terminate ssl directly at the load button.

C

Oh, that's nice.

B

But what we will do today is use it as a tcp load balancer and if you use it as an http load balancer, then we can also enable proxy protocol to get forwarded. The client, ip and stuff like this.

B

All right more annotations, so let's set those settings for now and when we go back here to this, I'm now back in the ingress engine x, where you see- and we noticed that we had only one one pot running for the controller and it would be nice to increase the replica count to two.

B

So if one of our nodes fails, there's another controller still running.

B

So let's do this. Let's set it to two okay and then now we we already installed the ingress controller. We can check this with helm list and then we can see that in our current namespace we already have this release it's in the state deployed.

B

So now we want to update this with our new settings and we can use hem for this again upgrade, and then we pass this values file which in our case is called ingress yammer.

B

Then we pass the release name, which we already used earlier, which we can see here and then we have to tell him again which chart to use.

B

I would love it if it was able to extract this from the existing chart from the installed release that the installed release has no knowledge about its source. So there's no way to extract the yeah which chart you want to use, so we have to tell him which one to use.

B

So if I press enter now, those changes would be just applied and there is a nice add-on for help. So helen has a plug-in system which is called helmdiff it's available on github and if you are working with helm, I strongly recommend to use this.

B

It's quite easy to install just hang plugin install and then the repository, and it allows us to instead of upgrade to run helm, diff upgrade.

B

And then it shows us the diff instead of applying it and that's quite nice short.

C

Question yeah, but mdf isn't directly supported right. So it's an extra plug-in yeah.

B

It's yeah just what I showed here.

B

You have to install it first uh as a plugin yeah.

B

I I kind of I'm kind of waiting until it is an official feature, because I think everyone who uses helm is using hamdiff, but I'm not sure if they plan to include it or not and yeah. It shows that what it would change so, for example, in the controller deployment it will increase the replica count to two and down here. It will add those annotations to the service and down here. It will also create something called a pot disruption budget.

B

So that's interesting because we didn't tell it to do anything with a pot disruption budget right. We only did those changes so in the chart somewhere, there is some template which checks the amount of replicas and, if it's more than one, then it creates a pot disruption budget. That's what I guess. So, let's look into the chart again.

B

C

Should be in the template of the import description.

B

Here's the pot disruption budget. Sorry, could you explain what a pot destruction budget yeah yeah? Definitely I will do this in a second yeah.

C

ah There is it the first condition: it's like yeah this one.

B

Right, no, it's for the auto scaling. It has to be ah at the back here. It's more yeah.

C

The other is an awesome should be yeah when.

D

Hot stain is enabled.

C

Or that's, that's the part that I don't like at all. When you write complex conditions. It's it's not a complex condition, but if you need to read a lot of templates, it's hard to get.

B

Yeah, but I think it comes.

A

What the down is doing, yeah.

B

Okay, oh yeah, you go go ahead, nicholas.

C

Okay, so we're starting first within with an if and for first um braces. I currently condition the first true value of an or and the second condition of the braces. um Currently um the second value and these boost values will be compared and afterwards, after that, a final boolean result will come out so and in the first one it's an end, so it will be conducted if, um when um auto station enabled is, has a value to a and end greater than also.

C

It's also first the end, and it's greater than so. It first would check for minimal replicas. um That's are greater than one so that you have at least defined for auto, stating a minimal, valuable set of replicas, and then it will check also if auto station is enabled.

C

So when no auto station is enabled at all, it will use the second value from the replica count.

A

It's their own way of describing conditionals.

C

Yeah, if you google, much probably to go, I use uh look for spec functions.

B

How is it written.

C

C

Runway spr, ig, yeah yeah, that's true! This is currently the template in language that uses helm um and these are built-in functions by respect language, template and there's an or for that reason.

A

Because, from uh from a user perspective, this is like, uh I break my fingers when typing that.

C

It's not intuitive, it's like. If you read it a lot of times it's different.

A

Yeah from uh from from a computer's perspective or like the the small pocket edition of your calculator back in school, it makes sense um it's easier to integrate. Yeah, mostly.

C

Probably some background about for templating um is that currently the whole template language uses go template because also help is written in dolan and they're using this split function. Library, as extension to do some more advanced features because door template doesn't have an or something like that. You can use a range you can use an if, but nothing like like comparison operators or something like that. So this is the reason why it's working currently yeah. But let's come to a topic. What the pot description budget does probably.

B

C

More interesting.

B

If we look at our div here, there's a bunch of labels which we can ignore and then it says min available min available one, and this tells kubernetes every pot which matches those rules from those pots. There always has to be one available, always that's a strict condition. We tell kubernetes to take care of. So if kubernetes, for some reason, decides to reschedule some of those ports to different nodes when it wants to change stuff and then it will always make sure not to kill too many of those at once.

B

So, if you have five instances of a pot, it will never terminate all five. It will only terminate four and then, if one of those is available again, it will terminate the last one as well, but there will be always one available.

B

This, of course, only works for cases where kubernetes does this as an active choice like we are draining a node, which means we remove all running pots from a node, then kubernetes can do this as an active decision to terminate all pots on this node. If, instead this node just crashes, then those parts are gone. So kubernetes can't guarantee the availability of at least one pot in this case, but for all automatic scheduling for everything, qrinatus will yeah.

B

It will take care of our rules here and we can use fixed values here, like minimum available one. We could also choose values like at least 30 percent of our pot needs to be available, or at most one pot is allowed to be unavailable. It depends a little bit on the application.

B

How how strict you have to be about the availability. If you have 20 pots and high load on those, then you usually don't want kubernetes to terminate 19 of those at the same time.

B

It's more like you want to take you, and that is always have 60 of it running, but you are allowed to terminate the rest and if it needs to recreate all pots after another, then it will yeah only do it in small batches to always make sure this 60 rule stays true.

C

And most of the other case that can happen is like when you have a lot of parts running on one node in juveniles, and currently the node runs out of memory or out of dist. So typically, then, an event in juventus will happen that chords that the pot that's evicted automatically to a new node, because um that comes from the quality of servers and with setting a port disruption budget.

C

We can ensure that at least one ingress controller will stay all the time over, so that every application gets traffic on it, because when we don't have any ingress controller running anymore, we can't reach for cluster from outside over http as the user, mostly yeah,.

B

And those kinds of things about kubernetes are those topics you need to learn. As for the day two operation like it's easy to get started to deploy a few things, but then you need stuff, like pot disruption budgets, to really tell kubernetes what to do to be sure that it does what you want all right. So we should. We displayed our diff now, let's really upgrade.

C

There's also a cruise site that was founded by hanging yard drops where um two anything's failure stories are listed when people get outages in germany. This is kate s.

C

um And there are a lot of various um stuff.

B

This one kubernetes, as I guess.

C

No um f or something different from the symbol: okay,.

B

Okay, for me, it always reads like humanities. As I mean, the domain is probably something different, but in this case I read it like this all right. So let's check our service again, and it now has a public ip address available, which is nice, because now we set this location and now our cloud controller manager was able to create it, and if we do keep ctl describe.

B

Then we can see now the error is still here, but the error is now old. There are now newer messages and those were successful. It ensured our load balancer with our cloud provider, and here at the top we see, are now new annotations.

B

One of them was that we wanted to set a name, let's go to the dashboard and display all balances, and now we see this name is used here. It's now called ingress, which is more readable than just a bunch of numbers and yeah. It's still those his checks. The heads now now send some tcp connections to our ingress controller and if those are successful, then it changes to green here which just happened now, so we set two more annotations.

B

One of them is use private ip.

B

If we don't use it, then the load balancer will send the traffic to the public ip addresses of our nodes, which is okay. It's not it's not bad, it's okay, but what we can do is to use the internal ip addresses, because our load balancer also has a private ip address here, and we all have this in our private network, which we created with terraform in the beginning.

B

We have all our servers in here, but now we also have this balancer in here and now it will send the traffic over the private network to our nodes, which, when it's possible, I consider this nice to have uh definitely doesn't hurt.

B

And if you do it like this, then you are able to apply a little bit more strict firewalling on your public ip addresses, for example, because the those nodes don't need to be accessible from the public ip address anymore. Now, because the traffic goes through the internal network, all right yeah. So, oh here we can see as well.

B

The inverse is in there and we can also see a bunch of roots which are created by the cloud controller manager, which is for each node in our cluster there's a new route created which tells the rest of the cluster which ip addresses are used on, which node.

B

But it's a little bit more deep dive into kubernetes networking. If we start here so back to our load balancer. It's now.

D

B

Green, we can see it has both of our agents as a back-end, and it has services for port 80 and 443 so for http and https.

B

So this sounds a lot like we can copy our ip address and visit it in our browser, and we can so it's a little bit boring. We just get the 404 from nginx, because right now we have nginx running as a reverse proxy, but there's no backend behind it, which would respond to any traffic.

B

So the inverse controller or the nginx inverse controller has a concept of a default backend.

B

Let's go to our.

B

Values which is disabled by default, which you can see here so let's just try to enable it.

C

ah This changed.

B

Yeah yeah, I don't know a few weeks or months ago. They changed this.

B

Which was a little bit surprising.

C

A lot of stuff were really surprising in the engineering's ingress yeah, so it's like when I remembered when they removed the support for tcp law by anything in edp. It opens. It was an interesting time.

B

Yeah, so let's do the div again and now it's a bunch of more stuff, let's scroll to the top. So here we have our controller deployment and there's only one line changed so in our controller container.

B

There's one additional option passed to our actual english controller, which basically configures the default backend for our reverse proxy and then further down. We create a service account again, which we will ignore. Niklas today asked me via chat if we can do a little bit of a session about service accounts and permissions in the cluster, and I think we should definitely do this at some point, maybe next week or another one.

B

I guess we will talk about this after the stream a bit um and then we see this new deployment. It's ingress in the next default backend, it's a deployment again all our labels, but it's sorry um here our container and it now uses a different image than the controller it uses. This default backend image, so yeah.

B

And all this code, just based on us setting this to true, which is a lot nicer than writing all this code, ourselves.

B

B

B

uh Depending on the hem chart and depending on how fast your cluster is, it can take a few seconds so now, let's check this cube, ctl get pots.

B

Now we see we have our controller.

B

Which, to my surprise, is ah yeah: no, it's all right um it we are now seeing. We are now seeing kind of uh the rollover of our deployment, because we changed the configuration of our ingress controller and it can't change this in place. So it's creating new pots with this new configuration.

B

You can see this those here 23 seconds old.

B

This one, for example, is one of the new ones and when this one is ready, it starts terminating the old ones and, for example, in this situation right now, our pot disruption budget is also there to ensure that always one of them is available, but I think kubernetes in this case kubernetes would also do this without support disruption budget. I think, but it definitely doesn't hurt.

A

So now I have a have a question. It was a little bit too fast for me now. What is the expected state we want to achieve with all the settings.

B

um The goal is to have now two ingenix ingress controllers. Therefore, we set the replica count to two.

A

B

Those are the two controllers: that's a nginx reverse proxy and it's quite the standard configuration to just take traffic and forward it to some backend, and in addition to that, we want to forward all traffic that is, does not match a better backend to be forwarded to this default. Backend and the home chart allows us to achieve this by just setting default backend to true, then the chart will take care of sorry of creating this additional pot and to configure those reverse proxies to forward the traffic there by default.

A

uh So right now I basically have two uh reverse proxies up front: local balanced and the back end is like the default backend, which is running and handles or all the stuff which is incoming, and the five would probably overload one of the parts.

C

So it's 54 page, it could be also the maintenance page or something like that. So if the url is not matched it's like, probably it makes more sense. If you go to the browser and currently now refresh the site- and you can see the difference right now and we got not in form four, we get another result of a default backend.

A

Yeah, I think I misunderstood reverse proxy. It totally makes sense if, if we describe it more like pulling up a running 404 error message or like a maintenance mode website, then you have something upfront and in the back end it doesn't matter which type of of port is running. You always have something visible and uh your users will see. Oh, they have a maintenance window planned or something else yeah.

A

I I did that for for this course um as well like putting an engine x proxy up front, um because when you're building the docker image users would say, hey the website is down. No, the website is not down.

A

Okay, please go on.

B

Yeah, maybe a short overview of which way our traffic goes right. Now it comes from our browser. It goes to the load balance size service from hetzner from there. It goes to one of our nodes, which one is random.

B

And last time, the next step was this that it went to our servers, but this time instead, I know it kind of I think it kind of goes to our first service, the contr, the load, balancer service and the service is implemented in ip tables. Again. So that's why I put it in brackets, because it's not something that really exists from there.

B

It goes to our ingress controller, so one of those two parts and then the inverse controller forwards this again to our default backend, port and there's a little bit more in between, like this service, ensures that at first it has to go to the correct, correct node, where actually an ingest controller is running and the same happens kind of happens over here, because there's also the service for the default backend and yeah. Then it could be that it goes to the node that runs the default backend port.

B

But because this isn't complicated enough, yet this service actually isn't really used at this point, so we can scratch this again.

B

So the ingress controller, the code of it, looks at the service and checks which back ends are in the service and then configures itself to use those back ends directly without the way over the service.

B

In reality, this is most of the time not important, but if something something does not work, it gets important. So whatever I mean with services and endpoints here,.

B

Let's look at our services, so we have a new service for six minutes now: ingress nginx default backend.

B

Let's look what it looks like.

B

And in here we have an endpoint this ip address and if we do cube, ctl get bots means output, white.

B

Then you can see that this is the ip address of our default backend pod, and what the controller which is running here does. Is it talks to the kubernetes api requests? The service passes the endpoints in here and applies this to its own configuration.

B

Now I can try to improvise a bit and look into one of those controllers.

B

So I'm not now have a shell in one of those controllers, one of our proxies and we can go to our nginx configuration which is over here, and I hope it's still in the way that it's in here.

B

And maybe, let's uh not sure, let's look for default backend, maybe it's no! It's not! Yes, we can probably search for this ip address. We had over.

B

Here so that happens when I improvise.

B

All right, so this is our ingress, con or nginx configuration and, let's search for it. It's also not here. That's sad.

C

It's a little bit tricky because a default or something like that, so let's say using it tonight by now I mean a separate file, so you can see engine x, default, chrome, something like that or chrome 50, probably.

B

I'm not sure because part of the config is evaluated in lua script on okay, while it's running so I think if we create an additional ingress, this one should end up there in the configuration file could be that in some way the default backer understand it a little bit different. I'm not.

C

Sure, but what we can do is to check the locks.

B

Yeah we can do that.

C

um And doing a request so that we can see what's happening mostly on a controller.

D

D

Change this so.

B

So it's a little bit tricky because it could and oh we could check the locks of the back end as well right.

C

Yeah yeah, probably it depends on what what you want to see. Probably so, uh um do you know of it to a tube tail.

B

Yes, but there are a few tools for this, but I usually I don't use any of it, but we could also see the get pot show labels.

C

And then, okay, here.

B

So with cube city locks, we don't have to define a specific port name. We can also define a label selector and both our controllers have this label app kubernetes io component controller.

B

So let's look into those logs and now we get the locks of both controller ports. At the same time and interesting, ah I think the logging for the default backend is disabled by default. So I think.

C

ah So that you don't spam your box, okay, yeah.

B

Yeah, okay, um but you can check those.

A

Out and I already started the while loop with the curl thing.

B

And I think the default backend itself is also logging, nothing all right. In a few minutes we will deploy an additional backend and then I think we will see some lock queries and then we can back to can come back to this. So as a first thing now, I will create a dns entry for this ip address. Give me a second.

D

B

Just imagine some background music playing if anyone has any news to share now would be the perfect time.

A

Any news ah news time: uh next week we have gitlab 3.9 on monday.

A

I have I need to check out what's coming, but it will be the best release ever as, as always.

C

Then we need to deploy next video to check it on kubernetes.

A

Yeah we could do that. um I only used the one click installer, but it would be interesting to combine the knowledge of what we've learned thus far and maybe yeah, but I think now we need to learn about um users in our pack.

C

We could go with one admin user so that we have currently one plus segment. It would be fine.

A

C

D

A

I think we need elevated permissions.

C

D

C

Will do mostly yeah, but we can sense a bit.

A

uh It would it would be interesting um to provision it or to use it and then to use, for example, ci cd pipelines to install something with a hem chart or something else.

C

Yeah that, but that would be at least two sessions I would say.

A

Yeah, probably we should extend to 24 7.

C

In live streams only.

A

I don't know how long a livestream on youtube can last.

B

I think very long, that's not the problem yeah uh yeah, if we should wrap it up uh sooner than just. Let me know.

A

No, I I just want to like getting things done. What else is missing um for like understanding the ingress controller? I think I get it now. I just need to like redo it on my own.

B

Yeah, so the next step, now that we have a dns entry for it, I now have a dns record for wildcard. Everyone can contribute dot for lli.com to forward to this ip address right now. It tells us it's insecure because we don't have a certificate yet, um so the next step would be to set up cert manager to yeah get an automated ssl certificate from let's encrypt.

B

We could do this now. We don't have to do it now.

A

It sounds like uh two minutes that.

B

Yeah more like probably more like 10, but we can do it all right.

B

I I will just go ahead right um all right, so top manager there's also a nice documentation, page for site manager, which I will skip a little bit now. But if you want to check it out, just look up sitemanager.io and it's the standard software to manage ssl certificates in kubernetes.

B

So we will go back to our shell. We will create a nice new namespace.

B

All right so again, we have to add a repository to our home. Instead, in this case, it's called jet stack, which is the company developing sub manager, and then we have to do something new. We have to have to install something called custom, resource definitions or often called crds.

B

So if we do something like cube, ctl get pods, then the object type. The kind pots is known by kubernetes by default. What is, for example, not known is cube, ctl get certificates. This should produce an error. What so I just don't know.

C

Sorry, you cannot write cert, it's the same. It's for sure, okay,.

B

Okay, all right thanks um and the server doesn't have a resource type certificate. So what we are going to do is extend the api of kubernetes and let's just accept this. For now we just applied some magic jammer files, and now we can query certificates.

B

They there are no certificates yet, but now the api server of kubernetes understands a few new objects. One of them is certificates, another is certificate, requests or cluster issuer or challenges or issuer or orders.

B

We don't have to take care of all those things things right now. Most of it is relevant for cert manager itself.

B

So next thing we are going to install the cert manager again with helm. It will now render a bunch of yammer files and then, let's see yeah, we now have three pots running one of them insert manager. The other two are also part of sweat manager.

B

So, if you are not familiar with, let's encrypt, let's encrypt is one of the best things that happened to the internet in a long time. It's a free ca certificate authority that provides you ssl certificates.

B

The only thing you need to do is to prove to let's encrypt that you are the owner of a specific domain and then they will give you a certificate to handle encrypted traffic and before, let's encrypt existed, this was expensive as hell and therefore a lot of websites did not support ssl and yeah we're terribly insecure.

B

So, let's encrypt is very, very good and it's a non-profit organization and yeah they're, very nice, very good part of the internet.

C

Also, there's currently also a second uh new free, ssl acme that you can get so there's also zero ssr.

B

Oh nice, I didn't know this all right so for let's encrypt, we need something called an issuer and which is kind of the account or let's we will create kind of. Let's encrypt account to to ask for a certificate, and now this is new one of the crds we just applied it's a api version, cert manager version one now and kind cluster issuer.

B

We will give it a name. Let's encrypt, we will have to provide an email address uh 30 days before the certificate expires. Let's encrypt will write me an email about it. We will have to define to what uh server we want to talk to. In this case, it's the production signing server of let's encrypt, and we have to tell how our challenges are solved.

B

When we ask let's encrypt for a certificate, they will give us a challenge which we need to solve, to prove that we own this domain and it's possible to do this via http and it's possible to do this via dns, and we will do this via http and we will tell our site manager that please should use the yeah.

D

B

Inverse controller running in our cluster or well to be specific, we only tell it there's an ingress controller running which uses a class engine and that's all it needs to yeah.

B

Do everything in a way that the ingest controller can answer those http challenges?

B

Basically, um let's encrypt will give us a string, a random string and we will serve this string from our domain and then, let's encrypt will believe that we control this domain and give us our certificate and all those things are done by sub manager all right. So we have this issuer.

D

C

B

C

I don't know I posted also the link to the um I've, seen literally where everything is explained, how we uh whole stuff or flap.

C

Let's encrypt it's working, so it's a protocol, it's called acme and um when you go, for example, to the http challenge going down a little bit, so this for my referred to in between four yeah and currently, when you currently see years before it works so literally, like you said, yeah token will be generated by the acme server and the acme server afterwards tries to do an http request to the um requesting server. So when you go down a little bit, um it will use a specific path.

C

So it will use slash well now well known, as mostly in the internet, for private stuff of a website and then add media challenge and there's the token that leaves post and you need to um respond within 200..

C

um This is how it works on a detailed level, mostly um yeah, and it's a really really interesting paper. So probably it's not the paper. It's like. There are a lot of other tools, also on premise environments. You can also use these acme projector to automate your whole um ssl insurance. If you go to a self-signed way, mostly.

B

Yeah, nice things um yeah, so I think about an hour ago in the beginning I mentioned the concept of an ingress controller and of ingress objects in our cluster. So now it's now we have a running inverse controller. We have a running cert manager now it's finally time to create an ingress to expose something from our cluster, and for this I want to deploy a bit of a hello world application.

B

We are going to use this hello, kubernetes application, which is a nice looking demo, which displays which port answered a request and here's a nice yama snippet for.

B

This right so at first we have a service. uh We are not using a load balancer service because we don't want a load balancer our load balancer is our nginx. Invest controller. Just want an internal service for this, then we have this deployment which uses the hello humanities image, and we can provide a custom message, for example,.

B

D

Contribute how many typos did I do? Everyone can contribute.

B

C

Mostly, can you write back to the servers at least the type, so it would be by default when you don't set anything, it would be set to crosstaip. So.

D

It's more intuitive.

C

For the user and but what it would probably also explain what the cluster ip is instead of adoptable and so uh yeah, you.

B

Feel free to explain why.

C

Okay, um yeah, really we don't use an open answer. um That means when we would use for type load balancer, it would mean that we, like we saw before already that really opens on hexa will be created, but currently um we want to root a low brain scene only internally in which we need this cluster.

C

So for that there are internal open lenses and they are called plus dips, but because the big problem of our pot is when a pot dies slowly, he gets every time he gets a new ip address, but, for example, for the ingress he doesn't recommend can't recognize over ip addresses.

C

So for that case he will ask the service that has a stable ip address. So when you check it literally, you can see it will every time users like um much already outsourced print before it will use for 10 43 20 78, instead of going to all the pot ip address. So when you or white we would also see the internal appearances. I think uh by that keeps you tall reports that should be also um yeah. You can see so it doesn't use the um 42 ip addresses. It will use directly.

C

The ingress will use only the 43 uh ip address space, and it's also important that uh separate spaces plus ips are in a separate space like the pod ip address spaces.

C

It's it's like it's a software operator in place, so it uses ip tables like we also saw last week to manage all the stuff, but you can also change this to a mod once.

B

uh Yeah, that's a crazy part. You can change almost all things and yeah.

C

Yeah right with other.

B

C

Yeah, like what we currently see is that journey. This is not a tour that you're using it's more like a framework for building something. So it's like. We can build our own platform that we want to have and use it for our use case. Instead of that, we used to sum products, and you can extend it like you want. So, like writing your own custom, cds and writing your own controllers for special workloads that you have everything is in place for that out of the box mostly, but you need to do it so for.

A

Me for me to to learn the customized message has a load balancer, we don't use that, um and the cluster ap address from hetzner is what we exposed earlier.

C

Now we exposed earlier the local answer, so that's the real ip address. So when you um checking the um ingress again yeah, it makes more sense. So you see well, the type is different so and at class ip doesn't have an external ip address.

C

It's only for internal kubernetes networking, ah okay.

A

So that you don't know the whole round trip.

D

C

Out of out of your cluster and going back in it makes no sense it's right and for that it's using ipvs, underwood, okay,.

A

But um what the load balancer type.

C

The type open so what it does, if you describe the log balancer again, probably we can see the event what's currently happening like repeating this from last week,.

A

But I thought we have a load balancer in head cloud.

C

Yeah yeah, that's the reason so because we are you can see. The events is mostly important because, when we're using a juvenile service like law, balancer, the controller takes this service and asked hetzner to spin up us a real low against the api. So a class ip doesn't do this at all it only it's only there. So there are no events when we would probably check the.

C

Yeah there we should also see the re-translation loop and wait when the object is happening.

D

And she looks like here.

A

Yeah, okay, so the ingress controller is acting as the load balancer and there are no other load balancers like created in headset cloud savings. Okay, what was the idea from last week.

C

Yeah but yeah, that's true and now coming to a big important point. So the difference between the service load, balancer and our ingress controller is a service level benches only on layer four. So it's only tcp ip. It's nothing special. But if you want to do an http request, we're going in the aussie layer higher, so it would be layer 7.

C

It could be also some different protocols. It doesn't need to be http, but um on that then we want to serve only about, for example, want to enter in url. So we don't need to enter an ip address and all the stuff and that's the reason why we have a low for after the lower and seven ingress controller, mostly so because we're going from layer four to layer, seven in the oc model.

A

Okay, but where is the load balancer? Is this the headset.

C

This is yeah. This is it's it's a real hex, not open, so we.

B

Asked each time we create a service of type load balancer, when we really have literally the word local insight in here, then the cloud controller manager will always create a matching ingress matching local lenser in headsnot cloud.

B

So there is it's a one-to-one connection between the object here and the load balancer service in our cluster and each other service of type cluster ip. It's also kind of a load balancer, but it's not a public load. Balancer, it's internal in the cluster like behind the service, could be several pots.

B

That's why it's an internal load, balancer.

C

Can you check now again the end points of our new deployment of our new workload? So then we probably then we can also see it. Also, it's more intuitive then, because we didn't.

D

C

Yeah on cheap situated ep is the same and we will create all the endpoints.

C

So really what you can see here now we can see if the um same service currently has the ip address. 43 and currently the servers has also multiple endpoints and these the real ip addresses of the pods.

A

Yeah, that's only internal to the cluster.

C

Yeah only for internet and because the ingress controller, the engine x, is also in the cluster, but also from from the front side. It's exposed publicly within real ip address. But all the communication from the interest controller will only be done by the internal ip addresses.

A

Can you max have the the network traffic flow description text editor thing.

B

Yes, I think it's to hear some where.

D

Yeah yeah, okay,.

A

B

This service is the load balancer service.

B

And back here, what the ingress controller will use.

B

Is the cluster ip service.

C

A

This is basically is a management management demand management thing, um assigning it correctly or like routing the traffic correctly actually yeah.

C

Mostly, it opens.

A

Yeah, I think I think I have a different understanding of load balancing um or I kind of confused it. So I'll keep asking lots of questions. Yes,.

B

Please, please ask all the time: that's why we do this.

B

All right, so the last part missing now is the connection between the ingress controller and our new service right. The inverse controller is not aware of it yet so we go back to our demo here and now we create this magical ingress object. I told you earlier about and for a second I will also the documentation about it again.

B

So because, if you want to learn more about it, you will definitely have to go to this page and there's a next there's this example how an ingress resource looks like and then a bunch of explanations, what the different options are and everything.

B

If you ever uh configured web server, then you will find the same patterns in here like in apache. You have virtual hosts and in our ingress resource we have a host which is very, very similar, and then we have different, for example, in nginx. I think it's then called a location.

B

uh Then this is which is a path in an ingress resource.

C

So bacon is mostly for upstream of an engines so because nginx or all ingress object doesn't allow us to set it only to specific ports, because it could be the failure that the pod could fail and we want to do it open soon. So we are only allowed to name a service to root for traffic. To that.

C

So that means that um we need to write it down. Yeah, probably we can deploy it now.

D

B

So let's apply this now we see the service deployment are unchanged, but the ingress is now created.

A

Can you show me the configuration the yammer again for a moment.

A

I would like to take a screenshot thank you.

B

And so now you have a few seconds ago. There was no address now there is an ip address and if we look in our ingress namespace at the services again, so you can see that it's exactly the same ip address like our load balancer service, it's the public ip address of our headset cloud cloud load manager.

B

So this is here now because our ingress controller now picked up this ingress resource and used it to configure itself and also wrote back to it that it's exposed with this ip address. So earlier I already configured a dns entry to this. So let's look in the browser I think over here. It's still no ssl. Yet so there's a little interesting hack in chrome. You can now type this is unsafe and then it reloads and completely ignores you know. Let me try again and yeah.

B

It completely ignores all the security features that are supposed to block it from accessing this page. So now we can see this demo application that we have running in three different pots still without https, and if we refresh the page a few times, we can see that the pod name here changes so each time our traffic hits the ingress controller, the ingest controller decides randomly, which back end is used.

B

That's why all traffic is served by different pots. So that's already nice. We can see our custom message here and now we need to switch on ssl.

B

And therefore we are going to add a few settings to our ingress.

B

One of them is the tls section.

B

um It feels a little bit redundant, but this one is for the web server configuration and this is for the ssl configuration we have to do it in both places and then we also add an annotation that annotation is necessary for the sub manager, because the third manager will also monitor all the ingress objects, and if it sees an ingress with an annotation like this, then it will use the cluster issuer with this name to create an ssl certificate.

B

So earlier we created this issuer called. Let's encrypt, that's the mapping. There.

B

And let's apply this again so now our ingress was reconfigured and now what we can do is cube. Cdl get pots, and now we see there's a new pot. This one was created by our assert manager and we also have a new ingress for a few seconds. No, because there's this ingress, which was created by site manager, now third manager created this ingress, and now the ingress controller again will pick up this ingress and configure itself to serve.

B

Okay, now it's already gone, but to serve this port um yeah, also under this host name and then, let's encrypt uh queries, our domain and this pod answers the challenge and therefore prove proves that we own this domain.

B

So by now we should be able to run cube. Ctl gets her certificates.

A

That's really nice by the way, because I really struggled with integrating let's encrypt into my docker compose container setup. Doesn't work yeah. It.

D

A

But the renewal doesn't work for some reason.

B

Yeah, I think this uh cert manager alone justifies to replace a docker compose setup with a single node k3s setup, just because you get the kubernetes api and all this stuff based almost for free.

A

Yeah because yeah.

D

A

I'm running the everyone can contribute.dev uh page, and this is a docker compose whatever thing with, let's encrypt and it doesn't work, um and you got me an idea, thanks, yeah.

B

Yeah you're welcome and now uh site manager created the certificate for us. It's now officially signed by let's encrypt it's valid for 90 days and 30 days before it expires, the site manager will take care of extending it. So let's go back to our website.

B

I think I need to open it in a new tab for chrome to realize yeah. Now it's not red and unsecure here anymore. Instead, our connection is secure.

B

We have the valid certificate from its encrypt valid until may this year, and still each time we refresh it. We get a different back end here like a minute ago, so yeah, that's the whole chain of things to do to expose this ingress.

B

So the nice thing now that we have this ingress controller is that we can use it to expose as many http websites as we want. So all we need to do is create additional ingress. For example, it uses a little bit of a different domain, not hello instead of the world and down here we use the same configuration for the back end, but if you have more complex application and we could define- I don't know- the admin backend, for example, is running.

B

In a different service, for example, then we could use this here and the inverse controller would automatically configure itself to match this configuration.

B

So if we apply it again, oh sorry, I think I copied somewhere something wrong here.

D

B

All right so now again.

B

We have this ingress, we have this challenge.

B

Ingress and if we wait a moment, for example, we should get a new certificate, we already have the new one, but it's still in the state ready false.

B

We can, for example, also see that there is a pending certificate request. I think now it's done yeah now, both already so, for example, if the certificate never switches to true, then it is a good idea to check the certificate request, which is a little bit off yeah under the hood magic of third manager.

B

So now we should be able to go to this website as well and it works.

B

It also has a valid certificate, and now we can do this with as many domains as we want, and this is very nice, because if we operate a kubernetes cluster and our co-worker, our friends want to publish some websites in it. We can provide them with one single inverse controller, which is all running behind the same ip address, one looper and software cloud provider, and then the ingus controller takes care of sending the traffic to the right, backend and taking care of all the certificates with sub manager yeah.

B

A

So my me- or I want to build now or whatever I don't- probably doesn't matter where it runs, but um a kubernetes cluster in a cloud and sponsor the everyone can contribute.dev domain, and then everyone can like come to deploy something from gitlab to uh to this thing. This would be interesting.

A

But we need to um uh so I cannot speak english today. um We need to sign off for today. I think we need to. We need to find the finished line for today.

B

Yeah yeah, it's I yesterday when I prepared today, I really really tried to leave stuff out and it's still too much.

A

No, it's not your fault, it's I think it's a learning curve. um I think I didn't understand everything, but I asked a lot and my problem or my like blocking point with kubernetes always was it was too much and I didn't have time for 10 hour courses or lectures and I'm learning by example and you're, giving me like the best examples I have so I can ask you, as my trainer, my personal trainer, um so I'm very thankful for for all.

A

You do and help us dive into the magic stuff of kubernetes and also headstart cloud. Of course.

A

Is there anything else we need before we wrap up what no coming next week.

B

um Not entire issue, I guess we have to check with nicholas. Maybe he just wants to prepare stuff for the user management and permissions for next week.

C

I can do so should we we can use data for that, so we can use github uh as the little piece I can login into the cluster, so we can use open id. I can also explain a bit how open id is working um and then we would use stacks. I think for that, but I will figure out how we can do.

A

That what I, what I want to do- and we can do this or we should do- is do this asynchronously.

A

I want to create a hexner group in my account, um so we can play around and I expand everything which we kind of do and find a way to. I think I can just add your account.

A

If you provide me with the account you can just indent us to a group yeah, I can add you to a group and then you can like we are working on the gitlab repository or the github group and on the headset cloud group, and I think it should be should be fine, and we can't like recreate what what max created before in the demo and yeah. This should should be working. If I haven't done this by friday, uh nicholas please ping me.

C

All right, I wrote me a.

C

B

A reminder to remind him.

A

Yeah yeah, it's it's! This.

B

Thing of time, um if it didn't happen under saturday, I will bring nicholas to pinker.

A

Okay and who else michael eisner who's, not here today, will ping ping uh someone on set on sunday, um okay, yeah. That's this great um anything else.

C

A

You use a management next week and then afterwards we have many different ideas. We are looking forward to that um next week. We can do 6 p.m. Again, um if it, if it's possible for everyone.

C

If I'm not doing something special, I think yes, but I will inform early enough. So if I see it on monday or something like that,.

A

If it makes sense, we can also stick with seven, but I'm not sure about it, because it's like family time in at least in europe.

B

For me, 6 p.m is okay or beach time for some people.

A

Yeah some people- um and I know where I at so I will like use my google map zoom and in stock. You.

A

C

Probably yeah, that's fine.

A

um Okay, then, if we have anything else we can, we can think I think um some sounds strange, um connect, async and yeah thanks for the session, and I want to say, bye, bye on youtube.