GitLab #EveryoneCanContribute cafe, 7 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 24. #everyonecancontribute cafe: Automate Kubernetes setup deployment with GitLab CI/CD

Description

We are learning how to deploy and secure Kubernetes into Hetzner cloud in this series.

- This week: https://everyonecancontribute.com/post/2021-04-07-cafe-24-automate-kubernetes-deployment-ansible-gitlab-cicd/
- Last week: https://everyonecancontribute.com/post/2021-03-31-cafe-23-automate-kubernetes-setup-hetzner-firewall-feature/

A

Okay, now we are live on youtube hello again for into our 24th. uh Everyone can contribute coverage for me, it's the vacation edition, um but for max it's like we're. Gonna continue uh what we've learned thus far of the q as setup and the automation and rumors do tell that. We are like combining it this week with gitlab cicd, but I don't want to spoil anything so I'm looking forward to marks and hand it over to you unless you, unless someone else has something against it. um No, so please go ahead.

B

All right, yeah welcome um yeah. Today we want to take all the stuff we built in the first few sessions and put it into one gitlab ci pipeline, and here it's mostly we do. It live just trying to figure out how it works. I tried the first few steps yesterday in the evening to make sure it's not going to be a disaster, but most of it we just need to try out and also, I think in the core here, like everyone knows at least the same amount of stuff about gitlab ci as I do so.

B

If you have any ideas, how we could do something better or different or anything, please feel free to tell me we can just try it out.

B

One thing I wanted to add to last week because I I think we were talking about the firewalls and we came across this topic that the where I told you that in my case, the wire guard connections did still work after I applied the firewalls in my private cluster and nikola's. First reaction was connection tracking, and my answer was yes, I thought so as well, but I restarted the notes and it still worked and yeah by now.

B

I figured out that it's the course is still connection tracking, but not the connection tracking on our notes, because this firewall isn't even applied on our nodes, it's applied on the in front of them and they managed firewall from hetzner, and so they don't care what we do with our nodes. If we reboot them fast enough, the connection can be established again so most likely. If we shut it all down and wait for a bit and start it up again, the traffic would be blocked as expected.

B

Yeah. That was quite interesting. All right.

C

So that's a little.

B

Bit of an answer answer to why this udp traffic still worked.

C

So I've also questioned regarding that probably or an idea, um because I was wondering in the last time, if you're using hetzner and you spin up a new machine, for example, you get every time you get also the same ip address. Mostly.

C

That could be also a reason because you, when you are in the time zone or in the time window, when you get a psycho p address when you are fast enough, you get the same ip address for the machine. If you are waiting longer, you will get a new appearance as soon as it's the same behavior, also on the firewall. So.

B

Yeah, that's possible yeah yeah, oh also. Sometimes I'm not like wow. I recreated this server five times and I still have the same ip address in the openstack cloud. I know as well there it's like yeah, you lose an ip and it's gone forever. You can need a thousand tries to get it back.

B

Yeah, that's about uh firewalls, so that's yeah, definitely interesting, and so our first instinct about connection tracking was right, but it's yeah at a different place than usual all right. So if you didn't uh attend last week, just forget what he said all right. So so far we have terraform to set up our machines and our not networking on hetzner, and then we have ansible to set up the k3s cluster in there and then we have a few.

B

I think we have a few yama files with just some kubernetes manifests. We applied with cube, cdl, apply and right now, I'm not even sure. If we also had some help charts already. I think we had not sure right now. Today at least we will put terra from and ansible into the github ci. I guess, and if we are fast enough we may add the installation of some kubernetes manifest as well, not sure um yeah.

B

Let's start uh it will probably start a little bit uh slow because I'm super exhausted, but we will see all right.

B

So you should be able to see the repository on gitlab.com right now and I prepared a little bit in the settings we have in settings cicd or variables, and I added an hcloud token and I added an ssh key and it's the token with read and write access to the hetzner api and our everyone can contribute project and then there's also a private ssh key and we use terraform to deploy the public key to our nodes. And then we use ansible with this private key to access. The notes, short question.

C

Yeah, doesn't it need to be written in capital, letters.

B

Yes, for for for the ansible entire form to work- yes, yes, but we will use, it has to have different names anyway for terraform and ansible, so we will just define the variable there based on those values. But yes, written like this is what it won't work with ansible or terraform.

B

It's a little bit mixed that the ssh key is and capital letters and h-clad token, not.

C

Yeah but the essay is the file, so it's also.

B

Now your sound is interrupted.

C

My sound is interrupted.

B

Yeah yeah we've freezed, for but now it works again.

C

Okay yeah, but the difference also that there's hp's a file, another variable yeah.

B

That's yeah, that's also a thing we should be aware of right, so the ssh key is a file. So what gitlab does here is to put the content of the variable into a file and put the path to the file into this variable, which is nice because, usually in ssh key, we want to use as a file when we pass it to a other program, because we will pass it to ansible but yeah that one is in capital. Letters and one is not as more random.

B

I don't want to change it now, because I think then it will reveal the values all right, but we have those two variables in here and um yeah. One thing I also changed is in general pipelines I disabled a public pipelines mostly because when you enable public pipelines, everyone will be able to see artifacts of the pipelines and later on, we will put some things in the artifacts. We don't want to have publicly accessible only for members of the project, all right.

C

Oh, that's poor. I didn't use it.

B

Yeah, I don't know if it's always enabled by default or if I'd enabled at some point in the past, but yesterday, when I because later on, we have some step in the pipeline which puts the cubeconfig in a artifact. So we are able to download it, and I was thinking way is this artifact public and it was yeah?

B

Okay, all right. So let me change the window.

B

Hopefully my browser doesn't crash again and because.

B

All right, yeah, some read me file. That's our yeah, just our demo repository we used for weeks. Why does my shell not work down here.

C

B

Teamwork session isn't.

C

C

B

Wow, let me restart visual code really.

B

D

We probably should rename the meeting to everyone can restart.

C

A

Okay, but uh uh choke choking aside uh today I figured we have the gtld.cafe available or it's like you can buy domains, and so I registered one. I just need to figure out how to use it.

D

Are you you're one of one of the people who um get a domain and then wonder what am I going to do with it now.

A

D

I I I feel uh a bit called out here. I'm.

B

A

Yeah, it's so better better having a domain reserved than looking at it and said damn. I wanted to register that two weeks ago and no no someone else has um so yeah. I I think I have fifty percent you in use and fifty percent are like. Oh maybe in the future,.

B

All right, so we we are back with a shell yeah, I don't know. Maybe I should stop doing stuff with the computers anyway, we will so. I will just a little bit assume that people know what a ci pipeline is right and maybe even worked with gitlab before, and so, if you watch this video afterwards and have no idea what the pipeline is then yeah, maybe look it up first and then come back and watch the video.

B

So first thing we will add a file dot, gitlab ca.

B

And then we put on some stuff.

A

I will be adding uh the urls to the docs in the blog post and link the blog post in the video description and then we'll save okay.

B

We will do, we will create several stages in our pipeline. uh Let's define them here, we will create for each like. We have.

B

Naturally, we have different stages like at first, we want to apply terraform and the next step. We want to apply ansible and for each one of those we want to have a step where we can see just the div in a draw iron that would be applied, and then, in the next step we really apply it.

B

So we have a little bit more stages here.

B

Then, if you remember, one of our ansible roles was to pull our cubeconfig file to download it to our local machine, and we will do a similar thing in the pipeline to download the cubeconfig and put it into an artifact in gitlab. So we can then download it in the browser of the.

D

B

All right, so one thing we do first, is to for the time being, we want to run our pipeline only if it's triggered via the web browser, so we don't run it for each commit not for each match request or anything only if we trigger it via the web interface.

B

This might be a little bit unusual for a ci pipeline, but our main goal is for now is to like have a way to create a kubernetes cluster with one click and so for now. It's like we just click in the web interface of gitlab, and then we get a cluster back and later on. If we want to, we can change the pipeline a bit to also yeah apply all changes with each commit, or something like this for now.

B

Let's do it like this, so this workflow setting in gitlab ci is like a it's like a gate before the rest of the pipeline.

B

The rest of the pipeline will only run if this is true, and if it's not true, then it will just stop running any pipeline at.

B

All all right, so gitlab ci has a nice feature of template jobs. So if we have several jobs with similar settings, we can create a template for this. So let's create yeah. If we want to create a template job, we can just start it with a dot.

B

Then it's not a job itself, so there will be no instance of this job created, but we can extend from it all right, I'm not going to type the full image name, because we are going to use the terraform image of gitlab because they are so nice to provide us with an image that is already prepared to use terraform inside of gitlab ci.

B

And then we will define some variables and if we go into our terraform code and our main terraform file, then at the top. We have this variable h cloud token defined, and I think this goes into the direction that niklas mentioned whether how this variable is called or not.

C

B

All right and for teraform to pick this up from the environment, we need to call it tf, var in capital, letters and then the name of the terraform variable, and there we put our hcloud to environment variable, and then we define a few more terraform settings. We set up the teraform root directory and there we can use this predefined, a variable which comes with the gitlab ci, which is the project directory.

B

So no matter where our project is cloned on the actual ci runner, this directory this environment, very, we will contain the right path to it and there we want to go into the subdirectory terraform, because that's where all our terraform state and terraform files live.

C

And um you don't need to specify vtfr h2 token. You can um save it, because the provider by itself can read directly from the h12 total environment. Variable.

B

ah Okay, but then we needed to set it in capital. Letters probably right.

C

Yeah, but now it works, so it's mostly for everyone, who's wondering you cannot services. So it's mostly that all terraform providers support the default variables, mostly.

B

B

Okay, so then, when we commit this, then it's noted there as well yeah for the terraform address, that's where the url of our state backend, and we can also derive this from some existing variables.

B

So this variable gives us the api url of our current gitlab instance, which in our case is gitlab.com api, slash, v4, slash something I don't know, and then it puts our own project id here and then you know we define this api api url to store our terraform state.

B

If you remember from our very first session that we did a terraform in it and there we also passed this url and it's the same like the layer variables will fill in this part and the project id and because in gitlab ci we need the same.

B

But it's nice that we don't have to hard code, this url in our pipeline, because this way we can, we could later extract this job into a external template and then use the same and it would work with different projects as well. So, let's use the variables here.

B

All right and then we can also define a cache which is nice to use the same. Terraform and terraform runs. This creates a few files and we want to keep them between our different.

B

B

And we only keep this hidden subdirectory of terraform and then we can create.

B

We can change into this directory. While I write this, I wonder we could probably also use our tf root here right. You could also do it like this. No.

B

But I think at this point it doesn't really matter all right. So if we apply a commit this then exactly nothing happens because it's there is no job here. It's only a template. So far, so now, let's create a few jobs based on this terraform template.

B

For example, we want to run terraform validate, which checks all our terraform code. If it's even valid code and we'd add this step to our test stage.

B

I can close the terminal for now so, and we can see here at the top. We have created this test stage all right and we extend our terraform template so right now it's the same as if all this stuff would be down here as well and yeah, then we create a script to tell gitlab what we actually want to run and we run to run gitlab tire from init and we want to run gitlab tear from validate so a gitlab terraform is a minimal wrapper around the actual terraform binary.

B

uh I haven't looked into it. I think it does. Some stuff like automatically tells passes the parameters to auto, approve the changes, so you don't have to type yes, if terraform wants to do any changes and stuff like this, so they provide this as part of their terraform image and yeah. There's a terror from gitlab documentation.

B

We can link it in the blog post as well, where it's basically most of it is an example from there. So how to that, you can use this gitlab terraform wrapper to run terraform in your pipeline.

B

C

B

Already our first job, so, let's create a few more more interesting than validate, is terraform plan and.

B

There we can put it into this stage, terraform diff, and we can extend our terraform again.

B

But this time we run a little bit different stuff. For example, we run gitlab tara for.

B

Plan and we can also run github perform plan json, which I think it writes it as json onto the disk, and then we can tell, I think, by default. Terraform apply users use it to json then, but we can check the output later on.

B

And then let's create a few artifacts based on.

B

B

Okay and call it plan, and we want to.

B

Put the cache of our terraform plane json command in there.

B

And then we can ah that's why we create the run terraforms plan json right, because we can now tell gitlab that we have a report in here and it's a terraform report based on this plan, because then, if we create, for example, merge request on gitlab, it will display yeah. This match request will do the following terraform changes: it will create five resources, modify one and delete none or something like this, so you have as a reviewer and gitlab.

B

You have some context that there is terraform running in this pipeline and that if you merge it will change stuff which can be helpful because you yeah kind of directly see it right and we can expire it. I don't know in two days or something, so we don't need to keep this for long.

B

Or maybe when we assume we review stuff, maybe let's do one week and then now we have validate, we have plan, let's also apply it.

B

And we do this in the next stage. Now it's not terraform diff, but just the terraform stage and again perform.

B

B

All right now we again overwrite our script to change what we actually do.

B

And this time we run gitlab terraform apply, which is just the same like when we run terraform apply on our own shell, like we did the last few times, but this time it won't ask us for any confirmation. It will just apply.

B

Yeah, I'm thinking yeah, let's we can add a dependency.

B

I remember that I did this in my own setup, I think by default, each job gets all artifacts from previous jobs and if we define a specific dependency or an explicit dependency, then we only get the dependencies from this job.

B

um I'm thinking in our case, I think it wouldn't matter. Maybe we can try to remove it and see if it still works, so we don't over complicate stuff for now, all right.

A

I think it should work, let's try it out and maybe try the dependency later.

D

So if I unmute myself, I can talk to you wow, it's exactly as you described. So if you don't specify it, you get all the artifacts and if you specify it, you get only the ones from the jobs you specified.

D

B

So, let's push this while it pushes. I will change my shadow window again and now I almost pressed the leaf button.

B

All right, we are back to firefox, we will go into our ci cd tab and we see that I failed a lot of times yesterday when I tried to prepare and let's just click run pipeline- and here we do this for the main branch, because we just pushed into the main branch and let's go.

B

If, for example, we wouldn't want to put our secrets like the token and the ssh key into the settings of the project, we could also like not put them in there and only put them here on demand, but for us, it's okay to set them globally. Here.

B

Then we can see, we get three jobs, therefore validate plan and apply the different stages and then okay, it already got picked up, because yesterday I don't know, I waited 10 minutes for some of the shared gitlab runners to pick this up um but yeah today it seems to be a little bit faster.

B

When we have our cluster running, then we can.

D

Think of adding.

A

B

Autoscaling runners- I think we talked about this in the past past- that we might do this, but right now we are fine with those shared runners as well. Okay, validate succeeded.

B

Let's look into this, for example, what we could do right now. We don't do this, but we could change it that the terraform apply only is run if we manually click here to run it like, we could check the diff and then, if we agree with it, we click to apply it, but for now we can just go and always run all the jobs.

C

B

A

C

This is the difference between continuous delivery and continuous deployment. So, if you have a button, you made more secret.

C

It deported the continuous deployment.

B

Then we can see in our diff that we would create 13 resources, so this looks familiar right. We did this a lot of times already.

B

And now the pipeline failed: let's look into it! Why the apply.

B

B

Okay, that's interesting.

B

It doesn't even make sense, I mean the token worked in the div and it worked for the first few. Let's retry this not sure why the head snare api didn't want to talk to us.

A

Maybe it's the same problem as last week with resource deletion, rest conditions.

B

No, I don't think so.

C

Yeah but plan doesn't execute it against the api. That's for mostly reason. So you implant, you can't see if your token is valid or not.

B

But it has to talk to the api during the plan right because it needs to check if it needs to change stuff. Yeah, that's correct! Okay, I mean it only access it with read access. So if our token has read access but not right access, then it could happen. uh Okay. Now our plan does not work anymore because we have this stored plan in our artifact and it does not match the actual state anymore, because the first few things already got applied.

B

I think we can see it here right, the first few got created and then, if you started to fail, let's just run the full pipeline again, I guess.

A

But it's good to see that it fails. So you know when, when you're in a like in a mixed deployment state- um and you try it again- you need to recreate a plan and then do a diff and apply it.

B

Yeah, so the diff matches what you actually apply.

A

B

I haven't seen this before.

A

So that's nice.

B

Theory, you know our diff should be a little bit smaller, because some resources already exist. You can also check here in the headset cloud console if we go to security and no no key here yet. Okay.

A

Theoretically, we could also like build something like unattended upgrades for the deployment. Then.

B

Yeah I by now use it for all the stuff.

B

Renovate we can do a separate session about it, or maybe uh morrow wants to do a session about it because he uses as well, uh which is super nice. We can even use the gitlab integration of it.

C

I have a sample repositories for this.

B

Nice because it's like the best tool ever it will uh yeah check if there are any updates available for dependencies of your git repository, be it python packages, ansible roles, uh docker images, uh ham, charts, everything.

C

Yeah, I have charts also, for example, when you update yeah.

A

I think I think niklas mentioned that several months ago. I remember that name um and yeah, but let's just it's just just a thought of mine was saying: hey, I probably want to install security updates and other things automatically and.

B

And you can tell renovate to it will always create a merge request for you with the changes and you can even tell it to auto merge it if it's just a minor version or something- and that's extremely helpful. I don't know for my private stuff. I get around 10 match requests each week and I would completely lose the overview of this without renovate, because everything would just be outdated, and so now I just get automated match. Requests to tell me here. There's an update for your parameters.

B

There's an update for the python image. Your ansible image is based on and all this stuff- and I don't know it's. We started to use this in my previous job and we started with two repositories and then we added it just to all of them and you can host it yourself or you can also use the hosted version if you have public repositories on github or github um yeah, it's very nice yeah.

A

I think I think it's a great great addition that you can self-hosted, because I've seen board being abused for crypto miners who generate merge, requests to pull requests and then inject something into your ci cd repository to mine, some bitcoins, or something like that and if yeah typically you you want to auto merge dependencies. But if it's hosted by yourself and you secured it by yourself- it's, hopefully it works better than uh anything which comes from the public to your repository.

C

Yeah, you can also do like not like standard dependencies. If you have like really crucial dependencies that are not in a standard way from form from the public, you can also write regular expressions to update them, or something like that, so you don't need to stick to to the convenience of the outside world, so we need also to use this on some.

B

Can show an example of this in my own repository, for example, here in renovate the configuration I can tell to it to yeah just take this file for in some ansible role, where our the default variables are in there and then match some substring of it, and then look in this github repository for newer releases and which is nice because now I don't have to check for k3s updates.

B

Then, if we look into this file.

B

D

You didn't merge the update, yet I see.

B

Yeah, that is correct uh a little bit overwhelmed with the new job and yeah. That's uh where it matches and it yeah it's just nice, because I just created get a merch request. We can show this example here as well. um No there's no voice! Oh well! Maybe I already merged it.

A

I think we can do a separate session around this.

B

Yeah next week next week,.

A

Next week we have upstream so like digging a little bit into monitoring and observability, and then we can see whether we can. We want to like stay stay on the loop and continue with prometheus and other things, or go into the direction of manner or managing dependencies and updates and other things. But.

B

A

Be linking your blog post in our blog post.

B

Yeah, it's a super cool tool. All right. Let's go back to our pipeline. We have or it failed again that's that.

B

D

Weird it, I also also it failed with every resource if you check the errors, this is the like. If you start at line 43, you can see that the error for the firewall is on line 65., so it may be some problem with the token.

B

Yeah it's interesting because yesterday it worked, I just checked in the headsnack console, so the token has read and write access.

A

What what would be the debug steps.

B

Right now I check the variable in a different window to not have it in the stream.

A

D

You do it manually.

A

With with the docker image uh from the gitlab branch history,.

B

Yeah yeah, I think the next step would be to do it manually just on the command line with the token to just run, terraform locally.

B

Let me check once.

A

Or is there a different way to communicate with the head snack cloud just draw curl or whatever, towards the service api.

B

A

I think the issue.

B

I think the issue is that it's a protected variable and the main branch is not protected, and yesterday, when I tested it, I explicitly protected my debugging branch, but that's super interesting because apparently the plan worked without talking to the api, then.

D

Yeah that that confirms what nikola said earlier, that it doesn't talk to the api, so the plan just takes the as far as I can uh see it right now. The plan takes the state you have which works because and takes the provider and then generates the resources and compares.

A

Them to the to the state and.

D

Not to the actual state in via the api.

B

That's why it's able to you know, that's why it's able to generate a diff, because it knows the state.

C

Yeah, it would fail if we had one of the resources already implemented, because then terraform would do a refresh on the api because he has something in the local state. It needs to call the api of agedlord to see the difference.

D

All right so now we're in that situation, where it doesn't work, because we don't have anything in the state. Yet.

C

Yeah, it's it's not! Okay! It's like it's like an extra step. I would say it's like an optimization so, but therefore, why should it call when nothing is in the state? So then you know don't need to different. If you have a big terraform state with 1000 resources or more, I think it makes sense um when you don't have anything on the api side, not not to do all the api calls.

D

Yeah that makes.

B

Sense so now I added the main branch here to be protected and then it should be should have access to this variable all right. Let's, let's try it one more time.

A

Learning by doing.

D

I'd be lying. If I said I I would. I do stuff differently on my job, so.

D

I guess it's 50, depending on what you do. You start like with each topic. You start with like 10 percent knowledge and 90 is learning. By doing and the more you do something, the more you know and the less you try.

A

It's I think it's sometimes I'm thinking to complicate it, and this is like me and ansible. There is nothing. There is no ansible for dummies for michael, um so I you need to find a use case and you need to try it out um and when you fail 10 times you fail another 10 times and then it works.

D

I have that really interesting uh topic of how do I get state information from terraform into ansible right now and I haven't found a good solution for it yet so that might be something where I'm thinking too complicated.

C

D

C

Ways to do that at least.

D

Yeah, I I thought one thing I did was I already did was put everything I need in ansible in as added as outputs to terraform and then call terraform from ansible register. The result which in ansible then gives you the outputs of terraform in a variable. But it's pretty complicated because, like I need some obscure sub property of something it.

C

D

Be nothing else.

C

Stuff you want to provision, does it have an api? Yes, probably, then I would um ask with ansible the api to get the information about that instead.

A

C

Because it's a separation of chronzone.

D

Yeah, that's what I'm actually currently doing. Oh no, let's check.

C

We get a new error.

B

It's a bunch of errors. I have never seen before interesting.

D

Wait for actions server is locked. um Did you possibly create some server with that name or is there a server existing with that name.

A

Or are you logged into that server? Why is this h? Maybe no, I.

D

I I I think, I'm.

C

Thinking more about yeah.

D

That that protection, that max is doing, but you should reload the page. It tells you to reload.

B

Yeah, which is just because it really is available, so there's no change.

D

ah Okay, yeah, okay,.

B

It's like a software new software version, not a new content. Okay,.

D

Yeah, I've never seen that before either so.

C

I think that's the cxt1, because it's not green and it stopped working.

D

Yeah, let's probably check the error again.

B

D

B

Powered off what the.

D

Okay max you broke it.

B

D

The other one is.

D

It's the ci pipeline. This is the pipeline broker. Yeah.

B

D

All right, yeah, we could check the error message again because.

D

Yeah, it's the agent action 168, okay! Well, that doesn't tell us much. Does it.

B

D

B

Yeah, I would agree based on the output here.

B

Yeah, I don't know we can. We could delete it all again and just run the pipeline again. We could manually start this server because I don't think it will boot by itself now.

C

We delete all everything and start again because we are doing immutable stuff not like trying to pitch anything here.

B

Yeah, I really I will not switch the window right now, but I run terraform destroy er on my local shadow, all right, yeah.

C

Do it because terraform is immutable, so it's like more when you try to patch terraform, it gets more pain in vs or, if you're, trying to walk around on that it's like, then, you need to import resources or doing other curious stuff. It's like.

D

The alternative would have been to manually delete the two vms, unlike the two servers in the hetzner cloud front, end and then remove them from the state. But, as you said, we're not in in a situation where we have any state that we need to keep so just throw it away created again.

A

Yeah- and I think it's also hard to you- apply something manually, you fix it in production like a hot fix, and then you forget about persisting it, and then you don't test it again and the next time you throw away everything. It doesn't work again because you missed doing something.

D

I I recently did something like that and I failed spectacularly what I did was notice that my k3s cluster server as we can see here, the service has problems creating my server was a bit too slow because I put a lot of load in it and then I was like yeah. I just quickly I'll just quickly scale it up and then change the configuration in terraform afterwards, and you can imagine what happened. I scaled it up in the headset front end and a week later I ran.

D

I ran a provider update um of the hack, h cloud terraform provider, which then in turn runs my gitlab ci pipeline to apply terraform, which then downgraded my server, which in turn crashed the server because it didn't have enough resources.

D

So if you have some some config management or terraform or whatever change it there and apply it again,.

A

D

A

Even if, even if you go into a git repository um or you manage something in your git repository, some configs and you change it in production and you forget to like update the git repository um yeah and then you deploy it again and then it's boom.

A

Marks you wanted to say something.

B

Yeah, I'm not sure, what's going on with the api today, I I ran terraform destroy and it told me it deleted everything, but the server the agent were gone immediately. The server was stuck in deletion or something I don't know even though terraform said it's removed.

B

I now manually deleted it here again and then it was gone, don't know, but now the pipeline is running again. We will see what fails now.

A

Some maintenance coming up uh on april 12 or something like that. Maybe that's.

A

Related or we are just hitting a time window, which is not good.

A

I provisioned or I clicked a server three hours ago, and it was up in one minute or less so I think waiting for the dns update took me longer than provisioning. The server.

C

I feel that the pre warm machine sets the clothes really fast so like if you're, using ubuntu or something like else. If you try to provision a machine with your own image, it takes it quite of longer time.

A

Yeah, to be honest, I'm quite lazy because at least my newest services are all in containers, and I only use the whole system for like installing docker and running docker.

A

um I've seen that the default ubuntu image from hats now also disables the the message of the day, news which forms home to ubuntu, and I normally disable that and I was searching the vm. It was like. Oh it's disabled already, so one one step less to provision.

B

Yes, it's the same error again.

C

um Could you downgrade the provider.

B

Yeah we can try that.

A

If I call correctly there's a cli command called hcloud, which you can use to to manually test the api right.

D

Yes, yes, it's written, it's a go tool written around echo library.

B

But either way it's it's super weird behavior, because it's just as a we created three servers and two are now running and one is in the shut off state.

D

Did you try to turn it out.

B

I can try. I click now.

C

Well, then, you need to import the resource on telephone yeah.

B

Yeah, I I I think I would delete it anyway again and try to create it again with terraform, but what the hell?

B

um I guess, if I open the console, you won't see it because I think it opens in a new window, yeah it's almost a new window, but now okay, the server, is running. I get a login prompt, okay, interesting.

B

uh Yeah, I don't know, I can't delete it manually again.

C

It would also no tate doesn't have much.

D

B

B

We can try to run it again, but I I don't have a good feeling about this.

C

Probably what we can do is to fix it as a workaround. So when we see it works now, then it could be a transcurrency problem right now for the time, um because I see this also for the first time. What we can do, then, is to reduce the concurrency level of terraform api calls.

C

So when terraform by default therefore make 10 api chords per second and you can reduce it to do it in a sequence way so doing parallelism to one so it means every user's created afterwards.

B

Yeah, let's try. This would be not for an imperial. uh I'm still surprised, though, because if the, if terraform would be failing and then the server would just like be slower to come up and then running, I would say: okay, something weird going on on the terraform site, but that it's ends up not running on. The hudson side is very surprising to me.

B

So terraform concurrency or something do you have the option.

C

From the dash parallelism equals one, I can write it. Okay, yeah thanks, but I don't know how you to add this: to the uh gitlab player from rupa.

B

I think you can just uh pass any options of terraform.

C

I you don't need to use the tfaa, it's my shorthand automatic apply so.

C

B

Well, this does not look good okay, so I switch windows again over to visual code.

B

B

Not sure if this is the right window, but no, I think it isn't.

B

This looks better all.

D

B

Now you should see our visual code again, let's go back to our galaxy iona and then in our terraform apply.

B

We add parallelism only use one dash right, yeah.

B

B

And then we go back to our.

B

B

All right, so how is it that's still? I don't know it displays as initializing. Still, let's delete it.

C

Probably um it should be a problem on the data center, but not where we are scheduled with our vm is currently having some problems. We can check had some status probably also.

A

I've been reading the gitlab terror from rapper script and it passes the the variables you are adding to it. Just one-on-one on terraform.

B

Okay, there's nothing on the status page.

B

So the server is deleted. Let's try the pipeline again.

B

And I guess if this doesn't work, we will try again in the.

B

B

So nice, when we started this kubernetes series, I think in january then it was always super dark when we finished and right now, I don't even have to switch on the light because it's still so bright at the window. I like that.

A

Yeah, at least in your area, in my area, it's snowing.

A

B

Had this earlier today,.

D

D

After all that, I know if it's snowing, it gets light. It gets more bright, not darker, so unless the sun has already set where you live, which I don't think.

B

It depends a little bit on the on the sky during the snowfall.

A

Yeah, true yeah, I I don't know, maybe it's it's blue sky uh after the clouds um but yeah the weather changed from a sunny easter weekend to s to rain, then snow and thunder yesterday. I think so it's it's a bit strange, but.

D

A

To to change uh to overcome the breaks between watching a pipeline's run,.

B

I was talking to a co-worker and he lives around your area and he was also like. Is your weather also so crazy? And I was like yeah. I don't know yesterday and today we had snow rain, sun everything and it changes every 10 minutes.

D

It's it's a big, a bit more set here in berlin, but yesterday we had a hail storm for like 10 minutes and then it snowed and then it snowed and then it was raining and suddenly blue sky. Oh, it's snowing again what the it's like classic classic april. First, two weeks of april weather.

B

Okay, I would say we stop here, because this gets annoying, not the talk about the weather, but the failing pipeline, because I right now I have no idea what to test uh but to wait, because I don't think it's something we do wrong because uh yeah, it doesn't look like an error on our side to me.

D

When you change the data center to helsinki, maybe just for test.

B

Oh yeah, we could change the data center. That's true.

D

That's actually a good idea for tests. Yeah.

B

Yeah then we will definitely get another node all right. So, let's.

A

B

Back to visual code.

D

All right, just as a last resort,.

B

Yeah, just let's move to a different country with.

D

All other stuff.

B

Just need some confirmation that I don't I don't know, share my sound panel with you or something um all right. Let's go back to terraform and yeah. Maybe let's run terraform destroy at first.

C

Can you specified in the provider the region so that you don't need to put it on every resource.

B

Right now it's on every resource, so maybe we could set it globally, yeah I'll.

C

Check the logs, if it's possible,.

A

Yeah but maybe it's also related to the provider which we're using because last week or two weeks ago we updated it.

B

Yeah there is even a newer version now I think so we could upgrade again.

D

There's yeah one: can you show us which one.

B

Yeah right now we are using 126..

B

D

Not sure if this one has been used, yeah.

A

This is this: I've learned.

D

I I'm I'm just checking my repository because I updated it like yesterday or something okay.

A

D

B

A

Latest one, okay.

D

A

Because musk updated it last week to 126 and opened the docs and and said use provider. And there is the version.

C

uh Max we could do it in a better way. I think, in a more readable way, we could use a data resource um our data source for location and then using this data source in every resource, because then you need to change it only one time.

B

Okay, that sounds good, but I have no idea what you're talking about okay.

C

I will post the stuff that we need. um I can show you the example typically when you're using other providers, so, for example, aws, azure or other you can directly define the location directly in the provider on hetzner. They have a different approach that you need to specified on the resources.

C

I probably I can also share my screen otherwise, to explain what would currently do. Okay now wait, but give me one sec for that.

B

Yeah, but I can already stop sharing mine right. I don't know it already stopped, didn't it now, it stopped. Zoom was so confusing. I don't know. I need.

C

C

I need to close too many.

A

Windows, maybe we are running into some sort of rate limiting.

C

D

That I deem that very unlikely, because that would be a really harsh rate limiting and I've definitely done things that are way more request. Intense work against the headset cloud, api.

C

Yeah, okay, so what you can do is um ah they have two data sources? That's more interesting. So, um for example, we don't now want now to get all locations or we want to ask a headstand cloud about all available locations and now, instead of creating resources with reform, we can also use terraform to get information.

C

And for that case we have a new type and it's called data and all data sources can be used to query information about the apis.

C

So, for example, now we would use data h, load locations to get all the rotations of all data centers from hetzer, and then we can add this to the resource, because this is locations will give us every all location ids.

C

Then we could easily say: okay, give us the first um element, or currently this example. What it would do is to create a server in every location on hexner, so one in nuremberg, one in in the other german falkenstein and one in herzinki.

C

So, but we could use the h-round locations and then using our own index instead of doing it dynamically yeah. That would be the case you can also, instead of getting all locations. If you were only interested to get the id, you can also use the data source h2o location. Where you can write for shorthand for the location, then you can use this data dot, h, cloud, dot, location, for example, r1, and then you can use this to um users in the resources and that's I think this.

C

No, it's directly formatted, okay, yeah and that's what the way. What I perfectly do mostly so that tear form controls everything so in terms of creating the resource, and I don't like to hard type everything so that we can use this type because therefore, can then easily check if it is addressed correctly most because then you can do type checking on this stuff.

B

Yeah that makes sense.

C

Yeah, that's it.

C

So, let's switch back.

A

B

All right, I already started to add this data. Each cloud location in here and I think locations doesn't make too much sense for us, because we, then we only get a list of all of them, but we don't know which one to pick right.

C

Yeah we can use only one so.

B

And then I think it's her hdl one for hezinki and then down here. We should be able to for the data, hcloud location, location and then dot name.

C

Or it could be also id.

B

Yeah, I I looked into the documentation and it says at least the documentation says: it's a string, it's a location name to create the server in. I would also have kind of expected that it accepts an.

B

B

Okay, so this way it kind of it doesn't make much of a difference if we define it as data source also variable right, but as a data source at least it can verify that the location exists.

C

Yes, if you would not type a wrong location there, it would certainly it doesn't confine the location.

B

So maybe we can run terraform plan locally.

B

That at least seems to work and it's trying to create an housing.

B

C

So what you also can see that not only resources created um but later when you push it, you can push it directly. um Then we can also see a change in our plan.

B

Right, I'm going to switch to browser again.

B

And here we are back, let's run the pipeline again.

C

uh So that's the reason why you specified the workflow rule to web so that we need to create via the web ui and not for every computer, yeah yeah, okay,.

B

Yeah we can change it in any way. We want later.

A

A

I just I just learned another thing: I was installing the hpr cli command just to test it out and it has a sub command which generates bash or as set as h, autocompletion code. That's nice.

D

I I think they they looked at control and were like. Oh that's, nice. We want to have that too, because I think the uh uh keep control is like at least the first one I can remember to have that and it works.

C

Really well, there's a drawing people are starting.

D

To add it sorry necklace.

C

There's the doll library for that that you can automatically generate it from your commands, because these all are go applications and.

D

Yeah, that's really yeah! That explains a lot. Thank you because.

A

Because I tried to do that in c, plus plus- and there is no other generator for that.

C

Can you go up max to the log, so probably we can also see what changed in our plan.

B

Do you want to look at the servers or is there a.

C

Higher converse.

C

Doesn't it show the data stuff, but it should say.

B

You can search for data.

C

B

C

B

C

But how does he knows.

C

Interesting because someone can you can you go up again? Probably I just miss some lines that I want to see. Yeah.

A

Maybe that's limited from the output just like to to put it on std out from from terraform.

B

A

It already knows, um but it cannot print it.

C

Yeah, okay, it doesn't get attached by the wrapper script. It would be reason to have more clear locks.

C

Yeah, but it's fine literally what you would have there would be a resources toward symbol, state refreshing and that's the data resources trying to get the latest data of the location.

C

But also our pipeline is failing.

B

Two servers failed: both agents they're both not coming up, even though it's a different data center. Okay,.

C

Can you check if it's yeah it's a different detail, don't you.

B

Yeah, it's both interesting, you know wow, I don't know what else to say.

A

Destroy everything all servers and then bring them up again, no just kidding.

D

A

A

It's a problem we can't fix now um and probably someone else who's on call is currently trying to fix it.

A

The least we can do is like wrap it up for today or wrap it up for now and um and wait for better times. Yeah.

B

C

Try again another time and next time we implement a multi-cloud strategy.

B

Yeah then this wouldn't happen right yeah everything would be easier.

A

Yeah, that's what you.

C

Can use for laughing.

A

You're laughing and.

D

A

Actually, a good idea.

C

A

C

No not for the most business, I would say.

A

It would be interesting- um and I was I think I was thinking about this a while ago, like to prove provision heads in a cloud when it works and then say: hey I we want to when want to do it with aws. For instance, I think someone responded yeah. Someone responded last week to the twitter thread around devops stack, devops tool stack, I don't. I don't really remember it, which is a a full-blown of a ready-to-use solution for terraform and deployments and multi-cloud and whatever, and I said yeah.

A

We can definitely look into it, but first we need to have it working with that snack cloud, um and I also want to learn it like building the lego. Bricks together is more compelling or more easy for me to learn, rather than reading the documentation of a fully fledged tool where I cannot see um the progress of the learning history and the trial and error and things around this. So I think, like.

C

There's another state that I know at least- and I know that someone use it really. It's called tooth. um It's like the same approach that the guy posted last week, um but you can also run it locally um to test for stuff and it runs multiple on on every cloud, but.

B

C

Uses the same approaches it's from philip strubel.

A

C

I know this was the firm first firmware that I saw that doing this approach.

A

I would again be boring and build it from from scratch just to learn it.

C

Yeah following it's not good to do this yeah.

A

So for like comparing how to probably do it at a later stage, when you know that you know the basics, I think it's great to have them, um but for the initial kickstart and say hey, this is how terraform provisions something- and this is how it works with headset right now. Headset is locked for some reason, so it's using the api and we cannot debug it um as a black box.

A

Maybe it's like how terraform fires, your requests um could also be a thing, but I'm not I'm not totally, not in the mood to use tcp dump now um or just attempt to do that um so yeah, let's, let's maybe um wait for better times, probably in one hour it works again. uh Next.

B

Right so we meet again in an hour. We we do a steady.

D

You just stole my joke max.

B

D

B

I'll call you in an hour.

C

Now we can, we can create a statue of pipeline, so it will do the job for us. So every hour try to apply.

B

It yeah, but then we do the destroy each time. So gitlab gets a invoice for servers created each hour until it.

D

B

We have to make sure to delete it every time afterwards, like just so, we create new ones every time. No, that's.

A

B

Now that's very important to raise the invoice.

C

Okay, to raise the invoice you want to raise.

B

Yeah, exactly okay.

C

A

Okay, so creating a server costs you some sense right.

D

uh I don't know yeah.

A

D

You always get one first hour on creation so that you don't just spawn servers like once you once you create one. You get built at least one hour so that you don't just spawn them and use them for 30 minutes and then turn them down. I mean you could make a business out of that and that's why it doesn't work.

A

Preventing crypto miners and other abusive things.

C

It's mostly it's it's relying on the business model, because um I know that you're, an aws you need only pay for the first 10 minutes is the fastest stuff that you did so you have 10 minutes. If you have workload that fits into under 10 minutes, you can spin up your whole machine for that and then use it in pain only for 10 minutes in a fun whole hour.

C

But it's mostly like on the business calculation, how you do the server calculations and so on. What's in the back, it's relying on the business model I'll see.

A

Here, for our case, I would say, probably don't over engineer it with scheduled jobs. um If, if you have the mood next week or we have the mood next week to say, hey, we want to verify it. um Maybe after the the ops trace session, um we can just click on it and try it again or who, whoever is in the mood um yeah. That being said um next week, is the 14th of april and sebastian will join us from ops trace.

A

um As far as we have discussed it, it will be, like um short, showcase. What is upstream, what are the plans, roadmaps and other things, and then we do like a hands-on session, and I also told him that we probably will be having some crazy ideas, um and maybe we find we find certain ones and then do it live, maybe just scrolling in monitoring the kubernetes cluster, which has built with ops trace.

A

I don't know if this works um but yeah, let's see about it, so I'm excited for next week other than that, um thanks max for preparing today and for debugging with us today. I really appreciate it and with that I would say, bye, bye on youtube, see.