GitLab #EveryoneCanContribute cafe, 20 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 13. #everyonecancontribute cafe: Auto-scaling GitLab runners in Hetzner Cloud

Description

- Documentation: https://docs.gitlab.com/runner/configuration/autoscale.html
- Blog post: https://fotoallerlei.com/blog/post/2020/autoscaling-gitlab-runners-on-hetzner-cloud/post
- CI autoscaling research: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27061
- docker-machine direction in GitLab: https://gitlab.com/groups/gitlab-org/-/epics/2502
- Hetzner Cloud curated lists: https://github.com/hetznercloud/awesome-hcloud

Workaround for Docker problem. runners.machine - MachineOptions

"engine-install-url=https://releases.rancher.com/install-docker/19.03.9.sh"

A

It's almost already uh my legacy monitor.

B

Let's stream, well that I have.

C

And it's really great because you're live on youtube now.

A

C

No, I just started the stream.

C

um As as a new adventure um and well, I don't know how to start it, um but anyhow, it's really great to finally meet gary who's, been writing quite a few tanuki to use their blog post this year already um bot, gitlab and cicd.

C

So you probably want to start with again a short introduction around and then we might just figure out what to do um or what to talk about, and I'm asking how is the weather in lower austria.

D

Snowy, so my name is michael wagner: I'm a product manager and a software engineer at a company called zkw, so we build headlamps for the automotive industry and I specifically built high performance computing apps we see plus plus so cfd engines, raytracer and so on and yeah. So I would hand it over to niklas.

D

B

Yeah, my name is citrus meats, I'm a detect hero. Currently, I'm working as a senior devops engineer for company called um swelty we're doing a lot of interesting stuff, so we're doing a lot of stuff with compliance mostly, and I doing only the hot topic, so I do blockchain and also on trivenitas. So I mostly um like to do the fun new stuff um yeah and I really like to contribute every week here.

B

So I will hand over to david, probably.

A

uh I'm david, I I'm from puppet uh currently with all the branding that I can cram into a video background. um I I'm a senior engineer on the content team responsible for all the puppet content, all the supported puppet content on the forge and as a tech lead just being the uh uh responsible for everything technical. I guess you could say um from yeah.

C

Okay, um I'm the crazy one who brought everyone together here. um It's been 10, 10.5, 10 and a half months now with gitlab yeah and I'm kind of trying to figure out what we can talk about. What we can learn from us and I have to be honest. I have no idea how we bumped into each other gary, um but maybe you wanna wanna, kick off and introduce yourself.

E

Okay yeah, so um I'm gary bell, I'm senior developer and tech lead at a company called e-cigarette direct. We sell their products online in the uk, um slightly controversial industry for some, but there we are. um I've got um the php background and that's what we build our platform on um and you know, use git lab for uh all of our pipeline stuff, so unit testing uh we're gonna, integrate the deployments for the new platform uh in the next couple of months um and yeah.

E

I I love gitlab as a product, so I blog quite a lot about it uh and hopefully can guide people into using it and moving away from other solutions just because I think it's the right one for for me and probably a lot more and we'll have bumped into each other michael on twitter, um either something I I shared or some comment I made um you know once I got up and running with gitpod.

C

Yes, I think you, you have been one of the first like contributors to trying out the gitlab gdk the development kit with github. I think.

E

Yeah, I think um I was given it uh given access to it about a week before it went public, um so that was pretty interesting. So I was very keen to get up and running with that and had a blog post ready to go and then giftpod got in touch and said we haven't done the press, release hover saws for this. Can you hold off a couple of weeks? It's like. Oh, I've got a complete exclusive and I'm not allowed to use it.

E

So no, it was. uh It was great to to sort of get introduced to those guys as well, um and I've got uh another git pod post coming out uh beginning of february, um I started you know, went through the process of setting up a completely greenfield project with um just to sort of see what it's like for somebody just starting out on projects. Is it a viable option for people that way um and had some good interactions with christian that get pod with that.

C

That's it's great. I think there is a docker rosenheim meetup which, which also covers gitport and rootless and and other things coming up also in february. um I just don't have to link handy now and I've seen that max joined us yeah.

F

Sorry for being this late.

C

No worries um we just did a short introduction round, which means say, hi and who you are.

F

All right, yeah.

C

F

Max uh yeah, I'm a systems engineer from germany. I recently moved to berlitz um yeah. I work at sys11. It's a company, doing managed hosting and managed kubernetes, so my day is usually spent working on kubernetes and my afternoons, my evenings, often as well, but uh more private, stuff and private projects. I hack on yeah yeah. That's what I do.

C

Okay um nicholas: do you want to try the hetzner provisioning.

B

Yeah we can do so well then I need to prepare some stuff at least.

C

Yeah, probably because you you've said you can sponsor one, how many calls 128.

B

Yeah I was but it wouldn't, it won't be only it's not really so because it can go to package. But when you want to go on hetzner, I can also set up a project, in other words,.

C

We could also try out packets or, like equinix,.

B

uh It connects around yeah. Let me check I'm not so prepared right.

C

Now give me one minute so sorry for like making this an adventure- maybe uh maybe the others have a suggestion to to try. So one of the ideas was to like install gitlab and autoscaling runners and try how things work or not work, but we could also dive into yeah anything ci, cd or best practices or ops, trace.

A

Can you can you? Can you give me a quick overview of how the auto scaling runners would work? uh I I don't necessarily need to see them in action, but just like uh how do I spend my money using those um I we're we're actually running into the problem where on our uh github installation or on our uh github pro account from puppet. We are running into the limits of their runners and I I just want to know how how we can do that better.

C

To be honest, I don't know how to do it with github and the actions runner.

A

um Yeah, I don't care like if I want to make it better with github. I know who I need to talk to internally to spend more money, but I would like to know how how the git lab action, uh the git lab, auto scaling runners work, um because that that sounds a lot more like what I would want.

B

C

Yeah there are so one one thing you can do is, for example, you use the docker machine executor, which which, for example, spins up additional virtual machines with the runner pre-installed in aws um or, for example, an openstack, and there is also- and there is a blog post from max actually which I'm currently um trying to find.

F

Yeah, I already posted it in the chat.

C

Perfect and as a shameless plug yeah, that's nice. um I I've seen you've uh reposted it to to dev.t.o as well so yeah. um Let me let me share my screen. We have this picture no, this window so.

C

Do you want to walk us through the blog post max, maybe.

F

uh Okay, yeah, I can try, it has been a while it worked so good. I don't need to spend any time on debugging. It actually um give me a second.

C

You can also share your screen if it's easier for you.

F

uh No, I think it's it's fine like this, um so yeah. Basically what I used there is a fork of the docker machine runner, because the gitlab runner itself does not support hetzner and someone in the open source community wrote a driver for now for it and.

B

Mostly to the architecture, it could explain a little bit to the ice station why it's a different runner, not the typical runner.

B

um It jumps from the topic that they're using for auto scaling the docker machine and doctor machine has a plugin mechanism, so you can drop in your own drivers and by default the detail burner doesn't provide free hits now driver. So this is the reason why you need to get in different images of official declared runner image.

C

ah So there is like a tab or which you need to download extract and then run it, and this is basically added into this uh headset gitlab runner image.

B

C

A

And basically, uh just to that that we were talking about right, I think basically, when gitlab triggers aci run, uh it would go to that docker machine thing and say hey. I need uh some compute to run this, uh this uh ci run uh and, and that will then use that driver to get a machine from hetzner and uh and execute that ci right.

B

No because vegeta runner aren't as deep doesn't push the information to the run of runner's pulling every time. So this is the reason why you set up in the first time the controller limit. Certainly it will ask if it needs to spin up a new job.

B

So dab is the control plane for spinning up runners, mostly.

C

The picture now.

F

C

Okay, there are different different possibilities, but yeah.

C

The the general idea is that you can like, when there's lots of traffic and lots of chops in in the queue that you spin up, additional virtual machines, um and you also shut them down afterwards, and one thing we did in the past with openstack was also to define offline times. So, basically, um the idle time of, like you, have only one virtual machine over the weekends, because nobody does anything over the weekend, which also can help reduce the cost.

F

um There's an example in my conflict that I keep the runners longer on the weekend at the bottom.

F

It's a little bit of the opposite of how you would do it for work. Probably there. You would keep them longer and more during the week and none at the weekend.

F

But if I use them then I usually do it on the weekend. So I don't want to delete them after one hour.

C

And that's true.

B

Aren't you doing for one more pattern.

C

Okay, what else do we need from the configuration machine options which are passed their passed as environment variables? I think somehow, our key value pairs.

C

um And the costs.

F

Yeah, the costs are ridiculous, low.

C

I I think I've seen a screenshot of yours, where you have like 10 pages of pdf for like starting.

F

C

Stopping something, and so on.

D

For 16 gpus nice.

F

Yes, the headset prices are very, very nice.

D

Never saw a such low price tag.

B

Of course, it would be truer when they do a calculation in 10 minutes charts instead of one hour. So then you could also save more money.

F

Yeah but you yeah with headset, you always pay one hour so.

B

F

Okay, you can't save money by shutting it down after 10 minutes.

C

Yeah, but still I I think, I'm using this or this somehow for my personal blog um and also for for a different hosting thing, and for me it was like. Oh I'm clicking there and something spins up, um so I was not auto provisioning it um and the nice.

C

The really nice thing was that it has built-in monitoring or like simple graphs, to see uh cpu and io and and things like, because I was migrating, a blog and actually a forum, and I was was not sure how how it would perform- and I just picked and said: okay, two cpus and and some ram would be totally fine um and it doesn't cost that much so it's really affordable in comparison to some years ago, where it was like dedicated host or something else. um Oh, I it's 20 euros a month.

C

That's too much for me, maybe 50. um so yeah! It's it's really nicely nicely built, and there is also- and there is a compiled tool list somewhere for the headset cloud, and I think prometheus also has auto discovery for the services api.

C

um And, and even more things um did I buy enough time for for nicholas, oh.

D

B

Yeah, probably you can do it, give me 10 seconds or something like that, and we do it live literally so no voice, um but I need to. I didn't thought that I need to do it but no problem. I.

C

Can I can do more marketing for headstone there's no problem.

B

I use it my arm, but I'm currently thinking how we can do it in.

C

B

C

Awesome hdr or some h cloud.

C

This is a a list of tools and then things where you can interact with the head snack cloud api.

C

And some abstractions and the like.

C

Just share that over here.

C

Terraform, to be honest, I haven't tried them um max. Did do. Do you use their use, some of them.

F

Yeah, I use the terraform provider from hetzner, so developed upstream by hezna itself, and I use the ansible module as well. It's also developed by hezna and their csi driver and the controller manager, so the the two links at the top top of integrations as well. So you need to use them if you want some more native kubernetes support on it.

C

Oh cool, interesting.

F

It's uh for a week few weeks now I run a k3s cluster on hetzner and it's provisioned by terraform and ansible. Oh.

B

I know a guy, he was a really true rapper about terraform and um juventus also, um and it was yeah I'm I probably am ready. So we do everything live um and we'll see how it goes.

C

That's the spirit. Okay, um then I just I stopped sharing.

B

um And I will start to share um so.

C

If you leak anything, um I will tell you.

B

No, no I'm! I think I will be no better prepared, because I do it on a different monitor. So this time it shouldn't be happening. I prepared for most important stuff um yeah literally um because we were talking a little bit about the architecture um and we could probably follow uh much slightly. um I want to do this.

B

um Let me check something: okay, um so because you um asking about the architecture um lately, what what can be done is I use this repository right now on github from appdarwin, because he has a simple um diagram right now and it's really the same on every other cloud. So what we will do now is probably we will use the default cloud to have our repository and connect our runner there and literally what we now do is to create um a simple instance that can spin up these vms.

B

Certainly so the detailed runner manager or vegetable runner by itself will spin up then virtual machines against the hetzner api, and for that case we need, of course, the hexner project. So um I have already prepared a project right now.

B

You can see there are no servers in here and I will click now through the ui literally um and do it so what's really cool, we could also do it in terreform or something like else, but first of all we do it like live um yeah probably can choose nova locations, so we will probably go into a german-based rotation to know back to to michael, so um that we are really near to him and having a low latency.

B

um What is really cool um in the meantime, hetzner has um pre-warmed uh vms. So if you're using the standard, vms so ubuntu or something like all these with thunder sign literally, um you will get them really fast because they will be deployed in under 10 seconds. So it's really cool for doing development um on on different machines. Yeah so, and we will start with a small machine because we don't need a so big machine for that, um because this machine needs only be intended to spin up all the firms.

B

So we go in with the smallest as possible um and then we need to um create our ssh t literally. So I have my hd there and then I create the machine literally and now we would sam sequence and the machine is available. So I can use now using this ip address literally and drawing back into my terminal.

C

Can you increase it a little.

B

Of course, no problem.

C

B

um Is it big enough yeah yeah.

C

B

um So then, I need to get access to the machine slowly. This is probably I hope this is the fifth standard user there yeah and now I'm in my machine. So this is everything I need to do um so, there's nothing on it. So an autocrat is there probably uh so the next step would be to install docker uh a better way of that is, instead of using it right now. In this way, let me check how to minimize it again and you can also provide on the boots.

B

So when you're creating a new machine open it um creating a new machine, you can also provide a cloud in it configurations that will do all the steps that you need to do, for example like automatically creating the users installing docker out of box, so that you have an immutable instance, and you can work right out of the way on that. But uh now we need to install docker.

B

Let me check how we install it ubuntu.

B

I didn't do this, okay, so right out of works, um we will now install and we don't need to update old versions. So we'll get him back.

B

Let me know where my terminal yeah, I would also don't need sudo right now, because we are um inserting the certificates.

B

B

Probably in the next cafe also, we could have a five minute um declare auto station runner all the.

E

B

We are currently doing um so afterwards. We install all these patches that are required, so these are packages mostly for getting the right software checking the software and so on. We can now um executing um the script to get the gpg key so that we can verify that the packages are now reliable, yeah. We don't, we won't check it literally now we add the um repository of docker to that and.

B

Okay, now we should be able to install docker, so um we don't need to set it.

B

B

So the promise that I can't see the chat anymore when I'm sharing right now, that's nice.

B

Cross I'm looking for the link from max okay. I got it nice, so everything is in place. um We have docker now and we can now connect it literally um so hopefully um are doing. Is that her compose no package these days?

B

C

No on on linux, you need to like and download it.

B

C

On on mac os, it's part of the docker for mac installation.

A

That command right there claims that you can just add, install docker.

B

Yeah, but mostly I don't like to use for ph managers because they're using an outdated version.

C

Yeah, but can you try that um would be interesting if it works.

B

Let let's check it probably ubuntu is so fine. They will provide us.

C

B

C

D

B

Upstream is available 1.828. We are now on 1.25.

C

And I think upstream we rolled it into golang, because previously it was python.

B

No they're bumping also some people, but okay. We will use this to be faster, doing the boring way.

C

Boring solutions.

B

Yeah so now we will create the um runner.

F

B

um Mostly, what is here so we will use the volume, so it will create in folder destroyed headset config that we need to handle. The okay now worries.

B

So I hopefully it will download all this stuff.

B

And I will explain in the meantime mostly what is interesting about all the runners right now they are using all docker machine, but docker machine is mostly not maintained anymore. It will be maintained from detailed right on now to use this federal machine binary, but doctor machine was intended or invented by docker, but it dodger itself won't maintain the structure machine mechanism anymore. So now, when we are jumping into the container, hopefully is this something yeah?

B

uh We can jump into that. I have a bottle of that um and trying to start with the process. Okay, um probably we can now see here's now of inner process and we can see that it runs as the user did lab.

B

So don't use a privileged um user for that, but all this stuff is run as privileged interesting um yeah and literally what I want to show you when you, when you type machine, I think yeah they're, using these commands literally to control all the stuff um they wrapped this into the clap, stuff and howard, can see the driver that are currently active.

B

I think it was also possible.

B

uh I know: okay, the chrome machine.

B

B

Help because you can see now here a lot of a lot of different options.

B

um And by default the virtualbox driver will be pinned up, but when you're now asking for a specific driver, for example for the um for the headset driver, it will show us the sub commands that we actually use for the headset stuff. So but I don't know how the driver is called. It could be something like this that all.

D

The virtual that all the virtual machines are spawn on the same machine or how is it distributed.

B

So the doctor machine driver that your machine will ask in the headset cloud api ah we'll create a vm will put on all the steps that we are doing now by hand. Will docker machine do automatically for us? Okay and vegeta lebron instructs the crew machine to use it. So this is the concept behind the auto scaling.

B

Mostly um probably it should be now that the user- it's not.

C

It's called hatsune, not hclod,.

B

Because now you can see also these um message: hey, it doesn't in the default path and all the drivers are prefixed with docker machine dash driver and just for binary, it's really the same.

B

They don't have special any outputs. I here you can see it so now you can see that all the commands that we are currently use. These are the special commands that can be used by the um little browner to instruct also hexner, to do special things and they will be reflected also into the um into the conflict. Comma. Mostly, when you are looking into the part, um we will copy this from the blog post. So I need to know where, to put it, I'm in the container right so.

B

Let's see there.

B

Yeah, okay and then we create the config tommer.

B

No, I will go. I will do the same stuff on the host. It would be the same.

F

B

And right now you can also see here the the command for machine options, and this is the same options that we can use from the help that we saw before so. For example, here's now the um api token that we put in which type and so on.

B

um Probably we don't need this.

F

Yeah, you can delete the whole cache section.

B

So then we don't use the distributed cache when we delete this. I hope so. This works out of the box.

C

It should, if we, if we would add some sort of s3, compatible bucket thing, we could edit again.

B

Okay, now we need two things. We need the api token, so we can fetch this api token from the um headset side. So when we are going into security api tokens, we will generate an api. I think we need read and write access because yeah we need to create resources, so we need also to have right access and gitlab.

B

Autoscaler and when we have this token, where we put it.

B

In and now the last missing part is we need to get the access token, and um probably we need also to update the url.

B

um We will do it right now with um declared from chrome, because it's easier in terms of I don't want to set up right now. I do have also, but we can use it afterwards and reduce that, because I need dns and more complex, so we will get the club.com and probably um on github.com. We don't have the options to use sharedrunner, so we design for all our environments for all the instances that the users are currently used, because the clip is currently providing us those.

B

But what we can do is.

B

um We will create a new project under.

C

You can totally go there. um You have permissions.

B

Yeah mostly so, then we can see it, and I hope I don't export anything yeah, it's okay. um So when we clicking uh when we are clicking on settings the icd, um we have now the option to install group runners. So this is the stuff where that we need to put into our little runner config and mostly what will happen afterwards. Every project and everyone can contribute, can use these runners. We have an also different option would be to do this on a project level.

B

So when we are clicking, for example, on our general project and there we want to run some ci drops, um we could go also to settings. Nci cd then also run us, and you see here is a different token, because it's a specific runner only for this project and um yeah, but we will use it from the group. So we go back use this from the group.

B

B

Michael made no reminder that we will need to reset the token or we get people at us, free runners for everyone.

C

No worries I've. I've noted two resets already, so everything will be cycled afterwards.

B

Okay, um yeah, probably now we can check the docker container that probably don't get the changes directly.

B

C

Get example.com.

B

Does it work automatically.

B

I think I need to restart it on it.

E

A

E

A

Still using gold placeholder as as access key so.

B

Yeah, so to be ensure that everything is loaded correctly, we will remove everything away, so we do a decomposed on this v and spin up the container again.

B

Mostly the dash v, what it will do is, it will removes, also managed volumes from docker. So if you want to clean up your docker compose file automatically with all the volumes attached to that, you can use fedex v, but it doesn't work for bind mount. So our head start. Config is still there, because it's not a managed volume from butter, and now we should probably see him a lot. It works.

E

Really my thing, but after the gitlab.com in the config does it need the trailing slash. I'm concerned get that dot-com z-r-u-I.

B

B

Mean here right.

C

B

Yeah, this is new to.

B

B

So we're doing the fast way so we're asking for.

D

ah I get it now.

B

It doesn't work, it can't.

D

Get the two together the key with the with the url.

B

Yeah, the promise is a different one. I think it's I forget this. um We have. uh We need to do an important step lately.

B

um This token that we are currently using is not viton um that will need for x's. So in this space for totem um in line 10, the number um is the x's token, but the token that we are currently receiving is the registration token. So what we need to do, we need to redisturb the runner first. It's also mentioned in the blog post. I overskipped this so to do that um we need to run the register mode.

B

And now yeah we can use the default. Doesn't it work? I have to type it out.

B

Of course, I don't have it in my space.

F

It's possible that it will completely overwrite the contract now, but we have.

B

Now it will uh append it yeah: okay, okay, that's nice uh autoscaler, so we will provide some state texts. Mostly. I really like to do this. I try to crosstalk to see that better in the ui um and then we say doctor machine.

B

And we're using a default image like I'm, really a fan of debian, so let's just dive in and now the registration is finished. um So first I check up in video. I have ever seen words and hopefully no.

D

Do you have to start the the runner as.

B

Well, now we point really not that we need to restart a runner.

B

You can see here's our runner now and you see the token is a little bit different.

B

um Everything is words is accepted.

B

I will copy this part.

B

Honestly, we did.

D

B

ah Yeah, you see, our runners also reduce that, um so it doesn't need to restart, and now we need pro it to run a job on that right, yeah, that's everything that we need to do so we will create our new, auto stator project.

B

Or should we use someone else's address template that we can use, or is it.

C

Project template.

D

No, that go micro.

B

Okay, you said it when it doesn't work. It's your fault.

C

Doesn't matter we can delete it.

B

B

B

B

Now we need also to adjust the ci pipeline because we want to ensure that it runs on our state runner slowly.

B

And I need to add this take so it will use to select the runner, because when we don't do this, it would use um the jobs from from it would use for shared runners from gitlab, and we want to use our own literally.

B

So I hope so everything is fine.

B

And now, let's see.

B

You can see now the little brother is connecting.

B

And probably, let's check heads now ah and there you see now for each job literally and all runner will be spin up.

B

And now it depends on the configuration you see. It takes some time to um get all the stuff done. So currently we don't have any locks on that to see what's happening during the boot phase. Or can we see also.

B

Locked think some res, but not the buddha blot, so you would see. ah I can't show you in the meantime. I can show you something before I forgot it, because I um talked about the topic that it uses: structural.

B

B

um And now, when we're attacking doctor machine press, you can see now the machines are also there and for easy jumping. You can now do docker machine ssh and you are now on the runner machine and you can see probably you know all the jobs.

B

What do I have watch cool and probably we are. We were a little bit too slow so that our job is finished.

B

Now, okay, do is currently do some preparations in the background because the job doesn't start it. So let's wait and the ci when our first um container will spin up, but this is the main idea behind auto stealing, so it uses um literally docker machine to execute auto scaling.

B

You can run it on every machine that supports docker machines, so it can be open state, aws gcp. um I did. I did this also for some customers on bare metal. So it's also possible to do it with kvm, but you need to do a lot of work more there, um but it works um pre.

B

It takes some time.

D

So in ensure they can do anything which docker machine can do, which is built in or.

B

Yeah, mostly when you have you need to put the binary into the path, so they found the driver so that you can use every driver. This main reason, probably we could try to contribute the headset cloud provider into the upstream, so it will be also there.

E

B

Default that you don't need to use a different image.

B

But in the end it should not work.

B

Or did I miss something.

B

Mostly, probably also no need to cheat where my sliders.

F

Why we wait for the runner to come up? There has been a question in the live stream on youtube.

F

Sebastian asks if gitlab will maintain the docker machine.

F

Runner for the future, so he's like want to know if it's a future-proof solution.

B

To use it um there's an interesting topic about that that I would want not to mention really, um I think, there's an issue about that. um How the future of vegeta lebron deployments will probably change. um I need to check with issues one minute.

B

More so because um let me check now where's my yeah, um so here's visually.

E

B

Also open it, which would be true um so there's an autumn, because this auto stating worked for most of the customers, um but it doesn't have, for example, it doesn't have ha out of a box. It could be not cross effective in terms of okay. You need to run it at least one hour on headstart cloud, but probably my jobs take only 10 minutes or something like how can we do a better effectiveness of all the jobs?

B

um All the stuff is certainly right now mentioned, but current the current state is um that detail uses the shared runner for itself. So all the shared runners that you are using are also using the docker machine, auto driver. There are also some window runners out there and um mostly it sees it will be maintained.

B

um You can also do I read a lot of stuff. You can probably also do it with um having senior runners on auto scaling groups on each of these club providers and spin it up, but on your own, but these outer stadium alternate faded up is really easy to use mostly, but it seems that it will mostly continued um currently an other alternative.

B

If you want to scale your runners is that you use two anita's for it on some parts so for, if you have simple, builds like web applications building or something like that, kubernetes is also fun um yeah, but there's also um you can check it also because I think invitative runner repository this information, which.

C

Is an epic for that? Let me link that.

B

The chrome machine they are using, but there are some other problems in the community because I think mini cube uses also the chrome machine for that, and they have also the same problem. To maintain dr machine by its own.

C

But I would I've linked the epic and I will also share it on twitter. I would recommend subscribing to it, so you get updates.

B

B

No, my windows flow, no, no, okay,.

B

Yeah, that's true, I hope so my job is running. Why doesn't it come.

B

B

B

B

B

B

B

Now this is an interesting topic, because now you can also see here now the provisioning steps. What it really do is like it's connected via sh. It will spin up um ubuntu, mostly and then installing docker copying the certificates and some there are currently some problems of setting up for certificates on the docker daemon.

B

This is why our job isn't triggered right now,.

F

I am 100 sure that I have seen this error before. I just have to try to remember the solution of it.

B

It's a long time ago, when I do this, it's like when it one time when it works, I don't touch it anymore, yeah and it also reduces like, like mark said it reduces my cost from 60 euros a month to 3 euros or something like that. So because I don't have so many workloads that needs to be run. Sadly, I need only on some time and for this auto standing really great in terms of also saving costs, not that I need to have a many feet off runners. I need only runners on some point.

D

So if you have many runners which run 24 7, it's not the best option.

B

Yeah, that's true: it depends on the use case. Yeah. Really, if you are, if you are sitting on your company, you are bare metal. Probably then it's easier in terms of cost effective that you use in your machines, but if you're in a cloud environment where you have only where on-demand is um expensive, more expensive than running it right in long-term, uh when running on, demand and long-term is really expensive, then um autosteering could be a good way to solve this problem.

D

But I can also mix it, so I can use my own demand runners as well as.

B

B

There are a lot of options that you can do.

D

But the cool thing is, you can always register any kind of runner to gitlab, and so it's.

E

Always hidden behind.

D

F

Stuff, so so there there is a workaround for the issue. It's basically to use an older docker version in the in the runners. I'm not sure. Maybe there is a an actual fix by now, but uh to get it running. You can add this to the options in the docker machine. Config.

B

ah Okay, yeah, okay, so let's do it.

B

F

B

F

Machine option, machine options, yeah.

F

I haven't checked for a while. If there is a better solution right now or not.

B

I could be the problem because in docker 0 19.3.10 they introduced the safe issuance certificates and so on.

C

Yeah and I think that's the problem.

B

I remember because right it's right now, interesting when I run docker also in didn't mode in gitlab, I don't try to update docker with every update anymore, because I need to fix it every time again. So it's quite weird so currently. For me, the newest version also not working okay. So then I restart the runner again.

F

I guess I should update my blog post, so if people try to implement it, it actually works.

B

Yeah for this for this, for cafe so we checked your word processing.

D

You can link the video as well.

F

Yeah it's con. I need a continuous testing for my blog posts if they are still up to date.

B

F

Mostly, you are having like a.

B

C

C

There is there are some people watching right now and they will try it as well. um I just I'm linking your blog post in the in the video description um as soon as possible.

F

Yeah I will update the post after the meeting so.

C

B

Feel free to do it. I will delete for all conflict that we don't need.

D

B

I'm so bad with them.

B

But now we have a problem literally. I think.

B

B

Okay, now it starts better probably, but we need probably to re-run the.

B

B

Let's start a new.

B

Pipeline um and most important check if the servers are still running, delete them by hand to them because they aren't controlled anymore and then you're wondering why your price gets high.

B

I will delete them.

B

And you can also see the new jobs are currently here. So, let's get back to my toner.

B

And hopefully now everything works.

D

B

Yeah, it's true.

B

C

It's the time for coffee.

B

It's mostly like waiting on pipelines, so it's like the xp cd.

B

ah We have no next step yeah. Now you can see the runner is executing that and the true stuff behind that is because we are using docker. We are very flexible to which image we want to use and we don't need to stick on to all stuff.

B

um We can choose it by our own and our operators needs only to deploy ourselves to runners and we can use.

F

F

uh A while ago, I also did some presentation at work about something where we needed to wait for the pipelines, and I also just told the other teams, so if they want to get paid for watching pipelines, so they should join our team because, basically that's what.

B

Yeah, mostly waiting on one pipeline.

F

B

B

Another there's a problem on on the on the project, so probably.

B

I think it would also break here.

F

I'm happy it's at least not an issue with the blog post. It.

D

Seems to be this, this crypto crypto library.

B

uh It's a draw problem.

B

Yeah we can see.

C

Oh, go home, go home.

E

B

And you accept yeah, but that's really the essential uh meaning of um vegetables. So.

B

And it's straightforward, mostly so most much through blog posts, um it's very easy to do this, and probably you also can do this on your own.

D

So you mentioned that you have done this on a parameter setup as well yeah. So then you have to write your own plugin.

B

Now, there's a kvm driver, but it's not maintained anymore. Essentially, okay, it's like mostly doing this stuff on bare metal is more like tinkering. So it's all like, like my favorite um naming of this, it smell like duct tape programming, so we're using duct tape for everything everything works, but it works in the end. It was um also scalable for the use case that we had, but currently was this approach. I wouldn't do this anymore.

B

But it depends so the library that I mentioned is um here: it's the doctor machine driver for that.

B

D

Machine will be maintained so yeah.

B

D

Parties to to have an interest in that.

F

Yeah, I think if gitlab would stop maintaining it, then they would provide iron alternative. I don't think they would drop it without another solution.

C

um I think there are ideas about writing, writing additional custom, executors and and like integrations, but it's really hard and yeah. You cannot drop something which is used. I don't know, maybe in in lots of installations, although all over the world for now it's reliable and I think we have three patches inside so basically technically, it's a fork somehow, but still there are some bugs which cannot be fixed. As far as I know, so we need a long-term solution, which is should be the the epic discussion um to follow.

D

Yeah, but it's interesting to see that I, if I remember correctly, docker machine was just built for in the beginning for having this docker tool boxes in on mac and windows. I remember correctly interesting how how far this project has come.

B

Yeah, mostly everyone in the community, use it so um and.

E

B

Important, oh no, the lucra is gone.

D

I've used it a lot with with mini cube and fighting against the company proxy, so.

B

But most important uh like every technology, you should choose it by the logo and docker machine has a really cool loader. So this is the original loader of docker machine.

E

I didn't know that.

C

B

And we'll see it's also stairway runner, so multiple ones yeah but um yeah. That's it literally before we get up runner, auto standing.

C

Yes, thanks a lot for like diving into this, um it's great to uh to see that it works so lots of motivation. um Probably we can either install a gitlab server next week or leave it as a home exercise to patricia hi.

C

um Yeah, I've updated uh the the youtube video description and responded in the live chat. um I've live tweeted, something so I think, do you have anything else in your mind, you want to share with the audience.

B

F

F

It's more like I wanted to ask you uh about a future topic, but uh maybe just after we've only if we are finished with the outer scaling stuff.

B

We are probably finished, so I'm currently deleting all the stuff. Okay.

F

It's too late for questions.

B

No then we do it again. No worries remember exercises okay, let's put it into interval or something like that, so that we can repeat it.

C

Maybe maybe we can try it next week with terraform, I would love to learn more more more terror, terraform.

B

C

F

The original reason why michael asked me or invited me to this meeting like a year back or half a year back, I don't know, was because we talked about k3s on hetzner on twitter and uh after christmas, I started to hack a lot of errors on this and I now have a gitlab pipeline which uses where you just give your hetsner api key and an ssh key, and it will use terraform to create vms.

F

Then it use ansible to provision them and to install k3s to configure everything, and then it uses help file and term to set up all the hetzner networking and storage stuff inside of the cluster, and so after 10 minutes, you have a running kubernetes cluster on hetzner.

F

So if you are interested, we could do this in one of the next cafes yeah.

B

E

You can do.

F

Okay, I don't think we should do all of it in one session, because it's quite a lot at least if you're new to terraform and new to ansible and new to kubernetes. Then it's a bit hard to do all the stuff in an hour or so, but we could start with the terraform and ansible stuff, for example, and then another week install uh yeah kubernetes in it.

C

That would be awesome.

B

And after this I will show them how to install gitlab on the kubernetes.

E

B

Signed certificates.

E

F

Yeah, that's nice.

C

Do you want to start next week well, you're free to like say no say say yes to anything so.

F

Yes, I I think I can prepare something for next week, because on thursday next week I will show some of it at work anyway, so I can prepare for both events. At the same time,.

C

Perfect, we can, we can battle test. Your presentation, then.

F

Yeah exactly yeah, maybe we should start uh asking hetsner for a sponsored account where.

C

He's supposed to vms yeah, I think uh lucas cameling, I think, has retweeted my tweet already so he's he's watching or maybe watching um yeah uh I'll ask him.

F

Yeah, if, if not it's, I don't think it's an issue. It's not that expensive, but it.

D

As we know, it's not that expensive, so yeah.

C

Yeah, let me know how many just send me: the the 10 pages pdf you get afterwards and I expense it for you.

F

Yeah then I get 50 cents. Yes,.

C

um I only paid trisha casey, you know, oh.

F

Okay, that's that's! I'm fine with that, because my resource, my the one you gave me and nuremberg, are empty.

C

I I think we have a deal to be honest. um So, let's, uh if you need anything till next week, just ping me um otherwise, um thanks for everything- and I would say we say bye to youtube now, write.

F

C