Rust Programming Language Rust Linz, 29 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: CI with Rust and Gitlab Selfhosting - Stefan Schindler - Rust Linz July 2023

Description

How can we keep critical code private but still use modern cloud technologies? This is a experience report including self-hosting and the development of medium to large size projects.

--

Stefan Schindler started over 15 years ago on the hardware deployment side of IT and has ever since become an OSS maintainer and Lead Software Engineer for tipi.build working on the next generation of compiler toolchains for reliable software.

A

Something from a Linux system called the steam deck. Oh there's so much delay uh yeah. It will be nice, it's a I, think a six or an eight core CPU. So it's an easy gaming handheld and now it's running presentation, uh yeah! You can follow my blog. uh What do I do so I'm here in lens to finish my master's degree uh should be done. Soonish, so yeah end of the year, definitely probably end of the month. I maintain couple open source crates on the rust side.

A

Here maybe has been threat, pool or parse int or pseudo. I can highly recommend Parts in because it just deals with all the prefixes I'm.

A

Starting a new job tomorrow. That's why I'm moving away from Lins, so the classical disclaimer opinions are my own, my employer TP. They are in the similar space, but yeah don't speak for them today and in my spare time, I organized another rust Meetup called turisi, which is the lake of Zurich. So all around that Lake there's also talks and rust. Fifth, if you want to join the rust Fest family event, you can in October in Bristol.

A

So in Brussels it's called rust. Dot. U tickets are available. Schedule is public, oh a book a day early, because there will be workshops that have not been announced yet, oh ah cool, so they have been announced. Wonderful.

A

uh Let's go to the table table of contents or not because I don't think you can read it and for the people in the Stream you will just get what you get. So, let's not waste any time so continuous integration CI, it's basically a thing to execute code on another machine. That's why I highlighted it here. It's a remote code, execution very important.

A

If you give access to someone with your to your CI, your CI will just run with the same privileges all over again and like we had with the Lambda cash attacks, where someone figured out that slash TMP is read writeable for all the Lambda functions and you can read stuff from other customers.

A

Same is true for continuous integration. If it's not set up properly uh and what does that mean? Etc properly well, Docker still runs as root by default and the default installation is in a Docker container. So your continuous integration job runs as root alongside everybody else.

A

That's why there is Portman which I can recommend, so you set podman up and then do the mapping properly and then it's not root anymore. It's just a user and of course you can use kubernetes k3s, for instance, and then configure that as the runtime or executor, but it will go over that as well.

A

uh Why is it so very, very handy because it's bundled with the repository if I have 15 different versions or branches or on pull requests and merge requests and all of the things and small things change? It will always build with the same code, even if I change the CI infrastructure itself. So that's very, very nice, and then we need some Advanced features right like we need a specific environment variables. We need to have secrets for deployment or for pulling down proprietary data sets.

A

uh We have caches. Sometimes that would be very nice and we may have artifacts. So the difference between those two-ish cache is read, write and you can manually delete it per repository and artifacts are outputs that are generally uploaded to something else like a website through the gitlab functions. There are similar stuff on GitHub, but it's called slightly different.

A

um We also have variables in the scheduling. That's basically the templating. Oh, is the mic back on? No, it's not! No they're fun am I gone when did I clip out.

A

Stefan is not here come on support.

A

uh Oh, he walked away with his phone right. Did it disconnect? No, it's still connected.

A

uh We're lucky today, they're very lucky.

A

So how far should I backtrack.

A

Can't wait for a third device.

A

Did we just lose the Stream okay, so uh yeah I was with the templates, so we can have symbols in our yaml file that will be interpreter. Yes, I hear myself again very good and another very nice feature is triggering events in different repositories.

A

uh Sadly, I cannot show that code because it's from one of my clients, but it basically works like this. They have a open source base installation and whenever CI starts to run on the main on Def and staging branches, it will send signals to a deployment repository with all the secrets and all the configuration for the production, and that part is very private.

A

So it will just forward the signal and you can see in the open source version if all your tests work and then in the closed Source part is the deployment. Did it actually deployed to the machine or not?

A

Now how do we think about stuff? So we have one gitlab CI dot yaml file, which is called a pipeline. This is one thing that runs through inside that pipeline. We can have stages that are just blocks and we Define the stages in order and they run in that order, but because that's too easy there's also another mechanism called depends on. So we have a full dependency graph, which of course we can build loops, and then it will crash or just hang.

A

So they have these two mechanisms I personally liked the stages because they're easy, but the graph is also very nice because it draws you a nice picture and it auto hides stuff, and it's very similar to github's Matrix, where you can tell make me a matrix over processor architecture and operating system, and then it just wants 16 jobs at once. For all the combinations or more another thing is we have per gitlab instance, so the web UI that stores the code?

A

We have shared Runners and non-shared Runners, and if you have code on the public gitlab.com instance, I would recommend just add your own Runner to projects that compile often or take long because they reduced the amount of free time you have on the system and yeah. If you don't want to be surprised, if you're close to this limit, it used to be 2 000 minutes per month, I think it's less now just add your own system. Let your system run all the the jobs and there will be no more surprises fun fact.

A

If you have a Workstation, you can just run it in a virtual machine on your local machine or in the docker container on local machine. If you're, the only one, that's linked to that Runner. You trust yourself. Hopefully.

A

Right then, you can just have it running on the background could even be on a laptop and with some smart trickery. You could set it up in a fashion that the runner would not run when it's unplugged from the power, for instance. So when you code on the go and then go home to your office, just plug the machine in and they will realize oh now, I can process the queue and it will run all your CI. Gitlab is smart enough to recognize when a job is no longer needed.

A

So let's say you have a branch and you add five commits to it. It will not run all five commits just the last one unless you tell it to, but usually it's good enough to run the last one all right, so costs last talk had it as well.

A

On the public instance, we have one hour limit and low lower priority compared to the paid plan, which is 30 Euros little less per user per month.

A

um If we compare that to some some random VM, which is like five euros a month and looking at it for like half an hour every month, just to apply updates for unlimited amount of users, it's a pretty easy sell even to your boss. Disk space just put something that has more than five gigabytes. 10 is probably tight. 50 good enough.

A

Networking is usually free with servers and yeah, because you usually pay for outgoing traffic and a gitlab runner is usually pulling a lot of data in so in-brand traffic is usually free with all the servers, so that's very nice, and for caching you can use an S3, compatible storage, don't have to use AWS, oh and if you use AWS, be aware that they charge you for traffic within their data centers. If the data centers are in different zones, and that can be very expensive just yeah be aware of that.

A

So another project where I advise sometimes now is, is trapped because they have a lot of storage in one bucket and another Huge lot in another one, and they cannot synchronize them anymore. So they just manually synchronize stuff that they know this is wrong, but yeah they cannot afford to just download. Both sides compare and fix it automatically, because it's a couple terabytes and it's just very expensive.

A

Another thing: privacy, safety, security, deployment, Keys- is a good thing license key is another thing, sometimes you're not even allowed to run certain software on certain CPUs. So if you're, for instance, have a I don't know Adobe license, that is for 16 cores and the shared Cloud spawns you on a 64 or 96 core machines. You, sadly breach of license, would be really hard to blame you for that, but kind of just put some old books in in the corner and let that run it yeah doing stuff local self-hosting yay.

A

So here we see a typical flow from the runner's perspective and it's all http who remembers long polling or a couple so for the young ones. This is before web sockets was a thing yeah and sadly it still works. It's still need it. uh Yeah. It's a long polling uses the TCP timeout, which is still 30 seconds, so it waits 25 seconds.

A

And if there's no data, the server will reply with no data, and then the client will pull again and do that over and over until there is data- and there is a chance that instead of waiting 30 seconds for the event that happens on the server side, we wait 10 seconds and then we're 20 seconds faster. It's still slower than lap sockets, because we have to reconnect all the time but yeah, that's the thing, uh the right side.

A

The far right side is the executor, that's communication that depends on the plugin and what plugins are these yeah? It's usually docker. Let's be honest. Most people running uh is a Docker container or have it as a systemd service, which I personally find very handy because with system TV can restrict the service a little more without touching the service itself, so we can, for instance, say private temp, so the service no longer sees slash TMP from the system, but it's locked away in some random directory that will change every restart.

A

The service won't see it. You can also restrict the capabilities, although to be fair, it doesn't make much sense because most of the capabilities like Reading Writing and network connections are needed and that's usually the dangerous stuff.

A

Yes, someone moved an umbrella, so if we add this, we can also make an ansible role. Probably can't read it, don't worry, so it's it's installing in GPT key and then adds up. Let me scroll down then enables the assistant D service and then we'll add the repository as well.

A

You see this is a distribution dependent. So when you are on testing, for instance like Debian testing or the I, don't remember what the Ubuntu Leading Edge is called. There is a chance that when the testing name gets announced, your updates will fail for a couple weeks until uh gitlab itself has a new folder, so you can also hard code. The distribution release or override it. Ansible is highly customizable, sometimes a little bit too much and you can override almost anything so the executors so the first two SSH and shell are the most dangerous.

A

Basically because it is what it is. It just opens the shell either remotely or locally and just executes whatever's in this yaml file, and that one is not contained at all. So, no matter what branch it will just execute the code.

A

You shouldn't use that unless you have a hundred percent control and don't accept any untrusted, merge requests at all another one is parallels or virtualbox, where you spawn a virtual machine. There used to be one for KVM, but it was discontinued. Sadly, there was also one for extend also discontinued.

A

There is the docker one and if you're just simulink the docker binary to podman, you don't have to worry about anything because Portman is API compatible and there's a Docker machine plug-in, which is the same as Docker, but just starts a VM first, that one is really handy, but it's experimental. Sadly um yeah and of course we have the kubernetes one and kubernetes is great, because you already have namespaces and whatnot and yeah.

A

The only downside is we have to run kubernetes somewhere and have all the Ingress and auto blah blah blah so for small installation. Docker is probably fine.

A

So, let's look at some code: this is from the sudo project. The suda is one crate, one Library, nothing fancy. Can you read it in the sunlight? Okay, nice.

A

The image is a Docker image and when a Docker Hub announced that they would charge for stuff and just throw everything over whatever I was a little worried, because I would have to update all these image paths like everywhere.

A

They reversed the decision. Luckily, because yeah it was very, very stupid idea and then yeah, you use the image. The image on the top level is the image for all the jobs we can override per job as well, and then we have the test stage, which is the first part, is here and then some name.

A

This syntax has been deprecated so now it will just reinterpret it as test colon cargo. Well, it works it's just one stage: that's all. It is I recommend outputting the compiler versions, no matter what language, but if you use node just say node version or Java or whatever this is printed, it will be locked and then you have a much easier time figuring out. Why some feature is broken or it gives you random errors or invalid syntax to check gift version is correct.

A

Then we run a test, that's yeah! What it is this one is in debug mode and then last but not least, we test all the example code, because this repository basically has a lot of example code because it cannot test it, otherwise, so yeah it is as it is.

A

If you have questions just like shout, they will skim the stream now.

A

Yeah, just some ad for hezner I, don't like heads no personally, they have an anti-tore policy or at least had one yeah use, IP projects or unglig.ch. If you want to be CO2 neutral, then you can host your stuff in the Swiss Alps yeah running of water power. So it's nice.

A

Okay next example. Parse end parseint is a little bit more complicated because we have features now and we have nightly the difference between nightly and not nightly is the image line. So we just add whoops, oh, come on double click you're at attack nightly at the end. That's it that's how we change images and for features. We just run the test without Flags and with flags yeah.

A

It's not that that hard right, it's good to do. You can, of course, do more stuff.

A

So this is from my master thesis and here I use a different image, so I don't use the stock compiler I use something that I built myself or this here so to change the registry as well I'm hosting this one on gitlab and I use J unify, because that allows me to print test coverage like how many lines are covered in the test.

A

Then Fork actual test coverage again: printing, the rust version, testing the whole workspace, this workspace flag. This is very handy so because it will tell cargo yeah yeah go search, go deep test, all the things, and then we can modify the output format. You can see here we have whoops Json, then we have unstable options. That's why different compiler report timings and then here's a normal shell pipe, oh yeah, and these little squiggles at the end. It's just new lines that I had to insert because it wouldn't fit on the slide.

A

And then we calculate coverage by just running a normal, build and then have llvm coverage to the actual coverage reporting but yeah. We need to have the tests running first, because llvm coverage will fail weirdly. If the rust code is somehow unsustainable like, if you have syntax errors, just in the tests, it will compile under normal circumstances, but then the coverage will fail because there's an error in the test. So we need to run the test first and then build again, but hey the CI does it.

A

We will never forget it and it'll just work and remember the artifact. This is an artifact, so the output of the artifact is some XML and sadly the latest version of gitlab broke this a little bit. I haven't fixed it yet, but you could just tell it here's the report template for junit, which is a Java testing framework.

A

Xml, is XML just look into that folder pull it out and then add it to the branch.

A

Yeah I will fix that some point: there is other formats uh I, don't remember, I, think four or five different coverage formats are supported.

A

Maybe I don't remember yeah, but you can find it in the docs which ones are supported now. Another example is color Blinder, it's it's tool, I wrote couple years ago to simulate color blindness and different kinds, and then it will output, a picture or a bunch of pictures or a combined picture, whatever you like and for the pure rust version. It's fine just like sudo, but for the UI for the gtk version.

A

We need some dependencies and now we have this before script and this one will get run before every drop and I use that to set up the environment, it's Debian based, which is very nice, so we'll just tell update, upgrade and then install, and then this is very handy. Install no recommends, so we don't get extra stuff. We don't need just gives.

A

The smallest footprint and smaller footprint means faster, download times faster runtime for the container, and then we install the libraries we need like Pango and gtk3 should update that the gtk4 at some point and then we just run our normal script. Just cargo test workspace. There you go and it just runs through and of course we can play with stuff. So one of the additional features is deployment and yeah I have heard of pages. Probably you know GitHub has it gitlab? Has it it's a static site hosting service?

A

That's integrated into the hosting platforms and the plugin is called pages, so we just have to call the job pages and put it in a stage deploy and then whatever we copy into this public folder will get transferred out or we have to mark it as an artifact and the plugin will then just look for this public folder in the artifacts and then put that onto the web server.

A

uh It used to have some bugs, for instance, when you would trash the web service, it would just be empty, and then you had to rerun all the jobs, but that's pretty easy. You just go into the latest one and then there's little I can just press it and it runs the job again, and the nice thing is, it will run it at the same. Commit with the same environment will be very much the same unless we pull something from the outside like time or download, will output the same result in the end?

A

Oh and before I forget this only is the list of branches where we say we upload the artifact only on the main branch or the release branch I used to do some magic with DirectX and searching for a git tag and whatnot, because it also runs it on git tags, but it's hard and the syntax changed, and this is way more reliable and then we just have to protect the main branch and then push to Main Branch, only with merge requests, and then we have release management as well.

A

So how do we deploy stage? 0 is setting up some sort of trust right and SSH is really great for that, because we can have verification in two ways. So we have one token it's the SSH deploy token. This is basically a private key which we can generate on our machine with SSH keycan and the output of that will be the value. It's just a one-liner and it will be saved in the other side. Is the server and we don't have?

A

uh How is it called the x509, the TLs trust chain right? So we don't have that we don't have a trust anchor with SSH, so we just ship it ourselves. We just record the known host files, put it into the environment, save and then uh we are ready. We have another script again, just set it up. We install rsync, we install SSH, we create the folder, we place them in there. This is an old script, because in the beginning, you could only have environment variables.

A

Nowadays you can have files, but it's a little tricky if the folder doesn't exist, so I still keep it that way. Then we change the access permissions. Then we have some make deploy. This can be whatever in the project. In this specific one. I use make files, because I have a lot of reoccurring symbols, so I use make files templating as well and then in the end, I just remove it, and this might sound silly because the container should get deleted, but with any garbage collection. There's no timing guarantees.

A

It will delete it at some point. Maybe next year right, we just don't know so. I just delete the private key from the container. It's they'll be still be findable on the physical disk or the virtual disk, but it will be much harder than just opening the directory and there it is.

A

So how much speed up can we get if we enable the cache? Yes, we do.

A

uh The caching yeah it's actually this slide.

A

Yes, so rust is, is a little bit tricky to cash in in the beginning, because you have the Target and the registry right so to Target is our build artifacts and the registry is all the dependencies and the index as well.

A

um If you have rust version, 167 and older, it will copy the whole registry all the time and that one I don't know how big it is. But it's couple hundred megabytes, so it will just download the index all the time and then all the dependencies, also, which is great for your stats on crates.io, but it's really slow so on a big project I had with arctics it just was pulling 15 minutes of dependencies and then building because it had the debug cash already um testing everything.

A

It was just three minutes of that, so enabling registry cash with this handy, Little, Sim, link nope, doesn't want to Triple click this one it's little hockey, but it works great. So it just say here move that into before script is important, because if we don't do it, the cargo command will already have created the structure and with this it will be redirected in our cache.

A

So that's very nice. We could also cache the release binary right, but I use that for deployment and there's some weird bugs, sometimes with caches, on release mode and I want to have a reliable deployment and I don't deploy every commit, maybe I deploy every fifth commit or something so I rather wait. The full build for release mode, which is I, think nine-ish minutes yeah. The other one is the release mode yeah nine minutes, instead of 20 still good, still an improvement by 50, but I don't have weird artifacts and I.

A

Don't have weird errors on runtime yeah. Does that answer the chat question.

A

A

Great great yeah I see uh git love, yeah gitlab has great runners in general, but they run in the Google cloud and I. Don't trust the Google Cloud, so yeah I won't use them, but sometimes it's funny. If you have a runner on old discs and not enough RAM, it will be very slow because the discs are just slow and sometimes you notice, weird lags in the lock and that's when there is a flush somewhere.

A

So if any part of the whole chain does a flush like pip does at the end of installing dependencies or some programs to it when they crash, they write the core dump and then they flush. It will get weird timing, artifacts in the environment, so I think we waited half a minute and I don't see anything oh yeah, moving on so an S3 cache bucket uh GitHub has a very good retention policy for artifacts, it's just 30 days and the latest one. So we don't have to worry about that.

A

Unless we deploy to production like 50 times an hour, it just works right, give it a little breathing room and it will clean up after itself. For the cache, it's not so clear, but it's a cache so as long as the transfers are complete right.

A

So what I mean is when, when the client starts pulling a cache file, as long as the client receives the cache file fully, it will be fine if we delete it right after because the clients will The Client means Runners, the runners will just search according to a scheme on this S3 pocket. Is there a cache for this repository for this job? We can even add our own mechanisms and templates for this pattern.

A

If it doesn't find it, it will just create a new one and if it finds it, it will download it and then overwrite it when the job is over. Plus we can manually uh clear yes.

A

Yeah, so the question was: why create a remote cache and not a local cache? It's a very good question. In the beginning, you could do that so I had a folder, slash, cache and I would just throw everything into that one I, don't know why it's probably to do with the kubernetes support, but I, don't know I'm, also a little sad, because now I would have to run another service just for the cache, which seems silly so yeah currently I'm, not caching. For most of my stuff, it's just the stuff.

A

That's already public open source code, anyways that uses the cache and everything else just takes longer. Yes, another question.

A

A

Question the yeah I'm picking up the cash at the end and we're deploying each other.

A

Yeah, so someone from the audience had their own experience with caching and apparently caching, all the things took longer than the game was with the cash yeah uh sad, but yeah was NTFS involved with any of that.

A

Okay, yeah yeah. uh If you run NTFS and lots of small files, you will be sad, so Microsoft's latest recommendation is use their new file system, the reassess in a second partition, because you cannot boot from that. Currently, it's supported in Windows 10 with the latest Service Pack I mean, what's it called now: it's not Service Pack, yeah and windows. 11 supports it too, and the server as well the latest one, oh okay. So of course not the home edition you're just talking to professionals, I mean.

A

Where would you be if the education Edition was still the same features that as the pro right it is not yeah I, don't know, maybe they changed that back, but we'll see if we run Linux, then a very popular menu.

A

uh It's just one binary just downloaded uh configure it. Please set a password. If you do it like don't be like the CIA. That loses lots of spy data over an S3 bucket or the TSA, which uses the no-fly list over an F3 bucket uh yeah, and so many others yeah. Please set the passwords. We don't need another FTP leak. Oh it doesn't scroll sad. Okay.

A

um We can influence the cache by the way in the steps in our repository, so we can say for the deployment cache.

A

We use a different pattern than for the testing half, and so we can prefix stuff if we want to otherwise all jobs in one repository share the cash but as I said before, I would not cache production releases, anyways, so yeah, but you can so conclusions yeah in my opinion, running a runner is very low effort. It just works. You get updates every now and then, but I have never had any trouble and manage this free.

A

If you want see, if it even brings you any benefits, we'll see and most of the stuff I told you, it works for any system right, like githubs or I'm. Pretty sure IBM has their own stuff yeah, and if there is no more questions, then I think we made it. We have more questions.