GitLab #EveryoneCanContribute cafe, 13 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 12. #everyonecancontribute cafe: GitLab CI/CD & C++ ABI checks

Description

Tweet to start: https://twitter.com/axccl/status/1347665411722989568
Repository: https://gitlab.com/tonka3000/conan-cpp-example

For loops in GitLab YAML with parent/child pipelines: https://gitlab.com/instantlinux/docker-tools/-/blob/master/.gitlab-ci.yml#L38

A

We are live, I think.

A

Happy new year, everyone, I would say in the new year we have our 12th um coffee chat. My coffee today and one of one of.

A

Hi and we kind of wanted to catch up on an idea which we discussed, I think last week somehow. So there was a question around c plus plus and binary interface compatibility and how to make it work with gitlab, ci, cd or generally speaking, with ci cd and.

A

I think I'll propose something around this and also michael eichner, and I think marty did some more things around this. Would you mind taking over.

B

Yeah, of course, um so I also read the tweet nearly at the same time as you, you, you read it and I've used your recommendations to try to build a repository to demonstrate how it is possible to use this api dump stuff so that you compare two different apis and um the tool which mr axel huble.

C

B

um Is mentioned is from the abi leopard laboratory and they build tools, for I, I would say mainly for the linux community, that they can track the api differences between projects and if they can reuse the same binary without recompiling, and there are two tools behind this.

B

Like you can click here, and here you can see. There are two main tools for that. There's one called abi dumper, which is the name, suggests dumps the api into a file um can be used directly from any packet manager which is available, but can also be used as a docker image, of course, and then there is a second tool which is called api compliance checker, which is more or less based on these stumps.

B

So you give this tool two different dumps and it will calculate and report, which is the difference between which things changed like there is an api removed which previously was public or something like that, and so I just try to use this and import this to gitlab and um it's not 100 complete, because I want that uh we can build something today.

B

But the general um thing is I've just built a completely simple example, so based on cmake, because I'm know simic very well.

B

So it's for me much easier than just providing make file for me and it's just a simple hello, build main file executable with a lib file with one class and I've already prepared the main branch that I have two functions so and- and I want then detect if I change this in the chat to and for example, double I want to see this in my gitlab repository and, of course, in my merch request, so that I can easily see what is changed because excel mentioned that he want to see this on.

B

The pure merch request, aka, pull request basis and, as I said, simple project just the api stuff. I tried a little bit around and the hardest thing for me per se was to convert the information from the api dump or api compliance checker to something which you can upload to gitlab, because there seem to be some restrictions in the api, so you can copy paste some html code in the markdown in the in the in the website, but the api is very restrictive there, so I've built my own wrap around it.

B

This was a little bit a little bit tricky, but it's just how to convert information to normal text files, and I have two versions of it. One where I prepare.

B

I reduce the html from the given report and the second one is this: uh where I can use the xml output of the api compliance checker and convert it into something which you can read. Of course, and I've tried to make merch request here and you should see if I left it there yeah.

B

This is the thing which, when you push something to the branch, I've demonstrated, how which is the base commit, which is the head commit, and I've just used the from the report. The binary compatibility table and the source compatibility table there is much more behind it, so I can show you the full report, so the tool do much more than just.

B

So this is the original report, so you can choose between binary compatibility and source compatibility, binary compatibilities, of course, ebi and source compatibility is api and in normal programming you can break the api and have a stable api in via versa, not an okay. In many cases, when you break the api, you break the api, but there are cases where this is not the case, and here you can see the test results and, of course I've never changed something in the branch. So, of course I didn't change anything.

B

So it's green hundred percent is gone through. I have no problems at all and to show you this little thing: the black magic is behind the gitlab eiffel. Of course, what I've done is I've just used the docker image from from conan. So that's my main packet match I use for for c plus for stuff on. They have neo for many.

B

Many distributions really really good prepared, docker images, I use it with cheese 7 and I just installed api dumper and the compliance checker is also included in that which is really interesting, and then what I've do is I've checked the api we've given one dump I've given the second dump, then I tell that compliance check.

B

Hey, please put out this html file and the second call is the same call, but with the dash xml and then I get xml representation, then I use the ci merge request iid to let my script convert the stuff and upload it to a the symmetryquest via the gitlab api, and I skipped one part: that's the part where I just built the stuff, because there are, it seems to be. There are two modes in in this api dumper. There is one based on just header files and one based on the on the binary.

B

I choose the binary way, because I think that you can only compare binary mode when you build the binary. I guess, but I'm not quite sure I never used this project before. I only know that it exists.

B

So, just simple as that I have my script. I just built with regular cmake my project and what I use is the artifact system from gitlab and tell the system. Okay, I want to collect all the aso files.

B

Of course I have only one the lib1 file and it uploaded to the artifacts, and I built the second one which built the whole thing and there I used the see what michael mentioned in the in the tweet, the ci merge request, diff base sha um to build to check out the the base, commit, build it and also collect this file, and in my api stage I, as I showed you, use this to feed the compliance checker and the dumper to produce them the file.

B

So that's the main, the rough concept. I'm not that aware. You can use many techniques to optimize this. It's just a simple step, because if you have a bigger project- and I think building always the old stuff is- could be really a performance problem, but I'm not using gitlab's cache that much. So I'm not an expert in that.

B

So I think you can cache the old results and use the cache uh normally, and if the cache is not there, then you can build it. So it's a kind of performance, optimization there and was what was a really interesting problem for me, which I never used before in gitlab. Is that the I use the only thing you can also use, of course, the the newer one, the the rules stuff.

B

I use the whole the only tag uh for the merchant stuff and the first thing which I have I say troubles with it was how gitlab handles this, because I, in the beginning uh I've get this detached state. I never have this before in github.

B

I use gitlab for many years now, but never have this detached state, and if you hover over this, it will tell you that this isn't in the context of the merge request and not in the context of the push- and it looks to me like it is an issue from myself, because now you can trigger two pipelines by accident.

B

So what I've done is I at the moment I've have the merch request only for every of these three jobs, so I get only one pipeline. If you leave this, for example here, then these are two pipelines and then I cannot use the artifact system to get the artifacts from one job to another. That was something which I have troubles in the beginning, because I never used this method before, and this was also something which and that don't really see that fast, because this variable is only set when you have this tag.

B

Maybe mike can learn more about this, but it was a little bit uh tricky for me and the only way I found when I have not. This thing was to to require uh to request the from the commit. I have to get back the merge request, but I think that's not the really clean way to do it, but this worked for me, but the problem is to find the root, because I now have the merge request, but it was really hard to find this out. I'm not 100 sure how gitlab filtered these others.

B

I would guess the store the they know which commits are in the in the merch requests and use this information to set this value, but I'm not the ultimate get expert, so maybe it could also be done with the command line.

B

um This was really hard for me and the second thing was, I found a really cool thing. I don't have the the just use the the community edition on gitlab.com, but there is one interesting way to make much more of this available because github and it has this report stuff and you can upload metrics.

B

This is normally used for things like open, telemetry or open metrics format, which is originally based on the format from promisius hope is correct, and I convert this xml report also to this stuff, but I cannot check it, but the cool thing about this. It shows you the the the text file at the the top of your merge request, because the way I choose for the community edition is that I just push the merge request. So the problem is you have multiple ways to do that.

B

If you always push the comment, you at the bottom of the page, there's always the newest one, the cool thing about this reboot stuff, it's always on the top and the matrix was the the cheapest way to do that, so that I just convert a simple promisius format or matrix like it's called today and just fill this out, and then you can see it at the top. But it's a premium feature. So if you don't have a premium or or at least starter, I guess, then you can't see this feature yeah.

B

So where should we start because the the interesting part, but I'm not the expert in this thing- is how to use now the cache to optimize the displaying, maybe michael, can help me that in in this field,.

A

Yeah, I was thinking of caching, caching, the abi dump or the object dump, and only checking whether it exists already for this merch request. So we need to use. Do we need to use a specific key yeah? Probably we need to store the the dump file with the merge request id or with the edi the internal id.

B

But the thing with the cash is, is the cash only sits the cash only on the runner or is the cash also.

C

A

Artifact should be, the thing is if there are multiple pipelines and I'm producing should be pipeline pipeline um sandboxed. um So the idea with the much requested doesn't make sense.

A

We've only used caching and not artifacts in that regard, so we kind of want to store everything inside the cache similar to, as you would do, with node.js modules and other things.

A

By the same job, in a subsequent pipeline.

B

Because there's a stored, wrecked, lab runner is installed and uploaded to s3 if distributed cache is enabled. So I would.

A

B

In the default.

A

We could use s2 s3, but it's not. I think it's not enabled for our case.

B

So then I guess it's, it's just sits on the runner.

C

Wait, sorry, I missed it. So the last question you're asking if they using whatever cash, is happening. Yeah.

B

But when I understand the documentation correctly, um cache only sits on the runner side, so it doesn't delete it and it's not uploaded to a century store with the exception. If you have this distributed, cache.

A

So for the sharedrunnerson.com, I believe you can, can you.

C

Yeah, if you have so um typically when you use um distributed caching for vegetables, is when you a really easy example. Is you have two runners and you want to share the cache between two, these two runners, and for that you can't use the normal way um because for attached, first of all will be you put under slash cache over files. But if you enable distributed caching, um it will be uploaded to an object, store. It isn't limited to s3. It can be azure anything else, but.

B

Is it in the premium stuff or is it.

C

No, it's free! So literally when you have to set up multiple runners, um you can use this as three buckets, so when you're looking into that or going back yeah all on the side, it's also, I don't want so literally when you this is the configuration of a runner. So you need to configure your own runner if you want to use the distributed. Caching, oh.

B

I have to configure my own okay.

C

Yeah, but caching is when you're using gtlab.com and for shared runners, uh distribution is enabled by default. Oh.

B

So I have to do nothing on gitlab.com.

C

B

To get this feature.

A

I would, I would believe that it's enabled by default, because otherwise our workflows wouldn't work either.

B

But but you have to you have definitely the premium one.

C

But it's not that doesn't depend on the level because chasing is in the free version.

A

Caching is core so or the free version um you can rely on that and there is.

B

I just can use this.

A

And I will get metrics like the metrics feature you mentioned before. This is in premium only a premium plus, but caching and everything which, which is required to run a ci cd pipeline, um is available for everyone. The thing is, we need to configure the cache, and this needs sort of um this thingy and probably we might want to use a cash key.

B

Is there any anything to limit the cash so that they can say, for example, after the 10 upload, the elements get the oldest get deleted or if.

A

It's just ribbon a little. There are different keys for cache with the cache key. We have a fallback cache key. You.

C

Can use all the predefined variables so in terms of that, you can use the branch name for as cache key yeah see.

B

Untracked one is also possible: that's cool.

C

Yeah, when you uh untracked is when you um don't have it in vegetarian in the current stage, but you want to catch this, for example like the binaries or so on. um Then you can use the untracked stuff.

B

But there is no like expire in or something like that or like in the artifacts.

A

Probably we don't need it, because both.

B

A

Kind of when the merch request is created for the is created um you want to like, create the file and if it doesn't exist, we we want to create it or we want to use it from the cache.

A

um So the the first step would be to define the cache yeah, and the second step is when we are dumping the file. um First, we check if it's already there, so it's cached and if not, we are creating it. um So.

B

But I have always to run a gitlab job by myself to check it via the code or is there any built-in mechanism so uh only build this, because I only start the.

A

B

If there's something.

A

Well, you can run the job, and this is like when you do when you create the batch request.

C

And mostly, the cash will be only saved by default when the job was successfully, so you can also cash on failure that so it means cashing will be also crashed when you have a build failure or job failure, or you always cash.

B

So I can use the cash here because then I can check if it is used or not, and then I can use it in the artifact store right or should we build it in the check api because both is possible.

C

So probably hash will be also, uh no, it won't be put into the next stage. It will be shared between all the jobs yeah.

C

A

You mentioned that it's.

C

B

Directory so I have to use a different.

A

B

A

Should we should uh use which should empathy error directory for it and then cached? It.

A

Or like we define the cache and then we dump the the api or the object dump in there.

B

Because I think you can use the same definition for now, because I already built this uh old, uh the different directory to have it easier and.

C

Then you don't need to, then you don't need for artifacts right.

B

Yeah, so I only have to check it and that's what my question is the cache like the artifact system so that it place it in the right place. So does the cache.

C

B

Fill this value with life.

A

The only thing is the the script needs a logic itself whether to create the file or use what's in the cache.

A

So if, if it's there do nothing, if not create it,.

C

If director exists, although if I write a shorthand.

B

Batch right yeah because I'm a regular windows user, so bash check.

A

Test test f is for file.

C

Test is not available on every stuff; probably it depends. What I think test is not is it opposites, or is it.

C

Image, so we can, we can write one outliner if so, no worries.

A

I have it on my mac os, so um I don't know.

A

But I also have um which test? um Is it a new local? It's a shout built in commands. It at least says so see now set this age. Has it built in um it doesn't matter um we can either use this or.

A

This, but the thing is you want to check for the directory, which is dash d um and you can use the or the double or and say hey if this doesn't return. True now, if this returns true then do this.

C

But he do, he is doing it in the odd way move forward.

C

B

A

Submit there are so many ways to roam in bash. um You rather write write the price and script now. um No, I'm bashing bash currently.

B

A

I think it's a german blog post from uh christian quintop who's working at booking um about why you don't want to write bash scripts, but need to look it up.

B

Yeah like this, this one I would go with because it's conan, so there is already python built in so it's also one way to do it.

A

Yeah but to be honest, we need to like fix it now like it's, not it's not beautiful, but it works and late and then we're doing another iteration and make it more beautiful.

C

But I posted the solution already good.

B

C

I think it should work because you terminate afterwards you're checking when only the exists, or do we also navigate it so when they not access create. So it could be awesome.

B

So is there something in bash to to return so something like return early.

A

Break, I don't know.

C

Except what is it one? Because you want to change.

A

Your job you want to you want to cancel entirely. No, you want no.

B

C

Now then, you would transfer the whole job. What what do you want to do? Don't clear it all.

B

Right, just just don't do the the lines above so if or should we just do this and use the yamo mighty line feature.

A

C

I don't would do it in yaml. I would write a sharestrip for that, because it's harder to maintain, because then it sticks to yammer. I like mod writing it in pure shell, then and execute for shell script, because I can test it all also locally. On my machine.

B

Yeah, I need a fast way.

A

The fast way would be to look into the gitlab ci templates, how it's done in there, which means you need to navigate into.

A

Ci gitlab.com, I think, is the path.

A

I'll just go in there and we navigate, it should be lip.

B

Lip led yeah is it lips.

A

Is it or was it? I think it's just.

B

A

One yeah: it's it's lip get lipstick. I template.

B

Script, syntax, maybe a faster way, yeah this one, okay, pipe.

A

Okay yeah. I I'm soon copying the the path from my shell history or from a browser history.

A

B

A

You, if you ever need this um to look up how how gitlab does things you also should find some. I don't know if there's mighty line in this specific example, but I would say in the at least in the order devops example: there should be something around this or in the in terraform yeah. If you do the 99 stuff in yammer, you don't need the leading.

C

B

Is it possible to use the rules which they exist.

B

C

During run time, so it doesn't exist before.

B

A

The rules are like the static configuration.

B

I would try it this way, so if and then I would guess I can write normal so do something.

A

I think that the security scanners are using that um because I was deep diving into the is it the sauce.

B

B

Okay, then, the echo here, let's, let's try this I'm not quite sure if this is right because, as I know, bash is really.

C

Now then, you don't need to the semicolon. The semitone allows you to use more stuff in one button: okay, one five for ending, and probably you want to do an else.

A

But you still want to like skip the job entirely right.

B

More or less yeah.

C

And then you can do x0.

A

Yeah then it's the next interior because I was reading okay, just copy pasting, another url.

A

B

So I can just use exit zero here.

A

If blah whatever, then.

B

And this I don't need to, I guess I can do that.

A

Yeah, you could, you could add, like.

C

Why this will not execute one built uh lib exists.

B

This way, yeah.

A

Yeah, you could also do an echo in between, but I think we don't need that.

A

B

So in theory, if we see this because it's coming from the cache, then we should have the message already cached, and so we should have it in the in the next stage, because every stage get the cache right. Yeah.

A

B

And so we should get this so and.

A

Otherwise, the next steps are being run and and something gets generated.

B

Should we, the key stuff, is used to to override the stuff? So when I use different keys, I.

C

Will get these indicator which chest should be used sometimes.

A

You can have.

B

A different channel, the last when I use this uh this key uh this thing, then I will get the one from the base. I would guess so that I get it. The question is: do you want it compared to the to the last tag?

B

To the branching point both is possible, it's just an absolute value and relative value, but I would personally use the branching point because then you see just the difference from the from the merch request and not from the whole stuff, because then you have always. If you break the api in in two ways, then you always see this in every merch request.

B

B

But I'm not sure I'm.

A

I'm not sure if you need the key actually.

C

So this means, if you want to share with cash or not between drops, because.

B

I have to be sure that that that that I use the cache from the branching point.

A

But you could, you can have multiple, mrs, which have the same base id same base, commit.

B

But then it's always right because I just need one.

B

I would guess.

A

Oh yeah yeah in this case it's correct yeah, but you can reuse. What's what's already there yeah.

B

Because you have, you have your stuff, and so this is my branching point. uh This is my branching point, so I want to have it pre, uh suffix, your prefix or whatever, with this key. So then I can. I know that it's the last, it's the point for the branch and then I can see the differences between this.

A

B

A

Even it's even better from a performance perspective. If you do it like this, because um each commit, even if you generate 100 mrs based on on this base, commit you only generate a dump once because they commit uh you cannot change a commit to something else like add a new file and demand.

A

um So it's really unique and it saves you a lot of performance. If you uh develop fast and have many much requests, the uh the base dump is only done once and.

B

Cached, so just to to understand this correctly, I use the key to mark my cache. So it's like a zip file or table or something like that, and then I use the cache folder key to tell the system which key I want right.

C

Now this firm protection exists, cache four, but he's a different stuff.

B

Because when, when I use this key stuff, how can I tell the system that I want the cache from this key.

C

So you're now telling the you're telling now the system you you want to use the charge from the currency that you have in place. So when there's no cash for this key literally, then you can also do before that key.

B

But this is the this.

A

B

A

We don't have any fallback keys, we just have one cash. um The only thing when I think we need to ensure is do we need the cash in the second job as well.

A

No, I think it should work, I think, for the key. We need to like double quote it as a string, um at least it should be safe. Then.

B

But for me, how can I use? How can I tell the system I just want from this key? If this is possible or is caching a global cache and last one wins.

A

You may need to check rb job.

C

I don't get it no. What's your problem.

A

Because because in the first job, the the definition that ending the cash also means to use the cash yeah, so the build the build old job knows by this definition use this cache writing to it. Reading from it.

B

So, okay, then this should be fine if it's not there, I'd create it, but so that, then I have to to give it here as well right, yes, because how the system knows the cache, otherwise.

A

B

A

Something like extents, where you just define it once and imported it, but yeah.

B

But it's now copy pasting. It.

C

But also at the um path and so on.

A

To be really, the path is needed.

A

You just copy two lines, not three.

B

So I also need to pass as well. Okay yeah. I had to guess that when I just use no pass so it's a star, it's used everything in the cache.

C

Yeah partition.

B

C

Like an extraction, okay,.

B

It's like a zip file and hey this. Is it out so yeah? Then, of course it makes sense to to make some some yemel thing or some extent stuff too. Yeah too.

C

And mostly, if if this are only your two jobs, it makes more sense to put the cash t, above all jobs,.

B

But the the question for me is okay: we can't completely skip the artifact this way.

A

Yeah we can use that for the debugging, but you can disable it now.

B

So I copy paste the same stuff here, but then I have to make sure that this is the build. So that's not completely the same here, so this should work so because now we always write to this cache with this key. So we but.

B

Then we get in troubles because when we have multiple match requests.

C

Yeah but then, let's, let's first, let's try if it works as exactly.

B

Yeah because then I would use this way because other way I now would cache also this and then I could get in troubles when I have multiple bridge requests on the same, commit base.

C

In series yeah, but you could use a different key so that the merge request id is based and or something like that.

B

But when I understand it correctly, then I cannot. You cannot use two keys here right.

C

You can control it so.

B

I can use it from two separated: okay,.

C

Now what you can do is you can insert that your t is only pointing to the ci merge, which is the base jar, something like that. What you should do is ci merge, request id dash and then the other value.

C

B

Okay, so I think let's try this out. I just want to check if this is correct, I can't use the the vs code building, but do it the old-fashioned way, yeah so seems to work.

B

Add little fci caching.

B

So, let's see if this works.

A

It should work because, right now you don't have two um competing virtual requests: yeah.

B

So when this runs so um is there michael? Is there any special reason why this only merge requests react? The way it is implemented because I'm not.

A

I think I think it's a scoping thing, because, if you're not inside a merchant quest event, you don't know if this commit is like or you just want to save the cycles.

A

um For looking up whether this commit is part of a merchant quest now and if this event happened and things around this, so I think it's in it's event based somehow so. Immersionquest triggers an event and you end up in a predefined scope where you have additional variables available in the environment, um because I think, if you, if you enable that globally, um it would slow down um the entire pipeline because you need to like collect all the stats, all the variables all the values and in the end you learn hey.

A

I cannot fill in this variable because I don't have any values and it's basically a microsecond, no hope which, if you like, think of one million jobs per hour it it adds. It adds up to the performance, and I I would say in in a specific given job uh job in the scope.

A

You either have merch requests or you have um the commit id and yeah. I think that's the reason um if it doesn't make sense and someone else says: hey, it's super easy to fix or to implement in a more performant way. um Do a big grab in the source code.

B

For me, for me, the only thing which which irritates me a lot is that it's called detached in the label, because it's detached for me as a git user is also always something which is not good.

A

Detached head is something else that is true.

B

A

I have no idea why it's called detached.

B

So I think our cache seems not to have worked correctly because yeah copy paste syndrome, because here I guess this is built.

C

B

Because I've already renamed this to have two separated ones, so yeah.

A

Probably we should define like define global variables and just use them.

B

Yeah, the problem is.

A

B

A little bit different, so it's not possible for every thing, but the interesting thing is it: if already.

A

B

ah It tells me because we cached this, that okay, that's the reason why this happens. So is there a way to to clean the cache, because now.

A

Our minus dash rf.

C

B

I have just do it my way, uh my own way so.

A

How you mean to clean an existing cache in an existing pipeline.

A

B

This change fix, um you can.

C

Share it via api.

A

Yeah, that's true.

C

A

You're going into the settings.

C

Page of your project, but I.

B

C

B

Sometimes a button- and I never know where this button is from now.

A

In the center it'll be clear.

B

Under cache yeah yeah, I've.

C

A

Button for years, but I.

B

Never know what it's doing it's playing with jesus, mostly on varanasi. So then, let's clear the cache, and so it should break fix cache.

A

Okay, I never needed to use that interesting button.

B

I see this for years, but I never really used the cache because we have no. We have an and self-hosted instance, so no no s3 or something like that. So cache is a real cool thing. When you have this bucket system.

A

We have, uh we have used it for c plus for caching.

A

For c cash you need to you need to store the c cash somewhere because you're generating it and the next pipeline or the next build, doesn't have it. But you want to speed up your builds with cash yeah.

B

It's a really cool, a really cool way to do it, so I've definitely tried to to to implement this in our pipelines. So.

A

The thing is, if you have lots of files- and this is typically happening happening with node.js- it could be that the cache writing and the cache reading is slow. There is an open issue to to improve that and I think there are many comments in it, um but yet it's an I o challenge um and to find ways to like how to deal with ten thousand hundred thousand files in a cache. It's it's not. um It's not super convenient.

C

Yeah, but in the last release there was an optimization on that, because what they're currently using for vacation they're using a zip algorithm that it's not so fast because in my company we have node.js and some of these um weird that you need to crash ten thousand fights or something like that, because everything will be downloaded and for that um there's a new feature. It's called fast, zip, mostly uh and where you can yeah it's currently, it's uh you can. Currently, you can use it by feature flag.

C

uh I posted for late until the chat, um yeah.

B

So the interesting stuff is the the now the caching seems to work in the in the build old job, but I don't get the build directory here yeah, because I have to specify it.

B

But now the cache should be fine, so let's get all cash passes.

A

B

But the cash seems to be really awesome too.

A

Yes, normally it's depending on which, like language and framework you're using, but if, if you run into performance issues where, like the job, install something or download something where you know you will be using the space, it doesn't make sense to repeat it all over.

A

It doesn't affect you that much if you're, using, for example, shared runners and you have endless ci minutes.

A

um But if you for example, say hey I'm using the free uh version with, I think it's 400 ci minutes in a month um you can easily like try to optimize that and say: hey shop doesn't need to run 10 minutes five minutes, but we can like make it push it down to one minute and the other thing to to to think of is, for example, you have gitlab runners in aws or you use a cloud resource, and then you get the bill and say yeah.

A

The jobs are running out of minutes and you're paying ten thousand dollars now, um just because uh the ci pipe lenses are not optimized.

B

Yeah, so uh we use simic a lot and then, as you mentioned this this, this simic cc cache. We don't use cc cache because in windows there is no easy or I don't know that it's working there. um But the caching thing is when you have, for example, you have your binaries compiled and you have your performance wire or something like that. You can share this with multiple jobs and get parallelism for free. So if your tests, like us, run 20 30 minutes or something like that, so now we can't access the build old.

B

Let's get interesting, so I catch the build old results. I catch the build results, but it cannot access the build old.

C

B

It's overrated.

C

Using the same gesture and for last wins, you uploaded the cash.

B

So is it better than to use artifacts for this.

C

Yeah, mostly an artifact is typically on the. um In terms of definition, an artifact will be used for passing the result between stages and.

B

The cache is produced. This would be fine here, because then I can know that this artifact gets to every stage, because I don't want to keep this artifact.

A

If I add a new commit to the magic quest, I still want to have the original base commit being cached.

B

But this is done with the old job.

C

Caching is suddenly used, mostly if you need to, um instead of caching, is for your dependencies, mostly.

A

Yeah, I was thinking of caching the uh the base commit for the match request. This should be cached the the build job which generates the commit. um This doesn't need to be cached, because this should always be generated.

B

Yeah because so I I use artifact and then I use the cache for the.

A

B

And then just I have to use it that way.

A

This should be working.

B

Because for me it's interesting so every time I write to the cache I I assume that the cache is and and and bucket, which is obviously our container. So every time I write to the cache is this a new container? If you just uploaded, I guess that it merged the results together, or is it just in in in one run or something like that.

A

Now this this is, this is for your project for the pipeline.

A

If you don't clean the cache, um I think it stays there forever and if you like, rename something in your ci job and and output files, um the old thing could be lying around. So when you, when you see strange behavior in sci pipeline, um it's better to clear the cache beforehand,.

B

A

Before you do some more testing.

B

Because I think about, I think we could skip this as a whole, ah not the check api, the the build old stuff, because when we, when this works this way, as you mentioned it, then I would copy this into a directory with this hash stuff.

B

And then I can use the same, build job to get both.

A

I think it's it's hard to understand, then um so, right now it's like it's. It's.

B

Easy to understand, but we have to type in.

A

Yeah, that's true. um Probably you can optimize it with like some extensor templates. um I think it's from a from a development perspective. It's really hard to go there and say hey. You have either this state or this state for the job, because we came here from this long flow chart. um I think, for debugging and for for getting things going.

A

Two jobs are not that bad, especially because you can define the artifacts, they don't hurt if you let them expire in one week or something, um and you can like download the artifacts and do the stuff yourself or if you are like developer and just don't trust the ci system for some reason or you just want to reproduce it on your own system. You can just download that.

B

It is it possible to get when you're going to have some artifacts. Can I just use the artifact system to get the specific one from where the api, via the.

A

Api yeah, but the problem is, then: you need to write curl in your ci jobs and while you can do it and reuse certain variables for the authentication, um I think it's it's not not the preferred way. I would it's.

B

Possible but not recommended so.

A

Yeah, you can, you can go with, go there and abstract the boy and say hey, I'm relying on the artifacts. The problem is um if, for example, and the policy for exploration is, is there um the artifact might be gone and you still need to regenerate it so um depends um so.

B

A

Could store after yeah.

B

The pipeline seems to work so I have something pushed here. So what I do now is I change. I make an api break, so I just use this class and the just make into double. I hope the api detector will detect this change. I never tried it myself, so be a good c plus plus programmer until the compile it is a double, and that way it should be the same, but it should break the api.

A

Yeah it should it's public yeah.

B

It should be great- uh let's, let's uh switch from ins to double and in theory this should detect stuff and um gitlab metric matrix metric yeah. The metrics reboot, which I mentioned before uh this is cool.

B

It's just it's a premium feature, but the cool thing is, then you can just use the repot feature in the git clip ci and if you convert it easily, so you always see it at the top of your merch request, which I presently would prefer when I have the premium tier, it's a little a little trick around, because I would found it nice that I can upload an artifact which is just in text file in the core version and to display it at the top.

B

Because then I can use or, for example, you can also use markdown, and then you can pin it at the top, as it's done with the unit tests and all this kind of stuff and.

A

The thing is end up: enterprises like convenience, so it's more convenient to have it in there, and so it's currently it's a premium.

A

What I would do is, for example, also like make a comment, while the api, um or just like, use some different methodology, which which can can become handy here.

B

A

B

Bots stretch the first comment from the bot and then replace edit. This thing, so you be in the first comment of the bot, for example,.

A

I think there are like different ways to roam again. um You can make it work, um it's. It might not be as beautiful as it might be. Look if you're, for example, using a metric report with premium, but in the end um you probably want to see a table and the table is.

B

Interesting, it detected the problem, but why it is not uploaded.

A

You didn't generate the matrix.

B

Warning so why is it doesn't detect this.

A

Change which artifacts did you define in these in this job.

B

uh Just the api folder, which is generated.

C

Exit with one and so the job will be whole cancer.

B

ah Yeah the api checker tells me that there is a problem.

A

B

A

You need to capture.

A

Do you want to ignore the error yeah? Okay, yeah? You want to ignore the error you just like your thing is collecting stats.

B

It should it shouldn't stop here, because it it process it. Then it tells me, uh let's upload the artifact for a field- job, okay, yeah, so how to the best way to to ignore the return code in bash.

A

Like like niklas said after at the end of the file or at the end of the line, two pipes, true has.

B

Ignore eba compliance checker return, code, error.

A

The thing is, normally, you should like capture the error and do something about it maybe later on, but then in this case it doesn't make sense.

C

Yeah, but we could also do that if.

B

You, if you, if you have a hard policy, you can use this easier, because if you have a no abi break policy for example, then you can use this to to to break the pipeline. So.

A

A

You can you should define a variable where you capture all errors, um so, like every line is captured in an error state and then in the end, you evaluate whether the error is different to zero or just exit. As as the the error code- and this is the this- is the specific scope where I would write a bash script.

C

Or do a simple approach? Writing it into a file and compare the file.

A

C

Yeah, instead of piping it into some variables and passing it around.

A

Sometimes I would wish we have a framework for ci scripting, like embedded python in go.

C

C

But please don't give us any dsl. I see the hell that some people are writing their whole own programming language in this dsl.

A

Yeah, I think one of the requests lately- and this is a total legend request- was to use for loops because you could generate chops out of that. I would totally love that, but I don't see fit in the configuration language if you want to go by that, you need to like invent your own domain specific language, but.

C

This channel will be invented because what you can do wait. Let me check, I have something: oh no, no.

A

C

A

Build doesn't doesn't support twists or with items.

C

No you, you can do it with stripe pipelines, yum.

B

So, as you can see, we found the I think I have no uh highlighting at all in in the stuff, but I I would guess I can just use normal html stuff to highlight this stuff, because yeah.

C

And also, I would write it here.

B

So it definitely worked to detect and then the interesting part is is it was the cache used for that case?

B

Was it cached or is it built yeah? It used also the cache. So the cache also worked.

A

Okay, perfect, so um actually can't just use this um alex uh project.

B

Brought straight to production.

C

My last link, you can see the generation of a child pipeline how it works.

C

We've only declared with only one for loop, um because this is a repository that I found really interesting using four generating this exporting this as an artifact and then later in each drop. The artifact will be included.

A

C

A

Yeah, since you can generate the armor from existing yeah.

B

Oh, this is possible.

A

Yeah that's this is what you've been using with parent track pipelines.

B

Yeah, of course, of course, yeah, but.

A

I thought I thought.

B

So it's the same strategy which is building, uh but it's not a yaml direct feature. It's a github pci feature.

A

It's very special actually so, when you, when you scroll into line 38, um this is bash.

B

Yeah this is bash. Okay, you just produce yeah. Of course, yeah. You can use anything to produce the yamaha that you use yeah. I just.

A

A

It's a cool way of doing it. If you are like bound to, maybe you should write a blog post about this follow-up. It looks interesting um if you just write a blog post, please, um let's.

C

It's not mine, so I found this only because I was looking how I can build monorepos before docker images very easily and maintain them.

A

It totally makes sense it's the only thing which worries me. A little is the stages definition.

C

Yeah, this is a problem because um you can't have dynamic stages in the club.

A

Yeah, okay, but I see there's nios inside that looks interesting.

C

Yeah, I think this awesome guy that um he's doing some monitoring or linux stuff repository is very well written at everything. So.

A

It also uses an audio sql, which is like an old uh ui for configuration stuff.

B

A

It's interesting.

A

C

A

One thing I wanted to share is: um I'm also copy pasting it into the zoom chat. um This was an example of how I singer uses c cache, basically just like defining path for c cache. Configuring c cache that it only writes the specific local project directory.

A

This should be hidden somewhere else, um and then you like copy over the c cache into the the actual cache, oh okay and this space. So when you so see, I think the default for cache is two gigabytes. You can extend that by configuration with some environment variables. I think, and the whole thing is being done in the in a docker-based image which has all the build tools and cache pre-installed um and if you kind of extend c cash to, I don't know eight gigabytes or something like that.

A

um You can also build big projects which might generate or exceed c cash all the time, and I think, when I started with c plus plus, I was looking at the compilation procedure for 20 minutes um and every time when I changed something and with c cache, you only like c cache detects when you touch the file or modify the file and only compiles uh the the object file, which would be generated out of that from the lip from um a specific shared library or something which is defined by the cmak file.

A

um And I think, when I didn't change much in in the system, except for a header file which is included everywhere. um The build time was like one minute. Two minutes: oh because linking linking is a different story linking takes longer than yeah, because you need to like link all the objects files into an executable binary afterwards.

A

But there are also tricks to optimize that different idea. um One idea which we actually did was um we kind of merged the source code of of all files in a library directory into a single big, huge cpp file, which then was compiled and generated. Just one object file, and this was then linked together. So instead of linking 500 object files together, we were just linking five, which is faster.

A

It consumes more memory, so the windows box might run out of memory then, um but in the end, if you compile lots of source code, you need to find optimizations yeah.

B

Definitely so if you, if you uh notice, um hopefully I find it now, there is the h hacker. Oh god, it's not direct for.

A

B

It's not it's, not. Google will help me. Hopefully yeah. There is for faster building.

A

The hitchhikers oh yeah,.

B

So there are multiple versions of that, as I know, but this guy mentioned everything which you can do in c, plus, plus to optimize in different compilers.

A

B

A

Xkcd comic is like this is story.

B

Of my life yeah definitely definitely.

A

Why is michael active on twitter? Why is smartly active on on the community forums yeah? The code is compiling.

B

All this compiling, so he he also. He also compare everything and it depends. But if you have a modern tool chain, uh unity builds is awesome, as mentioned that you put multiple files into it and you get you need more ram, but but the the the speed up is really big and he mentioned every tool for fast build for from for windows, that's more or less the cc cache for for the res other systems and yeah inline notations, and what? What not? It's?

B

A really awesome short slide about everything you have to know about performance in cnc, plus.

A

Yeah- and there are several like techniques to optimize the code, but the code quality is not that good afterwards, so you always need to like decide which technique is. Is a good one.

B

Yeah definitely, and but if your project gets bigger and bigger, there are also better ways to structure your code, so that then your dependencies, you have to rethink the whole thing and make it more modern or.

A

Use interfaces.

B

Or something like that, a lot.

A

One thing we encountered this previously: everything was a library loader um and you would load like shared libraries into memory on demand um and if you don't use a my sequel back back end, you don't load the library on startup and then there was a problem that the object lookup between the shared library files in memory was fragmented or was taking too long. Because you kind of you executed something in lib remote. Then it called a function in the base and the function.

A

The base called something in lib cli um and then it was jumping back and back again. So, like the symbol, look up to quite a long time, um we solved that by uh creating a static binary which is linked together. um It's not super amazing from the deployment thing and also not for debugging and finding the right address, because the release build differs from from what you get in your debug build then, but still it was faster, and this was for performance reasons at runtime later on.

A

It made the build process yeah more complicated, but in the end it was worth it because the customer didn't uh didn't say that it was so slow. um So.

C

A

Definitely existing times.

B

A technique that we we use, uh shared libraries which have only interface or abstract classes and just forward every class, because most of us is point as a references, and this way you can really easily break the dependency hell from the implementation.

B

But it's a technique which you use when you have hundreds and hundreds of modules like we have.

A

And it's it's something you need to teach your team, especially when someone is new on the team and onboarding say hey: why don't? Why are you doing it this way? Yeah! It's uh some historical reasons made us. Do it it's documented here and then you keep going.

B

We've just built a tool around it, so we the tool, teach the person how it to do it right.

A

One thing guna, and I uh did um I think it was in 2013 or something like that. So with java you, you automatically have together and setup functions for class members um over class functions now now for class members actually um and in c plus plus. You need to write together in the setters by yourself um and since we had objects with like 100 attributes or 100 member member attributes, it was like yeah. Can we generate that somehow?

A

Yeah, probably we need to generate the plus plus code um and since we kind of invented a configuration dsl for for I singer, 2 back, um we thought well.

A

We could use the same uh definition with flex and bison like passing something um into into object code um and we created so so so-called ti files where, like the class, was defined in an abstract language with all the attributes, and then it said well, it's a state attribute and it got a specific different flag or handling or something else like a meta language and this generated c, plus plus code.

A

um So before the entire compilation happened, uh the class compiler was actually generating uh the class c plus plus code, um and then everything was merged. Together um now explain that to someone who is new uh to a singer, but um a professional c plus plus developer yeah, or maybe like starting with c plus plus having read a book about c plus, plus and best practices, and then oh there's a class compiler yeah. What's that that is not c plus plus yeah. But it's like lazy lazy developers.

B

So it's the same thing which cute is doing with, with the whole mock steps and all the stuff, but the in cmic. They hide it completely now, so you never have to teach that it just works.

A

For us it was more or less like if I want to, for example, so it's support for tls. um As a configuration attribute you need to like define, enable tls as a boolean, then you have a string for uh to find a certificate and whatever- um and I said yeah, we said to our contributors here- look at the existing code- it's just there yeah, but this ti thing do I really need to like add one line and then everything works, and then I call the getter in the in the funk in the in the workflow.

A

So yeah just try it out yeah, but I don't believe that yeah just try it out. It works. It saves us time to look at compiling and do something else, um but yeah in the end. I think it saved us multiple hours and also removed errors from did you define this scatter, etc yeah, but the api is not compatible or something else broke. um The only problem was that the class compiler code was actually actually really ugly.

B

Generated code is uh code. Generators are most likely to end up ugly, but do you just generate the header files and you.

A

Know it's just: it's also the classes, the implementation.

B

A

um When you write your own, your.

B

Own, it's like typescript, so you write the overall stuff and then completely get generated out of it.

A

Yeah, I don't have the code, I don't have something to show you, but just imagine that, for example, maybe you know boost signals um when, when someone says on active changed like calling the function or like changing the attribute, you want to trigger a function in the in the back end um like someone creates a new value and you want to subscribe.

A

This is with a round robin slots um subscribe to that event and write it to graphite or to influx dp or something else, and we kind of used it everywhere. So anytime, an object attribute was implemented. It automatically generated these boost signal events in the background and specific other things like activate deactivate things around this, and um I think the whole cluster is active and load balancing system actually depends on that.

A

So it's it's super complicated and super complicated to explain the only thing is it generated everything in the background and there's a fresh contribute to the project. You only needed to know that there is like abstraction lots of extract abstraction and you just copy paste from what's there right. This is what I what I would call myself in the end. I'm a copy copy-paste developer, um I'm using best practices.

A

uh What what already has been there, though it's it's it's joking, but um in the end we kind of created our own c, plus plus library um with within, I think i2, and it made it super convenient for us.

A

The only problem is that the current maintainers need to learn that, um but yet.

B

A

B

It's a more learning curve at the beginning, but it saves you a lot of time over time.

A

And the thing is, they can always refactor it and they will probably do it. um If I see code which, where I think well, maybe we shouldn't use go to in that regard or something else you could refactor the code, make it more like error resilient um and then it's just like someone else approves and they merge it and then okay and they generate new bugs. Probably, but that's not my game anymore,.

B

So we have a acute code base and we use python qt, it's not the most popular library in the world, but it's also generating code for for python wrappers. With the cute object model, then you have an embedded, an embedded system and the abstract away the stuff. So you can write xml files to inject code or tell the system how we should transpile it.

B

And then you get a lot of c blast bus codes which you build then, and then you have an integrated uh python script, our embedded python integrator with your cute code. So it's the same thing, but we just use it for off the shelf. So building their own dsl is so, do you use any specific technique, or do you really make it completely new detail.

A

So the thing which is implemented in there I need to like I need to find it.

C

Okay, I need to take my chance. I will drop off um we'll see us next week. Probably I need to grow bye, bye, bye,.

A

Bye, um I think it was just using what was it using the tools.

A

Oh, we also embedded the config into the c plus plus code.

A

The class lex and the glass parts, um let me just get my screen now.

A

So I'm currently like in this project tools and class- and this was basically the definition. This is the actually the the token puzzle.

B

um You mentioned flex bison, so yeah.

A

Flex, python, so it basically like passes and tokenizes the whole thing um where it doesn't know what what it actually needs to do and.

A

Yeah different statements and it's a sim, it's a simplified version, basically of um of the actual uh config compiler, which is in lib config, and there is the config parser, the config lexa, um because this is like lots of things in the dsl which which are defined so.

C

A

It was not there in the beginning, it was not that big, but you like from like uh defining the scope and the workflow from an if condition you go over to like wild while loop and functions and everything crazy things in there. So we kind of reinvented python javascript php, somehow into our own dsl, and it was fun.

B

So is there an example from the from the yeah from the dsl.

A

From the the dsl itself, where is the configuration.

A

Well yeah, this is basically do you think that dsl looks like where is it come on plugins, for example, um it's like a template and an object and and attributes um it's similar, how how nazis configuration looks like, but it's in the whole new format making it making it like. You pass it and you you um can use macros and things like the macros with the double uh dollar signs. The thing is: oh, we could use hangman.

A

One thing we did- and this was I think the year when we implemented it um so kunda implemented it. There was an irc port um and the rsc port was written with a tickle script and within the zener dsl you can program, you can define functions, um you can like define it in a specific scope. So hm is the scope and everything inside lifts there.

A

Needs a global variable to, of course, and then you can do things and one of the ideas to showcase this was there is a console like the the python rapper and ruby and things you type something, and you get feedback um and you can play hangman in the senior two console. um Hangman, output, nine, okay um and the fun thing was um that there also was the possibility to wrap this somehow in a tickle script and the roc board was responding to it.

A

So you, in the background, it was talking to a singer somehow, um and it was fun. um Gunner is really mastermind what what? What was possible inside an example for the ti files- and this is the interesting part could be. This- is the implementation for the mysql backend.

A

This is basically how it looked like, so it was somehow the plus plus code like we're, using and and also interrupting things, but then um it generates different things. So activation priority is like configuration order, uh activate um object, starting things because it influences some things. It's a weird.

B

It's interesting because there isn't as well as faster now there is an proposal for c plus plus that you can make such things happen, I'm not not sure which state it is. But this is something like the meet the compiler from cute, but for dc, plus language itself.

A

I think we invented that in 2013 or 2014.

B

Steve standard is not that fast.

A

We started adopting the c plus 11 standard in 2016 2017, because the compilers on centers five now center s5, was dead. When centers 5 was dead, we could actually use certain things, um but yet more or less. This is all config. So this like defining this is a configuration attribute, um but, for instance, there is the single application class. This also has states for things and, as you can see, there are many ziploc cpp files. um This is also an object, and this has oh, this hasn't stayed there.

A

um No a host host object has a state um yeah, for example, a host can be up and down, and this is a timestamp which is stored as a state, and this kind of um added a boolean get a boolean flag marking this, as we need to save this over restarts. So there is like a cache file which is written. um If you shut down the application, this file gets written and and the next startup um you read from the cache and know whether the host is up or down.

B

So it's it's a meet, uh it's something c-sharp.

A

B

Me a little bit like c-sharp because they used this a lot. The language.

A

And I don't really remember what to get it. Was it to specify or to deny it I don't know, but you could actually like write inline getters in there, um which means the display name. If it's, if it's empty, so we created our own string class, um we would use the object name and otherwise we would use the display name.

A

Okay, it's uh I've been working on the singer, 2 from 2012 to 2020.

A

um So it's a long time and to be honest, if, if, uh if I'm not working on it anymore, I'm forgetting things.

B

Yeah, it's it's normal.

A

um But yeah it was, it was totally interesting to build it.

B

But I can totally understand that you build an old dsl because it makes this thing more maintainable. It's it's and the problem see blasphemous can do everything, but it's like with any language. You can shoot in your foot really fast, so I think yeah, it's it's it's harder for for beginners, which start with the code base, especially when you have yeah one contribution from one person in three years or something like that. Then it's much harder but for your core maintainers, uh I would guess it's much easier.

A

Yeah this shouldn't. One thing I wanted to share is- and this is like the last for the stream right now- um oops- um so I've tweeted about this- so.

A

Actually should should be pinked about it, so they can try it out and everyone else um yeah, and um so I would. I would like to say thanks for preparing everything and thanks for the time um and let's see what beautiful topics we will find out next week, so bye on youtube, bye.