GitLab Verify Group, 30 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Additive CI Pipelines - Proof of concept demo

Description

Fabio Pitino - Staff Backend Engineer - demonstrates a proof of concept about Additive CI Pipelines as experimental feature.

PoC Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/413435

PoC Merge Request: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121764

A

Everyone and in this video I'm, going to demo a proof of concept on an interesting idea that I called additive pipeline.

A

um Let's think, first of all a scenario, a typical scenario will be: you have a merger request and that has a failing job and and now we have also some jobs you have executed before and now you want to investigate why this job failed.

A

um You have a fix ready for it, so you push the fix, and now you have to run again the whole Pipeline and imagine that before this test job, there is many many more jobs that have to be executed again.

A

So now you have to execute the pipeline all the time and another similar scenario would be when you are building a pipeline from scratch, or you add first job and then you you run the pipeline, then you add another job and then you have multiple jobs and subsequent stages, and but all that any the more the larger the pipeline it gets the longer it's also starting to get to execute this Pipeline and another scenario once again would be you're making a larger Factory into the to the source code.

A

And so you push your changes to merge requests and they have like lots and lots of failing jobs, and so you want to investigate and fix them by one by one and without having to run the full pipeline all the time so with additive Pipeline and the concept is and I can show how this works. So you have a normal Pipeline and, for example, you have a job, one that creates artifact. Then you have job to that is also successful and then job three fails.

A

Job 4 is skipped um if you want to run a a negative pipeline. It's basically a pipeline that is built on top of results of a previously completed pipeline, and and what do we do is like. We first create a new pipeline in order to trigger an additive pipeline at the moment during the POC I'm, allowing understanding this flag from the commit message like we do for CI and Skip, and also as a CI skip.

A

We also support push options, but there could be also a parameter eventually through graphql or rest API, that you, you pass additive um parameter and that's going to be propagated to the pipeline creation logic. So we have this new Pipeline and we have created through the pipeline creation steps.

A

um But before we execute the pipeline, we process it. We are going to look at what previously completed pipeline we have, and if we find one we take the the com, the successful jobs, only we don't take the failing jobs or anything that is hasn't been run, and so we only look at successful jobs and we copy some of the results over. So these results will be the artifact. It could be the traces and also job variables, since they are that can be generated through dot m variable, dot, m artifacts and it could be.

A

You know, you're running a specific job with some variables and they need to be carried over to the uh to the job from the additive pipeline, because what we want to do is we want to simulate that we actually run in the pipeline from this point again and but with the new commit. So this is like the principle of running additive pipeline, so we copy this uh results over and we started running the pipeline.

A

So at this point, these two jobs will be in created stage uh State and we run the pipeline processing uh logic and which means we're going to have job three eventually completing. For example, if the fix is successful or otherwise will be failing again and we run a new additive Pipeline and we will try job threes immediately and then, if that succeeded, then we run job four and let me show you on a demo how this works. So we have this failing.

A

Pipeline I was showing in the merge request, and this is the code that I have um so you have a build job. That's um that is successful after 10 seconds, you have a test job that creates an artifact, a date.txt data which the current date and it fails, and then we have a deploy job that uses the artifact of test job.

A

um So now I want to fix this pipeline. So what I want to do is maybe identify the fix here. It could be in a script. It could be anywhere this test job since was failing before it's all, always going to be re-executed, so I have here. My imagine I have this fix here and then suddenly, the next one job we don't know yet, since we haven't executed yet to simulate that I want to now say: Okay fix, test job and I want to run an additive pipeline. So I do CI.

A

I'm gonna push the change. Without this it's going to run a normal Pipeline, and this is not what we wanted right now, so I'm going to push the changes to my current branch and I'm going to want to have a look at the pipelines here.

A

Created already here, we can see actually the test job. The build job is already successful and we haven't even run that and then we just run the test job right now and let me open refresh now we run we're actually running the deploy job and this is going to fail.

A

Yeah, okay, so we completely bypass, let's say the build job since we kind of took the results from previous build, and this actually is the is the artifact of the previously executed build it's not the actual artifact artifact here that was created here, we're kind of simulating this was running and we are running again the test job and the deploy fails. So we created the artifact they're going to actually keep this job open.

A

Since we have this article here, we're gonna need for later, and another interesting thing is that now we have this pipeline being marked as additive with this warning message, the speculate results are based on a previously successful jobs and now, once again, I know what the problem is now I want to fix this now, so I fix this and I'm submitting another fixed, deploy.

A

I want to run others additive.

A

um Okay: let's have a look at the merge request.

A

Now you see this I've been already successful, so we are not running we're going straight to the deploy job uh and let's have a look. What happened here.

A

So since this test job to create an artifact that is consumed by deploy job now, this deploy job runs correctly, but the artifact is using it's not from these exact instance of job, but it's from the previous instance of job, since we can we copied over the artifacts, and so what we are doing here is actually, if you check this timestamp 947.32 is the same as 94732, so we're actually using that artifact. This is kind of to show um how um the artifact passing Works, which are going to go now in details.

A

um So you can see how interesting this could be if you want to.

A

um If you want to investigate like a fix and then of course, if you want to go back to um making another change, just making like a silly change here at the moment and I want to push a commit, okay to run the whole apply and not come. You are running any identify plan at this point, so if I push now, I can have the full pipeline being created so refresh this page, and now we have the full pipeline.

A

So it's not additive anymore, um so I basically use this active pipeline to have very fast feedback. Loop go straight to a green Pipeline and now I want to run the full pipeline to see if the whole pipeline still makes sense and is still successful.

A

um So a few notes on this, of course, there is some something to still to discuss. The good side is that we are getting very feedback loops. This could be interesting if you want to again fix many jobs and very quickly or you want to build on a pipeline that you're running and that you are creating for the first time and and you you're gonna run only the new jobs or the failing jobs you had.

A

So all the successful jobs are gonna be um ignored from a from a pipeline processing perspective, so very fast, fast feedback loops. uh It again can be used to build pipelines iteratively you can save a lot of computer units and, like previously known CI minutes, and since we are not going to run all the pipeline all the time and the bad part is that additive pipelines cannot be trusted since they we are copying. Artifacts from previous jobs, we are doing some simulation there and it's not the the source of Truth.

A

uh So that means we can't use additive pipelines to learn, merge, merge requests, so they we need to have some logic in the merge requests that since If the previous, if the last pipeline is an additive, you cannot merge. The merge request- or you cannot set merge when pipeline- succeeds um because of this reason that the results cannot be trusted and the Ugly Part at the moment again.

A

This can be something where we need to work out some abstractions, uh it's the artifact reference I'm at the moment, I'm not coping like duplicating the artifacts, but actually referencing internally, so um the artifacts from a previous job. So it means, if we ask a job, tell me what artifacts you have it's not going to be the copy of artifacts that were that belong to that jobs, but actually using the artifacts from a different jobs.

A

There can be some some problems with this, like uh since artifacts and potentially can be owned by multiple jobs and an artifact can disappear because it expires and kind of expired for multiple jobs.

A

At the same time, and it can be deleted for one job and suddenly it can disappears for another job, but um identify plans are supposed to be used for a very short period of time, scope for um to reach a very specific goal of get to a green pipeline as fast as possible, and they are not meant to be used for audit Trails or things like that, um and then there's like a bit of a small technical that where Trace artifacts are not always propagated, because there is like a two minutes delay currently that this needs to be fixed in the code base.

A

But it means that if you run a pipeline very quickly, a new additive pipeline very quickly, we might not carry over the trace artifacts. So you might have jobs without traces. um I think it might be an example here, for example, this build Maybe, and this has it.

A

Maybe this builds yes, this doesn't have an artifact because we tried the the active pipeline very quickly, so I didn't have time to actually carry over the um the artifact.

A

uh What is called in the covert is primarily how the Cisco is supposed to behave with parent chart pipelines and there's some ideas like how to in uh propagate additive through child pipelines from the bridge job, but it hasn't been explored as part of this POC. So this is something that, like a little bit more, um if you have any questions, please let me know I'm going to link the an issue for feedback on this video and um and yeah. So I hope you find this interesting.