GitLab Verify Group, 27 Feb 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cross Project `triggered by` discussion

Description

The GitLab Verify team talks about the plans for adding `triggered-by` to the GitLab CI yaml to allow downstream projects to automatically build when upstream dependencies build or are released.

Spoiler alert: part of the discussion is around the name `triggered-by` changing 😃

Join the discussion here: https://gitlab.com/gitlab-org/gitlab-ee/issues/9045

A

Just takes forever, it goes to zoom, and then it and then it eventually gets removed from zoom when it gets uploaded to Google Drive. But it does work you just it takes forever. Yeah.

B

I started recording to the cloud, let's cross, our fingers that we'll be able to find recording later. Thank you, everybody for attending I realized. This is really difficult for some people, because it's really early in the morning, hello, Brendan, so right off the bat I want to talk about what what is this meeting and what we need to do in this meeting. So we're here to discuss the minimum bottle change for the cross product triggered by implementation.

B

The end goal of this meeting is having an approach defined for implementing this, and that approach does not have to be a finalized proposal, but we need to know which approach are we going to take at implementing this because it's really late in cycling? We still didn't, don't have a general general overview of where we want to go with this feature, and we don't have the approach figured out by the end of this meeting. Then that puts this cross project this tolerable for 11.9 at risk and it's basically a direction tolerable.

B

So we probably should focus on not missing it and I. Take a look at other. The deliverables and I think it's medium profile, so there's some community attention, but not a lot and I have a link in the issue and basically there's two approaches that we talked about so far. One of them is adding, so just so we're on the same page. This is basically for adding implementations for adding a section in the downstream pipeline llamó that defines that that pipeline is triggered by an upstream pipeline in a different project.

B

So there's two ways we discussed about going so we discussed the approaches furthest. One of them is adding a special stage at the end of the upstream pipeline, with all of the data stream triggers. So it's one stage at the end of the pipeline that triggers all of the downstream pipelines and yeah.

B

The other approach is just adding a trigger job at an arbitrary location anywhere in the pipeline. So.

C

Can I I think I can shine some light on this, because I feel and as well as looking at your latest comment. I feel that there is some miscommunication, or at least we're not aligned on thinking patterns here.

C

So, let's, let's first off define what is already inside the application right now right. We got triggered so not functionality that this issue will implement, but we got the cross pipeline project cross project pipeline functionality already in there. It currently uses the triggered keyword and it has it from that implementation already. It displays both up streams as well as down troops right and how I thought of this, and regardless of how many problems it might introduce.

C

Potentially, my thinking was here that we could reuse that existing functionality in there, though the direction that the triggered by keyword functions as this fight. This is the other way right. You define this in the downstream instead of in the upstream and at.

D

The trigger, whatever I, think that we are missing one important piece of a puzzle and that all existing code we have, and these two approaches we currently started, using triggers key wording in lab CI Yunel and triggering pipeline using CI job talking through the API. Both solutions, both approaches, result in having a multi project pipeline, are a little different than triggered by because in the former former case, this is always one to one relationship and what we are trying to solve here is one too many and that's a little bit different.

B

Yeah, especially because, depending on where we put this so if we know that we're gonna define they're triggered by section in the CI ml of the downstream project, but in the upstream project, we're not really sure where we want to put this job depending on where we want to put this job, we might not rely on the bridge implementation that we have were triggered at all.

B

We might have to do something different for that, which is why the approach actually defines the implementation and the implementation defines the approach, which is why we have to discuss this beforehand. So if we go with relying on what we already have we're kind of forced to put all of the trigger jobs at the end, the top stream pipeline period like if we rely on what bilberry have I, don't think we can put them anywhere in the pipeline. I.

D

Don't think that using pre jobs or trigger jobs, however, we called it. It makes sense because it means that a downstream pipeline is going to modify the arms upstream pipeline and put some arbitrary job somewhere in the pipeline and Wiley perhaps makes some sense, because then we have a connection between the job in the upstream pipeline and a job in the downstream pipeline.

D

Putting some job into the upstream pipeline means that we need to calculate the status of this job into the status of the pipeline itself, so I mean it's know, it's a little tricky because we are putting some arbitrary job into someone's else pipeline and we having a state, was there. We got a job, it's a status in the backend code. It's even called the comet status right, so it means that we are putting some status into someone's else pipeline and how do we treat that? Do we change the building status?

D

Well, now we're down to Python things so.

C

Perhaps an idea right what? If we skip the trigger job entirely, but still display the upstream pipeline that triggers that downstream, in which the triggered by job has been defined. So it won't be visible from a UI perspective, but it will be active from a functionality perspective, I'm.

D

Not convinced it makes sense as well, because I envisioned this feature to be extremely useful in case of, for example, a popular docker image that someone might want to rebuild.

D

Rebuilding all the down straw downstream observers might want to rebuild their images as well when the upstream is gets updated or perhaps there is some very popular library that gets included into your software, and you want to always keep your software up-to-date and, for example, update your software automatically when the in version gets released, using a pipeline, for example, which means that we might end with hundreds of downstream pipelines watching the one very popular upstream project.

D

And does it mean that we want to completely clutter the pipeline view for some pipeline that look for someone who is an outer or very popular library, for example. So.

C

I have one shot towards that, and that is the fact that I feel that we're mixing and matching release management with continuous integration and the use case that you have right that you want to automatically. You know.

D

Yeah I get lopsy is also CD, so severe.

C

D

So it is, you know, related to means. Yes, then.

C

I understand, though, I want to make clear that you know we the anchor points of triggering a downstream pipeline from like and I'm thinking here from an ideal ideal situation right, not something intermediary. But so when do you want to trigger your down streams or at least other projects that depend on your, for example, your package right or your darker image, etcetera? That is when you release that image, you send out a release. There's a new release that triggers all the downstream projects at reliant. Well, yeah.

A

Yes, but that, but it's the inverse case that this is I want to trigger my pipeline. You might not even know about it right when you release, and so that's why we have this like concept of on right, so I might say not on every pipeline. I might say only on tags right only when you tag your upstream thing. That's when I want to go online right.

C

The the thing that I'm thinking here is there's a difference between internal use and external use, so internal use, it would be make sense to triggered things based on the repository right, but from an external use case. It makes more sense, in my opinion, to trigger things based on release. Management am I completely wrong here.

B

The biggest problem here is that we're kind of changing someone else's pipeline from the downstream pipeline, like that, doesn't really make sense if I define something in my pipeline I'm going to change someone else's pipeline doctrine pipeline is going to be changed by my change and that's progress and.

D

That's very interesting because in software development we do have this concept called inversion of control and we usually introduce or invert the control when we want to decouple something, and in this particular case it means that we not necessarily want to tell the upstream pipeline owner who is using the downstream pipeline, because this is the decoupling we're we can can have it in this particular example. This is actually quite similar Forks. So when someone Forks your project, you might not be really.

D

You know interested in that, especially when it's some popular open source library with hundreds or thousands of Forks, but sometimes you might want to see who actually fought through a project, and this feature is usually quite hidden in github or github. You need to click some button to see. You know what what the forks are and I think that perhaps the most simple solution here is to just forget about showing that to the upstream pipeline, maintainer or upstream project maintainer and I.

C

Would be fine with that to be honest, I'm, more inclined like hey, we need to show something on the downstream, because that's where you configure it, you.

D

C

Which pipelines, for example, right you can still configure multiple triggered, buys in your your CI right exactly.

D

Wanna know which one triggered it exactly and you need to know which pipeline triggered your pipeline and when and on what conditions. It.

E

D

Be quite difficult to visualize that using features we already have, because this is one to many and not one to one relationship so.

B

Another like thinking about this, where we have some dependence that the rely upon others in shell scripting, you have a utility called watch. So that's how you depend on some change and then, once that change happens, you can do something else. The reason why that works is because it constantly monitors the thing that's supposed to change, and in our case we can't really do that. So that's the problem here.

B

It's really where the triggering the pipeline needs to originate. So how do we know that the upstream pipeline finishes it's going to trigger the downstream pipeline at real, really carefully without changing the upstream pipeline? The only way we can do that is by constantly monitoring if it happened and that's not really something we can do so.

C

We are now talking about the more like the functionality aspect. Right y-you want impacted, as it's.

B

Completely functionality and definition of the feature is kind of tied together. You can't really it have you think about the future changes how you implement it. So it's kind of we're talking about both of those things at the same time, because they rely upon each other yeah.

C

I'm thinking of like what is the impact on front-end and what is the impact on backend I bet that a change right I have.

D

Some ideas how to implement that on the back end and some ideas about how to reuse existing front end solution. So perhaps I can show that if you are interested so.

C

D

I think that we currently do have three jobs implemented right and we currently have all the front-end implementation that we used to show the connection between a job and the pipeline trigger. So it might make sense to actually use three jobs indeed like, but not in the case, in the form that it's quite intuitive for us, because it's quite intuitive for us to place a brief job in the upstream pipeline, because this is how triggers by works it.

D

It's now going to this direction, so I mean breed job that triggers the downstream pipeline is inside your upstream pipeline and it points from upstream to downstream like its natural direction, something that we understand. However, in this case, we are inverting the control, which means that perhaps the bridge job should appear in the downstream pipeline and point another direction.

D

This way, we can also take a status of the upstream pipeline into account into our downstream pipeline, because we have the status attribution feature on our roadmap and we want to make it possible to configure a bridge job in a way that the status attribution works right. Do you guys know what what the issue about stops attribution is it's the I think it's we do have that on the roadmap, it's about waiting for the upstream pipeline downstream part and to finish anyway.

D

So so we can use the bridge job to actually and create it in the downstream pipeline when the upstream pipeline gets triggered gets created, and we also can contribute in the in that bridge job the behavior, whether it's going to have status, stops, attribution or not, and we can reuse existing bridge up. It means that we would need to change the syntax of our future proposal because it's not going to trigger by, but it's going to be yet another job very similar to the job that has triggers keyword.

D

But in this case the the keyword would be different and we can have multiple jobs like that, and that's probably the most simple way to reuse, existing implementation and back-end and front-end.

C

From my from my understanding, you intend so first off we're not going to show any down streams in the upstream pipeline. We are going to show the upstream pipeline in the downstream pipeline and the job that triggered truck from that upstream right. However, there can be multiple of those trigger jobs. You say because it's the downstream, not the upstream right and we'll what I get from you I think is. That will only show the job that actually got triggered yes, and that makes sense to me as well.

B

Let's leave status attribution out of this discussion for now, because it won't be easy to build upon what say we build this, but we doesn't.

D

Actually, in this case, the status attribution code, implementation is going to be exactly the same, and this we could reuse that, instead of implementing another solution for status, attribution in case of triggered by because we are, we would use the same pre-chopped mechanism.

B

A

I think it's fine as it is today, there's a lot of work. We have to do with status attribution as Berger's mentioned, like we've got a issue open about deterministic status and a couple other things too so, but yeah obviously fixing the world of that is out of scope. For this.

B

So I think we kind of aligned on the idea of having jobs in the downstream pipeline, pointing towards the upstream pipeline kind of inverse of what we have in the proposal. Currently.

A

Yeah, you know always the kind of stage right.

A

Sorry I don't have a stage right like then, then you can have like a triggered State triggers stage. Basically, is my thought, like I'm, going to change the proposal right now. That I think reflect what we're talking about. So what I'm, just ones that kind of know we'll have that.

C

I mean we could even say the current situation is that from a upstream pipeline that triggers the downstream, so another thing that we're now implementing the job can be in any stage of the pipeline regardless. However, now we're looking at the downstream right, it should be one of the like the first stage optimally because it always gets triggered.

C

D

A great question and I don't have a MIDI to answer how this could look like. Yes,.

B

Both of those approaches have I.

C

Perhaps even would say that we don't even have to show the that job at all, because the upstream pipeline in the downstream pipeline detail view will be visible.

C

So we can keep the jobs fully invisible and it will just stream into the actual pipeline jobs that we already have defined. Just as we do now. Don't.

D

We have the issue about making the visualization a little more obvious in the way that we link jobs to the yeah.

C

I know and I want to keep that that out of this discussion very much but.

D

But this is quite important because if we decide to actually use jobs as a real bridge between pipelines, then the way that we actually visualize, that might matter.

C

I'd say in that, in that case that you know not showing something at all, is you know the safe choice and the minimal choice, and we can decide on something like that later when we get to tackle that issue, I, don't see it being a block or a problem for our implementation. Currently.

B

Yeah, just we were kind of running out of time, and we still have one burning question, which is: where are we gonna have a stage at the beginning? Where are we gonna have like a free-flowing?

B

D

The most simple solution is to, if you put the pre job inside the pipeline, instead of at the beginning, we skip jobs. That are I mean that are on the left right. How to say that in English you know what I mean we skip the jobs that should have been already processed.

B

A

That's simple:.

B

E

A

Yeah I think I agree with that. If it's simple to do, I wouldn't want to I guess the other thing I'm trying to protect is shipping right like and I think if we, if we ship- and we have a little thing outstanding like oh, we should skip jobs to the left.

A

I would be okay with shipping and then saying okay, but in 1110 we're gonna, you know, fix it, so I I would I'm just I'm just trying to throw my hat malingering that we should ship something better and if we've got a couple of UX and or other things we want to do after I would support putting that in the next release dish to ship, something you know stripping something rather than it just going.

A

Yeah I think I think we've hit all those really critical things that would make.

A

You know make a major mistake, so I think we should be able to assume that all this I'm she'll make you now work out and then I also updated the description that has, if everybody go, look at that real, quick I put the link back and zoom in case you missed it, but yeah. So sorry that was two different things one. We should take the path of least resistance on these smaller things so that we can ship and then to go. Look at the description. I think this is what Gregor's is proposing and everybody's anymore.

A

D

Also one additional concern: perhaps it's just something that appears in my mind, and perhaps this this problem is not clearly visible for us, but we do have pipelines on many different branches right, so we are able to define the upstream branch, but on which downstream branches do we want to trigger pipeline? Is that something that we consider it already.

A

So that's a good question. I would say if we considered it again, maybe not in the first generation. If we considered it, we could use the only syntax right like is it now? It's a job right so yeah and.

D

It's a critical job, even the only accept is going I will wait in the way that the brie job is not going to be created. We do not trigger a pipeline at all, so yeah really.

A

D

With putting this bridge atop bridge open to downstream pipeline, we can use all the features we already have to control the behavior of when we trigger the pipeline downstream. Oh well, you see so.

C

How would that function if the job is in the middle of the pipeline, so say the third stage, for example, I, would.

D

Say that the disparate job would have only accept and all the features that we currently support with pre job, and if our about pipeline evaluation mechanism tells us that this job should not be created in these conditions, we are not creating pipeline at all right. If it it gets created, then we skip all the jobs up to the bridge job in the pipeline stage, and you know, follow just normal pipeline processing.

C

I'm interested in what what use case is that my variable yeah.

D

For example, if you don't want to duplicate jobs, for example, you would like to have your downstream pipeline independent, but when it gets created triggered by the upstream pipeline, perhaps there is some, you know overlap into what gets tested.

A

This library is in some way completely independent from my build right so like say, I don't need to rebuild my application. I just need to make my doctor file again with it and my application. That would be in use case for it right, don't rebuild just repackage great, basically or don't rebuild just release. Yeah I, don't know how I don't know that that's the more common use case, but it is one that maybe yeah.

B

I mean as long as we have as long as we implement the MVC. We can build upon that and add different use cases. I, don't think this is especially different for the first iteration, but a weekend you know think about other used in the next generation.

F

I'm on board with this I think this is a solid, solid thing. Today, the.

D

Only last thing that I'm thinking about is renaming triggered by to perhaps subscribe or watch or vigorous, observe yeah. But you.

A

Know what's gonna I was gonna, bring that up mark actually pinged me I think on this issue.

A

Maybe rue was but mark. Puns actually came back to me and said: hey it's triggered by the right word. I was like you know what I don't know because I didn't write this issue and I just been kind of going along with him because triggered and then turned goodbye, but we should have problems. We did because we already.

C

Got treated by API key word, I feel or I believe so there's I'd say: there's good good cause to change it. Yeah.

A

Okay, so let's let's consider that, but hopefully we can.

E

B

There anything else we need to this guesswork while we were, while we're still on this meeting,.

C

Yeah I'd like to quickly define soul for Scott and me, if there's anything needed from you in order to like to move us to a solid implementation of this, because I feel we can reuse everything that is already existing in the UI. And this won't have like a lot of impact on our front-end right.

G

Yeah, as far as I'm aware, it should be just reusing existing components. It's mostly just determinating, based on the back in response. What what the she'll win.

A

Yeah I mean I agree like the night. That's another argument we keep coming up, for maybe we're got a little confirmation bias, but we keep coming up with new arguments for why we should do it this way. But that's another argument to do this way. Is the UI we'll just just work, because it's a job right think it'll work like the existing ones?

A

Okay sounds good, well, yeah, thanks for putting this together into you, which is really really valuable use of time, showing how a little bit of sync can improve on a lot of it anything. So thanks everybody thanks to.

B

Everybody for attending I'm gonna put the link to this meeting after I find it. So thank you and have a nice day, yeah.