GitLab CI/CD, 22 Jan 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Release Generation from API

Description

Meeting with Runner & Release team to discuss how to generate a release in the YAML.

Issue: https://gitlab.com/gitlab-org/gitlab/issues/39097

A

And then hi Darren? How are you hey.

B

Good morning,.

A

B

A

Guys have been up for like eight hours already, okay, so we're here to talk about our lovely release generation from the API issue.

A

The idea that we had before the break was that we would create a different mechanism for generating a release that wasn't from the rail side in the ammo file, but it also wasn't from the runner side. So we've created the sub issue of our parent issue, about release generation from the API, so I think, where we left off last time was Camille added a bunch of feedback with some code snippets about what we should be instrumenting and Steve.

A

You added some commentary and helped filled what we, what we should be implementing as our MVP, so when I wanted to get out of this meeting was understanding what parts should runner be completing? What parts should the release team be completing and what's realistic for things in 12.8 and 12.9 sound good.

A

Do you have anything else, Sean that you want to add to that or any questions that you want to dive into you right now.

C

No I'd be I'd really like to talk about the the code snippets because I'll read through them a few times, but there's still some areas which are not clear to me and maybe Steve. You might have some insights into some of those I. Don't.

D

Know you can maybe share your screen, understood.

C

Okay, could everyone say that yeah.

D

C

Mean you sure that it's at this comment? Okay, so yes, we've got. The the first part is done where we you know this job token. That's it's all completely clear to me and so then all right so now extend the runner. So there's a there's, a template to add an additional helpful command. So I guess my first part is I. Don't really understand why it's called a helper command, and why is it? Is this somehow it outside of the Rhino reserve yeah.

D

So I think you bring up a really good find out. It gives me really good how to say ammunition why I don't think it should be in the runner, because it's very confusing right, like you're saying what is this helper command like and it's not just you who's kind of asked that it's most people are gonna. Ask that because the helper command is just a command we have for the runner. That's like a skinned down runner binary that we use to distribute to docker images and things like that.

D

So we can upload, artifacts and and doing some other stuff, so, for example, using the docker executors, we don't just use one container. We use the container that you specify, but then we load a bunch of containers around it that are used to have get installed and like upload, artifacts and things like that, because in that way we can't assume that the user has all the software needed to upload an artifact. So we control that and that's why there's ticket LeBron or helper command that that's does that.

D

Basically, so that's the idea and I'd like to keep that like behind the scenes, or only technical people who know what they're doing so. That's why, if the comment below, if you look at the comment below I suggested like the drawbacks, if we have it and the hyper image and if that one or an aunt abandon CLI, so if we have it in the independent CLI, we will not have. Questions like you just are like okay,.

C

D

Is this helper commands doing like do I need to know any every command, because if we do release upload, for example, if we name it release upload, do I need to just know about that or do I need to know about upload, artifacts, compress, zip files, and things like that. Do I need to know about all that stuff. So it's like a bit of leaking abstraction on my opinion, but that's one of my suggestions on ways like I. Don't know what you think about no.

C

Well, I mean, first of all, when you guys are the experts first of all, and so you know entirely happy to defer to you, although I really would like to understand how it works, but I guess I guess sometimes really there's two things that are really completely unclear to me. Aside from that, so first of all, I mean we've got this.

C

What are we trying to do with this right because the example we've got here is it's just got like name and changelog right and so we're just going to create a very simple release with just the daemon and in the contents of the changelog. That's it so that that's how MVP is it? That's.

D

How I understood it so to reiterate how I remember it was being going to be done? It should be in this issue, so the first MVC would be. We will just have a CLI tool to make it easier for you to create references. Now we expose that CLI tool.

D

The users have to update their catalepsy, I am and file, and the script to use the CLI to now to see a lighter will have flags like name description assets, for example, and so on and so forth, and then in the next iteration you will define the CI ml file inside of inside of the CI m file and then transform that block that you proposed earlier so, for example, have the syntax you proposed earlier release colon and then all the information and then we'll have rails, transform that into the CLI command.

D

So, for example, when you have ray release, : name :, it will be transformed into release upload name thing: okay,.

C

So that that's the part where I'm a bit vague, so first, the first part of the new syntax is already it's already in production. Okay,.

D

C

Already committed to master- and you know it's just sitting there- waiting to be used. Okay,.

D

C

Then the part that I'm not really clear on is how how does currently how does rails and the runner communicate with this type of convey? Does it get arousal so.

D

Right now, it's not being pushed anywhere like the runner has no information about it. But what comments are just said? Is we have what we like? If you look at the jobs API, we have something what we call stats, I'm, not sure. If command mentioned, that didn't get issue or not.

D

C

In the in the jobs, resources, yeah.

D

So if you look at comments, comment, point number three. So the API is not documented because it's an internal API. It's.

C

Not documented okay, okay right, so if.

D

You look at extend definition of a step as seen from Runner yeah.

C

So this is all old, but this is all ranek taught code right.

D

D

Andreas, you can define a step. We do that, a few in after script, if I'm not mistaking so after script is defined as a step, and we have done. We run that step in the after-school. So yes, well.

C

Would this you know as a I'll have to look into where that is, but would that be something similar to jobs, for example, I mean? Is there? Is there some existing code? That's a bit similar to what we're doing we were taking some of the some part of the mo and then giving it to the runner.

D

The after script is going, but as far as I know so.

C

It would be doing it for that. The jobs no.

D

C

D

Not sure what's nodes are involved like I do not know. How do you mean.

C

D

As far as I know, the after script does the same transformation, so it takes a note of recalling its nodes and then it transforms it into a step and the response. So.

C

So but then it's so it's what it's doing is its transforming it and in presenting it in an API, endpoint and Narada makes a call and goes and gets it exactly.

D

C

Okay, that's fine. I can work with that.

C

So so, essentially from the rails perspective, it would be. It would be. This part here, convert them onto runner steps exactly and probably extend the API. This undocumented API Weaver that wherever that is and then make it. You know available somewhere to the runner. It's.

D

Or the API is already available, the structure is already there, and steps are basically just an array of commands that you pass in right, so you just have to append to the trade. Okay.

C

I think I mean once I find the entry point. I think there should be fine, yeah.

D

If you want to select D I can send you the file of where the API that's yeah.

C

That'll be awesome, so so so we do this transformation of the of the mo the runner. So the process would be then to run all the runner CLI or whatever it ends up being goes and picks it up picks up the configuration. Then it actually processes- and it makes another set of API- calls using an existing API public API to create the release exactly.

D

So the way it would work is going more into technical details. You define a step right. You know that will run on the image that you define because it to find an image first, and then we will take that step definition. The runner will understand it and say: okay, I'm gonna spawn up a container, for example, if we're using the docker container, docker executors, we're gonna container and run this command in that container. Now this command will pick up the CI job token automatically and upload it to the container.

C

D

Api first has access to J, okay,.

C

Right, okay, so.

D

It's like doing kind of like a docker run and then docker, exact and running the scripts manually and executed exiting the container.

C

C

D

Sense yep so from the runners side, what we would have to do, as we do not understand, understand the steps fully regarding the image part, so we would have to do that, but I think it might be fine without the MPC I would have to investigate fully how the steps are made to like give your salat answer. So I can do that. Aysen.

A

Do we need to create an issue for that Steven.

D

Yes, it's best to create an issue under on our board and I. Put some investigation issues: okay,.

A

You'll, do that part? Can you ping me on it when you.

D

A

C

Edit I mean how would we handle things like, for example, what, if there's no change log file right or it doesn't find all that doesn't exist or something had we handle errors that.

D

Would be up to to see a light? So if you want to fail on a if there's no change log file, you just exit code, that's something that's not zero and we will fail the job, for example,.

C

So that'll fail the job and then that'll appear. He failed in the pipeline exactly so this this. This will have its own step in own job in the pipeline and.

D

If you want it- or it can be in a other job, for example, and the build job at.

C

The end you want, at the end of the build yeah.

D

If you want it lagged out well,.

C

Because, when you think about, when should overlays be built, rather should be built after everything's passed, everything's grain rod yeah so.

D

Yeah, ideally, if I would be doing it like what we do would right now, for example, for the runner, we have our own specific stage called release stage, and it's just a bunch of jobs. That's uploading to us three package cloud and then a specific job. That's creating the release, object.

C

Well, that sounds like a pretty good model. Actually, you know it's on stage. Yeah.

D

Chris no Chris done we have, we can use that for it. So when the RPM packages are built, we just released their game packages and so on and so forth.

C

A

One clarification we do have the ability to support upcoming release creation, so I just want to make sure that when you say that it makes the most sense to create or release when everything's been done, people do have the ability to come to like creator release in the future. I do.

D

Not know that that's cool yeah.

A

It's called upcoming releases.

D

So that's even better, like.

D

If we have it in a separate CLI, we can have two commands right, create release and create upcoming release, for example, and this will be the tool that will automate a free every release- creation, for example, now going ahead with, if we should do it in on the runner side or on a specific cell, like I, gave my opinion and the comment.

D

If we're connect standard, of course, like with the upcoming releases, I I, don't think having upcoming releases would make sense and the our code base now the benefit we get as we have written get lab runner is that we don't have.

D

We will have some stuff already out of the box like picking up the a CI job token and things like that, but we can do that fairly easily as well right now by extending that into a library and have get lap, Runner and you're still CLI to import that library through, and this gives you another benefit right, you're completely responsible of the CLI tool and not well here and abandoned from us. Like you, don't know to ban.

D

If we change add a new flag, you don't have to go to the runner code base and add it and wait for a maintainer around our maintainer to review it. You just have a CLI maintainer and, like you, have your own priorities.

D

C

Absolutely, and in and as we as we extend this and start doing like we talked about binary assets and things like that, this CLI will just grow right, as we add more functionality, yeah.

A

And that was one of the questions that I asked. You know Camille if that was a feasibility, because that's what we would want to go and he he agreed that yeah. We can extend it, whichever direction I'd want to and.

D

That's why we would have like again that would be nice to have it. Indeed, it's on project, two cuz young. It can grow on its own.

C

And so does it so, given that we're talking about this, you know in this way, would that mean that would then become a new project that would be essentially part of the the runner team is that it.

D

Can be, it will be a new project, yes, and it can be either I, don't think it would make sense to be run. Our team responsibility would make.

C

Sense to be released, response release, okay, well,.

D

C

I guess technically you know we can certainly take it on, and but it would be great if I mean it'd be quite if I would be appreciated if we could take it on after it was more or less bill. Yes,.

D

Of course I do understand because, like building CLI tools is not an easy thing and like would go like the runner team, has a lot of experience, building CLI tools, because I think we've been like six or seven. Now at this point and.

C

Particularly if we're extracting parts on the brother, yeah.

D

D

So I wouldn't need to look at. Let me write some notes for this meeting so.

B

Are you guys suggesting that you want the wrong team Tbilisi I.

C

Well, I, I, guess: I guess: I'm wondering what the direction is.

C

B

C

Mean we talk, we're talking, I don't know, am I odd I'm not sure what the best direction is, but we we need to extract parts out of the runner right or would they be two different steps.

D

Technically, you can already import the libraries and their own libraries. It would be just you're, depending on the whole runner. Codebase is sort of just one library. So it's not the word add it's not I'm, not sure. If you're familiar how going for its work or not God.

C

Not my I'm, not really no, okay,.

D

So like, instead of like how to say on Rails I'm not too familiar, it writes so, instead of importing a whole big jam and then just using three four methods: that's what we have to do, but ideally just import one gem that has these three and four methods, for example: okay, that make sense yep.

D

But we can always do.

C

D

Everything it's gonna take a long time. Is that more clear? Does that bring up more questions and make sure that's comfortable? No.

C

It's it's definitely it's definitely clear I guess I guess yeah the question that Darren's raised about who's gonna. Actually, do it he's probably an important question to answer.

B

Yeah super important.

A

Well, Sean: what do you need to be able to do it because I think from from my standpoint, I want to protect the the timeline as much as possible and.

C

A

Much stuff that they're doing.

C

A

We do have a new teammate Jaime, who can.

B

A

Go and maybe this could be a go. You know project for him since its brand new anyways right, and it would give our team an opportunity to kind of Komine goal with with Steve to understand what, like some of the learnings but I, think the biggest risk that I see right now is, if we let run or do it, we don't really have control over.

C

A

Own timelines, we're kind of saying, okay, other team going ahead and instrument this, and then, if long-term, we're going to have to support it, it kind of means that were always beholding to you potentially asking another team. How did they build this.

C

A

The protectability of our scope, I, would I would be interested in seeing what we can do on the release side. I am a little nervous as far as timeline, though, because that means that we just need more time to figure out like what the heck we're doing, because we don't have experience necessarily doing like a CLI creation or any of that work.

C

So so hot yo, so new team member Jaime, you know I had a coffee chat with you yesterday. Actually so he's you know, he's got a lot of experiencing go, although he's only just joined good lab like.

A

C

Just onboarding at the moment Frank, and so you know so, the the other people who work with him see he's got great, go experience. I, don't know how to judge that myself, I'd certainly be able to. You know: I'd, be very comfortable working through the rails side and so I guess this con log free pass right.

C

There's the rails part there's the extraction of of the shared code from the runner into our module and then there's this new piece, which is the CLI which is also going to be written in, go, and so you know, yeah I agree. It probably should be kept inside our team.

C

But maybe if we could ask you to help with the extraction of the of the share code as well as kind of just maybe just be available to consoled a bit with the code- and you know you know know that we're going in the right direction, because I'm not ago developers yet and and ha mr. Billy just started. So you.

D

Know I mean at the end of the day, I guess it's up to Darren for me to like put my time, but I can act as kind of a maintainer for the whole thing like just guide you to the right direction and tell you that you're going well or like. We need to do this and start and start up like being like down the trenches with you. Okay, I can't talk to Darren like hey.

A

B

Gucci yeah yeah good time for full on questions previously say say: nothing else is on your plate right now, no community and nothing else, and you were doing this CLI five days ten days. We had something that solid. But do you think realistically I knew an expert, but the runner code and someone could go as well. I would.

D

Say a brand new CLI would take.

D

A week and a half being very very for trying yeah, two weeks being a bit aggressive, Messick.

B

Five working days right in your two weeks.

D

Yeah right, yeah right so ten working days.

B

Yeah, we definitely needed do the other model, then, and then the question we could how much we'll figure out the timing of the CT. It seemed in terms of your resources but but Jackie and show on to cease point. What's what's gonna happen next well, my Monday is that we're probably gonna be clearing the runner release boards, twelve at 9:02 focus on the next big thing, so we're gonna be stretched pretty thin. So obviously we do our best to help.

B

So we try to figure this out, but yeah we're gonna, definitely stretch then, in a short term. Okay,.

A

So this is probably one of the coolest things to be working on for us, so this is good. We have the opportunity to learn a lot, and this is me pumping myself up to kind of change my direction page so that I'm not committing to stuff that I can't deliver.

D

Some guidelines on our handbook on how to create your life with go. So that's a good starter pack and yeah I mean CLI tools are like go, is very famous for here, like them, so yeah.

B

Jackie another option we could cut off off line. I could talk to Elliot about it, but let's open you offline there and I, don't see what Elliot's thought. So he comes on. Okay,.

A

That sounds good, so Sean I think we should talk to John Hampton about having high, maybe in this I I. Think from up from a go perspective. I like the idea of having someone, who's, natively a go developer, and then you can get your go stuff up to speed, but then you can also have your ownerships like a pair programming. In that sense, yeah.

C

Yeah absolutely, but my only hesitation on that is lucky really has literally just started and, and we I guess.

A

It's a good question and I do think this is low risk cuz. Nobody else.

C

That's a great field tool, so it's completely.

A

New, you have to get ramped up to so I think getting him productive as soon as possible. This is a really cool angle for us to work around through.

D

A

Again from John's view, like that's, he manages the team so we'll talk to him about that. I do feel like that makes me a little bit more confident in you know, just our team being able to support all these different things like we have redundancy, then yeah. You and I can both support. This new project that we would be owning as a part of a maintenance stream is the last thing. I'd want would be you're the only person that's responsible for everything, with release generation, which is what we have right now pretty.

C

B

I talked about a back. What I just said: I'll wait for you to have your discussions with your en about the proposals for you guys to do.

B

The CLI before I have a conversation with Elliot, but the way I was going with seeing if our incremental resources that we're bringing onto a contracting company could be a fit at this ladder to know all that, but that you have your discussions first and to echo what you're saying my gut feel is if you, if we had on the path of the CLI as the option, my gut feel is there's a grid feature to give a brand new dev, because it's not gonna break anything right off the bat yeah.

C

I do agree with her I guess that that's the thing that's going forward, that is it's it's completely new and and we've got some code that we can probably look at because we can. We can look at the existing CLI riders as a starting point. Steve. Wouldn't you agree with that?.

D

Any repeated questions, sorry, you.

C

Know the the existing CLI that we've got. You know that's building, but errata I mean but they're all kind of perhaps be a bit of a starting point for this code. Yes,.

D

I can give you some pointers. I was just looking at the code now to see what actually needs to be able to extract it, and like one of the most complex stuff would be I would say, handling the error codes right like the status codes from the API like right now, we just introduced like responding to rate limits and things like that. So those kind of stuff kind of our rudimentary bus needed for github.com ski.

C

Yeah because, as this gets more complicated, there will be more of an issue: Schomberg I, guess it's just it worked or didn't proud. Yeah.

D

And looking things they don't seem very, very complicated. I mean all you need to do like an hight side is just pick up the CI job token from the environment, which is just one method and have dads, send its to an end point and then responds to the are like the status ecology, get from the response and I'm defined to see a live flex to be properly. Okay,.

C

Place to hike and when.

D

I add so maybe.

C

If we can, you know kind of you, have you as a maintainer on this and code with you or whatever I. Don't.

D

Mind so I think I can spur like. Maybe two hours like to kick this thing's off, like let's set up the project together, kick things off, I, have it in our own, like namespace, and then we'll move it to fill up the tork when we feel like it's ready, I'm, Bundy kind of thing, I.

C

Think that would be, if that's possible, OD an awesome user table, and you know if we get ha me on the same call and you know kind of all: do it together and I.

D

Can help out on on that part right? There.

C

B

Well, unless Darren.

D

Has any objections? No.

B

I think the initial to over is fine, but just yeah. But honestly we didn't have a lot of really have a lot of pressure, but we're gonna have a lot of pressure so that it makes for 12 and I'm going on for doing our team.

C

B

Gonna be fun. Does I see that in.

A

Sean you and I can talk a little bit about the direction of the product and how we feel like this should be timed out. I think there's a lot of risk right now in it. So I don't feel very comfortable like saying that this is well that's tied to my maturity plans, but I. Just don't feel like saying that we should commit to delivering it in the next two milestones. Given that I don't know how yeah.

C

I mean given it yeah. There are a lot of questions. Maybe we should kind of do a bit more of a POC. You know get some basic working code and then and then then we'll have a very clear picture on how long it's gonna take, because you know save says it takes two weeks for him. Then you.

D

Know I say two weeks, including code review and all that oh.

C

Yeah, of course, the whole the whole process right but say you for a new for a new engineer that list man there's nothing, that's doubled right. At least you know, even if they were ago guru right as it's, how does get lab work, which is different from other organizations and yeah.

A

C

A

B

A

Right, thank you all so much. Let me know if you need anything. I appreciate your all's time thanks thanks.

D

C

D

B

D

Down notes, what we discussed and the issue or shall I do that.

A

I was gonna, I was gonna post. This recording okay.

C

A

On what I know, we have sounds good, okay, bye, y'all thanks.

C