GitLab Verify Group, 28 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Combination of CI Snippets and Composable AutoDevops

Description

Kenny, Brendan, Daniel discuss https://gitlab.com/groups/gitlab-org/-/epics/889 and how CI Snippets relate to Composable AutoDevOps

A

Sure cool thanks gents for joining I, wanted to have a dialogue about two issues that I had seen between the configure and verify stages and teams. One was around our move to composable Auto DevOps, which feels like we're breaking up the standard.

A

Very robust gitlab provided get web CI mo file for lack of a better word and then verify is working on code, snippets or CI code snippets or recipes, and I really wanted to understand how the two interacted, if at all and so I, asked Brendan the product manager and Daniel the product manager for configure to join. This call to talk about it so Daniel you and I had kind of talked about this in our a recent one-on-one, but I wanted to I.

A

Don't know where would be the best place to start, but in my mind, they're both. Let me just say in my mind, they're both an attempt to distill down into more discrete chunks, our CI mo files and I just wanted to see like I, think we have that general desire from users today, which is how do I use different parts in the compose water DevOps. It's like I want to use git Labs best practices in the CI snippets.

A

It might be I want to use my organization's best practices in a more repeated fashion, so I don't know where the best place to start, but I'd love to think about how the two of those interact yeah.

B

So I think that what we're doing with our DevOps is that we are creating templates for each one of the stages. So that's what we're calling compose for our DevOps. So let's say: if you only want to bring in auto review, you use the includes functionality of mid lab CI and include that both stage. What I understand about snippets is that you can take it one level further and include a single job that lives in X stage and I.

B

Don't know if that's correct or not Brendan, but so I think that we're have to level two levels of granularity here and yeah in their own right. They're useful. If you don't want a full stage, but you just want a job and inserted I mean one of your own stages and.

A

Stage in this case would be something as large as test so inconclusive. Water, DevOps, I, wouldn't be able to say I want to include get labs SAS tests, but I want to inject a different thing and test I either get all the tests or none of tests right. Okay,.

C

Yeah, a little bit explains it. Maybe I will show my screen I, don't know if I will invaluable look at, but I'm gonna shoot. My will at least feel better that we're looking at something right, yeah, so I think to Daniels point there.

C

The the configure team is using existing primitives within verify the the ability to include a template file right and in order in order to break up what is a you know, pretty massive file that is Auto DevOps right, like here's, a project I have with Auto doubts, enabled the way that I kind of unwrapped, the auto devops onion for people as I say use. A template of you know, forget, lab CA, see, I am oh and we actually have our Auto DevOps template today.

C

Right and so this template is basically the file behind when I don't have a get lab CI mo file. This is what runs right to run Auto DevOps and it is 926 lines right and so that's kind of massive. And then there are a lot of I. Don't want to speak for Daniel, but I think that the reason that configure is looking at this is there are a lot of folks that want to consume parts of this and the way they do.

C

That today is put this whole file in there delete the stuff they don't want, and now there's nothing updatable right, like Daniel's team ships, something great new, follow DevOps for you, no dependency scanning and the way they have to consume. That now isn't just take a get. Lab update its go. Do a diff between Daniels team's new order, DevOps yeah well and they're modified all the DevOps yeah. Is that fair, Daniel? Again I? Don't want to that's.

B

Exactly right, so basically missing out on updates from gitlab is a big part of why customer or someone to maintain this process, and a teacher like composable, auto develops, makes so much sense. Yep.

C

Yep so- and this is using existing so that they're not really not to minimize the engineering effort- that's involved here because its massive I'm sure, because I know how they'll develop or understand enough about how order that works a day to know that it's probably pretty big, because there's a lot of reused stuff that needs to be smartly factored but they're, not building more into verify, right, they're, building, a smarter set of you, animals right, like I, always say or develops, is like the smartest yeah I.

C

Am we ever see they're using the existing primitive of include and then building really smart vendor Glee animals they're? Not the only team doing this either secure is also doing this right now. Right, secure and I am assuming there's overlap here, maybe for SAS, not kind of thing, but secure has an issue called I think they call it vendored templates, but.

B

That's right so, in that same issue, if you scroll down oh yeah, possible, auto develops issue you'll see the templates that we're doing as part of composable Auto DevOps are. Is that list that you have right there at the top there's lists here yeah and if you scroll, no, no the list below that yep, oh and.

C

See Figures bringing these because because correct already brought fast.

B

The first line below the the Bosch says, security product stages will be templated. Epic, yes,.

C

So this is another related thing to this whole conversation Kennedy right, which is that Security is using vendor templates, which is going to be a really smart thing for them to do, because of a lot of other things that they do right.

C

They ship a lot of their product as external docker images to get lab right, like the the scanning container or the Claire scanning container, are like other docker images we ship outside get loud and they vendor it today pretty well, but like this will help them be even smarter about about updates, and you know, beaten been during stuff going for it. Now all that's really great and all of that's built off of primitives. That verify already has today, like the include primitive.

C

Basically, when I talk about CI recipes, which I've changed from snippets, because I feel like sentences overloaded here and I, don't know that recipes is the best word either. But it's it's today's word. So today we can look at this video and see how silly I was to call it recipes. Maybe so.

B

C

Formerly known as students, Fuhrman informally known as CI snippets, yes, because snippet I think can conveys what I mean by it and so I now. This is still in early planning phases right like right. Now it's it's set in twelve three, because I want to get it done, it's a high priority, but it's not the highest priority. So I don't know that we know exactly what this is.

C

Gonna look like yet, but this is take that idea of including something get lag, gives you and and and come up a level from the make it simpler to consume for user right like this is more like today we have this is this is a silly mock-up. That is not how it's gonna look in the end, but today we offer you two things that are very valuable to customers.

C

We offer you this thing that we saw before, which is I, want to start from a template right, I have a node project and I'm gonna click, node and I'm. Gonna get a template that, like cash, is my node modules. That's a smart thing to do with your lab, see: I won't even work. It you know shows me NPM install, it shows me, services I might want to run right. This is very valuable for a game started from I. Don't have anything to build my new project, one on one IMO.

C

The thing Daniel is working on is extremely valuable for I have a yeah Mille I'd like to consume, get allows best practices for sassed or for review, apps right and so I don't want to mess with the rest of my mo and and restart with Auto DevOps yeah. Well, I just want to take what Daniels team has been doing with with monitoring or test or whatever and I want to bring that into my right, and so that's really great.

C

What and so that in that sense includes, can be as little as a job or as much as a stage in in the auto DevOps composable world. That's a stage right like that's. That's a decision that they're making about it and the right decision, probably based on their goals.

C

Ci recipes is, is more like I want to bring in just a small snippet or job of code and again I. Don't know that how this is going to land in the end right, the simplest method of this would be something like you know: text expansion right. This is an example. Dimitri gave from from an editor he uses where you literally have the ability to say you know boom and now I get the job right, and so in this example, if it was run maven test, it would output me a little just a little job.

C

That said, you know, may even install me even test right image, maven latest right, but I think that there's a lot more to be gained from being a little bit smarter than that even extracting a layer right way. Even more so examples of this are like github actions is similar to this. That's basically, a docker container that iran, bitbucket pipes, was kind of the answer to get hub actions.

C

Drone has a concept called plugins, which again are basically docker containers rather than code snippets, but there's also somewhere in the middle and I I'm, not convinced that any of those implementations are the right, one, of course, because I'm going to be get 11 and being difficult about it, but the general idea is I want to deploy to AWS and lambda right and today, I can do that with git lab by writing AWS. You know CLI code right, but what, if either get lab?

C

And today, if I was get lab and I wanted to support people for lambda I could add. You know an entire get lab CI file about lambda, but there's not one right right and do we really want? You know this list to continue to grow?

C

Oh sorry, I missed I also can supplement that list if I'm self-hosting, because I can say, use this repository as my template repository right. So again, I can supplement that list today, which is great but again it's still a whole yeah a whole get lab, see. I am oh right, so more of it more of recipes is I want to bring in a.

C

I was gonna, say, container a contained set of best practices for a you know, a specific thing, and then maybe you know we can build. You know an ecosystem on there, whereas you know the nice thing about github actions. Is you can literally just publish a github repository of them right like if we go to.

C

Air traveler has this awesome actions repo and like, if I find one like. Let's just wait for 200. Okay great. This is just a repository that I can add in like this right action. Wait for 200 uses boom. This thing that's gonna run this docker file, which is runs. Entry point is age. Of course, sorry didn't mean to dive this team, but all it.

A

C

Wait for some period of time right.

A

Your issue about snippets you, the original proposal, was more like a code include, which is which ends up being not vendored, but what what you just showed is actually something more like a vendor to include. Then, okay, so.

C

Again, I don't know where we'll land on that, because I we haven't had UX review on this. Yet at all.

C

Circle has a thing called orbs, which is more like a vendor thing. I think it's closer to the vendor thing is the right move and I think it could really play nice with a lot of the other things we want to do is verify like we want to do smart things around variables that are required. All right, you can see. Bitbucket pipes is doing something like that here when they collect a know. Sure where that deploy you see these variables come with it.

C

Right, like you, could be really smart in leveraging this against the other primitives we're building to be smart, I, don't know exactly what that looks like yet, but that's how it's a little bit different, maybe than then just a code snippet or just an include yeah.

A

Tell me why it's different than just a vendor to include like, if say, if we're able to do sassed in this model, why wouldn't you say, but what? What additional you again? Okay, maybe.

C

It's not maybe it's not maybe it's! Maybe it is an encoder, maybe that's all it is, but it's a way of curating and exposing that in the UI, that's nicer right. I could be convinced that that's the MVC for it right again, I, don't know what it is the MVC for it is, but let's say the MVC is the MVC could be this and and now I'm gonna say this in to recording, so I can write it down later. The MVC could be in the instance. Template repository.

C

Add another folder called includes, you know, get lab CI. You includes right because that's what it is today today, it's a directory of full CI, mo files or directory of get lab, ignore files. Add a directory to this called recipes, CI recipes and then expose in the user interface things that can be included from that. From that thing, that might be a great MPC minor, just come to it. I can eat again, could eat this later. But yes, in that sense, it could be a mostly front-end way of exposing includes a better way and but.

A

I guess there seem to be this, and that includes is gonna take, and maybe that's just you you were mentioned. That's Howard, Daniel and the configured team decided to scope it, but would take something that's at the stage level where, as you were saying, it could also be at the job level, but there's nothing preventing you from creating an include at the job level. Today, there's.

C

Nothing yeah you could include your include could just could be a lot anything today and so that's again, another problem to solve again, maybe not an MVC for this, but another problem to solve in this is today the include can be anything you can break your gamma like that, with their own included right like if I had.

C

Feel, like I, have a demo for this. Something really quick, but if I had stages to find in here right, I'm not gonna, try, but if I had stages right, one Brendon to and then I went and included some file, and that also had a stages definition now I would actually just get a failure. Right, I would just get ya, know invalid and I.

A

Have to do you include the full file.

C

Takes that file yeah, it doesn't I, don't think well, yeah anime merged up smartly. So actually maybe it would merge smartly, but, and there would be cases where I could break my mo dinner. That's right like in a way that's not intuitive and so I think again, recipes someday, maybe not NBC, but solves for that, as well as like includes, are very, very basic.

C

How can we and we- this is what we talked about with mark a little bit too right like how can we be smarter about including the whole of the pipeline, and how does that impact this pipeline? You know like, like all of that. It should be better about, and so again maybe that's not what we swipe at for NBC, but that's where we want to go.

A

C

A

Does I think it would help me to know some of the more specific use cases for both and Daniel? You have some of them in the includes in that we're listing the includes. Maybe maybe it's just that I don't understand how writing includes it. If you go back to the Daniels composable other DevOps proposal, if I have that file- and it just says include all these other files is each one of those other files, like literally just gamal content, that has that says, there's a test stage and enlist the stages, because it seems like.

B

You have, you have.

A

To do something intelligence to form all of those into a test stage by including all those files yeah.

B

And- and they do each one of those files effectively inserts a new stage into your CI and you're right. It's only more gamal content that defines seriously steps. Okay, yeah, it's exactly.

A

That and if you were to change the ordering of those includes in that list, would it actually functionally change your pipelines.

B

It wouldn't change like the process itself, I think just the order of operations.

A

It might change, it would change order of operations that you do code quality.

B

Before test put it ahead of exactly like the includes will get executed in the order that you list them in your in your file. Okay, well, the jobs I should say so the stages to run in parallel right, the jobs run in sequence. Yes, you would not change.

C

The stage outline, but you would bring a job higher than another job- that's right, good at saying: okay, all good confusing things for you to be confused about that. I need to solve for right, like a hundred percent, well you're right, where I want to like follow them like this is not into.

A

Yeah I guess is that my takeaway is that Daniels using existing primitives that solves a composable, Auto DevOps, question and pain point that we have from customers Brendan. You might in the future enable newer primitives that will enable a more robust way to solve that or we as a company might enable new, robust ways to do that. But today we're getting up Daniel. You feel like we're getting at the crux of the problem: compose water DevOps with this primitive I think.

B

So, um but for the specific use case of all of them, UPS, that's exactly right, but we've already seen customers templating their own internal practices, so, like all the teams that their company can take advantage of any updates.

B

That was, like you know the guys in charge of CI made it its template and like they don't have to do anything simply didn't they automatically consume all the updates that the CI team is is doing, but yes for the specific case of our DevOps, since it's like tightly coupled with kubernetes and for people that want to get started quickly and have a modern practice. That's like the way that they've chosen to do it and more and more we get more feedback than that's the way.

B

So yeah, that's kind of the the problem that we're trying to solve is that we recognize that a lot of DevOps and covered every single. This case, for example, there are some groups that have spent a lot of time in writing their build stage that conforms to their app, but they want to use. You know some other feature of audit of us. That's kind of more of the motivation like.

C

The thing I would use it for as Brennan tomorrow, I always run to the realist world project I have, which is my little side, app right, I, actually and I. Don't know why I come to that I've been having drawn with this pipeline. They never even water it, but I used to run the performance browser performance, testing job like as a job like I would love, but but I'm not running the restaurant of DevOps, because I'm like deploying to Heroku not to kubernetes and I'm.

C

You know doing my own weird mocha testing that I want to do right, and so I 100% would consume this though right I'd much rather consume that and get browser performance testing tomorrow and have to write this and then I think it probably broke, because we changed something and I didn't change it in my thing right so, but that's there's definitely use cases for those for sure, like I personally, believe it that way and I think that a lot of our users and customers have much more advanced uses, use cases than that.

C

But this way, especially for people that want to consume I, mean I. Think it's I think it's amazing for our debits. I think it's like critical for secure right, like I, think it's absolutely critical, because we want to make that really easy for folks to consider yeah. So that's why I think I think secured a quote unquote: first, that they buy a fire release that they get you to anymore. You all do it together.

B

Did I know that they started their work as part of.

C

C

But they're still open- maybe maybe they didn't so yeah I mean I, just think it's so much more. It's super critical for secure I'm, really glad that they're doing that I think it's definitely the way to consumer, secure and I. Think it's really smart to consume all that off this way to.

A

Awesome I think we're that satisfies me. I did like I said it's helpful for me to have you know two experts talk about it and it gave me a lot of clarity on kind of what the intent was and that's that summary of we're using primitives available and verify to meet use cases that our customers hit with Auto DevOps, really helps me frame it correctly. So cool.

C

And then I want to build better primitives, but then also better abstractions right, like I, want to go on both sides of that file. Right like from them, should be smarter, but then also it should be easier to consume. So, like yeah, just I'll just.

B

C

A

Right cool thanks, gents I appreciate it thanks.

B

Kenny thanks Brendan yeah thanks.