GitLab Verify Group, 7 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DAG live preview

Description

Fréderic shows us a demo/proof of concept he's been working on with Sarah (not present) that brings a live preview mode for our CI configuration.

A

A

Hi, so this is a video to explain a little bit about a prototype. I've been working on, which is to have dynamic, dag graph being generated when the user edits their yaml file, and so for now this is just living in the c island page, and this is just to demo that this can be done. This is the editor lite which what we have now when we're editing files- and this is a yaml file.

A

This is just a setup that I've made to have a graph showing, and you can see this is displaying just fine, and so what I've did is um I've worked on tying in editing the yaml to directly generating that graph of our dependency only so if you see here, we have, we have a couple of jobs, pretty standard, we have four build job and then we have three test job and they're dependent on each other here.

A

So if I go in the test, a job and I go well, you know what it shouldn't depend on blc and then I remove it. You can see at the bottom that it's updating what's being shown, and I can. I can go okay. Well, I wanna, I wanna have build c and you can see here. This is where right now, if the build is invalid, because I haven't added the c letter, uh there's a parsing error this there could be a multiple solution for this we could have.

A

We could keep the graph showing and just have a message that says. Currently your configuration is invalid. There might be an empty state. uh This is to be determined or we could have like a higher d-bound so that it's not just as you're typing right now, it's a quarter of a second. So if I just add c, then we see the graph and then you can see that we're back to how we were, and you can. You can literally have a new job and let's say I'm gonna.

A

I'm gonna have like a new statement, deploy and be deployed a and stage doesn't matter, and then, if it needs any job from, let's say a test a little space here, then we can see that the graph updates and then I have my deploy a here and you can still use the graph as you would usually do so you can highlight it, you have every functionality and then this is pretty powerful right now we're not showing any job that doesn't have any requirements.

A

So if I had like deploy b right and- and I don't make a lot of typo while typing it- which will be more useful right, if it doesn't, it doesn't need any job or it's not needed by any job, then it doesn't show which is the current limitation. uh But this is exactly what we have right now. So this is the same graph same editor, just tying them up together and then you can see how dynamic it can become and you can just keep adding whatever you need to add here we go.

A

Do you have any questions? Sam.

B

uh This is amazing, first off um so I mean you kind of I think you touched on it just there. This is using the existing components. Isn't it so anything any updates we make to the the dag graph generally will kind of come through to this yeah. It's not like a separate version of it.

A

Yeah, um definitely it's using exactly the same component, the only thing that has been changed and that can be changing this. The same component is we now don't just render on mounting the component. So when the page load- usually it's like here, we render the graph now, there's also a property, that's like if you update the data we render. So there are some some performance implication there that we will have to look into like uh whenever we regenerate the diet.

A

We can't just say here's the new data and then just update the path we have to recalculate the whole svg and then render that right. So there might be like a big graph. We have to see how that has. If that has an impact, but it's exactly the same code, uh nothing would need to be changed, but the act right now how it's working is. The yaml gives us a data structure, that's different from what the graphics back, so we convert it, and then we feed that to the graph.

A

So the graph is the same and there's this middle piece that take care of making sure that everything is fine.

B

That's really cool, so it's basically there's there's two pre-existing things and, and this proof of concept just creates a like a translator between the two sort of things. So we can see it all in real.

A

B

A

It's just basically it's a view component that has both of them inside and just make sure they they can communicate to each other and that updating one update the other like this one can't be updated right. So it doesn't it's a one-way update. But if I update the file then it goes up and then it goes back down into the graph and then it updates and we render the data so and you can see like it. I mean at least so far it's not a big graph, but it updates pretty quickly.

A

So it's not like it does. It takes time like some time to deploy, or you know, if I just add another need another test, a then it just it just remember instantly.

B

Yeah I mean I mean you're, saying there might be performance issues at this point. This is still a proof of concept right, so they're things that we can dig into when when this gets made, uh you know properly, but I mean this. This looks brilliant. Is this something that once we have this kind of thing in, we can start doing similar things with um you know the the child pipelines and- and things like that, like all the other parts of the application that could could benefit from a similar approach.

A

Yeah, I think so I think I think this is also just a proof of concept of live editing, which is something that um was started by dennis in the front-end team, where he would show you have your editor like, and then you can see on the right what's being generated, and so that kind of inspired us to just go and try it with the graph and it works just fine. So it could work with anything really we could have.

A

You could have this being the the graph that we're using in the pipeline page just that there would be more tweaks, like the dag, is really good, because it's really self-contained, um and so this graph is just like give me data and I'll I'm going to render everything while other parts of the application are more tied into our logic. So there might be some exploration to do, but the concept is the same. We could just tie in the editor with any kind of graph or visual representation that we want, which is cool.

A

It also means we could have live tooltipping in the yamaha editing. So if you're editing your example and then you make a mistake, we could you know here instead of graph, you could render like an error, you could make a fetch call to the api on the background or you could have button which is like just validate this or it could be as you're typing, because you know this is a mistake because we're we could be parsing the ammo as we go.

B

I mean yeah, I only have one question left and that's how do I start playing with this? What what what do I do uh I get? I assume I have to run this locally at this point. We don't we there's no published version of this yeah.

A

There it's not behind a feature flag, it's not even out yet like there's a it's really just a proof of concept. I have a branch which is called. Let me, let me show it here: where is it? Zoom is getting in the way, so the branch is dag preview, dot, dash experiment, sorry, dag dash preview, dash experiment uh and basically, for now, like I said at the beginning, this is on the c island page. So if you you go on that branch, you start your gdk.

A

If you go on ci cd pipeline and then we're gonna bear with my gdp loading a little bit um and then there's a c island button on the top right and if you click here for now, this is living there. Obviously it wouldn't live there. I've I've just removed this island and put that there, so I could play with like in an existing page, but here this is this: isn't even the yabble? That's in your project. For now this is just a pre-default table that has tag view.

A

So even if your project doesn't have a tag view, this will load up. So you can start playing with it right away. It was easier for a demo than loading your apple file with an api call and then putting that in there also to note that for now this is live editing, um but the library library that I've used for this uh parses yaml either live or like in the browser or with a file. So technically, you could also pass in the yaml file and then get the data like this graph out right.

A

It wouldn't be live editing, but you would also get the same functionality.

B

Okay, that's cool, I mean you know, there's there's a bit of weirdness, there's some limitations, but again proof of concept. It's a demo. um This is perfect for that. um So do you mind if I I started sharing my screen and you talked me through how I would get the demo running yeah for sure. Let me stop sharing mine.

B

uh Where are we uh I'll have to share the whole desktop? I guess so that okay, there's gitlab, where is firefox I'll, try doing it this way, so we can see both all right. Okay, so I've got the gdk running. I checked out the uh the branch while you were talking there, uh so it's dag preview experiment. This is the right one. Yeah.

A

B

Cool, uh oh actually, there's no code to do. Is it so, let's get rid of code, um I'm assuming there's no code to do. I just need to check out the branch right exactly and then you can go in any project. Basically any project yeah, I'm going here. Why not did I click that I can't see for the zoom sharing thing?

B

Oh, go away! Oh, it's loading.

B

What's your trick for getting the gdk to run smoothly when zoom's running, because mine just doesn't like it.

A

I I have no idea, I think I think it it's like for me. If I leave it hanging for like five minutes and I don't touch it, then it's slow and then, when I start again it's fast, so I think that's why sometimes it's well, maybe.

B

That's it right so now, where am I going.

A

So you're going to see icd and then pipelines yes and then at the top right. You'll have the ci lint button at.

B

The top, under your face with there, let me just move your face, see island.

B

And there we go well. That was okay, so it's just right here straight away,.

A

For for the demonstration, it was much simpler to just have like nothing to do except checking the branch, so you can start playing around with it. Well, that's great.

B

uh Okay, so let me see test it is dependent on if I take build a out of test a.

B

B

uh What happened there test b? I don't know if I actually changed anything there. Oh yeah! No, I did it just updated. While I was looking at the text rather than the.

A

And also the positioning for now, it's not ideal. You have the the graph under there are difficulty things that could be changed there. This is all like it doesn't matter where they're placed so it could be smaller. It could be bigger one next to each other.

B

And this this literally, is just the the actual bag. Isn't it so everything? That's like all the the locking and stuff just kind of work? This is amazing. um I was expecting a much longer time to kind of get this demo running.

A

I I I actually had planned a couple of days to do it, and then I mean to be fair. We had uh sarah and I worked on it like for the for the editor part, so just rendering it and making sure we understand how it worked, and then that took a little while, not that long, I think one or two hours uh and then after that I I took an afternoon and I plugged it in so it's very simple because, like I said earlier, the drag is really self-contained.

A

So as long as I could generate the data from the yaml to the graph that it renders and then it's just making sure that we we think of how we end all the in between state. So if you make a mistake, that kind of thing for the demo is not important, obviously, but yeah it. It went really fast.

B

This is great, I mean, like I said, there's obviously some rough edges, but, like again proof of concept, this the whole point right this. This is amazing. um If I'm going to stop sharing my screen. So where is there? We go yeah so like if we were to turn this into a full-on like proper feature, um do you see any any issues in creating it?

B

That would go beyond the proof of concept kind of thing like any corners that you've deliberately cut, because it's a proof of concept that would be hard to do in a regular feature.

A

I think I think it's more. There are a couple of technical considerations. uh I don't want to go into too much details, but there is the introduction of a new library, that's one, uh so a library that parses uh yaml to json. If that's not an option, then we might be able to do something in-house. We have to see, uh then the other thing would be how we handle the error state like I said, but this isn't really a concern.

A

Just we have to do it and then the other thing would be to just think of how we translate that data from the json to the dag, because it might be interesting because currently, the the dag expect the format of the data to be the api insert.

A

So there's a lot of you know it's stages and in the stages you have groups and in groups you have all of that, um but when you're parsing the yaml file and you get it to json you're actually much closer than what we need to render it, but because we're expecting the api response, we're converting it back and then converting it again back into like close to what we already had um that might be worth looking into.

A

I don't think it would be blocking for for making it like like making it into the product uh would just be something to keep in mind as we're exploring, but there isn't any there isn't any trick. I can think of like if I look at how it's done right now. The only thing is uh the no, not even that nope, because it's just it's a component that when there's both of existing components, so it's I don't think there are any shortcut that we need.

A

So it's just like performance uh making sure introducing the library is fine. If not, then we have to parse it ourselves and that's it. That's all. I can think of.

B

So it's more polishing up the proof of concept than than scrapping it and doing it properly for lack of a better way of putting it.

A

Yeah, it's it's. The only thing that would change is that the the file placement where they're at and we wouldn't like right now everything is duplicated, because I just copied the whole thing so that we wouldn't modify existing components and just.

B

A

Everyone, but then that would be the same thing. We would just import the component and if we need to add like a function there or something we would uh so that yeah it's it's nearly like. I wouldn't see why it wouldn't be ready to be tried out.

B

That's cool okay, um I'm sold yeah! No! Thank you very much. Fred um yeah you're welcome, yeah I'll uh yeah I'll, stop the recording now and we'll upload this to to unfiltered and things and and and get you know, get people excited about it. um Thank you for that. Thank you for doing this.

A

You're welcome like I'm. I'm super excited about this, so whatever I can do to help move. That along is great.