GitLab Incubation Engineering, 23 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Incubation Engineering Jamstack -- Weekly Demo March 23rd 2022

Description

TL;DR: https://gitlab.com/gitlab-org/incubation-engineering/jamstack/meta/-/issues/39

Join an upcoming Incubation Engineering update: https://calendar.google.com/calendar/u/0?cid=Y180dG1kZmF2dGtjYXY4c2IxdmxxYzUyMDFrY0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t

A

Let me quickly uh say this: is the incubation jamstack incubation engineering update for this week? um I'm glad you all joined. um I think this is the biggest audience for this update. Yet um so I'm going to talk about two things that I did this week.

A

um One is um the onboarding uh complete state uh update, which is necessary for the onboarding using the pipeline wizard for gitlab pages onboarding and the other is another smaller feature um for gitlab pages that I had been working on previously that I picked up again this week because I had a new idea, um but one after the other, um I'm going to start with what I did first this week, which is the onboarding state.

A

So here's my screen, um you see this is the graphql interface for gitlab and basically, what I did was adding a new mutation.

A

Which basically allows me to set um pages, marked onboarding, complete input for.

A

For the full for project path- um and this is a project that I have access to- which I am the owner of root- not for example,.

A

So basically, what I'm doing here is, I can now, in from the front, end mark tick box. Yes, the onboarding is complete, and this is working fine, now and and now the back end has this information that the onboarding is complete. It's really really simple, um but it needs another database column and I explained it last week why I need this because, right after submitting.

A

The after submitting the new gitlab uh pages, uh gitlab ci yaml file, and that should deploy github pages at some point. um We we're triggering a job, but it will take until this job is successfully finished. First, for us to have that information that the user has deployed a page or that there is any instance of any interaction with gitlab pages.

A

So between that time, where the user committed this gitlab ci yaml file for the first time until.

B

A

Pages instance was successfully deployed, there were kind of in a limbo um where the user interface has no information, whether or not we should show this onboarding screen again. So this should solve that issue, because now we can just once the user clicks commit for the first time we can just com, submit this notation and and thus prevent this onboarding screen from being shown again. Three other.

B

Hey uh yeah, so the minute I commit the pipeline.

B

Why don't I rely on the pipeline source itself as the state management so, like um I have a yaml file. Oh because I could be communicated in any branch, and not just the master branch, so is. Is that one? Yes, you could.

A

Be committing into the wrong branch you could be. um You could potentially submit an invalid uh file that doesn't create a pipeline just yet, because I'm giving you the option to interactively change the contents of the gitlab ci yaml file, so it could be invalid yeah, but it will still accept the commit. um So there's all those kinds of different reasons. um The pipeline just doesn't 100 overlap with what I'm trying to do here. So this is basically just an information. Have I shown you this this screen before and have you walked completely through it?

A

um There will be another way of accessing I'm imagining another way of accessing this whole onboarding screen again, um but after you've come you've completed it once, which is what I'm signifying with this toggle after you've submitted it once it will be hidden behind a second step.

A

Where you can say, please show me the onboarding wizard again, probably with a better text on the button, but um basically this so I can show you an in-between screen like um yes, you have already submitted configured your gitlab ci yaml file, but the pipeline is not deployed yet here's a list of things you can check or a list of things you can do.

B

A

It doesn't work.

B

Yeah thanks make sense. So where does this mutation start.

A

This mutation, um basically, um I have not prepared.

A

B

Hey by the way, if I'm derailing the recording, we can just do your recording first, and I can ask all these questions later. It's fine. I can cut them a bit shorter if needed,.

A

So this mutation is stored in the table project pages metadata, where I'm adding a new column, onboarding complete, which is basically a problem. Initially, this onboarding complete column will have all the values from deployed, so we're assuming that if the page is already deployed um that onboarding is complete so that somebody doesn't need the onboarding screen anymore.

A

So I'm filling these initially with the same um same values as on-boarding, complete during the migration um and eventually deployed and onboarding complete for a project that has been in existence for a little while and those values should align.

A

Unless um the pages deployment went or aura awry went wrong um or something's up with a pipeline, um in which case those two values can differ, but there should be some kind of eventual consistency.

A

Yeah nice. Thank you yeah. All right, I mean yeah. Here we have a case where onboarding is complete, but the page is not deployed which can happen if somebody walks through the onboarding but then um removes the page's deployment afterwards, so yeah those don't hundred percent line, which is why I need this new comment right. Then there was a different thing that I had been working on this week, um which is basically just the result of a shower thought. um A couple of months ago, I opened this, mr ad onboarding complete.

A

No sorry this is the one I've been working on, um enable the usage of any artifact folder name. This is an mr on gitlab pages, like the go process that delivers gitlab pages. Why.

B

A

Important because, currently, if you want to deploy anything with pages, your public files that you want to have pages delivered to the to the public um needs to be in a folder called public, which is the behavior, probably probably copied from github pages, and it makes sense if you have a simple setup, but it's an additional complication for any other static site generator, because most of the static site generators have some kind of output folder, but each of them calls them differently.

A

So I mean hugo gatsby and swelter do use a folder called public where they dump the compiled files. um React uh on the other hand, as the biggest framework uh out there uses a folder called, build, uh vue.js, naxjs, angular, astro and bit use a folder called dist by default.

A

Next.Js uses out and the currently very popular levante uses underscore site and so does jekyll.

A

All these frameworks can be configured to use a different form. That's not the issue, and I think that has how has been how most of our clients have solved this issue. um But it's an inconvenience.

A

You know it's not the most straightforward thing, because not only do you need to change the name of the output folder, many of those frameworks also use a folder called public for something else like for static assets or anything else. So you have to rename that one too um so and looking through the gitlab pages code, there's no real reason why we would have this requirement why this folder actually needs to be named uh public other than the fact that his that it has always been that way.

A

um So I'm trying to just get rid of this requirement entirely, um I'm playing through different ways of doing this. So my initial, mr, was like um changing gitlab pages in a way that it allows um basically any folder. So the original thing was I'm just. I will deliver if there's only one folder in your uh artifacts um just deliver that as the root assume. That is the root. um If it's called public, I don't care. If it's called this, I don't care.

A

um So the only way, the only thing where it was a bit difficult was if that artifacts folder contained more than one root full like what, if it has a folder called public and a folder called disk, which one is then the root.

A

um So I had this guest public directory name um function, which would always return the single folder if there was just one root directory and other than that, if in doubt it would just go through the most popular names uh ordered by most popular to less popular frameworks. So if it found a folder called public and the folder called build, it would go for public if it found a folder called this and the folder called site.

A

It would go for this, um which I think is probably rather elegant, um because it doesn't need a lot of overhead on the part of the user, um but there's a little bit of back and forth, because all this guessing happens when the user makes a request to gitlab pages, and that is a bit. um That's a lot of overhead.

A

um So this week I thought um how about we pre?

A

We move this whole um guessing to the job that deploys the gitlab pages um in rails as a rails background job um and, let's open that zip file, rename the root folder to public and then leave pages, the gitlab pages processed, as is or a third approach, which I mentioned here as a comment, because it's just pseudo code at the moment the idea was uh to just allow multiple folders and then uh use don't use any root directory at all, um but that um I just realized, while trying that out that um the artifacts.

A

uh Even if I say this like a blob pattern, anything in this folder will still remain keep their root directory in the path.

A

um So this would still need some some extra work on the rail side in order to to make this work. But it would also be very elegant because going allowing this would allow the user to skip the compilation.

B

Of static assets, so they could.

A

Propos provide multiple folders like dist, for the compiled files and static for all static assets, and all each of them would would be used as the root three has a question again.

B

My life is full of questions um yeah, so regarding option, one where we assume, so, if only one folder exists in the artifact, we just assume that is the root. um I know I know like asking the user to explicitly mention the public. Folder is not the most elegant, but assuming something is public could have a lot of security implications.

B

In terms of I know, I worked with a few people or quite a few people who are our customers or prospects and they're pretty sloppy in what goes into an artifact. So so many times you see like the git folder itself, the dot get folder as part of an artifact. So uh what would be your? Let's say?

B

How would you know that when you make this assumption of a folder like? Are you going to check if it contains static assets or you're, just going to assume that the only folder that exists has to be the root?

B

The the assumption is.

A

If there's only one root folder, regardless of the article, regardless of what it's called, that would be.

B

The root fold um I mean the very basic thing we could do is like try to see if it is a hidden folder or not. If it is a hidden folder, then don't make this assumption. So if it is a dot.

A

Folder, that is a very good idea.

B

Yes, yeah and- and I don't know but like in my opinion- if something becomes public by default, then I'd rather have the customer explicitly say that they want to do this rather than so. Maybe you can check if the pages are private or are they exposed to everyone before you want to apply this rule.

A

B

A

The thing is that pages still needs to be configured as a as a job, so somebody has to have gone in and say pages artifacts paths and just define this one folder as as the public one, um and only if, if that is in there, um you'll be surprised, this outdoor site folders would actually be additionally exposed.

A

um If, if we have multiple folders right now, but yes.

B

A

B

I I hear you and I'm not saying my recommendation- is the perfect way to do it. Yeah yeah. There are some new answers around that you have to. Like I mean if you go through a security review, probably they'll.

B

You know they'll mention more scenarios which I feel to mention, but just assuming something has to be public for the user. I know they have to still define the ammo, but we are making it really easy for them to define it in the yammer through the wizard people can.

A

B

Sloppy so we have to be careful.

A

So yeah there was another idea um which is adding another.

A

Item to to the um github.

B

A

Yaml property that specifies please use this folder as root, but I'm always a bit careful with adding more requirements to to that file because, as a user, I would assume that specifying this as an artifact under the pages.

A

Job state or job would would already be that.

B

Yeah, maybe if you can make it more explicit in the yaml definition, so if, instead of artifact paths, we said pages public and then they give the path, then at least they know that they're ex then the user is explicitly saying that this folder is going to be exposed right now, I'm just like this contract. You know, like hey gitlab, why did you expose my secrets? It should be very clear, a user because you defined it here, not because I made an assumption.

B

So I'm I don't know if I'm wearing the legal hat or being defensive but like purely like you, the user, have done something explicitly which makes this exposed to the world.

A

That's that's a good feedback.

A

Yeah I'll definitely count that in and yes, I haven't made any decision yet. I've just tried those different uh approaches and um weighing them against each other. um But yes, the ultimate goal is with as little overhead for the user as possible um and with as little going into and reading documentation stuff as possible and enable those additional sources.

B

Yeah, I don't want.

A

To negatively surprise.

B

Them yeah, no, I mean the the underlying idea makes sense right. There are all these different frameworks. Every framework has, let's say, a different folder name. um Maybe we I don't know this is probably a bad idea, but we just try to pick the biggest frameworks and support their particular folder names. That's an easy way out. um I like the idea, it's really clean right, I mean, if there's only one folder, then that obviously has got to be. The root makes sense, but there are like a different.

A

Another uh property like that behaves exactly like artifacts paths, but it's just named differently um so as to you go actively in and then we can have both behaviors parallel. Artifact paths will only yeah. I love that.

B

Yeah yeah and there's an explicit key that says public and then you build them, so publish, publish.

A

Right, thank you. This was good feedback that was great.