GitLab Secure stage Offline Environment functionality demos, 24 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: License Compliance and Dependency Scanning for Offline Environments Live Demo

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

Hello and welcome everyone to another session of the offline environment live demonstration today, we'll be looking at license compliance and dependency scanning trying to get our scorecard values to a five and with that Olivier.

A

Please do your stuff Thanks.

B

So we still have little things to validate for the currency scanning, which are mostly about documentation fixes, and we had another one to give to you ask a nice BG support, because there was a specific patch to improve the documented detection logic. But since this shipped into 2010 and thanks to more, we know of the 12.8 version running on the air-gap instance, the affiant instance. We have.

B

We no longer need this patch, so we'll demonstrate demonstrate that, as seen in the Kamel here, we had a Raynor issue, but we find out that while concurrency setting the issue on our instance- and so the remaining question here was about this before crypt- that you'll see is now gone so going back to the documentation issue, I think the initial problem was this missing. Merge requests, which was about some typos.

B

So this was requested minute weeks ago, when we could go so it's it's basically fixing some of the Year mold like we are using, and it's like you can see. That was a variable issue here, so it should took the copy/paste feature if we have faith- and we also added this missing link from the top of line Department the up line deployment, the transition page so I think this should address all the remaining four that we have around the doc.

B

Please raise. It should disagree.

B

And nothing concerning this, the only one thing that remains I'm, sorry about this. This was discovered recently. We might need this additional request. So for maven we have this variable. That says, we need to disable all the SSL check. So this is what we had in place, but the way the documentation is currently working. It's a bit immediately for users, because I'll show that when two minutes in skin, so this is something we should revisit, probably as soon as possible.

B

But the way we displaying this would say: hey if you want my up Java evidence report for offline environment. Here, here's what you need to do to your configuration- and it looked like you know- I still a lot snippet that we work right when you copy past it. So you could prepare that and it won't work, because this is missing an instruction to tell me even to use a specified settings XML file, where we tell me when, where to use the custom registry.

B

So this is the issue we have about: mixing in the offline documentation, mixing information from what is dedicated to offline environment, for our tool to work and what is generate any mailing project to work with a custom registry.

B

So what has been added here is this file that explicitly say that hey, if your copy pasting that snippet, you will have also to point with your correct sitting that XML file not a blocker to me, mostly to avoid some some feedback for customer that would just try to copy past the snippet and the other one which is more critical to me as the get SSL no verify. So this was a workaround to disable SSL check when fetching the gymnasium database and in some of the tests I made in the demonstration I've made.

B

I was not using this one anymore, because I was reverting the variable that we are, which is additional CSF bundle, so by providing the CSS bundle in that variable, you don't you know, you'll no longer need that instruction. The problem is we discover that we have three different sub analyzers for gymnasium.

B

This was only working with some of them and I think I didn't watch the recording and I think this is what Tanya uncontrolled in in this demo lately. So we might need this one. So let me make sure, even if it's not applicable to all of them, you may make sure that it's working in all cases for customers and then we until.

C

They work daisy-chained can both be present. Yeah.

B

You can, if you put SM, know verify it. We just bypass this, but if you put the certs on top of that, you have no differences and.

C

Then I absolutely think we should put an M R and to include that until we can thoroughly retest everything to make sure that they all work with the CA cert I agree.

B

So this is a pending request that would probably be validated today. So I'd say we would have five ones. This is merch, but we definitely need to follow up. We are all ready before I should about making sure the certs handling is working across all the different cases that we are supporting right now.

B

Hi hope this is career. If I'm any question, please don't be.

B

Okay, so could we just specify this in those comments somewhere and I will jump through the skies PT test project.

B

So the change in that we have for we is the previous one. As we were using the built in maven registry. We would get lab itself, but this was causing some issues because kayla is very dynamic and needs to do another ton of stuff. So what we gonna do is is set up an art factory instance again into our internal network.

B

So it's the same thing as the PI P registry, the NPM registry, that we have so it's not factory instance that contains some of the dependencies that we are using for SVT for Meilin and for Gretel. As you can see, it's containing some packages- and this is a package that the test project is- is using a lot quickly.

B

Hopefully, we'll let this load, so we just add. This scale is BT template again, which has been updated to point to this external repository. So there is a an option right here. It tells well. You can override the config with what's called repositories here and the replacer is a listed there. So it's pointing IV and maven repository to this I get luck. I gave T VM instance, which is where we are asking this artifact or instance.

B

So, as you can see, there are several AI V and M VN packages are there and I'm pointing out Jackson Taliban, because this is one of those that this project is leveraging. I look at the Builder SBT. You can see that this is using Jackson data behind from this namespace.

B

So now that we see this project is being configured to work with this external registry. I'm, sorry from click I would that into ah whatever group I created. For today we have the 24 yeah, so I just created this empty group and focus there.

B

That relationship.

C

We made a mid last time that we needed to update to 1210. So this still 12:9 or it's.

B

Too often, now, all right.

B

This is why the lowest to avoid D before screenshots kara.

B

This will finish.

B

All right so opening the web, IDE I, will add ghee CLABSI IML file, so I will go to the dependency scanning offline. Documentation takes a snippet paste it there again. I will ever write this variable to tell the runner web to fetch those images so see. A registry again is a variable filled by a populated by the runner. To tell to get this on this us name and the allow user is said where we are putting all our doctrine. Images here and.

B

We also have the comedian database being stored here, and this is another setting that we need to configure. So looking back at the documentation on top of this one, we are specific settings. This is another part that is a bit odd in the documentation. We might improve that too, so you have language specific configuration, but for all the languages you all know need to apply this change, which sounds a bit odd because well, you should just jump directly to to one language.

B

You might miss it, and this is my mistake because we did throw changes, I iterating, on the documentation, so it was making sense at some point, but not anymore, more improvement in community, so I will reply that with the URL to this database.

B

So again, this is part of the explanation here. The requirement is, you have to enough line, get copy of the change and database, which is what I'm pointing to right now, okay, so going back to the documentation, I want to look at what the specific for scale is bTW, so I will copy paste that snippets, and this is telling to actually liberate your Perl script.

B

So that we are bypassing remember, we are including the SSL certificate into the key tool, which is a different approach to leveraging and custom certificates when it comes to the language, Java, I, Gretel and SBT projects. So what I need to add here, and actually what I would do is just use some snippet, because there was different settings to do.

B

We actually need to provide two of them.

B

B

So the reason why I'm putting this as you can see it's twice the instruction that we are there. So this is generate inscription to say: hey. If you have, you have the need to connect to something external that have a certificate. This is a snippet that we provide to you to edit, but in the case of this gradall project, as a scholar project, am I mixing things I think this is a bad grade. Oh yeah, sorry I'm, going with to make too many different exercise.

C

We're in SBT right yeah.

B

We're in SBT- and this is mr. little tips because it is needed for Gradle not for it- I mean it's needed for our testing product of Gretel, because it's just incredible so sorry I don't want to mix you more with this forget about that for SBT, it's just 3/4 what we just have to link to not that else, but this one, which is the custom registry where we are putting the dependencies.

B

So this is necessary because we will have SBT trying to call this us to find the dependencies and since it's a self-signed certificate, it will prevent the connection to happen unless we import that to that certificate into the key tool.

B

So we should be good to go. I will just not forget this time to add the famous check, a gap job because I know you like it much and you've.

C

Done the number of times it's run, it's our favorite job.

B

And I'm still amazed by the number of pipeline that we already running that offline instance by the way it's a lot of testing.

B

So we have the structure of the check. I got one and the teenager maven one so check our gap again, we'll try to qw get command to dr github.com registry and will fail so the job will succeed in that case, and then we have the starch into maven job that will try to scan. This scan is bt project.

B

So, as you can see, it's fetching the image from this specific location that we pointed to it is trying to fight the gymnasium database over this URL and, as you can see, it's failing because the certificates is not it's not trust.

B

So we need to go back to the merge requests, and this is exactly fitting into the situation I was describing before, which is, if you want to use the gene agent database, you either need to configure the safe, seek aids from the asked where you're asking this database, which is this integral instance here or you need the additional environment variable so I can demonstrate both. But for this specific example, I know that the certificate works.

B

So would you want both SSL bypass or SSL certs.

A

So I guess I, don't really have a preference, Olivier I mean have we shown both in the past. We.

B

Started with this bypass until this set support was shaped and Santi rested, so in in this case equally work. It said that again, the solution is to make sure that we are documenting this one so that it worked in all cases. I'd.

D

Rather see the person I'd rather see the non bypass yeah seen the bypass before so so.

B

Let me go to the seating them. No.

D

B

Will just add a valuable we have this name and taking the certificate from this offline instance. So the reason why we need to certificate it again is because the continuation database is a stood on a git repository on that ass. So when we are an English pub line is trying to get your connection to it. I will just then leave this unchanged.

B

So if I go back to the internet request, you can see that the bypass is not here and I will just be run a pipeline. Now that the variable has been set up from this branch.

B

And now the fetching of the database for work.

A

And while we're waiting for this Kevin, the belief is that most people would just use their own self-signed cert right.

C

Yeah they're not really self signed. That's what.

D

Yeah, it's their own custom, full CA certificate. Changing is the way I would describe it, but I think on it. From a certain perspective, it behaves similar to a self-signed cert, but.

D

But they would argue, it's not self signed yeah.

A

Fair enough, fair enough.

C

It is a self-hosted certificate chain kind of like the self hosted dependency registries and.

D

An MoU talk to them directly because I don't understand certificates as well as y'all, so mo can confirm, but I think I. Think technically.

E

Speaking for all root certificates or self-signed certificates, it's just that we trust the certificate, self-signed sorts, so in their case, they'll have the root and then intermediates and that's the full chain. Sorry I.

B

Think you're ready to do a burn back station too bad sets for kids. I was.

E

Just thinking about recording one in the context of Roux, because I love Ruby and week and it has opened a cell binding so I think that would be fun. I can.

C

Talk about all the pain of installing it as a system administration.

E

That I would love to learn more about that.

B

All right, so the job now succeeded because it could successfully fetch the database, which is a stood on that system and that the fine instance- and it did the analysis installing the the dependencies from the TVM instance, which is where we are stat the arts factory registry.

B

So going back to the pipeline view, we now have the security tab and we have all those vulnerabilities and again, we cannot see that they're absolutely.

B

And I think it's that time, not so much request. Sorry not request is here: I will reload and to show the same information in the security widgets.

B

So we have the 55 new vulnerabilities reported, there's information there and we can merge that into the different branch so that it fits the dashboard dependence list and the configuration page so configuration page now tell that depends. Scan is configured because there is a pipeline on the different brands that adds this job and we have to wait for it to run so that it's filling the dependency lists and the secret to that one which are currently empty.

B

This one driving this there's a cleanup.

D

um Well, we'll wait a moment.

B

So check it got be still successful.

D

Well, we're waiting for that one to finish. I do have one question so I'm, assuming that the generate merge, request, option I, believe it's available in normal dependency scenarios. Is that is that available in in offline or is that not of it elbowing offline, cuz, I? Don't exactly how that works behind the scenes.

B

So first, it's only available for CATIA skating right now and for the princess getting it's only for the young projects. So it's really limited. It's a JavaScript project using the nut is a young parking manager and yes, the logic when it comes to dependency scanning. The logic is to look at the the remediation that we have. The analyzer is providing, which is telling if there is a newer version, the way it's working that the database the advisory database contains some information about.

B

Is that fixed by a newer version of the dependency of the package right and this the tool will look at this information and we'll try to fetch weight to change the the manifest file and fetch that dependency to update, also the log files, so that tells it simulates an engineer. Updating that file manually in sending the dependency locally and pushing this commit.

B

The problem is, it depends on all your custom registry is dealing with dependencies if it's just a proxy that fetches external version of the packages, as you asked them, it will work if you are obtaining this on the different schedule and you have this new version available before the job runs, it will work if you don't have those versions available in the custom registry when the job run, it won't work and even if the job itself can work, because it's simply just obtaining a manifest file and there is no real check about updating the dependency there when you apply that change.

B

So when you create the merge request, it's going to apply the change and then the pipeline. You try to build your project by fetching this new version, and this is where it will fail.

D

B

That's all but container scanning, it's kind of a similar process. It said that under the wood, it's a curling different things like, for example, we will try to call a apt to get some new version of a system package and thing like that. But I, don't know how the possible setup that you can have enough line environment to provide those this type of dependencies for your images, yeah.

D

And I just was curious and I. Don't think it's a critical, certainly not for what we're doing right now, but I was just curious. So I could answer like I know: I'm gonna get asked that question I screwed up insert sir no thanks.

B

Maybe we should I mean we have some documentation about remediation, but I don't think we are going so far. I mean the the there's no proprietary logic here the cut is public. Everyone can see what we are doing so I think it would be valuable to just explain that, particularly for third parties, I, don't.

C

Know we I have an item I'm, not traffic I completed, yet they better explain how a third.

B

C

Should use remediation, and so we might want to add that into that issue if that one hasn't been completed yet.

D

Great yes, sorry, my intent was not to derail this, though so I thought it would be a quicker answer. It's.

C

B

So I'm refreshing the dodgeball. Now that the pipe is finished on them, the master branch, you can see that it's filled with those abilities showing them the same in. For me they're professional dependents least, it shows all the component that have been installed and.

B

It's in you'd play out I'm late, I, don't know. What's changing 12 to 10, it looks like it's a newly I would about showing what packages are vulnerable great. It.

C

It's collapsing him so that if one has multiple vulnerabilities, it's easier to read because before.

B

C

Was harder to read? I was.

B

C

B

I was looking for the vulnerable component tabs, but I mean if they are all at the top. I mean yes.

C

Ladies are sorted to top, and then the indicator on the right-hand side says how many vulnerabilities with the yellow warning indicator. Oh no, it's supposed to be sorted by default to the top. Did you.

B

Read it I changed that yeah.

C

Okay, it's 34 story. Sorry I'm.

B

Screwing, the demos scare.

C

Me another spot all.

B

Right at least it's swimming up, there I think we went through all the steps yeah. So that's correct. Let's give it a 5 I.

C

Think we decided there was a couple more dock tweaks we needed, but at least it should be I. Anyone can say no, but I think at least it should go from a three to a four.

B

Putting that link to the chat, if you want to put that into the feedback cells yeah.

D

um I I would agree with the coal. Although the question then is like when the dock is. When that merge request is done, Kenny can we can we move it to a five without having have another one of these meetings? Does that work or or did.

C

I think if we share I would vote if we share the documentation in the slack channel and people review it and give it a yes. This addresses the concerns that we should be able to to change it. Unless anybody says No.

D

I'm; okay with that I, just not having done this with this particular process before I was asking yeah.

A

I think that's that's fair. I think that if we, if we put it in our slack channel, get the confirmation we can call it good.

B

The license complains I mean: are we good with that? The prince is king, then I.

A

Believe so let.

C

Me just make sure I got the note correct on it was the second one there's. Our main concern is that we needed doc updates yep.

B

And I did the Edit link to this finding read request: okay,.

C

So I'll just put in the same scores the last time for everything else and just say copied from last time. I.

B

Think we should point to this one instead because it was miss Ling, because this you look at the previous ones, we doesn't say what's missing currently, and so we put just they're all waiting. This specific merge request: okay,.

C

I can say that they're all waiting on you know B for our 12.

B

Cool license compliance. We had remaining for again about some making which documentation so.

B

Yes, five five for the four was about documentation about limited matching. mmm Do we have that documentation, move I, don't recall! We did that yeah.

E

The mr was opened like right before the last demo and then merged. Since then, I can get the link.

B

Thank you and Swan.

B

This is I think this is the merger quiz that what painting which is we're not telling people since compliance? So when you configure things for enough.I environment again, it's the same issue about hey. You have some specific things to do, because it's offline, which is pointing to the custom and either images here. The problem is we. You also need to set up your custom registry and we add different organization in the dark. So we have this link available for private maven repositories, but we were missing it for pison.

B

So this is what has been added it with this mesh request, which is adding the private Python registry and we added the corresponding duck on the top here, which say: hey if you want to use private or custom Python repos, you can do it is by setting up this peep index URL, which is already documented. It was existing before so it's just like yeah improving the dark power to help new customer setting this up just.

B

What this one just in documentation, specify the method step and more. You give me the length things, and this is the one we were talking before about.

E

Just what you're talking about.

B

Oh, this is a mess saying. Ok, so I was talking about the the comment that we've made there, which is a limited matching. The fact that we don't have you know the Schism pipeline fetching these PDX index, so we don't have the normalization of the license name. So it's fallback to name meshing, which is less accurate right now.

B

C

Said that was fine, though, as long as it was documented did we document it yeah.

B

I think we didn't.

E

Yeah pretty sure, let's.

C

Go find it just to make sure it's in there. Looking.

E

Their exact name matches are required for project policies. It's under the set license compliance CI, job, variable, perfect and.

B

Related issues- that's great perfect, thanks mo I knew you did it ever.

C

Been so many M arse I might meet you.

B

Just for the sake of making just clear try to find the crisp any much requests.

C

Although can I detour back to dependency scanning for a second yeah.

B

C

Seems like we changed our mind at some point during the process, because PHP composer is a5 and ruby gem is a5 and all the others were for because we were waiting on all of them to be done for them to be 5. So at some point we changed our mind that everybody could be fives.

C

So as far as I know, there were zero concerns about maven, Gradle, npm, yarn.

C

Hip-Hip and and setup tools, and so like, should all of those also be 5, with the note that we can't change steps, 3 & 4 to 5 until those are fives, because it seems like we changed our mind at some point in the process.

B

Yeah the thing that you comes from the fact that we were adding here that we were on were waiting for language specific documentation, but in the case of PHP, come father, Ruby and go. There is no language specific documentation because there's no like custom registry to set up so no pass and bypass to set up I think it goes back to a 42. The above-mentioned issue, with the get SSL bypass that was not documented, so I think we could align that with all the others. Okay.

C

So they're all blocked on b12 right now, yeah.

E

C

Just wanted to like make sure we had a plan so once b12 goes and talks about the s and sells all of those other scanners below can flip to fives.

C

Or were there any other concerns on any of the other scanners, like gradle and PM, etc?.

D

Not that I know okay.

C

Cool, so those are all flip all right. We can switch back again.

B

Yeah I just wanted to link to the specific comments I mean it could find back beyond the brown. No, it's super hard to find back the America quests. It's a bit painful, okay, all entity recommit anyway, and find the merge request later where it was. It was here.

B

So this would be a a five.

B

Short cuz because we don't need anything yeah.

C

Yeah Mr, okay,.

B

Wesker's because we already passes all the things I with yeah, we are the 24-hour.

B

Customer this metric Quest is no merge, so there's no longer need to wait for it. No.

C

Longer blocked yeah.

B

So we should get I can get five on all those things for license. Complaints.

A

Yeah I have no objection. Yeah.

D

I mean for the yeah for the languages. We agree to you for this first NBC yeah break his license is one of the ones that we're gonna do the other language is later right: yep yeah, so fives for the languages that are complete, which is those.

B

So you can look at the past MVC epic. We are there the lessons, compliance next steps for language.

D

Do you already have nickeled, you already have an order for those or do you need? Oh I,.

C

Do not have an order I do have which languages they use versus. They do not use documented in there and I put a proposed order, but actually it's only the first two that I care about. So basically we're gonna go with the first two, probably in the third. Just because we can dog food it and then whatever is easiest, just gonna kind of be like four, through thirteen as easy to hard yeah.

D

Basically, I'm sorry go, go is next, then, for.

B

C++ Java Script in C++.

C

Java Script because we had like the multiple Java scripts and we did the one that they use. We need to. Oh, oh.

D

I got you I got you sorry. We.

C

Have to yeah we did the one they cared about, but I want to take care of the one we didn't care about. So we don't have to explain to people like part of JavaScript is off line and the other part isn't so yeah yeah.

D

And then go was the next one, a couple other customers I've talked to you. I'd go number two, but yeah. So I think it's final orders I mean never just like they had it higher, but they were a smaller number of folks who waited on that. So I think yeah.

A

B

Couldn't get a five runs, this one I'm gonna find.

D

B

One should be a five.

D

C

Merged so yeah.

D

Yeah and it's that's all I was really looking for- was that that's the people wouldn't be confused.

E

And the crowd goes wild.

B

B

Well anything else, you want to ask I think.

C

Just a quick recap that we believe that we're gonna flip to fives once we have a asynchronous review of documentation, updates for dependency scanning, and we probably need one more run through of the QA environment.

C

Hopefully everyone s helped the quality team work through some of the the problems that they were. Having and they'll be able to do one more demo and then victory will be ours.

A

Yeah sounds about right.

A

Okay, last call before I stop the recording.

D

Before you, post, recording, okay.