Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The FOSSology Project – Open Source Software for Open Source License Compliance - Anupam Ghosh

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

The FOSSology Project – Open Source Software for Open Source License Compliance - Anupam Ghosh

In collaborative project development, it is important to get an insight of the open source software, to determine involved licensing and license obligations. FOSSology is a software project that is specialized in identifying license relevant statements. It can also scan source code for Copyright notices, Keywords and Export control relevant statements. FOSSology lets users generate compliance documentation according to the organization's needs, in a variety of data formats such as SPDX tag-value, RDF. FOSSology, is a Linux Foundation collaboration project, licensed under the GPL-2.0. Recently, new technology has been introduced for FOSSology, which is going towards more automation by providing a REST API for example. New scanning approaches have been developed which enables more automation at imprecise licensing. This presentation explains how users of FOSSology can use the new features.

https://sched.co/Nrv6

A

Good morning, everyone, my name, is Anupam Josh and I work with Siemens technology in India and work of the team lead I'm, a maintainer and developer for zoology. So today, I'm going to talk about physiology as an open source software for open source compliance, so the overview is, will I'll give a brief introduction of ethology?

A

Why we need facility such a tool and what the facility building blocks are, but what all they are in force ology and at the end, we'll conclude it so I think all of we are familiar with this with the screen, which gives you a legal notice. What is this? Actually so before you when you, whenever you install a software, you need to to proceed. You need to accept the legal legal notices, so all the software that are distributed. It comes with certain set of licenses, copyright statement involved and disclaimer.

A

All those things are related part of software in the licensing text. So we need to find the license text. We need to find the reference to the licenses written text. It may be not exact text. It may be a written text explaining the licensing and the rising license relevant test text. Why you need to find this that's important. So now it is. We use lot of open source software and the problem is the license. Pali finishing what is this?

A

So what we do exactly we use licensed software we use licenses, we use sorry, we use a software to build another software, and the problem here is multiple software. When you reuse, the software are actually built with some other licenses, so we need to take care of the license compliance here. We need to find out. The license have been used inside the software that we are using or reusing.

A

This is an example.

A

Example of Apache thrift appreciate fiftieth license under Apache 2.0 license, so do I achieve a very homogeneous license. But if you scan scan this approach it through to fossil ology, then it will give you 25 odd licenses. Okay, so that means that you have another 25 licenses inside appreciate thrift license.

A

So what is for serology pathologies are basically a web server for license and copyright compliance software component, so photo Logie was started in the year of 2008, its licensed under GPL. It's a Linux Foundation collaboration project, wave server based command-line interface. It provides you the scanning agent for license and copyright to relievin statement.

A

It's a multi-user, multi-tenant yvy. On the right hand, side you will be able to see that the present the different layer that facility uses it is a suppose with SQL of the database and we use c c++ PHP as the business to write or logic and on the UI part, is build on PHP, twig and jQuery.

A

So this is a workflow physiology work facility workflow starts with when you upload a package into a pathology. What do you mean by package package up the software libraries, the open source, libraries that you have when you upload it with a facility? And then you can start the agents? And when you start the agent you can choose it license: copyright ECC scanner, all those things you can choose once the scan is completed. It's ready for clearing.

A

That means the clearing person can now review the content of the package and once a clearing process is churned, they can generate a report. There are various format of reports supported like Espie, DX I'll go through them one by one. So here is one more slow here. Instable 360 is a component management portal from there also, you can directly upload software packages to force ology.

A

So this is the very same flow that we saw in the last slide. You can upload our ways as package in facility. Then it goes for a review process which is called clearing process, and then you can generate a report based on what has been done on the to review. What has been done on the clearing sign, so just I have added couple of slides yesterday, so it may not be deal in the actual slides, but I will upload this after the presentation, so facility has two main agents is called monk and Nomos.

A

So Nomos is mainly your keyword and its uses. Radix and monkey is basically more text matching kind of thing. So when you go from monkey Nomos, it gives you the flexibility, but the precision increases when it comes from Nomos to monk.

A

This is also a snapshot taken from the photo logy UI page. So when you upload a package here, it means a liver has been uploaded in Physiology. It scan the package. It's it a scan all there in each showing all the relevant license information. If I have some time end of the session, then I will give you a demo so that you can see the how the two works.

A

There is one more scanner in force: ology, that's called bar scan what it does. Basically I've talked about two licenses: no more Zen monk, though we have so many open-source license available. So it's not possible that a scanner may always hit the right license at the right time. There may be possibility that there is a new license. You may found a new license so how to handle the situation. Infertility gives you the bulk auction, where you can get the relevant text from the source file and Mark.

A

Has a bulk scanning agent marking the ball scanning agent choose a very relevant license from the list and scan it for the whole package. So in bulk you can also add or remove licenses.

A

So what are the new feature in photo? Logy the SPD x import, so what happens means it may happen sometime that some packages have been reviewed or some packages are already been seen, but you want the report for that library. So if someone comes and give us PD, X file so facility gives you the option to upload that package into force, ology and review that file. Ok.

A

So once you have got that package and if you you can yeah so in this in this slide, you can see the first option where you need to choose the package.

A

The word the package will be uploaded in the folder, then the package, for example, we've got a SPD X file for forcedly. Now you can select the jelly package and then the SPD X file that you have got may prevail in RDA format. You can upload it to the upload, deeper section and then choose you can choose therefore below options for the then they taste mean the experience. File may contain a new license, so you can choose them to be other, can't a license to be imported, and then you can import the speedy x-ray report.

A

The new report against that package. Now, once you have imported the report, you may you would like to analyze the file. What has been done? What deviants have been made on that package? Now again after analysis, you can actually generate a new report for that same package. Ok, so that can be compared and that can means you can reuse using this SPD X report.

A

So this is another feature this is after the clearing process is complete. You can to review the process. You can also generate a word document, so it's very important as you manage. Your licenses is very important to manage its obligations and pollie obligations and quality of the obligations and policies so facility also provides you a very good interface to manage your obligations and organization policies against our licenses. So you just need to define those obligations and using CSP import. You can import those in the facility interface and against a license and agenda.

A

The report obligations are automatically included in the report.

A

Facility, the rest api, the automation is upwards for a long time and the reason we come up with the facility rest api now its support, the rest api interface, so that it's become easier for user to integrate with the tools they are looking for or for uploading, a package or integrating with any other tool they want. So as of now, with the three from three four three four all version, we are supporting the rest api.

A

We have the endpoints like uploads, folder, search user, job, deport and tokens, so you can generate a token from Russell G wave ey, and then you can upload a package or post it and Digger a scan over there and generate a report, and you can also get the deport over the REST API. So here, I have given the link for the REST API calls how it works with sociology. I'll also show you if time permits attained now we see couple of agents like no more scan and no more spunk and box or bulk scanning.

A

Now, as a part of last year, g-shock we have developed one more agent, which we call its ji-sook. Is google Summer of Code, so our student Aman has helped us to develop this atarashi agent, which is not a rule-based command-line tool, but it's its uses, some statistical and algebric method to find out the license information in your file so.

A

We hear you use a couple of distance algorithm and similarity finding algorithm to find the license information from a file how it work exactly so. First, we process the input files, that's the pair files from the library and then extract comments from the file and the norm. They normalize it and then apply different distance algorithm and based on the similarity and the rank, we define that this file have this kind of what kind of Lies and so on file may contain. So this is a in wrong flow for same authority.

A

So when you have the normalized text, you perform a match and then similarity array, you get the similarity array and apply similarity array in the file and rank the findings. So that's all about more or less about what I wanted to present about the facility as a license scanner. So we have a lot of time and I can go through a brief demo of ethology.

A

Okay, so this is the facility web UI interface. So if you install physiology, it looks like this: it's very easy to install on your system, so we are in github, so you can just clone us from the github page. We are well very well documented.

A

If you go to photo majubi key, we have the difference, installation options, so you can install from source. You can use a docker container image itself, so there are various options available. You can try it out. Currently, in my system, I have installed it from source, so default user ID password is Fossey. You can login using Fosse Fosse to fossa logy and.

A

So the first thing you need to do as I defined in that process process view. So you need to upload the package with photo logy. So how you do so you go to upload, then upload the file. So now you need to choose the folder where you need to upload. You can, according to your need, you can create multiple folders and folder structure here, so just for now, I am choosing a deadly package and uploading it to.

A

400K and then you can just describe something test package or something. Then you can choose the scanning agent. This is copyrights, email, URL and then UCC analysis, QR analysis, the monk normals and you can upload.

A

Okay, so forth ology we'll unpack this package. First, then it will run the different scanning agent that we saw. So you can see this. If you go to my recent job, it will show you it has unpacked and now your package is scanned and it's ready for clearing process. So if you come to license browser tab, it will give you overview. Whatever the license has been found inside force ology inside this package, so it has found some ZD, BSO or just licenses, and then it has found the copyrights involved in this package.

A

You can see the list of copyrights that it has found. It is around 102 entry ECC also export restrictions. It has found sorry one base thing about this. You can define your copyrights and as well as ECC statement using from the configuration file. We have the configuration file, it's a text file stored in the library of ethology and you can define your own thing over there. If you need to if requres.

B

A

I'll just show you how the bulk scan.

A

So I talked about one more license, for example, but.

A

How this work I'll just show you here is the bulk interface for facility, which gives you the bulk. You can choose a statement which you think global. This statement looks like it could be a jay-z live license, so I can copy this text. I can copy into the text reference here and I can remove the deadly possibility right license and can add ze license from the list.

A

And schedule a bulk scan on this whole upload. Ok, so once this is complete, it will mark the file as identified. Ok. So it's not only this package inside its marked. Actually it's actually inside the whole package. It works. So you can now see before we start.

A

Ok, I think I took a bad example.

B

A

So it only actually marked on one file, but it actually found finder elements in two files, so it has mark to file apps cleared.

A

Okay, so now about the generating reports, so once a clearing job is journey, you can go to browse. You can go to the package that you have uploaded and, as I said you in the presentation that you can generate different type of Deportes. You have read mu SS. You have SP DX already. If you have tag value, you have unified report generation options. So let me generate a speedy XRD f5.

A

So you can see the reboot is generated and it's downloading so we'll put is download. It open.

A

So this is the SPD X report and you will have all the licenses that you have cleared. It will be here and follows SPD X information now.

A

You can also generate report in other format, according to your need. So one more thing: I told you about obligations into obligation, interface, PDX. It gives you a very good obligation management system where you can add new applications against a license or you can. If you have an obligations imported from some other place, you can do a CSV import or CSV export of applications. Okay,.

A

Okay and for REST API, you can generate a token from here.

A

Any gives it a comment, so you have the option to generate a token from here, give a name and a date and readwrite access the way one to gender adoption. The permission you want to give for your token and you can then easily use the rest in the interface, so the rest link I have given it's also their info, so logic org.

A

This is this is the photo logy share website where you will be able to find the REST API basic explanations, so here we have given some call commands that you can use over rest and you can. We have also given the swagger editor ID and the raw content of the EML files. So you can, if you put it into jumble, it will give you all the rest, API interfaces or well defined, so you can just try it out off your own.

A

It works pretty well, so it gives you a complete interface for automating your package upload with first ology or if you want to integrate with any of your CI CD you are, it enables you to do so.

A

So now this is a erotic, etosha project that we are currently integrating with pathology. Hopefully it will be done in by this year end and you'll be getting another sketch a scan engine that actually enables you to do more automation with facility. So you can conclude your license more easily. The goal of pathology is provide more and more automation, so that you don't the manual work decreases. The clearing process decreases so that you can only do the process review the lies on a license. Finding.

A

Ok, so so that was all about from my presentation. So if you have any questions you can.

A

No, we don't have a drinking slug it, yet you don't have a drinking flagon see it so you have to use raise KP. As of now. Ok,.

C

And then the question since I didn't have my microphone was whether there was a Jenkins plugin for Musashi. Sorry.

A

As of now, it's not available, ok, ready, yeah.

C

I'm Ali, we use fist ology pretty heavily we'll we've installed it with a lot of our customer base, but I really want like a full. A whole solution for yeah I came from open logic, and this used to be the business that we were in was nothing but auditing and now looking kind of replacing that with pathology. So thank you.

B

I just wanted to thank you again and highlight the fact I deserve lost over it, but in the current climate, it's very useful to be able to you do the ECC scans and in fact, that facility will find all the keywords that are relevant for that easily. For you as a useful skill, that's probably the only tools. I know that up there that open-source it does that. So do you want to talk a little bit more about how the Export Control on the scanning information is found.

A

Put off cannon scanning information.

B

How the Export Control scanning the keywords? It's okay,.

A

B

Some context about that: okay,.

A

B

Lot of open source projects, we need to refer register work across our international jurisdictions. Okay,.

A

So as part of facility, this is also open sourced our configuration facility used by default. If you go to facility, if you go to facility get a page and then you have to come under a subsea folder structure and.

A

Then come down look for ECC!

A

No, there is no ECC folder. You have to go inside copyright because for ECC we actually track the copyright agent into multiple region. So it's the same engine we use.

A

So if you go inside agent, you will find TVs ECC configuration file.

A

So this is the red X, that's mainly used by ACC so and we have the the reg X used by copyright. It's the same in the same folder. You have the rec excuse by copyright, ECC and keywords. So if you want to edit them, it's pretty easy it you can add your own stuff or it's it's very easy to do so as part of automations. Just I, just I didn't introduce it here to bring more automations. We are currently in a process integrating facility with software heritage, clearly defined and a couple of more things.

A

We are planning in future to build a compatibility agent. Also that will reduce work. A lot.

A

A

That's all thank you for joining.