CHAOSS CHAOSScon EU 2020, 23 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Metrics and valuation of open source projects: A crossover between Open Source and Open Science

Description

Metrics and valuation of open source projects: A crossover between Open Source and Open Science

Despite the evolution of the legal context (at European and national levels) to acknowledge and leverage the benefits of Open Source, Open Data and Open Science - especially in regards to innovation - the most commonly used metrics to evaluate the potential and the success of projects are still tied to old & closed Intellectual Property practices (number of patents, direct financial streams from proprietary licencing, etc.). There is a need for metrics more suited to fairly arbitrate between an open or a closed model and also to more efficiently manage open projects’ valorisation. This session will present our explorations for a methodology to reflect the value generated by Open Source projects in the field of public research. We will build on our work in an open working group dedicated to the valuation of open intangible assets and studies we have conducted for public institutions, including the French Space Agency (CNES). It will cover theoretical questions like the definition of value in a given context as well as down-to-earth implementation-specific problems, like choosing micro-metrics for code contribution. The presentation will also highlight how current research in academia on the evaluation of open science (good citation practices, tracability and reproducibility) can be a source of inspiration for further work in open source communities.

Slides: https://chaoss.github.io/website/CHAOSScon/2020EU/slides/metrics_valuation.pdf

-----
Camille Moulin
Consultant - Inno³
@Camille

Camille has been an Open Source and Open Standards advocate for quite a few years now; he is currently working as a consultant at Inno³, a French niche consultancy specialized in all things Open from an IP & organisational point of view.

Celya Gruson-Daniel
Consultant - Inno³
@celyagd

As a consultant at Inno³, Célya Gruson-Daniel helps various private and public institutes in opening up research and innovation by promoting the implementation of open science and data science practices and the co-construction of knowledge (science/society interactions). With a PhD in social sciences, she frequently lectures in various research and higher education institutes to provide training in digital methodologies (data collection, analysis and management) using a pragmatic and participatory approach. She co-founded the HackYourPhD community to create a space for sharing and exchange on the evolution of research practices. She also undertakes various missions of project management and communication strategy using co-design methodologies. She gives particular attention to the development of collective dynamics and governance respectful of individuals and organizations to build sustainable digital societies.

-----
About CHAOSScon

Learn about open source project health metrics and tools used by open source projects, communities, and engineering teams to track and analyze their community work. This conference will provide a venue for discussing open source project health, CHAOSS updates, use cases, and hands-on workshops for developers, community managers, project managers, and anyone interested in measuring open source project health. We will also share insights from the CHAOSS working groups on Diversity and Inclusion, Evolution, Risk, Value, and Common Metrics.

https://chaoss.community/chaosscon-2020-eu/

A

Oh and the difficult spot, because we are the last ones I between you and Leticia- will try to be quick. Okay, so, basically quickly. Fine, we work for a small French consulting company and we do opens stuff like open source, open data, open hardware and not technical fields like legal and economic all right. So what I'm going to do today to talk about metrics and valuation of open source project and its relation to open science?

A

So, first a bit of context.

A

What you might not be aware of is that in France we have a strong incentives to produce open source first at the legal logger. So now we have a load school like wall of mail in 2016. That's basically centered on open data that says that's. Every data produced by public institutions should be made available, but it should be made available in a format that is automatically consumable.

A

That is that, if you ask I, don't know for the list of something the constant you PDF with a picture of our strategy is that we send you that was it before I mean we really educate so now, I mean by zero. Supposed to send you proper formats, like a CPS, a JSON one answer and the other interesting aspects that we talk about later, but actually in those documents, and so you have to understand also that source code is considered as a document.

A

So now I mean when they produce software, they up to also publish the source code or a piece. If you request it and other are real open source license I, think that, then you have to connect it communicated to you as I have to license it and our license, which is an open source definition compliant plan and license. So that's one girl pointed like we have a open source and open date by default or by out of principle.

B

Law is a publication part and open science part because one article was dedicated to open access. I, don't know if you are familiar with research today, but you have to pay to access the publications that you write. That's why? Because it big publisher article and you have to pay for that. That's why, in this law also, there is an article about open, an open access and the link between open source and open science. It's at in the idealistic vision of science. It's also a community collaboration.

B

Transparency and the open source community are really inspired by this open science and this way of doing science and that's why there is a natural affinity between science and open source and also the researcher, push to open their code and the consul's. But no it's not enough, because it's a lot of these rules and norms, but it's not a practice.

B

That's why an important part is to think how to change the practice and we will get back to it later, but two things I answer: evaluation are really traditional with traditional metrics and that's why it's important also to get some evolution about that. So.

A

Yeah so first we mentioned the bright side of things, so the darker things it sets a beat and at the science level, at or at a more economical level, we still have old matrix and we I mean Research, Institute, slack, also economical constraints.

A

So on one side, I mean that's one everything but, on the other side say oh yeah, but you have to produce financial revenues with your activity, so the crime that okay, but how do I justify that I produce value when I publish a project as open source so to solve this, which we realize, because that's a problem of evaluation of open digital assets, something where the work before on that on my own business side of things.

A

But so what we realize or is that you have to take a global approach of value and continue complemented to just financial value. You and we Clint quickly realized that yeah we had to take advantage of what had done in the rich such feel that about measuring the measure of value which is not financial.

A

So, generally as a starting point, it's wick wrestle net of the entity you are studying and so because it's not a private company I mean the only goal is not to produce financial revenue. So you have to consider according to the goal of the entity. What is valuable- and the second point is where is that value that you produce captured because I mean it's very easy to realize that you are producing value but the cotan of actually who captures it? It's also key to the decision that you will take so and we were a commissioned.

A

Eyes.Look, Ness, I, don't know I, don't know well as the French Space, Agency, I, think english-speaking, say Cena's, so that produce a lot of open source software and some very willing to release as much as they can as open source. But they have to justify that. It's not wasting public money, so they wanted to have a pragmatic methodology to be able to decide which project will publish at open source and when, once they published it as open source, they want to be working just addressed the produced a value by taking this decision.

A

So the current framework that they work in is that they have an not objective and performance contract with the French state and at the end of this contract you have an appendix with a kind of table like that, and we just thought slated it with performance indicators and targets, and we see that yeah. The metrics are very traditional, the least like typically the number of patents I.

A

Think all of you do that it's a very bad matrix because there's a kind of automatic assimilation between having a pattern and being innovative, and we know that it's so much untrue and typically in software, because in Europe subscriber, is not supposed to meet at interval because of the business model of the European Patent Office. Sometimes they grant patents that they shouldn't. And so, if you want to reach your targets, I mean it's very easy. I mean you can just have a as much back to that one.

A

It's just that it will cost you money and so yeah, that's a wrong and after minute level. So, as an ID, that's if we want to have a better indicator, I mean the level of questions. Indicator is not exact height, so it was trying, and so the other type of harlotry stats about cellular methods or financial constraints.

B

Metrics about scientific prediction and again you can see that it's only about publication, the number of publications in the field of scientific research, but also they mentioned scientific recognition indicators to you, Citation Index, because inside there is a lot of metrics, for example, the impact factor and of h-index, and everything is based on on fabrication, commercial, science, scientific prediction and the excellence of science in a complete competitive context, because science is really competitive to to get some funding and so on.

B

But what you can see that the in the contract there is no, there are no targets, that's why the indica and giotto's war well free, and there was some flexibility to adapt and also to create new indicators to measure a scientific value. That's why the idea of the study was also to to explore how we can add also value scientific value based open source software.

A

So, as you measure the value produced by an open source, we distinguish.

A

So, as I was mentioning before the starting point, it was a to be clear on the goals and the mission of the medicine, and that boils down to three main categories and they are to foster economical activities and upstream and downstream, and so the main consequence of that is that you have to measure your value not inside the Institute itself, but also outside of it, and that's the same for contributing to science and research in space, because the usage of your software I mean you have to measure it. Also our insight and the last one.

A

That's economic efficiency which we mentioned before, so we had four categories of value that we are quite specific to dependences, the first that we go in that term. We're not we shot the best one. That was everything that is correlational, don't do it, which means that it's tightly linked to it. But you can't measure it. It's like Tron transparency and reproducibility for science. I mean it's not something like you can measure, but it has a fundamental value. I mean if you want to be able to make reproducible science I mean the fact of I.

A

Think open source software is a necessary. The second main category. It's using its value that you realize yeah, you are what you create. It takes value as it is used, so you'll have to measure that also the server lines contribution value is that because it changes as a work flows of value creation, that okay, if I, started, project that it's successful, actually I will add external contributions. So it's like I'm having people working for me. Oh I've, been working to help me reach my goals and the last ones.

A

That's a scientific value, which is a category in itself, so far is actually value and I won't say too much because and that's typically, what is a treat here in a kiosk on, so we have problems to to measure it. So that's the usual things like delete download. We know it's a bad matrix because I mean it's not because someone downloaded that we'd use it, but also you have a lot of proxy settings.

A

Typically, when you deploy your software in a large company, you will see one download and actually it's the internal company, so that will deploy it to ten. Thousands of desktop, so we know it's not perfect, but yet still one of the means you have to take into account just once. Telemetry telemetry is fine, because you know that as a person is actually using your software, but well that has drawbacks under privacy science. You can do surveys and which is interesting because it gives you find great information. But still you have a representativity problem and January.

A

The fact of qualifying your users is how that your interesting, knowing where they are based and also I, mean what kind of entities are if there are commercial entity, is a Research Institute and so on so contribution value. That's so about twin will just use the what is classical to dive. In said Tim, you are labs and I said I, don't know if I said it much simpler and we have the classical problems.

A

Typically, I mean attaching individual contributions to entities, and there are many things that had to be done manual and typically on the classification of ng difficult. You can attach it to a domain name, fine, but then how? What is a domain name attached to I mean? Is the SMB in Europe, outside Europe, etc? And one interesting thing is that because the CNS is using its own software and it's also directly captured a part of the value of the contribution. So you have to take this into account. Also in in your modern.

B

Scientific matrix and for this today the idea would be to develop new metrics, which are project open source projects and tell our researcher, because today researcher, when the Baptist a true open source project, it's not part of the evaluation of further research and it's a big issue for researcher who wants to get a job in research in academia, because I spent a lot of time, developing software and so on. But it's not it's not part of the evaluation.

B

That's why the idea would be to have some articles about a given software project, for example in science today now there is also specific journals and I'm is data science journal, based on the fact that you share on you, open your data and the idea will give maybe to have some software science journal about when you publish or when you develop an open source software, and also that you mention in your articles as a software and this that after this point, is mansion and evaluated in your research and it's a soft row index yeah.

A

One problem is that you have that you want to automate it and to police it. Yeah I want to know which research paper you use my software. You can do a plain thing text shots in your database but say: okay, you will have a lot of false positives. Oh okay, I'd love to have a way that automatically so two beers with that, then you are.

A

You have to have existing that at first, so you have to have a way to institute a researcher to clearly state the software they use and we see that traits a growing environment and because the software themselves are asking for it explicitly and the other prime, its the automation so with datasets. You have now the use of do. Why is a digital object, identifiers that can be used for software?

A

But it's not really a common practice groups like force while walking around it, but still it's not yet reality that the tribe that, if we automate it that will be but.

B

An important, but also for I, think for open-source projects and for all the research about metrics it's to learn from the past and from the history of liberation in research, because metrics in research, the first matrix based on genre and the articles started in nineteen seventies with the beginning of scienter metric and the main goals at the beginning was to read stuff to it.

B

Identify journal to help researcher to find the information. That's not an informational to get funding. But after there was a lot of misuse of this matrix and in research. We call that publish or perish that. That means that the researcher just try to use this matrix, but it became the objectives and they are not thinking about what is to do: try to produce good side, a better science to good science, but just to answer to the metrics. That's why it's also the difficulty today and in research today.

B

There is a lot of critics about this evolution and open science is focused on finding tracks, but also to question the balance between quantitative and qualitative evaluation that everything maybe can't be quantified, and we will maybe stop there. But it's a perspective. We give you that open science can give you inspiration about your practice, but also open source project can help research orphan project archetype closed. That's why open source is a good way to inspire or.