CHAOSS CHAOSScon EU 2020, 23 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Open and Reproducible Research with Community-driven Research Software Metrics | Emmy, Yo, & Mateusz

Description

Open and Reproducible Research with Community-driven Research Software Metrics

What do artificial neural networks, images of the black hole, and SETI@home all have in common? They all trace their roots to research software. In this session, we hope to explore metrics and best practices for evaluating open research software. Research software engineers are evaluated largely based on the numbers of publications and citations. However, metrics based on the number of publications encourage re-creating ‘new’ things or reimplementations instead of reuse, and there are huge variations in existing practices to cite software, meaning software isn't always cited or citations aren’t always captured. Better metrics that reflect the health of research software is crucial to avoid gamification, incentivising research software engineers to follow best software development practices by recognising and rewarding their effort appropriately, and ultimately advancing open science and improving the reproducibility of research.

Through engaging the audience in a discussion to compare research software and other FOSS, we hope to better connect the research software community to the wider FOSS community. By learning from each other, we aim to identify evaluation metrics that accurately reflect research software health and can be used by hiring committees and research funders, to encourage them to move away from current problematic metrics. Ultimately, we hope this will incentivise the creation and maintenance of good quality scientific open software.

Slides: https://chaoss.github.io/website/CHAOSScon/2020EU/slides/open_reproducible_research.pdf

-----
About CHAOSScon

Learn about open source project health metrics and tools used by open source projects, communities, and engineering teams to track and analyze their community work. This conference will provide a venue for discussing open source project health, CHAOSS updates, use cases, and hands-on workshops for developers, community managers, project managers, and anyone interested in measuring open source project health. We will also share insights from the CHAOSS working groups on Diversity and Inclusion, Evolution, Risk, Value, and Common Metrics.

https://chaoss.community/chaosscon-2020-eu/

A

I am presenting this talk with a couple of other colleagues, I've Emmett sang with me here and Mattos, and so I'm here today to talk a bit about metrics in research and if I say science instead of research, any point I apologize, it's an accident, I work in science by main research. So to talk about this and to talk about why research is associated with chaos or in any way, I'll talk about how science and how research actually use software.

A

Do they don't have any examples of software that is interesting, research software, maybe that you've heard in the news know by.

B

Open record by.

A

A conductor I, don't know if I've heard about that in the news, but I watched.

C

Strange channels.

A

Okay, so I'm, here's one that maybe you did see so many of you have probably seen the images of the black hole a few months ago and and that was all open source software. That was actually that did some exciting and amazing science and I. Don't know why, but both of my examples are actually astrophysics and I work in biology, so I have any of you ever downloaded and installed SETI at home, where you give the yeah wow.

A

That's amazing me too I haven't you said ages, but I did it like I'm over 2001 downloaded it donated all my spare processor cycles so that people could search for extraterrestrial intelligence once again that is open source software. So how many of you have worked in research or science or still do I've got your hands, but still maybe a third of the room. A quarter, excellent, ok right!

A

So I will talk a bit more about metrics in research and tell you what sort of metrics research tends to have because they are not the same metrics as software and sometimes they're, not always that compatible.

A

So some of the pressures that we have sort of said this slide before I came onto it is the paper. So I think this has been mentioned actually a couple of times already today, but as someone who works in a research organization, what I have to do is I have to write papers and if I do not write papers, I, perish, publish or perish I think Celia said earlier today and I think also Kevin.

A

You mentioned that you've got to write papers and people measure you based that and if you sort of think back to before the internet, I probably can't mention that in ten years, but right now most of us remember what it was like before the Internet.

A

So before the internet we communicated via actual writing. We wrote in journals, we shared what we've been doing in our experiments and our science and our research and that's still relevant in many ways today. So a lab experiment can still be written up in just the same way. But if you bring in software, it doesn't always fit in the way that you might think. Software is very different and I explain some of the reasons why so one apart from the fact that software isn't a paper and we have to write papers, we just do it.

A

We end up writing this really nice story. So here wasn't a problem. I wrote some software to solve it at the end.

A

Thank you very much and that's how I get my credit as an academic, but then a question rises up if I update my software right so I have a patch I have a hotfix I have a minor version is that the software is functionally the same: I've done hours and hours or weeks or months of work, but it's not enough to write in your paper on, and so how do I get credit for all this work? How do I tell my boss I've been doing something worthwhile?

A

Another problem is that you can tell a really nice story about your software and actually at no point show that software to anyone in the world. That's like well, what's the point: what am I actually getting credit for my storytelling abilities and so it'd be nice to fix this, and that quote is paraphrased, but it is 100% a genuine comment on a paper that I have co-authored that why they gave us a positive review and said: I've never tried. This I cannot try this well done.

A

So there's a lot of problems in terms of the way papers, measure, software and so another metric. That science has is citations. You write your paper and then people write their own papers and they reference the papers that came before them. So here's the work that happened earlier and you are better if you have more citations and software once again, it's not a national research output, so that means I may not get the citations that I need to justify my job in my time might pay.

A

So some people forget that they might eat us off society software. Some people don't know how to cite it. Some people- maybe they just ignore it, and so we can fix that in some ways like you might put a citation dot MD in your software repository I would recommend that, even if you are not an academic, because it makes us easier for us to reference your software, but still it is a problem.

A

How do I get citations as a software engineer, how do I play with the game that exists, so the oh you've even more the journal that we publish our papers in there again? May it may affect me? That's that's another metric. It's my journal, good enough! Is it minor?

A

Is it unexcited so where I even publish my work actually affects the metrics that I get and I think that most people who work in research software agree that these are not ideal and that there could be better things, but we don't necessarily have the power to get there and also the next question is so once we agree what I mean. This is a problem. How do we measure things in a way that is meaningful, so I want to try and help you wake up a bit I?

A

Don't want to just talk at you, I know it's a warm room and we're a bit tired, so in a minute, I won't be able to get us to do an activity, but one more metric I forgot to mention is in the UK. We have something called the research x excellence framework, because we want all of our research to be really excellent, which is incredibly vague, and it doesn't actually sound all that good when you think about it in terms of software as well. So software rather science or research needs to be original.

A

It needs to be significant and it needs to be rigorous, so I think we can agree what rigorous software is. We've got that one pin down so it might be tested or might have versions, there's all sorts of things about rigorous software. We're happy with that one significant that can get a bit harder to define, but what about regionality? So, if I want original software, does that mean I need to write new software every time for it to be meaningful? What about maintaining packages?

A

I feel that originality is actually counterproductive in terms of a software measurement, because it makes it sound like throwing everything out every single time and writing new software is actually what is meaningful, I'm, not convinced. So what can we measure? What can we find as a sweet spot? I think chaos has a lot of measurements that are very relevant to science, scientific software to research, software I.

A

Don't think everything is because a lot of its corporate oriented, and so, if you're sitting you read through the chaos guidelines, I haven't seen the new ones that were announced this morning, unfortunately about a lot of the times of justifications, for things would be like the Osmo will think this, so we're thinking about justifying things to companies. Sometimes that is relevant. Sometimes that's really useful, but I think that sometimes academia, maybe only maybe, can use other metrics.

A

So we're going to ask you a couple of questions, and rather than just talking for 20 minutes, I'm gonna ask you all to talk to each other and share some ideas with us.

A

So to do that, everyone stand up and stretch. Wake up, stretch stretch elastic. Thank you. We're awake right turn to someone to your left or to your right group. 2, 2, 3 4, discuss these two things. One is what cosmetics.

B

A

Useful, what cosmetics are useful to science or to research, and if you're nave and you don't know naive guesses are great outside view. Points are amazing and two, if you know the metrics that aren't KLC, that might be interesting, suggest those as well and slider. Here enter the code.

A

Chaos on 20/20 and just jot down your ideas really quickly like a post-it note exercise, but what I don't have to type up the posters afterwards and we will call you back when we have about five minutes left just to look at some of the ideas that y'all have suggested on to slide. Oh should.

B

We collectively do this on our phone.

C

On your fighters yeah, so if you have like into them for force and then one of the people will so.

B

They got the laptop and that you did after.

C

You discuss so 113 yeah and then you will have two questions. Yes, I was forward after the first question. So around half time so.

A

We have 12 minutes six.

C

Minutes from now switch to the next question.

C

A

A

C

Maybe before we look at that, is there anything that came up during the discussions that you couldn't put in but, like maybe thank.

A

Your no time for silence again.

C

A

C

There is there anything that you didn't think like you could put in here, because it didn't fit, but it came up from the discussions around research. Software metrics does.

A

Anyone want to vocalize that point. Yes,.

C

D

Guess I wasn't metric specific, but we were looking at the context of how often underrepresented people, especially women, are left out of the footnotes of scientific academic research work. So we were brainstorming a little bit on. How could we better represent smaller contributions to larger research? Projects may be doing things like automating or better, representing, like in the footnotes or citations, with automated acknowledgments to research work depending on what the software is like. That was some of none of us have research or academic normal experience. So we were kind of throwing ideas out that.

B

Was one that we were.

D

Thinking about is.

C

Better representation, yeah smaller.

D

Bite-Sized tasks.

C

B

E

It was I just hearing about scientific software and metrics, probably about five times in the last.

E

B

Increasingly important very.

E

Quickly, so thanks for doing this couple of the things that I have heard over the last couple days, what does downstream contingencies look at your software and the number of he says of supper. They depend on what you do another.

B

E

I heard was your software say as published in biology the field of biology and actually being also used in saying, ecology or geology that it's outside of your own field, so they translate.

E

We also came up with works.

E

Oh yet contributing so scientific software projects, they can oftentimes have very small contributor basis, so even just kind of incremental improvements.

B

In the number of.

E

Contributors can be a great sign.

C

B

There's a question of reproduce: OPM the issue reproducibly little research. So when you have papers that are published and commission sent to blind reviewers and they go, they don't get to see how you did what you did so. Do you do that everybody speaking of the software that is being used so.

C

Better possibility we mean that, with the same input with the same software with the same settings, we get that we can actually get the same result. Yeah essentially yeah we're slowly getting it's improving yeah. So it is a it's still a challenge.

C

One of the things is that, can you actually run the software, so I know it might seem strange, but in in the scientific word very often you can't even rerun the software like yeah that has been used so yeah I think it's improving that also because the skills of the researchers who develop software are improving yeah.

C

So one of the biggest I think that for me, in the context of quality of science, is the most important yeah.

A

Thank you, okay, thank you. Everybody I, just want to say, I could have made this an hour or two workshop and that's would have been really good fun if this is a topic that does interest you. This is us and please contact us or speak to us afterwards and we will hook up because we can talk about talk about this more and about steps that we are taking to improve things. Thank you.

D

Thank you for making it a little bit more interactive.