CHAOSS CHAOSScon EU 2020, 23 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Emerging Tech Open-Source Scoring System (ETOSS) | Cecilia & Justin | CHAOSScon EU 2020

Description

Emerging Tech Open-Source Scoring System (ETOSS)

This is a framework used by the UNICEF Innovation Fund, to evaluate and track the strategic position and growth potential of the early stage startups in its portfolio. The focus is on startups building open source frontier tech: Blockchain, Data Science, Artificial Intelligence and Extended Reality. It classifies the startups into four quadrants, based on two dimensions: the software developed (technology), and the financial sustainability of the project (business sustainability).

Essentially, each startup gets a final scoring for each of the two dimensions that helps position them in a matrix-kind of graph. When placed in the top right corner of the graph, the startup would have a high level of development in both dimensions. The final scoring is calculated through the weighted average of the score given to a series of key performance indicators depending on how strong they are. In the case of the ‘technology’ dimension, the calculation includes: GitHub Code Activity (Stars, forks, Commits, Contributors, Pull requests, Issues Tracking); Code Test Coverage; Open Source License Status; User Testing #; Prototype Readiness. In the case of the ‘business sustainability’ dimension, the calculation includes: Revenue, Expenses; Profit Margin; Number of Users and User Growth; Amount of funding received and number of third party funders (aside from them and UNICEF); Team sustainability (including tech and business skills as well as gender diversity).

Slides: https://chaoss.github.io/website/CHAOSScon/2020EU/slides/chaoss_fund.pdf

-----
Cecilia Chapiro
Investment Adviser & Blockchain Portfolio Manager - UNICEF
@ceciliachapiro

Cecilia is the investment adviser for the UNICEF Innovation Fund where she leads the sourcing and selection process of open source and frontier tech start-ups in developing and emerging countries. She also manages the blockchain portfolio of investments and developed the measurement model to track the progress of all companies in our portfolio (including extended reality, IoT, drones, blockchain, data science and AI).

Before UNICEF, she founded Yunus & Youth, supported by Nobel Peace Prize recipient Muhammad Yunus where she has helped grow social businesses in more than 40 countries. Cecilia has eight years of work experience in the for-profit sector (Johnson & Johnson), the non-profit sector (Junior Achievement, Ashoka, AIESEC, ‘Techo’), and the international field (Organization of American States, Grameen).

Originally from Argentina, Cecilia has lived and worked in Australia, Bangladesh, Brazil, England, Germany, Spain and USA. She holds a Licentiate in Business Administration from ‘Universidad de San Andres’ and a Master’s in Public Administration from NYU where was a founding member of the first NYU student led Impact Investment Fund and she attended as a Fulbright Scholar.

Justin W. Flory
Open Source Contributor
@jflory7

Justin W. Flory is a creative maker. He is best-known as an open source contributor based in the United States. Since 2018, Justin is a leading member of the LibreCorps program of the FOSS@MAGIC initiative at the Rochester Institute of Technology. He was the first, full-time co-op placement to work exclusively on community issues for UNICEF Office of Innovation. This was work he did in concert with UNICEF personnel and Red Hat Open Innovations Lab on UNICEF’s MagicBox effort. Since then, Justin led workshops for UNICEF’s Innovation Fund to introduce over 20 international start-up teams to the Open Source way. Currently, he works as a consultant for open source projects and community-building. Using Justin’s initial efforts with MagicBox as a starting point, RIT’s current LibreCorps mentors build interactive resources for clients.

-----
About CHAOSScon

Learn about open source project health metrics and tools used by open source projects, communities, and engineering teams to track and analyze their community work. This conference will provide a venue for discussing open source project health, CHAOSS updates, use cases, and hands-on workshops for developers, community managers, project managers, and anyone interested in measuring open source project health. We will also share insights from the CHAOSS working groups on Diversity and Inclusion, Evolution, Risk, Value, and Common Metrics.

https://chaoss.community/chaosscon-2020-eu/

A

Deston Florrie and I work at the Rochester Institute of Technology as part of the Libre corpse program where we do open source community consulting with humanitarian and nonprofit organizations. I'm here on behalf of my colleague, who unfortunately could not be here for the presentation: Cecilia Shapiro, to talk a little bit about emerging tech, open source scoring system for some of the work that we do at the UNICEF office of innovation, and it would be awesome if you could give her some social media love since she's, not here today. She would really appreciate that too.

A

So, let's talk a little bit about the background of what we do. So the unicef venture fund is a 20 million dollar pooled fund that invests in early-stage open source, emerging technology with the potential to impact children on a global scale. It also provides product and technology assistance, support with business growth and access to a global network of experts and partners. The unicef venture fund is the first financial vehicle of its kind in the united nations and also enables UNICEF to learn from and shape markets of emerging technology.

A

So the unicef venture fund is also one that exists at the intersection of a hundred billion dollar business markets and over 1 billion people's needs.

A

So, a little bit about what the venture fund does in action. It was founded in 2016 to make equity free investments in open source technology startups in developing and emerging economies that have potential to solve some of the most pressing challenges that are faced in those in those countries facing children. It also invests in UNICEF's country offices that test and develop emerging technology.

A

Solutions for children and the fund allows Yunus have to take small risks within particular technology portfolios and ensures that even if many of the investments fail, the portfolio is still a success and the venture fund invests in solutions that benefit from a small 50 K to 100k seed financial investment to significally significantly impact their ability to move to the next level of development.

A

To get an idea of the context about the work in, we have over 76 investments across 39 countries that mostly span the global SAP. As we know it so a little bit about where this all ties in to the the metrics conversation. So because the Innovation Fund is a seed investor, we take some risks on a number of companies and we measure how successful they are with a number of metrics something you are all probably familiar with for one showing success.

A

We want to connect them with UNICEF partners and country offices around the world to expand their solutions to a larger market. To do this, the UNICEF office of innovation uses a set of metrics to help us understand successful investments in open source solutions, specifically in emerging technology. To give an example of some of the portfolios that we've worked off of the past year, blockchain virtual reality and augmented reality, data science and coming soon a drones cohort. So currently we measure across two types of key performance indicators or KPIs on technology and business sustainability.

A

So in the technology KPIs, some of the things that we've been looking with or that UNICEF has really been looking at to help understand the sustainability of the technology. Solutions that they looked at is the github activity code activity around stars. Forks commits all the things that we've been looking at Ingraham, or we also look at test coverage around things of whether CI test coverage exists or unit tests are written for the project. We also look at whether there's an open source license that is approved, fits under the mandate of the standard, open source definition.

A

We also look at data such as the number of users that have actually had hands-on experience, testing the product in either local country offices or somewhere else in the field, and we've also have also evaluated on pro type readiness a little bit on the business sustainability KPIs.

A

They looked around a the percent of profit margin for the organization's revenue generation, third party funding and the amount and number of funders the team, the composition of the team, both around diversity and also skill, sets, and also the number of users, and sometimes the teams are coming into the fund with either an early stage. Solution. They've already been working on that they're looking to push forward, so sometimes they do come with already a few users in hand.

A

So, to give an idea of what the system that were using right now looks like we track a number of those metrics, the KPIs that I just talked about earlier, and we assign scores to help people across the across the UNICEF office of innovation, understand where our portfolio companies stink. So to give an idea.

A

We currently we use some of this going back to the Twitter talk, there's some, probably some cool stuff that we could learn to push forward our conversation to come up with ways to automate and better share the work that we're doing and the metrics we're also collecting, and we use some of the visualizations from the metrics we collect. This is used internally for conversations about our startups and also to do some outreach work, starting with Kaos Khan.

A

So what are the next steps for where we're trying to drive all like this? So first, we want to continue evolving the metrics that are collected currently by the UNICEF portfolio companies. One of the reason why we wanted to come here at Kaos Khan was that we wanted to learn from what the rest of the community is doing and what areas they're looking at to try to improve the way that we are studying and understanding the open-source projects in emerging tech.

A

Additionally, we also want to keep testing the relevance of the KPIs we're collecting already by looking at a wider set of projects beyond what we're just looking at in the UNICEF Innovation Fund. And lastly, we want to automate the collection process of this, hopefully with tools like grimore, and this is one area that we're very interested in looking to collaborate further with people outside of the UN and out of UNICEF.

A

This is something that has been worked on in kind of a little bit of an isolated bubble for a long time, and the momentum is starting to shift now, because we realize.

B

This isn't something.

A

That UNICEF can do by itself by its own. It's not something that can be done in isolation, so we're looking for opportunities to learn from others and also contribute some of our knowledge and experience with what we're working on with the wider community. That was a very high-level recap of the UNICEF Innovation Fund, but if you have any questions or comments, I'd love to take them, particularly. Thank you.

A

Eleven minutes, so we've got lots of time for questions. If anyone wants to ask them how.

C

Many projects have been funded so far, 76.

A

Projects to my knowledge- and that was that was this infographic we had earlier up here so for 76, I'm, sorry across 39, different countries.

C

Lots of difficult types of projects, if I say so.

A

Unicef works on different here. This is part of the digital public's good strategy. That UNICEF is behind I'm. Not super well-versed on this I only work as an external consultant to UNICEF, but the projects they fawned around specific sets of portfolios like I mentioned earlier. Some of those have been companies that are working on new blockchain solutions or working with virtual reality and augmented reality, data science and the upcoming one with drones, and some of that like, for example, with augmented virtual reality.

A

That's also been a great opportunity for us to work with organisations that are doing educational content. So that's where we've been able to introduce things like Creative Commons licenses, for you know the games that they're working on so the content that's being shared, is also a freely licensed resource as well.

B

B

B

A

Question was how many of our metrics that we have are currently automated and/or have been evaluated to be automatable. I'll, be honest, a lot of what we're doing is currently done through a mixture of Excel spreadsheets and other blackhat wizardry, but this is one of the things that by coming here, we really want to start looking more widely at ways that we can be smarter, with the way that we spend our time on this and not start from the ground from Ground Zero ground up from zero sounds more dangerous.

A

Any other questions comments concerns in the back I.

C

Wonder how you make the good.

C

What kind of services or products you have you can find strategies to make that fit that very simple, but depending on the products, you sell that's very, very hard, so I'm always interesting the connection between technical or community-based metrics and business. Once you clearly two sets, they might wonder how you try to connect those or the implications of devolution. I mean I, assume you measure them over time and how you make so.

A

The question was looking at the two sets of metrics, both technology and business sustainability. How do we connect those two together for how we evaluate the success of our portfolio companies? So it's a fantastic question and it's one that honestly I think would be better answered by my colleague, Cecilia Shapiro, but I'll. Try to give you my perspective on it, just as an individual, not as part of UNICEF I.

A

Think that a large part of this comes down to communication and messaging in terms of how you deliver these messages to internal and external stakeholders, like we've heard a little bit in the previous talk about having your internal and external metrics I think it's not just about numbers exclusively, but also the way that we communicate that value to our organization.

A

So it's not exactly probably not the answer you are looking for, but I think it's something that's actually not as much of a quantitative approach as it is a qualitative approach and how how we talk about those metrics and how we look at sharing them with others.

A

Any other questions.

D

Yeah, have there been any surprising stories or any success stories around how you use the metrics to help those projects so.

A

This has been part of what, once I was kind of embedded in the team. I started to learn a little bit about how they've been collecting these metrics so far and part of the work that I do is a consultant at RIT and through the Libre corpse program. Is we work with UNICEF on understanding what healthy open-source looks like so for some context? What we do at the Libre corpse program at RIT is we work with humanitarian and nonprofit organizations that want to commit make a bigger commitment to open source in their organizations.

A

So we started working with UNICEF two years ago and they came to us with two challenges for their. Their organization was one through the companies they are onboarding and and bringing into the fund. They need help to actually provide meant direct mentorship to these teams, who are trying to figure this all out for the first time. Very few of them have even a you know. A lot of them are coming from these developing nations that don't have a really an established technology portfolio or active technology community.

A

So I think we have to take it from a different perspective, but we have to help coach them and bring them into the open source community and set up their projects to be more sustainable from the beginning. The second piece that they that they came to us with is they're like we know we like we really wanted to open-source, but we don't fully understand what is a good project or what's a bad project.

A

We think we know, but we don't really like we'd love to like get a deeper analysis on that, so the metrics piece has kind of been from the last two years. We have like a more of a qualitative set of tools that we use. There's an open-source comm article about how the Libre corpse program uses a set of a route. We have a rubric and a milestone roadmap tool that we share with our teams.

A

So the rubric is another Excel spreadsheet, but it's all about the lot of the soft things that are harder to measure and that's one thing that we'd like to push forward into the metrics conversation. Yes, actually, if the Internet will be kind to me, I can try to actually show the rubric. Oh.

A

A

Yes, so we look at things like documentation, which we include like a code of conduct, contributing guidelines, developer, documentation, read Me's, we also look at user documentation and like five different other categories, project management, which includes things like a project or project discussion and I'm, just gonna. If you want to find all of these resources, they are all out there and published in this open-source comm article if you want to find some of those resources to bring it back to your question, this is where I was really like.

A

From my perspective, UNICEF has been doing some of these metrics on their own figuring. It out I'd like to push the conversation forward to see how can we automate the things we are already tracking through that the model that I presented on in the presentation but I'd also like to see? How can we move some of these things that we're evaluating on that are really hard to measure if some of the community ones at the bottom there's a whole section on outreach and how you engage with your community and a healthy sustainable way?

A

So I don't have a clear answer. Maybe a story to tell but I have ideas of where I want to start telling the story right.

D

Thank you how we end time we're used to have three minutes last.

A

Questions three minutes.

A

All right, thank you so much.