CHAOSS CHAOSScon EU 2019, 9 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hack Days...???...Profit | Sanja Bonic | CHAOSSconEU19

Description

Slides: https://chaoss.github.io/website/CHAOSScon/2019EU/slides/HackDays-Profit.pdf

There are three types of metrics: metrics, damned metrics, and KPIs.

Community health metrics, like any other data, are all about interpretation and perspective. The goal of this talk is to enable you to get your own point across to decision makers by using the numbers game.

Marketing budgets even within small companies are typically well over 500,000 USD while less expensive and more effective ways to marketing are disregarded. This talk explains the difference between marketing, developer advocacy, community management, and how you can convince your boss to let you and developers around you be more autonomous to increase business performance.

This translates to voluntary hack days, travelling to at least 1-2 conferences per year, and increasing the awareness for your company in a smarter way than a clickable blocked ad on the internet can. Number crunching and KPIs for each section included.

A

B

Only innovate, our technology, not our company, that's what a colleague told me not at Red Hat I'm from red, head I'm in Brian's team. Actually, and and we only innovate our technology. Another company was a very fun conversation I had about technology in general, when we tried to change things, and it's always about the technology, as if technology changes everything so yeah. I this talk is hack days, profit just because this didn't make for such a good talk title.

B

It's another way to tell you that stuff depends and other stuff and you're the one who knows best because that's what they tale about, but instead we call the tech days profit. So here are some numbers and facts. You can use to actually make profit or make data work for you, but keep in mind that I didn't have any really accurate numbers, so I just made them both of them.

B

That's how it works.

B

B

It does work, it works exactly like that. So just as a little background, I've worked in engineering for years before I now work as a community manager at Red Hat, so I do have some actual numbers to back up the stuff. It's just that yeah. They are sort of also made up. There are three types of metrics metrics, ten metrics and KPIs. Everyone knows kpi's our key performance indicators. So and if not the easiest note now, I've got this book with me. If you want to take a look at it, Brian's gonna get it afterwards.

B

It's the tyranny of metrics and its really really good. Mostly. It explains about yeah sure metric metrics are great, but you really have to know what you're using them for here's a little except the author writes about how metrics are used wrong in medicine and in crime-fighting and in education. So here's a really good example: when surgeons are rated or remunerated according to their success rates, some respond by refusing to operate some patients with more complex or critical conditions. It goes on that.

B

Then people die because they're not being operated on, but those deaths show up in the metrics because they've never been operated on. So a surgeon can be super successful with 100 percent survival rate, but he's just only ever gonna do like really tiny stuff like those operations or whatever, because he doesn't want to do the hard stuff. That's the problem with metrics and in the book. He also goes on telling about how in education teachers would just fake results of students in order to improve their own success rate. Stuff like that.

B

So this is what we have to look out for types of jobs that rely heavily on metrics because they don't produce anything. Our community management, developer relations and marketing, and these are all kind of related, and since we don't really produce stuff, we have to make up KPIs. So what are we gonna measure ourselves on?

B

Okay? So what kind of metrics are we talking? So there's KPIs key performance indicators. Those are metrics you track and report to your boss. After hoping the work out in your favor, then we have predictive metrics metrics attract because you think a change will warn you of something like your contributors base is going down might be because your community is unhealthy or because of other stuff, so you're gonna track those.

B

Then you have regret metrics metrics, you track to avoid repeating a problem you had in the past, which is, for example, diversity, inclusion or maybe something else, that's specific to your project, and then we have campaign metrics metrics, you track to see whether something you have initiated is useful and successful. You have like a lot of companies will have Google Analytics and then a special campaign on it which stable track. We all have procrastinated metrics. This is what we had in our teams like metrics.

B

You have because the data seems like it ought to be important, but you really have no use for them so says: maybe demographics, maybe such as contributor data, if you're not actually making use of any of this, then it's not gonna be useful. For you, a Community Health metrics, like any other data, are all about interpretation and what you make of them by the way I'm talking really fast. So that leaves a lot of questions a lot of time for questions later. This is a great xkcd one I used to think a relation imply causation.

B

Then I took a statistics class now I, don't sound like the class hubs, or maybe that's that's what we have with data now this one we can spend a little more time on it's, the actual metrics that are used. So I worked for a recruiting company as a business. Analyst I worked for I worked as a developer now worked, is community manager and I also worked in marketing and developer relations, so these are. These are actual KPIs that were used when I worked at these in these departments for HR, you have applications, interviews and rejections.

B

Those are the most used metrics, but you could also use, and some companies do use our satisfaction survey results so how Xavier a recruitment process and a lot of people, even if they received a know, if they've had a really Pleasant process. They will maybe apply again, but they will mostly answer positively to your survey results. So these would be important and also successful starts whether they stayed and stuff that's data you can actually use. It also makes sense that is not just about tracking your recruiters and seeing oh.

B

This is employee of the month because she had 20 interviews instead of 18 like the other person, so those are metrics. That would be useful. For example, in marketing we have all kinds of metrics. We have social media followers, social media interaction, impressions, CPC, CP, a CDR, like click through ratio for every link on the website. We have clicks per like cost per click. We have CPAs cost per acquisition, so I worked in a company where they have calculated exactly how much money they can spend on a new customer before it becomes unreasonable.

B

So in that case it was 400 euros. So if they spent less than 400 euro on a new customer, then it's a win. I, don't know exactly how that was calculated, but it was interesting because that's how they calculated whether they're gonna sponsor this or that conference, whether they're gonna sponsor this or that ad, whether they're gonna do red mark whether they're gonna do these images are gonna. Do any sponsored articles was all about how the conversion worked out to new customers. That was very interesting and what that could also be used for marketing. I.

B

Think most important is funnel conversion. So how much money do you spend how much trickles down to registrations, for example, for a product and how much them to actual users who pay? So that's the cost per new acquisition for community what we use or some of us use.

B

Everyone has different types of ways to measure their community health, which is good because every project is different, as we've seen with OpenShift versus other projects, so we have, of course, social media and how much interaction there is forums, new and continuous contributors, interaction on the various communication, media and content forwarded by the communities- they're, not just content created, but mostly also which contents get out and which doesn't that's a very important metric for for at least my project.

B

For for what it could also be is sentiment analysis. We've talked a little bit about that, and it's basically about how aggressive words like how aggressive is wording. What are the ethical emotions and we can use NLP for that and really good data science project. Do you want to say something?

B

Okay and comparative competitor analysis is really really good for community projects and comparative competitor analysis is when you check other projects, and you compare them equally. So there are different tools that you can use you type in all the competitor names, and then it gives you an exact analysis of how many interactions on Twitter does this one have how many different ways on the Internet does it have it's very, very nice to see, especially with open source projects which ones generate and there's always projects that generate more here and more there and more on Twitter.

B

Some are a mastadon. Some are just some Facebook stuff like that. So comparative competitive analysts analysis is really good for developer relations. Most of developer relations departments really just use how many conferences where we add and how much content to they push out, and sometimes the quality doesn't matter. It's just about showing to your boss. I went to 10 conferences in the last half-year and put out to a blog articles, so I deserved a bonus more done in the person hood than eight, even though they were better.

B

This is a problem a little bit of developer relations. Where is the cusp yeah, but as a customer survey of their acquisition journey. So, for example, if a customer meets the developer advocate at a conference and then they become a customer due to that, dad's would be a good data point, but usually you don't get to those, and it's it's hard to track these things for development. It's very easy. It's like! Usually you have Bucks Fizz ticket soft, how much time something to fix and how much like what the bug rate is.

B

Do we have 60 percent bugs or just five in six months bill times everything like that? What we could also measure for development is how much overtime has not been had as I. Have people consistently purchase their 40 hours, or has it been the consistent 60-hour workweek for everyone? This is an important metric. That also shows a lot about how much people are gonna quit later on. So it's also important for HR for product. We have features implemented and customizations resolved, like people request the feature and then they get it.

B

Yay a product gets one of their metrics up or not technical, that clear it should be a metric in product, but it's not most of the time for customer service. It's very easy. We had ticket soft and time it's time to solve.

B

What we could also do is continuous customer survey results like some companies do, which is really good for the costs, so I I'm not written down the cost for development and product, because that's just usually salaries, whereas for these four it's a little more interesting other than salaries for HR, an external recruiter you take to hire. Usually it takes about three gross salaries of the person, they're hiring.

B

So if they're hiring and let's say a developer developer earns 5,000 euro per month, it grows so the recruiter will get a payment of 15,000 and that's how much a recruiter costs. So these are the costs, for example, and whereas, if you're hiring or if you're, finding someone is a conference that you paid you pay for a booth at a small conference. You pay seven thousand euro and you get two people who are interested in company and they apply. Then you have basically saved half of what you've had for just one position for a recruiter.

B

So that's in that sense. Sometimes you have HR booths at conferences because it makes sense for them, because the proof is still cheaper than having an external recruiter for.

C

B

Mid-Level specialists for anything, but there's search, engine, optimization or technical writers who were gonna do some deep technical documentation. That's about 80 to 150 is really depends. Some are much more than that, and some mid-level is that were around 80 250 euro pour per hour for their work for social media search engines, there's always different. You have the ads and you'll just have to inquire how much it's gonna cost you and hear the difference between the cost per acquisition and the costs of what it actually took you to acquire. That's the one!

B

That's the metric that you should look out for, if you, for example, consistently spend 200 less than the maximum, then you're good, you're, good at acquiring customers for community the costs are swag travel and infrastructure, because you have forums to to do to maintain. You have servers. You have websites, that's all stuff, that's usually on the community side and all that stuff costs for developer relations, its travel equipment for testing stuff for finding out new stuff for programming, news, new things and paid content, because often developer relations also outsource things.

B

So they'll pay 500 euro for a person to write new content and which is usually not as good as if the person who's consistently in it writes it. So, most importantly, working on open source is a real privilege.

B

People in like we're all kind of in the area, and we don't realize in the last year's when I worked at as a developer, it's hard to work an open-source, mostly it's forbidden in your contract, but also it's just really really hard to get the time to work on open source and when you get home sometimes you just don't want it. So we can talk all about. We want as metrics and stuff like that, but I have talked to a lot of people who are not even allowed to travel to a conference.

B

They have to take vacation time just to travel to a conference to attend it, but giving a talk is something that they've, not enough, even think of if they think of it, their bus blocks it. So this is more of a talk to explain some costs to them, because sometimes people are not aware of how things work in terms of costs and and if you can go to your boss and say: okay I know we're paying. This is we're good connections also to the other departments, come in at the coffee table and other stuff.

B

So if you know that your department just paid thirty five thousand euro for that one move at the five thousand people conference, you can say look, but my travel costs just five hundred euro. Let me go there and I'll give you two people that I mean I will meet there. You can basically have it as your own goal to make sure that you network, with at least three people who are relevant to your business and then, if you do a report on that, and if you just include some metrics, your boss will likely be impressed.

B

Even if it's made up.

B

Have done this talk with colleagues actually present.

B

C

C

A

C

But they're primarily sitting you between probably and marketing whoops.

B

C

Their guys depend on the product. Are you seeing more.

C

B

So a trite hat we have different Deborah for and never managing has different KPIs. They measure their employees on when I worked in developer relations with for Red Hat. It's a lot of marketing focus because people don't quite get the difference and usually they think you're marketing.

B

The developers sometimes think the same so yeah it's it's really dependent on the manager yeah anymore. Yes,.

A

We distinguishing metrics that are useful.

A

B

Our case, like my boss, is really good and she doesn't. She doesn't look at metrics the same way as, for example, it happened to mean other companies. So for my projects right now, all I care about is how happy people are and that's our motives. Why are the forums and via the Twitter interaction people are very friendly to each other and there's constantly a flow off?

B

This community is great and as long as this keeps up, I'm happy and my boss is happy as well, because those are the metrics I focus on I'll show her look like these people said that and that's great if I just come up with new contributors I'm like yeah. So this week we had one new contributor and they fix the typo on the website. It's not a very impressive metric, it's nice, but it doesn't show anything about community health about how it's going. Whether people like it.