Cloud Native Computing Foundation Online Programs, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Reality Check: Is it Time to Raise Your Metrics Game? - Martin Mao & Yash Kumaraswamy

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Reality Check: Is it Time to Raise Your Metrics Game? - Martin Mao, Co-founder and CEO, Chronosphere & Yash Kumaraswamy, Senior Staff Engineer, Robinhood

How do you know when you have a world class metrics function? While we obsess about the metrics of our infrastructure and apps, how often do we take the time to inspect our own function? When was the last time you benchmarked your spend, people impact, and time it takes to remediate issues? This session will cover the KPIs you need to consider when benchmarking your own metrics function - and crucially, how to demonstrate the business value that you and your team provide to the business. Join both Martin Mao, Chronosphere co-founder and CEO and former observability engineering leader at Uber, and Yash Kumaraswamy, Senior Staff Engineer at Robinhood and observability tech lead to hear about their major takeaways from their past experience and lessons learned.

A

All right well welcome everybody thanks for showing up at nine uh for a prom day today. uh My name is Martin Mao I'm, the CEO co-founder of chronosphere. uh We provide a cloud native observability solution and we get to work with awesome companies like doordash and Robin Hood hi.

B

My name is Yash kumaraswami I'm, the tech lead for observability at Robinhood.

A

Awesome man we're here today to talk about whether it's time to raise your metrics game. So what do we mean by that?

A

Well, one of the ways we can value a lot of things like our our metric system is return on investment, really simple calculation, how much return or value are we getting from a particular system like our metric system, versus how much investment are we putting in and one of the interesting trends that we're seeing is when we talk to companies out in the industry there they're saying that the return or the value they're getting out of their metrics and observability system is actually becoming lower over time, while their investment is increasing, especially as the data grows.

A

So you can imagine this makes this equation on your Roa far lower over time, and that's really not great. So that's what we're here to talk to you about today, and perhaps some techniques in in reversing that and making the ROI sort of improve. So let's talk about the return component. First, when we talk to companies in the industry, these are some of the ways that people think about how to value your metric system.

A

Things like mttd, how long it takes you to detect an issue MTD, mttr and whatnot, and when we talk to companies out there- uh and we did a study on this about 68 of companies are actually getting worse outcomes, which means they're getting worse, mtgds, worse mtgrs over time, which, which is an interesting fact for sure. So, yes, how do you think about this at Robin Hood? And how do you think about the return of this particular system.

B

Absolutely I think, as as our metrics footprint increased and as we're as we're going through the hyper growth phase of the company, we realized that the availability in MDT are nmtd are very closely correlated.

B

So uh so we had to spend a significant amount of money to get our availability keep our availability high and uh that that that's, what prompted us to consider an overhauling our metrics infrastructure in Jan January of this year, uh which it directly resulted in dropping the mean time to detect and mean time to recover, which also increases developer productivity, because they're able to root cause issues more more accurately in a more timely fashion as well. That's.

A

Interesting, so you found that actually, the availability was closely correlated with mttd and mttr and root cause analysis as well. That.

B

Is absolutely right.

A

Nice awesome, uh so you know that's a quick, quick snippet on the on the return side. What about on the investment side and how much these things cost? So you know we're all adopting Cloud native we're all containerizing everything we're all moving to microservices and generally, that ends up producing a ton more metric data than ever before.

A

You know you can imagine high cardinality is a is a Hot Topic on everyone's mind and generally more data equals more cost, whether you run the system yourself and need more infrastructure to it or you pay your vendor, for it. More data generally correlates to higher costs. However, that's.

B

A

Only thing um when it comes to cost there are a lot of hidden costs as well. So perhaps you actually talk to us about at Robinhood how you thought about the cost of these particular systems. That's.

B

Right and Robinhood V Benchmark are costs uh on a continuous basis and and they're, usually, data driven and article, and that allows us to make decisions based on data and what we found was the total cost of ownership definitely increases over time with the metrics footprint and the other costs outside of what Martin just mentioned in terms of infrastructure. Cost operating costs is also developer.

B

Productivity costs right like how do we keep our developers, productive and feeling confident about the underlying infrastructure that allows them to move faster and and be more confident when it comes to product launches and so on? That's.

A

Interesting, so you found a pretty high correlation in developer productivity as well as cost of downtime. You know, as.

B

As you data grow exactly so like uh so keeping your downtime as minimal as possible as an infrastructure, core infrastructure team as a tier, zero infrastructure team is very critical. Nice.

A

Cool, so you know we talked about this equation and how the return is getting worse and the investment is rising. So what could we do about it? There are a few things here that we've just listed on some ways. You can approach this I'll go through one at the beginning, you know even very simple things like changing the collection interval, or your scrape interval in in atheist is a great way to improve your return. You can imagine if you have a use case like a CI CD use case, and you want to roll something back.

A

Sending the collection, The, Collection interval to 5 or 10 seconds allows you to detect an issue much better than if you set it to 60 seconds. For example, we have a very simple one there. Alternatively, you can imagine if you have a you know, long-term capacity planning use case, setting the collection, scrape interval to a minute or 10 minutes or or an hour is a great way to save on costs there as well, yeah sure what some other ways that we can look at improving the ROI here, yeah.

B

So we decided to significantly invest in the solid foundation aspect, which brings me to the uh done sampling and retention of data. We wanted to increase the retention of data from two weeks, which we supported with our previous infrastructure to over 13 months with chronosphere, which allows us, and- and this also ties in very closely with the query performance.

B

We want to be able to effectively query large period of data at a more consistent in a predictable fashion and another investment that we've been pushing for is developer productivity in terms of out of the box, monitoring and observability for our teams. So any microservices that get spawned in Robinhood comes with a set of red set of metrics red set of alerts and a comprehensive coverage of their infrastructural pieces and dependencies. Awesome.

A

uh Is there any five minute keynote, so uh you know there's plenty of other ways to improve return on investment. They wouldn't have time to cover today. But if you want to continue the conversation come find us at the chronosphere booth g15 uh for this conference chronosphere. We also created a video game called Ollie Legends uh just for the conference, so come and check it out and play the game. I think there are some prizes to be won uh every day there and actually one other announcement.

A

I want to make this morning uh this news just dropped about 20 minutes ago, but uh here at chronosphere we worked with Julius Voss, one of the co-creators of Prometheus in open sourcing prom lens, which is actually a standalone application that is used as a query, a visualizing query builder for promptql, as well as the query analyzer as well. So you can sort of look at what the cardinality of your particular queries are as well. As you know, which part you, your queries are efficient or inefficient, so this is open source right now under I.

A

Believe the Apache 20 license there at that URL, it's a standalone application. You can run with Prometheus today or eventually it is going to be upstreamed to call Prometheus as well, so very grateful to work with Julius on making that available to the whole Community here. Thank.

B

You thank you for your time.