Cloud Native Computing Foundation Observability Practioners Summit 2019 (San Diego), 29 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: New Relic Sponsored Session - Mike Panchenko, New Relic

Description

Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

New Relic Sponsored Session - Mike Panchenko, New Relic

At New Relic, we’re going all in with Kubernetes. That doesn’t just mean delivering features to customers that allow them to observe and monitor Kubernetes, but also, embracing Kubernetes as the defacto standard for orchestrating workloads running on the entire New Relic data platform.
This lightning talk will cover the trials and tribulations of planning and migrating New Relic’s massively scaled distributed database (a database that processes up to 1.5 billion data points a minute) to Kubernetes. You’ll learn how monitoring and observability as well as the tooling created for spreading our workloads out over many heterogeneous clusters has been critical for the success of the migration thus far. We will also share our perspective about what to expect and be prepared for in the future of this fast-growing space.

A

Yeah, so I'm, Michaela, panchenko or Mike or pancakes is what everybody really calls me. I am the general manager for SATA engineering at New Relic, which really just means that I'm responsible for the technical strategy and roadmap for our internal platforms and so real quick. What is New Relic New Relic is an observability platform and what is that?

A

What even is observability I'm just kidding I only have 10 minutes, but what we think of as an observability observability platform is a platform that absorbs the data that is emitted by your apps in your infrastructure, whether it be metrics events logs or trace it you emit it will consume it and will help you make sense of it.

A

That is our goal, and the platform that we run is open connected and programmable and I'm gonna show you I'm gonna illustrate that with basically a case study of how we ourselves monitor New Relic with New Relic, so neurotic aside from being an observability platform, is also a complex infrastructure. We have over 500 services, I think that numbers trending down ish, but until until recently, the joke, which wasn't a joke at all, because it was the truth- was that we had more services than engineers. We think the humans have started to pull ahead recently.

A

So that's a good trend, but the point is there's a lot of stuff to manage. We also have an R DB, which is a you know. The New Relic database is a propriety proprietary analytics database that we built in-house, which is for the intensive purposes of this talk, is a complex distributed system. There's several tiers to it. There's a lot of state there's a lot of coordination going on, so it is quite a complicated thing to monitor.

A

We have a maze dos marathon based platform that we use for deploying most of our stateless services, like HTTP services and q, consumers that we deployed around 2016 right before the choice became more obvious and, however, it's worked really well for us.

A

What we found, though, is that's, been we've been unable to serve our internal customers that run some of these more complex stateful services, trying to get that working on mazes is just too complicated, too brittle, and so, as a result, we're moving to kubernetes with an emphasis on those systems first, and so we're doing all this by the way, while we're processing, 1.2 1.3 billion events per minute, which is about 23 million per second, so we're keeping pretty busy, and you know so what are some things in our infrastructure that, amid data one thing, are obviously services and applications?

A

Those things emit tons of data, notably they emit traces, and so we being the good eaters of our own dog food. Or you know if you prefer the more positive framing of drinkers of our own champagne. We pump all of those traces into our distributed tracing product and what's cool about distributed tracing, which you know, obviously we're. Not the only people to do this, but you're able to see your request all the way through in it. I'll see you requests all the way through an infrastructure where it gets more interesting from the platform perspective.

A

Is that if you're also sending the logs for your services to our platform, which we do or sorry first things? First, we do support some of the more common tracing frameworks and we are participating in the open, telemetry standard development. So that's where the openness comes in we're trying to get data from anywhere. We can get it into this tracing system. The other thing is we send our logs to our logs product and because we have instrumented our services with. Basically, you know for the entirety of the platform and we've we've linked everything up.

A

We can actually see logs for a particular trace in the logs product, so you just click that button you go to logs product, which you know, here's just a screenshot of the logs product. It's like many other logs products. You send your logs there, then you're, able to search and filter. What's a a little bit more unique about our product, though, is that it builds on the technology we've already.

A

We already had, where you're able to assign attributes to every log line, and so that's what enables that linkage with distributed tracing, but it's not just that right. If you look at the left left hand, side, there's all kinds of attributes that you can filter your vibe that are already pulled out for you and that's that's really powerful right, you're able to I will get into.

A

What's what that looks like in a little bit, we also aside from applications and services have the platform on which those things run and more and more in the president, the near future. That's becoming kubernetes and one of the things we're doing is we're running many kubernetes clusters. We're rolling out a cellular ization strategy in the process where we're breaking up some of our large internal clusters into smaller clusters and we're basically kubernetes is at kubernetes. Cluster is becoming becoming our unit of scale.

A

Now a kubernetes cluster on its own is complicated enough, but fortunately we have we've built a school called the kubernetes Explorer that allows you to pull all the data from your kubernetes control, plane, I into the product and and then visualize it in a in a helpful way. This lets you see down to the container level how your how your pods are doing in the cluster right.

A

So here we've clicked through and opened a specific container in a pod, and we can tell you know how much CPU its use its using, how it's doing relative to its its memory allocation and and then we can also see that all the hosts are healthy right in this particular instance, but again because we've instrumented for the whole platform, we're able to then say. Okay, let me see the logs for that specific container right there in the in the in a cluster Explorer, that's really powerful, but that's not it right.

A

If you go back to that screen from distributed tracing when you look on an individual trace that happened in a particular service. Not only can you see you know through, like the the service level trace on that same screen, but over over here on the right you can also see which kubernetes node it hit and which pod is part of savior's. If you so you're able to reason about whether or not you know a particular node in your kubernetes cluster is having issues and it's causing these problems right and you have that all right there.

A

If that's not enough for you, that's not enough power, and you you want to ask a more precise question. We have our dashboarding capability and our query language, query, language, query, language, excuse me and RQL or as we call it nurple what nurko lets you do is query the individual samples produced by all of our different integrations, including the kubernetes ones, and ask it whatever questions you want.

A

So, for example, here I've written the query, I'll just read it out, since it's kind of small from kubernetes node samples, select, unique count, node name facet, cluster name and then the you know limit time, series etc.

A

That's less interesting, but what we're doing is we're counting the unique nodes per cluster, and so this is basically the adoption curve of kubernetes at New Relic, and that's just one of the questions that you're able to ask if you're just going to the Newark whole query bar- and you say you know from let's say, container sample, which are data about containers and communities, cluster select and just hit space. You will see all the different things than you can that you can query. You can count these things.

A

You can count unique instances or particular thing you can facet by them. Those are all the different attributes that we pull in. It's super powerful. If that's not enough, even you can also build now. You can now also build your own custom nr1 applications. Nr1 is our new platform that allows you to build whatever, whatever functionality you need. That is specific to your needs into into new relic and also share that with other organizations or other teams within your organization.

A

So, for example, so here's an example of a thing that we will probably make available when we've polished it a little bit more. The team, that's responsible for running all these dozens of kubernetes clusters has built an application that allows us to see when versions of a particular image diverge across the fleet right. So here we see that we have 42 total clusters and two images have diverged in version, and it even tells us the exact clusters that have sort of drifted from the main stream.

A

It's super powerful allows us to run these dozens of clusters like more like one, big cluster and reason about about what's happening and when you know where things have gotten inconsistent, you know what else in its data teams of humans, but you didn't think that, but they do and I don't mean data like you know how many lines of get you've submitted. That's you know that's really hard and unhelpful to to measure, but what we do know is: there's certain things that slow our teams down interruptions right.

A

So at New Relic, most internal teams have a a hero channel where people go to ask for help every time someone asks for help. That's a distraction from your other other day-to-day work anytime. You get paged at the distraction, and so we built a custom enter one application that literally tracks the metrics about our teams that we know slow them down and also tracks.

A

You know how they're doing against their reliability commitments, and so now you know our management and leadership is able to look at that and see when teams start to get in trouble, see when teams start to have you know page reload, that's too high right. We see a spike up to 14 off-hours pages on a particular day. That's bad right! We know that that's gonna slow, our organization down, and so when we see changes in those trends, we know that we need to invest and make things better there, so so yeah.

A

So if it omits data, you can put it in the New Relic observability platform. Even if it's humans now I only have 10 minutes. So there's a lot more to talk about. We will be at the expo will be at booth p12, a bunch of our product managers and engineers are here. If you have any further questions or you're just interested in the platform or you just want to chat pick up a t-shirt, make some new friends, please come by our booth, we'll be there.

A

Also, if you you know, if you want to ask me any follow-ups or if you want to just again need another person, here's my contact information, I used to put my Twitter up there, I don't tweet anymore, so LinkedIn is probably better anyway. That's it. Thank you.