Cloud Native Computing Foundation PromCon North America 2021, 1 Nov 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Telemetry is Not Having to Hope - Owen Diel, Grafana Labs

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Telemetry is Not Having to Hope - Owen Diel, Grafana Labs

Telemetry isn't new, so why is it so important now? In this nontechnical talk, Owen reflects on his unorthodox journey into software, how open source enabled him, and what monitoring means in the face of ever increasing complexity. In addition to the importance of monitoring and open source, Owen describes its natural progression from a reactive tool for managing complexity to a proactive tool for incentivizing and evaluating changes. The increasing scope of monitoring is only part of its success: it is also driven by the external growth of systems it need watch. Finally, Owen elaborates how Prometheus simplifies external complexity for both technical and organizational benefit.

A

Thanks for coming, this is telemetry is not having to hope it will be a little more informal, um but hopefully fun nonetheless, and double thanks for all those who made it past. The dramatic title, so this is telemetry, looks like it to me and it can be pretty complex.

A

This is actually an example of some stuff that I use day to day and don't worry by the end of this talk, we're going to make sure that we understand every bit of it. We're going to start right here in the middle.

A

That was a joke.

A

This talk is not about being an expert, it's actually about not being one, and so we start way way back when I was very fresh on the computer science scene, and this is me in the way that exactly four bullet points can describe, and so, as I said, our story starts with me, fresh out of a coding boot camp at my first job, which is a startup whose employees you can count on your fingers, and so we had just launched a b2c app our first. Actually, this is very exciting for us.

A

So one of the things that we had was the ability to log in with other providers, so think login with facebook, google et cetera.

A

Now it's important to note that at that time I didn't really have access to the degree of information of telemetry and monitoring that I do today, and so it was really just system logs which were but then whatever kind of stock ec2 metrics that we got out of aws, and so what's the first thing that you do right and I still do this to this day- we deploy, we go live and I tested it out and ultimately there's no real substitute for that for that visceral feeling of knowing that something works and that still holds true today.

A

But so great, that's awesome. Let's, let's head out, there's a beer garden down the way and we start we want to kick off early and we're about halfway there when we start to get reports that people can't log in and oh no right, my heart just sinks like a stone, and I am very worried. So we all get out our phones and again, let's try to log in we log in that works.

A

huh So what do we do now and for me this is actually in the story where I made the wrong choice. So I doubled down and I could see now how bad of a decision that was, but it was something that I understood in retrospect. I did that because I felt like I couldn't afford to be wrong. I felt very much out of place in the industry at my first developer.

A

Job, like I, whatever whatever I was working on, had to work and that's probably for another talk, but it's okay to feel that way and coming back to it for everyone, who's curious what it actually was who's the culprit.

A

It's a clock, drift right and I think that everyone kind of has this experience at some point or another. It's when two machines can't agree on what time it is and if they're used in the same system, this can cause pieces to fall apart in ways that you don't expect and so for our story.

A

What actually happened was that it fell apart in a way that a portion of users couldn't log in it's a relatively small amount but problematic. Nonetheless, and really. Interestingly, this kind of thing gets right by the kind of testing that we were using.

A

Realistically, we could have logged in a bunch of times and never seen it, and we didn't see it so that probabilistic nature made it really difficult, really really good match for getting past our defenses.

A

And so we're back here now this this is still telemetry, but how do we make it helpful and intuitive in a way that's helpful beyond when your investors walk in, and you want to point to something on the mounted on the wall no helpful in a way that becomes the first thing that you reach for to understand your systems.

A

And to really get there, we have to talk about telemetry as something that's way beyond successes and errors. Here, it's abstract it can be abstract and still be helpful and intuitive, and so we're going to go over just a couple. Examples that I wish that I had and try to relate them and set that up for the for the rest of this exercise, and so the first one here is something that we're all familiar with, whether we know it or not.

A

Latencies the difference between a website loading like this and a website, loading like this and everyone wants to be part of to use to build the former and hates using the latter, and that makes sense the next bit here that I want to talk about. Is I call them forks in the road? It's really whenever your application needs to make a decision to do one thing and not another.

A

Do you have an item on hand or do you need to go back order? It check if it's an inventory, that sort of thing, and so all of these things that your application does and when it makes these decisions? The timing information, the frequency it can help you cross reference and correlate and ultimately build a story as a sort of causality.

A

This is how I understand that I'm a pain to be around whenever I'm hungry and then the last one here is back pressure. So how? How long is the line to the bathroom? You know, I'm I'm at a conference right now, there's pretty boring talk, it's probably zero. Now, it's my time, take the hand y'all.

A

So these are a few examples that I kind of wish that I had at that point.

A

But so why are we talking about it now? Why today and to be fair, prom con is probably an aspect of that, but it's also the burgeoning complexity in every sphere kind of assaulting us on all sides, and ultimately, everything is consumable. These days we have hobbyist sites that are highly available and are distributed via content delivery networks across the globe.

A

All of our cloud vendors, all these cloud platforms that we run our software on everyone has compute for sale, but now everyone also has all these other provisionable building blocks of distributed systems.

A

So databases queues cloud functions which can execute thousands millions of times concurrently, and that's just our cloud vendors, that's not even to talk about the api economy. Think of all the sas companies that are consumable in a programmatic fashion. This becomes our authentication services.

A

All of these integrations, even in the case of my employer, our telemetry and monitoring, and this choice is empowering and incredibly freeing, but the cost of having everything at our fingertips is that we depend on everything.

A

So as an example, I have two 99 available legs and the chance that I can walk that they're. Both functioning at one point in time is less than that, and so complexity is the reason it enables us, but we have to pay part of that back as a cost of understanding it and by the way, no shade at python. Here. It's just. I love this love this example, and so open source is a really really good fit.

A

It handles this complexity very well, and don't worry, I'm not going to get philosophical about open source here, at least not too much. I see how far the drop is from my soapbox.

A

I'll try to try to stay pragmatic, and the first thing I see here is the cost of learning. It's really really low, generally you're, not paying for money, at least in a monetary sense, sign up fees or memberships additional head count.

A

You have to pay a different kind of cost, and this extends through to the design. The fact that when you start with a new tool, you're, probably not bootstrapping a sas account somewhere, you're, probably running it on your own machine.

A

And so yeah that extends into the design. It forces us to understand some degree of complexity, and that's that sounds wrong. Doesn't it isn't the whole point that we abstract everything away? So we don't have to deal with it.

A

I'm not sure. I think that some degree of complexity is important for us for us to understand.

A

It makes sure that, at least at a high level we're agreeing to the paradigms the choices that we're making for new tools that we adopt, and this extends really into our second point, which for me, is definitely due diligence and makes uh things like smoke and mirrors really a lot harder to hide.

A

You can go look at the source.

A

And then finally transfer the ability to learn something. We've shown that you can pick up open source, it's a lower barrier. You can pick it up. You can learn it, you can get hired to do it and then you can find somewhere else to do it too.

A

You can take these things with you and it's incredibly valuable, and not just for the individuals for the software engineers amongst us, but also, if you're, an organization it allows you to hire more easily.

A

But okay, this is one philosophical bit I'm gonna get into here. If anyone has built internal tools at a company before and loved them right or you worked at a company and used this internal tool that you thought was fascinating and then you change organizations and then you can. You can't use it anymore. That is incredibly debilitating, and so it allows you to to persist in this, whatever you're building, to see the benefits transfer across all these organizational lines and that that really lifts all of us up.

A

And so all these effects of open source really compound into a set of really low barriers to entries.

A

This makes new ideas more feasible, which is super valuable and for me, at a really small company. Initially, this wasn't prometheus, but it was always open source and the idea here is that we can use this to build on top of each other, and we can start building ideas and not dependencies.

A

It allows us to prototype. You know have to have new garage projects that can start at such a higher level. Comparatively you know kind of before we go into the next slide. When I was putting this one together, you can see a prometheus is at the top and then there's you know a couple templated ones in. I actually had real examples in there and then I realized I had this epiphany.

A

That would be the worst idea to get nerd sniped on stage for right, because I, my choices are not the same as everyone else's, but anyways prometheus is pretty much at the top there. I think it's a safe bet at prom con, so I'll deal with the fallout there and I think prometheus hits a lot of these points expertly.

A

It's incredibly simple. To start with, run the prometheus server. It'll watch your infrastructure for you and okay.

A

What do I really have to do to kind of continue setting that up? You can get some immediate benefits using these exporters and that's generally the term for how we get data into prometheus, be a bit of code that runs somewhere and exposes something. I left a couple of examples here which are some of my favorites, but there's so many too many to count official and unofficial variants, and so you can pick and choose start to use these more and more until you are actually comfortable and you're, seeing a lot of the value from this.

A

So what I guess that means is that prometheus wears complexity very well, and so now we're at the point say we have half a dozen exporters, we're monitoring a lot of our infrastructure. This way we're really seeing the benefits of it, but wouldn't it be great if we had some of those abstract metrics from earlier that we talked about things that really exposed the domain specific logic of our applications.

A

We had an exporter for our own apps and you can do that. Go instrument. A little code. You know write a little more code alongside the code, you already write to tell yourself the things that you want to know and it's pretty easy and then you can finally bring all of these things back together and you can alert on them. Let yourself know if a condition is or is not met, that's very powerful right to the ability to draw a line in the sand somewhere and say past.

A

This point like I need to know something's wrong.

A

But really we can also invert that idea and say if we knew a point past, which something is wrong past which, where we need to be alerted, we can also say that everything before then is technically still. Okay right, that's good enough, and so what I'm trying to get to here is the idea that using telemetry only for disasters only for remediation only for figuring out what's wrong. When I have an outage that leaves the majority of the benefits of telemetry on the table.

A

And so this is one of my favorite parts about it. It's ability to use monitoring proactively.

A

To say not just use this when it's wrong, but also raise the bar when things are right, how it allows us to be really effective with the limited time that we have to to work effectively to work on our applications, and so in this example. This is a kind of method that I'll use pretty often so I'll have some idea.

A

Oh wouldn't it be interesting to know x, y or z about my application that I don't know now. I think it might behave weird in this scenario, so I can go, add instrumentation and when I do that, then it starts to trickle in I start to be able to see that I start to answer these questions. I start to be able to understand if I'm asking the right questions and it doesn't always work out right. Sometimes we go back to the drawing board and that's fine. Sometimes it does.

A

You know, accept or roll back, no worries we're learning something either way, and it's really really valuable.

A

But so I realized that only in conference talks are all changes. You know purely beneficial like that. So a lot of time we have to make choices that are a little harder, a little bit more complex, but you can still use this this methodology here. So what? If you have one part of your application or one component, we decide to run more of them. You know a higher degree of parallelism.

A

Well, maybe that allows us to process more work to increase the size of that funnel proportionally, but maybe that also just pushes the bottleneck to the next stage of our system or what? If we didn't, even need to go that fast? What if the cost, the monetary cost, the cost of having more nodes more replicas outweighs the speed benefits that we get we're still under slo. So is it really worth it?

A

You can kind of answer these questions, no another, one, that this is kind of a canonical example. I'll use for this is we were evaluating different compression algorithms. We wanted to change our compression algorithm and, let's say we change from algorithm a to algorithm b and algorithm b. Is faster to compress and decompress? That means that our rights and our reads are faster.

A

That's great right, I mean who wouldn't sign up for that, but maybe we're paying for costs in different ways. So now it turns out that the compression ratio was worse, took up more space on disk and guess what we're paying for disk, and so it's a kind of a complex dance that you can make to be able to understand these things, but you can instrument them to be able to answer those questions to be able to transcend. Sometimes the just the engineering aspect to be able to understand the case.

A

The business case of is this: the right change or not.

A

And so ultimately,.

A

I think it's an iterative monitoring driven approach not just towards remediation but towards development. That really gives us a lot of benefit here, and I find myself increasingly using telemetry for this, as opposed to debugging problem scenarios and instrument. First right ask questions later. I guess.

A

So thank you for coming to my talk, everyone, I'm obviously a. I have a lot of good things to say about prometheus.

A

It's definitely empowered me and changed the way that I think about and develop software, but I'd love to hear any questions that you have or anything that you're using it for come grab me in the hall and I'll. Take questions now I'll. Take a pity. Clap too. You know that would work yeah.