Cloud Native Computing Foundation Observability Practioners Summit 2019 (San Diego), 29 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: LightStep Sponsored Session: Observability for Deep Systems - Spoons (aka Daniel Spoonhower)

Description

Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

LightStep Sponsored Session: Observability for Deep Systems - Spoons (aka Daniel Spoonhower), LightStep

Software architectures have evolved: applications are not just getting bigger but scaling deeper. Observabiliity tools must adapt to this new environment or leave developers with lots of responsibility but little control. I'll describe deep systems and where they came from as well as the opportunity that they have created for observability practitioners. All this in only 10 minutes!

A

All right, yes, good morning, I'm spoons, aka, Daniel Spoonhour and the CTO, and one of the cofounders that light step. I'm, not gonna, talk too much about light step. If you haven't heard of us, we are simple observability for deep systems.

A

I am gonna talk about observability, not surprisingly, I'm gonna talk about deep systems too, which I'll describe in a minute and that's because I think deep systems are really important to understanding how if we should think about observability I'm, not sure, there's a lot of observability skeptics in the room, but for those of you that are already all-in and observability, maybe or I hope that you can find something in this talk to take back and convince others on your team or in your org. That observability is pretty important.

A

My marketing VP made me put this slide in here, but I maybe should go without saying. I did want to remind people that there's a huge impact that observability can have on our users, whether that's understanding system, health, improving latency, reducing MTTR, just responding faster and all of those things have real effect on business outcomes right, whether that's happier, users more efficient code, just more productive teams, but wait what 70% seems like kind of a lot.

A

Actually, we see this kind of thing pretty frequently like there's a lot of room for improvement, and it's not as though this particular code is terrible, and this is this is a really smart person working really hard. So there's something sort of external I think that's happened and I think to explain that we need to zoom out a bit and try and understand what makes you know. 70% performance improvements, pretty common to do that.

A

I want to talk about two different kinds of scale: I talk a lot about scale, I'm sure you all talk about scale, a lot to Becca B's. You know scaling with more users scaling with more data scaling. Our teams may be scaling ourselves, but these two kinds of scale that I want to talk about our wide scale and deep scale. So as an illustration, this is a small body of water, and this is a larger body of water. So we take a puddle.

A

We had some more water, we keep adding water, we get a pond, maybe eventually a lake, an ocean, and this is what I mean by scaling wide and the thing about scaling wide is, though, an ocean is a lot bigger solutions, scale and sort of the same way as the problem. We just need a bigger boat now, in contrast, here's another example. This is a village, it's an organization of people, their work, their lives and here's the city.

A

This is what I mean by scaling deep, and the point is that a city is not just a bigger village. It's a different thing, so the way that I navigate a city, the way that I find information that all right, the way that I find other people doesn't just take longer. I, just I need new tools to do that so to step away from these kind of abstract examples.

A

Let me talk about software for a minute, so this is what software looks like when it scales wide and you can think of these as sort of embarrassingly parallel cases.

A

So trivial MapReduce is an easily Chartres double data store, and these do happen in real life, but from an observability point of view, they're pretty straightforward, but if you're adopting micro services, maybe you're in a server list, maybe you have an architecture that looks a little bit more like this services that depend on services the dependent services where each request is broken up over and over again, and that's why each of these services can focus focus on one thing and do that one thing well, of course, part of focus often means delegating part of that responsibility to other services.

A

If we look at real software systems, I took a handful of light steps users and looked at what their systems look like, and we see exactly this kind of pattern right. We have these layers of services built on top of each other and I think this is a fundamental change to the way that we're building software systems in particular. This means that micro services at scale aren't just wide systems, but there are deep systems, and this is the kind of complexity that affords the opportunity for these kinds of 70% performance improvements and I.

A

Think it's really important for observe really, but before I do that I want to just mention some other ways that you might know that you have a deep system, and that is what a deep systems sound like. What would you hear if you have a deep system? Well, you might hear someone say: don't deploy on Fridays, you might hear someone say where's Chris I'm dealing with a PC. huh No one else knows how to debug this. You might hear it can't be our fault. Our dashboard says we're healthy.

A

You might hear this. You might hear this. You might hear this. These are all the sounds of uncertainty right. The lack of knowledge about how the software was written, a lack of information about how its behaving in real time and what's happening. What are you feeling? What's your team feel in this in these situations? Well, you're feeling stress right stress can be defined as responsibility without control.

A

So if I draw the system diagram here again as a service owner, what you control is really your own service or maybe services, maybe have a handful of them, and these are the things that you can deploy new code. You can roll back right. These are the these, where you're at the driver's seat.

A

But when it comes to performance of your service, you depend on all of the services that are below you and so in terms of a responsibility that actually goes all the way down the stack and even, if you're, not at the very top of the stack as a service owner, maybe you're somewhere in the middle, there's still a whole set of layers below you, and this is really what causes this dress right this this, this difference between what you can control and what you're responsible for.

A

So how? If deep systems change this well as a service owner, what you can control has stayed more or less the same right.

A

You still own, just this one or two or maybe three services, but as the depth of your system, increased the scope of your responsibilities, got bigger and bigger, and not just linearly bigger, but because of the number of services that you depend on because of the number of services that those services spend and and so on, because of the interactions of all those services, the gap between what you control and where your responsibility lies, is growing geometrically, and this is I think why we need to think about observability in a different way, because the point of observability is really to shrink this gap right to provide the information to give users better control to let them understand their responsibility.

A

So what does success in this case? Look like? Well, we can go back to our users right, so success means that our users can quickly understand the health of their platforms. They can find the bottlenecks they can understand. Latency they can respond more quickly, they can improve MTTR and they can. They can deliver improvements in performance in latency.

A

So if we think about what that success means, it's really about these use cases right so as observability practitioners, how can we help our users be more confident when they release new code? How can we help them respond to incidents more quickly, so they can achieve less downtime? Maybe it means avoiding instance altogether. How can we help them optimize their code to provide a speedier experience to their users and telemetry is the raw material that can help answer those questions.

A

One of the things that's really exciting to me about open telemetry is that we're finally have a way of talking about logs and metrics and traces all in a unified way and well. More telemetry is often better I.

A

Think it's important to think about how that telemetry is addressing these use cases, how it's answering those questions, part of the reason that the responsibility curve is growing so quickly is that teams find themselves often responsible for monitoring and understanding the performance of all of those other systems below them, and that's not something that's realistic to expect in a lot of cases even feasible. So in these deep systems there are just too many signals and part of the role of of an observabillity tool is really to help identify which of those signals will matter.

A

Okay, so just to kind of sum up, so microservices scale deeply, and it's not just micro services, if, like I, said, if you're using server lists, if you're building on top of hosted services, anytime you're, creating layers upon layers where there's a separation of ownership, a separation of control, you're, really creating the opportunity for a deep system.

A

Deep systems are not just bigger, but they're they're different as well, and that gap that they create between responsibility and control is really what leads to the stress for users in these cases, and finally, observability has to help shrink that gap and I don't have a lot of time to get into it now.

A

But if I'm happy to talk about it later, if you grab me, I think that traces are really critical in providing the context for those kinds of use, cases and problems again, finding the right signals so that we can help users navigate to the solutions they're looking for quickly.

A

So I just wanted to quickly plug and remind everyone about the reception tonight, starting at 5:30 just across across the tracks at The Whisky house, please bring your veg I'll, be there I hope to see others there as well. Thanks for your attention, I'm looking forward a lot of good talks today and thanks.