Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Trace Driven Development: Unifying Testing and Observability - Ted Young, LightStep

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Trace Driven Development: Unifying Testing and Observability - Ted Young, LightStep

This talk proposes a fundamentally new approach to tracing and monitoring: Trace Testing. We discuss testing against trace data rather than code, and how this technique opens the door to an entire suite of verification practices which are difficult to achieve with the current unit/integration test dichotomy. We show that Trace Tests can span across multiple network calls, languages, and services, while still retaining unit-test-like ability to observe fine grained internal behavior. This combination is particularly useful in mircoservice and serverless environments. In addition, Trace Testing ensure that behavior which is important enough to test in development is also observable in production. We also propose that formal proof logic – which can catch new classes of bugs and help with live debugging – is not nearly as complicated once you have distributed tracing.

To learn more: https://sched.co/GrRF

A

Welcome everyone: my name is Ted young I work at light step and I completely geek out on distributed tracing all the time. So to start this off, everyone could help me just raise your arms in the air and wave your hands. I'd see a bunch of waving hands. That's right, because what this whole talk is this whole talk is a hand waving exercise so I want you all to participate in this with me.

A

So I'm gonna be throwing things up here and you're gonna, be like oh, my god, and just like relax just kind of roll with it. We're gonna kind of like get a little vision e here, so first of all, distributed tracing, I kind of presume. If you're coming to a trace, driven talk, you already know about distributed tracing. So I'm not gonna, dwell on this.

A

If you haven't seen distributed tracing before, please pause the video now and go watch a video on distributed tracing and come back if you're part of our live studio, audience- and you don't know what tracing is just think- logging we're talking about logging, there's an architectural model here.

A

The part that I'm interested in is this trace data, so, whenever you're exporting your traces out of your system, that has to come out in some kind of data format, with some kind of data schema and I want to explore some really interesting things you could do if you took this data schema and had some standardized way of doing it, something that was comfortable enough. You were used, you would be something that was standardized enough. You would be comfortable reusing this in other environments, like your development environment.

A

So what could you do with this structured data? So you've got traces. Traces have many spans. The spans can be decorated with rich semantic. Information like this is an HTTP client. This is a database call things like that. So imagine a data model like this. This already exists. It's just not standardized necessarily, but that's what we're gonna be working with.

A

So how could you take that tracing data that your system is Purdue, saying and start feeding that into your testing models during development? So the traditional kind of test flow is usually some sort of before test after flow you're setting things up. You're executing some code to test your code and then you're resetting things afterwards and we're going to switch that to a slightly different model. It's going to look fairly similar, but it's slightly different.

A

We're going to define our model, we're going to define the tests that we're going to make then we're going to gather some data and they're gonna check that data against our model.

A

So, let's imagine for the rest of this we're building a bank and for example, one thing you do with this Bank is you want to ensure that accounts can't withdraw more money than they contain right, I get! This is probably not how you actually build a bank but bear with me, but let's write a test for this, so we're gonna hang on this slide for a minute, because you're gonna see this slide. A lot now again do not freak out about the syntax on the screen.

A

There are many many syntaxes you could choose to do this style of testing any syntax in any language that is functional and somewhat declarative. I would argue would work for this and that's actually going to be an advantage that we're gonna see going forwards, but this basic idea is so in this particular case, we're going to create a model. We're gonna create a particular example in this model, which is accounts, cannot withdraw more than their balance.

A

So we're gonna say when you see a span called fetch balance and it has a tag called amount, and that tag is less than the amount tag on the withdrawal span.

A

Then you expect to see a rollback and you do not expect to see a commit, and you also expect to see a API level span account withdrawal and that should have a status code of 500, so that's kind of what we're defining to be the behavior for when you try to over, withdraw your account in this particular Bank and then we're going to check that model against some test data. So everyone with me here, like very basic test. You could write this test in a variety of ways.

A

So how would you then go about gathering this test data? Well, the first thing is what about a unit test? You want very fast feedback on this Bank API, so you might set up your an instantiation of just this in code.

A

You may attach a mock database to it, so we're just going to create a mock tracer and a mock database and create a new running Bank API, with that tracer in database you're, going to tell the mock database to hey, always return 300 for the balance and then we're going to run the withdraw command on bank server for 500 and then we're going to gather the test data and then we're going to expect this model to pass if that Bank actually refused to commit.

A

So if you had not written code yet to do this math, you would expect to see a commit you'd, especially this test fail. Then you write that code. You pass this test. Okay, so we're working in our little unit test environment, but everyone knows unit tests. You know these are very, very isolated testing environments, they're, not very reflective of reality, so you always want to cross-check your unit test with your integration test. So what does an integration test? Look like well now, you're in a local environment, spinning up real processes right.

A

So you spin up your bake server. You spin up a real database, but still like a local, maybe in memory database, something you can actually run on your laptop. You spin up a trace collector to collect your information. You populate that database with some test data, and then you run your test driven by the API client. So you spin up a client, you say get the balance, then you try to withdraw twice whatever that balance. Was you got back and then you fetch your trace data and have a look at it? What does that test?

A

Look like huh that's funny. It looks just like, though the last pest I wrote, it hasn't changed at all. The only thing that's changed here is what test data did I get back. This was test data from an integration test rather than test data from a unit test with mocks, but the actual test itself looks exactly the same.

A

I think that's powerful, but let's take this a step further to really see how powerful this might be, because that integration environment, you know that integration environment is better than your unit environment, it's more like your production environment, but it's not exactly your production environment right, like production is like this huge honking thing. You can't possibly run on your laptop.

A

There is a database there, but it's not this, like local, my sequel database. You are using in your integration test, it's some kind of cloud DB as a service thing that someone else is running for you, just massive crazy thing: you're running lots and lots of bank api's lots and lots of clients, there's data and things flying everywhere, there's all kinds of crazy stuff happening, so your integration environment, it's kind of like your production environment, but it's not really and we all know, there's gaps right.

A

This is why we're monitoring in production to begin with right, because you know it might be different from what you were doing in development. So what can you do here? Well, one thing we're doing in production is we're collecting a huge amount of trace data. Assuming we have distributed tracing turned on in production. You've got some large-scale tracing.

A

You know, SAS collector, something like this thats collecting traces, perhaps they're sampled, perhaps or not, and what would it be to test whether all the assertions you made in development are still holding true in production, even though you've kind of changed your production environment from your development environment? You would really like to know your assertions are still holding true wow. That's interesting same test.

A

I didn't have to change anything about this test to test against the stream of production data, we're just taking our model, we're running it against the stream of test data, and it's just running assertions on that test data over and over and over again nothing has changed. It's literally the exact same test, so I think this is crazy.

A

You can look at remote lis executing code and test it the same way. You are testing locally executing code, and this is the power of testing with data, because what we've done here really is I've kind of pulled a trick on you. This is generative testing, so if you've ever done quick check or any kind of generative testing anything kind of that test environment, we end up with this sort of purely functional testing.

A

It ends up looking kind of like this, because you're testing against data you're really just doing data analysis at this point, so any data analysis tools you have you could bring to bear against this tracing data and isn't that basically just testing.

A

So let's take it a step further. What about formal specifications? I know this is a kind of temporal Lamport logic and if anyone has ever seen this- but this is an interesting way of ensuring that your distributed systems are actually consistent functioning the way you expect them and it's really good stuff, but so hands who's ever tried to do formal verification of learning system. I see like one couple hands like how much fun was that was it hard? Was it easy.

A

Yeah exactly the first time you go to do this and I would argue it's like I know the squiggly syntax looks weird, but I guarantee you the first time you try to do this. It's not learning the syntax or learning how to do for formal verification. I would argue this, like the fun part, it's like literally. How do I do this like if you go, try to set this up in a day you will be so sad.

A

You will just be so sad because, like what is this tool chain like what is this stuff you're miles away from your finish line, when you try to do this, you have some kind of crazy, toolchain you're trying to set up. You have to learn a bunch of new syntax. You have to define your whole formal logic, all of this very basic modeling stuff, that's very far away from your application and then you're, like cool I, verified this formal model, but wait. What does that formal model have to do with my actual running application?

A

How do I ensure that my application is actually the same thing as my formal model? huh Okay, I guess. I could do something there, but it's gonna be pretty ad-hoc. I kinda have to make it up and then once you've done all of that work, then. Finally, you can start making assertions with that and perhaps fixing things and I would argue the big problem with some of these more quote/unquote advanced techniques, fuzzing generative testing.

A

Certainly formal verification is that it's just it's just such a slog to get to the point where you're actually getting value and doing work. It's just just huge backlog of stuff. You have to do.

A

But what, if you did this on top of distributed tracing? Well, you know, modeling from scratch is difficult, but distributed tracing is a very simple standardized model right like it's, not a model that you have I see something like ant, but it's much simpler. It's a much simpler model than say a programming language. Your programming language is very complicated model, but traces with spans, with references to each other, that's fairly straightforward.

A

If you've got standard ways of explaining what an HTTP request looks like, if you can start relying on what some of this data schema is going to be, then you can start building a lot of kind of standardized. You know proof logic and things like this, like what does retry with back off? Look like you can just model that in a standard way, and then someone else can come along and pick up that standard model and because the data coming out of your system is standardized.

A

You can check it against that without having to do any of that scut work yourself, so we can all start kind of sharing those part. The part where reality is separate from your verified model. Well, you're gonna run this against your actual code.

A

Like you can now I mean there's certain parts of model checking the formal verification that might be difficult to run against actual code, but you can certainly create a much stronger feedback loop than you could in an environment where these things are totally separate and it's kind of ad hoc and the tooling the environment. All of that stuff, you know it's a it. It's the same thing you were doing before so like. Let's have a look at this right. It's a group actually did this as two sets of slides, my bad.

A

So here's the visual version of what I just said right you've got the simple standard model. Reality is extremely similar to that model. You verified that, because you've extracted your model from your reality and the toolchain. It's the same thing. You are already doing like what we're seeing here looks more like generative testing than formal logic, but I would argue that's much more about you know. What are your primitives that you're working with in this language?

A

Is it less than or is it something like temporal logic, primitive, but I believe you could write this in a manner that looked fairly familiar to programming programmers without a lot of like math, syntax, essentially so I think that's that's something that's worth checking out and now the starting point is much much closer, much much closer to where you were before, like if all of that toolchain. If all of that work is standardized, you can go straight to making assertions and fixing things and I think that's really powerful.

A

I think there are some issues with this approach. I, don't think it's perfect necessarily, though I'd love to tell you that it's perfect and there's no problems. The two that come up for me that I think are maybe problematic, certainly involve some work. One is if you're going to test your code based on the tracing data. That represents your code. How do you sure that that tracing data actually represents your code? How do you ensure that, if you have a trace point that says X happen that that actually is a one-to-one relationship with X happening?

A

There's a little bit of work there I could see small unit tests or something like that. You know some kind of binding glue to ensure that, but shouldn't we be checking that anyways. If we're going to be relying on this data to monitor our applications, wouldn't you want some assurance that what you were looking at in your monitoring data did actually line up with reality. Isn't this a thing we should possibly already be doing the other issue that I think is.

A

You know a little tricky whenever you come to something that's sort of purely functional or side-effect free is the the data you can use only comes from the code you're running. So if you're going to be writing tests in this style, you can't sort of go around back and then check in the database to see you know.

A

Did it really withdraw the amount or not that's a Spile of thing you you couldn't I mean you could do that, but you would lose all of this benefits of them being able to go from unit testing to integration testing, to production testing. So there's a sort of style of insuring that you can assert everything. You would like to know about your system based on what your system is actually doing without adding an additional non system component to this test.

A

So I think there's a little bit of trick work that we need to do to get kind of comfortable with that.

A

But at the same time, if you're doing that kind of work that that's just more kind of artificial things like when you're in production, likewise you're- probably not going to be able to use those kinds of tricks. So.

A

Why does this matter beyond? Looking like a neat trick?

A

What why does this really matter from the perspective of observing your system and running it and I would argue, it's really important that we start merging our development practices and our monitoring practices right now we have this where we go develop code, and we have all this tooling that we use to develop that code and then we throw it over into production, and then we want to see what it's doing in production and we use a whole different tool chain to do that, and we don't test that tool chain in development.

A

We don't utilize that tool chain and development and I have found over the years that that means that tool chain has a high variance and quality is perhaps the polite way to say that, like any code, you at the end of the day, you need to write code in order to monitor your system is true, you know, automation can help, but it can only help so much at some point. A human has to get in there and describe what's going on so another human can look at.

A

What's going on and understand it and like any code that you ship that you don't test or you know, have any kind of feedback loop as you're writing it like the quality, is bad. If you were to ship production code and not test it and not really exercise it while you're developing it like how good would that code be when it runs not great, and it doesn't surprise me that you know logging metrics, all of these things mmm they tend to.

A

It tends to be hard to figure out how you ensure that it's quality data and I- don't just mean you know, logging too much or not enough, but just making sure that what you're putting in there is going to be useful to help you understand what your system is doing when you're trying to figure out what it's doing in production, so quality always suffers in this case.

A

The flipside is like well, why does this help development beyond being a cool-looking trip? The main problem I see with developing code and trying to ensure that it's correct and bug free when you ship it into production? Is that all your test environments are artificial?

A

It's just not reality, and if you could test against reality, I would argue you would do it. You would test against production if you could and I would argue that we do test against production. It's just the crudest possible test. You could imagine like smoke. Tests are so crude. They are. They are so coarsely grained in what they can tell you, just like. Maybe there's kind of something weird is about all you can usually get out of a smoke test s, ellos and alerting. Those are basically tests.

A

You're writing tests if latency stays above X for Y amount of time, send me an alert that it. That is a test, but it's a very, very crude test, often done through some kind of GUI thing right, as opposed to like programming and text where we can write things that are very nuanced in a very dense style.

A

So if running your tests against production wasn't hard I think you would really do it and if you were triaging a problem and you're trying to figure out which hypotheses you should pursue, you would really benefit from being able to quickly test those hypotheses.

A

So you didn't write the test before, but as you drill down without redeploying anything or doing anything in development, if you could continue to massage and change the assertions that you were making against your production code, I think that would be a very, very handy tool to have in your tool belt, while you're trying to dig into a problem, and, lastly, I think you know. The main point to take away here is that we've been talking about trace data like trace data is something special, but really trace.

A

Data is just data traits driven development is really just data driven development. There's. No reason why you can't take these style of testing that we usually use to test for a logical execution in our code and write the same kind of test against it. Sort of aggregate data that we collect in production, resource usage, latency outliers error rates and anything going on any sort of aggregate roll-up, and you want to start making assertions that that you know has some covariance with something else. These things are not straying from each other.

A

This is not ever going negative when I have you know ten instances, I expect resource utilization to be X on this process when load? Is why those styles that style of testing I, believe you could do that against all kinds of production detection tools that we like to use at the end of the day, we're just analyzing structured data, and we have a huge body of data science out there.

A

There's all these tools out there in so many different languages and environments for analyzing, structured data, and if you have a standard model of structured data that semantically describes what your software is doing in production. That is a hugely powerful toolkit.

A

I think that is more powerful in many ways from these more kind of Alucard things that we're doing, and so at the end of the day, this is just to get people thinking about how we can start applying data science to observing our software in a way that not just making pretty graphs but actually writing assertions and creating hypotheses and testing them.

A

Ok, so what's next? What do we need here? Well, we need trace data. There are a variety of schemas out there, but they're you know. Proprietary and air quotes in the sense that they're often tied to a particular implementation. You know light step. Has a data schema x-ray? Has a data schema?

A

You know, Jagr is open. They've got a data schema, so that's stuff, but you know like do you really want to have your whole development tool chain tied you like the x-ray data schema and how are you going to get things locally to produce stuff in that schema? I, don't know. I would like to see something standardized here. I think that would open the doors for a lot of these things.

A

Conveniently one other thing I've been helping work on is the w3c trace context, working group? This is a group I'm sure. There's people from this group in the room I see Sergey over there trying to define a wire protocol for in band context propagation. So you know basically standard HTTP headers for distributed tracing so that when you see traces coming in from another third-party system, you could continue that trace in your system and the user who's running.

A

Your collective systems might be able to take trace data from the tracing system in the first leg and trace data from the tracing system in the second leg and then glue that trace data together to get a more complete view of the system and I've sure have been saying, trace data a lot when I was describing that problem. So another thing we've been starting to you know. Work on in this working group is this trace data issue because having that standardized I think would be really great.

A

Okay, that's what I got next step to me is just a call of action. So if you're interested in this kind of stuff, you can find me outside later I think we've got some time for some questions but I'm going to keep updating on this something I'm gonna keep working on. If you follow me on Twitter and tell me about what you're up to I will retweet it I would love to see people start playing with these kind of trace based testing interfaces.

A

I would love to see more experiments with these more powerful forms of testing temporal logic model checkers, another fun, and on that note, I think we've got a couple minutes for questions so I'm gonna open the floor. Oh.

A

Yeah yay, yay, Vee and yeah. All of you could be interested in this.

A

Yeah I mean I think this is a fundamental issue with observability and privacy. These are like a union yang that data is in your monitoring system, if you put it in there and it's an aspect of privacy that I, don't think, gets addressed directly too much, but it is something we do need to really think about.

A

Yes, scrubbers things of that nature, but I would I would suggest that that may be doing. This style of testing doesn't necessarily make that problem. Worse though, we might notice it more.

B

A

A

Yes, what data you actually propagate from service, a to service, B I, think people just have to be smart and selective about what they're putting in there and that's just on the developers of those systems to do that. Yeah I got a question. Thank you question back there shout.

C

A

So I think the question was: is there you are seeing a leap between the the model that you get from distributed tracing to the kind of model? You would need to do more formal analysis, yeah I, think that would be an interesting exercise to do, because if we can't do this kind of analysis with this current semantic model of distributed tracing I would say that indicates a flaw in that model that needs to get fixed.

A

I believe you can, but I also know for a fact that there are rough edges around this current distributed tracing model. There are a number of things that are not present in it at this time. That I would like president for just even more basic monitoring. You know the the concept of a service that these transactions are ingressing and egressing.

A

Out of is like a sort of a ghost in the system in this current distributed tracing model, and that's something I would like to fix, but I think that's something we could iterate towards and I think this would help cool and I think that was me getting played off the stage cool well, I'll be hanging out outside and I'll be at the light step booth Oh afternoon, and if anyone wants just find me there.