Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Navigating the Distributed Systems Execution Maze With OpenTracing - Ashlie Martinez & Ilya Kislenko

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Navigating the Distributed Systems Execution Maze With OpenTracing - Ashlie Martinez & Ilya Kislenko, Kasten

As distributed systems have become commonplace, the need for organized, easy to parse debugging information from these systems has become a necessity. However, adding tracing in distributed systems presents unique challenges such as associating related trace information across processes and services, making minimally intrusive changes to add tracing infrastructure, and deciding when enough tracing has been added. This talk discusses the adventures and resulting battlescars the engineers at Kasten obtained while adding OpenTracing and Jaeger to their Kubernetes system. By the end of this talk, listeners will know what results to expect from adding OpenTracing to Go projects, understand some of the gotchas associated with tracing, and learn some of the differences between tracing with services meshes only and using a tracing library.

To learn more click here: https://sched.co/FuK3

A

Hello, everyone welcome to all the tracing talk. My name is Celia keys, lenka I'm software engineer at custom and.

B

I'm Ashley Martinez I'm a first year, PhD student at the University of Washington and last year. I was an intern at Caston. As you.

A

Can see I'm presenting on behalf of Julio Louis, who is not without who is not here today. He is about father of the newborn baby and spending time with his family, but I was helping to develop this talk and also work on the LEM up. So we should be good in this. Talk will describe our journey, how we were aiding tracing into our existing kubernetes application and let's do it at Castille, we're developing kubernetes native application called km. This application helps to manage stateful applications in communities and it followed in its micro services.

A

As you know, maker services I complex and really hard to the bottom. So our journey was the tracing started when fuel follow tests starting to be flayed him and we were getting some random timeouts, which were wasn't able to explain. So we spend a lot of time looking for the logs and metrics, but nothing was there. We thought that 13 close to that micro services would be easy to debug with the only metrics and login, but we were proven wrong to debug microservices. You probably need the whole tree factor.

A

Do you need logging unit, metrics and gimme tracing I thought, that's how we decided that we need it, and the thing with in our case was that we already had an application which was working and we needed tracing yesterday, and we wanted it really quickly and we didn't actually want to disrupt our development process or we write out the whole application to start supporting tracing some.

A

That's. How was our journey and we'll describe in this talk, what we need, how it worked and what the result? Let's look at the kubernetes ecosystem and what's the actually kubernetes native application, looks like if you look at the github, you will see that most of the kubernetes native applications following micro services Partin most of them return goal also, most of them using TPI, first approach and also most of them using code generations, because it saves a lot of time and it's just easy to do things to make.

A

This talk a little bit harder for us, but a little bit more interesting for you. We develop and assemble app called image application and we were trying to make it in the cloud native fashion, which means it has two back-end micro-services image catalog, which is metadata store. It talks to the database and faces all the information about images like passes with leaves in some metadata.

A

Also it has image store or the store service, which saves images, data on the local, hard drive, both microservices written and goal, and we're using swagger to generate our HTTP json api s-- services and the client. If you curious he's the guy coupling up with open source, it's in helm, there's some bar script, which helps you generate everything and it should just work. So I believe all the demos representation is supposed to be entertaining. So, let's start with the demo first and then we'll talk about it and also, I don't believe in the demo gods.

A

That's why we will be using video.

A

Let's open our application and see some beautiful pictures is VP if you're not familiar, it was created by matthan bailey and I heard the rumors there, the second part of coming. We can see the pictures. We can see the slides everything. Fine, everything works. What does the developers we keep changing our application and in the following update we're going to introduce really cool new feature code, random slowdowns, on the Fitch image from the storm, which means every time when you're reading it might be super slow or super fast.

A

So we upgrading our application via helm, because helm is amazing, if you're not using it, do it upgrade done, let's go and check how our application works.

A

We hit refresh and nothing there's any other normal user. You hit refresh again then you're getting scrolling. You waving your mouse you're getting frustrated, nothing is loading. Loading super slow, so I would say them for the really simple picture gallery. It shouldn't take more than a few seconds to actually load everything. In our case it takes longer than that some test developer. You have to go, and actually the bucket is them creator of this beautiful up. Trust me. You will not find anything. There locks all metrics and just some random timeouts but good for us.

A

We actually have tracing an hour up. So let's go to the tracing UI and see what it does. This is Jaeger and, as you can see, it has information about our two services metadata and store awesome. It shows traces. What they are I will describe a little bit later, but trust me they are good. You can see some durations. Some of the trace is slow, some of them faster. Obviously we want to find we sham slowness and where it is to do that. Let's find all the traces slower than 10 seconds is one of them.

A

As you can see, this is the get request to metadata and it takes 14 seconds which is super long. Here's our trace, as you can see, trace, has pretty much chain of actions through our application. The first one we get in income and get requests into the store to get image with some UID for our image, which triggers internal functions are actually fetch image which also slow the next will be find image, which is relatively fast at last. It only like tens of seconds, which means our database behaves properly and it works fine.

A

The next request you can see outgoing and incoming requests into the storm from the metadata and the final one is actually read: data from the local store, which is still 30 seconds. Probably we should go to the store, metadata and see what's happening over there. Another great feature about tracing and Jager in particular, that it didn't give you diagram of dependencies for micro services. As you can see, here's our two microservices metadata in store and that's how would they work metadata?

A

Pretty much calling the store for the images, so it was our up even that it's pretty simple only to microservices, but without tracing to find that slowness, you probably would spend a few hours because they don't need to compare logs from the different microservices find the same time stamps and if, for example, your microservices written in two different languages, it would be even more complicated, metrics bonus if you using only metrics, also will not help. It will show you a some requests hitting a threshold on your timeout, but not much info with the tracing.

A

It was relatively quickly because we just went through the whole chain of actions and we see what's happening. So what is distributing tracing on the high level? You'll be surprised, but it's also application. It's sometimes one application, sometimes two applications working together. What they do they automatically aggregate traces, we yup sense to them to collect them and doing some correlations.

A

Also, they highlights the execution part of the request. They helps to pinpoint failures and where the slowness are curved, and this is not a replacement for login or metrics, but it's super complimentary to them. So what you already heard, the few times I mentioned, trace, span what they are relatively simple. Think about the trace as the chain of actions. When you do something to the application. For example, you fetch an image.

A

It start the chain, the first one is you remember: it was the code to the metadata, then the code to the database, then the code further further further so pitch. Imagine is like all the actions which happening through your application for particular action. What the spends spends, actually the small steps or the functions so as you can see on the diagram application sends traces with multiple spans to the jäger. Here is a developer. That was me. Sometimes it's like 1:00 a.m.

A

trying to the sent way it's slow, why it's not working, but most of the time I just go to the tracing. Do I look at the things and fix them what we using and why we're using it. Since we're developing cloud native application, we decided to go to the scene CF and find what we can use from them to tracing into our application. We, like two things, think number one open tracing which is actually standard with them is the case and they're pretty much. Every popular language like go, Java, C, sharp and sorry.

A

Those is the key is helping you to instrument how you send traces into the distributed tracing collector and the second one is the Jager, which is the visualization application, which can read. Data from the databases allows you to do. Searches can do comparison, as you can see, helps you to create the diagram, but there's not only open tracing in the Jager in this world. This are the offers. If you don't like these two, for example, this Zipkin there's Google open sensors. Some cloud providers also offering you some facing technologies like, for example, AWS x-ray.

A

At this point, I want to hand our presentation to the Ashley the person who actually added support into our application. All.

B

Right thanks Ileana, so, as we just saw in our demo, we got a nice overview of open tracing and Jager and kind of how they work. But for now let's dig into more details onto the open tracing go SDK itself, as you noticed in Ilias demo, each trace is made up of several different spans, which are nested inside of each other. To show the execution path of the program open tracing accomplishes. This nesting by leveraging goes context object which it expects to be propagated throughout the entire program.

B

What this means for you as a developer, is that if your program doesn't currently propagate goes context, object, you'll have to do a little bit of extra work. If you want nicely nested spans, like we've, been showing you finally open tracing, allows developers to add extra information in the form of what they call tags. This information can include things like the status code from an HTTP request or other stuff.

B

Like request IDs, this information helps give developers a good idea of what the state of the system was when the span was collected, as well as helping them associate specific spans with the output of other tools like logs.

B

So with that information in mind, let's move on to actually adding tracing to our demo application, we'll begin by looking at our architecture, diagram and deciding that we want to add tracing for incoming requests to our image catalog and our image store services. If we open up our image catalog service, we can see that it's actually made up of three main parts.

B

The middleware, which is goes way of adding custom features to the incoming HTTP stack application code, which the developer actually writes and does interesting things and httpclient that helps push out requests to other micro services in the application, or perhaps outside services like databases or cloud providers. So, given that we want to add tracing for incoming requests, we decide that we should look at the middleware area here.

B

For this, we've created our own little custom middleware, which leverages open, tracings library, calls to begin and end a span automatically for us now. I said that open tracing actually uses contexts to store information about spans, but on this slide you see no context object whatsoever under the hood. Open tracing is actually extracting a context from the incoming request, starting a span with it, adding the modified context back into the HTTP request and then passing it down to the next middleware in the stack.

B

So when we compile and deploy this modified application to our cluster, we can see that we only get one lonely little trace, which is not really what we were hoping for from our architecture diagram. We can clearly see that our image catalog calls our image store service, but the span that we just collected does not reflect this in order to try and fix that situation, we decide that perhaps we should add tracing to the outgoing requests from our image catalog service.

B

Once again, if we look at the zoomed in view of our image catalog, it seems that perhaps instrumenting the HTTP client for outgoing requests would be the best place to start for adding tracing.

B

We decided that creating a custom, HTTP transport, which is part of the go stack for adding custom functionality to outgoing requests, would probably be the best solution. Here you can see the actual code that we've used. We first pull the context out of the outgoing request and have open tracing begin a span using it. Next, we place the modified context back end to the request and have open tracing inject special headers that it uses into the actual request to be sent out.

B

Once we send the request out and get a response back, we extract the HTTP status code from it and place it in a tag so that developers can look at it in iaeger later. As a reminder, the first time we added tracing to our application. We only got this one span. However, when we compile and deploy our newly modified code, we get three lovely spans clearly showing the connection between our image catalog service and the store service that it communicates with.

B

However, we know this is still not the complete story, because both of our services talk to some sort of store. The image catalog talks to a database and the image store talks to the local hard drive to retrieve image data.

B

So we feel perhaps we should instrument these calls as well, and we can see that we actually have two instrument developer code in this case, instead of some of the more generic code that NGO uses to handle HTTP requests, but before we get to the code, perhaps we should take a step back and ask ourselves since we're communicating with external services. Can we always just use the trick from our previous part of adding tracing where we create a custom, HTTP client?

B

But it turns out that not really it really depends on which API you use to communicate with other services because go has a wide variety of API sinews.

B

For example, we found that the kate's api doesn't take context objects, which means that, even if you do try and use the same trick from part two, your application will not create nicely nested spans, like we've, been showing, furthermore, DB with the API. The way that we're using doesn't actually take gos context objects. So there's no way you could use our little trick from the last part.

B

However, all hope is not lost. For example, the official AWS and GCP go api's, take both custom, HTTP clients and gos context objects, so indeed, using our trick from the last part will allow you to add tracing to those services.

B

Well, add tracing to the calls to those services. You won't get insights about the internals of what happens in those services. So with that, let's now move on to the code that we added to trace calls to our MongoDB database. We decided that the best way to add tracing is to add the three highlighted lines on this slide, the first to begin a span from the context that is passed in and then finished that span when their function returns.

B

The final line just or the final line in the set just adds extra information to the span so that the developer has a better idea of what was happening once we compile and deploy this to our cluster. We get a lovely view of all of the different spans that correspond to the actions occurring in our application. The first set the first three correspond to actions taken by the image catalog service and the final two correspond to actions taken by our store service.

B

So I know that was kind of a lot of information in a short amount of time. So let's do a brief recap of just adding tracing to our system. We saw three different ways to add tracing and when those ways might be good to use for incoming requests, we decided that using open tracing middleware functionality is probably the best choice. On the other hand, if you have outgoing HTTP requests, you should think about creating a custom, HTTP transport which begins and ends spans for you.

B

And finally, if your instrumenting other parts of the application you may want to check out the api's to see if you can use other tricks from the previous two for incoming and outgoing requests- or you may have to manually, add calls to begin and end spans.

B

Okay, so just a brief recap of everything that we covered. So far today we talked about using open tracings, go SDK to add instrumentation to micro services. We also saw how to instrument external services with open tracing, and these external services include databases. The Kate's API and some cloud provider calls earlier in the talk. Ilya talked about installing Jaeger with a helmet art, and he showed us a lovely demo where he used Jaeger to analyze and dig into some of the traces and now I'm gonna hand it back to Ilya. To finish off the talk.

A

So we just saw that racing actually helps debugging and understand. Where might be error as long as I could but I believe some of you probably still not convinced that we should use tracing. So let's try to talk to and read your minds.

A

So the first thing my login is amazing: I don't need tracing I can search through the logs compare logs time stamps, just amazing, probably, but we thought the same and we learned the hard way that probably it's better to use tracing if it's possible, if not and you relying only on the logs and metrics it's up to you but trust me.

A

It's a little bit hard because logs giving you a point in time for your application, when the trace is actually giving you small, tiny story line, how what was happening to your application or what was happening to the some actions or the requests and your application. So, even if you only rely on login I would recommend at least your try tracing I'm, getting SMS for majors, first of all, I hope, if you're using kubernetes you're, not using majors and you're, using something something more more than as the monitoring technologies.

A

But even if you're, relying only on the metrics and you don't want to add tracing into your application and you think, you're good, aiding tracing will give you ability to do more request. Sculpt metrics, like for example, you will get less amount of false alarms, because you can I mean you can strip your notification a little bit request specific, for example, not everything like you can skip health checks. For example.

A

It will be easy to do and you can really focus and trigger your alarms on really business, important things and requests, but my service mesh has it also true statement. Some sort of smashes like SEO, offers you tracing out of the box, but it's not deep enough in my point it might work. But if you want to go deep, you need to have additional instrumentation. What I mean by that is to actually only tracking the requests which come in between your services or in or out from your service mesh?

A

If you using some k-value database locally to store something, you will not see anything in those traces and if you're, using, for example, slow hard drive for some reason, you might see some performance issues and you won't. It will not be able to debug them through the tracing without some additional instrumentation for your application. So, yes, service measures does it and if you okay to using it, please do it my cloud provider. Do it for me for free, also true. Some cloud providers offering traces.

A

For example, Google cloud stack driver trees and AW x-ray, if you're using cloud provider integration with these solutions, almost seamless and super easy, but don't forget if you look in the whole kubernetes ecosystem right now, everyone most of the companies trying to use, not one cloud. Multi clouds, also don't forget about on-prem or hybrid clouds, and if you married to one cloud provider, you probably will not be satisfied which we require you to use some other solutions.

A

But if using only one cloud provider and you can use whatever they provide- that's perfect final thoughts, we're so tracing how it works, what it does, how it helps, but don't forget nothing comes for free in this life if you adding something it first requires time. Also tracing uses a little bit more narrow, not much, but some also because it's some additional CPU cycles.

A

It's also, if you really short on the CPU, it might affect performance, but slightly so be cautious and think about time and efforts and if you actually need it or not, but I highly encourage you to try tracing it's. It's really interesting thing at this point: we're ready for questions.

C

So sometimes I do the pollinating, so I send the south. Thousands of has that I know I want to find out the biggest ally repress them, which is acceptable. Okay, right now,.

D

C

To find one it's best, because the best.

A

C

A blind eye not to find that because it's a list of thousand over there, so my hope is the hawk I, get it okay,.

A

The question was: I can repeat, the question was what I supposed to do if I'm sending thousands of requests to the jäger, how I'm doing search, let me show you the UI and I can answer that. One rile them up will help us.

A

Give me one second, the pretty much in giggle you can search not only the duration on the request you can search on them. C'mon demo gods be with me.

A

Here we go as you can see, you can search for the text, something like HTTP exit code. So, for example, if you know that you're trying to find traces which failed you probably not returning something like 200, called you returning 500, which means, if you're looking for only those specific traces, you will see your faders also, if you've structured traces properly, you have many tags on them and you can search by the text.

A

C

Mean actually most of them are 200, but I want to find a specific, oh yeah,.

A

You can use on the scene, the text you can use HTTP dot issue yeah.

C

You are, can I use that your URL or professional URL I can I I can put a time step or in my in my in my URL and yes, yes, yes, yes,.

A

Yes, you could gig, it allows you to search by Ashley, pointed yes by URL, okay,.

C

I put is such a time step but to the attackers connoisseurship, yes, yeah.

A

Second, it still that one than that, so one two three four.

E

Things straight up: you said it never comes my fries Kev's a bit more word about benchmark result like how much overhead as to it and also does this a poor conditional tracing, because something might end-to-end request timeout, like ten percent America time. Well, then I like toggle the tracing all and something like that. Don't.

A

B

So we didn't do too many benchmarks, because we were just trying to get tracing running on our system for our testing purposes. So right now, it's kind of in our test cluster, but it's not deployed with any of the services that we give to customers as far as conditional tracing. There are ways that you can write your own version of how to turn on tracing, so we've been tracing. Basically everything that goes through our system. You can have a probabilistic tracer, so it will only trace a percentage of the requests that pass through the system.

B

There is also a way to create your own function that determines which trace or which requests should be traced. So if you were to implement one of those functions, you could perhaps monitor your system and say: oh I, see things are failing. Please turn on tracing for me.

D

I have a question that say: I have a drawing applications that is writing in different programming languages, services, writing, go and so speeds readiness, plus we still use of interest here or best practice to handle this kind of situation. You know aspects it.

A

Depends how you're, using it, for example, open SDK, was sorry open tracing is decayed. They do have libraries for goal. C++, Java, C, sharp salt, so.

D

The JavaScript, whatever you mix enough, oh yeah,.

A

Yeah, at the end, you getting everything into the same system into the Jaeger and UI will be the same, but traces will look the same and, yes, you can send them from the different languages. Okay, sure that's gentleman over there.

F

So here we show the example how to chasing the HTTP, so I. Think in a is it possible that we can chase message into the message message: queue such as repiy and kill the recent message in our system. If you come into the messaging Q&A arm and then go out, the messaging queue is t2. Is it possible then another incoming message, I think you are now outgoing, so.

B

You want to trace the amount of time that it's in the message. Messaging queue. Is that what you're, after yeah.

F

Yeah we have a message broker, so I would like to chase the messaging and kinetic Asians which subscribe to the to them message broker or produce the messages as due to the broccoli. Yes,.

B

So if you keep some sort of context object around with the data that is actually inside of the queue, you should be able to begin a trace when you place the object in the message queue and then perhaps once you pull it out of the message queue, you can end the trace.

B

So it's it's more of figuring out a way to do it nice and easily, with the existing and finding the right places to begin and end your traces, because really what you need from open tracing to actually begin and end a span or begin an end. A trace is this top line which starts the trace and this second line which finishes the trace. So as long as you have those two calls somewhere in your code, you will collect a trace.

B

The key is finding the right places to put those two lines: does that kind of make sense or.

F

Means that we need I need to start chasing in the producer and then something I've finished the chasing in the consumer. Yes,.

B

That would be correct. Okay,.

A

Sorry, a little bit addition so pretty much if you can add trace ID from the content into your consumer and producer and extension between them. Some additional tag for the message it will automatically be populated because the distributing tracing works on it has trace, has ID and the span has trace IDs and then it's automatically can collects everything and combines them together and that's gentleman over there I believe.

G

Well, what's your experience on a great increases with logins without Japanese resistance, so what man way, how what's your experience, our greeting, logins and Tracy's, without a jamming through all the systems you connecting the traces and logins? That's.

A

B

So we mostly I believe use traces kind of as an overview of the system.

B

So traces, traces kind of give you parts of the information that you can use to look up specific log messages, perhaps say you have fluent D or some other logging service, which you can insert tags into the logs as well. So you could use the tracing system to find the request that is causing trouble in the system. Like say, your failed request or your request that timed out and you can look at the tags inside of that request and based on the tags.

B

If you have like a request, ID or something that you've inserted in there, you can use that tag to kind of look up specific log messages. So it's you use them in combination, but one or the other is not necessarily all you need to solve. Your problem also.

A

I can I can add a little bit more. If you able to use Google cloud provider, they not only troduce tracing, they actually introduce really cool features like walks to traces to source code. If you whole source code on them and they do it automatically, it's super cool, but unfortunately, only so it's in Google, Cloud I, don't know if it works for you or not.

H

My my feeling is yeah Chase is important and love is important, but the things if I, because of if I, would like my systems of Microsoft space, then it wouldn't need a lot code.

H

Extra code to pre enable its function to make the work land. It means lot of a workload. All the course is leilani how to sleep and in the future, for example, to make simple, like combination walks with the message bar, so the waster log to correlate. So, let's make the code change as less as possible.

A

Sure do you pretty much one at tracing as easy as possible? What we will do, I, don't know your specific in the application. For our case, it was a little bit easy. We were using code generators and pretty much for us. We just edit tracing in one code in one template and that's it that was applied everywhere. Once we generate the code, that's it.

B

The other thing is, you can look for high impact areas, for instance, we traced incoming and outgoing requests, which actually gives you fairly good insights into some parts of the system, and so based off of that, you can figure out where you want to add tracing next. So the important thing to realize is that you don't have to trace the entire application to get data out of it.

B

You can trace just specific parts of it figure out which parts you want to know more information about and then figure out good ways to add tracing to just those little parts.

A

There's no magic peel. Unfortunately, it's all depends on your use case. Yes, thank you, hey you so much everyone, sorry, but we are out of time already I'm. Sorry, we can talk after if you want to thank you.