Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 8 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tracing is For Everyone: Tracing User Events with GraphQL and OpenTelemetry - Nina Stawski, Splunk

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tracing is For Everyone: Tracing User Events with GraphQL and OpenTelemetry - Nina Stawski, Splunk

There's been a lot of talk about the importance of observability and tracing for microservice-based applications. The use cases involved are focused on backend engineers and DevOps. But what about us front-end engineers? Often, we get blamed first when something breaks and the lack of consistent observability tools makes it difficult to debug issues. With the emergence of OpenTelemetry for JavaScript, more front-end developers are looking to instrument their code and connect their traces with the backend. A growing number of teams are adopting GraphQL as their interface between UI and backend as well. This talk will illustrate the process of setting up your app for tracing with OpenTelemetry, show what’s common in GraphQL instrumentation compared to other libraries and describe the potential pitfalls of the approach. Building on that, we will discuss how tracing affects user experience.

https://sched.co/ZeiJ

A

Hi everyone, my name, is nina stafsky and I'm a ui engineer at splunk.

A

My talk is about tracing and I truly believe that racing is for everyone, um and I will explain you how to trace user events with graphql and open telemetry, so you're, probably asking what a ui engineer is doing at kipcon and what does ui has have to do with tracing well. Anything happens with your product.

A

Ui is the first thing that the customers see. Ui is the first thing that fails and um the customers are always complaining that this button is not working. This page is not working. This tab is not working, so ui is always blamed first, no matter what happens if things are not completely failing, but they are a little bit slow. For example, it's also really hard to track, especially um to figure out whether it's a ui or something in the back end, something with the microservices so having some tools that help you do.

A

That is really useful. uh Of course, we have front-end analytics uh the most popular is google analytics, uh but sometimes it's not enough. You have to go dig in into logs. You have to go.

A

Look at microservices, you have to sometimes write custom, javascript or snippets just for you to figure out where the issue is happening and while ui is struggling with trying to figure out what happens and where the issues are back-end engineers already have really good tools at their um disposal and ops also have been looking uh forever at logs at different uh different services that help you figure out where the issue is, but for ui, while you do a console log, you look at your google analytics.

A

You look at your inspector to see the performance, but it's all a little bit disconnected and finally, um the products are getting more complex and front-end is not just a service anymore. um Front-End can be broken into different microservices built with different languages and it's much harder um to debug issues. If you have multiple different languages within your front-end application, maybe they they are stitched together uh and something is slow, but you don't know what, and yes we love microservices. So how can we make it work better?

A

How can we trace it better, and so this is an example of a stack that I use at my daily life. We have an application that is basically stitched from angular and react and we use graphql in apollo libraries to make sure graphql seamlessly works both in angular and react and in the back end.

A

And another tool that we use is this amazing project uh called open, telemetry, it's fairly new, it's only uh so it's in beta now yeah, it's only about a year old, slightly more than that, and I got to tell you a little bit about it before I go in a little bit more into how to use it.

A

So open telemetry is a combination of two projects that existed before: open tracing and open senses and together uh they provide a set of um tools that are uniform that are provided in one place that lets you collect your traces and metrics about your application. Again, it's usually used in the back end and a good thing. There is also project called open, telemetry gs. So I was some time ago. I was really excited to find out that there is already a library that lets you use it.

A

It's mostly geared towards node users, but you can also use it in the browser.

A

So far it's been kind of disjointed, so you can use it in your ui um and then you can use it in your back end, but correlating the traces is a little bit of a pain.

A

Still it's very exciting right in two big areas where open telemetry gives you tools are traces and metrics, and traces is a series of events that relate together that represent an end-to-end event flow through a distributed system, and you can also apply it to the ui in some way. So for back end, it's usually a request flow through multiple microservices.

A

This might be your request that goes from your ui um and you're, requesting a user info going through a middleware that queries. Another service that queries a database and the whole thing end-to-end is a trace.

A

But again you could also speculate that for ui. You can create traces um that are a little bit bigger than that.

A

And finally, metrics is a numeric final matrix is a numeric representation of data, measured over intervals of time, for example, a number of requests per minute, and so in a traditional approach. Ui is just one microservice here it is, but ui is just much more than just a service.

A

You can have a lot of um user actions that happen. You can. Users can go through the page for a long time before they actually click something that triggers a back end and it's very good to be able to track that users do a lot on on the modern application pages um there might be not just one but multiple backing calls and, as I already mentioned, performance issues are really hard to track, uh especially with multiple uis, multiple back and com calls, and um to be able to identify where the issue is so far.

A

Your engineers had the same tools at their disposal as backend and ops. Five ten years ago, right you go to the logs. You um try to figure out what is going on.

A

And so how are we trying to solve this problem right now again, open telemetry.

A

A

And so how are we trying to solve this problem right now? Well, open telemetry helps not just back-end engineers uh in ops and data scientists. It helps ui engineers, so let's switch to our sample application that I'm building to be able to correlate some traces from ui in the back end and to simplify it a little bit. I want to use both angular and react. I will just use react. I will use graphql and apollo libraries to simplify things a little bit and open telemetry um and our application will have a ui with some actions.

A

um Something would happen in the ui and we will capture these actions using the browser. Then we will send these actions to the graphql backend, where we will basically look at the spans, and we record this information and we send it after that to zipkin collector and we'll collect traces in zipkin.

A

And if you want to follow along or do it at your own time, here is a link to the repo, it will also be there at the end of slides, so feel free to go ahead and take a look all right into the wild. Let me share my screen now.

A

A

And in that repo that I just shared, I have uh two applications that oh we'll go through. So let me start my terminal and um there are two parts in that repo. There is a front-end application, that's built in react and there is a back-end application.

A

um So for front-end uh it's a react and typescript, and I have this uh used. Telemetry hook that I built that's gonna, be um posting traces into um two places. Actually one is we want uh these traces to end up in zipkin um to be able to see them there and another one is uh we'll, be exporting the traces um into the back end when um to be able to do something with them in the back end and then finally have my back end. Application is just javascript single page, um that's gonna!

A

Basically, that's running a polo server on graphql server and on a new request. It would actually retrieve uh my spans that I'm sending using headers. So there is a special header called graphql current spans uh and you can see.

A

In my application, that's graphql, but I'm gonna send graphql current spans um to the backend and so on. Every query that comes to the backend. I will retrieve uh through headers and I will uh create span with span attributes that contain trace id and a parent id. uh So we will post them to zipkin as well. Oh, so we would be able to correlate those.

A

So now, let's start my applications. Let's start back and first.

A

All right, uh it should be ready, let's check it uh and I once be actually yeah, let's check if it works.

A

Yes, we have our graphql there. There will be no traces because we're not sending anything from the front end and then we're in start now we're gonna start our ui application.

A

And, as you can see, it's a simple one, there's just a number of buttons um and um there is a button that will send some requests to the back end and the button and parentspan will just wrap up all spans and the ui and will not send anything to the back end.

A

So here we are trying to capture user actions and of course it doesn't have to be button clicks it cannot. It can be anything, um but let's try that here. So I clicked for buttons- and I clicked my get finished spans.

A

So actually my backhand should have received this pants and my you, I should have- should have sent those pants too.

A

Okay, so we got our spans in the back end. Let's take a look.

A

All right in tags here are is the traces um and parent ids that are propagated from the ui, but let's go and see if we got our ui traces.

A

Oh it doesn't, uh it doesn't always work as expected, but let's, let's see okay, let me start the ui.

A

A

Actually, to be able to post any traces from the ui, I need my course enabled. This is what I think didn't work for the first time. So, let's try again I'm clicking a few buttons.

A

I'm doing get finished spent. So now we actually got this response from a ui. Let's look at them all right. We got our button id, so we can provide metadata. uh There is a trace id and a span id here.

A

Same same in here, okay, so let's go back and click again, so this happened less than a minute ago and this is corresponding back in traces.

A

That actually has a parent id.

A

um And trace id so let's take a look at the parent id, so let me write it down somewhere.

A

600., basically, just yeah create a new new place to put my id.

A

And go back and look at my ui traces.

A

A

And you can see that they are identical, so we uh we can connect this data um and we can basically correlate the ui and the backend.

A

So let's go back from screen share to the slides and again um you can take a look at the github repo. um It's a little bit more comprehensive than what I showed right now and you can basically share this information with multiple different ways. So what what I did is I assign tags to my spans uh that that's with her parent id and I provided this parent id in my back end that I picked up from my spence sent from the ui.

A

um You can create more tags. You can send in actions, so you can do a lot more with this information so yeah what we can do, we can track user actions and we can go through maybe different user scenarios uh that we're looking into so in the ui uh it's easy to. So basically, if you're, uh you have some flows in the ui that you users need to follow um and you want to connect them to the back end.

A

At certain points you could do or you could provide these fans um and store them, and it will be a lot more detailed than just having this ui item on uh on your list of traces.

A

Then you can also try to track these actions and track back-end requests together and provide this information uh in uh have backend and the ui be a part of the same trace.

A

um You can basically time what's going on in the ui in your ui, so in zipkin we saw or that there is some timing already available for the spans.

A

And it will help with performance, especially if you have multiple front ends and finally, if anything uh caused an error, you can look not just at multiple different services, uh not just multiple logs. Not you don't have to solely try to look into chrome inspector and try to figure out what backends are working and what doesn't work for you. You can um export these traces into one of the multiple solutions that's available now um on the market and basically just visually see where issues are happening and how they are um impacting your user experience.

A

So it's not just the ui now and of course, this is a limited set of libraries uh this this is a ui with open telemetry that gives you so many more possibilities than before.

A

So that's it. um Thank you very much there. Us uh there is more details in the github repost. This is a short link or you can go through um it's github.comtelemetrygraphql, and this is how you can contact me. Thank you for your attention and I want to open the floor for all right, hi everyone. I hope you enjoyed the talk I can see. There are a few questions already uh and there is one um I'll answer right now and then I'll uh look at the other ones, so um someone's asking what is a span?

A

Well, you can think of a span as a unit of work that has some data attached to it. So let's say we have built a website that uh has a book catalog and we're searching for a book or something so retrieving book and four from this catalog from the ui um kind of the information about this action would be your trace.

A

But uh your request on the way of retrieving this and for might go through multiple, uh multiple different services right. The ui sends an api call to your backend. It might need to authenticate it might need to call a service that uh will actually return you. This book info, the service in turn, will call the database to get info about the books in uh so kind of it. It takes a few hops on the way. So each of these hops is a span and each of these funds might have some information attached to it.

A

So how long did the operation take? How um what what else happened? To? Maybe name of a service um anything? Actually you can. You can actually also add these tags manually, as you sew we did with uh when we posted traces to zipkin from the back end. We added some more uh info, so this hops on the way is a span and the whole thing is a trace right. Let me take a look.

A

All right, so let me share if people are asking about the repo, let me move to the last slide.

A

So there is a short link and a long link um both back end and front end are in the same repos, so you can use that.

A

Okay, so one more question: oh let me let me actually make sure I have too many questions going.

A

A

All right, let's start from the top um to use the same trace id for the uh so kevin libya is asking use the same trace id for the ui spans and back ends service bands. Wouldn't it be better for the ui to publish its own spans instead of sending them via their request header instead having the ui propagate the trace context uh to the back-end server. uh This is a good question. This is actually what we attempted to do in this combination of front end and back end.

A

As you saw, I was posting um my traces from the ui and I was also sending them as headers to the back end, and I used those uh to basically tag um with the trace id tag. My traces uh that I bought from the backend.

A

um The challenge with the ui is, of course you need to or you go when I run it on the local. I run the course plugin and I'm just posting my traces, but you need to authenticate. You need to make sure it is secure, it's harder to do from from the browser. You can still absolutely do that, but it will require a bit more work.

A

All right um one more question, um so: okay, um hamish robertson, hopefully I'm reading your name right- uh is asking you have to manually correlate the your actions in the backend traces.

A

um Well, we were doing that manually, of course, but you can do some automation on top of that. So if you use the same tags on the ui traces and the backend traces, there are solutions out there that let you basically filter by these tags and see these traces together. So you don't have to do it manually every time, but again in this small test repo we were doing it manually. Of course,.

A

All right, uh nikola bianchi, is asking whether open telemetry sets of libraries or just a standard. It is.

A

It is actually both uh it's a standard and it's also a set um a set of libraries or four different languages that are implementing the standard and if you go to github, slash open telemetry, you can learn more about that.

A

Yeah, oh one more thing I wanted to say about that: it's it's actually open telemetry is pretty exciting. It's open source and any contributions matter. So, even if you've been like, if you've been thinking- oh, maybe I uh want to do something for an open source project or maybe you're, just interested in tracing. Take a look at open, telemetry and any contribution even docs, anything it matters, it helps grow. The project helps take it forward. It helps make it more useful for people so contribute all right.

A

um Danielle manel is asking. How can I extract this data from tracing to a more dashboard oriented information?

A

Well, there is a lot of different solutions out there that let you basically, as you collect your data visualize it in a more comprehensive way than just uh in zipkin.

A

Well, I'm I'm from splunk, so we have our own open, telemetry implementation, um that that shows you pretty comprehensive dashboards and lets you identify issues within your system pretty quickly and well, there's other solutions out there. But, of course, I'm from splunk so take a look at splunk on microservices ibm all right in the real world, uh so vignesh rajya sakaran, uh hopefully I'm pronouncing your name. All right is asking uh in a real world application.

A

How do you pick which requests to trace uh since monitoring everything might not be performant in uh not necessarily not always necessary? uh That's true, and it would depend on each individual application in each situation.

A

Kind of well yeah that true answer it depends. um It depends on, of course it is. It is also somewhat similar to what you would trace with google analytics. Maybe right uh you have. uh You can specify some certain user scenarios that are important for you and then you can attempt to trace those. uh So um maybe look at it from a point of view. Okay, what would I track?

A

uh What user scenarios, what user actions would I track with uh google analytics and uh then kind of apply it to um what what you track, what you trace uh with open telemetry, it's very individual. I just um yeah it's hard to say all right.

A

Funny! Question trinivas uh convertor says what is it asks?

A

What is the difference between front end and back end, um so I'm calling front end, uh basically the ui side, your or your application, that user interacts with and I'm using back-end as a basically collective um description of all the services, my or microservices or services that are behind um that that this ui is contacting or um like every this big blob behind it um and your ui not necessarily contacting all the backing services right as in an example, as I explained earlier, um with the book catalog, your ui would talk to the authentication service.

A

It would talk to some backend servers that has that will return you a book info and then this backend service in turn also can contact the database so kind of um you have this and it's a simplistic view of a ui engineer um by having basically front end separately in the back end as this blob of info all right, andre stroud is asking in our own environment. We generate our own correlation id at the edge uh and then we pass it to downstream services. Oh, it's jumped um one second to downstream services.

A

For the time being, we use this to correlate logs across multiple microservices.

A

Is it possible to use a correlation id as the trace id.

A

As far as I saw in up until m3gs, uh you can provide your trace id. um So when you create a span, uh you can provide some metadata about that uh and one of the items that it can inherit is a trace id. But for that you need to send a span that already exists. If you don't have a span, your only option for the trace id is sending it as a tag, at least so far. Well again, it's an open standard.

A

It's all open source if it doesn't work for you, feel free to contribute and suggest the updates. uh But basically, in your case, uh what you could do is you could send your correlation id as a span tag or if you already have your spans uh with a correlation id as a trace idea. I assume you don't, but if you did, you could provide this span as a parent span for all the spans that you are creating and it would inherit them automatically.

A

All right, uh vincent despawa, again, hopefully I'm reading your last name right, um asking whether graphql back-end already pre-pre-integrated to open telemetry or did I have to instrument it myself. So in this case I did instrument it myself, um but you. So there is two ways um you can talk to your graphql back-end right. You can um make you can establish a websocket connection or you can um basically send http requests uh and for http request.

A

There is already a plugin for a open, telemetry gs, so it can be instrumented automatically for websocket there is no plugin yet, but it's in development. So hopefully soon um you will be able to use that.

A

For this example that you saw in the talk, it was all manual instrumentation. So I basically just yeah. Did it manually.

A

All right, ph badaria is asking: are there any performance impacts that you might be aware for open, telemetry java instrumentation, I'm not familiar with open telemetry java instrumentation?

A

If you meant to ask about javascript, of course, if you're collecting your traces in the ui side and you're, especially if you're posting it um into zipkin or somewhere else, it will impact your performance somewhat. So there is always a trade-off, um but uh yeah you well.

A

There is always a trade-off, so it it is kind of preferable for now um to post your traces from the back end and that's why kind of also part of the stock um happened because well, as you saw, I'm basically just uh sending my traces through headers to the back end and then back in, has a lot more resources to uh connect uh and collect these traces and post positive traces.

A

A

A

I do not see any more questions, um so if you have some, you can post them here or I will also be in slack um in the cloud native computing foundation slack at the channel 2 cubecon observability, and you can message me directly. So I'm happy to answer any further questions.

A

Thank you. Everyone.