Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro + Deep Dive: Jaeger - Juraci Paixão Kröhling, Red Hat & Prithvi Raj, Uber

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Intro + Deep Dive: Jaeger - Juraci Paixão Kröhling, Red Hat & Prithvi Raj, Uber

Jaeger is a distributed tracing platform, providing a way to monitor and troubleshoot transactions in complex distributed systems. In the first part, we’ll introduce you to Jaeger, showing what it is capable of. We’ll deploy a few microservices into a Kubernetes cluster and use distributed tracing to help us see what’s going on. After this brief demo, you’ll learn how to engage with the community and how to make your first contribution to the project. In the second part of this talk, we’ll explain in details what are all the moving pieces and how they work together. You’ll also learn about the different deployment strategies and advanced configuration options. And finally, we’ll have a Q&A session at the end with project maintainers.

https://sched.co/MPkv

A

Okay, wonderful, so welcome to the Jaeger project introduction. My name is Judith SEPA Sean, Crowley I'm, a software engineer at redhead, I work on the key ally project on the distributed tracing team, so I work on a mission to tracing components and I'm, a maintainer on the Jaeger project and a collaborator on open tracing and as a maintainer of Jager I'm. Here talking to you today. So this is a double session, so we have the project introduction and right after that we have the deep dive.

A

The plan was to have someone from uber here delivering as well, but he's got some visa issues, so he could not join us here in Barcelona, so the plan now is then to have like 30 minutes, introduction, section plus five or ten minutes of Q&A and then some five minutes of likely small break. So if you already interested only in the intro part, you can I skip the second part and attempt something else and as well for other people who know about Jaeger already, but refer only to watch a dip dive.

A

So we have a five minute break between those two blocks. Right then- and of course I'll be here, so you can still ask questions after that and for the second half we have it very similar, so half an hour for the dip dive and the remaining of the time then, for questions and answers.

B

A

Okay, so it's very hard to talk about eager and not talk about tracing or distributed tracing. So here is a tracing solution, but what exactly is done? A disability racing? So, if you've been to any other tracing talks this week or perhaps some open telemetry talks, then you've probably seen some introduction to tracing.

A

But for those of you who are still new to tracing, we use that to understand what our application is doing right. So we have Micro Services or we have some distributed system and we have calls between the services and we want to visualize individual business transactions as they flow across our infra.

A

So imagine that we have one HTTP request: reaching our franching service and as part of processing that HTTP request to make some database calls and we then go to an external micro service and that micro service goes through yet another micro service and so on and tracing then shows you the picture of that request that original request, as it flows across all these three or more micro services, including database, calls that we make in between right. So and in summary, this with the tracing.

A

It tells us the story of a request as it flows across all the services. Now we need by tracing- or we use tracing in general to understand problems right so to do root cause analysis. So we see that we have a specific or we have problems with our system across a specific time stamp or time range, and we then filter all the traces or all the requests that affected that service or that that machine on that time range, and then we can take into each one of those traces and understand what happened there.

A

So, as you can imagine, this is very useful for network latency or to visualize if we have unnecessary calls and things like that, and the way it works is actually quite simple. Right. Tracy works by propagating context. So when we call from one service when I go from one service to another, we serialize our context in one place and we sent this context along with the payload to the next service.

A

In HTTP terms, it typically means that we store the context as a HTTP header on the outgoing HTTP request. So if we are on a micro service- and we are calling the second one- we just add another HTTP header on the other side, with this serialize and whenever you create trace information.

A

He's asked me to ask you to go to the meet off there because we have people standing there yeah. Thank you.

A

Okay, thank you very much right. So then, how does it work? Then? We serialize context on one side and on the other side on the second third. Fourth level: micro-service. We look up the HTTP headers and if we have a header with the context with this realize that and whenever we create new new trace information, we use that as our starting point now. The context here might be a little bit too abstract, but typically it just means an ID right.

A

So we create an ID at the very first point of touch of in our system and we propagate that ID downstream. So that's basically how it works, but we are not here to talk much about tracing in general right. We are here to talk about the Jager and our final goal here, with Jager or with tracing in general, is to have a screen like this.

A

So I'm running now is a very quick demo, showing you what there is and how it actually works and that yeah so I have here or a hat here, planned to run a demo using East geo, because you know you still, but, as you can see here, some things are not working. So I have a backup, backup plan here.

A

So I have three micro services. They are built using a new Java framework called Quercus. So Perkins is a framework for cloud native job applications. I'm not here to talk much about the queries, but perhaps I should just say that applications built using queries. They are very small and they compile binary to machine code. So there you can just execute them and place them on docker images, so they're very fast to boot and consume very little memory. So that's key to called native applications right so that we can scale very quickly.

A

I have them a second micro service, also built using car keys and a third one.

A

A

Looks like it's working and I'm, then opening the eager website and I'm just copying. You know from from jäger I we go to the getting started page, which is this one here and I'm just gonna copy that docker command that we see there into mr. pubb's, which is this terminal here and I'm, not copying the ministry, because we want to see all the logs.

A

Okay, so Jager is running our three micro services. They are running as well and I've been just. Can you guys in the back to see the code here, the different okay, cool and I'm? Just then running like three HTTP requests to the first service.

A

Now we don't know this application and we know pretty much nothing about what it does, but we can use eager to visualize what happened right. So I have here two traces array that I created before, but those are the three new requests that we're in right.

A

So for each request that we have, we see all the data on one trace and I think it is now too small for you, Desna back and now we have here all the individual operations that were executed as part of that business transaction right so that initial HTTP request reached our service or on as a get request on it, slash so HTTP localhost 8080 slash as they get request.

A

So we see that information here and that then called the submit order operation- and this is something that I manually added to my code, this operation here so I'm able to do explicit instrumentation for those who know about what instrumentation general and then we have another automatic span here. So each line here is called a span in tracing right and that represent a represents a unit of work.

A

So this is basically okay right. So we see all the the information about individual requests. We don't only have this view here the timeline view. We also have the graph view, so we are also able to see what what is what are all the operations that were called as part of this HTTP request in a graph.

A

Do we see each call? Each service has a color, so we see that there are three services here being touched. It's a little bit difficult to read when you know when you have a lot of nodes here, but if you over one of those here, you see more details of what that's been, we can color them also by time. So we see which spans to most of the time- and this kind of view here is also the base for comparing traces.

A

So if we have one same trace and if we have one error trace, we can compare both or we can compare even more traces and see what are the differences between them and it uses the dif syntax. So in this case here we don't have any difference. So it's great right, but if we had spends only on the a side not also missing on the B side, it will be read following the diff colors and the opposite, meaning be having more Spanish than a would be colored in green.

A

Yeah- and so this is basically a here in the UI part right- it sees what it is. The final outcome, so once you have everything working once you have jäger are running in my case here as a docker container. Once you have your applications, instrumented in any way that you can submit your application with manual instrumentation with by a framework instrumentation beat by a service mash instrumentation that I wish I could show you here once you have all those instrumentation code in place, you should you would be able to see data here on here.

A

Besides the comparison and search of the traces, we also have a basic graph view or service graph view here right. So this is not a hundred percent accurate when you, when you have sources which are not traced or not instrumented, so I would not rely on that for architectures that you know where you're starting to trace things, because you are then missing points here, but it's still very useful. If you know for food applications, you see what are the dependencies between those services.

A

Okay, so this is basically a very high-level overview of what is here, but so, if you are a developer, what is what is your free right so either? If for developers is basically the the docker image that we just ran when you are developing, when you are debugging issues, so we start the Jager backend components as a docker container, but, more importantly, it is the tracer that you plug within your application.

A

So you instrument your application, you use framework, instrumentation, libraries and, and then Jager, for you becomes only the component that the actual tracer the thing that actually knows how to record data right. So the instrumentation that you use is your description of what should be recorded. What should be measured and the tracer or the client is the thing that actually knows how to do that now, if you don't want to use the other client or the eager tracer, so your client and tracer day, their de name to name is for the same thing.

A

But if you decide not to use the Jager tracer, you can use open senses or in the future, open telemetry and plug a exporter for Jager all right. So you can. You can have the same tracer for application and just change the exporter if you decide to change to or from Jager in the future.

A

Now, if you are an ops person, then eager means something different to you.

A

First, it means the things that you have to deploy, maintain in production like the whole infrared year itself, but perhaps more importantly, or more useful than that is the tool that you use to do the root cause analysis or to visualize what your, what your micro service is. I are doing. Basically, so if you, if you find, if you detect an issue using some matrix tools or or dashboards, you can probably zoom in into individual requests and once you once you, you know, you know that we know that M range, where your problems are happening.

A

You can select all the traces from that period of time and then looking to individual traces. So the Jaeger is your tool for that. You can deploy Jaeger on pretty much everywhere right, so bare metal you can deploy. We have wineries for Mac, OS or Windows, and for of course, Linux C's ya cube con, so I assume everyone here is running kubernetes, so you can of course run your own kubernetes or up or kubernetes distributions like openshift yeah, so the other is actually a. They built itself following the micro services approach right.

A

So we have quite a lot of components here. One of them is very separate from the others. This is what we depend on the situation. We call that the your client in depending we call that the tracer, so the ager client, we typically say when we talk about in in the context of talking to the backend component right. So we have the client in the backend and we use the term tracer. Typically, when we talk about from the perspective of the instrumented application, so your your micro services and then we have on the one agent.

A

So the agent is a is a component runs very close to your application and receives data from the tracer. We have a collector receiving data from either the client or they the agent any stores data into a storage. We have a query or UI and they're both part of the same binary, even though they are technically separated and we have an investor. So the interests are one of the list. Storage.

A

The storage mechanisms for the collector is kafka, so the collector can write to Kafka and in gestured and reads the stream from Kafka and writes to the primary storage so in terms of features that we have for the future. So right now, Jagr is doing very well the collection part of the things right so we'll average open tracing open sensors in open telemetry in the future, for an instrumentation part and for the collection part and then on the back end.

A

We we are very good at storing this data on somewhere right, so in the receiving is area in digesting ingesting this data and historian that summer- and this is somewhere currently- is elasticsearch and cassandra for production workloads, but also we support local storage via badger and in memory storage as well in right, I mean it. The version 112 was just merged was just released and include support for storage players. So if Cassandra or elasticsearch is not your thing, you can write your own plugin and use your own storage and it uses chart.

A

We see plugins or magic or planes, and basically it right there plug in and respond to a certain API and year then asks your plug-in to store data or to retrieve some data, and you you know how to do that stuff. With with a storage that you choose.

A

In this way, then, the eager does not need to maintain a huge set of storage mechanisms.

A

The second big thing that we have is adaptive sampling, so sampling is basically the technique to choose which traces to restore and which trace is to drop all right. So if you have a huge load in your application, you probably do not want to restore all the traces and then we use this technique call them simply. So we decide on its sampling strategy. So a like by the four year comes with a probabilistic sampling strategy of 1 in 1,000, so there is a chance of one more thousand a dirt.

A

Racer are being sampled so being stored, but the whole light the whole when it will be nice or very good to have the backend define the same thing based on actual usage right. So if you have a low traffic application, you probably want to restore everything, and if you have a high load or very critical application, you probably want to restore one a few traces and those decisions could probably be made by the jäger back-end, and this is then what we call adaptive sampling and the third big topic is data mining right.

A

So, as I mentioned, we are very good at collecting the data, but we don't have anything right now to analyze the data that we collected.

A

So if you are interested in this topic of data mining in general, in machine learning in AI come talk to us because we need input, we need feedback and we need to brainstorm this kind of thing and that kind of is a good hook to the next topic. So for those of you been to the keynote this morning, you've probably seen a slide asking for you to contribute to your communities right. So if you use Jaeger, we very much welcome your contribution contributions, no matter which kind of contribution you send us.

A

If you're a coder, you can look for the good first issues that we are going to show you right now, but we we don't need only coders. We need people who help other people as well. We need documentation help. We need brainstorming, help. We meet every couple weeks on Fridays Friday afternoons here in in Europe, where we discuss not only the problems that we are having right now, but also things that we want for the future.

A

So even if you don't contribute to Jaeger in a more traditional sense, if you have like half an hour to spare on a Friday afternoon, I would really like to see you there on our on our cause and then we I'm gonna show you how to really get started quickly with Jaeger. Our github repo is on the yogurt racing organization and in there we have quite a lot of repositories right. So we have 22 repositories for those of you who know golang it's.

A

You know you have a wide variety of options there to choose from, but we also have the UI repo with a Java Script and all this fancy front-end stuff that I don't understand anything about that. We have a yeegar operators. Operators are also in vogue here, so you probably heard a lot about operators and that we have a cooperator for you to help us shape shape as well, but also if you are not a goal developer, you can help with the clients right. So we have C, plus plus client C, sharp clients.

A

We need help with PHP as well: Ruby, Python, yeah or Java as well of course, and the the best way to start contributing is to enter any of the require any of the ripples that I've just shown here, go to the issues, tab and fight, one labelled good, first issue right, so we have here on the main repo. We have 18 such issues.

A

If you, if you need any assistance in getting started, we are on Gator. We are on the mailing list. You can just pop up there or even here on github issues. You can just leave a comment on those issues, ask for guidance or asking if the item is still free for you to start contributing, and someone from the team would be more than happy to assist you in your first contribution.

A

Yeah, so that's what I had for this first part now I know that those introduction sessions- they are quite rich in questions, so I left. So we have about ten minutes for questions, spend yeah, so I'm ready to take a rest for some.

C

C

Thanks for a very good introduction, so I think a popular question is things like logging versus telemetry or all traces, so auditing, logging and trace is like what is really the differences architectural II and when to use when and I can see things like sampling when logs becomes when that's ready and so forth. Thank you.

A

I'm not find it right now, but there is a very nice Venn diagram that shows when to use which, where I.

A

Can I can show you later basically because I have to look it up, but basically so we have those three pillars or three signals of their ability or the three sources of signals. One of them is metrics and metrics are things that you like. They are aggregate of all, so you can count things or you can store things in a histogram, so those are metrics and that's clear belongs there now logs. According to this Venn diagram, that I have in mind is a logs or event, and those are typically life cycle events.

A

So your application is starting up. Then you record an event on a log file trace is their request scoped, so everything that pertains to your two individual requests should be part of your tracing signal and that's typically called a spend right. So a span is similar to one log entry in the sense that it records it's one unit of work or one unit of measurement, and so that's that's the rule of thumb.

A

So if you, if you think, if you are recording something that is, was coped, then I see I see that as a belonging to tracing and if it is like lifecycle or related to your application as a whole, then it's logging not just to complement that I would not use extensive logging inside tracings right because I think it is. If you have too many, if your traffic is too low, is too heavy, then you kind of have too many log entries inside a span and that's not what tracing solutions were made for that?

A

It's not price. Now.

A

Any other questions so I have stickers for the best questions.

A

D

Hi, thank you for the talk for something. How do you make sure that it is representative of well the load that is hitting your services? What's the second.

E

A

If the, if the tracing is it simply is representative.

D

Of the mill yeah, maybe basically, if you take just one in a thousand requests, if I want to investigate a specific problem, maybe the issue is in the rest of the requests is not in that specific right.

F

A

That should come also with part of so this also belongs just simply in general right and how that is implemented. Then it kind of is different. We have a so you can implement your own same playing strategy and you can you can create the strategies saying that if it is about this endpoint here, then I want to record all the time. If it is not about this endpoint, then I want to record one 1000 or so so you can do that by extending or vice by specifying your own sink playing strategy.

A

I might be wrong here, but I believe that the sampling strategy that we support already so I know for sure that we support unloading as a JSON. Your strategies based on service names and I think operation names as well. So you can say that for all these bands that are named following this pattern here, I want to follow these strategy. So I think this already supported out-of-the-box but I'm, not quite sure about the operation. Name, part I'm sure about this service part.

B

Hi, so just to get that right, the sampling strategy is configurable yeah, but not at run time. At the moment, you.

A

Can change right? Yes, so we don't have the adaptive part, but it is /. You can configure it first service per operation. Question mark. There is a question you.

G

Know my question is about the topology of the calls your farmer can trace the HTTP synchronous calls. Is it able to trace the synchronous, calls using Africa as the middle yeah yeah.

A

Yeah so that belongs to an instrumentation part, that's kind of outside of Iger, so that belongs more into to open tracing or open senses and open telemetry in the future. But yes, the main idea there. So there is the notion this notion of contacts propagation right, so you create these a context somewhere and you serialize that context and say so in case of Kafka. We could send the the span context as part of the metadata of the message or of the Kafka item yeah. So you sent that and then whoever consumes that information.

A

Then this your lies, the the metadata. This relies, they spent context and then, whenever it creates new spans, it says: I belong to the trace idea. Xyz.

H

As you say, always there is HTTP Tracy. Can you also trace some lower level protocols like TCP UDP, like for the next quick or something like this yeah.

A

There is, there is a lot of talk about that, like integrating application level tracing with things like.

A

Cate-Cate rays and x-rays and peach trees, and things like that, some more to this level I think it might be possible I, don't think it might be desirable, it might be, might not be I. I don't have an answer if it is, but the thing is so it could. It could work I just so. The thing is I believe that the the target consumers of this information are different people all right. So when we talk about application level, tracing we are we're talking mostly about developers, are devops looking into.

A

You know, business reasons of why things are failing, or you know, application reasons we do see- network latency as well, which is quite useful. But then, when you go down to that level or when you want to see more information about that, then it requires an expertise which is typically held by another teams.

I

Question about what you see of multi-tenancy options out there: okay.

A

I

Have implemented one way we have yoga instances that are funded with kind of an annual phase, know an LDAP based authentication approximate, and then we share the same Cassandra database on the me with a lot of individual key spaces. I think they're called white so to kind of keep the traces individually separated. What are the yeah lieutenancy? Your managed service, Yaeger options to you see out there.

A

So we have a blog post on multi-tenancy or what are all the options for multi-tenancy when it comes to Jager and I'm, not sure if it's yeah- it's not here on there on the front page. But it's is our medium blog a publication and you can just look at their archives. But basically tenancy is a is a is something that means different things for different people and we we have a constant discussion or an open topic on around multi-tenancy for I.

A

Think two years now, mainly because you can achieve already multi-tenancy, just like you mentioned by by having different instances talking to different our key spaces in the in the back and C is certainly the most popular way of doing a tenancy. But that's one way of seeing tenancy right. So this Tennessee on data level. But then we have a Tennessee on an authorization as well, so probably I do want people to have access to traces belonging to two different tenants or I.

A

Have requirements like I'm in the healthcare industry and I have requirements to have data X strictly split from you know terms between tenants, so I. It is possible to do quite a lot of scenarios playing with how you deploy Jaeger, and it is very lightweight, so your itself is I mean the only one binary is 30 megabytes or something like that. So it's really lightweight. So it is not very costly to deploy a lot of jäger instances on your infra.

A

Of course it costs to maintain those instances, but the the costs in terms of RAM or resources, network resources and so on. They are not very different. Does that answer the question? Okay,.

A

Okay, so we have now then a short break for those of you who want to stay for the deep dive session. I appreciate for those of you, who've had enough for your figure I expect to see you on the community sometime. Thank you.

A

Right so I'm very glad that you are still here. This is now the deep dive session. Do we have people here who were not during the intro session? Can you raise our hands if you were not here for the intro session? Okay?

A

So for us for for you, so you know me.

H

A

For the other guys, there I'm not privy hello, which we could not join us, so he's not delivering the dip dive session I'm here on his on his behalf, I'm jealousy by Jean crony I'm, a software engineer at Red Hat, but I'm here today, as a maintainer on the Jaeger project. So for the majority of you who were here during the intro session, you've seen a slide just like this before, where I listed, all the components that are part of Jaeger but I, I, I'm, personally, very bad.

A

When I see these kind of lists and I don't see how things move so I try to come up with a flow diagram and this whole diagram is kind of show. It kind of shows how Europe might be deployed, but it's not 100%, accurate or on that term. On that aspect, it only shows the components that we had on the previous slide. All right, so we have our application is in the example from front previous session.

A

It is a participation and, as part of that application, we embed a a tracer, a client all right as a part as a library to our application. Now for a job application, it might be that the client might be in the same war file. It might be the same ear file might be in the same jar file for a go application. It belongs to the same binary, but in four other languages, it's very similar right, so it belongs to the package of your application. That's the client!

A

The client is a component that knows the how to record things. So you use of a tracing of entry of instances to tell their what to record, and the client knows how to record things.

A

Typically, when you don't specify otherwise, the client will just create this data, create those pens pack them into batches and send this ball of spans over the wall and expect something on the other side of the wall to catch all those spans and that's something on the other side of the wall is the agent right. The analogy here is because the client sends data via UDP and UDP.

A

We don't have any queries right, so we just fire-and-forget requests so packets to a specific port, and if there is nothing on that port, listen which and that will just get lost or if the other side or if anything happens, to the connection data is just lost now, because the agent is typically on localhost. We don't have much of those UDP reliability issues right, so it works quite well here now.

A

The point is, then, that the agent runs on the same machine, reducing the load on the application itself, and then the agent is responsible for sending this data to the backend, to the collector, in a more reliable way by TCP, perhaps using HTTP on top of TCP, of course, or maybe a RPC or thrift or you know so how it actually sends you say is a it doesn't matter, but it's in a more reliable way and then from the collector we store data into permanent storage, so Cassandra elasticsearch or some are ephemeral.

A

So in memory or perhaps local storage by a badger or even something new, that you write yourself using storage plugins, then once data is only storage. Our current component retrieves data from from the Fonda storage, typically on behalf of the UI, so the UI calls the permian library get things from these to merge and then there is a alternative path which is up there or up there. We have them a Kafka setup or Africa cluster.

A

Where then, the collector, instead of writing to storage, it writes to Kafka and then in gesture, which is a new component reads: a stream from Kafka and then stores data into the storage. The whole point here being that we remove pressure or we make the connection between collector and new storage anymore.

A

We make it more elastic right, so we are able to absorb more pressure now a little bit more details about each of those components, so the tracer typically. So we have the tracers for most of the major languages right, so Java C++, C sharp go. We have tracers or clients for other languages as well. We have Community Supported tracers and what they all have in common is that they support or they implement the open, tracing specification and API soar interfaces. So there is no eager API, so to speak.

A

Right, so your application is only talking open tracing and when you plug the Jaeger tracer, it's able to use your open trace, instrumentation to create data based on that and then there is, of course, the ability to use order, tracers, like open census and in the future again open telemetry, but I technically just not belong to the tracer component.

A

Now the agent as I mentioned, is a component running local to your application, and it acts like a bridge right between your application and a collector. The agent is a local host. What is actually then localhost nowadays, all right. So if you are running on bare metal, that's clear: that's your bare metal! But if you are running on kubernetes, then what is your localhost? Is it your pod?

A

Is it your node and you decide where to run the agent as long as it's you know, on the same on localhost or very close to your application, because remember, the whole purpose of the agent is to receive data via UDP, so it has to be somewhere where Europe is reliable. All right, so in kubernetes is typically the same pod. So it's a sidecar to your applicator plot application and that that's that works very well now, for the cases where it is, you have, you know thousands of pods.

A

That means they have thousands of agents and even though the agent itself is very small, I think it is close to 15 or 20 megabytes in size of the binary. It might still be. You know if you have thousands of pots and it's too much and then you may want to use daemon sets instead and then our own on the collector side. So the collector is a component that receives data from the agent by default, but the clients can also talk directly to the collector.

A

So I can configure my client to use a TCP connection to the collector instead of UDP to the agent. The whole purpose of the collector history receives data. It then cleans up the data. A little bit do some processing with with timings right, because this will this rigid systems, we cannot always rely on the clocks being in sync, and the collector is the right place for us to do some clock skew right.

A

So we are just a timing based on information that we know basically a spend that was reported as a child of another span cannot have started, but before such span right.

F

A

We do some some such logic belongs them to the collector. But if I know, the the main goal of the collector is to write things to a storage, so it receives data image in different range of formats, so it accepts the rain from your proprietary format or in Zipkin format and in the future. If we have such a standard for for this kind of data, we just add a new endpoint to the collector and then the second main goal is right: are those things to a permanent place and then we have an adjuster.

A

So the adjuster is a relatively new component and it reads basically data from Kafka, so the collector one of the backend storage is for for the collector is Kafka or rewrite their main gesture. Just reads this dream from Kafka: we have the query, so the query provides. The critical point provides some rest endpoints.

A

Now those rest endpoints are not documented or not well documented yet, and they are not part of the public API, but we do intend to build an API for you to interact with Jaeger with a Jaeger query so that you can build your own analysis tools and you can retrieve spends that you care about. Perhaps you are also listening to these dreams from Kafka and storing data somewhere. So you know the trace ID, so you could get the trace from Jaeger based on search, ID yeah.

A

So basically it reads from from the backing storage now backing storage might also be your own storage plug-in. So we call your storage storage plug-in by a jar PC, which is asking for data, and you return data to Jaeger and finally, the why so the UI cuts data directly from the query right. So the main consumer of the query is a why, when we have features that we've seen on a previous session here, so you can list a traces we a few traces. We have different types of views, the the Timeline view.

A

We have the graph view, we can do comparison of off traces and we have a very basic service graph.

A

No, those are quite a lot of components right and it's probably complex to deploy and manage all of that.

A

But, as I mentioned at the very beginning of that, at a slide with a flow chart that does not necessarily represent how we deploy Jaeger. So we can deploy all of those components into one single binary, so we have one binary called all in one and it just download that binary and you just run that binary and contains pretty much everything it does not have been gesture or the client tracer, because the client tracer belongs to your application and then gesture is not part of the all-in-one, but pretty much.

A

Everything else is there, so you can just docker run just like we did on a demo, blocker run and one specific image, and it's now when you move to production, though you probably want to at least split the query from the collector, because the collector has to be really high available and highly scalable and the. Why or the query is probably a low traffic application, so it can can be on its own pod with I.

A

Don't know one replicas, whereas the collector probably should be distributed with a lot of replicas, but how you actually deploy that how you you install this component also depends on on your interest, so you can either deploy in a very traditional sense, with bare metal or on bare metal. And then you manage that with system, D files or whatever process, monitoring or process management tool that you choose again, you specify which components you run.

H

A

You can install a Jaeger by a helm, so there is a helmet art for you. You can just specify the chart values and you install the the chart and I have a new Jaeger running there. A new word option for installing yogurt is via the yogurt operator, so you install you can install them the eager operator by a helm as well, and then you can install individual instances of eager via the operator, so you create a CRA a custom resource. Just like any other kubernetes object.

A

Saying I want a Jager I want with a strategy, production and I'm. My elasticsearch is here and then you pass that to kubernetes yeah and operator picks up this new CR and builds a new deployment for you, a new instance of Jager for you with all the right components on the right sizes on the right places, the user can be tweaked or configured I need you. We have a lot of options there and it's very it's pretty much impossible to add all those options in only a couple slides.

A

So what I have here tell you is for every component. We have a Help option as well, so with the binary or the docker image, you just pass the minus minus help option. You have a list, a complete list of configuration options, and hopefully each one of those have meaningful descriptions, but if not then open an issue furnace, but it should be self. So the description should be quite easy to actually quite descriptive.

A

Each one of those options can also be specified as an environment variable following the you know, typical cloud native applications, and but if you prefer configuration files, you can specify a yellow file or a Tom ll file with all the options and pass through the components.

A

Now, if there is in the operator, then you probably are going to configure the instance using the CR right so inside this yard, you specify that my elasticsearch is in this place here and the praetorium takes care of creating the right the right way of specifying that option to the underlying eager component.

A

Yeah now yeah also I mean, if you installed earlier in your you know: kubernetes or your parental beat on using all-in-one, be it using individual production, individual components. If you need to monitor that right, I mean you probably not want to just install you ever, but you also want to monitor it. Each point point has its own slash, metrics and point, and by default it outputs promise to use metrics.

A

Now the beauty of operators is that we can tie everything together. So if there is an prometheus operator running on the same cluster, the Jaeger operator, you just add the proper annotations to the components and the previous operator is able to also discover the Jaeger components. Alright, so you don't have to do anything. If you have both operators running, they just find each other and metrics just flow to Prometheus. Now, if you, if from, is Prometheus, it's not your thing, you wish to expose all these metrics are using X part.

A

We have quite a lot of options there pretty much everything that we have as an internal state. We store as a collage or as metrics or histograms. So it is, it is quite perhaps not easy, but it's quite whatever you want to measure from jäger. You can probably measure using the metrics that we have already so you can measure how many spans we received, how many traces we received and we have that information also per service. So you can monitor individual traces for individual services.

A

Yeah and then performance to me so how many of you are using eager already: okay, that's a good number and how many of you have heard about dropped, Spence or spans been dropped, I mean if you have faced decision yeah one here.

A

A

Well, it sort of yes, so the thing is for Jaeger was designed to be very resilient, meaning you should not be able to break Jaeger you should you should not. You should not crash, because you are sending too much data to it all right.

A

So, if you are, if you are sending thousands of spends per second, the Jaeger will then, if Jaeger is not able to consume that or to ingest all this data here we start dropping Spence, because it is desirable to not record things that we may have already similar data from the past or we are gonna have in the future. So we just drop data instead of crashing the VM or instead of having a om killer process from kubernetes just killing our pod.

A

Now the thing is, it is not the best default that we have there, because it is very conservative, meaning if you have a machine with 10 gigabytes of RAM, and you just run the de-ager only one, for instance, then, even with moderate load, you are having dropped spans because the queue size is more. So we store everything in queues inside.

A

Here and for each one of those queues, we have workers and we pick items from this from this queue. Each worker picks items from from the queues and process those items so in case of the collector each worker is store, is responsible for storing data on this storage. So if the queue size is not big enough, then data has been trough, even though you have more than a memory for that. It is very hard to come up with a reasonable default.

A

We have one github issue where we intend to automatically adjust the size of those cues based on information that we are able to gather over time. So if we know that spans are on average of I, don't know 2k and we are allocated 2 gigabytes than we are able to come up with reasonable queue sizes. That's not there.

A

Yet before this feature is implemented, I recommend you taking a look at the performance tuning guide and that provides a a nice rule of thumb, values for the penny or how to come up with those numbers for specific scenarios.

A

And then we we have again this similar call-to-action to the previous presentation. I cannot emphasize that enough that our we need your help. We need I mean I. Some of you raise your hands when I asked, if you're using Jaeger, so we special need you to tell us how we were using Jaeger.

A

What is your missing for you? What what do you want from us and perhaps even better if you can publish a blog post by specifying how you're using Jaeger right again, we have bi-weekly meetings so join us and talk to us and even before me to brainstorm and I'm not showing how to find good issues again, as most of you know how to do that, but basically on our github repositories, all of them we have for all of them. We have issues labeled with a good first issue.

A

Yeah, that's what I had for a deep dive now similar to that one. Typically, we have good questions here at the end, so I saved a good chunk of time for their questions.

E

Thank you very much for your presentation. I was wondering with the drops pants. Is that also being recorded in a way that you know that the whole trace is then broken.

J

A

So yeah, so the way that simply works saintly is made at the first span so at the root span. But if there is any other span that was chopped after that, then that span is not gonna show up in the UI now, if that is the root span. So if it is the first span that we generated for a specific trace, then you see a specific message on the UI saying like so people who face the the drop spans problem. Probably knows this trace without parent ID bug right, but it's not really a bug.

A

It's just that the the print span was dropped. So yeah you have a you, have a missing span in your trace. Okay, thank you.

A

Here, Oh, unless you have.

K

A simple question for me: does dagger support automatically HTTP.

J

K

Jaeger support automatic HTTP injection, where the request ID for incoming requests from browser can be traced in there.

A

If I understood the question correctly, so you ever see if a year supports that making the connection between incoming and outgoing HTTP requests automatically is that it.

K

A

And that is a instrumentation problem that is solved by open tracing or open telemetry, but yeah I mean so three yeah. So the thing is the the tracing or the instrumentation part. It calls the tracer to inject the context into the payload and Yerger just provide all the interfaces that you both inject and extract. The context from from the payload okay.

K

So in is that possible that this injection will fail, because, when the OSA, when when yeah, because sometimes when the HTTP header is injected, then load balancer or any other external control could drop that header. So.

A

If the load balancer is removing the context this pen or the context header, then there is nothing we can do. Then we're going to start a new trace, probably all right.

A

So you end up having multiple, because I mean the decision of to create a new trace or not is whether we have a spent context or not, and the context comes typically with you know it's a peer, requester or inside a a message, and if we cannot find a a context there, then we just start a new trace, so you may end up having multiple traces for the same business transaction.

A

Okay, thank you. I.

C

Think I mean I, had the expectation just like by looking at your explanation on the drop packages because of a couple of things and I'm sharing my view so that you can I can hear your comment as well, which is there is a buffering. You said like there is like those pans are sent in batches, so anytime the pods are recycled or the application crashed for some reason. We started non cleanly, then you would be losing some gems and the same was UDP I mean I, think UDP.

C

If it's localhost with the pub that is very minimal, but I mean, if it's a demon, set with very a lot of odds, then there can be a way sometimes that you can mess up a little too right. Yeah.

A

So, technically for a given set they're on the same host. So unless there is it wrong with the software networking level, then then, but I mean it's just so typically, Europe is not reliable over real network right over the cables.

A

In that case, that's why it is good to have on the same physical whole-step today at least but yeah you're right I mean if data is dropped somewhere, then we don't have a reference for that. There are some use cases for requiring data that is recorded to be stored. That's not typically how tracing has been seen because most of the traces, when you in stop to look at them most of the traces and most of these pens actually or I mean the similar spends they're.

A

The same right and tape may may be different in a few milliseconds here, a few milliseconds there, but if you compare most traces for the single business transit for one business transaction they're pretty much the same, so the idea of drop-in spans is that it's it's okay! If you drop one of those traces because it is similar to order 50 that happened. At the same time, and.

C

I think the conclusion then was traces like may be relevant to our in the discussion on my earlier question. We should not be expecting them to be hundred percent and 100 percent over time. We cannot I mean guarantee that, but we do. Maybe we don't want to have this to happen as well, but also it cannot be used for things like auditing or stuff, where you really need to keep track of us.

A

C

Was received right.

A

Yeah there are two sides to that: one is you right now? You absolutely should not rely on our audit data up in there, but it should be your goal when configuring eager to store everything. So when you configure a girl, you should provisioning a girl. You should configure it so that it it it's able to accept and consume all the data that you report right. So if you are dropping spends, it's a sign that you should be doing something about that.

A

If you're constantly dropping spins, then you should either change the configuration or change the size of your machine. Things like that, so I would not say that it's expected to have this appearance being dropped, but it's not I mean it is a symptom of something all right. So you should take care of that, but you don't have to you know: wake up at 2:00 a.m. because of some drops pants. The other side of the question is on use cases where you need all this data there.

A

So there are some use cases that we could probably fulfilling with traces, but for that we need wear until delivery. So all the database-generated has been reported in stored and things like that, and we don't have that yet yep.

C

And the last question for me is, you said the collector does that at just times or like like real times, yeah.

H

C

How this is this is happening. It.

A

Is very simple logic, and basically it's what I what I mentioned right? So if we have two spans one as a child of the other, and this one here is on the own timing information there. We say that it is. It has started before the parent. Then we adjust that to be like here. You know, so that's pretty much all we do yeah. We just guess there was another question: yeah yeah.

I

Maybe it's just a little bit of a comment on the differences between the UDP protocol and the HTTP protocol. If anybody is trying to kind of run Yaga as a managed service- and you want to have it externally exposed on the conductor side, the only protocol- that's kind of easy to add authentication upon that- you can also use Lars from some of the standard opens raisin paste. Libraries, that's HTTP right, so you can inject authentication authorization.

I

You know headers into the HTTP protocol, but that's not something you can do on the UDP part of on the backside of the agent. So anybody looking at you know running it with authentication on the connector scientists you to explore. You know just add a proxy in front of the on the connector, with a some kind of yeah authentication, credential, yeah and or baritone, or you know a classical HTTP authorization here yeah. So what.

A

One yeah that's much accurate one, one extra thing I would add, is we just like a couple months or even less than that? Where is support for the G RPC connection between the agent and the collector, and we can now do also education in between? In theory, I have not tried, but we can use the same approach there. So we should be able to place proxies to the authentication between agent, collector.

B

So I would be also interested since we heard in the in the keynotes already about this, the future. So where is it going? So how can we then in the future, correlate this stuff? So, for example, usually, if I see a service, degrading, I already see it in the the metrics that I get from the proxies that I have deployed with my services. So how can I then get this all together? So.

A

Tracy is very good to look into things that failed to look at very specific requests that failed. So if, if you know that that business transaction failed- or if you know you know the time range and then you you see like one trace with a exclamation mark, then you open that one. Then you see the whole transaction. You see why things failed, so that that's like the second step in the root cause analysis.

A

The first step is seeing that something is wrong when a dashboard and the whole idea there anything that's what most people are doing nowadays when they provide tracing solutions. Is you zoom into that time range and from that you see the logs and you see the traces during the keynote he was talking about linking log entries to traces. That kind of requires the context to be available for logs as well right.

A

So when you create a new span or when you log something, if there is a span opened around this log statement, then it adds a trace ID or the spinergy to the log entry, and only with that you can blink individual. So.

B

Then it would be, of course, nice to have this triggering that the disbands don't get dropped at all cost. Alright,.

A

B

So so, if your metrics reach a certain threshold, then you ensure that this service traces.

A

All the requests that there is one heart problem in tracing- and that is knowing when to sample things and be a hundred percent accurate all right. So we don't know we do not want to record, or it is not desirable to record to same spans or to same traces. If everything is fine, I mean we are just gonna delete them and in the heart problem there is determining after the fact, what we store and what we don't store and that's called in the table based sampling right.

A

So we decide to sample not at the head of the trace, but at tail right.

A

No, it's easy, a hard problem because, as you can imagine, on microservices, we may have one service running on one node, another service running on another node, and they don't talk to each other about tracing what they do is service a since de Lautrec Jaeger service B, since the other trigger and Jaeger, then I stitches, all those things together on the on the UI.

A

So if things fail there on a service B that service B has to talk, the chat has to tell service a not true or to trace, or not just simple, not too simple that now that, while that might sound simple for to services, when you have like hundreds of services as part of one transaction, it becomes very complicated. So there are kind of a couple of solutions, but no real solution. Right now.

A

Yeah right this and then you just roll it just didn't, do rolling of the logs and drop logs and yeah.

F

J

So if I ran some container or something in a poll, Kemah younger automatically attached labels for labels to expand. Yes,.

A

And it does it just so. You hear attaches Logs with attaches labels with the host name and with ap name by default, and also the same play strategy that was used for that trace.

A

Yeah. It does not by default, but it might. There is so depending on the client that you.

A

So if you see the Java client, you can specify in an environment, variable called eager underscore tags and those are static labels to add to all these bands for that application, and then you can use the I think it's called up upward API or something like that for from kubernetes just say well for inside that environment variable I want to store the plot name as well, but you know the plot name is kind of moot because it is the the container name and it is their host name, so it is recorded already right.

A

So the host name on that span already reflects the plot name, but you can use that API to to specify others. Information from the node.

A

No no well not automatically, especially because Jaeger was not meant to run only on kubernetes, but it's something that you know. I would encourage you to open an issue on the Jaeger project and discuss that, because I think we should be able to be it smart enough to determine whether we are running on kubernetes and if we are running there, then we can probably get those tags from you know from from from kubernetes, even if the yoga client is not doing that, we can probably do that with a Jaeger operator.

A

So when the a corporator sees your application, it can probably change the yogurt tags environment variable to include information open, an issue for us. Let's, let's discuss that this wonder.

I

It's on the question of what to do how to decide when to do a trade trace. If there's an error that you need to debug I mean if you wanna HTTP protocol, you can kind of inject in the header from the outside. We need to be able to control your client. That is initiating your request, but you can eject that you want to have a forced tracing started if you have a low sample rate, and you want to.

J

I

To actually make a trace on that, you can kind of tail on the Aster client that you would like to have a forced tracing started, which could be helpful. Sometimes, if you can control your your client, yeah yeah.

A

We have a question on the other side, so this side of the room dominated now. We are ready for the next side.

F

Surely good tracing comes for a price on resources, I need. Is there any assumption? I can take on how much more capacity I know I need in my kubernetes cluster, if I enable tracing for simply everything I'm.

A

Very glad that we finally had the performance question it was. It took I, don't know 40 minutes to get this question. We so she's not here, but we had a talk yesterday on that on that specific topic. So the talk was, what is the performance overhead of tracing and- and this was the the subject of Gabriella's internship, so the person who delivered the talk yesterday and basically the outcome is instrumentation itself is not costing you anything.

A

So you know instrumenting your application is the costs are very it's not very high or there's not any cost at all. Actually because most applications like in Java, it is the head code and the JVM would just remove that now. In terms of when you plug a concrete tracer like Yeager's, then of course you are doing something, then you are paying something.

A

So when you for in process tracing like when you have one HTTP request, and then you have all the database calls and you are instrumenting all the database calls, then those the collection of that kind of data is potentially costing you around 15% of throughput all right. So when you provision things- and you should provision, probably with a one or two nodes more for tracing I- mean one or two nodes- more. It's really relative, but I.

A

Think 15% is not a high price to pay for this kind of observability and when you, but then, when you do communicate over the network communication, one service to another service and you're measuring how long that communication is taking. Then the costs are very low and your network connection itself costs you far more than observing that connection. So it you don't need any extra resources for that.

A

Okay, so I guess that's pretty much it. Thank you very much for attending and once again join us.