Grafana Tempo, 9 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Grafana Tempo Community Call 2023-03-09

Description

Join our next Tempo community call:
https://docs.google.com/document/d/1yGsI6ywU-PxZBjmq3p3vAXr9g5yBXSDk4NU8LGo8qeY/edit#heading=h.3x2mcvpczj56

- Tempo 2.0.1!
- Adding Select to TraceQL?
- Future TraceQL work / Tempo 2.1 release

A

um All right, everybody welcome to Tempo Community call March Edition 2023. I apologize for missing the last one. I was feeling a bit under the weather, but I watched the video and Marty ran it quite well and I think you all had a great time a lot of cool discussion. It was right at the 2.0 release and I wasn't able to be there as kind of sad about that, but um so it goes sometimes I suppose.

A

uh So today we have a agenda up there. For anyone who is interested, um please feel free to add your own topics. I think we all have edit rights here. So don't hesitate to add your own topics and we'll always kind of end with an open time for questions. So uh if you have some thoughts or questions, you're welcome to ask operational with questions or just roadmap questions or really anything you want we'd, be glad to help people run their clusters or chat about anything. Tempo related or tracing I.

A

Suppose so think about some questions for the end um other than that we're going to talk about. We just had a patch that Mario's gonna review 201 and some of the small changes there. Just some bug fixes and some nice stuff Marty is gonna cost extension to trace ql that a Community member suggested and we'd also been talking about, um but he's just going to overview that and talks about some of those ideas and then I'm going to talk about some of the work. That's already in Maine for Trace ql.

A

That's that's exciting, as well as kind of uh show the to-do list and kind of get some baby. Some feedback from you all about what to tackle next there's a lot of things that could be added to trace guo and um we're just kind of prioritizing what we want. But we certainly want to hear what you want to so we'll start up with Mario Mario.

A

Can you do uh 201 buds.

B

Yeah sorry awful timing: I was just drinking water, I.

C

B

um All right, so we have a new release um this time, it's a patch uh 2.201 and it contains a couple of bug, fixes and changes. Although the changes are arguably bug fixes as well um today, sharing screen I can go over the points, I, don't know, wait or.

A

Without yeah, it might be good that way. The uh you know, the the recording looks it's more. It's.

B

A

To follow I guess on the recording. If you share your screen.

B

B

All right, so um we have a couple of changes. The first one is: um we've changed how we handle uh some special Fields between the whole search and in search. One of them is a status code, we're now using just status.

B

um It was kind of a similar change for error and we merge a change to fix that and to stick to to the new conventions, interest girl, and we had another change, which was uh we're now not streaming leading serials in Spanish we, which we were doing before um that used to cost that when we had uh when we were linking from trade skill results to the actual Trace, which now you can jump to the specific spam that match your query.

B

uh If you have leading zeros the the Lincoln wouldn't work correctly different, and then we had a couple of bugs bug fixes uh with some removed a Spam lock. Basically, uh then we were having, um or there were rather panics in Injustice and compactors, while reading some blocks that were written by a commit between 1.5 and 2. uh 2.0. So, if you've been following like tip of Maine, uh you may uh have written one of these blogs. Now the problem is in solve, it just fails more gracefully now and doesn't panic, which is nicer.

B

um Now we're also fixed how we were merging results in partly traces, and we have a trace level data coalesce better and then there was a bug in how we were sending trade school queries in AWS lambdas um yeah. Basically, we need to we needed to unscape uh three inquiries which we were not doing before and well that's pretty much it. uh We encourage everyone to move to uh to one and contentious a couple of back fixes, but all of them are worth the upgrade I believe.

A

Yeah we tried to keep what we back forwarded, 201 I, suppose, scope to the things that were most important, um that trace ql and Lambda is funny because we hadn't run it in Lambda, yet which is kind of surprising, but it's nice that somebody picked that up for us uh other than that yeah I think this is all pretty good. I've already had a good fix upstream and parquet go to prevent that Panic. That I think at least one person was that Lucas was kind of asking us about, because they were saying it most.

A

People in the community didn't see it. I might be making that up. Lucas I apologize. That's not you.

C

D

Yeah I think it was constrained only for anyone who was following along with the current builds like the dev builds for that exciting parquet functionality which totally paid off yeah. So um it's okay, yeah I, think this is a good patch.

A

Cool okay, um one of the gross Corners I, think of trace ql and the old tag based search is that first, one there status code and the way we handled it before was kind of to match jaegery kind of conventions, and now we're really sticking to otel and I think what we're doing now makes a lot of sense and it works a lot better, um but we still have the older tag- searches, kind of strange- if you do see weird behavior in that world or confusing Behavior- please let us know so we can continue to iron that out.

A

Hopefully we can move just entirely off the old search endpoint to tag based, search or sorry to trace, kill based search over time, cool.

A

uh All right with that Marty you.

D

Want to talk to us about sure.

A

D

Sure yeah uh well before that is um I. Guess I'd like to ask like is, is everyone running 2.0? Have you had any issues with it or using Chris ql have any feedback um good?

D

Hopefully you know some exciting stuff, that'd be cool, okay, sure, um cool.

D

Foreign runs the bleeding edge of tempo, actually yeah. Let's go that's great yeah good to hear that feedback um yeah. So there is a link in the document here, but I'll go ahead and share a little window too.

D

C

D

D

Cool yeah, so there's uh the link there. So this was something that we've been talking about and if you go through here, if you've used traceql, this will make a lot more sense. But if it's new, um it might not be apparent so I instead of running a demo to kind of show it off. um I was just going to actually just kind of like show this whole document here.

D

um So if the traceql UI will bring back dynamically kind of like whatever Fields you're operating on it will pull them back and show them in the table of results.

D

um And so, if you do like a regex or these open-ended Expressions, that's really cool because it matches the condition, but also shows you what was there. And so. If I did this query like show me all the five x axes? You know this is greater than or equal than 500, so I would see the actual value in the results table, but sometimes I want to see it, but without actually imposing any condition on it, and so there's this trick, um which we've been doing, and we realized oh yeah.

D

This should be part of the language, and it's also covered in that issue. You can kind of just do a condition that always matches and so it'll bring it back and show you whatever was there, and so this is cool, and so what we're trying to do is make this a little bit easier right.

D

It's not quite like, or at least simpler, um in maybe part of the language and um just like this is probably like the easiest, like kind of like a straightforward syntax that we kind of came up, or um you know that was kind of like popped up here in the ticket and um I think this makes a lot of sense and but I actually wanted to talk about it and just uh say that it's kind of interesting, because there's no or well, let me back up.

D

So what this would do is exactly the same as this trick above right. It would just kind of show it in the table of results without really doing any filtering on it.

D

um But what's interesting is that there's no typing on this, so that may be something we need to think through um I guess it would just look for that tag of any type yeah, um and so this is one way and I think this is cool. We kind of like the pipes um we kind of like the processing pipelines I think they make sense.

D

That's a part of the language that we like, um but before we go and do that I wanted to just throw out another crazy syntax, which maybe is something like this, because there could be a little bit differences in the language.

D

um This one doesn't doesn't necessarily imply that it exists on the span. So maybe that's what we want. The tag is nil like currently. We don't really do anything like that, whereas something like this would mean pull it back, but also make sure that it's present. So maybe that's something we would talk about, but so yeah. So if you have any thoughts or feedback um or ideas like I think there's definitely some cool things we could do here and we should um but it's kind of interesting yeah yeah.

D

So you could definitely do um you know the method in any of these yeah, yeah and um I. Think the example: what's that share.

A

That doc with me.

D

Oh okay, sure, uh okay, well I'll right now,.

A

D

A

D

uh I think I gotta click this button here.

A

Oh I I because I I, like your idea, I wanted to kind of Riff on it so I'm uh because I've been thinking about like tag existence and maybe tag existence is selection right, maybe just having a way to like assert. The thing exists could be the same thing as selecting I, don't know.

D

All right, we also do this.

D

Sure all right, thanks everybody for waiting on that. Sorry about that yeah! We wanted to share it.

A

Right now, this is seamless, that's right. This is all yeah.

D

Oh there we go.

A

Yeah I was kind of thinking this I like this idea for air. So this you know you want to say, equals a path. Right. I was thinking this for keg existence. Span.Hp.Stead, that's good.

A

I, don't care just as long as the span has it.

D

Yeah exactly I was thinking the same thing, but I think this syntax is reserved for booleans right now.

A

Great, that's a good point. We do interpret that to be like is a Boolean and a visible in true or false exists.

A

D

Yeah Loki's unwrap is kind of neat um the differences. Here um we only with the columnar approach. We only load the data minimum necessary, so we kind of need to know up front whereas with unwrap I think it's kind of already loaded, it's a little bit different, but yeah. Definitely that's something that we should look at again.

A

I always wished unwrap was called select to me. It is selected, so I always wondered why they said called it on their own yeah.

A

Okay, cool uh I do encourage anyone who's interested in this topic or Trey skill in general to jump in that issue and um give us some ideas. If you have any kind of input on the direction language heads we're interested in what you all think and so feel free to check some thoughts in there and we'll uh we'll we'll think about that. As we move forward with the language I.

C

Just had a quick question based on what's on the screen here, the first one uh that first condition you're, selecting spans that contain that HTTP URL. But then that's returning the traces, so you're, selecting status code and Method exists on the trace. Not the spans. The match in the first condition is that right.

D

Yeah I think it's kind of undefined. This kind of implies that we'll pull them back if they exist, but they may not, and so that's not something we actually do right now, if you there's no way to say to kind of select a nil in traceql, I think right now, even if, like this top one here, still make sure that it exists.

D

So it's kind of like a kind of an interesting side effect, and maybe that we want, maybe that we don't I, wouldn't say this is anywhere in the trace, because maybe you could have a trace where this exists. But none of these do right.

C

D

In this syntax yeah, it doesn't seem to kind of like assert that condition.

C

But after the pipe is operating on, the trace is returned, not the spans returned or the spans that match the original condition. Yeah.

D

That's the tricky thing this actually the way this would work. This would actually push it down into the storage layer and collect it here, yeah. So we collect it up front, but yeah interesting.

A

Does otel support nil or null keys? Can you have a a tribute with an explicit null value like not an empty string or not as like zero value.

D

A

D

It's been a while, since I looked at that yeah because.

A

uh If, if they do, then we definitely want something right. Some ability to like compare to nil, okay attributes have to be non-null I.

D

A

Of feel, like I, was a yeah yeah that.

D

Was my memory as well, but it's been a while, since I looked at a car.

B

A

Maybe maybe that's it, the status codes, not nil, is like a or like attribute, not nil as a way to ask for existence like if it's not there, it's nil. If it is there, then it has a value and we use that value.

C

A

I'm gonna write an issue about this.

C

A

I'm Gonna Leave, Your, Google doc, I'm, not gonna, link your Google foreign.

A

Tempo 2.0 trust, ql work. I guess is what I wanted to talk about. Next, let.

C

A

Also share my screen: real fast I'm gonna just grab this uh this. Whatever the stock. Is this agenda doc? uh No, not my entire screen. That could be a disaster there. We go.

A

Okay, so we have some cool things already in tip of Maine that are not um that are not into oh uh Marty did a really good job. The team did a really good job. Also just building this like layer to add features on top of very rapidly.

A

um We really just want to get 2o out because I added a bunch of other stuff, but we decided to just cut it right.

A

um So arbitrary arithmetic is now part of the language, which is fun because you can do silly things like you know, span Dot, uh http.status code divided by 2, is greater than 100, certainly a strange thing to ask, but you can, but it also has I think more interesting, like or interesting uses like span dot. You know megab bytes processed I like something like this.

A

Sometimes you want to write, um suddenly looks more human, readable right and something like that might be a little bit easier to understand as a human, when you're writing a quick query, so I think there's some really cool uh uses for this also, like ratios between tags, are now possible. Like span dot, one thing is greater than span dot.

A

The other thing oops thanks too so these kind of you know ability to compare ratios exist now where it did not before so arbitrary arithmetic was a shockingly small PR for what sounds like a huge amount of functionality and again like there's so much already done that we have like places to drop a lot of cool functionality, um I'm going to also comment or note this link down here this to do um this is our current list.

A

I, like I, said I'm about to write an issue for uh or I'm about, to write an issue for existence of an attribute. uh If you have strong feelings, please jump in look at some of those Trace ql tagged issues write your own. Please upvote! You know thumbs up the ones that uh matter to you and we'll try to prioritize the ones that people are most interested in um uh but, like I, said right, we could like min max and uh some would be very quick aggregations to add and I keep meaning to do that.

A

Spandot kind is one that keeps coming back. That I want to add. So there's a lot and we really want to kind of get a bunch of features in for the 2-1 release. uh This one is another one kind of a bug fix really like this before, but it's always false one equals 0.1 or one point I guess so. This is like compare. Comparing floats against number types.

A

uh Sorry floats against integers, so this PR really kind of coalesced, all durations all floats and all integers into one kind of number, which I think works a lot better. We actually have some people at grafana and I haven't tracked this down yet who are storing their HTTP status code uh as a float. I, don't have a clue why, but this would not find that if it was float, 200 dot. Basically, so if it was a float 200 instead of integer 200.

A

So now that with this change, that'll be in 2-1, um it's going to pick up all number types and be able to compare them. You know equally or across types which I think is pretty cool um and then, finally, this performance PR did some really cool things.

A

um The initial architecture was to pull everything at the very beginning and then evaluate all of your Like rules against it, and this performance PR only pulls exactly what's necessary to evaluate the rules and then, if those rules pass, then it pulls some heftier columns like the span ID, which is one of the largest columns, maybe the largest column in the uh in the block, um and it also pulls like the span start time and end time.

A

So this performance PR is really going to make a query speed up quite a bit, especially when you're doing like those needle and Haystack stop queries where you're looking for one or two traces amongst a whole lot of them. So these are three exciting things already in tip of Maine. Like I, said, there's this to-do list and there's a lot of directions to take this I'm, going to primarily be working on this for the next month or two um adding any features I can find doing.

A

Any performance, I can, or you know, kind of carving out any performance gains I can and then we want to cut a two one. Somewhat quick uh 2o was about two months late, uh not the worst thing in the world, but not the best either, and we really want to cut to one a bit quicker to get some of these features out because, like I said we're iterating, quick, we really spend a year building a platform to add value on top of and I want to cut it to one quick.

A

So if you do have thoughts, if you do have uh preferences, um please uh let us know, and we can prioritize those and get those out to you, quick in a 2-1 um but yeah other than that I I got not much to say uh at this point. The official agenda is complete. If you have anything you want to talk about, please feel free to ask in chat or unmute or Chuck it in the doc or you know whatever you want to do.

A

I think we're just all so stunned by the power of traceql, that we can't even speak. It's just like staring into the sun. Really, it is pretty awesome.

E

That I can send the a full uh rescue Elder and get results because now, on top of it, I build like a graphene uploading, where I can um write like a normal stress ql but then like, if I put some Fields with the wild card. It aggregate this field. For me in some uh time, so then I say: okay, I see uh some people ask me, like okay I see these spycon things. What are these things Within These attributes, then I can just do the aggregation and then say: okay.

E

This is that if we don't have Metro generator for this uh attributed and.

A

It works nice yeah. Can you show us because we might be working on similar functionality that we didn't want to quite talk about because it's a little early, but can you can you show some of this.

E

I, don't have sample data.

A

C

E

If I can share uh with the company data, but.

A

E

Can create maybe a small small app using uh this uh um sample data generator and just share it share a GitHub with with a sample project, but it's very simple. uh Just aggregate the data show it on grafana.

A

Is it uh is it included? Are you? Are you like um working through the blocks, the parquet blocks yourself, or is it a way you're querying tempo I.

E

Quarry Temple, so I just use I create like a small goal, application that query Temple and this one the aggregations and then send it like a in a clear format to grafana and then in grafana uh just presented like okay. These are the values in these times and then rafana. You can say like the front and two. So you know where to look, and you can say uh how many, how many traces do you want to look so I say.

E

For example, okay I want a sample of a thousand traces and then, with the sample check. uh What are the aggregations? That's awesome. Yes,.

C

E

Like it, because now we have this like a the trick that you were talking, uh I, don't use the the negation I just do like a wild card. If it's a wire car, he knows. Okay, I need to aggregate that uh and then yeah. Of course, yeah.

A

uh Yeah I'd love to see that that's really cool um and yeah we may or may not be working on very similar functionalities. Don't ask.

A

But yeah an obvious Next Step here, I think it's awesome. Fausto put something together is now that we have the column reformat. Now that we can kind of dynamically calculate some of these metrics to start dynamically, calculating metrics and kind of allow users to choose whatever Dimensions they want to uh see their see. You know, latency see, request, duration, C, uh requests per second and these kind of things and that's yeah again, maybe maybe a direction. We're headed right now, Marty, who didn't? Let me talk about it. We can all blame Marty.

D

uh You just talked about it: I.

A

Have said nothing, nothing is happening all right. um Thank you all for your time, I appreciate you all coming to the community call I. Guess it's a bit of a short one, but we had some cool news. Look out for the 2-1 um like I said we're trying to cut it quick and we'll talk to you in about a month or so. Thanks for joining us.

C

Bye, everybody, okay,.

D