Grafana Tempo, 10 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Grafana Tempo Community Call 2023-08-10

Description

Join our next Grafana Tempo community call: https://docs.google.com/document/d/1yGsI6ywU-PxZBjmq3p3vAXr9g5yBXSDk4NU8LGo8qeY/edit#heading=h.3x2mcvpczj56

What was discussed:
- 2.2 Exists!
Release notes: https://grafana.com/docs/tempo/latest/release-notes/v2-2/
Blog post: https://grafana.com/blog/2023/08/02/grafana-tempo-2.2-release-traceql-structural-operators-are-here/
- Attribute existence in TraceQL `{ span.foo != nil }`
- Demo of Hackathon shenanigans

A

B

Right, we're recording welcome to grafana Tempo Community call August 2023.

B

going to be the greatest of all grafana Temple Community calls it actually might be the shortest. uh We don't have a ton, but uh we'd be glad to talk about whatever is on anyone's Minds. So we'll talk about the few things we have on the agenda. I know Reuben wanted to talk about um uh searching tags from the back end, so we can get that in there and then I don't know if anybody has any questions or just wants to chat generally be glad to do that too.

B

I think the first part of this is something of the uh you know. Joe Elliott show I'll basically be running it unless somebody on my side wants to jump in, but I think we're pretty thin today on the grafana side, so I might just be doing the whole thing, um but anyways Tempo 2.2 was cut and um so far so good interested in people's uh shouldn't people's reactions to that.

B

If anybody uh uh has any thoughts there, I thought we did a really huge release, um which I'm really excited about to get all that in one release, but it did take a long time to cut. Actually it took me a solid I, don't know week, basically a little small test to get that out.

B

um People do is awesome, check out their release, notes I, think somebody posted it in the doc. Thank you. Blog post also has some cool content. There was someone has found a panic in the metrics generator that we have not seen I. Don't know why it's kind of odd that somebody would trip across us and we didn't, but they post it in our public slack um yesterday, I think so there might be a two two one. uh Once we find this Panic, we we don't see it anywhere, but they are stumbling across um this.

B

This concurrent access to a map, so you can't do that and go and we do apparently under certain conditions, so we will probably post a 221 if you're not seeing this, it's probably not going to affect you because it's never affected us. Basically, so this person just has a specific configuration, I suppose so possible, 221 coming up um and uh hopefully to kind of patch up this panic. But if you don't see it don't worry about, it would be my uh advice.

B

Secondly, a new Trace ql feature already shows shortly after 2-2, so everybody's gonna have to be said, and you know wait for two three to to uh uh to get it or to get the feature. uh Of course. If you are in the kind of person who pulls tip of Maine, then you already have it. Congratulations, otherwise, you'll have to be patient, but uh attribute existence is kind of a cool feature. We've been asked for a while, so you're not interested in a value. You just want to know.

B

If the attribute exists at all on a span, then we implemented it using and I'll type. This kind of in here, oh I, already put in there spandup food, not nil, so uh your it doesn't matter if it's a string or an INT or any data type, that's just a way to kind of surface uh any value for a uh for a specific attribute.

B

So the First new feature for traceql and T3, uh hoping to get a couple more, of course, before I cut it, but expect to see that I suppose. When we cut two three.

B

um Hackathon Shenanigans I'd kind of like to share this. It's something we've worked on at grafana, so we have a hackathon every two to try twice a year three times a year, something like that. I guess three times thanks Jenny three times a year uh and it's always kind of fun right to explore some corner of code that you don't normally get to.

B

Of course, everybody's normally just working on the the normal stuff the day to day and to have some time to be creative is fun, uh so I wanted to show kind of what we made. um This comes with uh all the caveats of um you know: I, don't actually have a road map for releasing this or putting it in an open source Tempo, but I do think it kind of shows maybe some of our ideas and perhaps where we're headed. So let me share this demo. Real quick.

B

Doink um so good old grafana, we have uh it. We we called it Mega, select because we think it's funny and it's a terrible terrible name and for no other reason, basically.

B

um So uh the basic idea here is you pick. We have this end point. This is all being driven from a repo called intro to MLT, which I would encourage people to look up uh if you're looking for just a quick way to try any of the Open Source grafana Products. This is a fantastic repo. Just Docker compose up boink everything works. It's really cool, so I'm using that intro to MLP repo, and it's all this kind of like fake data, basically oops. Let me not do the last six hours that might oh good.

B

It did return. Let me do the last, because I've only been running this for about 15 minutes, I think uh-oh, 30 minutes. Let's try that ooh don't fail me now. Computer.

B

Okay, there we go thinking. My computer is notoriously sad, so hopefully this will return soon.

A

B

I'd love to share this with y'all I was kind of oh there we go cool. I won't go much farther than this for obvious reasons. So the main idea here is uh at the top. We show kind of like a trend. You know so for the latency for a given endpoint. In this case it's called slash. Unicorn I think we probably see these huge latencies when I opened up a Google meet and all my resources are being taken for Google meet.

B

So to look, maybe back at this part of the trend line would be more interesting um and the idea is down here. You see uh this um same trend on unicorn, broken down by all kinds of different attributes automatically, so every attribute it can find on the spans. Basically, so here it's by the status, the span status right so like unset or error um here is the status code. We kind of seeded it with something called infra cluster, which doesn't really exist, but like a fake kind of a thing, a fake.

B

uh Let me see if I can't do this is this gonna die on me? Come on come on computer anyway, so the idea is basically by automatically breaking down by all of these Dynamic attributes. We can.

B

um uh We can very quickly identify right, which attribute is kind of correlative to the major Trend, and it allows us to kind of inform next steps like is this correlating to a specific cluster or pod or a namespace or a customer, or what so that's kind of the idea, and the point of the demo is to show that it correlates strongly to a customer but I think my computer's too bad to even let that happen, and we have way too many exemplars in here.

B

Look at that I can't even get over them there you go so um so that's yeah! So a simple idea, just something we're having fun with like I, said no promises, no roadmap. I do think this is an idea. We have log your turn for Tempo and we we had some fun putting it together for the hackathon.

B

That'd be cool.

C

Nice, that's cool I like seeing this.

B

Exactly appreciate that man.

C

I didn't know what the screenshot was you posted earlier, but.

B

C

The context is great.

B

It's way worse, the old screenshot I apologize for the slowness of my computer yeah, but uh you know how that goes right, um so yeah. So I think this is cool. uh This is all synthetic data and it's a little. It's obviously very busy. We've been working on cleaning it up.

B

It's all synthetic data I really want to run this into a real environment and see what what I actually see there, um because for me right now it was a good idea, that's very busy and needs a lot of fine-tuning to get to turn into a usable Tool, uh but I don't know. Maybe if I ran out on real data, maybe I'd see something different or you know prioritize it, taking it a different direction, but it was a. It was just.

B

It was just a fun project we worked on, so this is all Trace data, no, no mirror no Prometheus, no cortex. None of that stuff. This is being surface or this is an API directly returned by Tempo uh built on top of that local blocks. Processor. If anybody's messing with that thing,.

D

B

All right that was the entirety of the agenda from the grafana side, between hackathon and cutting22. We just don't have a ton at the moment. I will open the floor, though, to I believe Reuben uh you're welcome to explain your.

D

B

What you're looking to do with the and or the tags, endpoint yeah.

D

um Well, essentially, what I would like to to accomplish is, um accordingly, the tags are limited to I. Think there's a complete block, timeout I think this is because we are fetching the task for an investor.

D

um What I would like to do is to fix it from back end. uh I see you saw your comments about. um Maybe there are a couple of other things to do before preaching that that level or before, tackling that the problem, uh so I was thinking and first fetching from the back end and doing this down range, but also um there is another possibility of having well I. Think uh reading your comments, you, you mentioned some kind of page level. Caching I.

B

Don't know if this.

D

Is because this is um the same idea as I have on having some some sort of cache of tax in a separate I, don't know block or something.

D

um Those are the two possibilities that I can uh think of, uh but I want to to hear your thoughts and because I want to contribute with this I I would like to have some sort of plan or something that we can. We need to do sure.

B

It's something we've wanted for a while, but I've really struggled to get to um so the caching thing is for at the parquet level, we were thinking about starting to put like parque's kind of got. This idea of a row group in the row group there's a bunch of columns and those are separated into pages.

B

So we've been toying with the idea of putting those pages into cache to just massively reduce the amount you have to go to your back end your object, storage, and that would speed things up and would make this more tenable, which is why I brought it up. Another form of cache we were talking about doing is in the front end, and this would also make this better.

B

Basically, the front-end cache would, you know, see a query like, for instance, for tags or for traces and then like see that simple job and immediately reply right from a memcache off the front end. Instead of the whole thing now, if you're interested in working on this, what I think you should do that would be. Awesome is just build the feature, so we can make it configurable.

B

So it's off by default um that way, it doesn't impact our you know: Cloud customers and it doesn't impact our internal clusters and we can kind of just see how I don't even know my assumption is it's going to be too slow. We take 400 Mega. Second I, don't think it's it's going to be too slow for that.

B

Basically, that's what I think, but if we just have it in the code base, then we can start doing things like building the cache and improving it and have it as an experimental option and people who are lower scale can use it immediately like if you were a couple Mega. Second, it would probably work fine, so I think it's a good feature to add. Absolutely if you want to tackle that.

D

I can start doing um without gauge, and then we can have more Improvement right up about the performance right.

B

So if we just get it in the code base, get it behind a flag, so it's not turned on by default. Then right every time we make these other improvements, it's going to get a little bit better. Every time once we feel like it performs pretty well at the you know, couple hundred Mega second level, I think we'd be fine with defaulting that flag to true or just removing the flag entirely. Perhaps I don't know.

D

Okay, that makes sense, I think that that could work I I might want to reach you uh offline. If you can find me through at least some of the files that to look at for the code, because I was um I, did some smart contributions, but uh yeah I I've never touched the the query pipeline. The query yeah the query, but yeah.

B

Yeah, so the good news is that the code we use to access these blocks and find the tags in the ingester is the exact same function, you're going to call so the cut that part's done like getting the tags out of the block is done.

B

um It's really going to be the query pipeline part you're, just mentioning so like basically right now, when you ask for tags, we only send that query to investors. We need to go like see what blocks exist that fall in the time range make a bunch of jobs for those blocks, and that code already exists for the search part for sharding for searching, but we'll see a good way to kind of like also make that code usable for the tags, perhaps and then right so yeah, that's a good idea. Reach offline!

B

We can talk on slack, we can talk and issue whatever you're comfortable with and I'll point you to some lines of code and kind of get you started. Thank you thank.

A

B

Absolutely man, that's awesome that you're interested in uh giving us a shot uh is your wherever you are your org. Are you all just super interested in that historical search.

D

B

Like at your job, are you running Tempo at your job? Yes,.

D

I'm um redclap, so oh.

B

Okay, oh okay, Pavo was talking about this a while ago too. Okay you're at redhead, I didn't know absolutely uh yeah. We would like that awesome, Red, Hat developer. Please, um please contribute this feature. I appreciate it.

B

um I didn't realize cool all right, so uh pav ball you're up next with something about the GitHub account bot.

A

uh Yes, so in the operator repo, during the release or in the release action, we would like to submit use GitHub account to submit to to submit the pull requests to another repositories and I know. There is a grafana account that we could potentially use and if you could help us, maybe to see if that graph on a broader refunderbods account if it could Fork to repositories and if we could use the that account to submit the pull requests to those two lipos.

B

Ed I mean do you? What do you not just need, like some kind of API, key or token for this external repo? Can we not just save it as a secret in there and use it in CI? What what is special about the like GitHub or the grafana bot, or the GitHub account or GitHub yeah, exactly.

A

We need the token, but it needs as well to forward those two repositories so that we can so that the account could submit the two requests to those people's.

B

Oh, you submit a pull request to.

A

B

So you need just like a GitHub token I I, don't think we would want to use the same tokens we use for grafana bot because we're already overloading those Zach knows uh so I think what we might be able to do, though, is like cut you just a new GitHub token right and we could put it in there and you can use it in CI to publish you know just like to create PRS basically, so that might be the easiest way to do this.

C

um My platform team is pushing more for a um like.

D

A GitHub action.

C

Because the tokens on that bot, like you say, are very overloaded that becomes a problem right.

B

Pretty complicated, there's just like a token and there's some other idea that I don't understand like some kind of Bot level thing that I know less about, um but what we should be able to do is make a token well yeah. What account would that be under?

B

uh That's the hard part, I see why you want the bot. Now you just want the Bots account. So it's you don't want to be. You don't want the GitHub, uh you don't want the GitHub uh PR to be opened as Pavel or.

A

Me any other kind of accounts that will Fork those repositors and then I need a token of the account to submit the two request from interaction in the operator.

B

All right does anybody on the grafana side know who owns Cross or our GitHub accounts? Okay,.

B

C

B

Okay, if you work with us a lot I know: you're talking about I didn't realize he was working. Okay, uh I will sync with that person. I see what you want now. You just want this to be opened under a generic account. A service account not a personal account. Absolutely it makes sense. Okay, I will explain our need. We basically need to build some automation. We just need the token is that okay and it just needs to have basically public rights. All it needs to be able to do is Fork repositories.

B

Make pull requests easy things.

A

Yeah I mean they could Fork it by by hand, and then we would just sing the repo in the interaction and submit the pull request.

B

Do we need to Fork it? Can you not just submit a pull request against the existing? uh You want to make a branch on a local fork and then you're right, okay, okay, I, understand your thing now. Okay, can you send me the actual GitHub, whatever this olm account or repo is just to help me like frame this question.

A

Here in the document.

B

Okay, so basically need simple, PR rights and we can uh right and we can build a fork ourselves or whatever. Okay.

C

B

I will follow up with that and ask Julian I suppose he's as good of a person to start with with anybody.

B

Most of my conversations at grafana start with. If this is not the right channel to ask, this question feel free to send me to that all right and Andreas IPv6. You, sir, have the uh or grafana Tempo IPv6 expert on this call right now, so any IPv6 questions you have can be easily answered, and that's not me.

E

You have some. My question was if anyone is running Tempo with IP version 6., because we were running into an issue and um the next question would be if there's interest to like, um if I should be working on it, to fix it and like, if there's interest in Upstream, to have IP version. 6 support like.

C

We're running support, oh sorry, go ahead.

E

um No, um we run into the issue on an app P version. 6 only stack so I stack videos, no IP version, 4. enabled.

C

I think my network is having problems right as you're trying to explain.

C

um Yeah I would happily take a issue report if, um if that's something that we need to address, I will say that we do support IPv6 in both uh IPv6 only environments or dual stack, um though the member list only operates on one at a time. So that's going to be the the you know contention there. So, even if you're in a dual stack environment, you need to choose, if you want to run over ipv4 or IPv6.

E

Okay, so in general it should be supported, but yeah, okay, yeah, we'll take a uh I'll, say more exact. Look on it. I just got a report from our QE that um there was the issue was um error, getting interface root, IP plus net. No such network interface and.

C

That says that um the address range is probably out of scope, so that's something we could definitely talk about. um If you're in the community slack feel free to ping me I'm I think I'm Z Leslie in there and uh and yeah in the tempo Channel. We can totally chat through this I think it'd be great awesome.

E

A

B

uh Yeah all right, well uh anything else, any other questions. This is kind of the grafana red hat sync at the moment. Nothing wrong with that, but uh I.

E

B

Got some good stuff done, I guess.

E

Instead of public quality in us or something because I think usually there are like five more people here, at least.

B

I, don't know uh I think actually most of our most of our community call people are EU, I feel like I. Don't know why, but in at grafana we're in the middle of a hackathon, so we're a little low, partly because of that people are just.

D

Kind of off doing.

B

Their own thing a bit you know um and then I feel, like most of the regulars am I making that up are most the regulars EU yeah.

E

C

Just Mario's out, oh.

A

My vacation, PTO yeah cool.

B

All right: well, everybody take care. um Yeah, we'll see you when you see you be good.

C

D

Right sounds good.

C

Take care everybody.

D