Grafana Loki, 2 Mar 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Grafana Loki Community call [2023-03-02]

Description

Few highlights from the call.
1. New patch releases v2.7.3 and v2.7.4
2. LID (Loki Improvement Document) discussions
3. Multi-day query splitting in Grafana (demo)

Meeting notes are available in google docshttps://docs.google.com/document/d/1MNjiHQxwFukm2J4NJRWyRgRIiK7VpokYyATzJ5ce-O8/edit#heading=h.xma6rt7n0xbi

A

A

Hello: everyone Welcome to our uh March uh Loki Community call I am kaviraj and I am uh as a software engineer in grafana labs, um so yeah usually I mean I've shared the meeting notes uh in the chat for people who are joining for the first time. uh Yeah. You can use this link to see the agenda and if you have any questions or topics to discuss, feel free to add it. There we'll try our best to go through everything um yeah at the end of the session. We can also have open mic.

A

So if you have any questions, you can just uh ask away. Oh.

A

Okay, let me share the meeting notes.

A

And anyone see the little ones there or is it just? Well, let's see it when you access the access to sling, one.

B

Yeah I can access it so for some reason it didn't show any notes that may maybe pasted before I joined the call. So maybe that's the reason. Yeah.

A

I placed it at the beginning of the coin, maybe I'm sorry cool all right. So let me share my screen.

A

All right, so, in terms of like new um release, we don't have much. uh We have just a tiny patch portions, uh since the last Loki Community call so we'll go down with the okay.

A

Is that fine early on.

A

So yeah I mean I just put some highlights here, but uh I also linked the exam change log. You can see the exact changes that went into the releases there. um Yeah mostly it's about fixes.

A

um uh Yeah 2.7.3 has this uh if it related to the delete request, when you submitted editor closed with uh start and end timers exactly same uh yeah, which will panic and uh yeah, we have a fix for that um uh yeah. We also have some Panic fixes in 2.7.4, uh but also uh uh yeah, some other things like this. One I think this is the one I did I, guess the CRA tanks so yeah. This is one thing which reported from the community. uh It's it's ready to prompt him. So what happens there?

A

Is uh we added this support for uh CRI tags uh just a bit of a context there?

A

um So if, if you're using uh this specific runtime, uh so we have this extra Flags, which uh tells whether the line has been the log line you're sending.

B

A

Like uh because of our uh upper cap on the number of uh characters in the line, so before Loki don't have any support for this, it just treats everything as a single line. So we added this support uh last year, but.

C

There was a bug, meaning.

A

Like uh this doesn't care about the log line coming from which stream, so that is a chance. uh The log line from different streams can end up in uh some other stream and during Edition, so yeah. This release should fix that one um yeah and that is also uh fixed for the window event log, where yeah I think uh that's. It was scraping like uh incorrectly, and this is from the community um yeah and now the isolator should be scraping it fine.

A

uh If you're using Windows, Event log, there uh yeah also feel free to add any more context.

A

If anyone uh worked on this feature or if you have any more uh yeah anyone that on this appears, feel free, yeah I just pasted like some of the highlights, um if you want to go through, um but today, I think we can spend more most of our time discussing the lids and uh we have some thing from Ivana, also from grafana side, uh so yeah before moving on to Lids uh I just want to ask if anyone have any questions on the patch releases.

A

Okay, that's not yeah.

B

Also feel free.

A

To interrupt me doesn't matter, okay, so before jumping into the ladies for people who are new to LEDs.

A

um So what what this means is like low-key, improve uh documentation, that's what it means so meaning like I, think Danny created this idea, um where, if we want to add a feature that is uh uh like big enough for uh significant that changes, different components or.

D

A

uh Different components that need some consensus with the uh contributors or like Community maintenance. So what you do is you just go: create this lid and then before.

B

Implementing the.

D

Feature of course, and.

A

Then you get the Consciousness and then uh yeah. You can start working on the feature, so we uh so the LED can be approved or rejected uh for different reasons. So we try to keep the lids merged, even if it's detected for the historical reasons.

E

A

uh Yeah, that's uh what lid means.

B

Yeah Danny, if you want to add any more.

A

Feel free, uh if not, we can go with the howdy. Do you want to discuss about scheduler lighting.

E

Yeah, you did a great job thanks Kathy.

D

No worries uh yeah, uh I I can start with the first one, which is an improvement document for improving how the scheduler treats the quality of service across multiple users within the tenant so uh copy me. You may want to open the documentation link.

D

You know so. People can follow yeah.

A

D

um So the problem we we've been facing in the cloud is kind of that we have, even though we have low key running running as a multi-tenant system, um that certain tenants are so big that they have a lot of actually uses um accessing or using the same tenant.

D

uh The the query fairness between users is is established in this carry already, but if you have like a contagious user within within single tenant, uh a single user could theoretically block uh other users queries from the same tenant and the idea there is to make the the scheduler uh more aware of kind of um being able to make the schedule a way of like a concept of something like a user or doesn't need to be used, but any other differentiation between uh they are differentiation within the tenant, so that uh these users, for example, within the tenant, are treated all equally and they have all the same uh quality of service within the tenant um and uh yeah.

D

It's kind of The Proposal there is uh I, did the kind of hit the um it's just a second layer, introduce kind of this concept of users or whatever, which uh we don't really want to do, um and the other proposal is to make a fully hierarchical scheduler, which kind of can build a recursive tree of cues, where, with certain there's a certain API, you can, instead of only just specifying the tenant which is which is mandatory anyway, uh but as you can, you can specify in which bucket within the tenant, uh a certain query would be scheduled, and since this is hierarchical, you could like divide, attended into users and the user within uh multiple other sub applications or dashboards or whatever, and like each of these queues, then uh does basically a round dropping pull from one of their their sub keys, and so you can guarantee um the the fairness across like multiple actors within the within the tenant.

D

So that's the kind of the proposal for that are there any questions. I think.

D

Hello, I, don't see anything all.

E

Right, Christian, I, think I think this lid was accepted right so which which proposal did we go for yeah.

D

uh It was accepted uh going for the for the second proposal for the fully hierarchical queue, uh mainly because it's uh not much different to from the implementation point of view to to just add another layer, uh uh but it opens uh much more possibilities since it's much more flexible how you, how you can uh kind of to quality of service control, Within These um uh tree of cues.

D

Okay, um that was kind of the first thing. um The second uh proposal, or the second document, was about changing the sharding uh within the index Gateway. This is not on the website yet because it has been merged.

D

So the the problem we are facing is to index gateways. Right now is uh that we shot um index index requests of the Gateway per tenant, which means uh we hash the on the client side. We hash the tenant ID and then uh execute the um the request of one of the assigned index Gateway server instances, uh which means every uh every tenant hits always the same index Gateway and additionally, to that the charting across index gateways.

D

We have replication uh of the replication Factor, so it's not always uh doing it on a single and single server instance, but on multiple, it's usually three uh in the beginning, uh by default. uh The problem with that is that, of course, we have uh in multi-10 systems. You have a small tenants and large tenants and by sharding just pretending. The d means that every tenant gets the same kind of access to the same amount of resources. So it's just uh the amount of uh instances that the replication Factor basically defines and um so.

D

But in uh in the ideal case, you kind of wants that the more or the bigger the tenants, the more data it has, uh the more uh resources it should be uh should be given and just be shot it properly. So you can scale more easily horizontally. So what we have to do now for scaling the the index gauges is to add more index data instances and increase the replication Factor, but this is then also globally for every for every single tenant, which uh it doesn't make a lot of sense.

D

uh So there are kind of birthday portals again. The first one was to kind of Define a dynamic replication Factor uh for each for the um for each uh tenant. That.

D

So, basically, that that, instead of a fixed amount of uh replica, so if you have a yeah instead of having a fixed amount of of of replicas for each uh for each tenant, you do that uh dynamically based on the amount of available instances.

D

So to say, every tenant has access to thirty percent of the uh of the of the instances uh and it then it calculates the replication packed accordingly. uh This is the same system problem that uh that each tenant is treated equally, uh then, the other proposal would be to kind of give a replication effective per tenant, uh and then you could charge it accordingly.

D

Just add a shot ID to the to the tenant ID uh for creating the hash function, so a bigger bigger than it could be, could have multiple shots and therefore have kind of uh different, different hashes and different.

D

uh Correct amount of uh assigned index catering census, and the third uh proposal is: is com is different from them, which is uh introduced or removes the concept of tenants of tenant charting but um kind of thus sharding, uh based on the on the index files that it that equally um will access so and based on the on the index file? Name, the index file name is basically hashed and it's distributed across the index gateways.

D

This is a more General approach, which uh also gives the opportunity to say uh if, if a query touches not only one day but multiple days, it will would uh kind of uh touch multiple index files for multiple days and they would be, uh they would be also shouted across multiple instances, um all right, uh that's kind of the the three proposals.

D

uh The third proposal could be also extended additionally, to somehow count the access to to these files and um that factor in that into the shouting so files that are more of an exist, kind of uh distributed or sharded across multiple instances as well.

D

um Yeah, it's kind of was probably way way much into the details. You can read it. You can read the the pr as well so.

A

Is it this one already got uh accrued.

D

Notice, this is still in in open, State uh I think we have a tendency towards the search proposal, which is more generic and also is uh gives more options to to tweak it.

A

Cool makes sense. I have one quick question about the other proposal: the uh scheduling one yeah. So do we have any like uh um like a benchmark or like comparison? How it's going to impact the multi-dependent scheduling, one I'm, just curious I, don't need to find it on the.

D

A

D

Not implemented, they can't uh can't say it how it would uh would influence it, um but um so, but what we will definitely see is uh that did the idea that you have a sub queue for a single tenant, or do you have that they have multiple sub queues to a single tenant and uh treat all sub cues?

D

Equally, uh it will mean that um a single 10, a single kind of user, that's what we will pass on in profound cloud from regular found user will pass into the scheduler, uh as this is the second level um that a single user won't be able to kind of influence.

D

The other users within the tenant as much I need more I mean it probably still have an influence uh because uh you have just. uh If you have a extremely large query with a lot of sub queries, a single user will have a bigger queue than another queue than other other uses. But when you're doing round robin and not waiting the the the the query is uh everyone should get their equal share within the tenant.

A

So one follow-up on this one right, so this should also work seamlessly on top of software, starting that we have on query path like let's say, a single career has subset of couriers right. This shouldn't be a problem here right this.

D

Is the same thing, but exactly so the kind of the the contest, the shuffle sharding concept, Still Still applies. uh This is kind of the first level. The first level is slightly different to the to the other uh to the other levels, because the first level needs to take into account that a single tenant can have multiple queries yeah to uh to execute.

D

But, like everything passed, that the initial like the past, the tenant scope is kind of just a round robin and and equally.

A

Okay, cool good to know uh just I will stop with this question. Just one more onto this one, so I'll be planning to ship. This, uh do you have any this quarter or uh we are just.

D

Yeah, this is uh so uh started uh with implementation, and this is going to be shipped. This quarter.

A

Perfect thanks Audi appreciate it man yeah. If anyone else have any questions, feel free.

A

um Yeah, if you don't have any questions, then uh I think we have two other lids I. Don't think we have uh this contributor in the call down. Maybe if he's there, can you please shout out I, don't not sure.

A

Yeah, okay, uh I think caution has some review going on with this uh LED. Also, if caution is here also can give a some idea on this, but yeah. If.

D

A

Has any want to discuss about this go process? I have already reviewed the lid feel free. If you want to jump in.

A

Okay, then we can take it async way: synchronously yeah Danny go ahead. The remote evaluation.

E

F

A lot um so this one concerns the ruler component and just a quick recap.

F

um The ruler is the component that evaluates queries against the store and then either produces um alerts to alert manager or producers um recording rule metric samples that could sense to a Prometheus compatible backend.

A

Danny you there, obviously just me.

D

No can't hear you, okay,.

F

The way that the ruler works currently is just I.

E

F

um um Where did you lose me.

A

uh We are explaining about the ruler component itself. After that we didn't hear you, okay,.

F

All right yeah, so the the ruler itself behaves like a query in the sense that it's um evaluates, uh queries that gets applied to it and it knows how to interact with the store or, what's a query, um the the data, that's in memory in the ingesters um and as such, it also doesn't have any way of parallelizing queries.

F

um So what mimir did recently and what? What we're planning to do as well is to implement remoteral evaluation. So what we'll be able to do is have the ruler just act as a client effectively um like if you're, using grafana to query Loki or log CLI, we'll just have the ruler, execute the query and form out the execution of the query to the the full read part. So that would include the query front end.

F

The query, scheduler and all of the queriors and in This Way, recording rules and alerting rules will be able to be split and sharded and all of the query acceleration capabilities that.

B

F

Will will now be available for rule evaluation, so this is important because oftentimes rules get evaluated and they're quite expensive to run so, for example, if you have an alerting rule, that's being evaluated every minute, but for some reason is looking at the last hours worth of data or the last day's worth of data that can take quite a long time and because it's running effectively as a there's, no parallelization Happening by default.

F

So by moving that query execution to the full read path we get all of those benefits and that results in fewer rules missing their evaluations. That's pretty much what happens if, if your rules run too slowly, um they back up and then they um cause list evaluations which are are really not good. It means you can list alerts or you can have gaps in your recording rule metrics.

F

So that's really what this is aiming um to achieve, and this is something that we we hope to implement within the next uh within this quarter, which ends I believe um end of April yeah. So we should have this soon and I want to just reiterate about these Lids that they are really designed for Community collaboration. So even.

B

F

A lid has been approved and merged, or if or if there's one that's still outstanding. You know we really want the community to engage and to ask questions and to um really offer you all an opportunity to share your voice and to share your use cases around these features. You know so, if you have a specific use case um for this or one of Christians, uh nerds or legal jeans, you know absolutely.

F

We really want um as much feedback as possible and and please do read through the uh it's- the the first load over there, the introducing let's triple zero one and.

B

F

Explains like what we're trying to do with um with this whole process and what the procedure looks like.

A

Cool thanks Danny. If anyone have any questions like Danny said, feel free and yeah I mean if you have. If you want to go, read the dots first feel free and then you can also engage um in the issue itself feel free to comment on the pr and uh yeah we're happy to hear you all right.

A

So if uh no one has any questions for now, then we can move on to grafana. Ivana is still there. Yeah.

G

uh I'm wondering can I share my screen, because I would also like to do a small demo sure, okay,.

G

Okay, so hey everyone, uh my name is Ivana and I am part of the team that works on Lucky data source and logging in graphanet, and usually our team shares on this lucky Community calls either features or improvements that were recently released or are going to be released very soon. But today we have decided to do something different and we would like to share with you a feature or Improvement that is currently in very early stages of uh development, and we are looking for early feedback on it.

G

uh So let me switch to grafana, and the Improvement I would like to talk about. Is the query splitting multi-day query splitting and basically this Improvement is aimed at the users who run queries that span across multiple days. So let's say our query is hey.

G

I would like to see some Metric query within the last seven days or within the last 30 days, and sometimes these queries can take a long longer time for you to see the data, and so we are currently experimenting with improving the experience and the goal is to basically split these multi-day queries into smaller queries and then sequentially run them.

G

This means that you will be able to see first results much sooner and basically grab will be filling out for you and maybe at one point you will realize that okay, I, don't need last 30 days of data Maybe last 15 is enough.

G

um Currently, this feature is only available on Main Branch and it's behind the feature toggle. uh So if you would like to try it out, you need to enable Lockheed query splitting um feature toggle and set it to true.

G

So let me show you how it works, so I'm here simulating a little bit slower Network, and so, when I run, the query, as you can see now like each day, is filling out uh one by one um and like as I said like at any point. I can cancel the query and that way, I will only see the data for the already executed queries.

G

um This works for Matrix query for log queries, but, as I said, it's a currently in the development stage, so there might be some bags there might be. Maybe some experiences that you notice that are not best for you or something that you actually really like.

G

So if you find anything, I have linked here, the the graphene issues, GitHub issues, URL, where you can create the issue or share your feedback and I, have also linked the lucky slack Channel communities like Channel, where, again, if you have any feedback, if this is solving your problem, if you would like to see some changes or just like share, what do you think um we would love feedback of everyone who will try it out foreign.

G

A

H

um uh Hi Bella, thank you for sharing this. This is really huge. This will definitely improve um the way people are clearing Loki and we certainly uh have people carrying multi-tape things and getting gigabytes of data back. um One question about the implementation: uh is this shooting multiple queries? Is this somehow um picking packing on something new or lucky that I have foreseen? That is part of our API?

H

um How is it working basically to get multiple days separately.

G

By the way, uh Sven, if I say something that is incorrect, correct, please jump in because I personally wasn't working on this feature, but the people who were working who are working on it couldn't be here. uh So basically, what we do is, if you run multi-day query. So, let's say, as we have here last seven days, we split it currently day by day, so we run seven one day, queries- um and this is like currently only splitting- which we do, which is by day, um did I answer your question. Arsen.

G

Is there anything you would like to add.

C

No you're doing a good job.

G

H

Yeah this answers my question, uh um so this means, uh um if this is enabled by default, we will have basically way more queries against our Loki instances. uh Smaller queries, um so my question on top of it is: uh um have you thought about leveraging? This starts API or something similar from Loki to see? Okay, um if you, you may just want to shoot seven days, seven days of code, but it might be just two megabytes of data.

H

So why shooting multiple queries against two megabytes of data, maybe making it more data driven in future to say: okay, if I'm accessing gigabytes of data, then yes split it and basically lock is doing behind the scenes. The same thing, but having a little bit of voyeuristics here before shooting too many queries just for a very small amount of data. It's just the overhead may be just having over the wire 7 queries or X queries might be just not worth it I'm, pretty sure.

H

When tsdb lands, we will have may way more features at our disposal to answer query sizes from the index rather from the chunks. So it might be a thing to cooperate here and uh making this with couple of 46 more robust or at least have a look on this um uh over the next iterations.

G

So, by the way this is a. This is great question and if you, if you can see here, we already like in a sense uh use the stats endpoint to to show you how much data will be processed, and this is definitely something we talked about because, like on the other side, you can have a query. That is let's say last hour, but we'll touch a lot of data and we won't split that so uh so there can be. You know the the other side of this, um so it was definitely discussed.

G

It's definitely something we keep in mind, we something we would like to do or consider doing in the future, but at this first iteration we have decided to basically to test like the visualization, to make sure everything is working correctly. That um kind of like uh yeah like that that user experience is good. We have decided to go with this like simple approach, but that's definitely something that we will and we will consider improving in the future.

H

And also thank you for providing this in the first place, even if, in a simpler form, it's a it's a very good thing. Thank you.

I

uh I also have a question: uh do we run these requests uh sequentially or in parallel.

G

J

Okay, because, according to the network graph, it looks like in parallel, but it might be wrong.

G

As well, it's like my children right.

C

G

C

So once once the the first request ends up, we got a response from the first request. We send out the the second request for the secondary models and also, as you can see, we start with the most recent data and then go back to the order order points in time.

I

Okay, great thanks.

A

uh One quick question, though, maybe already you mentioned Ivana- is this already in 9.4 or.

G

I think it's only on Main I think it was um added after the branch out, so you have to be on Main and you have to enable feature. Toggle cool thanks.

G

So yeah looking forward for a feedback from anyone who who tried it out.

A

D

uh Yeah, my question would be since there is sub queries uh uh done in sequential in sequential way. uh What, if uh one of these sub query requests fails um for whatever reason uh understand the the rest of the queries, countless well or other other rest of the queries executed. So it's just one part of the the visualization is missing or yeah.

C

I think it is the the latter one that you will basically just miss one part of the um of the visualization we started thinking about. If one sub query fails, we would just cancel all the rest, but we currently in our implementation. It's just. We will miss all those. The part of the graph.

D

Yeah, because I'm asking because for for metric queries, it's kind of it makes sense for logic. Queries. You might not notice that the part is missing immediately when you see probably see your timestamps or the speaker Gap in timestamps, but um yeah.

C

That's also good feedback that we might need to improve on your ex or but also for for metrics guys, if I, if I think about it, would be good to at least know that there's data missing or the the data requested is not there and that not that there's no data, because it could also think that there's no data right yeah, thanks already for the feedback.

H

At least something visual like hey, this is a partial result. Something happened uh like a banner or something like that people will make I mean. Usually people are using the locks they make worth out of it, even if they are partial, um because they get at least something and yeah.

A

So, to follow up on Christian gushing- that's is this same, for this is only for metric queries or uh for both log and Metric currents.

C

Yes, for both currently okay.

C

Basically, it also might be good because we are looking for early feedback here, um there's a little hack like if you want to see like a chunked result or a split result and a non-split result you can to the letter a is blue, a Ivana to the ref ID. You can just add a do. Dash not Dash chunk and do do not chunk, but we're done with that and and not chunks, not split query.

C

This is super good for testing if you want to see results back in the dashboard or something um if you want to compare to two things.

G

Also, as you can see, also work volume is, is chunked or splitted as well.

H

Have you thought about making it independent from multi-day to make it let's say, multi something, for example splitting within an hour or within within a day, for example, um because if yes, there might be a coordination um Improvement, for example, if you split, if you set the query to login, you get get them split in half an hour by default, if I remember correct or setting, and so this means, if you really are getting small, the actual results of sub queries and you aggregate them and we might need to stop doing the job in lock it or at all, I mean you know, query front end is doing exactly this piece um splitting aggregating, so this might um yeah no need to do the job, for example.

H

This is just an idea for us for for food for thought. Let's say like this: it's not really something huge, but if it's get arbitrary, how you split the queries and it's configurable- it might be a nice thing to record in there with a sub query splitting in low key I'm, not sure. If mimir does this I'm pretty sure they have.

H

We have this problem cortex. If I remember correct, but um yeah and.

C

Yeah so at least from from the front-end side from girlfriend outside um there's another little hack, where you can at least also now to try it out Define the the time range on which you will split I'm, currently, not remembering how how it was set up I can share it in the dock afterwards. So, basically, you can configure the days or the the time range which will split and I guess. We will also, at one point add it to the data source configuration docky itself.

H

H

It might be also cool to have it as an ad hoc toggle to let the user decide, because data sources are uh most of the time.

J

It's not available for.

H

Normal users, um so um yeah.

A

Wait anyone else have any questions, uh not just uh grafana or any anything else. We discussed so far feel free.

A

Also, if anyone is new Johnny for the first time, yeah feel free to say hi and if you like, I, hope you like Loki. You can just share your experience. If you want something.

A

Or we can be back 15 minutes of time 31, then all right cool thanks folks, thanks for your time and yeah, have a nice rest of your day. See you next time right.

H

Next time, folks.

D

Hi everyone. Thank you.