Grafana Loki, 4 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Loki Community Meeting 2021-03-04

Description

https://docs.google.com/document/d/1MNjiHQxwFukm2J4NJRWyRgRIiK7VpokYyATzJ5ce-O8/edit

A

All right, thanks for thanks for joining every everybody, uh don't have a big agenda today, uh new time and I think we're probably gonna need to get. I think some books sorted out and do a better job of communicating this now that folks, in the more in the western parts of the us, should be able to make it, but um I think they're just sort of dig in kavi. I think you've got the ip match your design doc. You want to talk about that.

B

Yeah sure so just let me open.

B

That yeah, I mean yeah. The idea is like uh just to make it public and get the feedback from the community so yeah. I can just quickly uh uh explain the problem and the proposed solution. Basically, so we are also tracking that in the issue which also linked in the recent so yeah, the basic idea is like we are trying to provide a built-in, simple ip matcher for loki so that you can code the evil lock here.

B

So how we are doing now is usually we uh search for uh using a line matcher right if it is a single ip or we use some kind of request if you want to uh match multiple ips or like yeah, so with the wild card and stuff like that, so yeah, the the problem with that approach is like, usually it's hard to come up with the proper request, plus also it doesn't have any support for uh cidr pattern right.

B

So if you want to find, if you have like some kind of net mask or something- and you want to grab all the ip within that range, so there is no way to do it like uh yeah at least the existing solution is kind of like ugly, as far as you know so, yeah, which uh usually like turns up into like some copy pasting summaries from stacker flow, usually there's high chance.

B

It can be like wrong or something so that that's actually the problem you're trying to solve and the proposed solution is like uh have a single function called ip which can be used in both label, matcher and or line matching, and this ip function takes a format of pattern right and that pattern can be known of the three forms which also be explained in the descent.

B

So it can be a single ip or it can be a cidr pattern right, so both for ipv6 and v4 and additionally, it can also be like a range pattern like if you know exactly what range iv belongs to just a lower and higher range. So yeah. These are three pattern. We are planning to support, at least for now and uh yeah.

C

B

That's basically it and uh yeah. Here are some examples. uh There are still some open questions. Basically, for example, uh do we allow multiple ip function in the same lock, your query or we combine the patterns with a comma, for example, it can be ip of pattern. One comma pattern, two pattern: three or it will be ip pattern. One then pi ip pattern, two ip pattern, three, so yeah. Those are like small uh uh things like uh we want to get it right, even in the first iteration I think so yeah.

B

I think that that's it, and also that is one more open question like because if you have a uh ip there are like so many uh crazy versions, you can represent this ip right, so that is a good uh link which d shared in the design. You can read more about it like yeah. There are some crazy formats that, uh ideally uh the ib representation supports, but uh yeah we can keep it a simple, for example, one simple solution would be support.

B

The same format which go ipasser supports right, uh which also added a small command there uh for people who are interested. So it's just a three format. Basically, like a daughter, notation, hexadecimal and a mix of both so yeah and uh those are the two question I think, maybe if anyone has better idea, feel free and yeah. That's it basically, so yeah.

A

Okay, thanks gabby uh yeah that design doc. We should make it if we have. It make sure it's open for public comment and yes.

C

Basically, we need.

A

To figure out, uh like any good language, discussion, kind of figure out what we think the best out of trade-offs. There makes everybody happy um richie, I see you joined, um so I guess I'll. Let you make a comment on your uh your entry here uh or I can, if you don't want to.

D

Which? Sorry? Oh.

A

Yeah uh there's a issue. Well, I I don't know that you added it to the dock, but there is a the support.

D

A

D

Yeah, so um several aspects to to that topic, um if you click on it and if you click through to the other issue, 2722 um that talks about matching ips ready-made.

D

But I think this goes farther um where we would like to know what other commonly used things would be useful. As a matcher email addresses come to mind, does this line contain an email address or just match email addresses, so I can extract them or what have you? um There is certain to be to be other things. We got the suggestion to look into what greylock is matching, because greylock is obviously a log matching tool. So there is. There is precedent of people putting stuff in there and we can.

D

We can steal from this, but we would also be interested in other specific use cases which we can support by creating ready-made measures which obviously enables you to a uh be quicker about. Writing your your queries, but also makes mistakes less often like in particular. Ip address matching can be super intricate, even before you add ipv6, with uh like compression and the column for port and for the address blah blah blah blah.

D

But there are certain to be others, so that's yeah, that's what we want to know also, if you're on youtube, um just create issues and tell us, so we can do this.

A

Thank you, richie ivana you're up. You want to talk about grafana 7.5.

C

Yes, so we actually just released it a couple hours ago, and I was thinking about sharing couple of lucky related updates. um I can share the screen too.

A

C

Show with this one.

C

Okay, so the first thing that I would like to cover is uh lucky alerting so uh for grafana opens for open source grafana. uh We were supporting um other thing in the graph panel for various data sources, but this wasn't done for loki. So in this recent release we have added a backend for loki and therefore it enables other things, so you are able to create metrics query and basically create alert on those so as for, for example, prometheus or in flags or any other data source.

C

uh So there's one update. I'm going to have three updates. uh The second one is label browser.

A

Well, all these are cool, but this one was fun. I was playing with it yesterday.

C

So uh if you remember previously, there used to be a little like cascader button, but we have uh decided to replace it with this label browser um which allows you to, for example, if you know what value are you looking for? But you don't know in which label it is so, for example, if I was looking for um run log, but I wouldn't know if it's in job or job name, I'm able to kind of open the values for all of these labels, and I'm able also to search it in here. So it's like.

C

I have very tiny, lock instance here, but uh it highlights and shows the the matched uh values for those labels and you are in the at the end, able to either show log. So we would run this query or you are um able to run the shell logs rate and it would create a rate query for you, so um that is log browser and the last update or last change isn't strictly related to logi loki, but also to like all data sources uh that support logs and that is escaping of an escaped character.

C

So um we have had couple issues in grafana with users who had incorrectly escaped blocks, I'm talking about new lines and tabs, but it was very hard for them to escape them and they asked if there was a way how how we could do it and on client side. It is hard. It is hard to to know if it's actual new line or if it's just the same pattern. So we have decided to create this experimental feature and we are definitely looking for a feedback on that.

C

So if we run the log query, um so if your log lines have incorrect, possibly incorrectly escape new lines, this button- and this message is displayed- we are trying to inform users that they probably have incorrectly escaped new lines that, um as a first type or the best option, is to fix it uh directly in their logging database. But if they can do that, they can escape it right here and um let me see I I'm which are the logs within.

C

So, for example, here you can see, we have um incorrectly escape new line and it escaped it.

C

Let me see again yeah. So, for example, this line is going to be escaped and you can escape new lines and tab and, as I mentioned, this is going to be displayed only for the logs that possibly have this problem. But if you, if you would run a logs that don't have this pattern, it wouldn't be displayed here and that's all from me.

A

Nice um for sort of both of the or even I guess for all three, but probably for the label, browser and escaping if people have feedback. What's the best place.

C

A

C

We are definitely looking for all feedback if it's small thing, big things, let us know, and the best place is the grafana so directly, graphene, graphene, graphene graphene just create an issue, and- and we will look at it and, as I mentioned today, was a better release. So um we like uh have this, like. I think, like two weeks, is the beta period. We have this time to to polish uh this feature so uh yeah. Let us know if you have any feedback.

A

Thank you all right, so I added this at the end here, because it's looked and about on, for when I talk about a release and then we actually do it because it was a month ago we talked about 2.2 and started basically started the work of that today. uh I don't think it'll be done.

A

Maybe tomorrow might end up finishing it this weekend or monday, but 2.2 is in good shape. We've been running tests on it now this week um and part of last week.

A

Another thing I would mention that sort of got added at the last minute here that may be useful um and something that uh people are well yeah you'd have to have a latest master, build to try it out, but I think it's reasonably stable. Now, though this pack unpack idea where we've had some few forms of this in the past, where people have some label data that they don't want to make labels. So usually this is the case with kind of high cardinality data.

A

The use case we're really targeting with adding this right now is ephemeral, workloads uh systems that create lots of file names and lots of log source.

A

You know in a very short period of time- uh usually you you know file name or some. You know close derivation of that makes a label, uh but if you are changing a lot of those frequently, it ends up causing some index uh great grief for loki.

A

So what pac does is take labels you specify and put them inside the log line. We wrap the log line in a json object and then on grafana's side or in the query language. In loki, we have an unpack parser, which will automatically return the log line that was wrapped and make the labels that were packed as labels, log ql, queryable labels so more to come on this. It is in some ways not an ideal solution for this problem.

A

Ultimately, as we evolve loki's index and store, we want to be more accommodating to handling you know more series with with low volume, but currently you can get yourself in a lot of trouble, generating a lot of log streams rather than series, especially when they have small small amounts of data in them. So that is.

A

Well, I had on the list today, so anybody have anything else that comes to mind, questions anything else. We can add to the agenda, or this can be a rather short call, I'm fine with it being a pretty short call today.

A

I like the new time. um However, it's it's lunchtime for me, so I'm also happy to go: eat lunch um going once going twice, uh yeah! Well now that we do have this time and I think we're probably gonna be happy with it.

A

uh We'll do some more kind of I think promotion around it uh next time and uh see if we can come up with a little more a little bit longer list of things to talk about, or you know, tell your friends and add some stuff to the doc and uh we'll see everybody in a month. So thanks for joining everybody have a great day.

C

C