Grafana Loki, 29 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: LogQL v2 Public Design Call 2 2020-06-29

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

All right, um do you want me to share my screen or time you want to do like last time and do it on your own.

B

I mean you, you go for it uh still. I like your new haircut by the way so yeah. Thank you. When are you getting finished.

A

That's not uh kind, oh I'm sorry. It looks very nice.

A

uh I'm gonna shut us.

A

A

A

So there were a couple of stuff I wanted to talk. um Maybe uh to start I'm just gonna uh go through what we discussed last last time and what we agreed on. um So the first uh thing we discussed and agreed on was that we will use type for uh logision well, log operation, sorry and we're going to use function for metrics. So the first function that transform logs into metric will be.

A

The first operator will be a function, um and so we already have this for rate average over time and the current supported operator.

A

We were talking also about uh that. uh We want to be able to support uh this use case. um Do you see correctly my screen by the way yeah? So uh we were talking about the use case of being able to um multiple level passing to do multiple level passing um and I spot something by rewriting this. So maybe we can talk about this, which is in this specific case that we wrote.

A

We cannot use the duration here as a metric and at the same time, being able to extract the as a label and level as a label.

A

So the syntax is like this, and I uh just gonna tell you again that unwrap allows you to select which label becomes a new logline.

A

And in in this case, um if I, if I unwrap the message to be able to um extract new label, then I cannot really well technically, I could do an another. Actually, that's wrong! I'm actually wrong! Here! I could do another unwrap here. If I want to um yeah yeah, I could do that. Yeah.

B

Okay, so that's kind of gonna be if there's like naming conflicts like.

C

B

If the two levels of unwrapping in in the name space, but we have to do like a prometheus like style like original, what's it more than the keyword they're using exported yeah exported yeah.

A

Yeah, okay, um we were also talking about filter, so uh it sounds like everyone was arguing that we want to uh support filtering on extracted level and we really stopped there. We didn't really had a new uh syntax for that and that's pretty much what we discussed last time. So there's one thing that I wanted to start discussing before. Looking at the filtering, which was something we didn't discuss, is the rig x1, for instance, as explicit label names when you extract, but the log format or the json won't have well could have non-explicit.

A

So you could technically extract a lot uh of label, and maybe you don't want- um and we didn't talk about, how can we limit the labels that we want to uh extract in the scratch pads that we had? We were talking about, so I'm talking about this uh kind of example here where log format here is basically extracting all the possible uh keys of the log format.

A

In a case where there's like an android keys, maybe you want to be able to limit that, especially for so you know, especially for the the the the one that returns streams and not metric because metrics you can aggregate on and then we can do some sort of uh optimization there.

A

um So I think we were talking about. Maybe so that's that's more for that time to they will.

A

Yeah, that's that's something else, so yeah I wanted to talk about. uh What's your feeling about this, like um should we allow um to select only couple of uh key to be extracted in case there is too many of them um definitively uh ed suggested that we're going to return an error.

A

If there is like too many uh so too many could be like if there is like 100 keys that in the log format, then we're going to return an error, because we know 100 keys, multiply by let's say already 50 uh streams that could add up into a 5000 uh stream, so we're gonna have a limit to something like a thousand, maybe or something configurable.

A

um So that's the first type, but also, I think we should we should be able to uh select which one we want to extract. What do you think about this.

D

Yeah, actually, I kind of I guess I maybe even assumed that we'd be able to do that right, like have after log format, some comma separated list of keys that you could extract.

D

I think, by default, what you have where it just matches everything makes sense, but some limit on that for someone that has a I don't know: yeah 100 entries or something you can error on that and then they'd have to go back in and pick what's important to them. Yeah.

A

um So what what? How? How should this look like? We also said that we want to allow renaming and tom wrote this example here right. So in this case, does it mean that we will extract all delay the label?

A

You know from the json.

A

Document and only rename that one or does it mean that we only extract that one.

E

I feel like we, we we at least at some point discussed whether renaming should be a separate, explicit action. Yeah, I'm not sure if this was like an intermediate state. What we wrote down here- or this was something we decided on- do you remember.

A

uh We haven't decided on that. No okay yeah, but this is something that we talked about uh renaming is something that uh we should do because.

F

A

Be clash there could be clash between uh names all right. What I'm suggesting here is maybe there's like two way to solve two two different uh one way to solve two different problems. One way is like picking uh an implicit set of label and also renaming them. At the same time, right we could, we could say.

A

I could do something like this right, so this will mean that I'm selecting uh flap and although this is a bad example because json, maybe the syntax gonna, allow you to select a bit more complex uh property from the document, uh but that would be that would work for um like format right.

A

So if you have too many properties and and some of them are clashing, maybe we could have a way of renaming and also just selecting which one so this will select just two of them and rename them. uh But you could add one without renaming like this, so I just want to bring this subject see uh if you guys agree with uh at least having uh something like that and then, if this, if it's the case, how the syntax should look like.

E

I think having something like this makes sense. You covered a couple of cases except the one where you would want to keep everything, but only rename one.

A

Yes, so that's part of the design, though the original design.talks about this, and I don't know about the solution. You're right, you may have like a a big document and you just want to rename one, and if you start renaming one then it looks like you need to stop picking all of them.

D

You could have a wild card match or something.

B

I mean just like thinking the same: can't we just have it like log format, not take any parameters and just extract them all by default. When would that not be the thing you want.

D

If you had too many.

B

A

I mean why is true money, a problem, it's a it's a problem in the case where um you don't have a vector algorithm on top of it, so you may end up with a thousand stream. uh It's also.

G

Why is having a thousand streams a problem?

G

Well in case you don't want.

B

You don't want them, but I like in the logs view, you never see a stream right. You always see the merged result of all the streams in the logs view.

B

So yeah you know like so you want literally every message could be its own stream or it could all be one stream and we wouldn't really be able to tell the difference and then in metrics I feel like you know, we have a good use. We have a you know. We have a powerful system for doing aggregations here so like looking at raw time series is never a good thing. People are trained that way anyway.

B

I just wonder like: are we over complicating this, and should this thing just extract all of them and not take any parameters.

A

Yeah I mean that's, uh we can we could do that.

F

I would uh I would be in favor of it still giving it a parameter to resolve the uh to resolve the clashes like uh to allow any renaming within I'd. Be I find it weird if renaming was its own, uh its own little um keyword, mostly because if it's in a separate step to me this would this would be diff. This would be difficult once we have uh like a chain of block format and json parsing and unwrapping, then it becomes unclear to which it retains.

B

Okay, the only the only counter argument there right is that we're going to need label renaming and general manipulation of labels between every, like json log format, regex we're going to need the same thing and it I I would be upset and disappointed if we had different ways of doing label renaming between different systems and, like you know, the argument for it being a separate like label join style.

B

Thing is ones if I want to take a label, I've extracted from log format and a label, I've extracted from json and combine them, and it's like prometheus. So there's three arguments right.

F

um I don't understand the use case for combining some. um I I also don't see how the extraction will be different, because they're all keyword, keyword based like right. It's like you just give it a different key.

F

You say this key maps to this key now.

B

Yeah I mean what what you were saying there is that they should take the same relabeling format.

B

Log format, json and regex should all take the same re-labeling format. Yeah I mean the yeah. I don't know I mean I agree the argument. I don't have a use case for joining between different, like log format and json extracted labels, um I'm just that's just a hypothetical.

A

Mean we so if, if uh selecting just a subset of uh label is not um really important, then maybe we can introduce this feature later.

B

Because there's another.

G

B

Yeah, sorry, all I'm saying here is not: let's not do this ever I'm saying: let's not do this now.

A

Well, we can yeah, we can skip this for the first version.

A

um Another problem with uh being able to relabel, for instance, is the json like well actually x. Hatching also is a problem because um we haven't talked really about the json, but what will it how the extraction does work for the json? If there is a multi-def document? uh Should it's? If it's only like this, will it um extract only the first level of death.

F

um Like in my opinion, yes um and and then you just use unwrap right.

E

So would that work with uh erase, I mean not, I think or no. I think we said last time that aries is kind of out of scope.

E

I think we had it, we did have a discussion of some sort with it.

A

What what is a out-of-scope product.

E

That, like nested arrays, I think um something we talked.

A

About yeah yeah yeah, I think nested array is uh yeah, it's a bit complex, but if you have a nested property.

E

I think nested properties could could be, could could be in scope or it could be. It could be done immediately, not necessarily via unwrap.

E

As long as it's really just nested objects, that's fine, as as soon as we hit an array, it won't work anymore.

A

Okay, so using uh unwrap his description for nested property.

F

I don't even think we need to. I mean our wrap has a special semantic that it that it uh sets the value field right. What we're looking for here still is to see if we want any label extraction in a certain depth.

F

So maybe that was a good picture that on rep woman.

D

I think that could be pretty common. Actually, if you have an object that has, I don't know, a status code.

B

That's like two levels: deep. The big use case for this is the kubernetes events right and I am incredibly multi multinested.

G

Yeah, I can uh maybe uh get one of them right now, the the challenge with treating it as.

B

A um as a chain set of unwraps to to unpick like values, further down as you can't, you can't unpick two different branches right.

F

Yeah yeah, I think you need, I think you need like a path access syntax anyway, like we did with the flap quilts flipped off flop.

F

So maybe that was that was maybe the unwrapping that does work, and I, and maybe it's maybe having this path- syntax map it onto onto just top level fields. Maybe that's the way forward.

F

And unwrap just has the just maintains the special semantic about what what the new logline field will become.

F

E

It definitely seems awkward to me if I would have to write json multiple times for the same, like nested objects,.

A

Yeah, so the the alternative syntax is allowing some sort of uh uh you know, x-path or or gq language, where you can write uh where you can write, uh you know exactly how you accept this property, but then again that means that you selectively select each property right, which log format seems to be different.

A

So there's like two different behavior in the way log format, uh parser, um regex, parser and also json buzzer- will work.

A

I don't think it's a problem.

F

But hold on, I don't even understand the in lock format the extraction flap equals flip flop. What does that do is? What is why do we have like? Is that? Is it.

B

This was just sorry david. I don't quite catch that. Could you just repeat that please.

F

F

Do you do you see what I'm highlighting.

B

um Yeah either plex is a bit smaller. I can't quite read it. um Can you, um can you tell me what this does.

A

It's supposed to be doing renaming of the flap property into flip.flop.

C

H

A

H

Go didn't you flip that flower.

H

And put it into flat.

E

Yeah, that's that's how I read it as well.

F

Yeah so so I agree too like this. Should the what's what's highlighted right now should be the receiver right and uh and flip.flop seems like an access pattern, which, in lock format we don't have right lock from it, is a single level key value right.

B

Like we've had this in english, french and german accents, could we just could we just have this in.

D

American well.

B

D

Flip that flat yeah- I don't right, I think, yeah your points right david, like there's no dot in log four.

A

Yeah, that would just be.

D

Flap equals flop or flat people's.

F

Flip all right, everyone have fun here. um Where is this clashing now is what is this one up here? Can you scroll it up uh yeah? That was.

A

That was the old way of saying uh which, which uh label will become uh the value for the metric we. Finally, we finally decide that uh the metric will always be the log line, so the log line itself needs to be a passable float. So that's and you're going to use unwrap for that. Okay cool.

A

So this is uh yeah. We just need to uh uh remove that. One.

F

Cool but then, but then the interesting like in my eyes. The interesting thing here would be uh this sort of pattern right where you're really saying I am accessing multiple paths: yeah in adjacent tree and they're gonna land in these top level fields.

A

Yeah yeah, which I just passed at a big example of what we want to cover, and we already have an issue for that, which is the evan harter, which has a very.

F

Okay and then symmetric and then semantically here here we're saying we do want buzz right. Does this also mean here we're only extracting extracting flip flop and.

A

Barbers, well, I thought that that's that's. What I was saying is for the json. um If we go down the road of you, we use using a format using a language to extract. We don't really have choice. You need to be. You know explicit about each property. You need to extract.

F

Okay, so so we want to, we want to distinguish between either everything or only the ones that we select right.

A

Yeah yeah: let's uh sum up the issue.

F

Here feel like maybe we should have a second syntax there, where we say where we say we don't want particular ones. That would be like something like this.

A

Yeah, but it doesn't really work for jason right if we go down the language of extracting the label, because does that mean it json extract all the first level? Okay, sorry.

D

No, that was that would have been. Should we punt on the json, since that's, I believe, out of scope for the first iteration.

A

um I think I put it back on the first.

A

D

Okay, if it's back in scope, then I hadn't.

F

I would like I would. I would pick something like this, where it says, extract everything, except these.

A

Yeah, then, if you need to escape exclamation.

A

Mark yeah, hello, let's yeah.

F

I just want to be clear because then, because this makes no sense, I'm going to remove this buzz yeah.

B

I'm still, I'm I'm still pretty keen on not having any of this by the way and having it as a separate, very explicit step, because this this is kind of reading this, and I'm like. Does that mean I'm only extracting these labels? What happens to the other labels like it's just.

A

Okay, yeah, so should we remove json, then from the first.

F

I just want to see how it feels yeah, let's to do something. So, let's see how this.

F

So this would be, and then this would be first.

B

Where are we all writing? Oh here we are give her a name yeah replace rename join whatever you want to call it.

B

I mean I I explicitly would say I don't want this use case of like everything, but this label- okay. This seems like brittle and it's not clear what we'd use it for and it's not clear what the you know. What, when you remove some labels, what happens to you know you're, saying we merge these streams together again, I guess: aren't you this.

F

Is more, this is more. This is this is optional for we're not we're like dude. Do you see the huge message down here like where, like where it says message, or something or reason or like let's say, there's another field called detail or source yeah, maybe source, and then this would be. This would be a stream value. This would end up, and then, if this has like high cardinality in the data structure in the local response, this will be.

F

uh This will be represented for each for each stream right and in the first case you have you, have it a thousand times there.

B

That I honestly david, I don't think that's a big deal. That's an optimization there. Okay, we should for sure we should optimize the the way we return responses. If, if like repetition in the json response is a problem, but this is, uh we shouldn't build a language around optimizing. The wire transport.

F

uh Yeah, no, I yeah. Yes, that's a good point.

B

The removing of a label is something cyril was asking for wants to do under a different guise right, which is the merge operation to merge two streams together.

A

Yeah, that's something that I think is required at some point.

B

Sorry, let's ignore dropping labels.

B

I think we should exclude that. Okay.

A

G

A

The source is not a valid label uh right, so, if you leave it like that, you're gonna have an error. I don't think it's a valid uh label value. Leave it like what be specific, if the, if you, if you select source as a as a label from for a stream, then you're gonna end up with this big json blob here as a label value which, which is probably not a valid label value.

B

I mean we, we haven't defined what json does when it unpacks, but I would suspect it would unpack it to be source, underscore component source underscore host source underscore I agree or dot or whatever some separators right.

B

That's my expectation of how that json would work um and not that it would unpack it to be source and then adjacent blob, because, as we said above that would require you to have multiple json extractions in a single line which won't work. If you want to extract things from different forks, okay,.

F

F

F

Just a second uh in this label rename right um log format here so in this label. Rename this line here this this past bit here for equals far won't make any sense anymore, because we can't map on tofu anymore right. Why can't we map onto foo? Because foo is already a stream here.

B

In the front I mean it would just overwrite: what's in foo, I would imagine or you we we need to define the semantics. Maybe we don't allow it or maybe we allow overwrites, but um I don't see why it. You know it's a choice. All you have to make.

F

Yeah, the the question here was uh uh on the the parsing and extraction. If there was a clash, things will get. Prefixed was that was that was that the idea would export it or something or yeah okay. So this would then be exported foo.

B

But that's like that's taking bar and putting it in foo.

B

Isn't it so don't you more, don't you mean like.

F

I guess we got a private. I guess we should sort of right: yeah, okay, yes, yeah! So that's yeah, that's taking food from the message putting it into the fu level.

B

Well, it's taking food from the the exported food label and putting it in the food label.

I

B

There's no messages, there's only labels.

F

Well, yeah, at that point, yes, I mean there used to be before the log format step right, this yeah sorry this is this came out of the log formatting.

B

Yes, this log format took the the value and converted it into labels.

F

F

Okay, yeah, I mean if we have this sort of safety net there, then I'm happy with uh with it being a separate step.

F

I'm sort of I just feel like a bit annoyed about it, that I can't do it in the same.

F

In the same thing, because I feel like I'm doing this label extraction, I might as well give it give it some some hints on what to do in a single step.

E

But um I mean that's like a query: planner can optimize that away later right.

A

um If you do, if we, if we were doing a vector aggregation on top of all of this, then uh if you aggregate by uh foo only I would you don't need to attack all the other labels yeah. But I don't know if that's what.

F

I have to go by the way.

B

I'll, uh so if we reached a a rough consensus that having a separate step for this is, is more desirable than integrating it or are we.

D

B

D

B

A

Yeah, I'm playing with it.

F

Yeah, I'm probably here I don't know.

E

As long as the it seems like, we haven't entirely agreed on the syntax, but as long as I can remember, the syntax better than label replace I'm happy with it.

D

This is a low bar.

F

B

Arguments- and we also said no- no label.

D

Drop that's merged is that is that possible for json? I still feel like. We need a jq type, syntax.

A

Yeah well, regex is the same right. It takes at least one string as the argument.

F

All right, I gotta check in the other meeting, if someone's there, otherwise I'll be back in a minute.

A

Okay, thanks david.

B

That puts for labor naming jason regex vlog format, don't take arguments. Jason extract well, regex takes arguments. Obviously.

A

Yes, this thing at least.

B

um Jason extracts labels to food.bad up, as always, underscores I'm not particularly picky, on which one I mean underscores might be easier to parse.

E

I think people just may be more familiar with dots from the jq yeah.

B

And this causes shift.

E

B

On most keyboards, so might be a reason to use dots and we're not going to do a label drop. So what would be.

G

B

What's that town, what should we call? The label manage command.

B

What is its purpose purpose is to rename um maybe to join labels, maybe to format labels. You know, should it be label font log from label, fonts label forms like takes, takes like string substitutions and assigns them into other labels.

E

I feel like they should maybe be separate things. Label rename and label merge.

A

Yeah, I'm wondering why you cannot be renamed also for the json like why. Why do we? If the name is uh having a dot notation, and you want to remove that dot? Why can we just reuse the label and discovery name for copyright?

A

You can so I don't understand. The label means.

B

I mean I'm just what what is this function and what does it do? Is it gonna only do renaming, or is it going to support merging joining of labels, or is it going to support arbitrary, forming and formatting of labels.

B

Because you, these are all suggestions right. These are all reasonable suggestions. You could you could, you could think of uh you know your label, rename is relatively obvious and then is it equals or arrows like? How do we know what the which way around it is label merge supports more than label rename in that you could, potentially label merge and label join will allow you to construct labels that have multiple.

B

You know out of multiple labels, and we do this occasionally in the metrics world, although it's very you know very occasional, probably done it three or four times um label thumped would be like uh you know, you might label merge on steroids like in that you could string format a label.

B

I don't think we should do that right.

B

Well, the the value or the the label name, the label, the clues in the name is labelfund.

B

So let me give you some examples. So you know this: we've got a label rename example up here you could do label merge would be.

B

Would be like you know, would be the horrible label join the syntax of prometheus right. You tell me what that does I've no idea? Does it put flip and flute flip and foo separated by a spacing flap, or does it put black and flipping fl in foo? This is fun, um then there's uh label join label. Merger do basically the same thing, and I actually I mean a little bit me quite likes label front and right same front which would allow you to do like sense.

F

I really like little munch.

B

It sounds dirty when you say it like that.

B

You could do like that right, oh um god,.

B

Does anyone think any of these are a good idea.

D

Well, whether they're good idea or not, people are gonna want to do it.

B

Right, but it's more, this is a question of what we do now like. We can extend that if there's enough demand for any of them in the future, we can do it, but I feel like maybe there's no demand for any of these right now or admit all of them right now.

D

You could I mean rename, I feel like for sure and protect. You know you could start simply where basically, multiple renames accomplish what merge does right. So if you rename a label to a label that already exists, it's an implied merge.

D

So you could string a few of those together and then that's simpler, maybe than having a the label merged syntax that we don't understand.

B

Yeah, I think that's uh yeah.

B

No one we can all agree: the uh label join, syntax uh function in prometheus is bad but hold on. I.

F

Would have I would, I would have written this like this year. It's like flip and then calm a few.

J

E

Merge doesn't seem to make any sense to me. um Look like compared to label fmt.

J

Okay, you like why 3d printed yeah, but what is what is? What does lathe.

F

K

F

So I really, I really don't understand, is it merging or is it just producing a second label and then well I mean this. Would.

B

This would be taking what's in fubar and putting it in through again, so that would be override, but this would just be producing another label that contains a format of food. I guess bar is you're.

F

Just going to attach another label on both of those streams, that's what that's doing.

B

Well, there's we don't know how many streams are there right there, so it could be. There could be 100 streams here. There could be one yeah, but we're going to take the flu and fat labels and put them in flip and then we're going to do a you know we're going to do a some by flip right, which will effectively drop the foo and black ones.

F

Yeah but okay, yeah, okay, I see yeah, that's actually. I think, that's kind of nice to do that. If you put it like that.

B

It combines merge with the question of the separator it's much easier to read and it's much more powerful because you could put you know.

F

I mean it's a fake merge because, because it's not merging you just you just you're, just applying labels to a bunch of streams, they just happen to be the same labels, so you can later aggregate it. On top.

J

Of it yeah and you could, if.

B

You had like pod name, you could write, you know kubernetes pod right, you know you could you can put text in there yourself, because it's just a string format.

E

Yeah, I would imagine, if you take the kubernetes example, it would probably more often than not be namespace. Slash part right yeah, for instance,.

F

All right so everyone's agreeing, we need some function like that, where you can do some sort of magic. I remember we're agreeing we need it. I think we're just understanding that it's possible. No. This is awesome.

J

H

Looking at all of these, I.

E

I think I would argue even that, like maybe if, if, if we would introduce any of these only fmt label fmt, I think, uh would be.

F

The thing I start with hey: how is this not covered by label rename itself like? Why like? Why? Don't we just apply that same key value? Syntax up there.

B

Well, no, do we even need label rename? Oh, I see you're.

J

B

Label from or label rename whatever it becomes like.

F

I mean it did the same thing right you just you just added a templating logic.

A

In the value formatting, so you could support the string or a non-string, which is a another label name yeah.

A

I added the end, wrap to show that you can use that to actually format a log line, a new log line- if you want to and that's something that I think is useful- being able to format a new log line in some case, especially the cases you have a big json blob and you just want to see a couple of properties so being able to use a format on a label and then use that label as a long line is very powerful.

B

Yeah, so I don't like label rename purely because it's got an underscore in it and there's lots of things to type.

J

What, but, where is your love to prometheus keywords,.

F

J

We're in the log language.

L

L

I

If you take out that second out saves another letter, too, is.

M

I

M

A

I wouldn't mind.

B

Writing out format either here, but yeah, no you're right, that's probably if anything's taught it's like abbreviations are bad, especially, I guess the non-native english speakers as well might be harder.

J

But no, you can't call it vimt anymore. I do like saying thumbed.

A

I'm going to include it just for you.

E

Pronounce like.

B

Pronounced cube, cuddle right, yeah, exactly.

A

So, just to recap: uh we want to introduce only label format and for json all the label name will be extracted with dot notation uh and we accept.

G

A

B

We've decided dots.

B

um Are we going to go with label format like this? Are we going to go with these arrows, or are we going to go with equals? Oh dearest, I don't know yeah.

E

I think I like the equals better, but I do like able to.

D

A

So if it's a string, then you can do uh templating. If it's not a string, then it's another label yeah, which anyway you could do. You could do with with all on this thing, because you can have only the label.

A

B

Let's just tell like.

A

A syntax sugar.

B

Let's call this one roughly agreed and move on to the contentious one. The final one.

C

What's the final one.

H

G

No, I'm sorry. What's the uh um sorry, I'm.

A

Just gonna yeah, so I use the operator filter. uh Here's an example: let's, let's take that back into the document.

B

So I really really dislike that. We now have another syntax for doing filtering on labels.

D

I agree I really like to have our label matching syntax, but the the part that I'm not sure about is like this greater than or less than introducing that construct into the existing label. Matcher feels sort of just as weird and dangerous to me as adding another keyword, and this is you. This is because we want to only.

B

Look for slow requests right, so this is a hard requirement.

H

Yeah yeah yeah.

E

H

This is possible.

E

After after we've moved to metrics, why does this have to happen here? This filtering.

A

Well, because you don't, when you do metrics, you don't have logs and there's maybe more content attached to the log like a trace id that you want to be able to see.

G

Yeah, let's, let's just hear out, um hear that out because.

B

You might be able to do something like uh um unwrap latexy.

C

And warp: oh well, I like it.

B

um No or what what what keyword have we got for kind of turn, this into a time series where.

G

Latest value, I think, is what we said, although we didn't really no.

A

We said that we don't need one, that the unwrapped is going to be enough and we said that we're going to do like a operation on it, something like that then so we should have unwrapped. So that's the syntax, we're.

B

But like this is unwrap, is gonna, make the value still be a string? Isn't it.

G

Yeah but the fact that we you have this so the first function.

A

That um that you use will transform the the log line into a metric. That's the last thing we said last night.

B

I feel like there's a different I feel like am I I mean you're, there's, there's a bit of magic going on here: there's an implicit conversion between a string value and a time value and a sorry, a value value, a number value.

E

I'm not entirely sure we actually landed on it. I think we still have some notes up there that suggested um using the value function. Let's.

B

Just for the sake of argument, let's do the value function.

E

But this is now a time series I think we said the value function would be the first function. Yeah.

C

E

So even without latency, just json- I don't know, but we have to say um which one yeah exactly.

B

And then you could do greater than 10 like that right.

K

G

But then what this returns the time series this this here is logs.

B

This is a time series of time and latency as the value, and this then tells you only ones where the value is greater than 10.. You know you could even filter by status code by doing.

G

Oh, let's say we're looking for slow, but successful queries yeah. But how do I see the trace idea of those slow.

B

Queries it's going to be in a label, it's going to be in a label of the value, a label of that time series this is going to get you one time series per request, basically per log line.

B

D

So the question that I have with this because I like but like how would you turn that back into a query to actually see the log line? If you wanted, I guess you kind of can't right, like you're, just gonna have to look at all the labels to you. You kind of like.

E

I I realize I'm over overloading the word, but, like you unwrap it right, you re-wrap it you can you wrap it.

D

E

D

I, like that filter latency greater than 10, is there's just a lot of cases where just reading the log line is really handy right, like there's a lot of context, it's already formatted in a way I can read just show me the 30 slowest.

B

You're right, like that's, definitely a problem with this approach right, the the thing we've got away is no. I have we've managed to achieve here a solution to that problem without introducing any more keywords.

I

B

F

B

F

So back to back to where this is pretty produced into time, series you're saying that there's a json property with high cardinality and they all produce their own time series. Then this will be one one data point right.

B

Every if there's yeah, if there's a, if there's a trace id in a label, then probably every log line will be its own time series yeah, but with just one data point with a single date, yeah.

F

Okay, so you sort of have to plot this in a scatter plot or something otherwise. This is going to be.

N

Totally pointless to look at yeah you're, effectively going to get a list of.

A

Events yeah, but they are data points so unreadable.

B

Difficult to read right I mean it's like again: let's not let the like the json response from the loki api, like we can figure out ways of presenting this nicely. I mean it's not just well. The rest of the log line has disappeared.

B

The log line is just in the labels now because we lost it.

F

B

The the log line was well, let's call this log form instead of jason.

F

I'm just saying I'm just saying in terms of like grafana displaying this: this will be as many like single data.

J

Points as there were trace, ids right correct. This would be as many series as there were trace id's, which is as much.

B

As our request.

F

With each of them, possibly only having one data point.

B

I mean the way I would I would. I would construct this in the back end that there would be a limit on the number of series we could return right. This query would fail unless you narrowed the time range down or make the filter big enough, so that it was reasonable to display.

F

Yeah or you like, omit the field trace id via some other function,.

J

B

You want the trace id right because we're trying to find the trace id for a slow query.

F

Oh, I see yeah okay,.

D

So yeah I mean you: could we could reintroduce re-wrap right? You could re-wrap this on trace id and get back a list of basically log lines of trace, ids.

J

B

Trying to solve there with the re-wrapping now I don't know, I don't think we could do that right. I think I I think this is a good enough solution to the.

E

Problem, I think so as well, because, like the it's kind of similar to uh charting a histogram on a heat map or calculating the p99 or something right like you can do either um the p99 is something you would probably do first and then you look at it on the heat map or something- and you can do the same thing here right like you, first um see that there are these values or how many values there are of this kind.

E

And then you kind of remove this and find the actual log line and a trace id and like the user. Experience can still be kind of figured out. For that. I think, but like drilling down essentially is a possibility from here.

D

I think that it, I think it's a lot simpler to do. Filter latency, greater than 10 and easier to just be able to immediately see log lines that right.

B

But the the problem I've got here is: there's an implicit conversion between a label string and an integer twice right. This has made that conversion.

G

Really explicit, let's just um there's a there's another con here- is that this one here.

A

Is limited by 1000 by default? This one is unlimited here.

N

The pro heroes is probably easier to read right.

A

Well, it's also easy to you know, build the query as you go, because it's limited by a thousand results right. So the first time you get way more and there's nothing that you want, but as soon as you enter latency bigger than 10, you get another thousand, which is all the one that you want the value one here is going to return you so many times. If you have a high throughput uh logo.

B

I mean I'm because yeah here I think that we would have you know. If we went down this path, we would have to say you know. Metrics expressions coming out of loki can't return more than a thousand time series. We'd have to just put that limit in right.

A

Well, yeah simple, not just time to service all right. It's also.

B

My main time series there, you could have a time series with a thousand samples in each that's, not a particularly big time series.

B

And, to be honest, we're going to have to have a limit on the number of series we return anyway. I think uh cortex has it anyway. It's just like when people do silly queries that return a million series, they would rather find out quickly than break everything.

B

I don't the thing is I just really don't like this implicit conversion right, because what happens if latency contains a string? What happens if it contains an s on the end of it? So it doesn't return it right. The fill yeah in that, in this case the filter just silently, drops it probably right yeah.

B

In this case I would say you fail the query, because there is no way to to value latency out of the value the latency flag field.

B

I mean and that's a that's actually a really big question. I don't know, I don't know.

A

About most people.

B

Ms or seconds or so actually we're going to need some. You know this value here, yeah, that's, actually a really big problem like.

D

Yeah specifically and go, this is a huge problem.

A

Yeah but the the the other problem that I have is you're saying that if uh latency is not a flood, then we should fail. I don't agree with that. I think it's a bad idea to do that. The reason why is you cannot force everyone for a single stream to have the same level all the time someone's not gonna have at one point the latency, not in the log for the same stream and then you're gonna fail him just because the latency doesn't exist there or is different.

B

Right, but in that case I would if you're, if you I mean you, the the thing I don't like is silently dropping stuff, because I'm going to write a an sla query and there's going to be a failing, um there's going to be a log entry that says this request failed and it's not going to say the latency and I'm just going to. Therefore quietly exclude all failing requests from my latency metric yeah, but.

A

They'll dance is not valid too.

B

Well, the alternative, is you know we we, we tell users that you know we do this in prometheus right. If the query is invalid, we don't just silently return, no results, we tell them. The query is invalid right and like the great example here is, joins right, joins in prometheus, sometimes work and sometimes don't work right. They sometimes don't work.

B

If you get a sample set where you do like food times bar and foo and bars labels happen to match, then it will work. But if you do a few times bar and the label don't match they want you, then then prometheus will return an error. It doesn't just silently only do the ones that do work I've. This is all you have to do like join left group left or whatever. I forget it, but we have this in the kubernetes mixing all the time where it works on my machine.

B

But when you run it on a multi-cluster box, it fails yeah, you have to fail in a way. You know with an error message that users can see otherwise, like how do you trust the results.

D

B

I think there's a.

D

Middle ground here, where you can return a result with an error message right or with with additional data, so I think the trouble with log lines is it's going to be impossible to guarantee that you're not going to have inconsistencies in a log line and there's nothing. They can do about that. Right, like the option would be like. The query will always fail.

B

Well, there's two right: you're stating there's nothing: they can do about it right. Well, I wonder whether like, if we accept that the crew should fail when when it can't do the translation, whether we can then provide them with the tools such that they can opt into dropping stuff that doesn't match, for instance, like you, could do a filter here.

B

That is like label matches something right, and this is basically saying I've got these one way around. This is saying it must have a label, it must have a latency.

N

All right, I gotta go.

B

Right, you could, even you know if you know you've got late if someone's emitted, latency values of like I'm a teapot, you could say you know, latency is not.

B

ah I can't do not on this keyboard there. It is not, you know, you know you've, given them the tools to filter out the bad values, but they have to opt into that process.

A

You're right, this is controversial.

B

I really feel like we have to. We have to fail on invalid data and not just silently ignoring valid data. Well,.

A

I'm okay to meet you halfway and saying we send.

G

Back a warning.

B

So the prop the only challenge with that right is: we do this in prometheus. When remote read fails, is we send them back the data that succeeded and we send them back a warning? Do you know how many uis display that warning.

A

Yeah, but for the other argument you say we should not take care about the ui and uh the problem.

B

Is no uis ever show the warnings graffana's software ui doesn't show the world like this. This is a by design, bad api to to show you values with that and then another great example is the dynamodb api right, the dynamodb api. If you send it a write, you send it back to 100 rights right. It will return success and then, in the body of the success message it will say.

B

Oh I only I didn't write 100 of these rights by the way you're like how is that success, but it will tell you: oh it's a partial success, it's success, but there's a hundred rights I didn't write I'm like, but that means failure. Surely.

B

Apis that kind of have part you know you should avoid in in almost all circumstances, partial success and failure, because so.

A

Since, uh since we already have a way uh to like the you know, filter with the label to solve this problem, could we have uh this thing that tells the behavior of the of all of this? If it's like, very strict or not strict, and then, if someone really wants uh to not be very strict.

E

Right where we say like by default, um like a partial failure, is a failure, but you can optionally say for this one query: I'm accepting a partial failure and show me the result. That's.

B

The what I wanted in prometheus as well by the way except for brian, said no, I would like yeah. I would like a way to.

G

Change the behavior just.

B

Because I feel, like that's a reasonable, um a reasonable compromise in that there's a there's, an extra field that says I allow partial failures, but I would like give it a long, hard think about like having the semantics of functions, change based on an extra field, because you'll you'll have people share, queries and the queries will work. Some people are not for others,.

E

Yeah, this is a conversation that we have all the time that, like shareability, must never be broken, essentially like. When I share a link with you, you must see exactly the same thing as, as I see.

B

And I feel like the defaults have to be sensible, I.e. We have to not default to partial failure, in which case like we should really only be building for defaults. In my opinion, give it some thought we've got really. I think we're really close. I think this is the last thing left to decide before we're before we're done with the language.

A

Yeah, so don't you agree that having value here and then after on top of it average uh with a duration, is a bit a lot like a you need the.

G

Average, I don't see why you need the average. I think this is no, but.

A

Yeah there's a lot of all the operation that aggregation of the time that you want to do on those extracted data.

B

A

Talking about the value we thought.

B

You were saying make value implicit, I refer you back to no magic, like uh we rate is implicit like we could have just said right, we could have said rate is implicit, but we said rate is explicit, like we have to be explicit here.

A

Well, currency rate is transforming the log line into a camera. It's counting.

B

Value is transforming a log line into a time.

A

Series yeah, but the rate is a aggregation of a range, not uh a transmission of the time. Sorry.

B

Yes, they do different things, but their type signature is the same a lot because there's a range okay. The time series is significant. It's subtly different, but they achieve very similar aims right. They they help. You translate from the log entry space to the time series space.

B

And you know, there's really only three, two three functions that allow you to do that value rate count. I think right.

G

Well, yeah, there's bytes, also now: okay value rate, yeah, there's a very small number that allow you to move between these two there's, four of them.

B

Currently, what's the f, what so rate count bytes, both all the time.

B

Okay, bites and bytes over time seem like uh some overtime.

B

And uh and we're we're suggesting our value well, I've got to run I'm five minutes late for the next one. Sorry yeah.

B