Istio Extensions and Telemetry Working Group, 4 Apr 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Policies and Telemetry WG - 2018 04 04

Description

Working Group meeting from 2018.04.04

A

B

Follow everyone see.

C

What this is not the right days.

C

C

I, don't see any items on the agenda. I didn't want to share some updates. Primarily I talked about this I think classy, but I don't know if they did bad by admitting it was finished, but it definitely was finished earlier this week is. We now have mixer itself recording through two mixers, so the rhomboid proxies is sitting front, a mixer for them. The policy in telemetry service are calling two mixer themselves to report activity.

C

So now it's now possible to monitor sto mixer, at least in the same way that other services within the mesh are being monitored, and our next step is to add that for pilots as well. So that's.

C

You know sort of an update, I wanted to share I know my Ken. Did you want to talk more about the source attribute stuff or anything.

D

Yet so just follow-up status, so we had a few of us had the discussion. Yesterday, I'm probably gonna write a letter emails and it's a mailing list, so not only I think there's a few attributes who should remove I think we should rename some things as well for clarity.

C

Okay, Peter, you started on clients.

B

A few times, oh you masking that, so this is some pending question that I'm still trying to go through. Shall we add it on what we want on with you or just we don't care about one. It's a good question: yeah, yeah, I, don't know what the.

D

Easy ones II of what.

B

The consular report feature so we just added two I'm go with you or yeah yeah yeah, and also do we want to have client-side report for arm any service, call to these two components or just service. The service.

C

As well I think, although they want it first, one others trying to but I think one thing we wanted everything. But if we start with just service to service that would be a big win, then we can worry about doing less yeah, okay and.

D

I, don't think we want this one I, don't think we need to deal with it being one.

C

Darn stunning: do you guys wanna share? Any status of interest are going on.

E

Thanks man so and still replacing the data, the types and stuff in into API that are needed for the period after work, I think I think Thyssen all done. I'm starting to ingest them into into is 2 H 2 O, so people are specifying now living in the same. So we introducing some new types, working, foes and stuff which are, which is how you can specify the adapter information like descriptors, that support yet after compare so all that stuff now needs to be ingested into into the really cold.

F

Yeah hello, on the related note, I have just started or restarted, depending on hope how we look at it at the first part, which is kind of next step of super diaper transformation. In the context of given, given a an instance config and a descriptor step create create a structure that is set up to go so then that can be called on the runtime bot to produce points, and then sunny is coming again from the other side like connecting pressed on the mixer to it.

E

Talked to like Shri Canton's folks here at work, if they'd only takes is something which we can leverage mistakes, a descriptor and they say yeah, mail or JSON and gives- and we probably need something regenerate where regular. We have this on PV, which which takes I, am L and takes a photo message, and it can can transform it come on sugar, it will be ID, I mean if there's something already exist with instead of Jason.

E

Instead of taking a photo message, it takes a descriptor and there's a reasonable amount of stuff that you want to do. We just want to make sure that the field names match and it can be transformed, so it can be passed into so otherwise. We can probably think about continuing that piece of code to protocol go Tyra, do something and that emulator.

D

Do we want to talk about the topic we talk about sunny together about where the connection info goes versus other things.

D

Remember we said the sibling of neurons or it's inside and all the stuff. You know.

F

Whether connection belongs inside browse or whether as a sibling or friend,.

E

Right so yeah there's a problem making change we're like now in the when operated, specifying it handles one thing he specifies in Dominion rating as we custom resource you specify the kind as the adapter names and Prometheus, for example, and there's a spec and in this spec you directly write the image that represents the other topic right.

E

There is no special term that you intend that the conjugations is I mean for I, am or anything, but you just uses exactly say the first line inside they can be low-velocity or or or something else like this to make comparing a direct and building. So that's the current model. So now, in the same end of one thing, we also need to we repoed evolved and we have to add additional information which is not specified directly by the adapter as part of the adapter conflict.

E

For example, Martin was mentioning we need things like will it fail open up a flow, so this is a very static structure and also in right now for the GOP adapter. We need things like connection information, so each hands upon fate would have the adapter size of the combination, plus a connection. Information also connects to that handler for guessing right. So now, how do we do it? Because earlier inside the spec you had this full fuse, which was specified by the adapter configuration? How do we add new fields to it?

E

So this plan is, we can be, we create the V. Adapter configuration would be indented inside a special, well-known field called per am so so writing is fake. You write for an, and then you specify a low velocity or whatever their parameters. That after configuration is specified and final. You Quran will be connection so then you can search by a connection. So this is a breaking change for existing testing and I handle one which would break so we have a suppose both.

E

So we need to somehow figure out okay, this belong to the new format or you would format. There's.

D

Two variations to mention here is that in the there is more than just connection data at some point like I said so, the we want to specify per adapter mixer mixer controlled per adapter options such as logging debug, be one pure ball, oh yeah right, and so that's so so it's more, it would be more than just connection.

D

Second, the alternative of this breaking change is for us to steal a well-known name in in the users config space, so say all right, there's the existing fields and we reserved the another one called common or generic or base config, which is the standard mixer stuff. So that would be a non breaker. It not breaking change and it's just a little icky because we're stealing a word stealing a field out of the adapter can see.

D

Those are the kind of two choices there are.

C

Sort of hierarchy of configure we can provide at the top, and everything else in here is that field right there. Some sense which this is doing, yeah I.

E

Mean we thought someone I was mentioning. So if we do this feeling of being Tom, then the scheme is world by the by the adapter, and we know we don't have any schema we're just gonna do so. We cannot have data validation if we are feeling field, so the schema is coming together. At the contrary, there is no schema for the connection that Moses is B as part of that they kept on big schema. We would be loading it.

E

We can all do it very transformation from a camera to photo I think the meditation would be.

C

Tricky, as opposed to you know, structured it, so we do something where we generate an amplified: config proto doing it just just in the structure we wanna as a code gen and yeah. So like now, we.

F

Don't do anything a coding is like going deeper down, I mean if we, if we can come up with so we look at handle it up on the parsing side right which, which we will have to do for other reasons anyway, we are not actually going to just say here: is the ammo load in photo? It is always will be here is am all loaded in the generic structure? First do something with it then then load it, because so right now we do it for backwards compatibility reasons and they're always gonna have backwards compatibility right.

F

It always will be things which are, which you have changed so I, don't think we can just hand over the whole thing as it is proto anyway, and if, if we do that, then either option is fine, because then we could say that all these these fields are reserved, in which case it is not about what is not a breaking change right. Unless someone decides to use a connection running tools for grammar right, then yeah. If you look at those folks only or.

E

Are we only bind the bullet now and have it I mean what is more readable? Also, all I give me if you were to start from scratch. How would we do it? Yeah I think I.

F

Agree, this is the params thing is definitely more readable here. Here are your four like here is your configuration and then there is other configuration I.

D

Don't know, is it more readable, so it's so, but you just spec hole and you have all your stuff and then you have a common colon which has the debate. The based parameters seems totally usable to me.

F

Yeah and I think you're.

D

Agreeing or maybe you're, no, no, what I'm saying is I, don't want it. So whether whether we had a connection field, that's a sibling of params or we merely have. We don't add a params field and we just have a comment field underneath spec, it's the same amount of subdivision, but the common stuff that is separated from the generic stuff that they did on generic stuff.

C

Separated but I think if it was nested in the adapter can say guess what I guess to me is less clear than if it was like at the same level, it can say for the handler itself, yeah right.

F

Yeah I put it clearly separated, then you know which is which just for visual inspection. Otherwise you need to know which keys are I.

E

Think it would allow documentation. Do we use delivering these things? Then it would be handed in documentation as opposed to. We are adding it as a special field. Then the proto documentation would just give us and means no reason requirement. We can throw. We have acquired, not inquire or other constraints on this field. Oh.

F

Actually, so that's, okay, so what we can do and if kind of halfway between what the goals are just staying in Boulder yourself, is that we don't we don't generate whatever it. Is that so we don't actually add new fields to other people's furrows. However, in our documentation will say that you mean you want support this thing, and if they don't do it, then we pay the validation with a like. Oh the eventually we'll be making the part of the API and the improvement need you care and because photos don't have inherited.

F

The way we implement inheritance is that we table this is our big block.

D

So what that means, though, is that this is effectively instead of reserving one field called common that has our proto in it. You are now reserving an open-ended number of fields forever and you'll say you know we might maybe we'll have new common fields in the future that will force on you, I think I. Think if it's better to just have.

A

D

Field that we reserved upfront that has all of our stuff and then our stuff can grow independently over time.

F

Okay, I think that that makes sense. So we have yeah, we have our mixer driven in big or rather mixer controls and big, and then the adapter control can play, and then we can involve one for the world and do whatever we want with it without immediately affecting there not remembering.

D

Yeah so then it's just an issue of does that proto? Is that proto nested inside the existing adapter config, or do we treat it as a sibling of the adapter can see.

C

Yeah I, don't know I, guess I lean towards the sibling approach, unifed breaking change again today.

D

Yellow it's a breaking change right.

C

Yeah 15 is cleaner to me that that's different from config. Thank you. If I can think about I reach the handler or like what I want to have money to do versus configure the backend. Oh.

D

There's no there's another use case for this, which is the we wanted to support. Adapter specified request data, you don't we don't have a name right, so you can contain attribute elections that is writable by the fork for the adapter, so that would be.

A

F

Basically, an additional Newton that, depending which is attached to the unit that.

D

Would be itself another proto, that's attached to this, so you have the adapter config proto, the mixtures, adapter config and then the adapter instance data or something.

E

Yeah that time you will have to deserve. Oh no, if you're going with this indentation, if the adapter funding is part of Perl, and then we will, this later on, it's going to be, we have connection we have per a man, will have a reference to something as you do it. So there's no making change there then, but if you're reserving common now, then that that instance would be part of that comment. Follow as movable opening then also there's no biggie change. Yeah.

D

You could do is either way it.

C

Seems like there are nine years like if I looking at config between Jeremy seen on GRDC adapter is right and it has adapter components of its own. Okay. I've expect this sort of config experience new uniform, which to me means that this stuff should be siblings, of the trams and not is nested inside of the trends. But me that's just my little.

E

Way of organizing I don't know so here thing we don't introduce, but I am. But you have all these four balls in the existing bond thing and then you just have one the one leg expanded to it for connection or something yeah.

G

Like they know, making changes no.

C

I guess it's a breaking change us back and then you have the current thing we have. Then we have two other sections that are so the method instead of frames or whatever, and then you have the two other sections that are always played here: connection configuration friends and per instance, friends or something I, don't know what other people thought that that's I guess it's just the way.

C

E

Initial reaction is so, but just to know I mean it's actually not a breaking change, I mean even people with, but I am. We will have a just write, probably two lines of code for ensure that we don't think adjusting one thing we can begin part I mean anyways we're doing. We will have to look at some parsing on these stuff because we have, even if they reserving the key word we can on just know it's pretty cool. All you'll be loading in the camel has a spring or film.

E

I was drawing of the water and we are going to be getting each piece individually. So.

C

We're also talking about switching the siani and say our config model yeah, oh yeah, with this. Thank you, which is a breaking change itself, so we want to.

D

Do all that, together and and and I'm going to suggest some breaking changes for attributes as well yeah.

A

A

C

You probably write this out down and if we do it, we should probably do it. This month, yeah.

D

Okay, so who wants to write it down.

E

I can write down for the kappa ramm for the 11 in the config stuff.

F

Yeah I'll write down for the put a handler and instant and I mean eventually to make sure that building and I think we will.

D

Okay, so what Sonny could you start just one talk and we'll have three sections in it: yeah, okay, I'll, add a section on attributes, mother, Wilson and his stuff, and then we'll have an execution yeah.

C

Anyone want to talk about her agenda today, so.

D

I wanted to mention: I've got a PR outstanding for that changes, the log interface for adapters. So following last week's discussion, I there's now a debug method available for adapters and, oh that's why I got with the verbosity level left over from before, and it's now our debug enabled just like the regular logs.

D

So you can do a check if debug enable something something so that should be in probably later today, Sonny and I discussed this before in terms of so when that, when four out of process adapters, they're responsible for doing their own logging to wherever they want to so we're, not gonna, try and take the out of process adapters logs and pump them into through the mixers log, which means each adapter effectively has its own log stream and log parameters.

D

So that's why I want I want to be able to configure each separately, so the user can turn one adapters logs on and not the other adapters logs on and so that'll affect not only out-of-pocket actors but the in profit actors as well, so they so right now. The improv adapters are logging through the mixers, regular log streams and I think I'd like to separate that so they're, just independent. They go to their own files, they're AB, their own rotation and max size policies, and it can be turned on and off.

D

C

I think it all sounds reasonable at some point, we probably want to figure out what we want to do about exporting those logs. And what are you know like a word log, our unified locking story looks like four adapters right if I want ID on the log from all the mixer. Now, how would I do it if everyone's making all their own long streams.

D

Yes, I, well it so for all the adapters that makes your controller so the implementers and the coprocessor adapters, all those logs will be in the same in the same spot, right they'll all show up in the same container. What kind of thing oh I!

D

Think, if you can, if you can harvest one log, you can harvest them all.

C

Yeah yeah I think about that for, like 20 setups and things that are grabbing a lot, we did mean that yeah I'm going to end up making it harder. I hear your point it's like if I can get to wanna get all of them, but now I have to open big file. Okay,.

D

So that and that's a totally legitimate, legitimate concern, so the alternative is to say all right: mixer is actually the the all the adapter logs goes through mixer, including the outer proc ones, not not the ones where we talk to back-end service, that's their problem, but the ones where we manage the process. We could funnel it back through a mixture and going through the same spot.

D

So I I'd be okay. With that too, it just seemed like a circuitous kind of way to get the logs out.

G

E

Is that does that exist like a long train like foot I mean even if it's a separate file, if there's a trace, I believe for each long, they shouldn't becoming independent tool which can modulate six files into one file, but individual traits, ID yeah.

C

We can probably do that.

D

Well, yeah, so it's a matter of if, if the mixer, all we need to do is to have the mixer when it about is about to call an adapter to log with a monotonically increasing ID that it passes to the adapter, and then the log code in the adapter would also log with that ID, and we could do that transparently, invisible to the adapter. The code itself is just part of our logging support.

C

Okay, yeah: let's make sure we do that I guess: hey I'm, finally doing separate streams for now.

C

Let's do nothing, there's.

G

One long stream for remember.

D

Yeah Pharrell doctor Handler and who's behind them, yeah.

A

D

F

Waiting forever say that again, I really do think we're adapter or poor handler well I realize now it's a sleeper handler. Okay,.

D

And the log stream.

F

Thing so if and are these log streams, so the nice thing about having everything separate is that now they can go to ten different places like they can actually go to really different places, but when they go to the same place, it is usually better to have like one consolidated long right.

F

Thank you. If, for example, we want some locks to go to the producer side and some walks to go to the consumer side, I think that to support that models it likes to have actually multiple locks, not but just one stream, multiple mark or another. In that case right well,.

D

With the whole awkward I think we're confusing the log types here. These are more about development logs for the it's, it's not it's, not log for the code. It's not our log adapter, alright, I, don't think there's when a producer involved in this. In this, like.

C

Theater in bash parametric, ok, ok come on alien than their eggs, yeah.

F

Ok, so, and and when and where are these log headed typically like one log collection somewhere also.

D

F

D

Just dumped it's what it's Oh mixer is configured on the command line for where it's dumping, its logs, I. Think right now it's just died, oh yeah, and what, if that was logging, it's going to the same place in the same stream using the same consistent times yeah in this other model, each adapter would be opening with pot, would would send stuff, probably just to a file within the same context, I.

F

Think I think I think that it's better to actually annotate every log line, with whatever little ideas that we want and not worry about, having multiple streams at that level, because you can always, if you have the right ideas, you can always look up the stream and the post-processing with them or.

D

F

D

Okay, so and there's a reasonable step, a reasonable approach. That means that, for the out of proc adapters that are banned, the coprocessor adapter is that mixer manages we're going to want it to have these guys be able to call back the mixer to report log.

F

F

Or coprocessor for who processed, adaptors, you're saying right, not the true. The other promises are not going to report ending process. Yeah.

A

F

ah Yeah I did it, isn't it in Cooper awkward for sure.

D

Yes, a bit right, so that's the mental process I went through is all you all have them report. Well then, we'll all have the reports independently.

D

Yeah I mean it's not that big a deal to Expo to have mixer support a logging thing and in fact maybe it's just a piggyback when, when the adapter responds to a handle call, it can return. Some log data.

C

Yeah I mean the respond, their response message for that we can make it a response message and include cannery logs.

D

Yeah so then, as the adapter is calling log methods, we could accumulate that in memory and at the next opportunity, when the call comes in we'll just dump that and send it back or we could expose the G RPC service from from mixer to be called directly by the adapter.

F

D

Yeah so I think I think it's I hesitate to do that work because not only do we need to expose this API is. It means we need to construct some sort of a buffering scheme in the adapter. So we don't overload, don't send too many log requests and I'd, rather just dump this to a file right now and see if we can get away with that.

F

The problem with with top well as long as we feel support dumping to turn it out because don't bring to file in like kubernetes and in any other. If you want to send a stack time or like all, all those things become now some additional work right, be you have a configure the collector if collect from some additional path, so any of the boat I'm, not we're good yeah.

D

Sorry I confuse myself it's so the these out of proc adapters could report to their stare it out. It's just that the mixer cell right.

F

Little oily I think they're kind of it will be great so and then we're back to making sure that there is some ID that is associated on every line and standard out of the pod just become no actually on. If you send out of a container or part of a container, definitely becomes one stream, and it's more is anyway right, I mean Stan and out of a container. Is it one stream.

D

So the output of all the different processes is just randomly combined.

D

A

F

Yeah, it actually is very, very similar to like this blog right, even either any prophet can lock to the clock and the floor is just one one stream, and then you can literally become everybody like okay,.

D

Then there there's actually no issues. Then then these components, if the adapters, if they report to dump their own logs to their standard out or they talk back to mixture and mix or the pump set out to standard out it's going to the same place.

D

So it's it's going to go to that consolidated output file for the container.

C

Yeah they're on the pinky finger here: okay,.

D

Okay, alright, so let's just go with the so, basically all the config parameters that exist on mixer in terms of you can specify path, names, including standard out, and you can specify a max size if it's a file that kind of stuff those same parameters would exist for the adapt individual adapters as well, and presumably, if you don't set it up to something and Louis inherit whatever the mixer has and use that. Okay, all.

C

F

um Martin, do you want to mention your attribute thing or.

D

So I'm just gonna, I'm actually gonna write it up and send it to the mailing list. Okay,.

C

Anyone on the call was to chime in or discuss anything.

A

C

Well, thank you. We should just adjourn and meet again next week. All right thanks. Everyone.