Istio Extensions and Telemetry Working Group, 13 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Policies and Telemetry WG Meeting - 2019 03 13

Description

Agenda:
- Review 1.1 release notes PR (https://github.com/istio/istio.io/pull/3558/files)
- Roadmap for 1.2+
- Rough proposal: https://bit.ly/2EZlSWC
Looking for community feedback (what’s missing, what’s wrong, etc.)
- “connection.mtls needs improvement” doc discussion
- gRPC streaming message counter doc discussion
- Mixer v2 / WASM investigation update

NOTE: istio-policy is going to be off by default in 1.1

A

Well, we should probably get started before we go into the agenda. Eric I see that you joined. Did you want to introduce yourself and say hi or.

B

Well, Erica Norman, just starting contributing to heal this year, so.

A

Great is there anything in particular, you're interested in with regards to policy and telemetry um nope, just trying to learn more figure out. You know where the best place myself awesome. Well, there's plenty of good spots.

A

We'd love to have help with the policy in telemetry side for sure.

A

Okay, so I've put together a rough agenda, but not in any real order, but I think quite wanted. Chuck leave early about some of the stuff, so we'll jump tech collection, connection, MPLS needs improvement, doc, I, don't know. If people got a chance to look at this or not.

C

Right, so this is a proposal to improve situation around connection Angeles. We have right now it's a bit miss grinding at this point, connection Phyllis, really talks about the upstream downstream connections and then boy. That means, if you look at the article from the client-side proxy, looks like it's not enabled because it's a connection from application to the side card and that that is confusing and everything.

D

C

To probably an outcome, so what we want to do is actually define to two different attributes. At least one that describes properties of downstream and upstream, because.

E

Both of them are useful. They.

C

Are Jimmie's useful for client-side sidecars in that top to inside mesh and downstream is useful for web proxy, including gateways, so that that means we need to have at least two different properties. So.

D

The purpose is really to just define.

C

To populate the to substitute connection to use boolean, and we want at least for each pup with you these three different categories. So we want to know if it's I'm jealous want to know if it is just.

F

Regular jealous or.

C

Is not secure, so I started looking into anyway it, but it has some problems with this. Tv is inside and vehicle that makes deriving this stuff a bit difficult, but I, don't think it is nothing blocking a theory. The problem really is that when you try to understand upstream connection during the check call, we don't really have an upstream connection, because that happens asynchronously. So we don't cannot reliably tell what status, but we can tell is the desired status.

C

So if the properties of the upstream Buster tell you to establish, MPLS should be able to divide that property. So.

B

Is there a case where that that would lie where it would fall fall back to something that's less secure? The.

C

Only case, probably if it's more broad like if you, if it specifies you, can use either so that then you don't know exactly which one okay. So then we should report the least the least secure one to be on the safe side. Right, yes,.

B

Should be Dilys, so we also need probably the ranking.

C

Of the properties so tell.

B

Us is less.

C

Secure than MPLS, so this one yeah there's also another.

D

C

The a TLS which I don't know enough about, but it's something HLN system is a loss. It's basically offloading all the stuff or proxy. So we.

B

Don't even know what.

C

C

Still supports.

B

C

What supports it? Well so because they only supports it, we have to support it, and I used to be I, haven't quite solved that yet exactly this well.

B

For this kind of stuff, we there's a point where just don't make anything up. If you can't tell anything, send that information explicitly so developers so.

C

Maybe we need also the fourth one which is meaning it's. It's just opaque config and you don't know it is or could that gives be generically on the unknown or undefined there's something.

B

All a chance is better than.

C

Unknown because it's well defined it's it's, it's just a risk. Hardware the folding chair lesson. It just feeds them to some hardware and we don't know what it is so.

F

The only issue is that it mixes attributes that look like configuration state with actual runtime state right. The downstream connection is that children, time State, so even if you're in permissive mode, you will actually know whether the connection was actually made using TLS or not or or whatever, but I the yeah where's the other way round and if they're abstracting, everything in one place and kind of mixing configuration with state.

G

So you argue for the second set of attributes that are requested first requested versus observed.

C

Would be just a lot of I mean it's also.

D

C

Should probably not make it overly complicated because that the other they'd you be looking if it's secure not and.

F

So when it does so, I think so, I think that if, if we are, if you're certain that observed, so if they're certain that what would it be not deserved would be equal to configured, then we don't have to make that distinction. Then we can actually say that no, this is actually the observed state but which is observed with using some other magic. So.

C

We're certainly fasting because that's what then we will try to do if you cannot do it, it won't connect, but it's just if upstream says, maybe using TLS, which and WA doesn't support it. So if that happens, right super.

D

Wizard on the other side, yes,.

C

Right, we should pick that's less secure because we don't know which is.

B

G

B

We can have a different attribute for a check versus report right, yes, yeah, so for report. We.

F

Know what happened correct for report? We know what happened right so exactly so, then there is actually one requested and one was the observed one. In that case there is a distinction, but we just update the value or so.

C

For us them, both of them are the same because they both happen. Asynchronously.

F

Actually, you know I mean, but the time report things have already happened. So so we already know what happened. There's.

C

No way to get the exact connection that you and we used, so it's a listening problem, but right now you only have no method at about asking for day off you cool, okay, not what they do, do it but yeah. So this I think it's mostly the challenge. They were just making. Em boy behave in a way that we can collect stuff.

C

Smokey our designs.

C

B

So so do we need different attributes or did we decide we can we don't need this split.

F

I think that, right now we will we won't have the split but make a note in the in the docs that say that this is obtained. This is reading the configuration state, not the observed state yeah. Well,.

B

It's the minimum independent of configuration this that the values reported represent the minimum security level, so the connection could be more secure than specified.

C

Yeah and so it's little less I guess where chalice is less secure than Achilles right each of us is, you can think.

B

Different word that secure preferred.

F

I mean for upstream connection, tearless and emptiness is the same thing right. Yes, I will you happen to you, for it I mean I.

C

Can't even tell you yeah, you can't because.

F

You're not even looking in that Billikens Solaris.

D

In Genesis yeah.

F

It so just tell us at that point for outgoing right for outgoing port for incoming. We know the runtime states, so it's clear for incoming.

A

Acting it's useful to have is distinction, but I wonder if, if in the met you given the telemetry right, all people are gonna care about, is if there's some security well.

B

Because this is the fact that there's mutual TLS versus regular TLS make some of our other attributes trustworthy, like identity related stuff, if you don't have mutual TLS, you can't really trust those things. So, if you really care you might be checking both properties is.

A

E

A

The attributes, though, that I think that's always one of those.

C

We don't really use destination properties on the client and.

C

In the making decisions during reporting with the report, the metadata, but we don't okay, use that you check what.

B

I'm doing server-side check and I'm deciding do putting my behavior based on who's. Calling me if I can trust the identity of me versus not I can make different decisions right. So that's what connection tell.

C

Us right now is for it's basically only for.

A

Server-Side use.

C

And it's telling, if client or indicated so that's.

D

B

That issue, and we just need to preserve.

C

That correct, so they.

B

Okay, so nothing.

A

Else when path.

B

Is not as interesting yeah.

C

Also sign somebody asked for like more metadata about TLS, like which version of jealous was used, might need to piggyback as part of.

E

His telescope yeah, they also want other information. Exert exploration, insisted that yes, yeah.

C

E

C

When you'd like to basically get all the killers, maybe we need like a specialist so.

A

It sounds like we can go ahead with this. There aren't any real objections to.

A

The other thing, I guess quite before you leave- is the G RPC message: counters doc.

E

Did you want to talk.

C

To that quickly,.

E

C

So there is a problem right now that we don't communicate enough about streaming.

D

C

Calls because it they look like single long HTTP to request. In particular, we don't know how many messages were sent and that's a problem, because some of these two is actually streaming. Yes, yes,.

D

Xds is bi-directional screaming.

C

And that doesn't report enough metadata. That was one of the reasons why we use.

D

Library, instead.

C

Of proxy and so.

D

C

Is a solution to that because it's very easy to detect messages in your PC, so this miss proposal just suggest to add more PC metadata to report number of messages in both directions. So.

B

The one thing I'm wondering about this is whether these attributes need to be our PC specific or if they should be more generic. If you imagine at some point when, when everything else is done- and we start looking at supporting other protocol, that way are you gonna have the same issue so should we start building a set of generic attributes that can be applied with these other protocols as well? So.

A

Instead of requests here, we see, you would think they request that streaming message count or yeah.

B

Something like that right, so I'm, so you got this long long, running, open connection to my sequel. Then you sending commands to it. You'd want to use the same same attributes to report this stuff or or Kafka messages.

C

B

C

Think I will need to understand exactly how using some other protocol. So me, more sense, I mean Kafka seems reasonable. It's just a message based right.

B

But so right, so the idea is we. We already report the telemetry about setting up and tearing down all these connections. Now we want to go one level deeper and look at what's inside these these transports and provide some some information about that. So I think message: message related stuff seems like the 8th route, pretty generic kind of cancer right.

C

Do we wanna keep good question response as the top level out there object, that's Kafka using.

F

That notion request these phones, what I'm sure but yeah we at least need to have two or three before we can safely abstract yeah.

C

That's I, don't wanna like all the quest in the confusing, so maybe we need to level extreme message.

C

That's why I wanted to understand. How are the protocol used? What's the terminology, do you like this so.

B

Request is about the overall kind of TCP level connection right, H to begin, and then you look at the thing below is a different category. Yeah I think that's.

C

Also implemented at the end, what an invention so I Risa.

H

C

Poverty PR and a test and there's a I measure, the performance cost. You know very heavy Derbyshire, Stephen Harmon and it's it's about four percent, so it's not insignificant, but it's not bad. This.

A

Is reporting at the end of the connection.

E

So we don't know what that rate would be if we were had a time on there or something so.

C

Overall, cost of the function to process data buffers and at the end of the proxy is about 4% I once ate supper.

B

We shouldn't be sending these messages at the editor connection, but.

C

There is a loop that counts messages, I were just measuring. How much does it function constant, but the real of limitation should be sent the messages every once in a while that the time will buy stuff. It can happen in parallel, because we also need to report like bytes, and we don't already so so that best pull up a token on, because within only even two, it work, I'm Tanner.

D

C

Not dependent on messages, okay,.

F

But just counting the Murph message takes 4% CPU.

C

In a very heavy to PC base, that only does this particular counting of a person okay. So it's not insignificant.

F

Also also, it's a bit surprising tip, so we also do this in for TCP, like the TCP, we.

C

Just do for byte count. So come the public counting the length buffer is fine. It's just kicking into buffered. That's what caused all right.

F

So yeah so that so it that it actually has to go through framing of messages.

F

So it's doing parsing, which it.

C

Otherwise would not very limit the person, but all it does is just like five bytes what's wrong right and that cost four percent, so it's surprising but I'm I want to ask and what people that's my bad mutation.

B

Because you're touching cache caches casual eyes that haven't been touched before.

C

My miles my job is, his dream is also quite synthetic, so it could be just bad courtesy.

B

Okay, so if we, if we enable this, if we put this in this, is just a applied systematically on algae or PC connected yes,.

C

And the cost is primarily for streaming if it's not streaming unary, it just is five bytes ten bytes per each critical, so.

B

This is another case of something that we need to turn off if nobody's looking.

C

Well, I can measure it for a union for nonce not streaming, and if it's like zero, then I don't think we need to turn off for me, but I guess we will indicate if of courses, if.

B

Somebody is using streaming, but it's not looking at these metrics. We should not reflecting.

C

B

Yeah, that's just wrong yeah, so we'll have a bunch of those in the mix. Ruby I think yeah.

C

That's probably most the most sophisticated count counting you're doing makes it feel.

F

Even more oh well, yeah, most of the other things are really available doing the other thing that we count like I said, is TCP bytes, where we on data, we just look at the buffer and see how much side.

C

You don't wanna collect direction with us.

F

C

The box, you just okay, so then.

A

Wondering if it about this extraction, whether it's valuable to have high-level counts or if they should be protocol. Specific attributes like if they map well onto each other I'll, be interested to see.

E

So I think I'm looking.

C

E

C

Interesting because this is technically not necessarily HTTP to base, but it is in practice, so I mean stops. Package received its own category makes sense. If you read by the practices, it's pitiful, so yeah I.

A

I just want to move on first time. I put on the agenda was the release, notes PR, so I want to encourage everyone to take a look at the release, notes and see. If we've missing things there I think we have some documentation that we need to clean up with things, especially in the policy until I'm, chair Adams. This is a p0. If we're getting release one one out, it's getting the release, notes, solidified, so courage, everyone take a look at that. Yeah.

B

So there's several items in there that have TBD or links to Docs and if we don't have any Docs for those things right now, let's just file an issue and then remove the TBD.

B

But if there are Docs, that's for the proper pointer.

A

Yeah- and we should, if there's anything, that we don't have there, that should be called out. That's that's the other big thing right, yeah.

B

But if you do, if the, if you do have a favorite feature, that's mentioned, but not documented right, the talk today it it is otherwise nobody's gonna use your feature.

A

Okay, the next thing I wanted to talk about was the roadmap for went to that I know: we've had a bunch of people join, so I want to just pause for a second and see if there's anything else, people think should be on the agenda. We should talk about before we do that.

A

Okay, so I put together this sort of draft set of things that I think we should tackle in the 1/2 timeframe and then beyond that, as well as sort of try and give us a little bit of thought ahead. This is in line with sort of the direction that the working group should present to a larger community. What they're working on for the next couple months, so I thought I'd try to organize thoughts.

A

Here, I tried to stick to the theme of 1/2 being about focused on stability and improving the user experience what we have instead of adding new features. So that's mainly how I've tried to organize the work items. I can go through them, one by one I guess, but so, if that's the high-level overview, so we've had a lot of reports about a lot of questions on the discussion forums about add-ons and how to use them. How to integrate with, say I already have Jaeger installed. How do I get it to integrate with this?

A

Do of that kind of thing, so I feel like a lot of. What we need to do is focus on documentation around how to use yeah the add-ons links about how to do production. Ization of them, then maybe do some support of existing things like operators for those those add-ons there also a number of issues that people have opened about things they'd like to see in the dashboard, so I added them here is sort of addressing the the known issues, the dashboards.

A

So any questions about that or things that we don't think are appropriate for one two in the first section.

A

Okay, going further is the best sports part here, so I.

F

Know you had asked.

D

About separate.

A

Release workflow I think we can improve the performance of the dashboard queries by doing pre-computation for Prometheus.

F

E

A

That'll help it's killed.

F

A

Then there are some open github issues that people have complained about so I'm. Switching the points versus lines all other stuff looks the separate.

F

Release workflow about so one one issue that we've had frequently is: we either have an enhancement to the dashboard which otherwise satisfies rest of late care. It doesn't require any new counters or anything like that, but that has to wait for an Easter release, which is which is basically unnecessary. So but a new dashboard is very much useful to people right like some someone added something and now yeah we want to get it out kind of orthogonal to when the next is to release goes up.

A

Interesting yeah, you don't have a good way of labeling, which version of the dashboard goes with post version.

B

Excusing is it just a dashboard or so what? What is the dashboard? In fact, it's.

A

The set of Griffin and JSON files you can dump in and see dashboards right and your father. Okay, because.

B

There's also the binary right, the key, the which version that the binary doesn't that is not affected, which version of which binary sorry Jiali. Oh.

A

Oh so I think we're talking about the graph on that dashboard key, also there's something that needs to be done there. Probably as well and I know. I saw John on yeah I'm here, okay, so this is.

F

Just right so with Griffin ah right, it hasn't actually at least I've never heard. Anyone specifically saying oh I wanted the latest version of Griffin of binary and I have to wait too long for that. Okay. So it's just our configuration state, but right.

A

Now the co-founder team is asked us to keep uptodate. Yes. Is that correct?

A

The only thing I think this sort of hidden in here is this first class tracing API, and that is sort of a new feature, but as we start to support more and more tracing options like lights, that people are asking out for open tracing based on boy integrations, I feel like we should have a unified way to talk about tracing, so mixer isn't doing its own separate tracing flags that you know have to be configured differently than the rest of the system. That kind of thing okay,.

B

So by API here you mean Co, DS, yeah and.

A

In not doing it through a pilot environment variable for sampling, for instance, are you having some sort of proper support.

A

The section is about just custom customizing. How telemetry in this way, probably also include policy, is done per workload. I think we've had a lot of requests to control at a finer-grained resolution than we currently have, whether or not we're generating access logs for debugging. How treat you know can I some work loads from tracing turning policy on off that kind of thing, so we've used annotations for some of this and I think we should expand that support and then clean up. How that all works in this document.

D

A

That's sort of my static describing what needs to be done there. I know: we've talked about the the stats generation filtering how that, but we don't have sensible defaults there. Maybe we improve that so that there's sensible defaults. We keep people from treating themselves in the foot. That kind of thing.

C

Does I think what the enable policy only visible sure yeah.

A

Gary I thought you wanted to add something about: adaptive sampling, I.

I

Mean I think it might be covered kind of what you're talking about with the the tracing API CRD. It depends what resolution you were thinking about going down to, but I mean in Jaeger there's a beginnings of an adaptive sampling mechanism that allows users to configure the sampling rate based on services or individual endpoints, and so that's to enable them to you know users to be able to focus on where the the errors may be occurring. So you know to reduce the amount of unnecessary tracing information. That's been captured so thinking.

I

If we can build a similar type of mechanism into this do and then maybe have you know other high level, tooling, possibly key ally that would allow users to easily you know enable a high level of sampling when they find. You know a certain type of request. These causing problems. You know it allow. You know more fine, grained adjustments, that's.

A

E

I, like that, a lot.

A

Okay, the next section I think that we can do a lot to sort of improve the way we collect things, including if there's some PR, as it floated to exempt some metrics from liveliness and readiness checks, but I think we just need to universally apply that the third complaints about the fact that we have our logs out of the box when is installed so I think the next two lines are about auditing when and why we generate logs across all ICO components and try and clean that up.

A

A lot I think that that's something that this working group can take on and also improve their value right and then then the next bit is about what extra information we can expose to based on data that we already have you know, should. Should we make regions own locality, information available for logs traces et cetera?

A

What else do you mean we talked earlier about? What else should we report about in TLS? So that's why I included that information here and then we started when we were talking about policy on and off we didn't know.

A

We didn't have a good way of answering questions like are there policy rules in place in mixer, so I want to add some some metrics around that, as well as providing examples on say if I'm monitoring pilot, how do I know when things are going bad and should we provide example, alerts to show people how they can monitor proactively, monitor these steel components themselves and maybe extend that to the mesh. So that's what that last bullet point. So, though, does that make sense.

F

Two more things: what one is the I think? Maybe you mix not in the lightness but mixer restarts when Kali is not available. Is that it did you cover think of it, okay, so so, basically, the the out-of-the-box experience sometimes actually often is that you see mixer has already gone through a couple of restarts right away and then then the operator needs to go and check by okay. What why didn't? We start if it's something bad happened like so, we do have to do yeah, so I think we need to do something there.

F

So just start up the pellet see so yeah there is a startup dependence on galley and sometimes galley is not ready and then yeah mixer I saw something violently starts everything restarts essentially and then, when you, when you look at the cluster even half an hour, let us say you see: okay, these have restarted and then that's like almost a compulsion to go and investigate what happened.

B

For a couple minutes right, yeah.

D

A

And then I, just sort of had this catch-all category at the bottom or I think there's a lot of work that we need to do. I think we can improve our documentation and test coverage for the process. Adapter with a couple of scenarios, I've been asked questions about how it works across clusters, boundaries and things like that. I don't have great answers there, so it'd be nice.

A

If we could put those into FAQs or updated on from that there's some cleanup, then we need to do I mean we deprecated adapters to your DS, but not really because it took so long get one that one out and I think it's time to. Finally, just turn it off stop pushing CDs in that also improve our install experience.

A

We've had a couple issues with the ingress gateway where people are calling from inside the mesh and it's forwarding sto headers and then that's changing source information and hiding in so it looks like traffic is coming from two places. Instead of one place: oh okay, yeah that I had okay and I know: we've we've. We thought.

G

Before thought about it,.

A

Before we've been yeah at least yeah, we didn't think that it would be an issue right away right, but I think in the one two time frame you want to get out and then I write below the bottom I think it fits in with some of the v-2 stuff. That's going on where we're gonna have these configuration free implementations of default, metrics and telemetry, and so maybe this is the time to start documenting that and providing sort of here's a simpler way to adopt sort of partially adopt SEO. So I don't know.

A

If that one thing we want to do in the 1/2 timeframe, but threw that out. There is something we can consider.

B

So when we for v2, we can't order the existing in process and there's we're not quite sure when v2 is going to be ready. When should we start officially deprecating the in process, adaptors o.

F

In 1.2 timeframe,.

A

Yeah I mean I think we should move on everything out of in process everything, except maybe the standard, IO right right, yeah, just just some debug stuff, oh I'm,.

B

Thinking in terms of the like the multi process, the multi release messaging, we when V 2 is available we'd like to to not have the any burden of the legacy. Adapters need to be removed from the system by the time T 2 hits. So we need to have some advance notice to tell people. Maybe in 1.1 we say: hey in 1.2, we're gonna delete those. So so so it looks like.

F

Computer, correct me if I'm wrong, but you already have you you have had that PR kind of ready for a.

J

While or yes correct yes, but the problem is I have that too, but I cannot try it for effect, they took how how is all adaptors but I'm not sure, like whether also doctors who worked well with those generated I mean sulla adaptor, author right.

F

Right, okay, so I think I think that right after 1.1 is the perfect time to cue that up and say: ok here we did migration with the caveat that what you mentioned and then ok, adapter authors, please like please, go and make sure that all your adapters are still I mean. If the adapters have good test coverage, I think with integration test, you should be able to tell if it works or not. If they don't, then we can ask the reporters to bring it kind of back up to code.

B

So so do we want to make a statement in the 1.1 release, notes and maybe somewhere else to say we are we in process adapters are deprecated, do not write new ones in 1.2, they will be removed, yeah I think I. Think that's. That's.

F

Yeah I think we should definitely put that. Ok, ok, so now the question is: do you think that, because 1.1 release notes are so dense, it would be I mean the alternative. Is we send out a separate email saying that a like this is happening and okay now we just moved everything out using a tool also.

B

So we can put it on the the wiki pages to describe in process adapters. Okay,.

G

B

Using those we'll put it in release notes in a deprecation, maybe you should have a deprecation section dedicated to this kind of stuff, because there's a couple more things that are being deprecated and there's nothing wrong with having a blog post as well. Okay,.

A

And we can also put on the forum's right I think that's something else.

E

A

The place okay.

D

Is there or should there be a framework for developing these adapters, or maybe maybe an example? Well I, don't want to say example, maybe maybe a project that that could be a base. You know for creating these things that you know takes care of a lot of the details that you end up having to deal with so.

B

Basically, an empty repo kind of thing, yeah, something.

D

Like that well, yeah.

F

D

But, but not not example in the sense of it's not really for primetime, but one that's actually. This is a good base to start from. That's a great idea.

H

Do not have my thing that already I don't know again.

F

So so, for example, we have we have a couple of adapters right that we don't officially use.

J

So Messick's I see example, oh so well.

B

Hey I think it's cause looking in more than just an adapter right, so the we could do. We could do a full stand-alone repo that building yeah.

D

B

D

F

D

But I'm not sure you know I think there should be an official one. Okay, we.

A

Used to have one that we were using as examples for how to compile and process ones from out of tree all right, I.

D

Don't know what happened to that repo.

A

But we could kill that and repurpose it or strip all that ugliness out of there's. A lot of that was basil, based I think even right.

B

A

That time, I don't.

B

Think that repo exists I think it.

F

Kills that okay.

B

So we have the contributor repo, that's right, but this this would be I. Think a standalone repo that just builds an adapter I think that's a put the big file in there and everything yeah.

K

So, a few months back, I had suggested that we create a sub command to sto cuttle that could generate the scaffolding for raptor. So I think that is something which we could do in addition to an empty II repository.

B

So, what's the benefit of having a tool spit out naked text file, rather than just give you the text file I mean.

K

I think that's a better word for yeah. That's the only reason yeah, but we can build some things around it maybe like to package I mean I, have more ideas of you like. We could create a standard packaging tool along with this, and no things like that. Unless.

D

Unless it relies on some maintainable scaffolding, outside of that, though, you, how do you, how do you deal with the bugs and updates to it? I I would like, like maybe, if there's like I said, if there's a framework or a library or something that could be relied on, that could be maintained, and that would you know, have downstream effects or could have downstream effects on existing adapters that you could pull from. Actually.

B

Okay, so that so that's it that's a little different though well.

D

It it is it's a little different, yes, I mean it could. If you base your project on the scaffold project, then you could choose to pull things from it. Yeah.

B

Yeah, we need to invest more in the doctors.

D

Speaking of which has there been any other thought to secure configuration for adaptors.

G

Configuration like.

D

Like secret stuff, because a lot of the adapters have to connect to external things right- and we have talked about this before- but but an idea of having secrets that isn't tied specifically to the implementation. So if I create an adapter right now for me to have secure configuration, I would have to say: okay, where I'm going to use kubernetes secrets. But then that only works in kubernetes I mean.

B

It's a society.

D

B

Would the security working group to have them exposed a canonical secrets API, so an abstraction on top of kubernetes that we could use I, don't think I saw an ETA for that I'll I'll Pingdom.

D

Yeah I did the same. Oh yeah I started the conversation about it, but I was going to actually put together proposal. I haven't had a chance to do that. I just wanted to raise it here.

D

As a you know, I'm going adapter issues as well, oh and one other thing, while I'm talking I had a proposed an issue raised a long time ago, like a couple of years ago, now I guess to have more requests and response time stamps to be able to bracket performance information or incoming and outgoing things like the stuff that you that's useful in a in a proxy setting for like start and end of received time sent time. You know that kind of thing so that you could fully measure the performance of every hop.

B

So those would be new attributes, yeah yeah.

D

I can this was the old issue. It just ended up getting time down as.

A

You say I feel like we had a fair amount of discussion on that issue.

D

D

They just put it in the chat yeah.

A

I'm trying to look for these- oh oh sure, the chat here. Yes, 17:01.

A

A

Okay point: the section of code: didn't really do anything.

H

Easier conceal this.

F

Yeah, oh, it's actually unstated on and reopen. Well, it's worth this. It's closed yeah, so we have to reopen it.

B

A

I didn't know that there was such something. Well! Okay! Oh yes! If you, if you really like an issue, you can mark it as no scale and oh I'm,.

H

Just gonna do that systematically.

H

Kills those well.

A

How they no no Reaper tender? Okay, so those little hacks of the whole label system, including you, can just assign yourself the DTM and approve labels and.

D

Bypass the whole system, too,.

A

B

Those are good, that's all for this kind of stuff is that's. Definitely, it's gonna be easier. Once mixture is in the envoy, because we're not paying a big tax to sand the Byzantine attributes and I think we do need to have the support to turn off functionality. If we observe that there's no nobody's consuming the particular attributes.

B

Then that'll give us the freedom to go crazy. It's.

A

As any work started anywhere on tuning attribute generation by a system, config no.

B

No that'll that'll be in galley, 41.3, I. Think basically, okay,.

A

Okay, well, just to complete I mean we can I mean they should put that in here. It is.

A

So then, sort of my thought about stuff: that's going on beyond the immediate improvements. Obviously any of the v-2 and ecosystem related development I feel like it's going to start to show up post one.

A

There was a request to have different failure. Modes per a doctor and I think that's an interesting feature. I just don't think that's something we can get done in the 1-1 in the one time frame.

A

Maybe maybe we can yeah I mean it's.

F

A it's a it's a simpler future and in in some sense, cross-cutting. Well, you can just the API I think that stuff yeah, that that's t-shirt, change, deep yeah well,.

B

It requires us to introduce the notion of mixture controlled per adapter config state, which we don't have right. I.

B

Guess, well we have the connection state now correct.

F

B

Will place to put.

F

B

F

Okay, this was asked a year ago urgently and then I suppose the other agency went away if.

A

Something she could clear a state fight, maybe.

F

It came back up again.

A

With the Redis, that's right, yes,.

B

A

We were, we were failing close, but when errors are happening, that kind of things that remind me of this issue Oh what I did so.

F

I, don't think we need this to solve that right, but this is a semantic difference between quota and chicks. Yes, well, this is sort of my.

A

Father like what a.

G

Man is that we should.

A

Support something like Eva's: it's the same thing with the sampling support like changing API or even we sort of had the support hidden in the agai I was actually doing the implementation of staffing support.

A

That could arguably be done in a 1/2 timeframe, but I just didn't know where to draw the line, and then there's been requests recently for but rate-limiting at the ingress and treating ingress specially so I don't know what else we want to do there and sort of ill defined, but I put I called that out and then I was curious about whether integrations add-ons we want to adopt and support.

A

We started to have a very limited WebSocket support now and so it'd be nice to provide attributes around when connections are upgraded and maybe start supporting some basic analytics. So I'm gathering signals to make recommendations for mesh at some point. There's a question about mirrored traffic and I. Don't think we have any good answers about what the right way to monitor your traffic is.

A

That's something we should look into post sort of one two and there have been suggestions of providing sort of proxy Lissa steel. Fridays may be the mixer v2 functionality inside of GRDC, and so I don't know. Well, we need to do there so sort of group that, as a possible thing to work on and then sort of I was also thinking. We have this experimental command line for consuming sort of providing a dashboard. What else could we do on the command line policy into?

A

Why don't you really just sort of make that more interesting, so I don't know if other people have other ideas of things that should be looked at or we should be started planning for, but this would be the spot. So please comment and add to this. Just sort of just things that came to me came to my head. The long list, Yeah right, yeah.

D

A

Think it's useful to sort of start gathering that list, so we can there. We know what to do next time around, for this exact exercise.

A

And that's that's it so then the only other thing I haven't the agenda and other people please suggest items as they come up. That was on the door. Maybe do you want to provide any update on the waz on v2 work and where that stands and we're headed so.

F

So they so the sandbox API is building up to be coming useful and we are to kind of test it out.

F

We are starting the implementation of a of a Prometheus adapter, using that that API and and part of it is that you want to demo something quickly and, and the other more important part of it is that go to access the API and kind of iron out any kinks that let me see right away so so the idea would be that the idea is that that adapter is going to is going to generate as close to standardise, to metrics as possible.

F

And then the layout of process, adapter adapter, will follow after that, that that is the way for proxies to directly call and they are post adapters.

A

Issues running into or where help is needed, or so.

F

I think I think right now, right now we are okay, I mean, but right after we do, the do the know can take like going from no config to config is gonna. It's going to need a lot more plumbing in lot more thought. So that's that's kind of right afterwards. Wait but I think right now, it's your find. It's.

G

Mid March till the target or is that's, the religion is 1.

F

Y is Pradesh mid-march.

G

F

No longer the 5x monarchs in end of March yeah, we are, we are with Mars.

F

Yes, I I need to go and update its focus.

J

So for that adapter face all mixer configure will be na by the Muslim or, like all current mixer coffee. Yes, so so we I mean.

F

Inherently, we were only collecting, like the configuration for mixer, that we are collecting and we have been collecting. Is it doesn't.

G

Really have much to do it like specific implementation, right yeah, but.

F

As you say, there will be a lot of promise- Mike, yes yeah, so so the so. Why the time out of process that after kind of fully start working? Yes, we we, we do mean the the pilot side and, like all that for me to start there are. There are some easy ways.

F

So, for example, it still already supports the on work will Drake, so we can actually configure on what they drink and like push these adapters in with some things that are statically configured up front and have a working system without all the config being fully dynamically in the world. So there are, there are still multiple paths and it won't block right away, but eventually, it's being given the lead altar.

A

That's it for the agenda. Is there anything else anyone wants to bring up or talk about before? We know.

F

Is tio policy is turned off by default? That's that's in the release, notes and it's not yet I guess it would.

D

Mean their release, notes.

F

Shortly so I just want to make sure that this group is very well aware of it. So if we hear things like oh, this is not yeah. This whitelist is not working or something like that. Then that would be one of the first things to check. Saying: hey. Did you turn policy back on and.

B

Hopefully, by one point three: we can just remove that option. Then it'll just be automatic, yeah.

A

Kurt, but to be clear, this will still install the policy deployment. What's.

F

The way this time actually I hope you ought to rename all that, but I don't think it got in.

F

So let me let me let me just check that quick, well I will I will update one go one way or another and it will be in the so what the current state will be in the release notes, but I kind of lost track of that here yeah. We should probably.

A

Have something on the website for those policy tasks or separate tasks and says here's how to turn policy on what.

D

What will actually happen if you had the policy CR I see, are these in place and policy is not on nothing. You get no warning, nothing, it just nothing happens or yeah. No.

F

One, no one consumes it and nothing happens. Okay, that's why I said the the.

D

F

Is not working okay first thing to check is see if policies turn like you need policy, and is it turned on okay,.

A

B

It's kind of sad because they're, given the, if you don't pay attention, your cluster keeps working. None of your checks are applied right.

A

Yeah and if we don't have the PRN for the deployment you also, the upgrade itself is wonky, because you'll you'll be using an old version of mixer. Okay.

F

Further so I I'm glad you know the PR did get in and and that the policy.

D

Deployment is there just.

F

The checks are turned off, so you used to see the parts and all that, but the checks will be turned off by default.

B

Okay, so is there any way we can them someplace, where we could check for the user thing hey, you have. You currently are using policies in your cluster.

F

So we so did the operator steel could be yeah yeah, so the operator would would would would do that, but I mean since we don't have it right now, but the operator would exactly be able do that. To say hey looks like you, you don't have policy, so it's safe to switch it off when I move over or you do have policy and yeah I'm not going to switch it off. So if we had operator it could have done it. Okay,.

B

So, but right now, I'd feel better. If we had a if we could tell the users run this command line and it'll make sure everything's okay for you, so that command line will check to see if there's resources. If there are then it'll turn on the policy check, if there's no resources leaves it off. Okay,.

F

So you want so you want the operator command to be made available as like an ad-hoc thing. It's just so we can stick it in there. Okay, okay! So that's a that's a good idea, so I I think we could. We could couch it as as a release like new release, validation tool or something like that right that says I mean the operator will do that. Actually, but even if, without the operator, someone can run and say, I, just upgraded and I want to see if the stuff that I have is still.

B

Yeah, it could be a it's your cuddle thing, how little make sure that works, status, I.

A

Have to think about this again, isn't the problem that we can always check for check templates that we know about yes,.

B

That's right, we can't well, we can check for the existence of any check template, can't we. Why not? We.

A

Know the company needed: how do we easily do that? We're gonna, pull down the the proto descriptors and pull out the annotations so.

F

So, actually, to do to do it correctly. We have to do what they're saying. However, we don't have to go all the way right. What we can very easily detect is whether you were using previous default configuration which had no check adapters deployed right and if we find anything else, apart from that we'll say: hey it's unsafe, you go figure it out, so I don't know in order to do it completely.

F

I agree we'll have to discuss, but yeah the simplest tool is just see if there are no check adapters and just see if it's the previous list and then you're done actually.

B

Could this just be an option on the mix mix s?

B

It reads all this series? Okay and then what would it do just so? You can just run mix s specifically saying check my deployment and then it'll it'll grovel through the CR DS, and tell you whether you have any any policy checks going.

A

So I think all that is possible. My concern is whether or not that's gonna be done before at least probably not.

D

A

Know we need something that can be done in the next half day.

B

Okay, then, the cheap and easy stuff, okay.

A

Okay, well, I think we're right up against time. So yeah you should probably go.

E

Already, thank you you later so on Thanks.