Istio Technical Oversight Committee, 7 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Technical Oversight Committee 2021/06/07

Description

Istio's Technical Oversight Committee for June 6th, 2021.
Topics:
- Discussion of breaking change on Header Case.
- Nominations for 1.11 release managers.
- Grafana inclusion
- Environments roadmap for 1.11

A

B

A

Yeah, so still working on an experienced volunteer for the to build, um so I'll do some following up internally.

A

Actually, let me assign that to myself.

A

Kevin's on here, all right.

A

Brian um nate, I think we talked about istio with mcs the last time. I don't know if you've gotten the reviews you need yet. I certainly haven't had time to do it.

C

So should we just leave it um at this point I mean I've already got the approvals I need. It was really more or less just to kind of inform the community, so we can probably remove it from the.

B

A

uh John, are you here you want to talk about header casing,.

D

uh Yeah, I am here um so this is a kind of similar topic to the one. We talked about a lot previously with, like the local host forwarding, where we have a new behavior that I think most people would agree is better, um but we have backwards compatibility concerns, and so we wanted some guidance on how people feel about this one.

D

um So, basically the background is that envoy for http, one will lower case all uh header keys, which is valid in the spec that they're supposed to be case insensitive. uh The problem is that a bunch of people write bad applications that don't follow the spec, um so they get broken when they move to east geo.

D

um So envoy kind of resisted doing this preserve casing for a while, because they wanted to call the specs, but eventually enough enough, users were broken by it that they, they kind of caved. um It seems like, as we want to be fairly transparent. That may be a good idea for us as well, so there's less noticeable changes. When you know users drop these through into their their existing applications.

D

Now there is the risk that someone has come to depend on it being lowercased and is now actually broken if we unlowercase it. That seems kind of odd, because I would think that your application, probably originally was broken by expecting it to have.

D

You know the original casing and then you install envoy and you realize that it's broken and then you fix it properly, instead of just changing it to be lowercase only, um but I'm sure there's at least one person out there. That will be broken, so I guess yeah. The question is: how are people feeling about this? We can add an api fly. That's like you know some global or per proxy fly. That's like uh change the casing uh yeah. I just want to get some some feedback on this.

A

Go ahead mirrors.

E

A quick question john, I think I asked this before, but it will be good for everyone else on toc to hear it. So first is.

E

Does it affect any authorization policy or routing policy.

D

uh That's a good question. We should verify that. I expect no, if not it's probably a bug um or security bug, because it you know, everything in envoy should be doing things case, insensitively on header keys.

D

Now it could affect if they had like external authorization, maybe and that's changed or if they have like. I don't know a custom, lua filter or wasn't filter. Maybe that was doing the incorrect uh matching on the header, but I think, like the core stuff, I mean we should definitely double check, but I would be surprised if they did so.

E

Can we capture that, if you don't mind so my.

D

Request would be to.

E

Basically understand those behaviors and document it either way whether.

F

We make it the default.

E

Or not so that people understand what they're gonna be the effects on.

E

B

A

So someone that has written my two sentences- those are your two cents, john. I presume.

D

Oh yeah, those were mine, yeah.

A

I kind of mentioned.

D

That uh that's what I was describing.

A

Okay, so there there are some tricky things with this right.

G

Oh justin, do you want to go ahead and just just quickly? um Yao ming is doing some work around some of the security vulnerabilities that have been popping up around normalization. So I would just talk to him because he has a huge spreadsheet now of sort of the behavior of lots of different back ends and how they respond to different things and like what flags we want to set. So if you haven't talked to him, I would talk to him just because I think this will play nicely with some of that work. That he's doing.

G

A

So it's probably worth mentioning and feel free to correct me if I'm wrong, but this this problem is an http http. One only thing right, because in http 2 and in quick http 3, the the lower casing of header is a strictly enforced by the user.

D

Yeah, this is only h31.

A

So, no matter what happens right, there's a broad class of user agents that are going to dump everything back down into lower case in all cases. And if we start using http 2 as a transport for http 1.

A

By upgrading right, like the client's heart, sidecar uses http 2 as the transport I don't know, we can do this right, I'm not sure it's actually possible to do this.

D

A

Right in envoy, like I, I would be very surprised if they said retain the header case when they put stuff over http 2 transport. I don't think they they would or could do it.

A

um So that makes me a little bit wary about trying to do this. Like I understand it has broken some clients. Have we received a lot of complaints about this.

D

Mirage, I feel, like you had mentioned a few people complaining about this. Is that accurate, or did we all just assume that someone else was.

E

No, I mean my complaint is generally proxy should be transparent and they shouldn't be mutating things unless they have to uh whether people are relying on this or not. I don't know but, like you said, they're pretty, you can assume somebody will rely on this behavior either way, but I don't understand why we should mutate if we don't have to coming back to louie's question about http 2 transport, that's interesting when it converts things back to http1, to give it eventually to the backend louis.

E

What kind of guarantees are given for.

A

It follows the spec right, so the keys are maintained in lower case there's. No retention of case.

H

A

In http 3., it's very clear about it.

A

Right so like there's, there's a large number of http 11 and http 10 user agents out there that are.

A

Uncompliant in lots of ways.

A

As more and more traffic goes over, http 2 and 3 they'll have to adapt and solve the backends.

A

Anything that's taking user traffic from a browser today right is seeing the headers normalized, I believe, all right. We can go and verify that chrome, any webkit-based browser anything right is going to see normalized lowercase headers right. So that's! You know a large percentage of you know: user agents so really you're talking about api clients at this point that aren't doing that, but anything going into a cdn at this point is probably going to have this effect. Anything coming in from a mobile device is going to have this effect.

F

But yeah we still need an exception. I mean for some special cases. I mean not only like.

A

I'm okay with having something, but it would be http 1, 1 and 1 0 only and it probably shouldn't be mesh wide.

F

And the follow up is age 1, because in h-bone I'm not entirely sure how it will travel through the whole whole hopes, uh because.

A

Right if we put http 1 over http 2, it's like we're not going to retain the case.

F

We can't, but but still in special cases, we can have the option somewhere, but maybe not as an api but as a yeah.

A

Right, that's going to add a lot of complexity to any transport right in hp yeah. So we're probably not going to do that that soon, which means it's almost better to give peop like retain the existing behavior and not oscillate when we add h-bone right.

A

And you're mirage like, I think it would be fine to do this for gateways yeah, but like gateway's talking to side cars if they use h-bone. That will be that right unless we explicitly have them forward http 1-1 and then that makes mtls more difficult and we are having a problem again.

F

Well, we can do http one one over html, but it's going to be very complicated to support each other right.

A

It's gonna be a lot of transport selection, complexity,.

E

Yeah, I think the direction is clearly http, 2 http3 at the transport and then do whatever you can on it right. So if the headers are going to be lost, so I think the problem is. This is a long tail of applications and a well-documented behavior of proxy shouldn't change things right. That's my.

E

Concern so so, john, what's the current state so currently we lower case everything and is there an option, that's provided or is there no option.

D

uh No no option in easter. You can onboard filter, though okay.

E

So it looks like then defaulting will be tricky, but providing an option is not bad right.

F

Well, it is bad because it cannot be supported, so it prevents us to move to hbo or make it impossible to support the api.

D

Yeah, it does seem kind of like the type of thing that we would maybe say, hey if your applications are broken by this just use on with filter yep.

F

And fix your app.

F

And reminder that we have the concept of blessed field as we could. We could add where we have a filter. That is doing exactly that and saying: hey just apply this filter.

A

So so we would add a filter that would say replace this lowercase key with this mixed case key before forwarding to the back end over http one one.

F

Yeah and include it and have it like, like the telemetry filter, so we support it and you know.

A

So does does envoy header rewriting retain case.

A

The existing envoy header, rewriting.

E

D

The key right, I don't know if you can rewrite a key yeah right.

A

Like that would be something maybe you could we could push upstream or as a a basic built-in filter.

F

Oh, you mean you mean at the end the last hope to rewrite the key. So if you get the uppercase, you expect but have all the transport. That's not a bad idea. Actually.

E

You will have duplicate keys in a way, but I.

F

Don't know you replace it, I mean if you know the back end, you could take a copy yeah. That's an idea.

E

I don't know if you can do that. Currently.

F

E

Okay, so for the transport you still get lowercase, then you convert it at the server side.

F

Security security applies to lower case. Everyone is happy lowercase, but at the very end you rewrite them.

E

That's not bad.

G

I'm just going to make a comment that I I agree in general that a proxy shouldn't be changing the behavior or the what's on the wire. But if we're making this look more and more like a security device and doing things like authorization policies, I I can see us doing more normalization so that the way that we're interpreting something that is impossible, that we normalize it to how we we parsed it and then forward that on to the back end,.

E

So that's the tricky piece, a tricky part right justin. I mean yeah normalization to account for policy and then what's given, eventually to the back end right most of the cases it makes sense to both of them are the same. But sometimes, if you fundamentally change things, then the back end is not going to be happy right, but h2 is defining things. That's lowercase! So there's no error in this case. It's fine right uh headers, but some other normalization. That might not be true.

F

Slash and other things but themselves yeah.

A

Those are, those are header values right, so exactly like, we already have, as justin mentioned, a lot of work in flight for that um right. So this is a much more narrow thing and it has. The spec is very well defined.

A

It's been well defined for about a decade.

A

Okay, so I guess there's there's two things in the short term right, so one let's go and check out the behaviors of h2 and h3 if they're forcibly normalizing right, that's basically that the class of traffic that's now not going to be captured would be.

A

You know, api clients that are not hdp2 capable right.

A

And then we can see if we can push into upstream envoy the ability to do header key rewriting.

A

Right as- and they might actually want that right for the same reasons.

A

Right just to satisfy application back end, um it's not any different than a form of canonicalization. Really it's just we let the user do it so and then we, the third thing we have to figure out- is that what scope or grain do we want people to do this.

A

Obviously, what envoy did didn't break the universe right, because envoy got a lot of adoption right. So there's enough people complaining that still probably a relatively small percentage of the market right so still, probably a relatively small percentage of applications, and that would indicate that this should probably not be a mesh white behavior right and it should be per workload. Behavior.

F

Does that sound about right, yep and also not an api, because we may not be able to support it.

A

Are they what sorry question.

F

Not an api because we may not be able to support it.

F

I mean a field are not not a first class apis that we guarantee to support.

A

Because yeah I mean.

E

E

So it looks like I think, at least from a user point of view, rama from salesforce took this over and his comment right now in the pr is they had a very high severity customer issue because of this.

A

And right, but I believe I believe, but it's probably quite important, that they fix it right. I thought like if that weren't the case. I would be arguing for not doing it at all.

A

I think we should do it.

F

And he has a filter already I mean he can apply the filter today. It's not that it's.

E

Blocked yeah, that's true question, but you know coming from him. I mean they have been burned with onward filter a lot below so.

F

But we'll also be burned if we, if we do zero positive, because who knows what will break and how many people and we've burned the age-born plants, because then we have even more difficulty.

I

He didn't make a comment that he was not able to do it for our future because of the proto merge limitations.

F

I

um Nested type config, so I'm not sure if filter would work for him.

D

Let's fix the filter, I mean it should it does.

F

D

There were some comments on there that it didn't, but we figured it out. um It was some bug and we had an old version of envoy or something like that.

I

Okay, so that's good, so at least he has a option to do it per workload and it.

J

I

Been running fine, you know without this for a long time, so they don't have a need to enable this for every single workload.

F

uh Workload and gateway, because you also need to if it goes through a gator. You need to be surprised. Yeah.

I

F

But maybe not for the entire measurement, just preserve it for that particular host and.

E

All right, I think this is fair, uh louis, uh let's keep it at onward filter for upper workload basis. For now uh I get like I guess the decision where we say can we make this on by default? That's a no right.

A

um Yeah, so what we should do is document how to do this.

E

This is reminds me overall louis. What are we losing by transporting http? 100 is gp2. What else are we losing.

A

A

Well, if people like to check content, transfer, encoding, chunk, headers um yeah.

E

A

Down conversion from h2 to h1 is uh you know it is canonicalized and standardized right if people were sending things that don't need to be chunked and transferring coding shunt. That would go away if you have an application. That's sensitive to that! Well, good luck! Frankly, but.

F

Boundaries cannot be enforced. There are nothing for that proxies, so you're already screwed. If you.

A

Yeah, like I would, you would be in.

F

F

Slightly different because of the randomization and I'm not sure which one is faster at this point.

E

Okay, I'll discuss this offline. I think I don't want to take up the list of dlc, but.

F

E

Interesting thing to look at when we start promoting it to also document the side effects the user might see because of the protocol. Translation.

A

Yeah they they'll see, you know the header normalization for sure.

A

They will see, they shouldn't see, header key reordering.

K

A

K

A

No, I'm sorry value reordering within within the scope of the key.

F

A

Most application code doesn't deal with that very well anyway, right they, some of them, give you just the first header or the last error right. Yes,.

H

A

Yes, okay, so no doubt there will be something, but it probably doesn't affect the overwhelming majority of users. Yeah.

E

I mean it's basically reconstructing at the other side, so yeah I mean the sequence of headers are going to change, but maybe the relative value within multiple.

E

Keys and does it do any sort of like condensing like coalescing or something like that when you pass it.

A

What do you mean coalescing.

F

You you can have a repeated hair that is concatenated. As far as I know,.

E

Yeah, that's what I was going to say so suppose.

F

E

Client x-4 r on two separate lines: right, that's valid in http 1.1.

E

How will it show up on the other side after you transport it on http 2? I think it will be put separately by commas.

A

Yep, um it's actually that's user agent dependent. It's! I think they were allowed to do either.

F

I think I standard this is congratulations. If I remember correctly, I can check, I think they have a uniqueness. Let me double check.

A

I don't think it's it's not a requirement of the spec, because either form was allowed in http.

A

Okay, so yeah, okay, so john, I think you have the answer to your question. Can we make this on by default? No yep.

A

Do we need an api to configure this? No.

L

I don't know right.

A

Now we should do some verification about what envoy's implementation in h2 is doing.

E

H

E

Interesting thanks for bringing it up now, I'm thinking so many ways of my legacy. Apps might be broken.

A

Yeah I mean that's at some point, I I don't know what the deprecation timeline for http will be 10 years 20 years, but it will happen.

A

All those sub clients out there are gonna have a whale of a time.

A

um I think we have an answer right so do we have many user reported issues on this in github.

D

It's it's not that common, okay.

M

A

Right so I mean the other thing there right I mean it's a pretty easy answer to users, which is think, look, here's the http 2 spec. If at some point your client goes to a cdn.

A

What do you want us to do all right if it's internal, east-west? Yes right, which is why we could we will offer them a workaround, but and we can try and make the work round easier. So I think that should probably be our focus, but.

D

Yeah, they can't always declare the port as tcp as well. If they're, yes,.

M

D

Http, even though they're very, very close to hdp.

A

Right, which is fine, actually yeah, maybe you want to capture that in the options.

A

A

um It didn't clear the port of tcp and they had protocol detection, though they would have a problem.

D

Right, if you exclusively declare a tcp, then you turn off the detection. Okay,.

A

Maybe that's probably the best recommendation in the short term.

I

So would they lose a lot of education, metrics, yeah, yeah.

A

Sure but right I mean they're, not sending http traffic and they're they're violating the spec.

A

B

Should at least motivate them.

A

Right right I mean if it's a small percentage of cases like the workaround is easy just to clear your port as tcp.

A

You know when and if you get some time go back and look at your app all right and then, but you know this is the easiest work around and then there's a harder one, which is something that does header manipulation. So you can turn it back on.

E

Yeah, I I think this is good. I think the nation became it should probably.

A

Lease managers- john: you brought this up.

F

D

I think I saw there's an info at the top of some people already volunteered.

A

Yes, we have two volunteers and I have an ai to follow.

B

A

Upgrade grafana to the agpl licensed version.

D

All right, so, basically, we somewhat shipped rifana like not really, but we shipped dashboards and like a sample deployment um using them they in grifana, like eight, I think they switched their license to a gpl which may be problematic, um there's some discussion on other projects which integrate slightly more with grafana about whether this was okay or not, and it seemed kind of unresolved. um So I just wanted to get confirmation from someone that we should do this so that I don't get in trouble if I bump the version.

D

um So we probably don't need to discuss it here, because I don't know if anyone here is like the right people to do this, but it'd be great. If we could get in contact with someone that can kind of give us the green.

A

Light steering has talked about this, um but kevin is probably the best informed.

A

E

You'll see kevin.

A

B

A

H

In me, yes, uh okay, uh so I mean I know that we've had discussions about it within red hat, but I'm not sure whether that would apply to issue. I know that from our side we so we ship griffin as part of our product, and we looked at agpl and we were told that because it's a separate part, that's not integrated into any of the source, it doesn't impact the source that we we ship. So it's a separate deployment, separate code base and everything.

H

So that was the advice that we came back with, but I'm not a lawyer so take that with a pinch of salt.

A

Okay, so red hat so I'll just record what you determined.

B

It wasn't integrated into code, it was a ship.

H

If there are specific concerns that john has, then I can go back and double check with the lawyers about that. Just in case there there's more context that might change their minds.

I

So we don't ship with grifana either right. The only thing we reference grifana is in the samples that we point to their image. That's it.

D

Yeah, I think very even we're even less uh like. I would be very surprised.

F

D

We were broken uh like, for example, linkerd is something similar, but they actually ship like a link or d, slash, grafana image, which is just wrapping it with like. I think they just have a docker image with their dashboards in it. Basically, um we literally use the upstream ground image directly, so it's yeah. It could be a big stretch if you can't even reference any yaml or something, but.

I

D

I

So we're a consumer of it. I mean we're not like shipping for them we're just promoting for them. Basically,.

A

Yeah, so it certainly seems like likely that we can continue to do what we are doing.

B

A

A

We just we need a lawyer, that's the job for steering.

A

So I we can take it up on friday. um I would be surprised if there's a problem given how all right we ship their distribution. Well, we don't ship it right. We just reference it in its original form completely. What's this configuration.

A

Okay, maybe I'll just it's kevin dude.

A

Yeah, I guess who, whose lawyers do we want to be the lawyer for this uh I'm happy to talk to google's lawyers uh kevin I don't know, does it do you want to talk to red hat or mirage? You want to talk to your lawyers, then.

J

uh Our lawyers are gonna, not come back. Okay, yeah.

I

We we, we probably won't, have a good turnaround. I don't see this as a problem. I mean I've worked in open source for a long time, but certainly good to run by a lawyer.

H

A

H

Larry I'll I'll see, if I can I'll ask the lawyers that I spoke to to see if I could share their advice with you, so that you can see what we have been told internally, uh but I I would certainly suggest you talk with google's lawyers about it. I suspect the answer is, as other people have said, we're not going to be impacted by this, but.

A

Yeah yeah, I I expect that it's, I guess the question right now is yeah. We can treat the the combined knowledge of google, google's lawyers and red hat's lawyers and just say that's. Those are the issue. Lawyers for the moment.

E

K

Check the steering committee.

E

Minutes- and it looks like the action item was louis- will get a description of how we use griffin.

A

You're expecting you're expecting me to remember something that happened two weeks ago.

E

No I'm just saying like so we should maybe get a description of how we use this to how we use grafana here so that you don't have to go around in circles.

A

I have a description right right. It's it's referenced as a configuration in in install tools right.

E

Yeah, I think that's good enough. All.

E

B

B

Type: okay uh shouldn't be too.

A

Hard, uh okay uh environments roadmap.

A

I am happy to present.

N

Okay, jacob is here yeah, I'm here.

A

Will be this time.

N

Let's see the right thing, uh we're still seeing the the toc meeting notes, yeah.

B

A

How about no excellent.

N

uh Yeah we do okay.

A

You want to just start working.

N

Your way down through it and tell me when you want me to scroll uh yeah, uh keep scrolling down uh until the environment's on is on the offset.

N

All right, yeah uh right there and then scroll to the right a little bit: okay, yeah uh all right yeah. So here's the current plan for uh 1.11 for environments, um we'll start with revision-based upgrades, um so revision based upgrades um have been alpha for a while. uh We hope to actually uh try to get it to beta uh for 1.11.. Unfortunately, we don't have an owner um for it um to actually push this promotion uh forward um and it's also pending uh cross version compatibility testing, which is listed above um so yeah.

N

Just given kind of you know, feedback uh that we've received from users actually getting kind of better support for for for multi-merger uh cross-compatibility is a concern in order to make this beta.

N

One of the other features around revision based upgrades that is currently in progress. Sam is spearheading the effort to get stable revision labels that were introduced from 1.10. um They are currently experimental, but the plan is to get them into alpha.

N

So uh I think the overall you know goal is to actually get this into beta, um but yeah again we're just looking for an owner for this um yeah and then kind of going on down um multi-cluster uh again multi-cluster.

N

uh Oh I'm sorry. I skipped that um externalist dod. uh This is another one that needs needs an owner as well, but the uh you know the hope is is, is you know if somebody can actually step up and again push this into beta? This has been alpha for a while, uh I believe, over a year. I could be wrong, um so we have kind of two action items uh that are p0 uh that are kind of needing an owner at this. At this time.

N

um I'm not sure exactly what that means, if, if, if we should kind of, uh you know ask for more input- or you know, put them down from p0 to p1, but you know the hope would be to actually get these uh both actually put into beta 41.11, if possible, um going on down the list as well. um It's a multi-cluster nate is going to be leading the effort from taking a multi-cluster from a beta to stable, um and I believe he has an istio enhancement, uh doc, ready um nate.

N

If you're on the line, uh I hope I'm not speaking out of turn.

C

It's uh it's actually transitioned to stephen landau, so he's going to oh.

N

It has okay all right, so I will update this document as well.

N

Thank you for that update, um yeah, uh yeah and then going in into helm. V3 um hello, v3 uh going from alpha to beta, was delayed for 1.10 um there's a little bit of rework that is being done for helm.

N

um Things have have moved from the base chart into istio discovery, uh and um we are currently evaluating kind of our approach to the crd and kind of how how helm manages the crd folders.

N

So I'm hoping to get this from alpha into beta for 1.11, and I should have a doc uh pretty soon here on on exactly the things that are required for for this to be done. um I know that there has been some concerns um around some of the uh changes to naming um that are currently underway. uh So hopefully we can mitigate that going forward and actually make this a bit a bit more stable and add more testing and whatnot and.

F

I'll write word on this topic that we have started to review the api surface for the helm and there is a document that um that is proposing, because we we have values that yaml and a lot of you know the the options in in installing this under the sum of all the options in the world. So we are trying to move to better proxy config parts of the values and other things, and we need all groups to be involved in this, because it's a pretty core api.

N

Yeah exactly um kind of speaking along, I guess similar lines: uh yeah just mesh conf, big cleanup, uh prox, config uh proposals that are currently in the design session or design discussion phase right, um which uh constant is kind of uh spearheading that effort.

N

um And then you know kind of the last thing um that we are hoping to actually get done for uh 1.11 uh john recently got um gateway injection um to alpha. I think it was last week or just a few days ago, and the plan would just be be to continue this work uh and try to get it into beta for uh 1.11..

N

So these are the major things that kind of themes that we're focusing on for 1.11. um but yeah. I I think the biggest thing that puts things at risk is is just kind of lacking an owner um for uh some of the topics um at bay. Right now really.

E

Take a quick question here or actually uh so looking at the operator row row 58 is ostromat is he's still working on istio. I thought he's no longer no longer working in the stu project, so you might need a owner for that too.

K

Wait, that's that that's martin ostrowski he's he's working on this too. Oh, he is sorry. I think I I confused the.

E

Github uh get her nicknames then, and then uh that's.

F

An operator again, the api surface is alpha and it's too large to move to better and part of the discussion about uh which, with what we can support long term, because it's it's. You know that one thousand dollars.

J

E

No, it makes sense and then uh revision based upgrade.

E

So, what's remaining from the user experience point of view.

F

From user experience, they're mostly out of the user view because of the tags uh and the ux for tags is still in progress. I mean it's, I know experimental missed your cattle, but it seems to be the future. I mean that's at least the consensus in you, environments.

E

So louis question, for you: do you think if, if you, the concept of tags remains that alpha right now, it's experimental suppose in the next release we make it alpha? I don't think we can make the whole revision based upgrades beta. If that's the main way, a user should use revision based upgrades.

F

You know tags depend on on divisions, I mean they use under. So underneath everything there is a revision uh so ux. It cannot be better, but.

A

So I mean, but it's it's allowable right. You can come to the toc for an exception about future progression right as long as you have met the requirements for beta, it can be considered right. We don't have to write blocks.

A

We don't have to do that. um So if it's, if it's clearly in the best interest of the project, to progress this faster, then we should make progress on that um and I think.

F

We only agreed to.

A

All agree that tags are better than the what was done right. It's easier for users than.

E

A

E

um So can we add that uh louis or jacob, basically that we should have the tags based? I don't know if it's another role, it's a part of revision based upgrade. Basically, they should kind of move in lock step. In my opinion, yep.

F

So we can try to make the plug button. It's it's it's! uh The underlying mechanism is similar. I'm fine with that.

I

Yeah, that's reasonable, so quick question. The revision based upgrade is that only for it's still cuddle not for helm, and I assume that that's only for control playing or does that also include the gateway.

F

It is, it is for study, it is also for help help supports revision-based, upgrade.

K

F

Well, but for gateways, that's different because we are moving towards the injection and and it's a separator with no 63.

J

F

Cnn is actually, I think, it's networking, but I don't know, uh but.

I

Ci is in place right.

F

Here is in place of course, oh yeah yeah, absolutely yeah, it's impossible to. So I think yeah.

N

I'm sorry, I'm sorry go. Firstly,.

I

Sorry, I was just going to say it's a little bit confusing because you have helm version 3 as a promotion, ad item and uh there's also some dependency or how you want to cut. Also revision based upgrade also schedule for promotion.

F

The xero token I mean helm, supports revision based upgrade and and it's it can be promoted upod independently of the revision.

I

Right but when you have that spata, you would assume you know, the revision-based upgrade for helm is also beta.

N

And I think that that's probably something we should add to to uh the beta document right for for helm, right.

E

uh Yeah, so in my, I think, lynn, what you're saying is actually very important, but in my opinion it is the whole helm based workflow. Whatever is done through helm is currently alpha.

E

It will go to beta if jacob and the team can actually meet the requirements. That includes, in my opinion, division based upgrades. They will also be alpha. They'll move to beta so yeah. So some of these features which you install and manage via helm, it's confusing, but I think that's the least confusing part. If you use helm, we should just say: hey the helm. Workflow is currently alpha.

E

uh Hopefully it gets beta at the same time as revision based upgrades, but if not, then revision based upgrades can move, but if you use helm to do that, you are basically choosing to use an alpha workflow.

F

Does that make sense so.

J

You will use.

F

Revision based, but the helm itself will be alpha if it sleeps exactly right.

I

And if revision was promoted to revision-based upgrade, it was promoted to beta before helm, then that beta is only for istiocado.

F

Is closer than the other?

F

The only problem for him is having people I've been having to see and approve the reduced api.

A

Okay, well, I I don't think we necessarily need to get into the line item details. The bigger item I hear from jacob is that we don't have owners all right so, like my instinctual reaction is the people who were the owners for bringing the feature into existence in the first place would be the default owner.

F

If they had the time.

E

Well, I think sam might be sam, has his hands with so maybe he might.

O

Be talking about.

E

O

uh Yeah, I just I wanted to say uh so. I think another thing keeping revision-based upgrades from beta is an api to control which revision handles validation and uh where we have like features on the way for that. But I think that before those are stable and tested in, um I don't think we can call revision based upgrades, beta or uh anything above experimental.

F

Okay, I mean not not everything must be better. At the same time, and I mean we have alpha fields in beta apis, it's a mix. We should not wait until everything is perfect. We can have.

I

Yeah but validation web book is pretty important.

F

Sure- and we have solutions for it- I mean you can see validation itself for validation apis alpha because it was just introduced.

E

I I think, whenever the promotion steps are done, you can just include the exceptions and then we can decide whether it meets the.

F

Bar or not, but uh sam.

E

Were you kind of volunteering and saying that you are working on the pieces that is, in your opinion, at least required for revision based upgrades to be uh to be promoted? So should we put your name against it or you are not volunteering, to put your name on.

O

Yes, you can put me as the owner of I'm uh definitely working on pieces of.

L

Promoting it so.

O

Thank you. Okay,.

A

And then for external sdod.

F

For external studio, we propose to to move it to leave it p1, because it's it's not critical. Vendors have their own solution. It's mostly for vendors that that run this to the outside and they have their own testing infrastructure, their own criteria.

E

Who was the driver for this lin? Were you driving it when you were part of ibm.

I

Yeah, I mean certainly I I remember one of the key thing we had trouble to promote to beta was the testing environment was different and we were using uh kind cluster. We didn't have real key and search uh for the gateway, so we had a little bit of difference between what's tested instead of io and what we recommend user to use. So I do agree with what constance said.

I

It is a little bit vendor specific um and not all the vendors offer uh istio as a service at the moment um and the key answers and the gateway configuration certainly varies for different clouds.

D

I don't know if it's vendor specific in the same way that, like we can't test an external load balancer with an acme certificate in kind either, but that's a stable, non-vendor-specific, core feature of east geo to you know, expose the tls service on a gateway.

D

It's not like external e2d is like vendor, managed ecud like people use it on their own self-run, estio instances right.

I

Right, yeah, that's a good point: the airbnb also use it. I mean they manage internally themselves and people use it too.

F

But really external studio is more like a concept, not necessarily as code, because it's just using stuff that already exists, and you know we just point the injection to an external url. But that's not.

A

I guess the question is what would be required to progress it right if we need an exception to say, look what in our test environment, you know we can't do an acme, cert, okay. Well, we have other ways of testing acne, search that we would or can have other ways of testing acne sorts. That would allow us to feel comfortable progressing the feature.

A

So what is you know? We can progress a feature with an exception right, like the test environment, can't do this. Okay,.

F

But my point was that external history is more like a concept. The fact that this year supports managed versions of east georgia externally and really what we, what the uc would should probably do is to say that hey vendors, that have managed are responsible to test their own platforms and uh you know, submit any fixes in case. There are breakages that disallows them from from from managing.

E

The custom lin just said that right, airbnb is not vendor, but it does so. It looks like this pattern is used beyond providers and vendors.

E

So if that's the case, we should advertise the status of the feature as accurately as we can and answer lewis question: can we promote this feature either by exceptions or by actually meeting the goal and meeting the you know progression steps. So are there ways to test it? Is there an owner for it and the person who started this? Are they willing to work.

A

Right I mean, what's the difference between this and multi-cluster, with a single masters duty.

F

I

You're not watching the re-issue resources right on that cluster. When you run external hdld, that's the key difference.

A

Right but if I have a an external cluster which has no istio resources in it at all right, but is that watching multiple clusters? How is it different.

F

It's the same thing, no external studies using exactly that same mechanism. The main difference is that it's using atmos certificates and you don't have the certificate distribution. So there are, some steps are simplified. Exactly.

A

Okay, so then it does sound like we should be able to progress it and it does sound like we like would have test environments that are capable of testing this, because we have test environments capable of testing multi-cluster.

F

If we treat them as the same size and it's already tested, because the body test.

A

Right? Which? Which.

L

Is fine with me that makes.

A

Progressing it easy right.

L

That's reasonable.

A

um All right, but still needs an owner.

A

Lynn, seeing as you helped create this monster, can you help jacob find an owner.

F

Portfolio cluster.

A

Right, like that, like there are lots of ways of solving the problem like we just got to figure out someone, and I do have one agenda item I want to get to, even though we're just out of time.

I

Yeah we'll have to follow up offline.

M

I

You know I think it would be. Whoever has most interest in this feature would be great because we're not using this feature at the moment, it's so low. uh It would be hard for us to step into help with this.

A

uh Okay, uh but you you can you and carsten can still figure this help jacob figure this out offline.

I

Yeah, we'll try to sort out offline see if we can get some help from other community.

A

Members I'll catch up on the notes on this in a second um so the lap and the item I have on the agenda. um Some people probably already know this, um but josh is actually leaving google and he's going to go and take up a fancy job at a startup um and that startup is not an istio company, so he will be stepping back from his istio responsibilities uh as part of this transition.

A

We at google are obviously very sad to see josh go, but you know we're also very happy for him and his his his shiny new vp of engineering title at a company in the networking space. So obviously you know josh has done a lot for is still in the community since he has been with us and so there's. Obviously you know some broad shoulders to fill um but and and the toc slot to fill. So that's you know something that we'll obviously need to take up um and josh is not in this meeting.

A

Unfortunately, otherwise we could all troll him terribly, but hopefully there will be opportunities to troll him and make him bemoan his life decisions. The next time you see him. um So I just wanted to make sure everybody knew this and obviously there's there's lots to follow up on after this.

E

And his contributions.

F

E

Immense and we'll definitely miss him in doc and the product security working group.

A

Yeah, so I believe josh mentioned this at the product security working group last week.

E

Yes, he did, and uh even though his new uh fancy startup is not using, his team definitely have a lot of people who worked on this too.

A

E

A

Yes, that is true um anyway, so.

I

That's big news, which company is he moving to? Is that allowed to alliance.

A

Well, it'll show up in his linkedin profile at some point, so.

A

The company is called aviatrix.

I

Wow he'll definitely be missed.

E

Yes, so louis in the interim uh for filling in the shoes uh are we like, especially around resource allocation and stuff, like that, josh was really helpful, since he was actually also managing the folks. Is it coming to you or sven, or somebody else,.

A

Right now, you could kind of assume a hybrid with me and schweda and maybe a side serving of mandar a little bit um okay. Well, we figure this out.

A

But yes, I I will be wrangling the machinery of assignments to some degree. Okay,.

A

So yes feel free to bring me back questions perspective.

A

Okay, I think we are out of time.

B

B

B