Istio Technical Oversight Committee, 27 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Technical Oversight Committee 2021/09/27

Description

Istio's Technical Oversight Committee for September 27th, 2021.
Topics:
- Adding additional company representatives to Product Security Working Group (PSWG)
- Requiring Istio early disclosure list to join Envoy early disclosure list
- Rewarding PSWG contributors
- Moving in-cluster operator to maintenance mode
- Open TOC seat nominations

A

Group over the past, you know six months uh and you know just active membership is, is kind of dwindling, so you know, is it fair for us to? um You know, solicit them, uh and then that leads into the second one, which is you know? Can we actually make it a requirement? You know that it that if you want to be on the early disclosure list, uh that you should also be a part of the envoy early disclosure list, um so yeah, that's just an open question uh to tlc.

B

Go ahead, justin.

C

For the second one, I just wanted to provide a little bit more context, so the the issue that came up is that um when we do our cve releases, a lot of them often contain envoy releases, and so, if we do an announcement for an istio release that includes the on or sorry for the early disclosure.

C

If we do an early disclosure release and they're, not members of the envoy early release, then we're violating the envoy embargo that we agreed to so the reason we wanted to make that a requirement is that otherwise it's going to complicate our announcements for uh early disclosure.

D

I think that sounds reasonable to me.

B

Yeah, I think we should probably double check if the three people that are already on our list that are not on voice for some reason can't get on the envoy list. um We should probably double check back in to see what we should do, but I'm guessing that they can just apply for it and they should be fine and we don't have to worry about the repercussions of removing anyone. Anyone and you know for future people that are being added. I think it's a very reasonable requirement.

E

Yeah seems reasonable.

A

Okay, um so yeah so I'll email them that uh as well as us solicit them. I'm not hearing any feedback uh saying that that it's um that we can't do that. um The last question- and this was from oliver um oliver so over the past. You know I guess, we've had reporters in the past.

A

Ask us you know, you know if there's any uh bounties or you know, swag or um kind of anything, for you know reporting some of these uh cves to us um just kind of uh bringing that you know forwarded toc, you know. Is there the possibility of uh of doing that in the future, or should we bring that up to tsc or um or you know, should we just say that you know just thanks and contribution on this deal website?

A

Is you know the only thing that we have you know at this time and uh we aren't evaluating other other things so.

B

I I'm actually a bit worried about providing rewards, because I think that may encourage low effort reports, um which we already get a decent amount of, and you know people reporting things like. Oh, your grafana dashboard is exposed and it's like well, we you know we did that on purpose and like your email is configured in a weird way and that's not really a priority for us.

B

um I don't know if it will encourage someone to go, find like some 10.0 cve, that's like forward to eco and then you know we'll give them some a reward. Unless, like we had, you know substantial rewards that actually uh like promoted someone that actually to invest substantial time but, like I don't think, a t-shirt or like kind of the lower effort rewards, is going to provide any benefit to us. That makes sense.

B

Like, for example, envoy, has a reward and it's a like a real high cash reward, but they have a very, very, very isolated setup that you need to exploit. It's not like anything is game, and so I feel like something like that would be useful, but I don't know how we could actually set that up for ustio.

B

So I would worry about what kind of reports that brings.

C

Yeah, I guess I could imagine some some things like uh you know, set up criteria like authorization bypass or something or sort of a similar gauntlet that they have to pass in order to claim it.

C

This may be a discussion more for google, but I mean I guess: if it's related is you know, do you know how are people handling like cloud services? You know like if someone found an exploit in google cloud, do we know if you know, since we have a version of it of istio, you know. Is there compensation available for that, because then it could also create sort of weird things.

C

If there can be an incentive to report them in one one place, but on another maybe we get the same end result, but still it might be strange.

B

I do think, and I'm not certain of the details, but that google cloud does have such a thing. I don't know if they have a scope that would include or disclose east geo.

B

I'm curious how other projects do it like you know? Obviously, google offers like some mysql database or something do we offer bounty on that and does the open source mysql offer a similar bounty or vice versa, or even on voice off really or kubernetes?

B

I don't know, I guess what is our goal of of offering this reward like? Are we trying to encourage more cve reports? Are we trying to thank the ones that we already got or.

A

Yeah, I guess the incentive would be that you know there are security researchers in the world that that um may be focusing their efforts on on uh like other projects.

A

um Just because that's how you know, perhaps they pay their bills uh and you know to get a little bit more eyeballs um into the seo framework um to see if there's any any other issues that are there now, that's speculative right, I don't know if people actually are looking at istio right now um they may be, but they have been fruitful, obviously for other projects right, but you know um other projects uh kind of may operate in a you know in a different way or have more visibility for people.

A

So um I'm not sure if the you know it's necessarily like, um we would even see the benefits of um actively doing this uh plus, I think you're right. The number of reports would go drastically up, or at least initially would go up um as kind of people see something you know of this caliber.

C

But I mean I think, that the amount you know if we're talking about giving a t-shirt or something that's not going to really help a researcher. I guess it would just be more sort of like a the swag that you get out of speaking at a conference. You know sort of a recognition or thank you, but but yeah I mean it's not gonna. I doubt that it's gonna drive real security researchers to spend significant time on on this deal.

A

Okay, all right! uh Well, uh thanks for the feedback, um I think we had talked about this a while back so yeah just confirmation that yeah this isn't the plan that we want to go to for now. So we can. We can move on.

B

All right, sam, are you here.

F

Yeah hey, uh so this is kind of a continuation of a discussion from the environment's working group. uh Basically we're calling for like a soft deprecation of the in cluster operator installation and upgrade methods, um we're not calling it a deprecation and we're planning on continuing to fix bugs. The main difference here would be. We are not going to recommend that new users use the in cluster operator, we're going to kind of shove, the docs somewhere more hidden and basically put a warning on that installation page.

F

In terms of the reasons, the maintenance burden for the working group has been pretty high and there's less and less expertise on the end cluster operator, um the fixed bugs and yeah. That's that's! Basically what we wanted to run through the tlc.

B

Yeah, I guess to me, there's really like two different aspects here: it's just the general concept of having a maintenance mode as kind of a state between an active feature and deprecated, where the difference being that deprecated implies that it's planned to be removed, whereas maintenance is just uh like we're not going to remove it, but we're also not actively supporting it as much as other features, uh which I think is going to be more and more important in istio in general, because at this point it's very very hard to remove features.

B

Now, as we've, you know gotten critical mass of users.

B

um I I can't even think of any features that we could really remove at this point, and so I think, like in the future distant future, perhaps like things like virtual service and gateway, may go into maintenance mode once the whole gateway api is fully stabilized and recommended, I'm sure other things will probably go down that route as well. So I think it's kind of a useful state to have, and in terms of the operator being the first one in that state. I totally agree.

G

Yeah, that's, it seems pretty reasonable. I don't think we've documented maintenance mode as kind of a feature state, so we probably want to go and expand on that a little bit. um Do we expect to put this notice into the release notes for the next release?

G

B

Okay, all right yeah, we'll have to be very careful with the wording, though, like we want to make it clear that you can use it if you are already using it and like not scare people off too much, but also not encourage new users. I think so very precise wording.

H

uh Speaking of precise wording, uh one thing I don't know if, uh if it's clear from the discussion uh documentation right now, we have you know three modes operator is the same parity with the other two. um The the goal would be to hide some of the documentation or put them in a separate site, or I don't know how we can market. But it's not only this operator mode they're.

H

All kind of we have command line parameters documented on the site, for uh you know, auto completion, pilot agent or all kind, a lot of features fit in the same category where we do not really want to maintain total to support them long term.

H

We don't want people to use them, but because we documented them and and we have them, we need to keep maintaining them, but hide them from documentation to prevent further use of them, uh especially if we want to make artificial changes and- and uh like we discussed in the previous meetings with microprocessor or other things, that, uh because the api surface is huge, I mean that's really. The problem.

G

We can have a very long deprecation right if we were clear about it in advance um right instead of our normal, like the fastest, you can deprecate something is you know, depending on the future status right up to one year right for something that is marked stable, but we could actually go longer than that, while still giving users the indication that eventually, we do intend to get rid of this right.

H

But uh I think the first problem is to stop the bleeding basically to stop. You know a lot of features, not only this is just an example, but a lot of other things. We kind of want people to stop using them and not be so prominently documented that my my concern.

G

H

We can do the other.

G

Things like moving the documentation around etc. Right. That's that's fine, but I am wondering about what we should actually communicate as long to the intent.

G

Is maintenance mode, a a just, a way of saying a very long deprecation.

H

The cost is not huge to maintain it, I mean it's just we have the same template, it's just uh it's relatively okay to keep it undeterministic like just like the virtual service. I mean there's no need to to scare people into deprecation.

B

I do think that it's probably largely the same like if we say we're going to deprecate in two years or if we say it's in maintenance mode and then a year and a half from now we say: okay, we're actually going to deprecate it in six months. It's the same end result, but the messaging to users is a bit different and I think that when people see the word deprecated, they get scared and I think that there's also confusion on what deprecated means and different people interpret it as this is removed.

B

This is not recommended and going to be removed, or just this is not recommended. um So in many ways I feel, like maintenance mode is just hacking around that confusion, rather than actually doing something completely new.

B

I see all the time people like it's deprecated, so I like it's. I can't use it and it's like well, you can you just shouldn't um like they think it's removed.

G

Yeah and some of the the concern around deprecation is because we have deprecated, for instance, apis that were in alpha or beta right in under six months right and then so that it would be reasonable for users to assume assume that I can't use that thing, because in reality you can't for very long.

G

If deprecations were you know one year or two years right, then users would probably be a little less concerned.

G

D

All right, lynn had her hand up first.

E

Okay, thank you um so yeah. I have two comments. So first comment is maintenance mode. I do think it could be pretty confusing. I was just looking at to see if kubernetes has a maintenance mode, um I haven't been able to find it. I I think it's another thing user would have to learn. Unfortunately, um so we we have to be very clear on the application maintenance there, I'm leaning towards not having a separate mode uh beyond deprecation at the moment.

E

um So that's point number one point number two is I saw mitch just joined. I I was just looking at one nine upgrade survey. I don't see we have like percentage of how many users are using in cluster operator. I don't know if mitch, you had any insights as far as you know how many users will be using and what are the percentage using in cluster operator.

I

No, we, we did not ask our users whether they were using the n cluster operator or the operator on the command line. um We might be able to get some of that data out of gke I'll need to take it to up my management chain for permission to share.

E

Do you guys think that would be any useful to pursue.

B

I think that it I mean I would be interested to know the data, but I'm not sure that it would actually change my opinion on the data, because we're not saying that we're going to remove it anytime soon. If ever and I think we do even if 50 of people are using it, we still want to discourage usage, because we don't feel that they should be using it right and we're not saying that those 50 that are currently using it should stop using it.

B

We just don't want any new users to use it unless they have a very, very, very good reason and understand the implications of that.

D

Do we do we have a page somewhere that, like at the top of the documentation for it that says what they should do instead, like do? We have the document.

B

uh No because right now we just say here's our three install options use any one you want. So this effort is basically adding that section that says no, you probably shouldn't use the operator. You should do this because of xyz and if you really want to you can do it, but here's the caveats.

D

Well, so I think we could separate that from the main input thing right we could we could discourage its use.

D

Using a new mode that we haven't done for.

B

uh Yeah, that's true. Yeah.

E

Yeah- and maybe we should have that first before, like the maintenance and the deprecation, because that documentation- that's missing is really you know user confused, so they wouldn't know what to pick so they could easily land. It was in cluster operator without knowing the caveats and limitations.

D

Getting that feedback from the current user finding out what's going on um and if they move and all that, I think we need to do some more outreach.

B

What is the goal of ours to find out if we should deprecate the operator or can keep it alive or.

D

To like, if we think it.

D

B

Yeah, I would be very concerned specifically about saying like in especially in 1.12 release notes that a operator is deprecated, even if it's a two-year deprecation or something and then at the same time we're going to have a section. That's like oh.

H

B

We got new home repos like helm, is great again like the tweets just write themselves about like ego flip-flopping between operator and helm and operator. I don't think it would be a very good look for us. um So I would I'm very against saying that the operator is deprecated 1.12. We can do almost anything else, maintenance mode or just not calling it a word, but deprecation is.

D

Yeah yeah, I think, pointing people towards alternatives and explaining why um and talking to users and see if they are planning on moving or willing and starting to get that feedback yeah. I think that'll all help you know feed if we think that that's the right approach.

D

No eric has a head eric.

I

All right, so my only other side comment to this actually had two, but the first one was. We only have it listed as beta currently. So is that enough of a distraction without adding you know the maintenance mode description it's in beta, we could deprecate it within three months. If we wanted to.

I

Obviously we don't you want to let it live longer, but then that sort of leaves this long-running beta thing, and I know we've had this discussion to try to you know promote features where you know this one we're just sort of letting sit in beta for a while. um So I don't know, if that's enough of a distraction or a deterrent, I guess, um but I was thinking between that and then the comment.

I

um You know that we might recommend something else um without mentioning deprecation might be sufficient, but I'll leave that up to the toc to figure out overall.

H

One more point is that east geodes turning itself into an operator I mean it's going to operate uh all the gateways. You know create gateway deployment and all the other stuff. So maybe deprecation of operator would be a bit uh premature because east ud will become the operator and uh the api is a problem because operator api is the one that is mostly causing the process. You know it consists of mesh config, proxy config and all the others, but uh I don't think duplicating the content of narrator. It's uh it's a good idea.

G

All right, it sounds like we mostly have consensus, clearly not to deprecate and to also not introduce a special new mode just because it would be confusing with respect to deprecation. But we are going to go and reorganize the docs to provide guidance.

E

Yeah and if there's any data from gke, I think it's a good reflection of the data you know beyond gke, so that would help us at least understand how much percentage of users are using. This thing.

F

Yeah, I think I think, whatever we call it if we have like a red warning on the top of the operator page, that it's not the recommended, install that'll go a long way in keeping new users from adopting it. So it'll kind of stop the bleeding there.

G

Okay, so the ask is, for the working group to go: take a look at all the the pages that guide users to install flows and try and make sure we steer them in the right direction.

B

That sounds good to me. Sam, are you taking that on yeah? I love this all right. Cool.

B

Does anyone have any other topics.

G

um Just a psa, the steering committee will be meeting to discuss the open toc seat on friday. So if you want to self-nominate, please contact a toc member or a steering committee. Member, probably the best way to do it is to go into slack and to just note, your interest in the toc steering questions chat.

B

um If no one has anything else, I did want to bring up something that cost didn't mention. Let's see, if I can reassure this.

B

uh About east god becoming an operator and what that was about- um since I don't think we've mentioned this terribly broadly. um This isn't necessarily something that we have. I get conflict we need to resolve with the tfc. I just want to bring it up to the tlc, since it has pretty large implications.

B

um So, historically we have had you know you install the gateway um through helm or operate or whatever, and then you configure that gateway with a gateway resource and those two are kind of decoupled, but they are actually quite coupled logically, like you have to keep them in sync, with the ports, um et cetera.

B

So, with the new gateway api, there's been a lot of movement towards having the actual gateway resource represent the underlying infrastructure, both in obviously out of cluster implementations like the gke one, for example, you create a gateway and it provisions a load balancer for you and you just get an ip address and the status, and you call that and everything kind of works with one resource, but also in cluster implementations, like eco, have also been moving to this model, where creating a gateway resource also provisions some proxy running inside the cluster to handle the traffic, and so the idea would be that east geo would do the same and that if you created uh some resource like this, for example, you would automatically under the hood, get some service.

B

uh That's exposing these ports and you get an envoy deployment with whatever you think we need to run the deployment and that way we're kind of abstracting. The fact that we have this deployment and service in the cluster and the users can more think about the features that they need in the gateway api. You don't have to worry about keeping things in sync et cetera, so this is not terribly complicated. Obviously we're just turning one resource into two effectively, but it does represent a kind of large shift in kind of the api model.

B

So I want to bring that up if there's any concerns or questions or anything.

E

So john, is this mostly for app uh dedicated gateway?

E

H

E

Shared gateway across different name space for different apps.

B

I would say: no it's not mostly for that. I'd say that it is even better for those users, because if you have more of them than making something like, if you do something 100 times and it's easier, then it's a lot better than something you do once it's easier, but that doesn't mean it's not still useful for uh if you have a shared one like, for example, in our east west gateway, install guide, it's a lot of steps right like that. That whole guide would be replaced with like a 10 line.

B

The ammo file, like this essentially um same with like on the ingress and egress docks. We can just say, deploy a gateway and we just give them this yam ball. We don't have to go, say, go to the install docs and then pick your install method and then configure these parameters and then configure a gateway and then make sure the ports open it's. You know it's quite a quite a bit more work, so I don't know. Does that answer your question.

E

Yes, uh it does, I guess one related question I have is uh so let's say if I have this gateway input on your left side. What, if later on you know, I decided I'm going to poke up another port, maybe with a different protocol on my gateway, and it could be. Maybe I support an additional hostname uh now. Would you perform uh upgrade to the service and deployment.

B

Yep, so if I change this to say like this is now one two three five, then the service would also update or same if I added a port, um so they're kept in sync, and so, if you wanna add a new port, it's like it's very simple, whereas today you have to update two things, um which I mean granted updating two things is not that hard, but it still actually causes a lot of pain for users um and developers.

B

I know that you and I have both struggled with that in the past, when it's not very obvious. Why something's not working you realize, like the ports are out of sync or something annoying like that.

B

It especially makes the docs really hard too, because there's despite the fact that it's just updating one port in a service, it's actually sometimes quite hard, depending on how you installed to change that port easily.

E

E

If no one else have a comment, another question I have is, I guess this means your gateway resource must reside in the same name space as your gateway deployments, uh which today we recommend people to do, especially when you have tls.

B

uh Yeah, so that requirement is already with a new api. That's our requirement, regardless of if we do this. So one thing I should clarify too. This is uh this is one option and it's the default, but there's also an option that a gateway can point to an existing service and deployment um if they want to manage it manually.

B

This kind of mirrors like something that a out of cluster implementation like gk, would do where you can point to an existing provision, load balancer ip, so very similar concept there.

B

So while we expect most people to do this, if they don't like it or they need to migrate from an old install or whatever, then they can, they can opt out.

E

That makes sense.

B

All right, thanks for listening. uh If there's no other topics, we can probably end a bit early.

G

Yeah sounds good to me.

B

All right thanks, google.

G

I