Istio Environments Working Group, 22 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio Environments Meeting 2021-09-22

Description

Istio Environments Meeting 2021-09-22

A

All right, yeah uh before we get to the agenda, just one uh quick process thing: uh the environments working group owns about five docs tests that haven't been automated. I think we went through and manually verified that these docs are working for the last release, but um yeah. If we can get owners for some of these that'd be great. We don't have to assign owners now, but uh just to make you all wear well in advance of the release.

A

um And then yeah looks like john. Has a few topics.

B

Oh wow, we got a lot um yeah yeah. I wanted to discuss. I forgot, which one this one is um oh yeah, so there's been a proposal to, uh as the description says, terminate envoy when the connections become zero, um so the idea is kind of a more graceful shutdown right now we just always wait five seconds, which is either too long or too short.

B

So the idea would be that we instead keep waiting while connections are open and then we will exit only when they're closed. So it gives us an approximation of exiting envoy when the application does.

B

The kind of open question is: how do we like what knobs do we expose to configure it, because we we kind of need, like a minimum amount of time, to wait otherwise, there's a race condition where, like, if the very second, that cubelet kills a pod, they send a request or something, um and we already have an existing configuration flag. So what do we do with that? One um there's! If you scroll down a bit there's a long comment that I had about various different options.

C

How do we detect if the application died.

B

Well, that's the thing is we don't detect the application being alive, but we detect whether there's open connections, so it may not even be a good idea. I think it's certainly not 100 reliable. If you have an application that, for example, it shuts down it, does a bunch of work and then tries to like write into some database at the last second, because there wouldn't be an open connection.

B

While it's doing a bunch of work, I think would help the common case, though so it may still be worthwhile to do it, and then, in that case they could configure it to always wait.

B

However long they they're setting it today like they would already have to set that today um to wait a long time to stay alive.

C

So they are going to application is going to close all sockets when it dies if they're going open socket. So if we are not.

D

Are we lame duck, uh I think uh yeah so immediately we go.

B

Into draining mode which will send like the connection close headers, uh okay,.

C

So no new connections will be accepted, at least.

B

Well, they'll be accepted, but then they'll they'll be distributed. If the client behaves well they'll be closed immediately after.

C

So it sounds reasonable to detect, at least if there is no connection. It's clear. There is no point in waiting. I mean there is no, if we, if, if it's possible in a way to get this signal,.

B

Yeah, I mean the way we get it in the pr is somewhat hacky like it's using the stats, um which is a bit suspicious, but.

C

With uh with the injection, unless I mean having the starter in the application, it's very easy because application, we start the application. I don't know if we can inject some.

C

I think we discussed in the past by adding some some binary to the container, either through to unique container to mount volume or something yeah, and that will be something that will solve all our problems. Basically, because it will launch the application it will detect when it dies it, it can yeah.

B

That is another option.

C

And it's consistent with what we are doing with with uh docker vms and other places where we have full control and where we don't have this problem in the first place, we.

B

Do the starting to like waiting to start waiting.

C

To start is another thing that is perfect, because then you can you can. uh That would be a clean implementation of way to start actually because yeah.

B

C

Would be transparent.

B

C

Be transparent.

B

C

I mean if it's a docker that doesn't have any dependency.

B

No, I mean we don't even need to do like docker image. Building like we could make it yeah. You need volume, yeah volume, that's shared between the two containers, exactly.

C

B

C

And it will perfectly consistent with with uh the injection less thingy, but I mean if you build a docker image, you probably have the binary already you don't have any problem. If not it's. uh I love this idea because it's it's.

E

C

And the startup is actually more important than shutdown in in many ways, and I think we already can play always complain about uh well.

B

We do have the hack too, a.

C

Hack, that is very similar is doing a shell. No, no.

B

Well, it's kind of relying on cuba pre-check to like then move in, which is a shell.

C

Yeah, which this trolls probably doesn't have no.

B

Oh, it's it's a command and pilot agent, so it does work for just release.

C

B

It's not modifying the app container, though, but.

C

It's kind of just oh, it's not modifying you know.

B

No, it's a pre-start or pre-stop hook, or something like that.

C

Other opinions, sorry, we are in a room so.

A

uh Sorry can we go back uh for some context on the existing options and the option of this pr ads? Yeah. Sorry, I just uh don't have all the context here.

B

uh Yeah, the existing option is basically there's. This environment variable called termination drain duration and defaults to five seconds, and basically we just wait that amount of time and then we exit there's also a kubernetes setting which is somewhere named termination grace period seconds, which is basically when the pod exits will send a sig term to the pod and then, after the grace period, it will send a sig kill. So if you don't gracefully shut down within 30 seconds as a default, then it will kind of forcefully kill. You.

C

Because that means that every time you stop a board, you need to wait. Five seconds I mean, which is a problem for rolling upgrade is a problem, for I mean it's not a major problem five seconds, but.

B

Yeah, but it's also like it's, neither it's always either too too slow or too fast right. It's never just right so, but in this I mean the way this is implemented, it's actually at minimum five seconds and could be longer, so it doesn't actually help the shutdown quickly case, otherwise we're at risk of like a last second request, and then we see, there's no connections and we close too early yeah.

F

B

The kind of wrapper may attribute would be more effective there, since it actually looks at the process.

G

If the application doesn't doesn't close all of the connections, uh does it hang or is there like a like a force like a force, kill well.

B

Eventually, that termination like kubernetes, will force, kill us. Okay,.

C

But after 30 seconds.

B

30 seconds yeah right.

C

B

C

Application doesn't handle seed, kill, sig, stop six sticks up is the first time uh I thought.

B

It was a stick turn then six.

C

So if it doesn't exit on sick terms, then it's it's going to hang for 30 seconds, but uh and we'll die anyway after five seconds. Yeah.

B

It doesn't matter if we like, we should live just as long as the app there's no benefit of exiting before the app right, because the pod won't go away until the app exits. So it's just like it's not like we're saving time to exiting earlier and.

C

The application is expectation of the developer is that they will get 30 seconds to clean up and do whatever. So if they expect that their connection, without these years, they'll have 30 seconds where they can still keep the connections alive and whatever and for long-lived connection. Sometimes it's useful to have this.

C

Should we bite the bullet and do the hack, I mean not the hacks.

B

I'm a bit concerned about the like making the hack a default, but I think it's a good option. It's certainly.

C

For sure I agree with you for startup, I think it's a much better solution than what we have.

B

Well, I'm just concerned about modifying their container, um we're not we're not like modifying the docker image, but we're modifying like the container.

B

I think it will work in a lot of cases, but I could imagine at least it would cause fear in some very security paranoid people for some reason um or some obscure edge cases. I don't know it's definitely worth experimenting with. It's like.

C

Pilot agent, it's it's the binaries that we run. You know.

B

Oh, I think it's definitely worth worth checking out and maybe we find out it's it's great and we should just turn it on for everyone um or at the very least, an option like right now. The waiting till we start is an option and there's downsides, but there's also plenty of upsides, and I think it's so.

C

So effectively we are talking about pilot agent. Here I mean we are going to pop a pilot agent to do some uh volume and then mountains, application and run pilot h and ram, which I think someone already implemented the next option pilot agent. So it's kind of like we'd signal.

B

Somehow from well, we could either run pilot agent fully what a substitute yeah I don't know if we'd want to run a fully. Actually, then we'd have no cypher or I think.

C

B

Signal uh to the container like we mount some shared bds socket or something, and we we send some signal like I, I've started up and then I'm shutting down.

C

Okay- and that happens to be the same thing we are doing for injectionless for docker, meaning that we uh probably this will also binary the process. Grpc bootstrap and- and so we don't have to run a proxy container.

B

Well, it's it's amar right. One of them is signaling.

C

No no, but this binary can do few things. Okay, I'm sold. Everyone has opinions on.

G

um Yeah I mean this sounds great yeah. The only thing I guess I would say is like like: where should the user be able to configure this right? Is this? Would this be a candidate for like a proxy config? uh uh You know part of the crd. You know that you guys are implementing or like, or is this going to be like a global thing? um You know when the user does configure this. I think this is a good step forward.

G

Honestly, um the only downside I guess I would see is that I, uh after working with a lot of people, uh people don't do graceful shutdowns, very well in their kind of environment like they're, not listening for sick terms, uh and you know a lot of users have poorly app poorly written applications that basically have to be forced killed. um You know, I I don't know. We can't worry about those kinds of things like if people are doing bad practices and and things like that and they're just going to get of course killed.

G

Then it's like yeah, that's fine, but.

B

It gives them the same behavior they have today, though, like if there.

G

C

B

Seconds then, we'll wait: 30 seconds get force, killed and no problem. Well,.

C

It is better than we have today, because it's application is hanging for 30 seconds. We also have for 30 seconds and- and so it's better than oh.

B

Yeah it's better than today I mean it's the same as today, without east geo, like they won't adopting, east geo won't slow down their pod shut down. The bridging shuttle, okay.

H

Okay, so doesn't mean I no longer needs to configure this at stop time, because now we have this intelligence to terminate envoy when there's no active connections, especially for digitalists, because they, our users, are struggling with how to config this for this service right, so we do 45 seconds, but so, if user doesn't do anything, this is pretty transparent to them. Right, if I understand correctly.

C

Absolutely- and I think on the previous question- if it should be an option in proxy config, I don't think it's something that we want long term as an api, because if ideally, if this works, that would be the default user doesn't have to worry about anything either startup or shutdown.

C

So we may remove the options that we have for a startup, because that will take care of it. Naturally,.

C

Is there any reason not to have it by default in uh at some point, I mean after we test it and including startup.

B

I think we'll have to see like I think, if short term, it would be some environment variable or, of course, option whatever and then global yeah long term, but we'll see if it's something that we can just turn on or if it belongs in proxy config.

H

Yeah, it would be really nice if this is the stable and we enabled by default. So this triggers me a related question, so you know how we had like hold the application until your proxy is ready right. So we talked about like the need to hold the product. I guess it's a whole day application and it's, I think it's hold the proxies from stop until the application stuff.

H

Would this actually eliminate the need to for the stop sequence between the envoy cycle and your application container, because when your application containers are running you're, not shutting down your envoy.

B

Yeah it would handle both the startup and the shutdown.

H

Oh, it wouldn't handle startup too. I missed that yeah, so it doesn't even need that code application into your proxies ready. Then.

B

Nope no, I could.

C

B

C

It will replace it and it's much better because uh the starter can know uh you know it has far more control. I mean you can communicate with the proxy to some socket or some other mechanism, and- and uh you can you know you don't have to have the weight slips. 100 milliseconds try something.

H

Okay, your pr says terminator, so I was confused. I thought it was. We just.

C

H

B

Oh yeah, so lynn, I don't know if you you join me if you missed so this pr is, would only solve shutdown, but we talked about an alternative that would solve both where we kind of insert our binary. That kind of does the weighting on both ends.

H

Okay, cool wow, that's that's very powerful!.

G

Oh yeah, so my question was: um what are we going to use to say you know to turn this from? You know false to true right. uh So if it's an environment variable or we're just going to like give it some time to kind of incubate or uh you know, get some feedback from you know some users or or like how? How do we intend to measure that so that we know um I don't know, because if it was up to me in my world, I would just turn this on right.

C

Don't eternal master, let it trust well, you know nobody complains and we leave it on. Otherwise we turn it off before release. So.

B

Yeah, I think, like this is the most requested feature in eastgas, so we'll get plenty of feedback. I suspect, uh okay.

G

Okay, then that's great, then.

B

Yeah, I will does someone want to experiment with this.

C

So I'm experimenting with this one on the um for the docker yeah, so I can. I can do a small experiment as well, so it's basically a way to start the injection less because it's exactly the same thing yeah and eventually this can grow to also fetch the content. Okay,.

B

All right sounds like we should move on to the next one.

C

So john I'll I'll try with my my uh starter: that's the one I'm experimenting with and I'll. Let you know if it works, it doesn't work and then we can sync up and figure out what uh how to move forward. Okay,.

B

A

Let's just skip.

B

The next one, that's mine as.

A

Well, because we have a lot of stuff I'll move it to the bottom, okay yeah. What is uh what is the case today, where the proxy exits before the application uh with so that this would fix that case as well.

E

A

Did you want to skip this one through or.

B

uh No, I mean this is just real quick. Just we publish the helm charts now um it would be there's a lot of easter charts that people were unofficially publishing, so it'd be nice if people went and like started or something so, it will show up first so and use them. Of course,.

A

That's it cool. Do we want to do an announcement or um put it somewhere, where I think a lot of community people would probably start anyway.

B

Yeah, I put it on the there's, a big issue with like a bunch of stars for uh publishing the home repo. I posted it there once we actually published 1.12. I think in the like release announcement. We should highlight it as one of the big things so, but right now, it's only for 1.12 and.

C

So, which one is it the one from howard, johnson, john? No.

B

There's an official.

C

B

There's official easter org, you should say official, verified publisher.

C

G

So kind of along similar lines. um Lately you know: we've had you know people you know, request annotations and labels and other things um that aren't visible in helm. uh My my feeling is that once we actually go live with this, uh there's going to be kind of more requests, for um you know other things that istio does support within our helm charts um should we come up with with some kind of a kind of a policy or a mechanism, or are we just going to?

G

You know just approve a lot of these like configurable items that people are going to be writing issues come on because, right now, what I'm thinking happens is people put the charts into their own cacd and then from there they're able to um kind of modify them. You know as needed versus if they're, using an official home repo, I think there's going to be a desire for them to kind of not kind of work in that old workflow.

G

Should we take a stance on that before we go live and if so, we can defer this.

C

You know, I think I think we discussed, I don't know if you missed, I think few meetings back. We discussed this same subject and someone was sending a pr or something my opinion at that time, and I don't think we anyone disagreed strongly was that we should support everything that the normal health charge. I mean if it's a standard pattern like annotations labels that is commonly used in the helm world. We definitely should support it and annotation labels are fit in this category.

C

So if it's part of I think john did the experiment with uh creating a helm chart from helm, create init or whatever yeah, and there are a bunch of things there that are presumably common.

C

uh If it is not in that list, then we need to be very very careful, because we need to support it long term and you need to make sure that we um we really want it.

C

Knowing that people have the alternative to use the old helm, charts which are not published, but they can download from the repo that has 100 plus options.

B

You can also point them to the customized plugins.

C

Customize plugins that we also discuss.

G

Exactly so yeah so yeah, so we should probably just like put some yeah going to customize yeah, so we should uh probably include that as part of the the home documentation I uh I can take that. I can take that action item, but.

C

Not as required thing, I mean it's just that if you really want to do things that are not commonly done, sure you can use customize or you can use old charts. Both of them are fine and and good solutions.

G

Okay, yeah, no, that that actually, that actually sounds appropriate yeah, so we'll just make sure that we document that. So so we don't get here's every single option. I want added to the you know: values or whatever. Okay sounds good.

C

Let's write it down: if there is agreement, let's write it down, so we don't rehash it next time.

C

C

I was not seeing it.

C

C

Okay, next topic.

A

uh Yeah, so I just wanted to kind of bring to the working group attention. uh It seems like there's a few pretty severe issues with the revision-based upgrade using the in cluster operator, um and I don't necessarily know uh who should own these issues right now or who has expertise in this area. So I just kind of wanted to get some visibility on it. um I looked into those these issues when they were reported and it seems like the wrong revision of enclosure operator is reconciling um revisioned istio operator resources.

A

So it's it's pretty broken and I guess overall um maybe to raise some discussion on whether we want to continue to support the enclosure operator. um Yeah. That's that's! Basically, that's basically it because I I don't know who uh owns this feature right now,.

C

In cluster operator upgrade, do we still want to support it.

B

I'm a I'm a bit skeptical of saying it's deprecated, just because of the churn, but I I could do kind of think that we should go and on the dock say like this is not the recommended approach. We strongly recommend use other ones because right now, if you go, it's the second one on the list, it doesn't really have anything telling you not to use it, so people will just use it. I think we should uh disincentivize people from using it because there's a ton of bugs it's not just this one.

A

Yeah, I think the community has kind of come to understand that it's not um as supported or at least from the issues and slack threads, I'm reading uh it's kind of people warning each other, but not not using the end cluster operator, but yeah.

B

But it's a bad experience to find that out by wasting eight hours, trying to get it to work and I think it'll be way less useful. Once we have the gateway stuff, because, although we'll install is easter, od and if you've already installed used to od or you've already installed the operator, you could have just installed eastwood directly, so there's kind of minimal benefit to use it. Now that we'll have one deployment to install, whereas when we first made it, we had like 10.

C

Especially if the git is automatically created, that's.

B

C

B

Like if you're just creating you're, creating a deployment to create one other deployment, so what's the benefit really.

C

Which creates another deployment for gateway, so it's kind of uh yeah.

B

You might as well, if you can create one deployment, then you can create a different one. That's easier to you know.

E

uh So so, regardless, uh regardless of um the future of in-cluster operator, I think I think these issues you should take and you know either I can take these or actually tongue. uh I should be able to uh uh take care of any any known issues.

E

um Yeah, and I mean this is notwithstanding what what we designed with with operator. I don't think if, if there's something, that's obviously broken, we should fix it.

C

So I think uh that will also reduce the number of bugs if we kind of de-emphasize it in the documentation.

C

I think the important thing will be to help to see if we have a migration path from in cluster operator to either helm or histogram. I think that that is possible today, no.

E

uh Yes, so so helm, I I guess it's not really straightforward. uh There's there's not uh uh you know, there's not really an explicit path. I I think we could work on finding something uh for operated. Istio, cattle. That's that's! A very straightforward move.

C

The thing for him is also because install doesn't have doesn't interfere with operators, so it's just a different revision. Only base would be effect, so you can just do a helm, install histod operator and everything will be fine.

E

I think that's that's largely true. The the only possible uh wrinkle could be if somebody started using the non-held fields, um in which case there's some manual migration involved, but it's it's not anything terribly difficult. Okay,.

B

Yeah, I think you can definitely do it. We just don't have a docker test or anything. So is there any concern with putting some warnings and not recommending in the dock on easter, io.

E

uh I I don't have any concerns, I I think it sounds like that's the consensus, so um you know I I think it. It makes sense to uh have as few installation paths as possible and if, if really this, there's no longer uh anybody that absolutely needs operator, I you know it used to be that that some folks, we were using that as as a uh as a kind of a basis for uh for a service offering.

E

But if, if that's no longer the case, I I don't think this any compelling reason to keep it going and and the fewer installation paths the better. I think.

C

Okay, so we're left with two installations: there is your cattle method and uh yeah, that's good enough.

A

Okay: let's go ahead.

H

So should we allow the application, maybe first next release after.

B

I don't think we should deprecate. I think we should just tell people not to use it, which sounds like the same thing, but.

C

So we continue to support them. I mean dedication, it means scares. People means that we are, you know, removing support. There is not a huge cost to continue to support it at the level it is today. So if it works today, it can work five releases from now and it's less scary for users and minutes yeah.

B

We just want any new user going to the dock should not install the operator basically, but it we don't need to tell existing users that they need to get off it or say that we're going to remove it anytime soon.

H

Okay, that's cool.

C

And I don't think we should plan to remove it. Even I mean it's, it's fine, it's as long as it's working and we can support it and it's not a huge cost to fix the bugs, because it's very close to what this geocache is doing. So it's not yeah yeah.

A

Did you have a hand up.

I

Yes, yeah, and I just wanted to- I call what martin was saying and the fact that we have this issues opened by a customer. That means the people uh still using it, so um we should fix them, and I have no issue to take on those uh issues and signed to me and uh trying to fix up.

H

Oh, that's so much appreciated. I I do know there are users using them. It's just, as you know, who's willing to maintain them.

I

Yeah yeah absolutely and I've been trying to fix the operator related issues in the past few months. So.

F

Thank you tom. Do you mind if I tentatively assign this to you? uh No, no.

I

Problem at all.

C

Speaking of deprecation, uh one thing that we do need to deprecate- probably I don't know if it's official or it's uh part of the istio cartel operator api. We have the gateways, we have all kinds of options that are probably not doing anything anymore yeah and I don't think we have any doc or any any indication that those are just uh empty strings. Basically, I mean do nothing.

C

um We probably need to go over the operator and mark whatever is still doing anything at all and what is deprecated and what we want to keep long term.

C

A

uh I so I do have a quick question on how operator in cluster upgrade is supposed to work. um If you have revisions, because I noticed when you, when you're doing an upgrade um it'll just reconcile older and newer crds over top of each other kind of it'll just kind of flip-flop, um I don't know how that's supposed to necessarily work.

E

uh Well, it's the best yeah connect, so canary was was never uh I mean it's it's supported in theory, but but in practice it's it's very tricky to set it up so that it works correctly, um which is, which is why you know it's a area. It's. It was never really advertised as uh as something that uh we we recommend operator for. um But but the short answer is that you can install operator sorted, so it only watches its own revision um and you need multiple copies of operator to have canary work correctly.

E

uh So it's you know unless it's broken. uh I know that it used to work in the past, um but it's it's a challenging setup and you know not something that we would recommend as a good ux.

A

Okay, yeah right yeah. What I'm saying is, even if you have the um multiple revisions of the enclosure operator when they reconcile my understanding, is they'll overwrite, the base of the other.

A

um What the other operator had just written that makes sense, uh but I don't I don't know if we need to do some leader election, maybe between uh in cluster operators or.

C

Let's not get too far, I mean we want just to fix, bugs and try to reduce. So if it flips flip flops, we can just not touch them. You know not update it or uh so idea is to minimize the amount of changes and support we need to do for the operator if it's no longer recommended.

E

Yeah, so just just on that, specifically, if I remember it's been a while since since I actually set this up, but uh I I think the way we dealt with that is is just not to install base in one of the operators.

E

Okay, that makes sense.

H

Yeah yeah, I think what sam is on to something, though, because if the in-cluster operator has a big limitation, you know we should highlight, uh and it would help people make decisions too like if the same you know we don't have enough people to support and we're not recommended- um and this could be one of the critical reason why.

B

Should we maybe be it somewhere between not recommending and deprecating and call it like maintenance mode or something that we'll continue to fix some bugs, but we're not actively developing it? And you know you probably shouldn't adopt it.

E

Yeah, I I think that that makes sense, given that even that canary is is, uh is in white use now um and- and I think we could- we should probably explicitly say that it's not recommended for canary upgrade yeah makes sense.

B

C

I, like the term maintenance mode, because probably for some of our other apis will also go to maintenance mode.

B

Yeah, like gateway virtual services, long term, but I think that is a good thing like we like we're at this point in the project, it's very very, very hard to deprecate something so that I think maintenance mode will be kind of the future of the future way. We deprecate things and.

C

And also we should advertise and make clear in the website. I don't know how how we can explain to the users that we are committed to support all the features that we launched, but some of them will be in maintenance mode meanings that will work for a very very long time, but we still do not recommend people to use it, but we recommend the new apis kind of.

H

Yeah we need to craft their messages pretty clearly and also making sure user knows you know there are alternatives, hopefully, are way better out there, which, in the case of the networking api, I'm not sure you know, you know the kubernetes gateway api. You know user would be willing to switch at the moment.

B

Well not yet I'm talking like in a year from now or okay.

H

Yeah a year for now for sure yeah, but now it's not, but this operator thing especially like canary update the revision based if there's a known issue, you know we can communicate that in the maintenance, yeah.

C

So people who already use it they can keep using it as long as they want we, they don't need to be worried that it will stop working but we'll not add new features. If we are not fixing major things, we just make sure that it we test that it still works with the features that it has today.

B

It's basically like the oldest release kind of, but for a specific feature like we'll fix bugs security issues, but we're not pushing features.

B

And we can fine tune that more, as we start putting more things in maintenance mode, but I think operators are good start.

B

You can probably bring this up at c2 if we want for the higher level.

H

Yeah we should.

C

We may make some changes, for example, but the base we can. We can say that operator is only operating on history d itself, but you will need to do helmet installed or stock for the base, so we have full control over the base yeah and that will solve some of the current form because the canary will install it with uh helm or these two catalan. Okay, let's fine tune it. I think we spend enough time on.

H

Yeah I like that that thoughts right yeah I, but I do think it works well too, if we could support just for ecod. I think that would be very valuable.

A

I can um I can.

H

Own documenting the maintenance.

A

Mode uh change.

H

Yes, sam, I can help you review too.

A

Awesome yeah I'll, probably I'll, probably leave a bunch of you all in.

C

I did other to sneak it in uh for gateways. We will recommend the new new way. I mean standalone user, operated independently or automated when it's available and we are going to dip. Well, I don't know deprecate or maintain I mean I don't know if we would still want to support the operator managing gateways. That's my my concern. When we have both helm, clean charts and uh automatic mode and.

B

Yeah I mean it seems like it's more work to remove it from the operator. If we're going to keep it easy, we definitely you shouldn't, do it, but.

H

Yeah I mean we don't need to recommend people to do it.

A

C

A

Yeah, let me uh represent this next one um yeah, so hopefully this is uh pretty quick. I just wanted to get some opinions from environments. uh So basically there was a problem in our web hook, patching where we removed leader election from the validating web book, and we we never had it. I don't think for the mutating. I could be wrong.

A

We might have removed it like a year ago, but uh basically this is a problem, because when you have multiple replicas of sdod they can they can fight each other when they're updating their respective web hook. So, basically the thought was. We don't need leader election, because um only one revision should be patching and then the problem there is okay. Well, you can have multiple replicas in a revision that you do need to leader elect among, so uh this pr basically adds a new leader election type that says: okay.

A

This is a type of leader election, where only one replica in every revision can win it. So if you have revision red revision, green both of them have two replicas one replica from red and one replica from green will win the leader election.

A

um So that's the general idea, and um so I got a comment that perhaps we want to do this for every leader election uh in istio, okay, I guess john actually made a separate comment since I look.

B

Only for things that are patching revision objects otherwise like for it doesn't make sense to have multiple leaders of a global object right. I.

C

Mean it's, for example, it is for for objects that are shared. It will be catastrophic because we already had the bug where yeah two revisions were fighting with each other, because they are not in the important yeah. uh I go ahead. Go ahead, you're also working on having the default uh revision, uh so so the default to win and and to become the validator and to shouldn't the default uh version, also be the one that is patching and doing all the other stuff. Instead of having a random red green, whatever.

A

Not exactly so for web hook, patching um only the given revision knows the cert to patch in and each revision handles its own web hook.

A

C

Don't think that's necessarily true or or because we still share all the vision share, the same c root, ca and all revisions are using the same citadel certificate. Wow.

F

C

F

Not true, no, no, absolutely.

C

Necessary because otherwise they cannot look at each other. So if you have work, I mean the whole. The premise of of uh revision is that you at the end, all workers can communicate with each other. So the.

B

The mtls certificate- yes, but the control plan certificate could be different. It's typically not, but it's easy.

C

Typically not, but then I don't know because the downside of this is that uh it will be very desirable to have a default or or some dedicated hype, because that that's a dangerous part, that's the most dangerous part in istio.

C

If the revisions wouldn't participate in election- and we had- I don't know- we don't have another controller, but but at least we can say that the default is your or a dedicated issue takes care of the okay.

C

Let's uh table this, I have another topic which is very closely related with this. So I can. I can discuss the parameters so we'll go ahead and do the other stuff and.

A

Okay sure yeah yeah, so um basically this is fixing a bug. That's present in 111, so I I just uh wanted to see. If you, you all had any thoughts that this is unsafe. Somehow, because changing leader election is actually hard to make changes to in later releases.

A

um So yeah, are we all right with this as a 111 fix that gets back ported.

B

How does it work with upgrades since the other? The older versions are using the older right.

A

It does not so.

F

A

I mean it's fine in that case, because the old revision is broken among, like the older vision is going to be uh fighting each other for the old web. Hooks like it was, but at least the new revision is fixed, so it doesn't, it doesn't make things any worse. I think it makes things strictly better for the new revisions.

C

But uh any reason not to have the default revision, I mean. uh The only problem is that uh the certificate may be different. Do it we.

D

Yeah, I think that's issue, especially maybe how can the certificate be different? This geode doesn't like.

B

For example, if one of them is using the kubernetes signer and one's using the self sound.

B

Yeah, I think the origin of this was that someone had like a bug that somehow caused them to be out of sync. I don't know why, and it was like just retrying infinitely.

C

This is a feature that I mean this whole signer is.

C

There is another alternative really uh which is to I mean: are we going to install the east-west gateway or some similar gateway by default? We discussed in the past. I know you didn't necessarily agree that we should have a gateway and have the web hooks and everything behind the gateways that is actually dispatching. So, instead of having.

C

I had a pr at some point to have the default version. Also, you know the same product studio deploy a gateway. So basically there will not be. You know one different web per uh revision. There will be only one web hook that is a default and default can dispatch through to a gateway route to different revisions.

C

um That has other benefits. I mean you know you. Can you can put a real certificate, so you don't have to patch anything. You can support better multi like externalize your dna and because external city is really a gateway. I mean these dude is behind the gateway, and should we explore this again? Should we discuss.

C

It is not hard, I mean we had it before in in one zero. I don't know three releases back before releases back uh yesterday had the side car, which you know, acted as a gate or can act as a gateway.

A

uh So is the idea there that, because we know the default revision, we can just have validation, requests, get dispatched like proxy.

E

To the default.

A

J

A

An interesting idea again, we could only have we that would let us have just a single validator and single mutating web hook. I guess.

C

Yep, the default will be it'll, be a fixed, fixed, url, same certificate always, and then we use mtls and the regular issue between gateway and http.

A

Yeah, that's that's, probably a whole separate proposal, but that's a really interesting idea.

C

It also means that we can implement uh percent switch between canadians, because right now.

G

C

uh We have a very strict routing between uh revisions and histories, but with a gateway I mean having a default gateway in the terminate, xds and all requests. Basically in the default we can have, you know, 20 percent go to a canary and then it kind of is down, go back to the principle, so pretty much what we are doing for for revision for regular applications when we do traffic shifting and subsets.

C

Let's think about it, I I don't know it's it's probably more work, but it's it's it's going to solve many problems and create more uh opportunities in especially with multi-cluster. uh You know, fallbacks and a lot of other things will be possible with that.

A

Right: okay, okay,.

C

This this can also become the east-west gate. I mean if it's vandal, if it's another container in istio d, it could be the default out of boxes twist uh for vms and other things. We don't have to do another, install and and because right now, it's a bit painful too yeah.

C

Let's think about it,.

A

Okay, cool yeah. um That makes sense to me if, if we could review this pr, though, and either um say like it's, it's you know maybe too dangerous or uh move it forward, just because it's a it's a 111 bug so yeah. Thank you. That's that's everything on that topic.

H

What's the chance to hate this they're just curious.

A

Oh sorry, what's that.

H

Is it is this the intermediate issue that people would hate? Oh.

A

um I think it results in a lot of error logs in sdod, um because it'll, try and like update the web hook and then get a conflict. I don't think it's like a release, blocker um type thing, or it's not like a huge issue, but it's a it.

C

Is a huge issue: if you have uh twist yods that use different certificates, because then then one of them will be broken.

H

Right and that normally happens when you are in the middle, for I guess rotating yours, no.

C

No, no, if you are trying to be too smart and try to do things like uh use, one is to a deal with uh adele. One is to deal with something else. So different I mean if you mess with undocumented apis, you may get into a state that.

B

Yeah, to be honest, the only way I know this would happen is if you might go to the google map the google version. I don't know if any other user would be impacted. Okay,.

H

That makes sense, because I was wondering you know I haven't heard people reporting this, but now it makes sense now yeah, it's only when you're changing uh your certificate for hdod.

A

You can okay, so even then would it be an issue just because I guess if the replicas roll out at different times, but this is only a problem with two replicas of the same revision stomping on each other, not two separate revisions with separate certificates.

A

B

Then that should just be that's a bug if they're different, like two replicas.

A

Yeah yeah yeah.

D

A

So then, this is not a problem. In that case, uh this is just for the same revision.

C

A

Let's not worry.

C

Too much about it it's, uh but we still should still look at if, if using a gate, we wouldn't improve other things, but not in this context, not for this bug, but.

A

Okay, can we can we think of any other cases where we want a single replica from each revision to win a leader election? If not, um I'm happy to close this with won't fix.

A

Or maybe just remove the error log actually.

H

Yeah make the arrow lock less chatty, because I believe I've seen those in the outer log. I think that's all.

B

We added back off in another pr, so it's not doing it every second, it goes up to like a long back office.

A

Okay, so maybe this is uh not a big issue: okay, cool.

A

A

And yeah uh constantine and john, you can get your uh yours in a head at this one.

C

Oh validation, yeah, just a quick.

C

uh Are we validation, or are you still discussing uh validation, uh so uh the validation url? Does that there's a few things about validation that I'm? Actually? I have question about people more familiar with validation. If they can answer, my understanding has been that validation is not uh actually doing anything any lookup in the in the kubernetes cluster is more or less stateless. I mean you can have a validation web hook.

C

That is, you know, holding all virtually it's version specific, but not revision specific and it's not dependent on what you have what else you have in the cluster. So someone could have a single. You know: east uio, validate that all validation in the world are pointing to and that will validate revisions without any uh well there's some privacy implications.

C

You don't want to have as this ui or do world validation, but it could be something provided by the net is provided by someone or someone else, not necessarily that we uh is planning to have such a central validator for uh gateway.

B

Yeah, well, that would be on gke, probably only but on.

C

Gt but other vendors, probably you can do the same thing so basically they don't expect engine x to improve, to install uh gateway, validate or easier to install a gate, evaluator and you know, link it to install a data validator, so the one per kubernetes install the question is: if we can do something similar for easter, where we kind of move validation, you know maybe out of history, even or in any studies that is dedicated for addition or or um you know, for centralized geode or external studies.

C

And decouple it from from the current model, where you know we have election. A particular study is validating and it's pretty complicated. I would say.

B

So I the election, is only for patching the web book not for serving it, and we could definitely have like validation as a service right now,.

C

It's not only for patching, I mean it's also. You need to select which one it's a default version and because we we who's validating, I mean you have revision a revision b. We have a lot of complexity around it.

B

Right right, I don't think that in the default install we should ship validation as a separate thing, but I do think that being able to separate the code, so you could run. It is fine and I think trivial to do it. Probably a 10 line change. I suspect.

B

um One thing we could also do is we could probably move probably eighty percent of the logic to the open api schema. So kubernetes will do it for us, um although if we don't move 100, then it doesn't really help us that much because we still have the webhook well.

C

It does help us, because you know the 80 will be validated, even if you don't have something which is and it's faster and yeah. So I.

B

Think we could probably do that and should, but we won't solve these problems. Yeah.

C

And the other question is: if, if we can make it backward compatible, so if you have 112 uh if it is capable to validate for 111 and because we had the discussion about which validator do you use, if you have 1 11 112 at the same time in the cluster? Obviously you cannot create one to have resources because 111 will be confused, but if we use the 112 validator and 112 can validate 111.

B

And so what the euro would be like validate, slash, 1.11, yep.

C

Because that's the only thing that validation care about, I mean 111 112 and the version.

B

Seems reasonable to me.

C

Okay, let's explore it, let's, let's, uh let's uh spend some because validation has been hasn't changed in a very very long time and and with open api validation. Improving it's not even clear how much it's.

B

Yeah, I I'm, I suspect we can't move everything I actually know. We can't move everything to open api, but probably a lot of it. We can.

B

Which, I think is worthwhile.

C

Probably aligned with what, with kubernetes when kubernetes gateway, we moved to bennett's gateway six months from now or one year from now, whenever, uh since kubernetes will take over gate, probably that'll be a good time to schedule for, for, for validation, changes to involve and keep it stable until then, but uh yeah, because then we'll no longer have to fight. At least your gateway and probably other apis will will migrate upstream and uh become yeah.

B

If you look at the gateway, they have a ton of validation and they have exactly one check in the web hook. Everything else is open api, so it's pretty capable these days how hard that will be with our protobuf stuff. I have no idea, though, could be a huge pain.

J

uh What's what's the proposal here, I'm sorry I might have. The proposal is to to figure out if we can.

C

Improve validation and and uh turn it into a you know, revision independent service that could be provided by you know. In a multi-cluster environment you have only one validator, you know running somewhere and we you don't have to to worry about you know. Validation is revision, a revision b and it's you know.

B

I'm curious if google provided like validate esto.google.com, if people outside of gk would use it. Oh, but.

D

We can do east uil, bloody.

B

Date, that's true. I would feel more a lot more comfortable having a runtime dependency on google than on eastern io personally, but it.

C

Will run in google.

B

C

Don't know it's.

I

B

I don't know if it would yeah I mean it certainly won't be like you have to be able to run in cluster for a lot of setups, it's private whatever, um but uh something that we could consider. I don't know how ruling people off either to use it, but I think it would work for them if they wanted to.

C

Another benefit of a central you know: east ui of slaves. Validated. Is that in a cicd system you could just call this with proper certificate, and you can validate your own uh configs without actually going to a specific cluster yeah.

C

It doesn't think anyone will I mean if we host it under east ui url, I mean doesn't matter if it's google or ibm or someone else that actually is providing this for multiple validators.

C

uh It will help a lot of people because they will, you know, be able to use it in automation and other things.

C

Okay, no proposal, actually it's just opening uh for discussion and and ideas I mean if we, if we uh are interested.

A

Yeah, I think I think that's awesome uh right now. The way validation switches is like the tag command. Basically just swaps around the you know where the singleton web hook points. So if we can make that better yeah be awesome.

C

Okay, john, you have 30 seconds.

B

Oh yeah, so I talked a lot about this gateway, automation, stuff. um I have a pr. I think it's ready to go.

B

That's pretty much it I I think we should do this. um Is there any concerns comments, just review the pr one thing we kind of talked about before is like: where will it run and kind of the conclusion that I personally came to was that running it in east?

B

2D is the kind of only thing that really makes makes a lot of sense because, like the whole point, is that there's zero friction to getting started and if you have to go, install the gateway operator to use the gateway, then we've kind of defeated a lot of the purpose. So, with this, the user will just do helm, install htod, create a five line, gateway ammo and everything will just automatically work out of the box. So it's pretty nice experience.

C

uh I mostly agree there is one one thing that uh in another context as well, uh it may be useful to have this as a standalone container, that is version independent and it is not tied to istio whatever and still put it in history as a separate container, because.

B

Yeah, so I definitely agree that this should be able to be turned off and run standalone, because there are security implications except it's an easter d and it's very reasonable. For someone to say I want this separate, but I think in the default install issue no.

C

No, it should be, it should be included as an option in in in in http, but as it's not not part of history binary, but a separate controller. If we can do it. So so it's if I, if I have eq 110 yeah and I want to use gateway controller, I can take gateway controller 112, put it in in a different class because not and then then it's you know faster iteration. Basically we can. We can iterate on gateway much faster because, especially if you start generating materializing.

C

The uh is your resources right now it's in memory, but you could. If you have a controller, you can create your gateways to virtual service and everything that is grant created in memory, and then we cannot support for gate much faster and without.

B

Yeah, so I guess there's two parts like is the deployment part is fairly unlikely to change. It's like 100 lines of deployment service and having a separate container seems to be complex. If we completely split out the gateway stuff, then it may make more sense. Okay,.

C

But let's keep it in a design and I guess with master election and other things we can have basically and okay.

B

Yeah, hopefully the gateway stuff will stop changing so rapidly too, as it stabilizes right. Yeah.

A

B

C

Okay, let's ship it, I think it's super awesome.

B

um Yeah, so that's that's pretty much. The main controversy is where it lives and the permissions, but if we're okay with that, then it's pretty straightforward. um There is some not invented here, uh but I have a long comment explaining why I did that and I I think it's for the best. um It's actually it's extremely simple, like just creating these three resources, but it it's it's very hard to write a controller and kubernetes. That does the right thing. Apparently, but I've learned.

C

So we're going to document this as uh it it doesn't work for it for each okay, no.

B

D

B

So right now we have pretty much no docks for the kubernetes gateways once this merges I'll, probably update the current dock to give a lot more detail right now, it's like 10 lines and to discuss this, and I think the the way we should push this to users is that in the kubernetes gateway api, a coup, the gateway is the infrastructure and it represents everything and there's an opt-out mechanism.

B

If you want to manage it yourself, as for advanced use, cases but like that should be kind of the default is like a gateway is the actual infrastructure as well, not just a configuration object.

C

I was wondering if in is a three installed mechanism for gateways that we have today for easter gateway and should have a fourth one that say: hey, create a bernates gateway and then use virtual service and use your gateway. In addition to that. But instead of you know, using help, install gateway or whatever is to cut. Operators say that I create a resource, kubernetes gateway and that will automatically create the in.

B

A home chart, though,.

C

The user creates a kubernetes gateway for yeah and that will create the gateway. You know yeah and then creates virtual services.