Istio Technical Oversight Committee, 25 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Technical Oversight Committee 2021/10/25

Description

Istio's Technical Oversight Committee for October 25th, 2021.
Topics:
- TOC review of Istio Security CRD graduation to v1
- Follow up on extended support window - good adoption of skip-level upgrades
- Delaying 1.12 branch cut to update Envoy, delaying release by 1 week to accomodate.

A

Okay, all right perfect.

A

Okay, release support health uh release manager. This is something we should be discussing. I think we already have the candidates right with kenneth and daniel yeah they're good. Here, that's for offline review feature phrase uh jacob. This is your documentation.

B

Yeah, there's a pr uh sven wanted that so any reviews on that would be good.

A

Okay, I guess you still want more people, there's only two people from the us economy without.

A

Yeah, do you want to mention anything else for this vr give us an overview of struggles, review offline.

B

uh Just review it offline, I don't think anything is- is uh out of line or necessary to call out.

A

Okay, perfect. Thank you jacob uh yeah. I think you have a family agenda, so we should probably get to united nations foreign.

C

Yeah, so uh I just hope, like a tlc, uh could take a look at the talk and uh give comments and feedback and approve, if you think that is, uh that is good and uh there are two main takeaways from the work for graduating security. Crd from we want beta 1 to a1.

C

The first takeaway is that we plan not to make mutual changes in the api, because there were comments for asking if we should align with the new gateway networking api, but we decided that we want to graduate first and and make those alignments later.

C

uh The second main takeaway is that majority of the work mentioned this talk is actually about testing stability, documentation and uh we do not introduce new features in graduation work. Yeah, that's those are the two clean takeaways.

C

uh There are more details in the talk about like each of those uh things we prepared to do to graduated to be white, so you just take a look. What you think.

D

E

B

That's true, but I can repeat it, uh she asks. Is there what are the other types? Is it pure authentication, request, authentication, and I forget what the other one was.

C

It the type of crt request and the authorization policy.

D

F

So after one year of testing and feedback from users and everything else, we haven't found anything in the beta apis that we want to change. We just want to promote them, as they are. Everything is used. Broadly, everything is the same level.

C

uh No, would we actually have made something right.

C

Like the authorization we'll see, um that is a few. You feel no feedback.

F

I'm not worried about adding to them. I'm more worried about from from the original eti is everything broadly used. Is everything works in the promotion I mean additions we can make at any time is not a problem. My my concern is, it seems a bit strange that you know I mean luckily, maybe maybe we did an excellent job of designing it initially, but you know for all apis. We typically find some. You know parts that are not commonly used. Sometimes they don't work very well.

F

It's it's! It's um I'm I'm! I'm surprised that that, when an api moves from beta to to v1 without any significant change,.

C

A request for significant changes.

G

I mean custom. Normally it would be expected that there are very few changes between beta and stable right. I mean that's, that's that's working as intended.

F

G

No, that's definitely right, I mean we expect changes between alpha and beta. We don't expect a lot of changes between beta and.

F

uh Stable forget that many of the apis were kind of the jump to beta directly. I mean it's. It's changed to workload. I mean.

D

F

Workload, selector is the one that I was worried about the most, but apparently we are okay to move it, but, uh okay, uh if the usage of all treatment. My question is: if we move to to age point and for to kind of uh comparative cps and other things uh we may have to broad api surface.

F

That's uh I mean permissive mode. Is it's something that we want to promote to have v1 as uh supported, or is we we can just select a subset of the field like we did proxy config? We we didn't, move everything. We just went over the field and selected the ones that we felt most confident that you know socially stable, that can be implemented by others and so forth. But it's okay.

G

So young man, you you didn't mention peer authentication, correct right, you said: request, authentication and authorization policy. uh Sorry I.

C

I think peer association is also included here.

C

So it refers to all the three meaning: security, cld pures and requests- also n n, o c.

C

G

I certainly uh request authn and auth. Those definitely make sense to me. I think they're.

G

Peer off and hasn't had as much iteration as the other two have had.

G

Right both of these is an entirely new revision of you know the previous form of authorization policy request. Authentication, you put a lot more time and effort into. I think constant has a point that you know there is some inherent conflict between auto mtls and pure authn that we do need to be careful with.

F

I know for sure that, for example, approximately grpc can support and will probably support our z. So I have no problem with odyssey. It's it's. You know it's pretty standard, uh jots they're, fine, they pretty much have no support, but but but the permissive point is the one that gives me uh concern. I know hedge bond would not use it probably proxies will not support it. So it's it's uh dangerous. I think.

C

But we will uh so the interesting thing is you think if we promote it to we won, we we don't want to. Let you support permissive more than we want. Is that what you being costing yes.

F

I mean permissive will probably go away soon and will not be supported broadly and it's dangerous for users to believe that permissive mode is a v1, stable api. When a large part of office, you will not even be able to technically support it and then three like will go away and it's also the transition mode. Anyway, it's not meant to be something that is permanently used.

B

Yeah I mean uh the way I see it is permissive mode is almost not even an api, because it's the default behavior and it would be. I just don't see how we could ever change that from the default.

H

So that is the api yeah, that's the problem we have here. We made the wrong thing a default and it's now difficult to not break users.

F

When you switch to each one when you switch to hbo, you know there'll be a migration path. There will be some settings, I mean, but it's not necessarily an api. It's an install option is not.

B

I I'm actually a bit confused why we seem to think this is the wrong api. I don't understand how east geo would really work without permissive mode in the real world.

B

No one has clusters that literally everything is ego enabled, except for very very few users, almost everyone. I talk to has kind of homogeneous workloads. It.

H

Depends on everyone, how you're introducing sto to your environment and how you are migrating workflows right. I.

B

Mean I totally agree, people should use strict, but to get rid of permissive mode seems like a very extreme.

F

I agree with that now, I'm not saying we we need to get rid of it. For you know people migrating to eastern, just saying that the ap I mean. First of all, I don't know if it can be technically supported, I mean, even if we want it, it is not possible to support it in some environments with with our implementation, with the port being either.

G

We will have situations in h-bone where we will allow traffic on the original ports.

F

Yeah but but it will not be auto-determined, not the time is eventually. If it's, basically, we will do we sniff. We, we use uh the crazy stuff, alpn negotiation, all the other stuff it to be a different implementation, different mode.

G

I'm not sure, but I'm not sure that part of the api actually changes.

I

Go ahead, justin yeah, I mean I agree with john. It seems like the transition. It's important to be able to have this permissive mode, but also I worry a little bit about having some particular data path. Implementation, then dictating what apis we want to support. I understand that it can, if it's just hard to implement, that, might be an issue sometimes, but it seems like a fairly new addition to the data path and sort of letting that drive and modifying apis seems a little to me.

I

It seems like it'd, be better to add that support to the data path, implementation, as opposed to modifying our apis.

B

I would also mention that we don't have a technical solution to change fields between v1, beta, 1 and v1, so to say something like we will just not add this one field or something to the v1. Api really means that we're going to block this, for you know a substantial period of time until we figure out that solution.

F

We can hide fields from the docks, that's all we need to do. We have annotations of fields now we have.

B

You can hide them, but they can still be used.

C

I mean if we I'm still not sure like what's the benefit of just hiding in the v1, I mean like what do we want to achieve by like hiding this like a permissible mod? If we like fully support it.

F

We are not fully supporting that, I'm saying proxies jpc doesn't have it because it's a very other.

C

Is something new right that comes after we have the permissible mode like that.

C

And uh it loses.

G

So so one question I have young man um is: do we know how often the api is used? Is it one per install.

C

Typically, um we probably that you mean, like someone like explicitly specify permissive instead of using uh by default, no.

G

I mean how how many on average, how many peer authentication resources does the user create? For instance, oh.

C

I think we have those numbers, but I don't have it right now. I can get it and add it to the dog for a reference.

F

G

Yeah I mean if the answer is close to one per install: it's not an aps.

G

Right, it's a configuration setting and it would be right promoting something. That's used that rarely too stable right. We might just be better off deprecating it and moving it into. You know proxy config, slash management thing.

C

Yeah maybe look into the number and push it into torque for for this usage.

G

Yeah and the other two, I think everybody agrees that they have broad and diverse usage, but I think that one, I'm a little less sure about yeah.

A

Yeah, can you guys hear yeah?

A

Okay, one thing I do want to say is uh like when we recommend user to adopt istio for pure authentication policy, we do recommend them to change strict per service first and then per namespace and then look at globally. So there are a transition phases that we wouldn't recommend people to enable strict materials for the entire mesh.

A

Without this gradual phase.

F

I mean we can have this field in other places, not in the v1 api. That's my point.

A

Yeah, I just want to make sure when we have this, if we think products, because because a better home we we need to make sure it also allows users to do. Granule streaks tighten up their mutual trs.

G

Yeah, I think that's a fair point. um Okay, so I guess that should be some assessment made in the document.

G

I mean it's a beta api right. We can't get rid of it that quickly anyway, but if it, if we could, then that would actually be beneficial.

A

I mean there's no proposal out there on proxy config to replace it at the moment right because, ideally, we would like to see the proposal implemented before even at least you know have the design in place before we allow the application I'd assume.

G

But we would also have to make that decision before we decide to promote this table right.

A

Yeah agreed on promotion to stable yeah.

F

I'm only proposing to keep it is better. Just like networking api, so gateway virtual service will probably stay at beta for a very, very long time. So can this api can stay there.

J

I

Networking apis may have been at beta for a long time, but that was one of the things we wanted to avoid was staying and perpetual data not to say that this shouldn't not necessarily say that we should rush to move to promote this. But on the other hand, I don't know if the past past ways that we've treated apis is the right way to think about going forward either.

G

Yeah I mean I I would agree. We should promote this if we don't have a concrete plan to reconcile it with proxy config, but that should at least be assessed the other two. We should definitely promote.

C

uh That's just a small question, so, regarding the like the process, confusing kind of like a replacement user documentation or anything like a poc, I should contact with to like learn more details about this, because I feel it's a uh it's only a video to me.

F

I I don't even know if foxy configures the rest, I'm just saying that it's it's good to have additional uh discussion. Thinking about this, and you know two out of three- is not badly. We are pretty clear about the other two, this one. If we are not convinced, maybe it's good to delay, release or two there is no. No I mean nobody will die if we keep this one for another order or two releases.

G

Yeah, I mean one other way to do it right would have been to in authorization policy say you know whether mtls was required or not in a deny policy right.

G

um So that that is another way of rationalizing the api.

C

Let's see yeah we can we can, I mean you mean we can always use possibly to like uh uh kind achieve the strict mod so that we uh we still have to do that right.

G

Okay, if traffic is not mtls, then deny it.

C

Yeah yeah, that's actually like in one of the like uh reconditioned configuration we have in his total, I o the user. They can apply additional osa to achieve this. In addition to the pure also.

C

Okay sounds good.

G

Yeah so I mean like yeah: we we should do an assessment to see if that is also a better, is an adequate way of expressing this requirement. So we can have one fewer api, which is usually a good thing.

A

Yeah, that's good. Okay, great. I tried to take some notes um so essentially for pure authentication we're going to keep it as a beta until we have a plan, whether we are keeping it or move it to proxy config, request, authentication and authorization. There's no objection from the team here.

F

I mean I think that the current status is proxy configures out of the uh selection. We don't plan to move it proxy config uh louis proposal. It's actually perfect, I mean it's just. uh We will imply it from the authorization. If you have authorization policy implies, you have strict because plain text, you cannot verify the principle. If you don't have authorization policies, then it can be permissive.

G

Well, yeah, or something like that, something like that, I don't think that's quite right.

A

Okay, that's perfect yeah! I added that portion too. So, um but let's let's, uh when we have a plan, then we can discuss whether this is beta or it's going away with beta or anything anything else we should be discussing for this topic.

A

Okay, great uh it's not do. We have nature on the line to talk about extended support.

K

Yes, I'm on the line. Can you hear me.

A

uh Yes, we can go ahead. Nick.

K

All right, so I'm on mobile, because there's a storm coming through the area right now.

D

K

Internet is not reliable. I apologize if I break off, uh but I just wanted to give a brief follow-up on our decision to extend the support of the 111 release. You may recall that we were aiming at a behavior change on the part of our users, we'd like to see more frequent upgrades, or rather not more frequent upgrades, but more frequently we'd like to see our users on an up-to-date version or a supported version of istio, and so we extended the support of 111 by six weeks.

K

That support ended on october 5th, and I took a snapshot of gke data around upgrades on october 7th to sort of capture how we did um and what we found in terms of upgrades was twofold one. We found that uh the skip version upgrades one eight to one ten, one, nine to one eleven has actually been a common user practice uh since far before we supported it so about ten to fifteen percent of users in any given release that upgraded would do so across two or more major versions.

K

uh That was interesting to see even before we supported it in one nine to one eleven. We see 28 percent of uh of all of our upgrades are coming directly from one nine, so that is a user behavior change, but also we're seeing that we're supporting the existing behavior of users. Additionally, one nine has now had the fastest turndown of any istio version. Yet and 111 has had the fastest adoption of any istio version. Yet so I think overall, we we have moved the needle we're seeing more users stay up to date.

K

On istio, we've got a long ways to go in that we still see a good number of users on the one nine release now that it is not supported.

K

I think our first zero day for that is expected early next month and we will still have a substantial number of users there so room to improve, but we've definitely moved the needle in terms of upgrade experience.

A

Well, this is great. I have a quick question. I do remember: tke had a rapid channel right. So are these replicano? Oh, are these normal gke.

K

I don't have any data correlating to gke channels.

K

Those are like.

A

K

Data sources, so I can't.

A

K

Them thank you.

H

So this is in general good match and thanks for sharing it, what's the recommended action, if you have any for us, would you like the six week window to be extended further? Should we stay on course.

K

I think staying the course would probably be what I would advocate. I haven't heard any complaints from users that six weeks was good, but not good enough. um It would be interesting if you've heard from users. I would love to see that feedback, but we certainly moved the needle. We could try a 12-week extension at some point uh and see if that moves the needle further.

K

The risk, then, though, is the the messaging to users of this relief is released, is extended by 12 weeks, whereas no now we're going back to six weeks, we don't want to vacillate and confuse our users.

K

H

I agree that makes sense.

A

But I think niraj is asking a really interesting question. So if you guys look at this, when we have this announcement out there, we did kind of say: did we get any actual feedback on any of these, because we did say you know, based on more data, we're going to decide whether to continue this path or not.

K

My efforts at collecting feedback have been around empirical data rather than you know, sort of anecdotal feedback from individual users.

A

Okay, that makes sense.

K

If you would like anecdotal feedback, I can direct my efforts towards that in the next release.

A

No, no, I think the data you have was great. I was just curious if you get any data through these any of these channels that we mentioned or upload some freaking, not really.

A

So I think nietzsche will do for 110 right for the extended support. Are we going down to the extended support for one time or not? I assume when today is going to be soon.

K

Yeah, my my understanding is when we decided in august. Our intention was to continue with this policy unless we found that it was not useful. I.

D

Believe, based.

K

On the data that I've been able to share today that it has been useful, and so we should stay the course with the 110 release, extending its support to six weeks beyond the release date of 112.

K

F

K

Or less, unless we see concrete reason to change, I would suggest that that become our policy moving forward.

A

Yeah, because only the extension allows them to do the skipped version upgrade, which you are seeing. A trend of people adopting.

K

Right, the extension allows them to do it while still supported um you know not not leaving the support window at any time. We have seen the users have been doing this for quite some time, meaning that they've been doing it without support either for the skip upgrade itself or for the older version at the time of their upgrade.

K

So in some cases we're simply taking an existing behavior of our users and making sure we support it. Well,.

A

Any other comments from anyone.

L

So, what's the one comment I have is that I would like to see similar data from other participants. I guess solo.

L

uh One data source.

A

Yeah, because we're not the cloud provider, it's actually super hard to collect these type of data, and I mean the only approach we can do is ask individual users.

A

Like we couldn't get a massive data of what's running on a given cloud for the kubernetes.

A

I don't know if these data too.

A

I just realized that eric is on vacation, so that would be more question. Go ahead.

F

uh If, if the user's customer behavior is to do you know skip version, I mean from one line to one eleven one and basically to have fewer upgrades, basically uh wouldn't make you know kind of reopen the issue of.

F

Do we want to have all d1 releases and apparently odds is more popular with these two, because 1 9 and 1 11, but wouldn't make sense for 1, 9, 1, 11, 113 and so forth to have even longer at least to give them more time, and so so we kind of keep the pattern of having uh skip release, but uh with with the deterministic deterministic, uh you know which questions are mobile are more. uh I don't know how to say yeah popular.

K

I I don't think that we have enough data to say yet that users prefer odd releases or even releases before one nine became our most popular release. One eight had been our most popular release, um so so we see a fair amount of distribution, but I can keep an eye on that as we do the 110 to 112 cycle to sort of see if there's a sort of tick-tock cadence to upgrades.

F

I think the number to look at is how many people do each version versus how many people do uh skip one version, because it doesn't matter which version they skip, but if they, if they do skip version because they want to have fewer upgrades and they feel confident that skipping one version is safe, then uh it will be better. For us to you know not have all versions be skipped versions.

F

I mean to have one night, one eleven one ten to one twelve, but to control a bit the the virtually because it's easier for both users in our store.

K

Since um I am don't have permission to share the raw data with the group, the broader group but cost, and if you reach to me offline, I can share the raw data with you so that you can see those cadences thanks.

A

Okay, it looks like there's no further question from mitch. Thank you so much for giving us an update. I think that really validates uh the plan we laid out early this year on extended support on one night has been working out really well with that we're going to be moving on to daniel to talk about 1.12 uh branch, cutting daniel, take it away.

M

Hey there yeah that's right um ken, and we have been discussing this and we so we've been delaying the branch cutting for 1.12 because of the issues around the envoy upgrade um there had been, and there still is an automated pr that hasn't been merged in three months um this week there has been actually end of last week. There has been some progress exactly this is the the open, pure that supposedly fixes the issues uh that stem from the envoy update, but it's not merged.

M

Yet, uh in any case, we are now almost two weeks behind on the branch cutting so kennen, and me would like to propose delaying the release by two weeks so that we have uh appropriate testing time once we cut the branch and get the first builds, uh because otherwise we would only have time for a single community testing week and no buffers basically. So basically, the community testing week would end of the day of code freeze. If we find problems late in the testing week, we can't fix them.

M

So adding two weeks would allow us to do two testing weeks and it would give us some buffer time to fix any issues that might show up during these testing weeks.

A

Is there november 11? Is that oh nice, with our current release date for 12.

E

Yeah november 9th is the current.

J

November 90th.

E

E

We have been discussing this by one more question uh along the same lines. I know that envoy fixes, like keep on delaying. How much um confidence is this, that this uh fix will be available soon, although it's merged it was merged last week, so I'm not sure.

B

We already have the ego test passing, but we just need to merge it, so I think we're very high confidence.

K

M

But but it's still a good question, john: what's your confidence level on any remaining issues that might pop up that we might not be catching in our tests. I know we have some issues related to changes in envoy and 120 that might be causing problems.

N

um Yeah, that's a good point so.

B

I mean we may find something community testing uh more realistically. If there is something that's not caught by our test, it's going to be caught by users after we ship it just historically speaking, I think there was still one change where the envoy removed a bunch of filters that our users used through envoy filter that we decided we would add back, but I don't think anyone's picked that up, but that's like one or two hours of someone making some basal changes. So I don't think that's terribly high risk.

B

That's all the issues that I know about, but there certainly could be more that I don't know about.

M

Right so we yeah go ahead.

C

Oh sorry, I just want to finishing another thing: is uh we probably have a few prs like uh waiting to be merged to the 1.12 branch after the only dependency is resolved?

C

So those things might, uh you know, introduce some like a new changes in the 1.12 that is not fully tested because it's being like a blocked for a long time because of the outweigh dependency issue.

M

Right, so these sorts of changes might uh increase the probability of us hitting anything in the community tests.

A

Yeah so yeah I mean the changes you mentioned. um Are they release blocker.

J

C

I'm so the one I know is, I have another pr for implementing the job. Claim-Based routing, that's one! Pr like waiting. I get blocked by the only dependency update, uh that's a feature. We are planning to roll out in world war, 12, so uh yeah, but uh it's kind of like a. We really needed that one in one to twelve.

C

So it's it's not like a release broker like a bug fix, but it's something we have.

A

Yeah makes sense a new feature, but you depend on this envelope dependency to be able to do your full validation, so we will have to call you know: are these still reasonable to drop these type of changes for one child? Given one child is already at risk.

C

So that pr has been like a reading for probably at least one month, there's only one pr implementing this feature. It touches a very small code base in the networking code, so I will really want to like get this in u1212.

C

Because it has been like us for many people and we have done all the things to get ready for one or twelve. If it's large the like our dependency issue, it would have been like again one or twelve four, at least one month ago,.

A

I see yeah, so what do other tlc members think about the delay? I just look at the calendar. If we do delay two weeks, it would be the tuesday on the thanksgiving week.

H

I don't think that's a good week to release.

B

Maybe delay, but just by one week instead of a few.

M

That's an option: yes, um if we delayed by one week, so I'm guessing I'm hoping that this week actually middle of the week, we will be able to cut the branch and get the first builds out. If we then immediately start a community testing week.

M

Then we might be able to even do two of them before code freeze, if we, even if we just delay by one week, so I I understand that the the date in two weeks might not be ideal. So yeah, maybe one week, is a good compromise.

A

Okay yeah: let's focus that and then see daniel. Hopefully you don't need to come back, uh certainly if we have to have more time. Definitely you know back again in the tlc meeting.

M

I'm sorry, uh I didn't quite understand that.

A

Yes, that's just a delay by a week first, and hopefully that would give you sufficient time. If not, uh you know, you can always come back to the tlc meeting a week and let us know in a week or two.

M

Right yeah sounds sound, sensible, yeah, uh mandar.

A

M

I don't have that information. Somebody just asked in a chat how many issues we usually find in the community testing days. um Maybe john, are you on the call john wendell.

H

My question was also similar to mandal. Basically, if I understood correctly many of the things that we have now are automated, so we don't actually have a lot that's manually, tested during community days. So if that's the case, I don't even know why we need a week or two weeks.

L

Right I mean the the community testing day. One of the things it does is make sure that the documentation is aligned with what's built um because the automation doesn't read the text of the documentation, but apart from that yeah any any real kind of any real issues. What have we been finding so so yeah? So maybe one week is actually enough and not really even a compromise.

H

But this shouldn't impact the text of the documentation right like this is changing.

L

All rights correct so this this particular thing shouldn't involve, but any text that has already changed is not subject to community review until we declare uh community testing day and that that is one of the clear functions to make sure that.

H

But if I understood correctly, for whatever is automated, we say, don't even do anything for the doc page. That means we only actually look at the documentation text for conceptual pages, architectural phases right.

H

A

The case not every single time.

H

Now we can't hear you at least I can't.

L

Okay, so so it looks like the trc already voted for one week, uh I yeah so so maybe the decision is already there and we don't need to continue further. I I just wanted to kind of see why we would be dealing it, but I don't think that changes the decision one week is great.

H

I I don't think we have enough people to even agree to it, I'm okay with leaving it one week, but I'm with you. Actually I don't understand why we need to delay.

M

Well, somebody maybe somebody from my last time as a release manager. A lot of the testing was manual, so maybe somebody from the most recent releases can speak up, but my understanding was that the more time we have for this release to get into the hands of people and have somebody playing with it the higher the probability that we're not going out the gate with somebody that will pray break production deployments.

M

um Of course, if we have I, I know, there's been a lot of effort going into automation, of the tests that we've been running in the past and if that is, if we think those tests are enough to justify a release, then yes, we might just say we don't delay the release.

M

That would mean we have one week from branch cutting to code. Freeze.

F

Go ahead, sorry gusting go ahead. Now I was going to ask: do we have enough automation for stability and uh because that's usually the kind of bugs we find when when we, uh when we let it run for a weekend and start using it in practice and find our kind of memory, leaks and other kind of weird.

D

F

Which we cannot capture with the grand testing framework did anything change in this area. We do have stability. That is automated. Now.

E

So one one thing to add: here: um the community testing has changed and um based on the recent feedback, last two quarters back or two releases back from the toc when we discussed is um most visited pages, are the ones which are prioritized so two quarters back.

E

Our p zeros, which were identified before has all been automated, and then we had a course correction and we decided that the pages which are visited mostly are the ones we want to mark easier and if they are not automated, we should automate them at this point in the community testing. We just make sure that if they are not automated, we automate them or manually test them and other than that we don't do any feature functional testing so based on that um other than three.

E

I think everything is automated, so we may not need one extra week, but I would like to keep that as a buffer. Given we are only two weeks away from the release and one week normally, we need for the code, freeze, running the long running tests and it's always good to have that buffer.

E

um So um I think we should be still be okay with the ninth november, but it's good to have that one week which toc has granted so neeraj and mander um that one week I think it's mostly for the extra buffer, but we may not need it.

E

It also depends on when this envoy uh fix is also getting merged and when we are doing the code freeze, but I'm not expecting a lot of changes in the community testing just because uh from the recent releases I haven't seen much uh bugs found during those most of the time they're automated when we find them during the releases anyways.

E

So costing a question to you when you talk about stability, what do you have in mind? Is this something you relate with the community or is this something you relate with the longevity tests and you see after 24 hours to 48 hours, you see those stable lung, stable tests like how do you connect them with it? So.

F

So, typically, what they expect when you start community testing people from different companies, you know pick the the release and they also install it in some canary cluster, where they have their own application, and if we have tcp first broker or have any kind of the web feature or interaction with a particular crazy application that we have no clue they will detect it in that interval. uh Second, is yes long running, I mean you, let it running for a week.

F

You install it today running for a weekend and who knows what will happen on memory, leaks or other things, but the first one is the more important because that community testing is basically the interval where people may try other things, not only our scenarios which are already tested. I completely agree that we don't need to worry too much about the the you know documented install upgrade it's just how to interact with others.

E

So then, um in in that sense, constant, we don't run those kind of community tests anymore, because we had a very poor attendance that has been stopped for a very long time. This community testing is very much about the docs testing and that's what we do um and most of those are already automated. So we really generally don't do this.

F

I I know we don't do it, but my my expectation is that when we announce community testing, our users will do it on their own, because, if you, you know kind of uh you know vendor center, it's not only what we do in that world community testing, where people sit around and install it. But it's also, you know it's a signal for vendors and users to try the new version before it released them and.

A

Yeah, I think it makes sense to provide that extra buffer and, like vendors, I know sometimes like to try them right just to make sure that their is your offering is going to be compatible with the upcoming new release. So having the release candidates out there and having that time enables vendor to try them.

A

So if there's no objection, I'd like to propose that approved for the delay ball gave daniel and the release team one week delay by the one week beyond november 11 on november 9th, which was our prior release date.

F

And and maybe maybe we should make it more clear that that our expectation is that vendors will actually do these actions that I hope they are doing, because it's in their interest and kind of translates all community testings that we used to do and to hey the new release is coming. You have one week or two weeks to actually, you know, do this and that and then verify your application is not going to break with any release. Basically,.

A

Yeah, I think it's good to reach out to the vendor, actually, especially people who are building solutions on top of the issue.

A

Really we're out of the time to uh schweither we'll have to line up your topic for next week. um I do know seven and the toc team do have a roadmap, so we can perhaps review it next week too.

A

Thanks everyone for joining.

M

Thank you, bye, bye,.

D

D