Istio Technical Oversight Committee, 30 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Technical Oversight Committee 2021/08/30

Description

Istio's Technical Oversight Committee for August 30th, 2021.
Topics:
- Finalize Istio support window discussino
- Review 1.12 timeline
- Discuss Istio 1.10 and Kubernetes 1.22 issues

A

uh You know either lyn and christian.

B

A

C

C

C

For that you get from the meeting.

A

Oh, no, I have no energy, sorry man, I'm gonna drop, maybe a few. I just wanted to give an update here.

C

Thanks for watching.

C

uh All right, yes, I guess.

C

C

The release manager recruiting.

C

So you said, john you mentioned uh gauche from huawei is interested.

D

Yeah they reached out to me in slack.

C

Okay, um can you put either their slack handle or their gear panel or get them put in the sheet? I guess actually. I should link that sheet shouldn't. I.

C

Let's think about sliding okay, john.

D

C

uh All right, uh robot presentations are there any left. I think there might be.

B

Yeah, I think mostly done.

C

Okay, so much for my memory and then okay, one doc to review offline, the feature-free stock. Oh okay, uh all right, extended support, discussion.

C

So I think we we gathered several of the toc members opinions offline thanks john for reporting, uh niraj and sen. Do you want to record your opinion.

A

Sure so this is basically do you want to support 1, 9 or start from 110.

C

uh Well, it's do you want to be n, minus 1, n minus one plus six weeks or n minus two? Do you wanna be uh when do you wanna start? Do you wanna start with one nine or one ten.

C

How, or do you want to do odd, even.

A

Okay, let me just write it them down, though we can.

C

Move ahead with the next object.

C

Spend if you want to do the same.

C

For weakness of your opinion, um all right, schweder, you have a glorious reminder for us to work in a strategy document um which is due tomorrow. That is not going to happen, at least not from the googlers doing perf right at the moment.

C

um That's that's at least to push that out by a week.

B

Okay, do you think that would be enough time.

C

C

But it's it's enough time to maybe at least get started on it.

B

Okay, brian started a document, but there is not anything yet.

B

uh Okay, I'll share it with you offline.

C

Yeah or in the tlc select channels.

B

I am not in the dlc slack channel so I'll say all.

C

Right brian, can you do that or maybe brian's gone.

B

Yeah I'll share it with one of you and if you can share in that uses, live channel.

C

Yeah, okay, that sounds good actually.

B

C

And then uh well, let's get this the last thing the kubernetes.

C

B

Are you reading 8, 23, section, louis or 8 30 section, I feel like it's.

C

Oh sorry, I scrolled too far didn't I well. We did the extended support discussion which that's made me and then okay, you want to review the 112 timeline right. Sorry, yes,.

B

B

That's a visual version but and the table is more readable if it is.

C

Okay, thank you. So we started our kickoff.

C

What is tc priorities.

B

uh Review doc test case clarities, oh.

C

B

Well, the the p0's were done, and then we we wanted to change the um tc's for the most visited page and then we end up creating few more p zeros for the team, so we are trying to get nailed down now.

B

um I think there are only three or four left rest were done by the team, so I was talking to brian and eric last time, so we were planning on giving them early on so that they can be completed versus in the last minute, but yeah 30th september yeah by then we want it to be done and then a few other dates based on how we normally run the releases. But the plan is to complete that by 9th of november.

B

The only thing which I did not have is, if you have any conferences, um if anybody's aware I tried checking, I could not find, but if you are aware, if you want to double check the dates.

E

Like the 14th of october is during kubecon.

B

Okay, there you go.

C

D

Like five days so it's on monday, if well.

B

C

D

The people that are doing the code freeze are kubecon, then I don't think they're going to like that date. But if not, then it's probably fine.

E

Since we don't have release managers, I don't know that we know.

D

E

D

It on what's that, I think thursday versus monday is not a big difference. Anyways we usually get delayed a few days anyways. um So whatever ends up working there for the release managers, I think, is fine.

B

Do we have entities managers.

B

I was saying: do we have any release managers? Are you talking about code freeze right, john.

C

B

C

Any other opinions about the schedule.

C

Okay thanks, sir.

C

Okay sounds like there's nothing too controversial here, but we might adjust the bill date a bit, but we can figure that out when we get some of these managers. But this is the general general flow. Okay.

F

I think what I have one really minor question, which is like what what's the latest it can slip to before. We have to skip past thanksgiving holiday like how much how much buffer do we have here.

C

We've got 10 days, yeah.

B

F

Okay, yeah, it's quite fine that if we start slipping we'll need to adjust, it seems fine to me.

B

Yeah, the good news is, we have not um other than the last release, which was done one week, one or two days later. Most of the releases were done on the same day, which was quite achievement so and going with that history pattern.

B

We should be good with 10 days off thanksgiving.

C

Okay, um last year, we shipped release in december right. The am I doing my math right.

G

I don't recall a ship in december.

B

I remember 19th is what I see the date chloe. It was announced. It's istio 1.8.

C

I'm probably off by a year, um okay, um anyway, there's these are the dates. They look like the dates. Normally, look, I'm gonna go back to the other agenda.

B

Can we add the notes to that one pager that the eta is off by one week now, so we are moving towards um check the date six of september. That's this eta! Now, our 7th of september. That's a tuesday.

G

Are we meeting next week? I think it's enabled day.

C

Right, probably not, then.

B

Okay, so are we saying by 9 6? The draft is ready and the following week we are meeting to go over. It is that the plan.

C

That is a plan that is the best time we currently have.

B

C

C

Okay, coming back to extended support.

E

Let's see so louie taking everyone, except for sven, registered, essentially like ranked choice, voting in terms of the duration and which releases to extend uh summing. All of that up, the the clear winner would be every release at six weeks, which is everyone's either first or second choice.

E

C

That is fine by me. I guess it's fine by everybody by definition, who's their first or second choice, or they don't have a preference.

C

uh So I should capture a note.

G

Now we just need to decide which release to start with this.

E

And if I could summarize a consensus there, louie aside from, I think your oh wait. No, you said you're agnostic about 1, 9 or 110.. In that case, the consensus is very much a dry run on one nine, with official support starting in 110. I don't know exactly what was meant by the dry run. If that's like, are we going to release those binaries and cve patches, or so we might need to elaborate a little bit on what that means.

C

um Yeah, I wasn't sure what that meant.

H

Either so I think, I think that the dry run means that we just make a commitment for this release and then so, rather than announcing that this will be the way we'll do it for all of the releases. We just do it for one and then like based on what you find, I thought that's what tyrone meant.

F

Yeah, I think it means don't, don't announce it, but do it so I think it would mean building binaries.

E

So would we be releasing the binaries then and just like we're spending one night, but we're not telling anyone that we're extending it effectively.

F

Just put them on.

E

F

D

Who does that help well yeah exactly.

I

Why would we do that.

I

I was going to say I thought brian's dry wearing was, we would maybe build them, but we wouldn't actually release them, and that way we would know how much time it took to do. The.

D

Additional builds yeah, but we can't use this dry run as an exploratory thing to make a decision here. If we're already.

B

Well, at this point, I think you're right.

D

Yeah like, if we're already deciding that we want to do it for 1.10, then there's no point doing a exploratory thing.

G

Well, we just announced the support for one night, so the the the good thing about one nine I think as for mesh uh jacob, is lined up to do the work. So we don't have a staff issue.

A

There's a difference between announcing hey x, just to announce that the release is built and available and announcing that we officially import every release and plastics or whatever. So it's one thing to just announce a release and another thing to announce that will forever support the particularly schedule.

D

All right, it seems a lot simpler to me. Like 1.9, we've already said it was out of sport. uh Six days ago, it seemed simpler to me to just put out a blog post, saying, hey from now on, starting with 1.10, every release will be supported six weeks longer, here's the next, you know three or four releases when they'll be out of support and then hopefully, everyone's happy having like this partial state is kind of confusing. I think.

A

But they can still build one that releases, even if it does support it. I mean nobody prevent us from doing it. Even if it's not supported, we can still publish a release, something.

G

Yeah and don't forget most of your user on one one: nine from the survey image collected.

E

Yeah, one nine has only just peaked in terms of onboarding, um so as far as strategic-ness or strategy, to borrow a phrase, uh we're going to have a lot of users who need to upgrade to one nine. If we're going end of life now, making that process as painless as possible would behoove us. Although I mean whether we extend it six weeks or not, they they need to upgrade and fast.

J

uh Yeah and to just clarify so for 1.9, uh we did publish the announcement that it was going to go eol on the 24th, but the eol announcement did not go out um just because of uh the status from last week. So uh you know it looks a little bit in limbo. If you were to just go to istio.io, uh you know and as far as what is going on with one nine anyway,.

E

So we we have someone to do the work for one nine. We have users who would like us to do this for one nine. Is there a significant drawback to doing this for one nine.

E

John, I think you've been pretty pretty vocal in advocating for not doing in one night. Would you care to.

D

Collaborate, it's kind of confusing to users that would say like hey this version's under support and then just kidding it's you have six more weeks, but then, by the time we announced that it's actually like four more weeks and then the release managers have kind of already stopped working on it, but now we have to bring them back. It just seems like we've already added support or we're already going to add the support for 1.10 plus. It seems simply just just do that.

D

I don't have a strong opinion if we want to do something for 1.9 uh programs trying to keep things simple.

J

uh I think the hardest part can be- uh and this is more for- like the pswg right like for us um getting a release out doing qualification testing. um You know we do accept security patches still, but we don't build anything uh typically for older. You know, release branches if somebody were to put them up, um but you know during the release cycle, uh you know on patch day: that's when things can get a bit. You know chaotic and, and you know, we're kind of flustered. You know getting.

J

You know in this case 1 9 110.

J

I guess m111 not really 112.. I guess it was a little bit more just because we did three this password, this past past uh patch cycle uh versus. uh Typically we do two so um so yeah. I just wanted to throw that in consideration.

G

So how about this.

G

How about this? Let's start with one nine another alarms going forward just along. You know why nine we're adding like additional support for a few more weeks and then 110 uh we can allows for just depends on how one line goes. We can allows for every single releases we're going to do this and it it would allow us to utilize the data we have from one line to at least gauge the workloads.

G

uh How much workload on the table test release will google and how much workload on the product security work group to give us the confidence. If six weeks is the right number is. If every release is the right number.

F

I'm I'm in favor of that. I was actually thinking. I think about this. A lot like just any other kind of feature of this deal. Right like it needs to go to experimental before it goes to stable. So we can kind of announce, experimental support for longer releases, we're going to start with six weeks and we can say we're asking for feedback from the community on it right and see. If people respond, I mean we could also not announce it, but I think it.

F

It could also make sense to sort of announce it as an experimental thing that we're trying.

C

How do we want to gather feedback.

C

Because the one thing we know is, you know, anecdotal feedback doesn't allow us to make useful decisions.

E

Yeah, the difficulty is getting feedback from targeted users, users who actually took advantage of leveraging one nine into this new extended period and then did a direct upgrade to 111. That's a fairly targeted group of users.

E

What I could do is circle back to users who have particularly called this out in our upgrade notes in the past and asked for their opinion on it. There's no guarantee that they would be in the practice of using it. As a matter of fact, if they've filled out, our upgrade survey and they've finished their upgrade to the whatever version they're upgrading to it's likely that they would no longer be able to take advantage of the one for one. Nine.

F

So can we can we ask something relevant in the 111 survey, or do we already.

E

Know the 111 survey is live, but I last last week we didn't have any respondents, so it's not too late to modify it.

F

Yeah, I think we should at least put something in there to ask if they were.

F

Upgrading one nine, if you don't already have that in there and then ask them if they like how long they needed. I don't know yeah think of some way to ask.

E

I can modify the 111 survey to include that and maybe even to reference, the blog post, saying: hey we're extending one nine by six weeks and we'd like feedback.

G

Yeah, that sounds good. Is there any value to also update the 1, 9 and 110 survey, because I assume we use upgrade to 110 when they look at our survey? They could also ask be asked for a choice of hey. Is the extension of one nine support helpful for you, as you upgrade to 110.

E

Yeah 110, it might make sense to modify the 110 survey. I don't think it would make sense to modify the 1 9 survey at this point because the survey is designed for you know when you're upgrading to a new version. uh That's when you provide the feedback. Does that make sense.

G

Yeah yeah, except a lot of our users, haven't even made 219. They may start looking at an upgrade to one night, so it could be useful to collect those data too. So, like you like, like the diagram you produce right, a lot of views are not even on one nine today.

E

E

Okay, I can do that.

E

And I I will work on a blog post. It sounds like announcing this for one nine and requesting feedback. uh Is there anyone in particular who'd like to kind of collaborate or review that that post.

G

I can collaborate with you if no one else wasn't here, thanks lynn, yeah sure louie. I know you were going to say something. I think I just don't know.

C

My network's bad can somebody summarize what we uh what just happened? If you want to take.

G

Note yeah: I will do that.

E

Perhaps the most exciting thing about this is that in two weeks at the next toc meeting, we won't have to talk about extended support. I'm excited about that.

E

C

So I did hear the word experimental.

C

I don't know what that means. Right. Experimental means something to be taken away later, which is going to completely run against user expectations, for how people manage releases right.

C

Right, I think we already have strong feedback right, that we have to do something and right now, n, minus one plus six weeks is the minimum. We agreed to do right.

C

So I don't think we can call that experimental.

C

I'm not sure quite who made that comment, but.

E

I think what sven was saying was essentially to put it out there as we're doing this for one nine, and if, if this is not the right number or the right number of releases to extend, etc, we're looking for feedback we're to be honing. The policy over the coming releases sven. Does that sound about right.

F

Yeah, I I just don't want. I don't want us to say this. Is the policy going forward right, like you're going to you know, do every release at exactly six weeks right? Look. We just. We want to get feedback, so let's actually just say that.

F

It's it's an experimental extension. It's not experimental support. It's an experimental extension.

G

Okay, we're moving experimental, sounds good louie.

F

Yeah just say: extend one nine to collect feedback.

F

I think louie is having more network problems.

G

Okay, I think I captured everything so mitch you're going to update the survey for 191.111 you and I are going to collaborate, a blog to explain this to our user and then we're going to look at alarms. Official support, starting with 1.10., sounds good.

E

Thank you everyone. I know that this was a slog, but I really appreciate everyone's attention on.

G

Well, this you for driving this! It's a lot of work and force.

D

Are you going to take on writing like a blog post or something to announces jesus? Yes, lynn and I are going to collaborate on that.

I

And we are not officially posting an end of life statement for 1.9. At this point, correct.

E

Yeah, we should probably post an updated estimate for end of life on one nine for the new six week out date, which is approximately early october.

G

Yeah that sounds reasonable, so essentially it will be captured in that blog we're going to collaborate which we're hoping to publish in the next few days.

E

G

Louie, are you back? I think the next topic is yours.

F

Oh, that was that was last week or did.

G

We did, we know that, could we knock you over.

F

We didn't get this last week, we didn't get to it. Okay,.

G

Sorry I took the notes on last week.

F

Well, that's yeah, but.

G

F

All putting that's because that's where we had the place for the opinion.

G

Yeah I'll make a comment for people.

F

um I know some of the stuff around one to two, but I'm not sure exactly what we meant by that louis. Do you want? If you can, do you want to type it, and then I can yeah, I mean it's just or you can take.

C

Good, uh but one two two, we know breaks uh istio one line and below uh seeing as one nine is now end of life. um We just.

F

We just unended end of life, but okay. um Well then,.

C

C

And what we, the other thing, was one two, two um sorry 110 was not going to be qualified in the support window, one two two, but what we we were testing it, and so we indicated on the release status page that it was tested against 122, but it wasn't supported.

C

So there was kind of this chasm for users right if they were going to be on one two. Two.

C

You know what release were they going to be on.

C

They would have had to have been on 111 really to be on a supported version, which is a pretty demanding set for people in some cases.

C

But I think it's okay, that we indicated that it's testing tested.

C

But if we extend the support for one nine by six weeks, right, we're theoretically, gonna have more users who are gonna, be on one line for longer and therefore may be more likely to run into the one. Two two train.

C

But I don't think we were planning on we're, certainly not planning on backporting the support into one nine.

C

So I think we already took the only action we can take, which is to say that one time is tested. We could do a little bit more and say that 110 is supported.

C

Right, so that's really the only decision we could actually make of elements which would, in further 110 release, expand the number of number of kubernetes versions that we claim to support by one.

C

I'm not sure it's material, john. You have any thoughts.

D

Yeah, I'm kind of saying we officially support it. I mean we already did all of the work um to do it anyways so right now we're just doing the work and not getting the benefits of user adoption. I've seen a lot of people say like ah this product doesn't support 1.22, I'm going to switch to something else. It does uh that'd be a really dumb reason to lose an easter user when we've already done all the work to support it.

D

So I'm in favor of saying that it's officially supported, uh we certainly shouldn't say like going forward- will support five versions forever, um but I don't think that putting one extra version says that so.

F

Any any other opinions, so I'm I'm kind of interested in the like generic version of this right. So um you know, should we be generally doing some forward compatibility tests when new versions of kubernetes come out so that we can support it so that.

B

We can announce that it is.

F

Tested and or supported in older releases right and like how far back should we go? I think that's to me that's an interesting question. Can we set a precedent for how we want to handle this in the future.

C

Right, so, if is there a new kubernetes release that comes out during the existing support window of an istio release, should we qualify and support it? On that event,.

F

Yeah I mean it's always the case that when a kubernetes release comes out, it's not supported by stu right so like whenever a new release comes out. Our default behavior today is we don't support it until the next iste release, and so the question is: do we want to change that.

F

C

And that that would be the process right. We we perform the qualification, or at least some some of the qualification right- all the automation right, we're already doing it as john pointed out yeah.

C

So is there anything more that needs to get done process wise to just say that that's part of the release.

C

Release manager's requirements is that if a new kubernetes release comes out during the cycle that it's their responsibility to make sure the testing, the automated testing is run on it.

F

Well, how far back do we go, so I think it definitely makes sense to do that for sort of our current our current release. Should we go any farther back than one release.

C

Well, I think it should be. You would expect it to be any currently supported release.

F

So that adds a number of new tests that we need to be doing, which I guess maybe we're already doing but like the matrix gets bigger and then I guess the other question is: is this a provisional one where we say like if it breaks?

F

Where do we actually try to like go and fix things with patches to make them work? And where do we say? Oh sorry, this break, because you need to use a newer version of istio.

C

So, with their four month elite cycle, we will have a release that releases one month before their release right once a year.

C

Which means that release will be out for two months, while their release is out.

F

Yeah, I think it totally makes sense to me to say that we want to try to support that one, I'm just I'm wondering about the old ones so like what 110 supported on one two two makes sense, because one two two came out while 110 was the current supported release.

F

But one nine is more questionable to me, and one aid is even more questionable.

C

Right, I think that they're, the concern is the major vendors pushing like having things like gke and rapid channel right, where they promote the kubernetes release, two months after right to users who may or may not be updating their istio releases, yeah, all right. It's somewhere in the six weeks to two months window right.

F

Sir louie louie are you thinking we should just do it, for we should do it for all supported like we should do for like the existing back compat, but then do forward compat for every kubernetes that comes out. While our release is supported.

C

Decision was: let's make it an incremental step and see if it satisfies people, we could do the last release.

D

Do you mean the current like one, not 1.9 right, because 1.9 actually does not work with 102.? So so, when a kubernetes.

F

Release comes out, we say whatever is whatever the most recent issue is.

D

Yes, yeah, I think that's fine, as long as it actually is supported. If we find out it doesn't work, then we may have to have no.

F

Yeah, well, that's the question. Then. If it doesn't work, do we do we try to back homework to it right? Do we have to backboard right.

D

Yeah we might have to, we don't have any known breakages until 1.25, at least where we have only a few non-core or non-stable features we use now. um It's hpa and pod description budget, which is pretty easy because it's just part of the install. So unless they make more subtle changes that break us, we should actually be pretty good at going forward on compatibility.

F

And I think they're they're trying to not break people too right, like I think they don't want to be breaking istio, and maybe we should talk to some folks in the kubernetes community about that about how to make sure they don't do anything that, like breaks us the day before the release or come down there.

D

Yeah we do test their pre-release builds, um so we at least give them pretty early feedback when they they do impact us. um That doesn't mean for changes that they've planned and that they'll undo unnecessarily, but we've got like subtle, bugs that impacted only uco, for example. Before so, that's been helpful.

J

uh Yeah, so um so what do we mean that we'll test it? Does that mean just uh just running things in kind for that version, or should we also do um qualification, testing.

C

So all the automation, plus whatever functions the release manager would run to qualify or release, but not the bug bash or anything like that.

C

That seems the appropriate level of testing right.

J

Yeah, I would say so um I know eric eric is in charge of testing release, so I like his thoughts on it as well.

I

Yeah my only concern right now is: we don't do testing of all releases after we've, you know, after the dot zero release. As far as I know, we just.

C

Run the the automation well.

I

We run automation, we run qualification. We only run qualification against one kubernetes release, so we don't so usually when we we used to do the dot zero, we would run against you know in this case you know 20 21 22, um but when we do the point releases, I think they only run against. You know, say 22.. They don't run all of them again.

I

So it sounds like what we'd want to do is when we would pick up one of the older ones. We'd want to make sure at least we requalified that kubernetes version.

C

Sorry when you say that right because we're doing this when there's a new kubernetes version, so presumably we're going to want to run against that one, because that's where the most of the risk is right, the older ones would change less, and so the need to requalify against them would be lower.

C

I

So so I I guess the question there's two points. One is in in the case of qualifying a new version. Then yes, we just want to qualify that new version. I was also thinking you were talking about picking up additional support going back, but if that's not the case, then we should just have to qualify the new version, and typically we've already been running, I mean john, is usually very quickly getting our automated tests to run against. You know the beta versions before they come out so from an automated standpoint, we're usually way ahead.

I

It's just the qualification where we're behind. We don't call it. We typically don't qualify until um we release the point or release a minor release, the patch releases we don't so you're. So I guess I'm saying here is when we pick up the release for the new version.

I

So let's um let's say we want. You know we were behind. Let's say 122 was was came out after us. um Then we would pick up the 1.22 qualification testing in the next patch release. I think, is what we need to do right.

C

uh Yes and then we either well, we might not need to patch right to support it.

I

Correct, I, I don't think it's, I don't think we need to make a patch to support it, but I think what we'll do we would do is we would run the qualification testing and whether we run it on the next available patch or we want it on the last available patch that was out, then we can add it into that supported releases document in the on that page to claim support.

I

Does that make sense? So so so we either run the next patch release or the prior patch release from when the new kubernetes version comes out to verify things work, and then we would update the supported releases document with that statement.

G

Yeah the only thing with just qualify on the patch releases.

G

If you look at our documentation, we don't tell people like one nine dot, maybe five or only that version supports kubernetes 1.21, for example, um it could be pretty confusing.

G

So if there's no issue with dot zero, really it might just be easier to qualify back to the dot zero and just along support for all the patches. For that zero minor release.

I

Could do that as well.

C

Yeah, that might because we only indicate support at the minor release level right.

G

Yeah, it's just easier for user to consume that data. I think.

C

Right, but if we did need a patch to support it,.

G

Then it would have to be on that patch, yes, and on the apache later.

I

And then we just have to add a note in the supported releases document.

G

Yeah, hopefully, that's a very round case this.

I

G

Mitch, you had a hand for a while.

E

Well, I just wanted to point out, I think, in the case of 122, all of the incompatibilities were deprecations that had been around since 116.. um So this is the sort of thing that we could probably plan for and not be responding to granted. There are always problems that occur with subtle, incompatibilities things. We did not expect and those you know we have to deal with as they come, but for deprecations we should be able to plan in advance. I think.

G

C

Yeah and that's.

C

Is there a good way to be testing the proposed deprecations john, or do they kind of just creep in over time across multiple releases in kubernetes.

D

um I think there actually is a flag that allows you to fail on on deprecations uh that we can maybe start testing with, although right now it wouldn't help, because we know that it's it's deprecated like if you do like just like helm, install easter, it'll spit out a bunch of warnings if you're on a recent enough kubernetes that things are going to break in 1.25.

D

But nothing else can really be deprecated that we're using other than those two hpa and pod disruption budget, because they're all core apis.

H

That we depend on now.

D

Well, they could, but it would be the longest deprecation they've had probably I wouldn't expect it.

C

um Eric do you want to have a look into that? I don't know exactly how we want to define that from a well. I think the build test and release working group should figure out like if there's a way to start testing with deprecated features disabled.

C

um That would be some good hygiene. I think the the struggle there would be if the deprecated features are replaced by non-stable apis, then, obviously, that creates churn on both ends and that's hard, but I think that's something worth looking into.

I

Yes, we can do that.

C

Okay, so I think do we have agreement that we would just qualify the newest kubernetes release against the most recently released miner version at the zero patch level.

C

So we need to go and update some dock for the release. Managers.

C

Is that correct eric? Could you could you take on an ai just to make sure that that decision gets captured in the appropriate dock as well.

I

Yes, I can, I can do that and I guess the question is we're only doing this for the most recently revert released.

C

I

F

So so the algorithm here is like kubernetes 122 comes out. We test 110 0.. If it passes we're good and we just say hey, this is supported. What what exactly do we do? If it fails, do we test like do we go re-run qualification against one of our patches that we think it passes it or how do we want to match that? Well, hopefully,.

C

We know why it failed and yeah need to patch to fix it. Then then we assess the risk of making that patch right, but ideally we make the patch.

F

Okay, so there's.

C

Not an automated.

F

This is a like, maybe bring it up to tlc.

C

Type situation, or I mean if it's minor, then probably don't bring it up to toc, just go ahead and do it and just put it into the normal backlog.

C

uh If there's an issue.

F

Like that, okay, we do a patch, we do a patch, and then we do a very uh qualification of that patch, using our normal qualification of patch process.

F

That sounds good to me and then yeah there's a major issue bringing it up to toc.

C

C

I think that about captures it.

C

Okay, folks, I think we're about out of time anyway, so we are at the bottom of the agenda, but two decisions. Look at that.

F

And we put both of them in the previous week. Oh wait! No, we moved one back.

F

I can move this one forward if we want, but yeah move that one up so it looks like we did the work and in the morning also it helps me when I finally get around to publishing these videos, because I use the notes to say what we talked about and, if they're, in the wrong way it doesn't work well. For me.

C

uh I'll bring up the upgrade experience one and then put in a note, so I think it's okay, you can leave it there. Okay, that's fine! Yeah.

C

Okay thanks: everyone thanks have a great week.

F

And we're skipping next yeah. I capture that already, but yes, see y'all later.