Kubernetes SIG Node, 15 Aug 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node 20230815

Description

SIG Node weekly meeting. Agenda and notes: https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#heading=h.adoto8roitwq

GMT20230815-170507_Recording_1920x936.mp4

A

uh Hello, it's a weekly signal meeting, it's August 15th 2023. We just entered second half of August, so participation is not set huge, um we're doing uh mostly retrospective. Today uh we have a couple more topics, but um I think with respective for 128 will be the biggest uh topic for today. uh So let's get going Raven take it over.

B

Yeah uh thanks, okay, so uh as before I uh uh put together all of the Caps that we tracked for 128 and I marked them as a merged or not merged. So we have an overview of uh how many caps we emerge and how we're doing- and we have some history data so first it will go through all the caps and then uh we will do some retro on what we uh did well and uh What uh didn't go too well. So, uh let's start with uh caps.

B

Here is a list of all of the Caps. Let's just quickly go through them. We have a uh 31 caps tracked uh for 128 and out of them we have 16 merge uh I, have to say that uh some of the Caps are marked as tracked on their cap issue, but I just uh take them to the comments a little bit to see if they really merge or not. So, if I misunderstand the status of any cap, just let me know, look through them. uh Yeah the first one uh support username spaces.

B

This is an um existing cap is staying in Alpha in 128, and this one is submerged so.

C

I think what I would like to add here is we merge the support for volumes, but we missed one piece, which is the Pod security policy piece.

C

So we need to continue working with cigar to get review on that PR.

B

Gotcha: okay, thanks all right. uh Moving on to the next one support memory, Qs with SQL V2 is uh promoting to Beta and it is not merged here. uh Next, one yeah.

C

I think we can quickly have a summary of that one, so it didn't work out so dikshita I know you're on the call. If you want to briefly uh summarize what happened.

D

Sorry, so we won't be able to uh move ahead with the beta, because we found an issue uh in how we were understanding the usage of memory, dot High um it it. If we use memory dot High, it will actually block the processes forever and hence we don't want to use it. I'll also go ahead and summarize it in the cap and uh as a bug comment.

B

B

And moving on hot conditions around Readiness to Star container after completion of pots and path creation, uh this one is merged, as Alpha 2.

E

Yeah I mean I actually uh yeah, that one is good to go. uh The one thing I had to do was the the original author was uh wasn't able to get a hold of them, so I had to kind of create a new ticket to track this and get this more up to date.

E

So hopefully I kept this one in a more better state.

B

Gotcha thanks: okay, uh moving on improve a multinum alignment and topology manager. uh This one is uh promoting to Beta And. It is merged in 128.

B

And the next one uh well, this is my Gap. A limit on Parallel image pool is trying to be promoted to Beta and it is not merged. uh I didn't get the time to write the test. There is some change on my schedule so yeah. That's that I'll try to do it in the next cycle.

A

I think for this uh we have documentation change that accidentally got merged. We need to reverted.

F

uh What change documentation.

A

B

A

B

Okay, thank you and the next one is. Oh sorry. Let me just uh document.

D

A

B

uh Next, one is uh the node memories. Web support is promoting to beta1 and it is merged.

B

And then sidecar containers uh it's uh there are some changes, but it's still in it's um still Alpha and as the changes merged the next one uh State advisory, crifle container past status uh is uh trying to be promoted to Beta and it's not merged in 128..

B

And then fine-grained supplemental groups, control, uh yeah attempt attempting Alpha again not merged next one retrievable non-retrieval pod failures for jobs is some changes but still staying in beta and it is merged and then in place. Update for uh pod resources uh is sustained Alpha for 128 and I, don't think there's any change emerged for it. I might be wrong on the status of this one.

A

Okay, we wanted to support Windows for that, but uh we failed to merge that.

G

B

Okay, uh next one CDI devices to device plugin, API uh Alpha and it is merged uh and then uh split uh standard out and the standard error log stream. uh This is not merged. I, don't think the author uh made too many comments or too many updates on the issue, but yeah it looks like there is just a uh not too much raisin update and next one is so Insurance equal, put images, uh yeah, uh Target and Alpha, and it is not merged.

B

And then uh introducing sleep action for pre-stop hook, uh Target and Alpha and not merged, and then discover C group driver from CRI, Alpha launch and merge, and the next one is support for dropping complete configuration, Alpha launch new feature and it is merged.

B

Next, one is the decouple 10 manager from no light cycle, controller I'm, targeting Alpha new feature and it's not merged.

B

Next one is a field status host, IP, add a new uh few to host eyepiece per pod and Target in Alpha and it's merged.

B

uh The next one is also a new feed Max to restart times. uh This is also Alpha and it's not merged.

B

uh Next one Cloud dual stack, node IP handling um yeah targeting beta and it's not merged.

B

uh The next one is uh uh graduate coupon resources endpoint to GA I, think this is an old beta feature and it's targeting stable and it is merged for 108.. uh The next one is uh well. This is more of a policy chain to support uh Otis the node with newest control, plane, uh yeah, and that is merged.

B

Right uh next, one X10 Paul resources API to report allocatable resources uh also on launch in a future to be promoted to stable and it's merged, and next one is uh non-gorithm, note shutdown uh promoting to be stable and it's merged, and the next one, uh configurable Grace Period 2 probes, uh trying to be promoted to stable and is not merged.

B

F

B

Think this did merge. I'm, sorry, which one yeah.

H

Yeah this one GA the grace period seconds.

B

This one is merge or no yeah.

I

B

I

B

Merged okay, yeah.

I

I need to do a doc change uh for.

B

All right cool thanks, uh yeah and the next one uh dra Dynamic resource allocation, uh Alpha and I- think the changes are merged yeah and the next one is to support third-party device monitoring plugins. This is also a long-standing feature being promoted to Sable and it is merged uh next, one incubator resource metrics endpoint of promoting to uh stable, and this one is not merged and uh the next one is uh sub.

B

Second, uh more granular probes, uh new feature, Alpha, uh not merged, and the last one is um uh Kos class resources also Alpha new feature, and it's not merged so yeah. These are all the uh 31 uh caps. We tracked for 128, and so we have actually 17 of the merge, okay and uh yeah uh to compare these are some of the history data. uh We had way more caps, attracting 128 compared to previous Cycles.

B

Okay, so with that, let's go to the Retro part.

B

Yeah on how do we want to do this? We cannot take first to take a quick look of what we found out in the previous retro for 127, and then uh maybe we can uh sync if we have achieved them in 128..

B

So this is uh uh what we found out in the 127 red shows think that went well. uh We did uh in place pod vertical scaling. uh We had some uh good progress on graduations cap, graduations and uh yeah in 127 we had more caps track emerge, and now we have even more abstract, uh better overall planning. uh Everything goes on. Time uh did a good job, applying milestones and yeah. We had more chairs and yeah. We decided to not take sidecar for 127.

B

and for in place. We decided to not log on Windows support uh cap reviews. We did that earlier and I yeah. We identified issues through tests before releasing.

B

uh Things could have gone better in 127 from merged in place late uh due to the review Samsung bugs and not too much going on for fixing the In-Place bugs right and we broke a standalone cool word yeah and also we did most of the reviews uh right before the phrase yeah. Okay. So that's what we found out in 127. So let's uh talk about uh 128, then.

A

um If you're talking about what could have gone better, we still not very good in uh merging early. So sidecar is an example when we wanted to merge as early as possible and we were ready, uh but uh we were pretty late.

I

It's the function of reviews.

A

um Both uh reviews actually revealed something, but uh we only get good reviews closer to end of the cycle.

I

Yeah, you said only got good reviews close then.

B

Okay, um anything we did good for the last cycle, trying to think well.

J

B

uh We we we tracked more caps than before. I think this is the uh it's the most accounts we have ever checked for a single cycle: I'm, not 100 sure, if that's like a totally good thing or not, but uh I won't just want to mention that in the things that went well, part.

C

I think probably we should look at the ratio right so.

D

C

Completion is less ask one percentage 13 out of 19, and this is 17 out of 31. yeah and.

F

We get spread for nothing.

J

C

F

But at the same time we need to work. Also, it was size of a caps were small and straightforward ones, and most of uh big ones will just postpone it.

C

Right I think, like we have capacity, was I. Think sidecar was the cotton this time, so we need to identify early, which.

A

Because, oh I remember.

F

A

Is a function of better planning, so we guide people more on how to get things into.

D

A

Then, during the release, we don't have as much review power.

F

But what actually repetition of what was some previous uh like what we can do better, so we can use the time between our releases to do reviews for vowels, which will not fit into previous ones, so like not not to postpone it until the last moment.

H

One area I think we struggled, and maybe this goes back to the big ones common or maybe what you're alluding to here is um some of the Caps I think that we present are not.

H

um They present.

D

H

Idea- and maybe they have like uh 80 of the idea of thought through and I- think we struggle to get um a broader set of folks engaged in helping to flesh out that final 20 I think we're all pretty strapped in that area and I think that's where we seem to break down a little bit like the more the idea is presented and uh covers all the use cases, the better we are in reviewing and closing it, but helping close those gaps and ideas that are presented.

H

That I think can cause some friction between people who want a big item and those who are trying to make sure the thing is uh I guess as baked as possible. But I don't know if I'm alone in in that perspective, but um I don't know. If that's also what you're alluding to.

F

um Oh I was just referring to uh what was what was written on the previous retrospecting, so, like most of the reviews are done in short period of time, just before the deadline, and practically before we have more time between the code is complete and under the deadline for my new caps.

H

Yeah and part of the reason why human factor I think on why some other reviews are done near the end is because folks we're trying to think through how to help present.

I

H

Pathway to close the 20 that could be missing in something, or you know, the chicken and eggs there are sometimes like get presented in like bootstrapping problems but um versus everyone having the time to maybe collaborate on Big, Ideas.

H

uh uh uh At a faster Cadence or what um I I know personally I struggle with that I, don't know if I'm, the only one so I and I know, there's a few others that we read some caps and we're thinking what to do on this and there's probably some guilt on on trying to figure out how to how to close some questions, and sometimes the Caps present things as Alpha and then defer the resolution. The big problems until beta and I.

D

Don't know personally.

H

I, don't like that um so yeah, just food for thought.

C

Deadline also acts as a motivating factor that, if you don't review it, we have only a week left. So that's when everyone gets lit up too go and review and try to get things much I, don't know like what other motivation I think we need to come up with other motivating factors to kind of spread. That reviews out across caps.

I

There's also, you know a little bit of a bleed that happens where you could split up certain caps into parts, and we don't because we're looking for you know to boil the ocean a little bit too much, maybe or at least we have different opinions on what that would be.

C

Yeah I think we're generally good a fee is a problem, is well understood and scoped and contained, and we are able to move faster, I'm, not sure if it's always possible to take a new idea and break it down cleanly. That way,.

I

Yeah it almost feels like for some of the Caps. We need a different track, an architecture track where we're deciding what the total contents should be, the end game. You know you know, and then the parts that we can make progress moving forward right, I think sometimes we're too concerned about putting something in that we have to support forever. But it's even if it's just an alpha.

A

Do you know things that went well.

C

I I would like to call out that the qos testing was pretty good. It caught an issue in time and we were able to pause the move to Beta. So while that, while we don't go to Beta I think it's a good thing, we caught a testing the memory queries.

H

Can I have a question on that um and I want to make sure I was uh uh understanding? Was the behavior of memory, dot High?

H

What was actually expected by the colonel Community, or was that a bug in the kernel, community and memory.I will match to what we actually had thought it meant, which was you'd, be throttled but not fully stopped.

D

So I think when memory dot high was launched, uh the documentation was incomplete and then eventually uh there were a lot of people who had questions around the usage of memory dot high and that's that's when it was clarified uh by the kernel uh folks, I would say so uh memory dot high is can essentially be used to throttle the workloads, but there needs to be an external workload that should react to uh the throttling basically say it: it can be used in a feedback loop mechanism.

D

Basically, when the workloads are throtted, there needs to be an external uh something external that takes care of either moving memory, dot High to a higher level and sort of uh help in increasing the increasing the resources that the workloads would need. Does that make sense.

H

Basically, um the expectation is memory.I should halt your process, but then because you're halted, you can increase it and you know control the knob at that level or you could have written your own boom killer to do an action at that.

D

Time, that's good, so I was thinking. Maybe when we have uh are you like our user space in killing in place? Maybe then we can use it but uh yeah. It cannot be used for just throttling.

H

Nothing when reaching memory.I is inducing memory reclaim by the kernel at that time, or is it? Is it still trying to do like? Is there memory pressure before you get to memory.hi.

D

I'm not I did not check whether there was memory pressure before memory dot high. But yes, there was memory pressure after it reached memory, dot high and there were kernel reclaims uh beyond that point, but it never uh reached the memory.max level that would have uh resulted in um kills. It always stayed below memory.max. That's why it was just stuck.

H

I appreciate you sharing all that details. I know, there's other issues that um uh at least here with my my red hat hat on where I I would foresee us as we try to start using more Secrets V2 features like the cube Community finding, maybe the unexpected, with the behavior in the in the course c groups, or vice versa. I know, there's an issue right now around um uh maybe some CPU sub behaviors that we're trying to think through ourselves.

H

So maybe that's a good role model on just the importance of testing, um so I agree with murnal.

C

Yeah I think we'll need something like umdi integration here. For this to be more useful and I think I mix it up on it all right. The clarification in the corner documentation only happened like month or two when people started asking more questions about the behavior, so even within the carnival Community, it seems like not. Everyone was on the same page on how this behaved.

C

A

As one of the action items do we want to try postponing and rejection mode caps next cycle unless we have clear approver committed to that like right now we released approvals, but those approach may not necessarily track implementation and give early feedback.

B

D

B

Of the planning right during planning, we should reject more caps unless.

A

Yeah I'm asking if we want to do that, so if, if there is a reviewer, an approver who clearly committed to spend time early in the cycle to review caps PRS, um we cannot accept anything what it will help or would make it worse.

A

We may have we might end up with very few caps, because approval Powers is very little foreign.

H

If we can expand the approver powers after this release as a separate action item, I don't know if there's any interest in folks looking to uh help expand the roster there, but I think anything we do to improve like the social contract, for people is a good thing.

A

Okay, maybe at least we can like.

A

Explicit next cycle, so we we take it as a little bit of uh just like put any name if nobody committed like I, assume, Ronald name very often, and the approaches, if nobody else sign up to review. uh But maybe you need to be more explicit. There.

G

We also need a reviewer right, so not just more approver yeah.

A

J

I also find myself wondering like what is the definition of success here with, like you know, try to the object of trying to reject caps unless they have career. Approvers implies that, like our success, metrics are planning the amount that we feel like we can take on, but also I would consider. Like you know, 17 caps in a cycle is like or other than that, whatever the number is that's lost.

J

It um is like really good and kind of like the best that we've done, and if we had been more conservative we might not have been able to get that much done right so like is it necessarily that bad of a thing for us to have the ratio bad or is it you know? How do we want to define success here?.

G

I agree, I, actually, I, think um I think we did a good job of this circle and we track more racial, maybe drop, but look at how much we merge and how much make a progress. I still think team Community here actually did a good job.

G

And I really worry about it. Just for the velocity become to the big driving here and then destable. Our quality I mean kubernetes signaled on so many components already have the reliability issues right. Everyone heard that and then we are. We are rushing for velocity.

H

That's the goal is to try to maybe attract fewer things and get a closer. You know batting average between merged and track and yeah.

C

H

Think that's good, but I agree with.

C

H

Know everyone here that 17 is a pretty good number like I, feel I feel good about that.

G

I

A lot of background.

A

We will not try to enforce it or we'll just be more explicit on when calling couple hours.

A

I kind of agree like that's: why I ask like, will it hurt more or will it help uh to have clear up over some reviews.

I

Well, the thing to to the point here: Commander was making uh I think we've got two things going on right, there's two goals: one one goal is to have a long-term support, very stable release and another goal is to have all these. You know great new features and that we think are important for future releases and because we're only you know really having one release. One main release we're not doing uh any, you know out front experimental releases and we don't have a LTS release per se.

I

I think because of that we've got this like pulling back and forth between what people want to get in and- and you know what is actually ready and stable right. If you don't merge something you can't, you can't really test it very well.

I

And we have feature gates to try to alleviate that, but we're holding out a lot of Alphas with feature gate default offs. Just you know, out of risk right.

A

I think the police could still really good like everybody. We still have at least two other features from 110 and 112 releases, which uh not being looked at, and nobody wants to easily remove or continue with us.

A

That may be a accumulated check that to accept more features.

A

I think another point I arguing with myself like uh it will start rejecting more caps and considerate on fewer caps. We may end up in our own consensus here, uh but we accepted cap and we try to force ourselves to merge it, no matter what, because we already made a big commitment and even if you see a big problem with that, we still trying to merge it, because there is nothing else in the pipeline because we rejected everything else. I think uh incentive may be a little bit uh off if you start doing that.

F

Well, it's okay web actually can be handled. Probably so like imagine, we made a commitment to to do something like to. We could personally merge it, but when, during the implementation we figured out what it's not what we want, so we can revert well cap, merge and put it back to the under review.

A

Yeah I agree with that I just I think incentive will be to merge it, even if there is a problem that you may not fully be comfortable with. So you can force yourself to be comfortable with that, because there is nothing else in the pipeline and you want releases to be meaningful.

A

So in a sense of a little bit slight uh like a little bit off here,.

J

Yeah, that's another reason. I kind of feel like the definition of successing is important to Define is like you know you mentioned. We have our features that are, you know, have been stuck since, like 110 or 112 or whatever like. If we reduce the set of things that we aim to work on like the life cycle of the Caps, that we've already kind of committed to will extend, because we have. You know this set of caps that we've already agreed to work on.

J

But if we, if we cut down the number, then like those could maybe not make progress in a way that, like over time, will kind of lead us to having Perma betas and, having you know, things kind of languishing in Alpha. So I I, wonder if, like really what we should be aiming for is you know, emerged caps, like you know, emerged like little pieces of work that incrementally move on kind of like what Mike was talking about.

J

um You know the our small little changes um over time, taking off like not as more not more than we can chew, per little change, but like aiming to take on those little as many of those little changes as possible, um because those can like don't as much threaten the stability of the code base, um but also like have little improvements over time.

A

Is there more ideas and opinions on uh these are metrics or success, or uh what went good or bad.

E

uh Excuse me, this is Kevin I. Think one thing that was confusing for me was: if there's no owner on the issue or they're hard to get a hold of. It was not clear to me like what the progress is for that and I noticed as I was working on the cab. Having your kept issue up to date really uh helps the release. Team actually know. What's going on so towards the end of this release.

E

I just ended up uh basically copying the issue and not getting it up to date and that made honestly stuff a lot clearer. But I guess, like an action item, is making sure that the the cap posting owner is available because uh one of the issues I found is if the the person that creates the issue is only able to edit it, and that was kind of a little bit of annoying one.

J

Yeah, that's a good point that was encountered, the swap as well.

E

Yeah I tried to ask the GitHub admins or whoever whatever the organization is called to see. If there was an option- and it really was just uh the owner of the cup should be, the owner of the issue should be at least involved. Otherwise, you will have to kind of create a new issue.

A

Yeah I think uh I, as a chair has a permission to edit the issues so I try to keep them up to date, especially if there is a comment uh issue asking for that. But I agree like you: it's not scaling that. Well, so maybe some automation may help here so.

E

Yeah I didn't even know. I asked on the the signal Channel and then I was I was told that the the real wrong way to do that is to create a new issue. uh I didn't know if who had access to those issues but yeah, it's probably it's probably better to have make sure the issue owner is actually creating it, because it is crucial to the release team of knowing what steps are there and yeah. There are a few items there that I think are important to keep up to date.

I

A couple of items, I thought went really well um won the refactoring of infra on the cnci post, I thought that went surprisingly well, the the other that I thought was went really well. This release more so than prior releases, was all of us help. The signo team has been doing to push down dependency requirements to The Container run times and runtime engines a little bit of a shout out. You guys did a great job. You know helping push us to get the feet, the changes we need in the runtimes and engines.

I

You know sufficiently done for your features, for example the the whole stuff around you know, username space with ID mapping, I thought that went really well, even though we don't run scene release. Yet you know the issue that that the effort was.

B

Mike did I capture that correctly. It's a refactor of cncf.

I

And refactoring sorry about the puppies, the re refactoring of test infra for prowl.

I

Oh, it was moved since yet most times.

I

All right, but I'm babysitting a puppy today.

A

Okay, if there is no more opinions and things to share um I, think extra item would be to better understand what our goal is and uh better planning would be interesting challenge like we need to um be careful, how much things we commit to and how many things we want to take at least.

I

B

All right, so uh that is a retro for uh 128. uh I, will organize my uh notes a little bit after this uh okay I will uh another word practice yesterday.

A

You could even um have uh we have two more items on our agenda today. First is uh Peter.

J

Yeah speaking of uh taking on more things and uh features and stuff, there's a so a couple of what was it uh months ago weeks ago, I don't know sometime in the past right at the uh beginning of the 128 cycle, we're talking a little bit about um you know some reworking of cubelet image GC.

J

um You know it's kind of been a long-standing idea that people had to like have different um schemes, there's two kind of priorities that like putting my red hat hat on, um there's two kind of priorities that uh we kind of have uh in related to the cubic GC and I wanted to see if other people had other sort of perspectives. So like one of the things that we want to push for, is alternate schemes for like garbage collection right now. Part of it is just uh dispersantage space uh and um I think we.

J

We have a use case where it would also be useful to have some sort of like time-based um garbage collection, which kind of implies the existence of maybe a plug-in system which could over complicate things depending if there's like no one else with other kind of ideas, but also, if other people have sort of uh garbage collection schemes that they can imagine, they would want like.

J

Maybe we could so I'd I'd like to in part, discuss that um and then there's like a separate one, which is uh we'd like to be able to separate the, but basically the way that the uh images are mounted or the container.

J

The images are mounted to create the containers to separate the Image store, which we're like also calling the read-only layer from the like writable layer, separate disks, so that we can have a separately mounted image disk um so and I I met with Derek uh and some folks in Red Hat internally yesterday, and it sounds like that. Second Use case might actually be possible, but there might be a couple of things that the keyboard is assuming about the way that um the containers are mounted.

J

That will not actually be the case, so those are like we're gonna investigate that second case, that's kind of like a separate. There are two kind of separate but related things and I think they'll end up, ultimately being two different caps, but I wanted to surface them both because they're, both kind of related to each other um beginning for the first question like for the image garbage collection schemes like, are there other use cases that people have thought of, but people are looking for with?

J

Like you know, uh cleaning up the images are there other ways that other um triggers that people want to kind of see happen in the cubelet for starting the garbage collection process.

B

Images to protect them uh from being garbage collected by kublid, we can do this already on the container runtime I decided we had that PM flag- it's just if we have a more uh I know systematic way to do this. That will uh be better just one um yeah.

J

And kind of the scope of the container runtime piece is more like I want to reserve images that I care about as the runtime, that's less of like uh I'm, a user and I want to Define, like these sets of images are ones that I just never want to garbage collect like those are I mean you can achieve it with that, but like as in terms of like an exposable API for instance, that would it's like they're kind of different use cases.

J

So I can't imagine that that would if we, if we came up with a plug-in scheme, then we could tie in that sort of use case in there. um But it could be like a a plug-in system that ignores certain images and just doesn't remove them.

I

uh And Derek brings up a good point in the chat, the some of these policies. Things need to be multi-tenant based or you know, isolate that one user is only allowed to access this private pulled image. We can't and that's part of that insurance secret, full image stuff we're working on, but without the policy it's sort of hard to.

C

I

Know to match that to the runtime handlers and in the Pod specs we're sort of guessing a little bit too much. um If we had the right policy definitions for security and for caching, you know performance purposes, then then we could do that. We also have a use case in that specific. So far, just a container today because of our snapshotters, we can set them on a Handler basis.

I

You know- and we might be using a different file system and I'm we're finding that the the status of an image is right, now isolated, only to a default file system, not to the actual file system, where it's unpacked okay, so we definitely have some issues there with respect to you know, is the cash policy going to be, you know, pull just the zipped images and then the unpack can go on into end file systems that it can get really detailed right.

I

um But if we have the right kind of cash policies in the Pod, spec I think I think Peter I think we can solve at least the program act perspective at Kublai, um I.

H

Appreciate you chiming in there Mike uh do you remember who I was uh hoping to induce um uh one of the things I wasn't sure when you were pushing for you know or Ensure secret pool images can't be, are always re-entitled, I, guess or revalidated when, when already cashed on a note, was.

I

If you had we're going with the flush access, pretty quick yeah.

H

Yeah, what I was.

I

E

H

Think, through is, if you had heard any uh for those who are using kubernetes and, like maybe highly regulated environments or like different compliance environments. If anyone has had a requirement that says uh this pod, that's using this secret pooled image, the secret pool image must be garbage collected or shared the same fate as the pod, like does the life cycle of the application image need to match the life cycle of the the run time itself. So, yes, the.

I

Answer is yes, but we're solving it with. You know, by pulling it into a different kind of VM as opposed to having it. You know in the Local Host cache.

H

Okay, but but the point being like uh you folks, have heard requirements around image, GC or when it should be done or not done, and if that image was entitled or not where the policies might vary or need to vary. Whether.

F

A

Not everyone's.

H

Taking different implementation approaches that that's cool, that's fine,.

F

Yes, confidential containers, we have the requirement what the image should be dropped as soon as we were 15 Financial stuff is destroyed, so we'll put the purple destroyed.

I

So you're on it there when, if, if we have the right policy set as per Peter, was talking about, then we could also cover multi-tenancy issues with respect to the security right in that policy. For the you know for caching policy, and then you could pass that down to the runtime and we could go ahead and clean it up right right now. The garbage collection is done by Google it. Maybe that's the right place, but I'm, not I'm, not sure yet. I'm, not convinced I.

I

Think for two reasons: one the security information needs to be kept in violent right. We shouldn't be passing Secrets down the credits over the apis I, don't think we should probably come up with the cash policy that says use. You know, uh resolver level key rings only um and then Google. It would know. Okay, I'm not going to try to use the old model of using image, pool Secrets right.

I

um So it's that's! That's one part of the security thing when you clean up the other part is the yes. If, if you pulled it from one person, you have to pull always if you're in a multi-tense scenario. Today- and we can fix that, but yeah it's going to require some more work on that and sure pull Secrets uh I'm. Sorry uh yeah.

A

Thanks yeah well for the same way. No discussions I for one of the things uh we discussed internally. I I wanted to entertain an idea if we can put a mirrors config into kublet. Like do, we have to have mirrors config in runtime, so one of the reason for mirrors config, like the problem with mirrors config in the runtime.

A

If you have pool secret for specific mirrors that you use, you may be want to use this from cigarette uh when you use this mirror, especially if it's some kind of ambient cool secrets from uh like in Google examples.

A

Gcr IO, like you, may want to use your ambient secret to use with GCR Io if it's configured as a meter for specific uh uh image or like specifically registry, um and there are other use cases that may be useful like if you can reuse certain images and stuff like that, and also it will be useful for or end users because, like they don't need to learn how to configure mirrors different environments in different runtimes and the logic is very much the same across different runtimes.

A

So it may be just forced to migrate all the logic, to Google it and make runtime Slimmer and more efficient, like.

I

More like more.

I

More registry murder, routing information in pod spec, as opposed to Less in passing the policy for pulling the image down to the runtime. You go both ways, but then the are you really wanting kiblet to do the resolve in the poll and then push it into uh yeah, a location for the container runtime to use. That's.

A

No API will be definitely more elaborated between uh qubit and runtime.

I

A

Is there any ideas like immediate any direct reactions.

J

Well, I, wonder, like you know, kind of a pattern that we've been leaning into, but not necessarily like committing to is like the opposite direction, where we're moving more uh housekeeping knowledge into the runtime instead of the other way around um and the motive they should be. The runtime is the one that's ultimately responsible for those operations so or so it's like you know closest to it knows about.

J

uh You, know the confidential VMS in a way that the qubit will, like probably never be taught how to know or like not it's not in the scope of that those enhancements. I wonder I do agree that it's awkward right now, but basically the way that you have to do mirroring in the runtime.

J

Is you basically lie to the Cuba and say like, haha like you asked me to pull this image, but I'm just gonna ignore you and do something different um so like reconciling that sort of like weird, like sleight of hand, would be good but I'm, not sure I. Think talking through the details of like who's ultimately responsible for that I think would be um I think we might need to do some more discussion on that.

I

Yeah, it's a tricky problem.

F

Well, user interface of fetching with secret and passing it down to runtime or, like mirror, configuration and passing it down to runtime, it's it. It's probably solvable it's not a big deal but like moving to couplet logic of how images should be fetched or like when teaching like different runtime handlers. What is the specifics to Google it? It's not really a good one. In my opinion,.

I

And tell them why this is important. This is only going to get more complicated and more urgent, as we start doing, artifact support for scan results, or you know, signatures s-bombs requirements that you're going to need to have in the pot in the Pod, spec I think going forward right that it required you know all images have been signed.

I

You can do that with plugins, but again it's starting it's getting more and more complicated. How.

J

Do we deal with the.

I

I

On pod, spec definitions for caching for images, artifacts routing, mirroring all this sort of thing right.

J

Yeah and I think it's especially complicated because we have the split brain mode where, like currently qubit, is where half of the things and telling the CRI yeah so I think it knows, and the CRI is either listening or not depending on like how obscure the configuration is and that I think I think we're really good like with these kinds of more obscure use cases we're going to just keep hitting these walls of like who is actually responsible like. Why, like who?

J

Who is the entity that should actually know about this and who should just follow along um we're coming up against time? I am wondering I, like part of my wanting to bring this up was just to see like who's thinking about this. Does anyone want to be involved in the conversation? It sounds like yes, um the second question would be like what is the appropriate Forum to continue this conversation?

J

Should we have a working group that is talking about it, or should we like just continue in the you know this meeting for a little bit until it seems appropriate? This is something that we want to like. We as, but you know, like my team at Red Hat, wants to like pursue the garbage collection piece, at least um in 129, but um I want to make sure all the other voices are heard.

J

Should we set up a separate meeting, or should we just reconvene next week and you know um see if we can, uh you know, come up with a sort of Direction, maybe with a proposal, that's written out a little more concretely.

A

What worked well with sidecar is separate meeting, at least for a while to teaching distribute tasks to write documents, and once we have more written down, it's easier to bring it to bigger forum.

J

Right, um okay, well, I can investigate sort of starting at least like it, uh a working group in the intermediary intermediate time between like now and when we have to start thinking about cap process so that we can get the we can have larger discussions like this and kind of fine tune. Our all of our thoughts before bringing it to the larger forum.

A

Thank you cool. Thank you. There's a last uh item. I wanted to remind you that sidecar working group is happening. uh We will uh we post meetings but we'll restart them starting next week uh it's 9 A.M just hour before this meeting. uh If you're interested in sidecars Evolution, uh please join that meeting and with that we reached the end of agenda. Is there any last minute notes?

A

Exclaimers? Okay, thank you very much. uh Thank you Raven again for driving the retrospective bye.