Kubernetes SIG Windows, 6 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Windows 20201006

Description

Kubernetes SIG Windows 20201006

A

All right, hello, everybody and welcome to another sick windows: meetup it's the 6th of october and as always, it's a recorded meeting. So please adhere to the cncf code of conduct. uh We don't have a huge agenda today, but let's dive in the first issue um amber privilege, containers.

B

So we have a couple of updates uh since the last meeting we kind of- let you guys know of a couple of the networking challenges that we discovered kind of out of conversation and discussions um to kind of reiterate what those were is that uh in our current implementation, um there is a scenario in which, if a privileged container is added to a non-privileged pod, the privileged container will have the host network namespace, whereas the non-privileged pod will have its own pod name, space, which kind of breaks the uh the same name, space, kind of assumption and kubernetes.

B

um This was something we originally thought that we could uh would not necessarily have too many impacts on alpha. But after some discussions we realized that that was a necessary thing for us to figure out how we can align the namespaces of a privileged container uh within a non-privileged pod. um From that there was another couple of scenarios that came up, which I think deep mentioned the csi scenario and as well as kind of the init container scenario for service meshes.

B

Where, in this exact scenario, where a privileged pod or a privileged container is deployed to a non-privileged pod, it needs net admin. Access to be able to configure certain networking things or csi might require different access, um but not necessarily need um host networking, but there's there's kind of some nuances as to how we can enable that access within windows um with this kind of networking situation and how we align those networking name spaces.

B

So with that kind of investigation ongoing and us seeing what is possible in windows, since there are some differences between what is capable of windows without like capabilities or things that are available in linux and just different designs, we have to go back and kind of look at that investigation a little bit more closely before we can roll forward.

B

The reason why this is kind of blocking a lot of us potentially moving forward at this point is because we anticipate um some significant changes or, as a result of this investigation, some changes in either the cry api or perhaps other layers that we've previously identified as things that may or may not need significant changes.

B

So as a result- and we are pushing it back from 120 to 121, so we're not pursuing gonna get into alpha in 120 anymore.

B

However, we are kind of looking into other ways for us to provide what we have currently um in 120 for people to kind of try and get additional feedback and and kind of figure out. If there's other. You know nuanced scenarios that uh we need to take a closer look at similar to this namespace one um we're thinking about using runtime classes or some other way to kind of expose this to people to kind of work and test out with.

A

Absolutely having some experimental support or actually some experimental release without any support that could work and then obviously, then we can validate certain scenarios around csi with deep and and cluster api uh that ben and others are looking into and kalia and potentially uh identify more more issues that you could fix in 121. yeah. I I was a little bit surprised the other day when I saw the the the the the give up ticket changes, but your explanation and this data makes a lot of sense. Thank you.

B

Yep and we'll we'll try to get that updated on on this experimental release.

A

Anybody else have any concerns or questions.

A

All right, that's moving up uh maz. What's the lineup for 120.

C

Yeah, so I was just uh wondering because we are at the enhancement, freeze and like what we actually landed up for 121. Do we have a list or something.

A

Try to see if I can enumerate them and you guys can correct me uh deep uh csi, we're taking it to stable, uh that that was updated right. The cap.

D

uh So I'm having second thoughts on that, uh mainly because uh we can do that. uh But the main thing is like we wanted to see how privileged continuous is doing, and we should combine that as in like if privileged containers is coming around, should we keep it beta for one more and then once privilege containers alpha. We do all the changes entry, if, if anything, is needed uh to support csi and then release a stable that supports both.

A

uh That could delay things by like almost a few more months right, because if privileged containers might not be in alpha or beta stage for for quite a while, um I I don't know I'm looking to you to determine. Like I mean if csi is stable using the current model and if we're thinking that potentially you might be able to use privileged containers to run the csi proxy.

E

A

Regardless we want to get folks to utilize um to make forward progress, because even if privileged continuous ends up being our our final plan, uh privileged containers being production, great and ready- might not be until kubernetes 123 right.

D

I agree, I agree yeah that that's definitely going to take longer, but I.

A

Was wondering so, would you keep csi in beta for that long.

D

What what's the like, there's a guidance now right around, like you cannot stay beta for more than what is it two releases or three releases? I.

F

Think it's three releases without an api change, so you can publish like a beta, 2 api or update to a beta2 api or whatever um yeah. What that means for something like.

D

Csi proxy right, so that's the kind of other wrinkle in all of this, because all the work that we hope to like that the incremental work will all be in csi proxy. At this point nothing internal in kkk. So we can. We can mark that issue as going to stable for 120..

D

I I think this is where we need a little bit of guidance.

A

I mean so so, if you're, if you're thinking deep, that you should just keep it to beta one more release and wait until some experimental support around privileged containers is out, so you can see how it maps to ssi approximately I'm. I think you know it's not an unreasonable ask. I think we can leave it that way for one more release and see how what happens yeah. I think that's.

D

That would be my preference, and I was that that's.

A

What I mentioned mark, what do you think.

F

Yeah, I'm kind of leaning that way too. I think that since it's not really enabled or disabled via feature flag, it's not quite as jarring for maybe potential users to use it still in the beta state and if we don't have really significant changes to to make now, I'm not sure but but we're planning on them in the next. You know six months might be better to keep it that way, but also um if there are folks who kind of would want to see this, like.

F

I think if we have reasons to believe that promoting this disabled would increase adoption or make like folks just feel better about it, and then we can pursue that way, but other or that. But if not, then probably just keep it in beta for uh 120.

D

Yeah, I think peter posted a question. Does the csi team anticipate any problems? Transitioning kubernetes cluster, with the current csi proxy approach to a future approach using privileged containers? um So yeah? That's a good question. I think there were some thoughts and I think maz had a really good idea that you know we have the existing ap client api.

D

uh We can probably just uh instead of talking to this proxy. We can like make it talk to privileged containers. So, instead of having the same pipe and the proxy binary in the host, all of that can shift within the container itself uh but yeah well.

D

My preference would be to just like if it's just a matter of a single release and by that time in 121 we have a pretty good idea of how privileged containers is shaping up. uh We can probably have a provide a much clearer pathway as part of the move to stable to uh other csi authors.

A

Make sense all right, let's, uh let's, let's call it then um the next next thing is obviously continuity. So um I think all the prs have for that have been merged, so it will go to stable in 120.

F

Yeah, at least uh like all of the test grid results, at least for um windows. Server 2019 are looking really good, and I think most of the work here now is documentation and um I've started to take a look at updating the uh cube adm that what we have in sig windows tools to to have directions for, using that with continuity nodes as well, um and I'm making some progress on that.

A

The next item, david, is a local traffic policy. Gonna make it to uh uh for this release for 120. yeah. It should okay. So is there a cab for that, or is there no need for a cab.

E

um I don't think there's a cap needed, so it's already like.

A

It's already a feature that exists: yeah, yeah, so so yeah since it's a linux feature you're, just mapping it okay. uh Is there anything else from networking? Besides local traffic policy, I know there were some dsr enhancements that your team was looking at. I don't know if there's anything that will that will be called out for 120..

A

No, no we're still.

E

I don't think there's anything official like scheduled. I.

A

Thought there was some work to support more low, balancers and, and some other work is not official or not scheduled.

E

uh That was the dsr work, so that has already landed.

A

No, I thought that there was still more work left in dsr to support more low bandwidth or do more comprehensive testing.

A

uh Maybe I'm misremembering? Okay, no, no worries, okay, no, I'm not sure yeah. I'm not aware of that.

G

Oh david yeah, there was a pr where betsy was trying to toggle the dsr feature for the other load bouncers. I don't know if she has since merged another pr. Okay,.

E

I'll talk I'll sync up.

A

Yeah yeah so yeah. I remember that we were trying to do some dslr enhancements. Okay, so uh anyway they don't need a cap, but uh so basically I'm gonna call it the networking work uh no kept needed.

A

Yeah like I, I remembered basically what kalia said as.

A

A

All right, um and then are we uh I'm sorry like I haven't followed as closely on that and on the cluster api, so uh kalia and others are you working on that. Are we gonna be able to claim any alpha support for cluster api, for windows, for azure and uh for this release.

H

Yeah, so um I've got it mostly working. I've got a pr for the image builder. I can drop in the link here and then we also have.

H

A cluster api that I can drop a link in as well um both of those um are in progress. I think the as far as the uh like cape goes. It's I think, we've got mostly agreement. I just have to do some final updates. I was out the end of last week, um but uh yeah I'm working through one last thing, with uh inter node connectivity for um uh for using flannel.

H

uh It looks like our our docs upstream uh you're able to deploy windows containers, but when you can try to communicate across nodes, it's not working. So I'm looking into that and once that's resolved, I should have those pr's ready to go.

A

Cool thanks james, so should we uh comfortable is a cab merged that it will go to alpha.

H

Yeah well so it doesn't uh apply to the same life cycle as kubernetes, so um yeah. I think we have general agreement that it's in pretty good shape and should merge in the next week or two.

A

Got it? Thank you.

A

All right, I'm not aware of anything else, that's going in 120. Is there anything anybody else is aware of.

C

All right moving on next item uh zinc, uh mike michael there were. Are there any test changes? I know that that's infrastructure, claudia and elena working on should we be tracking that, as.

A

Well, I don't know if we should be tracking the test changes, at least not through the cap process, um but uh definitely track this deliverables that we're planning to to to do. Okay, okay, sounds good all right, so you can talk a little bit about the the privileged containers and volume access.

I

uh Sure so this is a question related to privileged containers. So for css proxy right we have um disk uh api volume api. The main task for them is to access those volume or this host. For example, we get volume with certain volume id and then we can format or get status from it mounts green bundle etc.

I

So for privileged container inside of container right, even though it is privileged, can it um access those volumes uh disks uh on host right, let's say get a list of volume on the host get a bit of disks on host?

I

Is it supported because the the main purpose of csi proxy is doing on those kind of apis on behalf of the legacy as a driver?

I

Even though provision container has privileges, but does it mean it can legacy directly the disks or the volumes that is on the host.

B

I would imagine, sir, but I I think that's something we might have discussed in the cap as one of the comments as well.

I

Okay, so um initially like we, I'm more focusing on the file system set, but actually that's not a problem, even though, even if it is not a privileged container and with host pass um container can see the host directories but like um get access to the disks or volumes on the host is actually different.

B

Yeah, um I think the only thing that I would say now is like, if we're aware, if job objects can do that, then this would be possible, uh since the only thing that's kind of happening in the privilege container scenarios us enabling job objects as a type of container like essentially I'm, if that's within the bounds of job objects, which is something that I'd probably have to go back and check. Then I think this would be addressed too.

D

So, just to clarify.

B

D

Be some way for say, get volume or get disk to for the container to report its own volume and own disk that it's seeing in the container context, as well as everything from the host as well. Right.

F

I'm not entirely sure that the container like that the container will have context. That's separate from the host in this case.

A

So what you're saying, if you do get volume you'll.

F

Get both your own as well as a host, possibly, I think we need to confirm, but since it's just running as a like an entity, job object on the host, it shouldn't be all that different.

D

ah So you're saying mark that the container won't really have its own os volume at all to any.

A

So job objects are not it's not so I we used to work a lot of job objects back in the day, so george is just another process on the host with specific privileges and okay. So it's not a container. There's no encapsulation.

D

A

C

Yeah, that's that's correct, that's correct, michael, michael and and as far as the that goes to amber in the goals, we have explicitly called doubt that um the privileged containers will you know, um providing a method to for privilege container to host to access host resources, including host network service devices. This and under disk. We have host paths and volumes. uh So I think that's one of the goals for privileged containers.

I

Great well, it doesn't need uh like um special api or it's just. Let's say you get inside of container right now, right you you that volume gun disk, you won't see the host once and if I want to get the host ones, you need to separate api or separate commands to do that, and also I need to like format, ether.

F

I

F

Kind of, for all intents and purposes, the privileged containers will act just like any other um process that is running on the host and should be able to see the disks on the host. um I think we can try and confirm that, with some of the sample, um the prototype code that that danny carter cantor had, um though so I I think, I think what we're thinking is. Yes, I think we expect it to work that way. We just need to confirm yeah.

A

So, let's, let's take this offline um and and uh amber and mark if we can have someone try it out in the next couple of weeks. That'll be great.

A

All right david: do you want to talk about this dslr reporter problems and regression.

E

Yes, so there was a regression in the 8c and 9b patches on windows, server 2019..

E

um The issue is that basically anytime, the hns service gets restarted or gets rehydrated uh a vfp rule gets dropped and this uh dropping of the rule may cause the pod to service traffic to fail in some cases.

E

So um this issue is specific to windows server 2019, so you can continue using the latest patches on windows, server, 19, h1 and above 1903, and higher this issue will be resolved in the 10c release that is coming on december 2019 later this month, and uh you can. If, if you suffer from this issue, you can uh work around it, um it's not really an optimal workaround.

E

The probably the best solution would be to avoid installing hcn 9b, all together on kubernetes on windows, server, kubernetes nodes, but the workaround would be to pause, basically uh pause, cube proxy and remove the uh the policy lists and then restart.

A

Your proxy or restart windows is that a kb article from microsoft on this. I think that that would probably be very useful to point people that hit this or.

E

I don't think there's a there will be a this will even should be mentioned in the 10c release that it was fixed. I don't think it is mentioned that this is a a regression in the existing 8c and 9b.

A

So so, if, if we have someone that hates it possibly might work on having a kb article, if nobody hits it and nobody talks about it, then we can just silently wait till end of october when it gets fixed, but people. If people hit it very likely, you might need some form of kb article or something to point to people.

A

If it's not a kb article, it could be a sorry, a github issue on the github, slash, microsoft,.

E

A

So we can, like you, know, articulate it there like uh like on the wiki or something it doesn't have to be yeah.

E

I can I can file an issue and link it in the um in the sig windows notes.

A

Yeah that'd be great.

E

B

A

Wait to file it after someone hits it if nobody hits it and we haven't heard about it at all, then I mean I'll file it anyway, just okay, so cool all right.

C

That's a good idea: uh yeah thanks david.

D

I I agree with filing it, because I've already had internal inside microsoft inquiries about people trying to use that release.

A

Okay, so so yeah, okay, so david, let's file an issue and then something here. Yes, thank you all right. Last item on the agenda. uh Graceful notes, shot down mark.

F

Hi, uh so I linked the cap that is going through uh signode right now. I believe that it got merged on friday.

F

um This kepp is kind of was specifically focusing on linux scenarios, um but after I was reviewing the cap, I think this could be applicable to windows as well, um and I commented in the pr about the windows apis needed to light this up, and it was suggested that that I just um make a follow-on pr to this kept, which I should be doing hopefully this week.

F

um But the point of this kept is that for linux, nodes that are enlightening, the cubelet um via systemd inhibitors, currently to be able to detect when um the process is being requested to terminate and then um delaying that for some time and then issuing graceful um and then gracefully terminating pods that are running on that node. To just better have continuity when nodes go up and down um windows has similar apis for um delaying shutdown.

F

You could probably see that if you leave like notepad open with an unsaved document, um it's the easiest way to trigger it. um So, as part of the discussion for the cap, they've agreed to kind of make that to split up the os specific implementations out when the cubelet um changes are being done, and um I just wanted to raise awareness because I think that'd be um pretty cool to if we could get windows. Support for this, um and also I was just.

F

This might be a good um kind of opportunity if anybody is interested in um making some changes to the cubelet um with some kind of windows- specific uh knowledge here. So I just raising this for everyone's awareness.

A

How does this interact with the cloud provider interface and some of the cloud providers? Are they basically gonna utilize this for for basically setting the signal to the cubelet that the shutdown is coming? Is that kind of the workflow that is expected.

F

I'm not exactly sure, with that. Most of the discussions were focused around once um once a shutdown is actually like, initiated I'll have to double check those parts of the cap, but there there were talks in there about uh automatically triggering cordon and drain of the nodes.

F

Once um the keyboard is aware that the shutdown's coming.

A

I think that's kind of I think I think I think that's kind of reverse, though right, because we expect the cloud provider, uh like you know, whatever your is you're running, underneath to kind of drive that execution of basically cordoning the nodes and draining them and that will instruct the cubelet to start executing the action around the graceful, node shutdown. So not the other way around like. I don't expect the cubelet to start that process.

F

Yeah I'll have to review the cap, and but anybody else can do, and those are probably good questions to add.

A

If you can add them, that'd be great mark.

F

Yeah yeah I'll, add those if it's not already addressed when I um update with the windows apis god. Thank you.

A

Yeah, if anybody's interested please uh reach out to mark, I guess.

F

Yeah or just comment on the issue or kept directly that's right.

A

um Well, we have about a minute and a half or two left any other questions or concerns anything that anybody wants to bring up.

I

A

Go ahead, go ahead.

I

I just want to check like for privilege container. The current plan is 121 for alpha and how about beta, so it will be like next 122, etc.

B

uh Yes, that's the current plan.

A

And usually kubernetes asks that we stay in beta for two release cycles. We can see if we can get an exception for that and go to ga earlier. If it's warranted right now, obviously will have to see how how everything is working out.

C

uh Michael just a quick question, and maybe embryo, but just adding the windows cap- that pr is still under review. I know we have enhancement. Freeze. Does this pr need to be merged before today, or which, which pro uh just adding the cap uh windows kept to the to the to the cab directory? I just pasted the link to the to the pull request.

F

I think we should be able to merge the cap in a provisional state after the enhancement freeze,.

A

Oh, this is a privilege yeah, I mean if we're not going alpha this this release and this there's no there's no urgency here right. Well, we still want the.

C

Cap to be there probably right like with that.

A

Yeah, so we can I'll we can get the cap in. I think okay, like I want to make sure that the cab calls out that this is 121 right.

F

Yeah, I think that there's some comments with the the enhancement leads about this and we can always reach out, but I think the deadline is a lot of it's so that the enhancement leads can start um knowing what they need to track and what they don't need to track. I don't think, there's anything stopping us from checking in a pr or for emerging of the pr just to get the dock in there. um No, it.

A

Should be explicit, though right, it says, target 120 or 121. We should change it to be explicit.

F

A

Otherwise, one of the comments.

F

um Was we need to add a cap.yaml there? Recently? There were some changes to the cap format, um and that should say that should be explicit about what release we're targeting alpha.

A

Okay, who's gonna own doing this amber is that you yeah that'll.

B

A

Sorry, cool yeah just tag us once you're done and we'll see about getting any merchandise, but since we don't have specific uh milestone for 120, I mean, if you can merge tomorrow or the day after it should be fine. I think very cool all right, everybody all right we're out of time. Thank you all for attending, see you all next.

A