Cloud Foundry CF on K8s Forum, 28 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CF for K8s Working Group Forum 28 June 2022

Description

[KB] How to proceed with Pod Security restricted level and kpack?
Workaround with webhook selecting label “kpack.io/build” and setting SecurityContext: https://github.com/cloudfoundry/korifi/pull/1284/commits/4008001b1e8b68744a9e7cf25457d52d53d4f0d1. Is this acceptable as a temporary solution?
Kpack issue: https://github.com/pivotal/kpack/issues/972
[RI] Some website changes were proposed. I will follow up on them. Thanks for keeping an eye out on the content!
[TD] When should we do a 0.2? A few big changes like HNC removal
[TD] Proposal for an initial labels and annotations track Label/Annotation Support in Korifi
[TD] Want to start talking with the App Runtime Interfaces WG about some changes we’d need for this. Draft Proposal: Support Instance ID / K8s Deployments in Korifi
[DB] K8S version support policy (slack question)
https://kubernetes.io/releases/
[JH] Draft (incomplete) proposal for CFRoles: CFRole Custom Resources
I’m still working on this but if you check it in a couple of days it should have an actual proposal in the proposal.

A

B

All right welcome everyone to another cloud, foundry on kubernetes working group forum. I will be linking the document that has all the discussion topics in the chat.

B

If you would like to talk about something, you can either drop it in that document or you can drop it in the chat and I can add it for you.

B

We have a few topics already, but we can always add some more or you can add them into future topics if you'd like to discuss them another week. So with that said, let's get rolling with our first topic, which is how to proceed with hot security, restricted level and k-pac.

C

Uh-Huh guys, just in time sorry.

C

Yeah, so what's the situation on this say it summarize the problem um so we're previously using pod security policies to enforce the security on our workload. Pods um that's been deprecated since kubernetes 122 and it's going to be removed at 125 and 125 is coming in in august, so the clock's ticking. Now um the replacement is something called pod. Security, uh psps work by creating a a role that links to the pod security policy and then the user that runs the pod needs to have a binding to that role.

C

So it works the user or service account level. uh Pod security is loads, better you've got three standard levels of security and you set them at the name space level.

C

So that's privileged baseline or restricted restricted. Stops you becoming root, doing privilege, escalations, enforces a couple of set comp profiles and forces you to drop all capabilities and only add one that allows you to bind to privileged ports. So it's all sensible stuff.

C

And we want to run workload pods using the restricted policy, and we've got a problem that our kpac build pods also created in the workload namespace. So they also have to adhere to this policy.

C

So we've been we've just done the um the work to switch to uh pod security. It's all fine, apart from these uh k pack pods, and we need them to go and set security context on all the containers uh inside the pod.

C

um So yeah we contacted kpac a couple of weeks ago created an issue and they seem to be up for creating um pods.

C

With these security contests set, they want to put a few little config capabilities in there so that you can specify a different level other than restricted inside the build resource or potentially in the name space, but it sounds like they wanted to do a bit of design on that and see how it's going to turn out um so we, but as we were right at the end of the work, we've done everything apart get apart from uh getting all these to run.

C

uh We just did a little hack to see if we could uh write a web hook that added the security context to the kpac things. um So we've just done a pr for that. There's a link there I saw tim, you commented and said- could be a bit more. Could we be a bit more specific on what pods we're targeting so yeah? We could look at that, um but yeah after that. So the question is: how do people feel about that web hook?

C

Because it's, I know it's pretty grim, targeting core resources like pods and what should we do with kpac?

C

um I haven't contacted them again yet because I'm not sure what we should be doing like we could suggest that we could do the work and pr and everything, but I don't know how people feel about that because I don't know limited resources at the moment. It's only me and danny on this side, um so we'd have to stop all the task stuff to do that um yeah. So any thoughts on that.

D

um I think well that will that's good context. That answers my question. I think, on the my top level question on the issue um like I, my personal opinion is like I feel like the web hook is a little gross but like if we can restrict it to only the builds for that like we're, triggering, which I think we can.

D

I think we put like a build guide label on those or even like app good and stuff to get logging working like we could leverage some of those things for that, then it's probably fine as a temporary thing.

D

I think we should reach out to them about contributing it because, like even though like on on the mia side, you have like limited bandwidth, like I think um folks, like like we could just open up to everybody like on this call and like on the american side, like some of us could probably pr that I think it's like fairly well understood.

D

It's mostly just working with them to figure out what the configuration interfaces should look like right yeah, um so it might be worth just like reaching out to them again on that front, because I think I think they're in agreement that it should be done, but they're also in a place where they have limited bandwidth too. So it, I think, that's the main problem. We're hitting yeah.

D

B

Yep, I think that makes sense, uh definitely spread the burden a bit, especially since our time zone uh for those of us in the north american region probably aligns a bit better with the kpac team. So highly appreciative of the exploratory work you've done on that, and we can definitely coordinate moving that forward so that it's not so difficult um from your side. Given the constraints of time zones and having limited bandwidth so yeah.

A

B

Try to get that moving forward, um maybe open up in slack in our dev channel and then also uh go direct to the kpac friends and work with them because you're right, that's the kubernetes versions march on and we're we're getting there sooner than later. So definitely good to work on cool.

C

So and that pr is probably okay to go if we address the um the label thing.

B

I'll need to look through it as well just to have context, but it sounds like we're on the right path, so yeah just tidy up any comments and make sure everyone's cool and we'll go from there.

C

Yeah, we won't get a chance to change it till tomorrow, but okay, cool.

B

Yeah, thank you.

B

uh If there are no more comments on that, we have uh some website changes request. I think ron says to follow up on them, thanks for keeping an eye out for content, uh any thing to add on there from or is it mostly just a signpost there ron might be away.

E

Yeah I just wanted to uh mention that I mean thanks for keeping an eye out. That's like a a big thing. I was aware of the upcoming hnc changes. I just wasn't very sure about what we were changing to them.

E

The pr I think hasn't been merged. Yet so I'll speak to like the folks that take care of our website and have that live and updated before the end of the week.

E

um I think there's some automation that should just kick in and help us cover the release, part of it, but I I have to make sure, though, but yeah I'll do that, but in case there's other things that folks want to include or change or you know just feel free to- I guess open more pr's and we can find the best way to orchestrate the changes into the live thing. So yeah, that's what I had.

B

Awesome yeah! Oh sorry, I think I stepped on someone's toes go ahead.

B

Okay um for context, uh some of those changes that we have going through into the documentation website are the big removal of hmc, which was a big effort to sort of streamline our process and remove some dependencies we had so um that was sort of the impetus for that.

B

So uh moving on to the next topic we have, uh when should we do a 0.2, and this also calls out the few big changes like the hmc removal.

D

No, no, my comments. It's just like donut out there.

B

Cool um yeah, it's always hard to determine what point is the right point for a release and I don't think I have a straightforward answer for you right now. If someone has a, you know very clear mark to put forth as a suggestion. Please let me know uh probably should do it sooner than later, but we probably need to tie up a few loose ends.

B

um It's probably an ongoing discussion that we probably have in the creepy dev channel.

B

But if anyone has thoughts, please speak up.

C

I mean it should just be a case of you know. Ci is green. We can uh cut it whenever we like, I mean it's. The hnc thing is a good reason to do it. We've covered that edge case with deletion now as well, so I I wouldn't say any reason not to press the button. Now, apart from georgie and giuseppe not being around at the moment,.

C

It's true: when are they back uh gyorgy's back next week and giuseppe the week after that? I think.

B

Okay, so if we don't feel we should release before then then, maybe when all our friends are back, we can release it so that everyone can see it go out the door.

D

Yeah, like I don't.

A

D

There's like a big rush or anything, it's more just a question because, like yeah, my understanding was uh what kieran just said, which is like: we've got the automation there and it's just. We just pick a date and make sure we don't break it on that day when we cut it.

C

Cool well, it's tied to the um pipeline as well. Now, so we can, we can cut it from the last green as well. Well, you can only cut it from a green one.

B

Good, I like the idea of only being able to kind of at least somewhat assuredly working release um cool all right. Well, we'll keep monitoring that and when the right time uh occurs, we'll just go for it. um So moving on to another, one from tim is a proposal for initial uh labels and annotations track.

D

uh Yeah, um so this has come up recently, as we've had like more people, try out, karify and and stuff um there's so cloud foundry already supports this concept of labels and annotations on most of the endpoints, which is conveniently the same as the kubernetes concept.

D

So this is a proposal about like a track of work, to start allowing people to actually set those on the core um like cf custom resources, so that they can like show up in various gui's or or things that require, like a label selector to be able to to grab something. um The other thing about this is like, as we were thinking more about it. We're thinking like annotations could also be like a better like front door for like enabling some experimental features or like additional config properties on on our things.

D

So, like an example, is the cpu um requests and stuff we were talking about a few weeks ago, like there's no way to set that through the cf apis, but we could, um because the cli lets you set labels and annotations on things. We could let people like set like a cpu request, annotation on on the process and then like use that or something um so.

D

This is just uh a proposal like I want to start getting feedback on it because it it should be a fairly like low scope track of work. It's basically just supporting some additional um patching points that the cf api already supports, but it is like fairly widespread, so like every resource supports this thing.

D

So if anyone wants to look at this, that would that would be nice.

B

Cool sounds good um if it has not already been linked in the curfew. Dev channel, I would say, drop it in there test. Let's set feedback. Take a.

D

B

At it later, uh probably at this meeting, so thanks for putting that out there all right uh moving along, uh I want to start talking with the app runtime interfaces working group about some changes. We'd need for this, which is a draft pr to support instance, id.

D

D

Sorry, I got my terms wrong because the draft proposal um so like we, we started talking about this, like previous working group meeting, maybe two away about like what would it take to stop using staple sets.

D

um So I I went through some of the work um you irene folks had done like a year or two ago about what this would look like in cf for kids.

D

It's actually a lot simpler now than it was back then, because, like we, no longer have to worry about logger gator, like um cf metrics, like the diego protobuf compatibility stuff, like that, we mostly have to just worry about the cfcli um being able to display like or let people choose like non processes with a non like integer index based identifier.

D

um So this this is like a draft proposal um just like talking about the current state of things and like how we could leverage the diego um lrp instance guid as an alternative identifier and the changes we'd have to make the cloud controller and the cli to support that. And then once once those things can support that. Then it would be a lot simpler for karify to like allow people to opt in to like alternative like deployment strategies like deployments, um and maybe they could use an annotation or something to do that.

D

That's that's a side question. So this this one's a draft, but I know like a lot of folks here- have historical context on it. So it'd be nice to get opinions on it and stuff because we'll we'll need to get buy-in from people in the app runtime interfaces working group to be able to like either get them to make these changes or like contribute them ourselves. So just wanted to get this one out there. Sooner than later and and like bird rock said I'll drop links to views on on slack too later.

B

Cool yeah thanks so much for putting that all together um definitely want to move towards a world where.

A

We can have a.

B

More uh flexible interface we can plug various bits in, as we see fit so definitely worth working with them all right. So we have another topic for a kubernetes version. Support policy that someone asked on slack.

F

Is the question asking basically what version of kubernetes do we support? We? I don't remember discussing that. I can't even imagine how we can support incompatible, kubernetes versions as well, but that's probably another story, but maybe we should have an agreement how or how old kubernetes we want to support.

B

Yeah, I don't think we've really discussed this. I think the running assumption was just whatever the the current supported main version is that's kind of what we're operating on. um Definitely don't want to be in the business of trying to run on end-of-life versions, but I think probably a couple version spread is a good target, so maybe like the the newest, stable and then one back from there.

B

It's probably a good starting point, but I'm happy to have it.

F

So this question on slack says that it just supports 120 2.9, which I think previous is 123, which means that this is even older.

F

So does it mean that we would not be able to run flawlessly or make ikeas.

G

I guess it's kind of a question for the open source of what is support mean like if we're not testing it, I'm not sure what support means so.

G

Like we know what version we test- and we can say our test pass, but they're not buying support from us. So what is support right support, maybe means like we're, not gonna close your issue out of hand when you raise it, but uh I I think basically, we shouldn't really be claiming stuff works that we're not testing.

D

And fall on to that, I don't think we have the bandwidth or necessarily like the funds to do fan out testing for every every version of kubernetes either. Like I, I feel like my like personal answer to this question would be like we aim like we soften the language. We aim to work on like any like the current non-end of life, like this, like kubernetes, supports like the current and the past two versions, which is like less than a year's worth.

D

To be honest, so I don't think it's too bad like we could aim to like not break on those, but.

C

D

Said we can't test them every permutation of them.

C

They're slightly out of sync, with the um the hosted solutions as well. If you go for a stable gke, you get 121, which is um not supported, and it's, I think, we've only just been able to get a 124 via gk in the rapid channel, so they lag behind a fair bit um yeah in terms of what we're doing with kind testing on 123 and gke testing on 122.

C

In arena we used to yeah, we did fan out and try the three different channels in um gke, but that's just an expensive thing to be doing.

G

Yeah, I wonder, is there any way for us to put that version somewhere where it ends up in the readme or something, but the versions that we're currently testing into the readme.

G

um Wow, that might be maybe.

F

These notes, this release has been tested on that release.

B

Yeah dude no guarantees, but just saying hey we tested on these. This is what we found that we we have ourselves a little bit of maneuvering space.

H

We had some precedent or policy on what kubernetes versions we were deliberately and coincidentally, testing against with cf for gates might have to look it up. Burdock. I don't know if you remember what that was.

B

I don't recall um I could look back in our ci and try to figure out what that spread is yeah.

H

We might have said something in the cf for kids repo as well about what our intent was at least.

B

Yeah, so we yeah, we can definitely try to pick that up and see if we can hear that.

B

But I do like the idea of especially at this point not guaranteeing compatibility with versions and just saying we've tested on these that way, if they, if the kubernetes community makes changes that are significant, that break things like the pod security policies and stuff like that, then we have a little bit of room to say: hey. You know we'd like to support this, but it doesn't work for various reasons.

D

Right and like we can like welcome issues for previous, like non-end-of-life versions and fix them. It's just like you can't test them all type of thing.

B

B

Yeah great uh great question: thanks for bringing that up um little there's some more digging in discussion, I'll see what I can find and if I find anything interesting and see if kate's that's concrete, I will drop in the slack uh to sort of further that discussion.

D

There is a supported version, supportivecaseversions.yaml file in there. Maybe that's that's it. I don't know.

B

um That hasn't been updated for a little bit, so take it with a little bit of salt.

B

Cool all right, unless there are further comments on this five four three two one gone.

D

B

Have uh another entry, which is a draft proposal for cf roles,.

G

Yeah I was hoping to have this a little bit more evolved before this meeting, but so it's still pretty pretty uh lacking in the actual proposal part of the proposal, uh but I should have more details in there in the next couple days. But uh basically the idea is just making it either a custom resource or a property on orgs and spaces to uh have a an abstraction for cloud foundry role: membership um in order to kind of solve some of the the ugliness we have with managing the kubernetes roll bindings.

G

So, anyway, you feel free to take a look now, but you'll probably be more satisfied. If you take a look in a couple days,.

B

Cool all right I'll, uh keep an eye out for that and uh sort of try to offer some feedback to help along yeah.

G

Feel free to anybody to pipe up, even though it's not done yet.

H

I think we might need to have comment level access on the dock.

G

ah Yes, well, that would be a good thing to do.

C

All right, let me see.

H

I wonder if google changed some defaults globally, because I've been noticing more and more cases where the default seems to be viewer, and then people can't do anything.

G

Oh no, I I mean it is the default, but I also just I just did it and I didn't do a very good job. Okay, you should have comment level access now, cool.

G

I can see all sorts of anonymous whoever's popping in.

B

Looks good all righty, it looks like we are at the end of our list of topics. If there's anything you'd like to sneak in at the end here or bring up that we wanted to discuss next week at our next final weekly meeting. Please do so otherwise I'll, let you all go back to whatever you plan on doing, depending on where you are on the globe or what your schedule is thanks. So much for attending great discussion, one.

G

One one one last thing: I think we have an outstanding pr for the karifi website to remove mention of hnc.

G

B

um Rom is going to be looking at that and working on trying to get that merged and then finding out all the machinery that goes into deploying that to the website. So definitely we should do that. I think it's being tracked uh thanks for the reminder, I'm gonna let rom speak since I saw the microphone mute, go off.

E

Julian, that's on me um I'll, follow up on that and um keep folks updated on slack about what happens.

G

Right yeah, I think the changes are pretty trivial, but uh you know from from the looks of it but great. Thank you thanks for looking at that.

E

Yeah, it is a trivial change. I think the big thing for me is it's the first change, and so nobody really knows how that works.

H

Yeah, I think we've been a little unclear about like you know if this gets merged. How does that relate in this getting published? But I thought there's some discussion.

F

H

Actions or something on that pr.

B

Yep cool awesome. Thank you for following up on that julian and rob um we'll figure it out. It's always interesting to see uh with a new process how things work out yeah.

B

So uh thanks everyone for coming, offering your thoughts, great discussion as usual, and I look forward to seeing you all again in about two weeks so I'll catch y'all later.

E

Thanks everyone bye.