Cloud Foundry CF on K8s Forum, 16 Nov 2021

Previous Meeting

⏯

youtube image

►

From YouTube: CF on K8s WG Forum 16th Nov 2021

Description

Recording from the CF on K8s Working Group Forum call held on 16th November 2021.

Topics
[TD] HNC propagation woes making it difficult to stage (and probably run) apps
- https://github.com/cloudfoundry/cf-k8s-controllers/issues/243
Service accounts / Secrets don’t propagate. This is an issue with Kpack.
Disable SA propagation and have shim create SAs?
Lends weight toward CRD abstraction for Orgs/Spaces.
HNC webhook is also aggressive https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc/config/webhook/manifests.yaml#L95-L116
Needed to monitor everything under nested namespaces
[GC] Binding Service Accounts
Users vs Service Accounts -- perhaps add a defaulted field/parameter on requests that would allow end users to create service accounts. This should maintain backwards compatibility with current CF API.
Create Role https://v3-apidocs.cloudfoundry.org/version/3.110.0/index.html#create-a-role

A

Hey good morning, good.

B

A

How's it going.

B

You're all right, apparently.

A

We're already being recorded.

B

Yeah the recording starts as soon as someone enters the room so.

C

Oh interesting, okay,.

B

And then uh ron, let it sit down later to the pertinent portions, okay,.

B

Grab the document link and drop in chat periodically.

B

We've got one from tim cool tim's here.

B

Give it maybe another couple minutes a lot of people show up, but if you have additional topics you'd like to add, please do so.

B

B

So we're just giving a couple minutes for anyone else to show up um I'm going to drop the link to the document and chat again if anyone would like to add a topic to discuss, probably going to be a little light on topics looking at it so far, but always good to have the forum and then we'll take whatever time. We need and then go on our merry way.

D

Do we know if uh giuseppe or georgie is trying.

B

I do not ah but looks like we got someone uh at least new to me.

B

Welcome john thanks showing up I'm again drop the link to the document of topics in chat. So if you have one, you would like to add feel free to do so. Also just sort of have an open floor. If you wanted to talk about sort of anything in general, so we'll get started in.

A

B

B

B

All right well, if anyone else shows up, then we can bring him into the fold on the fly. But uh thank you. Everyone who's here for again attending the cfan cates working group forum, uh we're we're just trying to provide an open space for everyone to discuss the various topics that are germane to the cloud foundry uh on kubernetes space. Oh, and we got one more welcome, two more okay I'll, be dropping that uh chat uh document link again for everyone to add topics feel free to add them.

B

I think we can have our first topic rolling with tim on hnc propagation.

E

Yeah um might not be interesting for many of the people who have just joined and some of the people. I want to talk with, aren't here so I'll, give a brief update here. um We're planning on using h c to which is higher the hierarchical name, spaces, controller to propagate um service accounts and other objects um across the namespaces that are created with cf, create space and cf create org, and, as we've tried to use it in in earnest.

E

I've found that, like its ability to propagate service accounts is, is a little wonky, and that is because um service accounts themselves are are reconciled by this core kubernetes controller called the token controller which generates secrets for them, and the secrets need to like be in every namespace in which they exist and hnc has like hard-coded, um some some logic to avoid propagating those secrets, because I think that would actually break things more, but it also makes it like difficult for us to actually propagate service accounts with this.

E

um So I was bringing this up here to see if any of the um the folks over on the umea side like had more thoughts here. But in my opinion, it's like kind of like not looking so good like I'm, not sure if the benefits we're getting from agency are are super useful if it's like, if we might have to like manually, manipulate these and propagate the service accounts themselves.

E

um I also like was surprised to see how aggressive it is in terms of permissions like it. um It sets up a web hook that, like validates on on the wild card, object type, so literally every every creation or update or anything of any object on the cluster has to go through h and c, um which is also perhaps not ideal.

E

um So tldr is, I might put together just like. I don't know a quick little thing about like perhaps not using okay, we got some some folks.

C

Hey sorry, the we had another meeting that ran over with the supposedly new garden team, so.

B

All good welcome to uh the party we basically just got started. Tim was explaining some of the difficulties he's been having with uh agency propagation.

B

um I don't know tim if you want to try to briefly summarize that, because I think they might have some context. Yes,.

E

um The tldr of what I just was talking about was um it I'm not positive. The agency works very well with service accounts, and I think that might be a an issue given how that's like our primary use case for it, um I'm wondering if, if y'all drink, your explorations on it like got it to successfully propagate them such that they are usable or like, if I'm just doing something wrong with it.

F

um We weren't really doing service accounts when we were exploring it we're looking at role bindings generally so- um and I just had a quick look at the code earlier and saw the whole um secret the code for the secrets as if it's a service account token, then no, it does not propagate and there's no way to override that.

F

So I see your problem there.

F

It's, I guess the other thing is just so. What is the setup for having to have service accounts in every single um name? Space, so is, does that mean you've got controllers that are running like one per name, space or.

E

um No, it's a it's a requirement from kpac, um so to be able to stage apps in in the appropriate spaces that line up with them. You need a service account for kpac to use and I believe it runs the build pods with that service account and that's how it lines it up with the image, pull secrets and everything.

F

Just trying to think how, like irene, used to cope with stuff like that with dynamic? Well, it didn't really have dynamic namespaces. So did it. I guess that was the thing you knew them at yeah install time, and then we just gave permissions across them and.

F

Yeah, so nothing explains, to my mind there on what we can do about that.

C

Does it mean that we need to like the what we? The outcome that we want is every every space namespace has its own service account.

E

C

E

For for capex sake, yes, um and the reason why is we, we really want to be able to run the staging pods in the the same name, space as the apps, or at least in discrete name spaces um primarily for log output. Reasons um like like we want the same r back to be in play. That is normally there like for staging as it is for running, um but also like yeah. I guess this is the same concern for it's a tendency, related concern.

E

um Yeah sorry, maybe.

A

We should be setting up the service account, that's needed for kpop when we create this space.

C

Yeah, that's what I was thinking like.

C

Maybe we should just because there is. There is a specific reason they don't propagate, which is, I think, the comments in the code basically say if we, if we propagate this, would be basically working against the kubernetes controllers, and we don't want to do that so that kind of suggests that we want to achieve the same outcome differently in a way that doesn't like go against the grain of what kubernetes usually does so.

C

The first thing I was thinking was exactly that, like we just created it's a bit annoying because like if um and if you use the the shim, then I guess we could add that to the shim and just the shim just creates a service account every time possibly like if you want to. If you don't want to use the shame, then of course it becomes annoying, but we've already discussed having crds for orgs and spaces. So maybe that's where yeah that's one extra point for going in that direction.

E

Yeah, I think so I think.

E

I think there will be value in that, like I, I was also surprised to see like I'm, I'm not saying like we really like rip out hnc or anything, um but I was also surprised to see that, like it sets up web hooks like on on every object in the cluster, and I could see that being not agreeable to some some operators out there so like. If we had the crds, then there could be like also a an alternative to it.

E

Where, like you, make the crds and then like an operator could set up the namespaces or whatever and and the controller could put the appropriate service accounts and stuff in those um that's a separate issue. I think I think what you're saying about maybe for expediency like in the gym it could. It could do this for now and then, like.

C

Yeah I mean they, they have web books, but those are needed for them to, I think, keep a coherent hierarchy.

C

Among the other things I don't know what all the web books, uh but like the one I hit from time to time playing with it is like, if you try to delete something, and this thing is not empty or something it tells you, you can't do it or stuff like that, like you, it doesn't allow you to mess up the hierarchy uh so well. If we replace this with something else, there's a chance that we have, we wouldn't need the same validations. No, I.

E

Guess the one I'm talking about is this one: that's a web hook applied to the wild card. Api group.

F

That's because.

E

F

Dynamically assign propagation so, okay, so I guess it has to they've set it up. So it listens on that absolutely everything. Unfortunately, there's no like initial conflict that it looks at and stays like that you can change it by modifying hnc configuration at any point in time.

F

E

I mean I I get why they need it. I I'm just like when I saw that I I like I'm saying like thinking just like in terms of an operator. They might not be too fond of that one.

C

um Yeah I mean it's, I guess either we we would have to think about an alternative to those to replace them or maybe a way to make it less invasive. Maybe so it only applies to objects belonging to a subset of the cluster. I don't know how you would do that.

E

Yeah, like that's, I mean for the sake of the immediate problem like I, I made a story where we just want to be able to run and stage apps and spaces that we create with the cli. This was just like something I I found well. I was going on my my journey last night, but.

E

um Anyways, it sounds like um we should probably disable the propagation of service accounts, because that doesn't work and they will like probably step on anything. We do and um like look into updating the api to to try to create these.

C

And this is just for spaces not for orgs.

E

Yeah it's for spaces, because um that's why the apps! That's where the apps run and stage.

B

Is it worth looking at kpec to see if it would be possible to alleviate some of the immediate concern like? Is there a way to configure that? Do we think.

E

No, it only takes a name, um so it it will only look in the same name space as as the kpac image.

B

All right, that's unfortunate, maybe a feature request or something down the line, but yeah.

E

Maybe I think um um I don't even know if you can do like cross name space like image, pull secrets and stuff like I. I think it in terms of how they're actually using the service account. I think it might make sense like for it to be scoped to the name space.

B

Well, some good information things to look into um any other comments on this topic. Questions concerns.

B

Going once well so this.

F

Is going to be an absolute sort of killer fate for hnc, then I suppose it's a bit of a shame because then well, I don't know, there's a fair bit of work to get that whole replication of roles working nicely.

E

It's not a ki, I mean I think, as long as the roles work for it like, we could still use it for rules. I think it's just not sufficient on its own. I.

F

See yeah so then we just do a manual service account and secret thing. Yeah. I see oh yeah that'll keep us going for a while anyway,.

E

Yeah, but I think it is like definitely pointing towards the like we should. We should make those like oregon space controllers and and move some of that and out of the api. And it's.

A

E

Definitely pointing to some value for having an abstraction there.

B

Yeah, I definitely would uh caution against immediately going and trying to rip. Hnc out sounds like it probably still provides value in certain areas, and then we can disable certain facets of it and then supplement it with some of our own stuff until we decide well, maybe that will be what we go with long term, but uh at least would give us some room to breathe so cool. It's.

C

Also, an api that it might just become part of core. I don't know exactly what the if that's going to happen or when, but it looks very official yeah. So there's.

E

C

That, eventually just becomes kubernetes, so there's no option anymore. It's that's just how many spaces are going to work no way, because you don't have to use them right.

E

Yeah, I think I think, right now, it's just early days for it, which might might be what we're betting up against like I was. I was also surprised to see it's like. There was an issue for like it, propagating some istio config and like stepping on istio's toes and their like work around, for it was just a hard code like the names of the resources that istio was making. I was like what the heck, but but anyways.

E

I think um I think you're right that, like as a it's still like pre-one, oh and like as it matures, um I think right now we're just hitting a use case where, like its previous service account, behavior was was worse than what we're getting right now, because I I think it was like locking up the cluster as agency and the token controller were fighting over over these secrets.

E

So so this was what they did in the interim, at least.

B

Cool well last call for comments on this one.

B

All right, then we're moving along to. We have a topic about binding service accounts yeah. So we've.

C

um We've noticed that uh we have this uh bit of discrepancy between uh cf and cates when it comes to types of users, because ncf there's just uses well in kubernetes, there is two types of users: one is actual users which is humans and then there's service accounts which are treated kind of differently and at the moment I think in our role binding code, whether you try to assign a role to some user. We just hardcore the fact that the subject of the broadbinding is a user.

C

But that means that via the api, you can never bind any role to a service account.

C

So we were thinking about kind of extending the api with a field that wouldn't make any sense in cf, and it's not part of the cf spec, but it wouldn't make sense on kubernetes if people want to leverage service accounts, uh for example, for automation, because I think in cf people automate by just making the cli log in and passing the password by a standard out or something well in kubernetes.

C

There is proper ways of doing this uh by using service accounts, uh but I've now really, as we talked about the issues with hnc's, I've just realized that maybe this might not work as well. If we have these hnc problems,.

C

Hopefully the message go through. I.

F

Just wonder what the hnc problems were. I mean I just had a little stab at it as well. So I was changing the end to end test code to use the api rather than create kate's objects directly, and I got a bit of a surprise when half the roles that were created by tokens were not being recognized and then it's because it was setting them up as users instead of services accounts, but it's it seemed.

F

It seems like quite a nice thing to do, to have a kind field in the role in the create role request and it can default to user if it's not set and we can set it to service account if we need it to be service account.

F

So at the moment I could see people doing a cf curl or something maybe if they really wanted to do it from the cfcli to go and set that and then maybe in the future, we can add an option to the cli, so you can specify that, but it I don't know it seems to fit kind of secretly and nicely at the moment. With the case side of things and.

C

The process the problem is always like: if service accounts are namespace, that is a bit hard. Isn't it like? How do you know like which service account? Are you talking about if they're namespaced, because.

F

You always pass this org or space in the request and that's your namespace ah right very good point. Yeah.

C

So that maybe works.

C

Nice yeah in general, just a shout out. This would be the first time we would like kind of enrich the acf api with kubernetes specific things which don't break backwards, compatibility but kind of add new stuff on top of cf, which are kubernetes specific. I don't know how people what people think about these kind of things.

B

So I'm just trying to type up a note or two about this.

F

And while you're there on users and service accounts, you can also bind groups which takes it even further out. So there's there's no real way of doing that in the cfcli. But if, if somebody wanted to assign groups or of them, users would be another way to do it.

D

And those are things that would just be going on to like a rule binding in that name space. That's uh we're managing the the api layer or maybe in the future, like an organ or a space controller.

C

D

That's the idea: okay, yeah, so like right now, people would have a workaround where they could. They could add those other subjects directly on that on that or any other rule binding right if they wanted to set that up.

F

I mean they'd kind of be forced to do that. So, if you're trying.

A

F

Unattended stuff, our recommended ways to create a service account and use the token from that to provide the authentication um and then you've got to go and authorize that account somehow um so that would either yeah. That would be literally creating a role binding in case directly. At the moment, this opens it up, makes it a bit more user-friendly if you like, all goods.

D

Yeah, I I think, being able to um certainly like officially register a service account for operations within within a space like that. Definitely sounds really valuable, um given that uh you know, assuming that whatever we're doing to propagate user roles into the appropriate role, binding wouldn't interfere with those uh like. If that's the case, I would think that extending the api wouldn't be a very high priority, but it seems like something we could certainly figure out how to do and.

D

It might be worth at least discussing with the runtime interfaces group just to see one you know make sure they wouldn't have any objections and to see how this might play into any changes they could be thinking about, or you know, might have inklings of on the cf for vm side.

E

Yeah, uh like one of the things I'm wondering, is how do they handle today, the uaa user versus uaa client.

A

E

Here because, like the cli has a special flow for uaa clients which, like I know, they're different but conceptually is similar in use case to a service account, um but maybe that's just abstracted away on the api.

E

um I I had a question on the dock on our on our node stock. What um what in point were you thinking of adding to specifically.

F

um The post on roles, these.

E

Wheels, why did they not have a documented.

F

Well, create role in the api.

E

Oh, it's on create role, yeah, okay,.

E

B

If you have the uh uri for that feel free to add it tim.

E

Yeah, I just got it.

B

Yeah, definitely um if we want to add some functionality like that, when I have the default case, the the expected cf api.

A

B

I could see enhancements working and, like you said you could do a cf curl. If you wanted to sort of get that functionality without making a direct change to the cli.

E

Yeah um the way they structure this api with this um relationships object makes it pretty amenable to things like what we're suggesting. So that looks good cool like even if we like, like name spaced, our our type or whatever, with like kubernetes like I. I think it would be very unlikely for us to step on toes, not not saying we shouldn't reach out to runtime team, but this this api looks better than like what I what I expected for for doing this.

F

Okay, so you'd recommend instead of having user you'd, have a service account there under relationships and then be able to deal with that.

E

Yeah or even like kubernetes service account or something like. I don't think there would be a cf for vm service account, but just.

C

E

Disambiguate, it.

B

B

Well, um probably add any more thoughts you have on this topic or we can move on.

B

uh We are at the end of the topics in the document uh we're about a half an hour in uh so unless there are last minute thoughts on what we've already discussed.

B

I'd like to open up open the form to any thing anyone wants to discuss questions comments, concerns um welcome to all the people that are here for first or second time glad to have you please show up anytime. You want to talk about us and think about what we're doing and get involved yeah. So any topics anyone wants to bring up.

B

Sounds a little quiet on the western front uh if anyone's thinking about please just uh raise your hand and speak up, but as sort of like a logistics thing uh we're sort of heading into holiday season. So I expect that the meeting schedule will get a little bit interesting as we sort of dance around the holidays.

B

uh Current plan is still to mostly run the meetings, but I would understand if people were not showing up with a torrent of topics to discuss, so uh don't feel too badly. If you aren't going to make those meetings, um yeah still still planning on running them. For the most part.

B

B

As much as I love hearing myself speak, um I'm gonna, let everyone get back to whatever they're up.

A

B

Now their lives uh thanks again everyone for coming I'm discussing this is a lot of good stuff to think about. So I'm looking forward to digging into that in the future.

B

C

Thank you thanks.

D

C