Cloud Foundry CF on K8s Forum, 11 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CF on K8s WG Forum 11th Jan 2022

Description

[TD] Proposal: User Provided Service Instance Support in CF on K8s
Explore story: https://github.com/cloudfoundry/cf-k8s-controllers/issues/367
[JH ] Prefixing guids in namespaces & statefulsets–should we create CFOrg and CFSpace CRs first?
Some preference for establishing a pattern of `prefix_guid` or similar. Optimize for practicality.
cf-space-{unique-id}
cf-org-{unique-id}
Context around the original GUID decision
Some concern about clients expecting the GUID format or database schemas using GUID types.
If we use well-known prefixes these clients could always just trim off the prefix
[GC] Should we get rid of the authEnabled feature flag?
Should set up an admin user when initializing CF on K8s - role and binding.
Start with minimum privileges?
[CC] Update on timeline for public roadmap and beta release?

A

A

A

What's up, are you recording this ben nope, okay,.

B

And we're recording.

C

Awesome, thank you. uh Welcome everyone again to the uh qualifying on kubernetes working group forum for the uh start of the new year january. 11Th 2022 looks like we have a couple topics on the document.

C

I will link it again in chat, so you can get to it that way. If you have anything that once you want added to the document feel free to drop it in the chat, and I can add it in as well, but the first topic we have is the proposal for user provided service instance support tim. You want to give us the tldr.

D

Yeah, um so I'm not expecting everybody to just jump in and start reading. This just wanted to share out um that some some folks of the on the team have done an exploration into how we might support user-provided service instances which, as you may or may not know in cf, are a way of manually configuring. A service instance um that you can bind your app to so like no service broker api or anything just like manually, providing some credentials and configuration that you can use for service bindings.

D

um So the tldr of this proposal is to leverage some projects like the kubernetes service, binding spec and some of their resource. They call them duct types such as the provision service, deck type and use that to build out some initial support for for these user-provided service instances. So anyways um we're we're open to feedback and we'd appreciate it. If, like everyone, could like, take a look and comment on this proposal,.

C

Awesome sounds interesting. uh I personally have not been able to look at it yet, but we'll definitely do so, because this sounds pretty cool any questions or comments. People have maybe things that they want to ask tim since he's here.

C

A

C

We'll, probably uh I'll have to take a look at that, since you know it'll take a little bit of time. uh So the next item I see is about prefixing goods and namespaces and staple sets and says: should we create cf oregon cf space crs first.

E

Yeah, so uh I don't know if this is the best forum for this question, but uh we had an uh spike to try to prefix the goods that we use for the names of um kubernetes.

F

E

And stateful sets with some prefixes that allow people to kind of identify them, and I think this was probably mostly to do with name spaces, because it's kind of hard to tell when you're, using the kubernetes.

F

E

What's what you see this sea of goods, um so there was apparently feedback from users customers that like to be nice, if we had some clue whether these things are orgs or spaces. um So the first thing.

F

We tried to do was just.

E

When we generate the good, we just added a little bit of goo at the front of the good to make it into you know.

A

E

Word or see, if space matt, the problem with that approach is that that then comes back through the api as the goods that we use for orgs and spaces. um So if we want those goods to look sort of traditionally like a good, then we needed to everywhere. We were interfacing between the internal representation and the cf api. We had to strip those things out and then re-inject them on the way back in which seemed like it was brittle and terrible.

E

um So we were thinking uh we need to. If we're gonna. If that's an actual requirement, then we need to move the implementation down into the controller layer, preferably, but as it stands right now, we don't have controllers for orgs and spaces we've been talking about. Maybe we ought to.

E

uh Maybe it makes sense to wait on this until we do so that we could then have a cf word with a a name that just looks like a good and then handle in that controller handle the internal representation like prefixing it on its way in and out.

E

So we thought we'd see. I think one of the big questions for this is like how hard is the requirement that we have goods that look like goods in the cf api, because a good isn't like a prescribed format. That's globally unique.

A

As a user of these goods, I actually prefer just having the prefix in front and just knowing the the convention. Let's say a convention of prefix underscore good. If you use that type of convention, then I always know how to strip it off and get to the actual good.

A

If I actually need to go in, but if you use internally, the prefix underscore good for everything that you're doing, then I don't need to worry about stripping it off, just use it and send it through, and I don't think it's any confusion that anybody you don't have to call it good. You call it a unique id right.

A

You don't have to call it, though it has a special meaning right how it gets formed, but in this case you're adding uh a prefix onto it, call it by some other name and now, let's use throughout the whole interface.

A

uh The only thing has to change is, if you have any databases that are restricted to saying this is a good type right. You want to change that to be a string of whatever right.

A

So I I I. If you're looking down approach, it's trying to be uh traditional versus just practical, I would go down the practical route more than the traditional route, and people won't be confused by it. uh Github, for example, does that for its uh access, personal access tokens, for example, other tokens, it's it's a guide, but it's a prefix in front of it.

C

That's uh some great feedback. Thank you. um I think, as long.

F

As we can get.

C

Away with that, and there isn't some hard requirement somewhere that we haven't sussed out yet then sounds like a pretty good thought to just optimize for practicality.

G

So we are just talking about prefixing with org or space. Are we not with the actual name of the organ space.

E

uh Well, it would, it would end up on the name space. It would be the the name of the names like the the requirement. As I understand it, was the name of the namespace should have a cfo or cf space on it, so that it's easier to understand what that name. Space is.

G

Yeah I mean if this escalates into saying when I create org foo, I want it to say cf or and then a good, then there's trouble because then we've got this.

E

Oh yeah of names, yeah, okay, it's not the it's, not the the name, the cloud foundry name. It's the cloud, foundry good, slash, kubernetes name.

A

It's it's the internal naming it's the internal names, not the external names.

A

From from that standpoint, right, the user interface doesn't change you. Don't most users, don't look at these goods. It's only people using the uh the api internally and stuff like that, but they'll have to worry about it uh directly. Otherwise, no users be affected by it. They wouldn't even know it.

D

Yeah, and and going back like kieran's concern is, if we put the like cf name, those are mutable and on kubernetes, like the name fields like once, you change that it's no longer the same resource. So we're just talking about like static, well-known prefixes for each type.

H

If the purpose is to make these things more recognizable to cube cut or users, let's say: can we solve this with labels, for example,.

H

Because I guess it's less visible because you have to look at the resources to see which levels it has instead of like in in a list. It wouldn't be immediately visible, but.

A

Again, using the practicality approach, when I'm trying to use the api- and you know- walk down the tree of goods in order to get from the org to the space to to uh to the apps, whatever they'll have their prefixes, and I'm just going to look at the at the good that comes back, and it helps me when I'm looking at that data.

A

uh Okay, I'm looking at the app school here right now, I have to you, know, base it upon looking at the url string of where the good is located within the string kind of knowing what's going on by having that in it. It makes a little easier that I know what type of gui this corresponds to by having something in front of it versus just having a traditional straight 64 character. Good in front of me again, I'm going back to practical versus uh you know, traditional.

E

Yeah my suspicion is this is coming from somebody who's kubernetes operator, who has a cluster with a bunch of other stuff on it with a bunch of other name spaces and it's kind of hard for them to uh to tell which ones are the ones that we polluted their cluster with, um and so we could put. We could.

E

We could come up with a bunch of different ways to solve a bit, but um probably the least obscure would be the best like you know we can put a label on, but then they have to know to look for a particular label.

H

Yeah, and if I remember correctly, it was also get the hierarchical namespace thing to propagate labels through to the actual namespaces. So maybe just picking up a name. It's just easier.

E

Yeah that that particular approach is like three lines of code or or six if we want to extract constants, so it's pretty pretty uh pretty simple.

C

Yeah, so I guess the the following question would be if we do want to do something like cf space, either uh dash or underscore the guide and same with the org. If that pattern seems like something that's good to us, would we want to propagate that to all the other resource types, or is it only something we find is needed for sort of the org space.

F

It's it's something that when julian and I were talking about this yesterday, we were looking at just the non-cf resources, because that's the ones that we thought were the most confusing, but based on norm's comment, it might be worth just doing across the board and saying: okay, everything has this prefix. So that way, you know what you're looking at um without having to think about. Okay, what type is this and looking at other other values?

F

A

Also makes a little easier for testing purposes you if you put the wrong guide in place, you can't tell that by just looking at a traditional good, but if you put that prefix in front, you expect it to be a user-provided service, school versus a marketplace code. What, if there's a difference right? You could have that prefix being different. You can see it right away and you can test for it and you have better test cases.

H

You will be quite redundant to have a cfr resource with the name with name cf app blah. So that's something that I wouldn't like to see honestly like the for any resource we reconcile to.

H

So I don't know deployments like knowing that a deployment was produced by a cfr is useful, so calling the deployment cf app, something I think makes sense, but calling the cfr object resource cf, app, something feels very redundant and same for namespaces, which I again target resources, or I don't know whichever we probably generate a whole bunch of other resources that are not around custom cds, like our customers. Here, these reconciled a whole bunch of things. So we could use that approach for those destination resources. I wouldn't use them for the source resources.

H

If you see what I mean like we're talking about, for example, cforg and cf space crds, using like putting prefixing the name of those resources, wouldn't make much sense to me like, but the name space they reconcile to, or even the hn hns resources that they consult to then. Yes, that makes sense, because then you, you know where they come from, the problem is kind of tracing back where they come from, but like this cfap doesn't come from anywhere.

H

It's it's our entry point right, so I wouldn't, I don't think it makes sense to prefix those, especially if we expect those resources to be created to be created by users directly one day. Potentially, I don't want to ask people to prefix stuff, like that. Doesn't sound like a good ux.

A

It's it's, not the user, that's doing the prefix thing, it's the internal subsystem, that's doing it and again, I'm I'm correct. If I'm wrong julian, this is offering internal naming conventions and keeping stuff straight. So when somebody's looking at the api data or the internal databases and stuff, they have an idea. What they're looking at right off the bat.

H

Yeah, I guess it depends on the definition of internal versus external. I think we do have. uh We do have the shim, which is at the moment I think the our outermost layer, but we've also always talked about having a layer of crds that is effectively a user interface that people can use.

H

Instead of this shame, especially one day replacing the shim all together, like I don't know in a very distant future, so I think we should treat those that that external layer of crds as public interface like someone, should be able to create those objects like by themselves.

H

The the the the api at the moment creates them for you, but like we're, I think we are converging to a point where, like we have once here, deeper reversed source in the api, and I have having a close resemblance so that one day people could use this here this directly instead of hitting the api.

H

uh So I would, I would be mindful of the experience of creating those objects uh manually because yeah and.

B

I think that was because they.

H

Could end up being a republican.

I

If you look at uh some currency subjects like pods that sometimes gets created by deployments and the naming that they have chosen, I guess they do something like if a deployment creates a board and it's called my deployment- you get like my diploma- my deployment dash random string, so they embed the name rather than the type. So maybe we can just think of something like that. Maybe I don't know how applicable is this for auxin spaces? Obviously, but.

B

I

For example, it wouldn't be nice as episode it will be redundant to call the app cfap dash my app it.

F

I

Be more like calling it my app dash uid, so that's unique, but I just wanted to make the parallel tool like we should probably be looking at what currency is doing already, because it will feel more native if we follow their patterns. But orcs and spaces are kind of a different beast. I think.

E

Yeah, well, I guess they don't they don't have to be. If we had a cfo are going to see a space crd, then it would be yeah. Then I think you would have that that the same parallel, because you could just call the org by the raw guide and the underlying subname space anchor by the prefixed good.

E

uh The only problem there is that, of course, the namespace name leaks out right. You use that everywhere through all the layers of our code, so there will be. There would be some changes, but.

C

Yeah definitely some good discussion here uh on my side. I'm gonna probably need to do some thinking to sort it out and map it out, but um that's yeah a lot of good thoughts here.

A

Yeah we all know naming is hard.

C

This is true, I'm happy to let this conversation continue going for a bit, but we do have a couple other items we want to squeeze in at some point in the next 10 minutes or so so final thoughts on this until later,.

C

Cool all right, so uh we have another item that is: should we get rid of the auth enabled feature flag.

H

Yeah so uh some some context, we started implementing the shim and the shim was uh at the beginning. The shim was always hitting kubernetes with a what we call a privileged client, so this client had uh was using like a system service account to hit kubernetes.

H

So I had privilege just to do any everything right and then we started actually changing the api so that uh the user sends a header to the api with their with their authentication information and that's used to create a user-scoped kubernetes client that we then use to hit kubernetes so that every operation that hits kubernetes, even if it's through the shim, it's authenticated as the original user and we leveraged kubernetes our back for our authorization, which is pretty cool and we're doing we're still in the process of doing this and we're doing an endpoint by endpoint and our issue.

H

Our initial impression is that if we just started doing this endpoint by endpoint, then we would get in with situations when, like some endpoints are authenticated and some endpoints are not uh so we introduced this feature flag called off, enabled if you turn it on, then we do what I've just said. Otherwise we just always use this privilege client, just like we used to do, but today having a chat with kieran.

H

We kind of realized that maybe the future flag is not needed and there is never a case where uh doing some even like using mixed end points. Some authenticated some not might lead to problems. So the question: if the question is, does anyone get any one thing on the spot on of some situation where, like that could be problematic, because if not, I am very keen to just get rid of the future flag as soon as possible, and because it's just extra complexity that is sitting there.

H

um I couldn't think of anything so like it's just sometimes some things will happen in admin. Some things will happen as yourself.

E

So I guess the only the only thing that that flag buys us is an escape valve for when there's bugs, and you can't test.

A

The thing you're.

E

Doing because you can't set up your cluster and yeah.

B

E

I would point to is um there's a bug sitting in the backlog that about interacting with empty orgs, and I I'm not sure if it's all types of clusters, but certainly it comes up with kind so like if you just.

F

Create an org and.

E

You haven't made a space in it yet then that org is unfetchable with auth enabled turned on now, or at least it was as of last week, so that makes it very hard to test other features.

F

E

Having said that, I 100 on board with getting rid of this. That is a goal.

H

E

H

The idea was like we authenticate all endpoints and then take it off, but if it's not needed, we can take it off immediately. uh I'm wondering if there is any like problems that you've described can be solved just by logging in as an admin. Basically so.

G

Yeah, okay, I think that's the case. We I always find I have to go and give give a well. We don't have an admin role at the moment, but a space developer role in the cf namespace to the um kubernetes admin user is the first thing when I create a cluster and deploy the api and then, as you, create an org that space developer role gets copied into it, and then all the logic works, there's filtering on orgs that make sure you've got at least at least got a binding in it.

G

um So that's probably why you don't you can't list it, and you can't see it um we're missing that whole step, like you get in normalcf, where the admin user is created with an admin role.

G

So if you manually do that it works at the moment, I think we need a story to sort that out so that when you get a fresh deployment, there's an admin that works and you you kick off from there and everything else fits into place.

G

E

That's interesting I'll have to try that um yeah, documented or.

H

So that means that the that proper way of getting rid of the future flag is allowing people to kind of set up an admin user when they install the cluster. So when they install the customer, they say this is the admin and we set up the binding for them and even if the admin role is empty for now, at least that our code just checks, you have some binding in it.

H

So role probably doesn't even I probably can can just displace developer one like or can can just be a copy and paste like it or it can be like a very permissive role or whatever like we. We can improve the actual permissions of the admin role later, but like just having a role and a binding kind of solves all these issues.

H

So maybe we need to think of something- and maybe that's part of our.

G

Provisioning- I just hit this now in the end-to-end tests, so as we start putting off onto orgs and apps, and things like that where's, the thing that can create them to set them up in the e3, I'm missing an admin user and missing the admin role. So I think we need to go and push this along. While we add all these other things.

H

You need some sort of story to introduce the admin role, even if at the moment is not perfect, like just something that can do everything and.

G

It can be emptied stuff and we can just add permissions to it as we need them.

H

Exactly because otherwise you try to do things and you won't be able to do anything like if it's at least as powerful as space developer, then it will will work for now.

H

I think no, you need to be able to create orgs, so no, but whatever we can think about it, but we need the role and then we need a way to when you, when you provision to say this is the admin user and we need a way to automatically set up the binding to that user, something that eventually will have to do right. When you install cf, you get an admin user, it's just in our case.

H

We want an existing user of the kubernetes cluster to be nominated the admin, so we can't just create the user. We need to ask whoever installs to tell us who's the admin. Basically.

G

Yeah and it's it's also a bit tricky way to put that logic. So we've got the config package with some yaml files in it. um I remember trying to create a cf namespace in there, but it was impossible because um customize was changing, namespace names, prefixing everything was cf system and all that so I think now in the um deployed dependencies script it goes and creates the cf root namespace.

G

So that's later on, and it's only once you've created that namespace can you go and create the admin role binding in that namespace, so it feels a bit messy at the moment.

H

Yeah, but it needs to be something that we distribute like it's not just for us. It's, like the users, wins the operators who install this thing need to be able to to do this.

G

So it should be in the sort of config area. I think.

H

G

H

Okay, well at least we can put some thought into it.

C

Yep, that makes sense.

C

I just shot a few notes down there about the discussion about privileges, starting with minimum or sort of opening the gates, because it's an admin definitely should think around that, but it can probably be driven out of the normal flows.

C

We have a couple minutes left and one item left, which is an update on timeline for public roadmap and beta release.

C

Does anyone with a authority here want to speak about the public roadmap, since we did have uh the sort of public beta release on friday, but as far as roadmap, I don't think I'm authoritative.

J

I think you're talking.

B

About me, to some degree, I mean not not to put anyone on the spot here, but I guess, generally speaking- and I think chris is on the call and he'll agree with me. um This whole cf on kate's effort is kind of our biggest story for the year.

B

So um one of the things that we're really looking forward to is to try and get folks to sort of um start engaging with the project, at least by consuming it, and just you know, putting it out there in the hands of the people and get some feedback and get some buzz going in general. And I guess the two big things that we are looking for is a the beta release tag.

B

I guess doesn't matter what it has what it doesn't have, but you know just as a milestone um give people the feel that you know the project is moving forward and we have something to show and tell, and also the whole notion of the public roadmap is kind of very welcoming, so that we have the opportunity to again showcase what people can expect in terms of engineering effort and how our ideas and theories from the end of last year are actually translating into something actionable for the community to consume largely.

B

So I think that's really the background in terms of introducing that item for discussion here today.

J

Yeah, that's fair, so so I mean I I would say for right now, I'll get you know, we definitely have plans to do betas right and release those in the open source side. um I think we need to really kind of nail down exactly what we want that one to look like where we're at a good spot to kind of draw that line and say publicly like this. Is you know it's a usable beta? It's a place to be.

J

I don't know that we have a good handle on that exactly just yet, um but it's definitely on top of our mind, so um I think we'll take that feedback and and maybe prioritize that work to to make it more public and make it more official. So it's a good call out.

C

Cool yeah, I think, uh of what phil said, and we just want to make sure that we have something that is friendly to use rather than frustrating, so that's kind of the high water mark, because we gotta make sure it works sort of out of the box.

C

um But with that said, we are at time in a minute over, I believe, or do we actually have the full hour that we have four hour? I forgot we did uh half an hour for the past several months, so I was trying to keep it to 30 minutes. So my apologies for compressing that time, but we can continue talking about topics that people have interest on or we can adhere to that uh accidental constrained half hour that I did other people feeling.

J

One way or another, I think the only thing I want to just check in real quick julian, if you feel otherwise, we talked a lot about this, the oregon space crs. I don't know that we really got like what our next steps are on. That like is it to put together a couple of options and the pros and the cons you know, I think we just kind of we had a good discussion, but I don't know if there's like. What's next, you know what I mean.

E

Yeah, uh I I feel like we're missing representation from product work, also we're missing the person who fielded the original complaint, so um it seemed like the discussion sort of mapped closely to my interior monologue.

E

While I was spiking on this like yes, this is the cheap and cheerful way of doing it, which would be maybe not too bad, maybe fine, but maybe also kind of gross, and then there's this other thing and then there's the right way to do it and um so yeah. I agree. We don't really have a resolution there, but I think it would be good to get a uh maybe a closer read on the requirements from eric.

J

Yeah, I think that's pretty good call just to get more definitive unless tim, you know specifically like more definitively like what the desired outcome of this is.

D

um I mean, I think, from eric's perspective. The desired outcome is mostly cube: cuddle users or people who have installed, like guise and stuff on their cluster, just to be able to see like what all these random non-cf resources are, which we talked about a bit but, like. I think, a good next step would be to just put together like a not too heavy, but some dock that, like outlines like the different classes of resources like ones we don't own ones, we do and anything in between and how we might name them differently.

D

And just so. We can visualize this better and and comment on it um yeah and then I think, there's a second point about like um internal representation versus external, like through the shim.

D

um If we want to change that like because I could imagine, there's a world where internally, we make these things more friendly, but the shim continues to present them as as goods for like databases or clients that rely on that or there's a world where we decide that use case is like pretty niche and not that important so like it would be good to just like dig into that with a little more contention tensionality. I guess, but um I don't know, that's that's just like the next steps I see and then like.

D

We could get eric's opinion on that probably easier sure.

E

Yeah, I I guess at some point the you know, but part of part of the question around this is like how much do we want to invest in it? um My my fear is that like, if we don't actually care that much about it, we might end up investing quite a lot more in it than we intended to just to figure out that we don't think it's worth a lot of investment.

E

uh So that's kind of.

F

Part of the you know.

E

The read I wanted to get from eric yeah, that's.

F

E

To do you know before.

E

Spending the time it takes to kind of write, you know, make a total roster of all the resources that would be touched and which ones would be touched under what scenario and how much of work it would be.

D

I guess I was thinking lighter than that, just like classes of resources like like once, like literally just ones, we don't own like like names faces and stuff, and then we would just say, like we attach a prefix like like nothing that like enumerates every single resource or anything and how we we might treat it.

D

I I just think like there's like a cheaper world, where we only touch those and leave the cf ones alone for like um the concerns that folks raise around like redundancy and stuff and, like maybe there's less value of renaming those. But I don't know.

E

uh Yeah well kind of I'm.

H

E

I I my read on this: is that there's really kind of two avenues: one we just generate some goods with some extra stuff on them. That's incredibly, cheap has some weird side effects. Maybe maybe we care about the side effects? Maybe we don't, but we're really talking about three objects right and then there's the oh we're gonna go.

E

Do it the right way, approach where we find the boundary between the things that we own and things we don't own, and we inject prefixes at the point where we hand it off to, for example, the areni lrp from the cf process were the namespace sub namespace anchor from whatever layers creating the subname space anchor, uh and that second approach is a bit more expensive.

E

It requires us to be more careful, has some aspects where it leaks right like even if we're really trying to sanitize it. We still have kubernetes name spaces referred to everywhere, so we have to be aware that those things have a prefix now, um so there's more work right and- and maybe we.

F

E

Until we've implemented orgs and spaces in order to make it cleaner, but without knowing like oh, what's the urgency around this, it's a little hard to make that call um so yeah. I was kind of hoping that eric would be in this meeting, but we couldn't bring it up again and keep forum. Maybe that's the next step.

C

Cool yeah, it all makes sense.

C

um I think, for my thoughts, part of what we're running into is, as mentioned that divide between the customer resources and kubernetes resources, and if we really wanted to just overkill it, we could make a custom resource that wraps each kubernetes resource, but that's just a whole bunch of mechanics. We don't want to deal with, so this prefix is probably if we divide it between those that probably works so.

D

uh I'm like I don't even I think the some of the feedback was like people care a lot about namespaces and they care a lot about pods because they want to there's like security and like runtime implications of those. So it really is those like those core ones. When you have a bunch of like just random stuff running, you want to know where it came from better.

C

So maybe even further, it's things that are generated from our controllers versus things that are directly created with our api might be a way to look at it.

G

But there'll be a feature request for arena as well. This will be a first in a while.

C

All right: well, then, it's not seeing more topics being added, so I might at this point actually let everyone go give you the 20 minutes back and uh we'll see y'all in about two weeks or whoever can make it, and hopefully we'll have some more of them and all of the stuff.

H

Thank you folks, for.

C

Sure everything.

J

H

Into this, thank you.

B