Cloud Foundry CF on K8s Forum, 14 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Foundry on Kubernetes Working Group forum June 14th 2022

Description

2022-06-14

[MR] Inlining eirini as statefulset-runner vs writing it from scratch
[GC] Rewriting eirini tests to get rid of telepresence
[MR] The consensus is that we should rewrite the code as statefulset-runner. We may want to take this opportunity to move to deployments from statefulset, though it likely makes more sense to do that once we have statefulsets working due to the CF_INSTANCE_INDEX
https://github.com/cloudfoundry/korifi/tree/main/kpack-image-builder
[GC] Repository tests - envtest vs kind

[AK] Validating webhook for duplicate role bindings for kubectl users
https://github.com/cloudfoundry/korifi/issues/694
Building webhooks against core k8s resources
Context: On AMER on tues/thurs a pair works through the chinbox
Chinbox is not ordered by priority, but rather by grouped by topic ^^ we’ll share this out with more of the AMER team
Label less fleshed-out chores as “draft”
[DB] Pod security standards
Pod Security Policies (PSP) are deprecated, will be gone in K8s 1.25
Replacement is Pod Security Standards
PSP configured at Pod level, Pod Security Standards at namespace level
Kpack issue: https://github.com/pivotal/kpack/issues/972

A

B

Yeah I mean I'm gonna, wait a little bit for more folks to join, but here's the start. The.

B

Cool all right so, um as I mentioned earlier, I'm going to fill in for a bird rock today and just facilitate going through um the dock that I've I've dropped in and as new people join I'll. Just keep dropping in links into zoom chat to get to our agenda doc, uh but that said um folks feel free to keep adding stuff, but we can get started. I think matt you've got the first topic on here.

C

Yeah, um so just to provide background for for those who who aren't familiar, um so we we've been using irene to run our you know our lrp workloads staple sets for apps. um There's this work under underway to move to another crt called the run workload. That kind of fills the same niche just represents it differently, and so, as part of that we we have this story. That's in flight to take irene kind of inline.

A

It inside of karify and give.

D

C

The staple set runner- um and I think you know that sounded really straightforward on paper. I think the uh actually doing it has proven quite a bit more complicated.

C

um For one thing, there were some issues running the tests- we've mostly gotten through that now um now, there's issues running um building the images because we're just building them very differently than we're building in the rest of the carrifie project.

C

It also just occurred to me that, while in the main project, we just use a kate's client in our tests and in our code to fetch resources in um in irene, there's these, like generator typed clients that are tied to the lrp type, and so when we actually go and eventually make the change to use the runway cloud, we're gonna have to figure out how to regenerate all those and work with those, and just every every step of the process has just proven very difficult.

C

Unfortunately, because it's so different from the way we do things in the rest of kurifi, and so this prompted discussion um among the american team last week around like. Are we doing this correctly? Like? Should we actually just start over?

C

Take the same functionality that we have an irony and write it from scratch as this new component, but follow the same patterns. You know, use coup builder and just kind of go with the patterns we've established already. So I just wanted to throw that out. There see how the um the emea folks felt about that just talk about it.

E

Yeah, I think I think we should rewrite it.

E

I guess you can do some heavy copy and pasting, but I would take the opportunity because also, I think that there are some issues with just the design having been inherited by the old irini, which was a rest api. So there are some. Some stuff were just moved over for convenience, but I think there is. There are a bunch of abstractions in there that shouldn't be there. They only existed because they used to support a rest based api.

E

Now they kind of I mean they still make sense, but just the whole thing kind of became a lot shallower in a way like it used to be rest. That goes to the creates a crd like just a bit big. It's never been a very it's never done a lot, but like it used to be more, no, it does even less so maybe some abstractions are not needed anymore.

E

I've added a sub item because, like just just in the last, maybe 20 minutes me and georgie have been looking at an issue with telepresence, because um we need to make a. We need to make a change to irini because we realized it has a bug like it's ignoring a couple. The the memory and disk fields for tasks are being completely ignored, so we're like okay, how do we run the test?

E

We need telepresence and we happen to have uh upgraded our image image for our development stations and we now we can't install telepresence anymore, because the version we were using is deprecated and the new version is completely incompatible. It will take probably forever to figure out how to make it run, blah blah blah so yeah.

E

It doesn't feel like it's worth, investing time into upgrading to the new telepresence and in general, just carrying over with all this old stuff. We might as well just redo everything, and until then I guess we we just run the old tele presence. uh The package doesn't exist anymore, we'll have to build it from source, but we're hanging there for a bit longer. Basically, and then we just dump the whole thing like that: dump all the all the old stuff.

C

Yeah, I may, um I may be able to help a little bit outside this meeting with telepresence. um Just we were, I was able to get it working, for, I think the eats tests um with the modern telepresence, but I did end up with a pair rewriting or not rewriting moving all of the integration tests over to use test m, um but that did mean commenting out a fair number of tests that we're, assuming that, like pods, were being stable, sets are being reconciled to pods and things that don't happen in intestine.

C

um But I I might.

E

Be able to offer some of that honestly like it's, it's just not worth it if you're gonna rewrite it anyway, okay, just it's just a matter of finding a way of installing the old one. It's just. If you use the old install instructions, the packages disappeared, they just just it's deprecated aka. It's been wiped from the face award. So now you have. If you want to install it, you have to check out sources and compile it.

E

I hope that we can do that and then nothing else needs to change and then we'll go from legacy telepresences to not only producers at all, because the new temperature is just very different. It's it installs stuff. On the cluster side, cars it feels way too complicated and the way we were using it was already too complicated. It's like the command is like 15 lines. Long, it's it's very, very complicated. I wouldn't want to touch it honestly.

C

So um I we heard giuseppe's opinion does: does anyone else have a different opinion about potentially rewriting the iranian controller code as stateful set runner? Anyone object to that.

D

And we were talking about this uh here in emir and I think our kind of uh feeling was that it's better to start from scratch, because it also has the added advantage of not copying over stuff that we might not need so, basically simplifying, potentially so just a very natural fit to our strategy of like incrementally delivering value.

D

uh The other thing, like the big bank is something that we've been avoiding kind of in the past, so.

A

If we are going to rewrite every controller, does it make sense to take the opportunity to use deployments instead of stateful sets.

E

I guess we're back to the.

E

I think: it's it's it's the same conversation about dropping some features, I guess, or stopping providing some environment variables or stuff like that, uh honestly, like the way we, I think, when we spiked in a really the deployment, this deployments thing like the changes were minimal, which were because, like it's a part spec that we really it's the pod spec we care about, and I think whether it's embedded in a stateful set or in a deployment it really makes, doesn't make a lot of difference for us like in terms of like how we generate it.

E

So I think it's a change that we can do whenever we want. If we want to.

E

I think it was like we literally just grabbed for stateful set replaced it to deployment and it kind of works more or less, because the pod spec is identical.

C

Did the I know, there's a there's, a um a mutating web hook that changes the the pods so that they have uh that vcap uh index set right yeah? That will break if you start doing deployments, because it's.

E

They it it's kind of.

C

Hard based on that, that's it from the name of that name.

E

Yeah, so that will break because pods will stop having that number.

C

uh I think eric and.

B

C

Discussed how we might potentially just remove that index entirely and make that work, um but I don't we have to consider the backwards compatibility implications of that yeah like like.

B

You have explored in the past so you're, aware of, like you have to get changes into cli and everything. That's assuming integer based things.

E

Yeah we we had a very long thread back in the days and like we even tr, we we tried to see if we could produce numbers as weird as they were, so that we wouldn't have to change everything, and we were like, oh that weird, like five or six random id five or six characters id that kubernetes generate what, if I turn it into a number using some weird conversion. Can I do that and I think we we realized. It was a problem because sometimes we need to do the other way around.

E

I think we have to go from the number back to the thing and I think there was an issue with the mapping, so we had to give that one up as well. So yeah we've talked about this extensively. I.

B

I think, in terms of like immediate steps so like going from a stateful set runner to a deployment runner in terms of code, isn't that different, like we just like it even seems like something like the same code base could support both like it's like a you, opt apps into it and stuff.

B

um So, like I, I don't think in terms of like rewriting irene controller. The deployment versus staple set is like if we just defer that later. That would be a huge issue. Yeah.

E

That's what I'm saying like yeah in terms of generated resources. They look very similar, it's just in one case. It's called deployment. In that case it's called stateful set and in one case you can use that hook. In one case, you can't, but that's like a different conversation.

B

Yeah and like I think for the deployment thing, I think they just need a more target targeted like way to wire this in, like we'll have to like um get some changes into the cli um into cloud, controller, etc.

B

um Yeah, but anyway, that, like I, I totally agree with where danny's coming from there, like, um as others have pointed out, also like not only are the staple sets like kind of non-idiomatic, but they also have runtime differences around like placement and restart and stuff which have caused people headaches. So.

E

Yeah, the the cf app is a very unique beast, like he behaves like deployments in a way, but it has more guarantees at deployments, but it has way fewer guarantees that stays for sets. There's nothing in between. We have to write our own thing if we want it to be like exactly behave exactly the same, we need like our own controller, that orchestrates pods.

B

But that feels a bit overkill, yeah right, I think it's I. I also think, as apps migrate themselves to be more kubernetes native too they'll probably rely less on, like things like a specific instance index and stuff yeah.

E

We should somehow push thoughts that direction like try to.

E

I know, maybe even start saying, if you're relying on these things well they're going to be deprecated next version, we're going to break the behavior this way and kind of trying to nudge people towards a more like a more kubernetes native world.

F

Yeah, it's a good opportunity to get rid of some extra craft on the um crs as well stuff like the organ space. It's totally because this came from a place where there was one arena works um namespace and everything got created there. And so, if you wanted to see what belonged to an org you'd have to search by label. So that sort of thing is just redundant now, where things get created in the correct place. Initially, um yeah.

E

Yeah, I know so I think we have the the crd the new serial is is looking pretty nice. We just we're just copying the implementation over right so like we, basically we're gonna, throw away the arena crd we're not copying it over. Are we.

F

Yeah, I was thinking those things end up as labels on pods as well, so they're totally irrelevant as well.

C

Leaves a bunch of those as nil right now too, so I mean it seems like if we, if we're leaving a field as notes probably assigned it, doesn't have to exist anymore.

E

Yeah pretty much.

C

E

That is specific at that level of obstruction should not.

B

E

Reason to exist.

B

To be honest, adding cf labels isn't the worst thing, it's probably actually a net positive, because there's like plenty of like, um like third-party, like tools like like spotify, backstage and stuff that, like aggregate things by by label so like if someone using one of those tools were like, I want to see everything.

B

That's in the same like org like I know they have name spaces, but most of those you just pass it in a label selector and and that type of thing works so, like I I think, even if stuff is redundant, if it's in a label and not in the spec, it's actually not a bad thing. Necessarily.

E

Yeah, I think at the moment we have in the spark at least for tasks it's full. I think the lrp was cleaned up a little bit, but we didn't have time to clean up the task. So I'm looking at the tasks here, it's full of stuff craft like the app guide that kind of stuff which doesn't really belong there right.

A

Also, what do you think about the idea to distribute this reconciler as an optional component, so that, if you want to plug your own, I think.

B

A

B

The other one um I'll, let that talk yeah, that's.

C

The the the idea with these run and build workloads that we're introducing is that we could eventually get there. I think the thought process is that we'd start with like the default with no alternative and then once we've got the the defaults in place. We can start thinking about how to disable the defaults and deploy alternative implementations that maybe other people write.

C

So I think we're on the same page. There just might want to start without having to to, without considering the swap in workflow and then get to that after we've got it working. I think the idea.

E

Was we still ship it in the same code, but it's just for convenience, but you can turn it off just because this way you have one repository and we don't have the problems we have now with irene, but, like you have to change it.

A

Well, the thing is that we're probably going to put that controller in the same controller's binary the same monolithic controller, binary that our controller deployment runs. Don't.

C

Know if that's.

A

C

Way that we structured this for the for when we did the build workload, was we had this um kback image builder subcomponent? That even though it's in the same repository, we treat it as though it were its own project, so it um it shares the go mod file, for example, but it doesn't depend on any other files from the main project um and, and so the idea there is that, hopefully, we've done a good enough job of isolating it that in the future, we could make that a separate repository.

C

If we wanted to it's just a lot more convenient right now to have them all live together, so that all the tooling can assume it's there so make sense. I think we could do the same thing with staple set runner and then make that same decision down the road.

E

Yeah we could even build separate images for them and even if they're, in the same repository, you can still build separate images for them and then uh in either the install instructions. Or if we have some sort of automated thing, we can turn it off in the ammo.

C

That's exactly what we do for the image builder, so.

E

I think I we have a pattern to follow there that way, you don't even pay the price of having it compiled into the binary, although I don't think the price will be a lot like, even if we just compiled it into one big binary and turned it off at runtime. I don't think the price would be very high, like probably be fine but yeah.

E

If you wanna, do it super clean you build it separately.

B

uh Cool I'm gonna do a time check on on this topic. I think I think we're aligned, though um okay cool, uh so let's, let's move on um next one from giuseppe um test versus kind for repository test yeah. I guess we've.

E

Been having this conversation again, because the current tests are a bit, you know a bit weird like they.

E

Currently we test reports, we integrate repositories again, step test and test is not just doesn't have all the things running that it's that the real system does, and we've partially, I guess solve this problem by filling in with like fake implementations uh in the tests like we have a go routine that runs and pretends to be the organ space, controller or stuff like that, but it it's.

E

We still don't have the hooks, it's kind of bad, so we've been trying to do something more extreme like what, if we try to install all the real like to run all the real dependencies in those tests all against stem tests, but that doesn't seem to be working really well like it's kind of complex to to to gather all the dependencies and run them etc.

E

I think the natural direction is just run those tests against real clusters, although that makes everything a little more a lot more cumbersome in a way like you need some kind running and uh it becomes a little bit like the end-to-end flow. Where, like you, need to spin up a cluster basically- and I know that locally- that is a bit more expensive, etc, but in, on the other hand, I'm I feel like if we are integrating, we should integrate with the real thing.

E

Otherwise, we're not really integrating like at the moment we're integrating against a fake organ space repository, for example, we're and we're integrating with the thing that doesn't have the hooks like it's not the real thing. So it's not it's not integrating, so we should probably try.

E

I don't know, maybe experiment see how it works or because the alternative is, I guess, going in completely in the complete different direction, accept that we don't integrate, accept the risk of not integrating, and at that point just mark whatever we you need to mark and and have like unit tests.

E

At least you you don't even pay. The price of m test I feel like m test, is just a bad compromise. It's it's expensive as a name, it's almost as expensive as an integration test, but it was also not integrating, because it's missing all the things that he needs to have.

E

So I would say we should probably try to go in one of the two directions and settle the thing, because now I think we're not very happy with the tests.

C

I think it makes sense to experiment um I, although I I have to admit.

G

C

Share a lot of the concerns.

G

C

I'm hearing, um I think the way I think about it is that we're getting the full integration from the end-to-end tests, so you have confidence that everything's working together properly there um we've tried. I think we maybe even have some unit tests for the repositories we don't have viewing tests, but we've tried unit tests in situations where you're you're using kate's client and those.

C

I I'm really unhappy with the way this turned out honestly because you end up having to stub like the get and create and all these different methods, and then you have all these type cases inside of there and like you're. Not really all you're really doing is writing a unit test to write a unit test. I don't feel like that actually provides any value personally.

C

So the the thing I like about the testimony test is that you get a pretty high level of um of real it's of realism.

C

I guess but you're just how you have a firm boundary where you know none of the controllers are working, so you're testing your responsibility as a repository of I'm creating these objects correctly, and then there are cases where you have to simulate the controller to, for example, um if you're waiting on a state transition or something but again you're kind of just testing that boundary where uh the repository has a responsibility that when a certain condition happens, it responds this way, and so I I personally I know it's a little um confusing at first to understand those go routines.

C

We have that they're updating the state, but once you understand the pattern, I don't think it's that bad, um but that's just my personal thought and I'm familiar with it so maybe coming to a cold. It's.

E

It's, uh I guess it's several, it's a bit. It's it's a weird boundary. It's not just! I'm gonna create this idea. I'm gonna be fine. The city sometimes will like come back at me with mutations, and things have happened, and sometimes there is a actually there should be a hook or something like that and it's not it's actually not there. So it's.

C

Yeah the web hooks are definitely something that we haven't found a good solution for, like sometimes.

E

It would be nice to yeah yeah so.

C

E

There's no labels.

C

That are set for us and it's it's messy. I agree with you on that part.

D

The other thing.

E

I think of the.

D

E

D

Web hooks, it's a bit a little sidetracked, but speaking of the web hooks. I recently looked at those tests and like it looks like we have a complete application of units and m-test tests, because you have like, for example, web hook that defaults a value. So you have a unit test where you just call the default method, and you check that it mutates the object properly, which is fine. But then you also have the amp test test, which pretty much creates the crdn checks that it's properly defaulted, which is one and the same thing.

D

So I have created a chart to simply remove the unit test, because I think I don't need it, but that was a small side track. Speaking of weapons.

E

Yeah we could try it's just uh it kind of becomes a lot more complicated. Do we? We do not need any telepresence, because you really just need a cluster, and then you can run the controller locally and the controller talks to the to the api. So it doesn't really matter where the controller is running.

E

So that's good, but still you do need like a full-blown cluster, that's kind of annoying. It becomes a it's, not a self-contained test anymore. You have to run it with a thing that deploys the cluster spins it up. Just like we do with the entrance, which is the annoying.

D

Plan so basically we arrived again at this stop because we had the chore to experiment with uh what was it with kind uh rather than f tests. But I just wasn't convinced anymore, that it's worth so we tried to pretty much run like we. What we didn't like is actually these simulations, because they're confusing like if, if it's a test, saying that my repository creates this resource with these fields, that's fine and then there are no controllers. I get that.

D

That's fine, but then sometimes we we have this these simulations and we just wanted to get rid of them or maybe not get rid of them, but just run the real controller. But then this would entail pretty much all kinds of like creating real secrets and it just becomes very hairy, really quick, and then we realized. If we're going to do that, then we pretty much are going to implement the same thing as kind. So uh no yeah we should. We should pick, I guess a bit on the fence about.

D

I guess that's what, when we're bringing it up but like if we this, if, if we are going to rely on the end-to-end tests uh like because they're providing us with like guarantees that we're integrating and all that, do we even need those m tests, then, and can we just go with units? So I guess here is your point about ugly mocking. I guess maybe we can experiment with things there. There was an idea that we can use just the client from m test and just call the reconciler manually. So we can.

D

You know we can test error cases and all that I don't really. We never tried that, but we might experiment with things like this.

G

It's kind of occurring to me uh thinking about this is it like? Maybe the fact that the amp tests are so hard to write is a signal that, if maybe I mean I, I think in a lot of cases, we don't have any choice, because the cf api guaranteed that it would do something that is quite complex.

G

So we're not really supposed to come back from the cpcf api call until we've got all that stuff done, but uh in cases where the api shim is doing a complicated dance in order to like make a thing, wait for the thing to finish, and then do this other thing and like those those are the cases that are a little hard to to mock, and I guess maybe that ought to be a signal to us that like well. Maybe some of this logic belongs in the controller layer like if we could have a controller that.

C

Would just wait, you know it would just produce ready state the one that comes to mind with that pattern. Is um works and spaces right, and so we we do still have that simulation going on in the m test, but it's much much simpler than it was before when we were directly interacting with sub-name space anchors. I think now it just waits.

C

We just have something that fakes out the the ready condition becoming true right, whereas before it was like creating uh service accounts and doing all this other stuff right, um so I mean we have gotten better in that department, but maybe there probably still is room for improvement.

E

Yeah, I guess we could introduce crds that you know don't match one to one with the rest resources that aren't and do the stuff for us. But then how would you test those like and then because that would then have some logic to wait for something.

E

So you have kind of the same problem, uh because the other controllers wouldn't be running anyway, like.

G

Yeah, I mean, I think I think in some cases we we just run those other controllers right.

E

Yeah, I guess it's easier if it's just if it's a matter of running two controllers, instead of one, maybe yeah, because the way I see it is like okay, the shim relies on is a layer on top of a cluster with the crds and the controllers installed, and the works and everything.

E

So it's almost as if, like the crds plus the hooks plus the controllers, like all that stuff, is one interface and then the shim is an adapter. On top of that interface right and like mocking that interface is very complicated because there's all these mutations, but so we integrate with the whole of it with the whole of that behavior. Now we integrate with like a very partial, slash, reconstructed version of that behavior.

E

So to me, it's natural to integrate with a real thing for the repository layer, and the real thing in this case is the whole thing like it's just. I think, if you, if it were easier to do it, it would be a no-brainer, it's just complicated, because you have to spin up the cluster and program. You can just run our system like of controllers and hooks and everything easily locally and integrate against it.

E

Just like you would integrate against a database or imagine this being postgres like you could literally spin up the postgres and the postgres maybe has some. How do you call them like some uh stored procedures or whatever, like you, could just literally spin it up integrate against it? It would be easy like this is a. I see this like it's, mostly a technical limitation of kate's being very complicated to run locally.

E

I'm very complicated kind is actually fine, but it's still a lot more complicated than just spinning up a process.

E

I I I it's been a day of having these kind of conversations where, like there is no clear good way to go like, like everything, is compromising a lot. It's been frustrating.

E

But yeah, maybe we should try. I don't know see how it goes, see how that works out and if it's terrible, we try in a day of putting up a script or whatever and make targets see how it goes. And then, if we hate it, we throw it away and we wasted a day. It's fine.

E

Also I try to port over. I don't know a couple of tests see how the difference, what the difference would be, how much better they look like. Maybe by looking at it, we can convince ourselves that it's either a better or a good idea.

C

I agree that it makes sense to try and experiment. I think the only caveat I would throw out there a concern to keep in mind is that it's, it's really nice um for focus tests like that to be able to run them from the ide or to run them without the um without make. So if in the experiment, y'all can find a simple way to like set an environment.

C

Variable that says like this is the kind cluster I'm using and just run it um go only and not have it I'll, be basher or make file driven. I think that would be that would address a lot of my concerns about going this direction.

E

Yeah I mean: do you have issues with the entrance tests, not allowing you to do that so, like do you but like that means, we should probably change the end to ends as well. Then, like yeah.

C

In tests also take like 10 minutes to run so they're, not something I'm they're, not my inner loop they're. My.

E

C

Loop right but yeah.

E

Yeah everyone's.

C

E

G

I think yeah right, but we could. We could change that if, if we just decouple the to playing to kind with the running of tests,.

E

So, which I thought was already the case but like maybe you don't see it but like it, should be right.

B

Like the script.

E

Should deploy the kind and then call the tests with some information about the cluster address right like they should not be so, coupled that you can't run one without the other right. I always run the script, so I don't know what's going on under the hood: that's because we have a shell-based development process, but like uh that, should be either already the case or easy to do. No.

F

Yeah, I think, if you're going through a loop of ets, you do the deploy once and then keep on running, make test dte with skip, deploy, mvar set, and then you can just modify.

C

It yeah that's where you'll find the test code. My issue is less that it's slower run. It's more that you have to the only option I found for running. It is cli based and that's not my normal workflow. So that's all.

G

That's what I'm talking.

C

About like, if we could make it so that you can, these new tests can be driven by go as long as you have like the right environment variable set to say, like here's, the kind cluster I'm using and all the scripting does is set up the kind cluster and set that environment variable. I think that.

E

That should be the case already in a way, but maybe we do it the other way around, but like.

C

We might even be able to. I I'm sorry like I just read this little conversation, but um you might even be able to do this with the end to end test already, and I just haven't tried because it's it's again, it's like my outer loop um and.

A

C

Just doing an experiment I don't want to, I don't want to throw a big wrench in all this, so maybe maybe forget what I said. Thank you for being open to it, though it has ways to run.

F

On any um cluster, because we use the same script, to run it against gcp, so gke and kind so yeah, you can definitely flip it like that with mvas just needs to know what they're called all right.

D

Sorry is the same kind of cluster like if we end up running everything on a gun cluster, we could reuse it and we could have a pre requisite script that just installs the dependencies once because that's what you usually need and then you can even from an id or something should have a relatively fast thing- that pretty much does like rotates from the same cluster. Deploy really undeploy then run. Some controllers somehow should be possible.

E

Yes, I'm looking at the script and it's kind of possible, but like there's a bunch of steps that the script needs to do that the script does that maybe we should move into the go so that you don't need the script basically like it sets up a bunch of stuff like creates, like an account.

E

It sets up the fqdn root, namespace, all those things right. So if you don't, if you skip the script, you don't have those things and they don't work. So maybe that's the problem. We need to.

C

Yeah, I'm um I mean, I think it for me. Personally the intents aren't that painful, because I don't run them that often um so I think I if it does become super painful for me, I'm happy to write a tour about it. I I was just more thinking about like I think, if we have that in mind as we build new new testing solutions on top of kind it'll make this an easier process, but I again feel like I kind of derailed the whole conversation, so I'm happy to to yield on this.

F

C

F

By the way, we also did a fix last week, so it doesn't keep on crashing the second and third time you run it without recreating the kind cluster. So uh the registry thing was losing its images somehow and then k-pac was silently just failing on us yeah anyway, so you can keep on running, make test dte over and over again now and it's pretty stable, which is nice.

B

Cool all right, so um it sounds like there's a like chore, slash experiment to just like try this out on some tests. um uh Does someone want to take the action item to like write that up make it.

D

uh It's in already.

B

Okay, great right.

D

D prioritize, this short, like do other experiments, and then this conversation popped up. So I guess we just need to keep trying and bring it up again if we have more interesting results or raise the traffic or something.

B

Cool alrighty. Well, that sounds good. um Let's move on to ashwin's uh topic.

H

Yeah, so um I was working out of this chain box the other day, and I noticed the story about adding validating webhooks, uh to avoid duplicate, troll bindings right, and this would be specifically for the cube ctl user, and I wasn't too sure about the priority, and I saw that it was sitting fairly on top of the chin box right.

H

So to give you a bit of background, so if you're, a cf, cli or an api user- and you know you want to um bind a particular user to a particular rule- um we and if you use the cfcli or api to do it, we take advantage of the fact that you can't create um duplicate, kubernetes resources with the same name. Given that you can't do it.

H

We have this logic in the cf api, which sets the name of the role binding to be the shard 256 value of the role and the user that you're trying to bind with, and that takes care of making sure you know when the cfcli user is creating a duplicate one. It gets the same name and then the kubernetes rejects it, saying that it already exists right. But for the cube cuddle user, um you don't have to follow the convention.

A

H

You can give any arbitrary name on the role binding and then you end up with this uh duplicate role, binding between the same role and the.

E

H

Right so this story is about hey, you know, let's add a validating web hook, that watches the rule bindings and I was a bit concerned about the complexity of implementing this story and then the load it's going to put on our controllers right, because when we say we want to add a validating web hook, it's going to reconcile and queue up requests, regardless of whether the role binding is related to cf or any other.

H

um You know system in the cluster right because we don't make any assumptions about um the fact that the cluster is dedicated to running curry. If you see a fight that can be other apps and things that can be there and then this is going to reconcile on every robotic, that's being created and um yeah. I was, I was thinking hey. um Do we want to do this now or does it need to be on top of chin box? And maybe you know what? If we don't implement it and probably um pick it up later.

D

uh For what it was actually the shin box doesn't really it's not really ordered by any kind of priority or something maybe things are grouped together by topic. We haven't pulled this in the any kind of to do so.

D

I don't like we're. Not. I don't know if we're going to do it soon, because you're saying top of shin box like there's no such thing as.

A

D

Park there, so it's otherwise, thanks for your input.

D

Is it the one by danny.

D

Yes, but they duplicate roles, yeah.

H

Yeah, that's right.

B

Well, that's good.

H

G

Hasn't prioritized.

B

I don't think we knew that yeah. I don't think we knew that.

D

Well, it's just like working place where we I mean, I mean the product has been this way when it went.

A

Usually, when I create a chart, I just drop it in the chin box, without really carrying where it.

D

I think it's good, like it's good that maybe you just put a comment uh describing this briefly on this, so that we don't lose this context. I guess, and then we can decide based on this. We can decide to just uh nick it or maybe you know we can find some solution to this, but yeah. That's all I can say because I I didn't really mean to to prioritize this thing necessarily in the near future.

B

Well, that's uh that's good to know. I want to just like uh agree and elaborate on something ashwin said which is just like, as we were like thinking through this one like we were thinking.

B

It was actually pretty risky um for for role bindings, particularly to put a web hook on them just because of like the ordering in which things typically get installed or like a cluster gets set up, such that, if that web hook gets applied first, but our like actual web hooks not running or crashing or something, then you are like you're blocked from being able to like apply any rule bindings at all and and everything. So that's thinking through some of those things where we were like thinking.

B

uh Oh maybe this one is more troubled and it's than it's worth, but ashland added a comment to the issue and and stuff, but thank.

A

B

It might be a place where, like a more dedicated like cf role concept or something actually makes sense, that, like shields the the core resource or something.

D

Yeah, that's such a good point. I think it's not the first time where we come to the idea that maybe we need a role I think in front of the the natural role binding.

D

Yes, yeah, thanks for bringing that up, uh you're saying: there's a comment there, so I guess we'll think about it.

H

G

Awesome yeah, I guess the the other question. I I think that came up as we were discussing this is like. Is this really a chore? um I guess for me the way I tend to think about. It is like if it touches the end user experience like it changes it changes functionality right for the end user.

G

Then it's probably a feature and we should probably prioritize it in a feature backlog.

D

G

D

Make makes sense to me, you know, maybe it's probably a chore, because I mean we're just recording ideas with these charts.

A

And I guess we.

D

Were started, introducing these these concepts and all that and we did it for a thing or two and then it started looking more like a chore, but when I think about it, it's actually not. So I think you're correct about this uh yeah. I guess we can promote it to the future with the context in it and we can decide if we want to do it. We want to do it. Maybe we just don't do it, maybe maybe we come up with a proposal about another user-facing object.

A

D

Actual binding, but I guess we'll have to put more serious thinking in that direction.

B

Yeah I I added something to our stand up just uh so for for context um at a previous one of these, just that they mentioned chin box was getting long and on our side at least we weren't actually working on them, because we didn't pay attention to them.

B

So we've been doing this like tuesday, slash thursday, like uh more like community pair type thing that looks at any pr's if there are any from outside or just works through the chin box and we've been operating under the inception that they're, like ordered by I, don't know why we thought they were ordered, but we'll we'll tell people to use like their discretion and not just like do something immediately, just because it's in there so.

D

I guess if you wanna pick things from there I mean yeah. I don't know how we solve this because we split the backlogs. uh We could potentially arrange for some charts to be in the american to-do, or maybe we can just pick some of them. That you think is important and I mean dropping some line since like so that we know could also work.

D

I mean it's awesome that you took some time to look into into these. I guess we'll have to maybe uh yeah when we had one backlog, it would have been easy because one backlog means everything's prioritized, so you could just pick the next chore and you could ask questions and all that if you want to collaborate like this, I guess we'll have to put some more thinking into it, but I'm also fine.

D

If you pick stuff that you like from the from the gym balls, the thing is that things might not be flashed out at all might be just some vague id.

B

Yeah, I think that's just what we'll have to keep in mind and.

H

B

It do our own, like uh put some thought into something, and if it's.

D

Basically, if it's.

B

D

Description- maybe we thought about this- maybe I thought about it and fleshed it out, but if it's.

E

A

D

No guarantee is there: if, if it's not, if it's, if it's parked, there no guarantees pretty much so.

E

Yeah, I would have, or we'd have to introduce another column, basically like a shared chores thing like.

D

E

D

B

D

Parking lot, where things are pretty much, you know hanging waiting to be fleshed out and all that yeah.

E

Exactly so, it would have to be an extra column for the prioritized chores which we can share.

E

G

My feeling is that we should absolutely not do this.

D

Like if it's, if it needs to be.

G

Prioritized and it's probably not a sure right like and if it's, if it's a chore, then yeah you should be able to look at at the chin box and pick the thing that pisses you off the most and work on that right yeah. I think.

E

Here the problem is more, some are not completely fleshed out more than some are more important than others like. I agree that they don't have a priority in the sense that stories are their priority.

E

They could be still be ordered in I'll. I love to this is probably more important to be done, like it's probably good. If we do this first, but like it's not.

D

I guess uh something that we could do is we could mark the ones that are definitely not fleshed out or, conversely,.

D

G

D

Maybe yeah we can just draft label on chores and then, if it's drafted, just don't waste your time on it, it's not draft. Maybe it's worth picking up about priors I mean I don't know like uh coming from trucker a long time ago. Everything was kind of nice and ordered, and banker was like.

A

D

So that that's the priority I'm talking about, as I said, I would be happy for you to pick the thing that you like the most. You think you have context, but I just cannot guarantee that it's in a prepickable state, but maybe if I go over them and put drafts on things that are definitely bad or not. Maybe then you can just pick up non-draft things.

B

Yeah I like the idea of draft because, like we don't want to like prevent, like put so much friction that people don't add things to there, but if you're making us a chore and it's just like a title and a little bit just feel free to put draft on and also if you have a chore. That's like you really want to prioritize. I feel like you, can just talk to people and get it pulled into a normal backlog too, and then the chin box can be like what julian and people are saying is just pick.

B

What suits your fancy that is actually fleshed out.

D

I mean, as a matter of fact, we still do the interleaved backlog like chore storage or something like that, but that's still that still means that you can pick up things from the gym box. I usually keep a relatively short horizon, I'm sure, because we often change track like now. It's concord then something else it was evolved before and like it's always like. Oh, I have stuff to to catch up on.

D

Yeah I'll definitely I'll definitely have a go through the the stuff and stamp them out. The draft ones.

B

Cool uh all right- well, that was that was a good discussion. um Let's move on to the final thing from danny the pod security standards.

A

And this is just the heads of the that are deprecating put security policies with uh 125 they're going to be gone and they're going to be replaced with bot security standards.

A

um The big change is that, with both security policy, you can define security context on pod level, meaning, for example, you can limit the port so that it cannot run as root and stuff like that, while with port security standard, the configuration inspire namespace.

A

So there are three bot security standards. Privileged baseline and restricted to a restricted is apparently the most restricted.

A

So we are looking into putting those uh security standards on both the workload namespaces and the nice spaces that are running our core ports, controllers, api and so on.

A

This approach currently has the implication that k-pop ports are not going to be able to start because they don't set security context on their ports. We have raised an issue with kpoc to think about. It dropped some suggestions on ways to approach it, but yeah.

A

Just to let you know that we are looking into this direction.

F

This also means the responsibility for doing that. It was in irini, and now it goes totally into kirifi, who's, creating the workload name spaces and the re and arena or the stateful set controller won't be doing anything at all. Apart from creating pods that got the right settings on them, so they comply to the policy.

B

Cool yeah that all sounds good good good to know. I linked the kpak issue you had logged here um looks like they actually responded fairly recently, but I'm not sure they quite underst at least reading it. I'm not sure they quite understood that there's an issue with the the prospect that the build pods use, but maybe um at least shows up there they're like yeah, we should support restricted. So that's good.

G

Yeah, eventually, they'll figure it out right like as soon as they try and test with 125.

G

B

G

B

Yeah, I think I think 123 is the first one that we'll have this on um via the correct admission controller by default. Someone correct me: if I'm wrong, I think 122 you can enable it with a flag or something. But that sounds right.

F

E

Even 125, if you don't annotate the 9 space, nothing happens right.

A

E

The baseline is not enforced by default. Yeah.

F

It's just the controller is active well inside the.

E

Api yeah right: it's.

F

Turned on to look at the labels.

E

So it's uh very plausible that they will just go through their radios until someone some user tries to do this and it blows up.

F

If you look at a cluster with karify and all its dependencies installed, the only pod security policies are the two that irini set up. So I doubt if any of the other components are actually thinking about it at all, so it won't affect their testing really probably.

E

Yeah there might be like a soon as this thing becomes widespread and people start annotating the main spaces. There might be like a storm of issues for like all these projects that suddenly realize they've been running in in very insecure configurations.

E

So it's a good it's it's definitely a change in the direction you can just annotate up a namespace and have guarantee that all pods in there are running to a level of security that you at least understand so and it's been set by someone who knows what they're doing, because writing your own psps was a nightmare.

B

Cool well, um uh unless anyone has anything else to add to this doc or just a talk in general, I think I think that's it. I think we've another successful cf on kids working group meeting so.

D

I'm just letting you know that I moved that story about the webhooks to the box and it's not no longer put some comment on it. So good.

B

Cool. Thank you.

B

All right: well, I think that's it, then um I will see everybody later.

D

Thanks everyone bye.