Kubernetes KubeBuilder Controller Runtime and Controller Tools, 20 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes KubeBuilder Meeting 20210520

Description

KubeBuilder Meeting for 2021/05/20. See https://sigs.k8s.io/kubebuilder for more details.

A

All right, hello and welcome to the q builder controller, runtime and controller tools. Meeting for what days I probably should have looked that up before thursday may 20th 2021.

A

As a reminder, this is being recorded and uh will be posted to youtube. uh So you know don't say anything that you don't want posted to youtube for all the world to see uh all right. So it looks like we have a pretty short agenda today. um So we have two items. I think the first one comes from kevin about the conditional controller design. So kevin um I'm gonna hand it over to you.

B

Thanks sully um yeah, so this is something that I think has been in discussion for a while, um and the kind of design that was approached a while ago uh hit some hit some hurdles with with uh with kind of some upstream stuff.

B

But I just kind of wanted to first kind of revisit that and and before diving into the actual design, just kind of discuss the use case and try to understand if you all in control of runtime are supportive of of kind of supporting this, and so the idea here is that more and more people are writing controllers, where the controller is not running on the same cluster that uh that it is controlling and- and you know it could be controlling uh multiple clusters so like when this is the case.

B

I you know the operator admin who writes the controller, who runs the controller manager, is different from the the cluster admin and when that's the case, I don't have any control over what crds are installed on the cluster. um And so, if uh the crd, that my operator is watching isn't installed when the operator starts it'll, just air out with a with a metanokin match error or if the crd gets uninstalled during operations, um the informer will just spam logs and spam.

B

The the api server with with uh list requests that'll always fail, and you know, could you know ddos the api server effectively um and so basically, what I'm trying to ask is: do we want a solution for uh gracefully handling when a crd gets uninstalled from a cluster that my operator is operating on and and if so, then that's kind of where the design that that I've offered in, uh as you can see in the dock, um and then I can discuss more about that.

B

But I think I just first wanted to start a conversation around that use case and see if we're on board. For for uh that or if there's you know, objections to.

B

C

So I just uh I read through your proposal: real quick just in the few minutes before this meeting started and the discussion started. What was.

A

C

To me, in particular, is trying to to map this to the multi-cluster story.

C

um I was 100 clear again just on a quick first reading of this, how tightly well, how comprehensively we want this to fit into a multi-cluster story. I've, admittedly not kept up recently with what controller runtime is supporting uh as like a first-class kind of um support for accessing multiple clusters at the same time, but that seems like the critical point, the critical side of this to me.

C

If we were just talking about a single cluster like I'm just running a controller cluster, watching resources in that cluster and on rare occasions, apis come and go, and I need to deal with that like that seems like we might take different or simpler approaches than what's here. So I don't know, can you maybe elaborate a little bit on how you see this fitting into multicultural support in general.

B

Yeah, so I think I think you're right that uh the the guidance that has previously been given on this use case is definitely use like a simpler two, like uh run different different operators, kind of approach, if you're only running on one cluster and kind of the need for for this really comes from from that that multi-cluster uh case and- um and I guess the I so I think I think the way that that that we've been thinking about this is that there's there's no good multi-cluster story and controller runtime currently, and hopefully something like this can serve as uh kind of the first step in making multi-cluster uh more supported.

B

I guess solid, correct me if I'm wrong or or if I'm saying anything that that doesn't make sense, but um this you know the the current in in the multi-cluster case. uh Basically, if you wanted to do something like this, you would need to have some sort of orchestrator of of controllers that can get kind of clunky, and this provides a a uh you know, a built-in way to handle that.

B

Soli what I know, what would you add there.

A

uh Yeah I mean I can I can well so to be clear. We've had like a little bit of previous exploratory work around multi-cluster.

A

um I think there was some initial work around uh separating the cluster objects, kind of, like the cluster connection, details from the manager object so that you could theoretically instantiate a manager uh and and like have clients from that manager. That connected to multiple controllers, um I think one of the things just to kind of supplement what kevin said is one of the one of the patterns that we've we've seen with some of our um internal users uh is, is kind of a an approach where we have yeah.

A

We have like a multi-cluster manager that talks to that, like um has a runs, a bunch of sub managers per cluster, um basically, and so like the idea of of crashing uh the entire binary.

A

When an api comes and goes, is like that's not, you know, that's that's not really going to fly for that use case, and so like one of the things that we would like to do is like kind of make that use case a little bit more viable um so that you can have that kind of you know some sort of, and this is like a pattern.

A

I've also seen externally for for some of for for a couple of users before um like we, you can have that multi-cluster manager, binary uh and kind of just like easily not have to worry about apis coming and going.

A

Does that? Does that make sense.

C

That makes some sense to me. um I wonder if you could comment on like. Is there a viable option in there, based on what you've seen to have a boundary at this sub manager?

C

Layer where, like I might naively, just think, I'm going to create this sub manager that is going to interact with a particular cluster, so I think there's a sub manager per cluster that I'm talking to perhaps, and if something changes about that cluster, like new apis, appear that I need to start dealing with or apis go away that I need to stop watching.

C

A

C

Restart that manager, like not worry so much about reconfiguring and you know, doing these kind of changes. But what, if I just actually let it exit or return out of that manager and start a fresh one?

C

Is that possible.

A

I mean you you we could you could theoretically do that um the places it starts to get more clunky is if you're, if you have like multiple controllers running in that manager, sharing a cache, which is a pattern that we have to some degree encouraged right like run all of your managers in a binary, because they're using some similar resources, like you, have to control or sorry run, all your controllers in a single binary, because you know, like you, have two controllers that need to look at pods might as well share a pod cache uh stuff like that.

A

um So, like you, you're, theoretically, just restarting is fine. Practically that can be could be a disruption of service. um It's probably not a huge deal.

A

um It's we would also like part of part of the work like uh I think kevin has put in is making the flow a little bit easier. So you can declare like this controller is conditional on this api existing or whatever, and so I think in either case. We'd probably need to have like doing that right now, without like kind of support and control runtime, we found to be a little bit.

A

Clunky um so like in either case, I think we need to have support for that like we need to, or we want to like make that a little bit easier. I think at least um the under the hood details. Theoretically, we could just restart the home manager.

A

um I think it's a little bit more clunky, but.

C

Yeah, it's definitely more clumpy uh it. What what I like about! It is like the just complete simplicity of it um that and I think it fits other declarative kind of patterns and immutability kind of patterns like when things change, rather than trying to keep up with every incremental change. Let's just you know start something fresh. um It also assumes that these are going to be rare. Events like apis coming and going from a cluster is presumably a rare thing.

C

We know that our control, our entire process, is going to get periodically restarted anyway, just based on upgrades or getting you know: node maintenance getting migrated from one another like this kind of stuff, so like occasionally you're having to lose out on the cash and start fresh, especially if there's only one clusters worth uh seems like an acceptable thing.

C

If it's worth it, maybe it's not worth it, and maybe this kind of um deeper approach to a deeper understanding of what apis are coming and going and what controllers and what caches uh correspond to the apis and trying to like carefully restart the right things?

C

um Maybe that's worth the effort, but just wanted to at least explore what I think could potentially be a simpler alternative.

A

Yeah, that's that's a that's a fair point to bring up. um I think. There's there's also like these are like there's a there's at least one kind of optimization benefit to this approach as well, which, like it is kind of strictly an optimization, and it um only applies in certain use cases, uh but the um the the.

A

If, if for some reason, you are concerned about the number of requests you are making from your um manager binary out to a given api server kind of hot looping on an api that doesn't exist, uh isn't great and doesn't uh in in certain cases, and so like the other nice thing about this here is: is it uh like compared to some of our traditional approaches? Is it stops at restarting? The manager would probably also help with this we'd.

A

Probably be you end up using the same signals, and then we just have to block the manager. I guess block the manager from starting or block the particular controllers from starting uh until until the api appeared, um I I would be. I think I would be idly curious um to see how much actual overlap in the code is, because we'd probably still want like a lot of the work kevin has done.

A

I think this pr is plumbing through uh figuring out how to plumb, through the signals that we need to deal with like stopping and starting the controllers, and we need those same signals for the manager. I suspect a lot of the complication uh or complication. It's a lot of the a lot of the deeper approach would kind of have some overlap with the restarting the manager approach. Just from that perspective of we still need to plumb through the signals we still need to be able to detect hey.

A

This thing has gone away: hey we're waiting until this thing appears stuff like that.

C

Yeah, that makes sense, I've definitely seen, and I think, maybe even a couple times. I've done at least half of that pattern of, uh and probably some others have seen this as well like a particular controller at startup like in your main.go at some point, you'll use the discovery, api and, like put it in the loop, like wait for some api to appear if it ever appears before starting this controller kind of stuff um that side's relatively straightforward, probably the api going away, is more difficult.

A

Yeah it's! It is yeah that it is it's straightforward, but it's nice. I think if we see it we're seeing it enough now, at least I don't like, I can't speak for everyone else on this call, but like it's something that uh I think I've encountered from a lot of our internal users is like that. That kind of desire to wait and so like encapsulating it in an easier to set up a pattern, seems seems like it would be valuable.

A

Even if it's not you know terribly complicated off the bat just making it a little bit easier and being able to tie in with the stop right.

B

um Okay yeah, so it sounds like um I can.

B

I think that the restarting the manager idea is is interesting and worth uh me looking into if that simplifies it or just kind of at least spelling out in the design, why that might or might not be better or simpler and um I'll try to spell out more also around the the kind of multi-cluster.

B

uh You know how this fits into multi-cluster or like what it would mean and um and probably go from there and maybe revisit when uh when that's more fleshed out, but also feel free for anyone to take a look at the design and the proof of concept itself and and offer feedback on that.

A

All right, uh unless anyone else has any comments, I think we can probably move on to the next thing.

A

um This should be, hopefully click, uh so um I don't know if, as as you all may or may not have noticed, there is a new tool in the controller runtime library uh for helping to manage nfs versions.

A

uh So for people who maybe aren't familiar, we stopped bundling and test with a cube builder for keybuilder v3, um which just released a few weeks ago.

A

And right now, cue builder projects are generated with this uh scripts or are our use this script that lives in controller runtime the shell script uh to download appropriate versions of of um the m test tools like cube, api server and xd and cube cuddle, uh and this is not great from a number of reasons.

A

uh First of all, I think perhaps most of us, if not all of us, can be in agreement that uh running arbitrary scale. Ships from the internet uh is is maybe not the best, um and you know there are all sorts of weird issues with shell compatibility that occasionally hit and stuff like that, um and the other thing is like.

A

Historically, it's been hard to discover uh what versions of of m test binaries we have built and what's available for download and kind of managing those, um especially if you're like me, and you need to flip between versions on a somewhat frequent basis, debugging issues, seeing if something happens with mt with one version of the cube api server, but not the other, or something like that, uh and so um we went and rubbed this tool set up m-test and the idea is hopefully it will eventually make its way into cube builder, but in the meantime, we'd like people to try it out.

A

So I can give a little demo of how it works um and talk about how it works and yeah see. If anyone has any questions, let me let me share my screen share a window great. uh Can you all see this well font size, big enough yeah? That's fine! For me! Wonderful all right! So um all right! So it's! The idea is hopefully it's pretty easy to install it lives in its own um module inside controller runtime, so you can just go install it. I already have it built.

A

uh So it's that was a quick install, but it's actually it's not huge. So it's pretty quick anyway um and then uh it's you know on our inner path, like normal um all right. So if we, if we run the command, we can kind of see how did some help in some examples.

A

uh But the idea um is that we can. We can go and list so m test list. We can see all available versions you can see. I have two versions installed: 119 and 120, um and then I can say use- and I can say like 1.20.

A

A

You know what uh that's gonna cause an issue. I have a shell script set up that that that wraps this a little bit for my personal environment. Let me uh let me actually just use the binary directly, so you can actually see what's going on, so uh what you can see um is that it kind of downloads them into a cache. The cache is operating system specific, so it'll put it in the appropriate place for your operating system for app data.

A

Basically, um and you can switch between these versions easily and it'll print out, uh you can tell it's either print out information um or you can tell it to print out a path just the path or you can tell it to print out a little shell script type thing that you can just source like like that um you can. This will accept all sorts of selectors.

A

So, if I want to, like you know, say uh the latest version greater than 119 or um you know, if I want to say like about 119, I can say 1.19.

A

uh And I'll select the latest version. If it's not available it'll go download, it it'll attempt to go download it from our our kind of gcs buckets where we host all this stuff and then, of course, there's also a delete command.

A

If I wanted to clean up versions and stuff like that yeah, so hopefully that should make it a lot easier to manage your uh m test binaries and make it so that you're, not you, don't have like if you work on like five or six different projects, you're not like, and they all use 119 by default. You don't have like seven copies of 119. You just have this one centralized copy, that's sitting around.

A

Does anyone have any questions?

A

Oh, I should mention too, for people who are concerned about offline usage. There are flags and environment variables that can configure it to only use offline versions or to look at a different source or allow you to side, load, kind of side, load tarballs in the format we expect into kind of the internal cache uh so that you could theoretically side load. You know private versions or if you wanted to work in an air gap, environment or something like that.

A

uh Anyone anyone have any questions.

A

See some examples.

A

A

uh All right, if not um hopefully, please uh give it a try. Let us know if it you know, breaks on on your operating system or, if there's some weird use case, that doesn't work for you or if you just find some sort of flag completely unintuitive or something. Let us know um we're currently considering it kind of in an alpha beta stage and hopefully we'll promote it to staple ish as well and start using it in q, builder and etc.

C

A

uh And with that, I think, unless anybody has anything else, they want to say pause for a moment just in case.

A

All right uh have a wonderful rest of your day, and I will see you all in two weeks see ya thanks.

A