Kubernetes KubeBuilder Controller Runtime and Controller Tools, 30 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes KubeBuilder 20200730

Description

KubeBuilder Meeting for 2020/07/30. See https://sigs.k8s.io/kubebuilder for more details.

A

All right, uh this is the q builder controller, run timing, controller tools meeting for thursday july 30th 2020. As a reminder, this meeting is being recorded and will be posted to youtube. Eventually, so don't say anything you don't want recorded for all of time.

A

All right um so looks like we have one uh agenda item for today from alvaro. So why don't you take it away? Alvaro.

B

Yes, so I basically have a question, so the background is that people apparently use what's in package builder, to create webhooks and what's in there is very little with regard to webhooks. This is about validating and mutating webhooks, which then resulted in someone creating a pull request to extend this, um and since I personally, never use queue builder to build a web hook.

B

I've looked at the history at this and found out that this functionality did exist in the past, basically using the builder to create a mutating or validating webhooks, but was removed by you in january.

B

So today we have some examples that do not use the builder, but that seemed to be like be able to cover the validating and rotating webhook case, but it seems that, at least for people that use this coming from cuperta the cli. This is not the best way to go.

B

um At least that's what the author of this prerequest commented in the end, and for me it's just unclear what the vision for webhooks is, if what's in package builder for webhooks, is just a reminder and should be removed. um Yeah.

B

And I think sully would be the only one who can answer this.

A

uh Okay, so I think the background on this.

A

A

Let me let me try to page the rest of this info information into uh my my memory, but the background on this is kind of, I think, was kind of that the builder. So the purpose of the the builder is supposed to be largely to remove extra boilerplate from the implementation, so we kind of wanted to make the, especially when there wasn't declarative validation.

A

So some a lot of this came about from before there was declarative validation, but especially when there wasn't declarative validation.

A

We wanted to make it kind of have an idiomatic way of specifying validation on your um crd and so the idea of being able to have just like a validate method on your crd object and being able to use the builder to just be like. Oh, this crd object conforms to the standard interface. Let's go with that.

A

That was kind of the idea around having the the builder and the current form of the builder for kind of the the generic case of I want to run a validating web hook against arbitrary, uh like an arbitrary type.

A

I think it wasn't clear what the builder provided over just directly like constructing the web hook, um and so we kind of removed. We removed that because it wasn't like. I don't see much purpose to the builder there. I haven't reviewed this pr yet so, but I don't know is that did that come through? Clearly, I feel like that was a little family.

B

Yeah, I think it makes total sense um for me. Most importantly, the author of this pier in the very end said that he uses queue builder, a user's controller, one time coming from cube builder, and he quote would still like to move forward with some design that jupiter can take advantage of that abstracts away the request handler, parsing and response logic.

B

um I've never used q builder, the cli, so I don't really know what's expected there and how the current state of affairs doesn't match that as someone who directly uses controller one time, I find it completely. Okay, the way the books are constructed today, but I don't know how that looks. If you come from cuba, though,.

C

Sorry, but by today you mean like uh that the the type has the web, the validate methods.

B

So that's probably another issue there. uh If I saw it correctly, two conflicting ways of do it. The one is this an example it's built in and then validating mutating webhook, where you can do whatever you want. You get the request and you can do whatever you want and there's something that somehow implemented on the type itself, which is what is transiently used when using the builder. The latter is actually severely limited, which is what this request was supposed to fix.

B

C

B

B

Maybe the issue is just that we should remove this. This thing, that's based on adding or having validation on the types themselves.

A

It has generated a lot of confusion where people have been like how do I, how do I get this equivalent on on the high level types, and then we have to point them at for how do I get this uh equivalent to the high level and built-in types, and then we end up pointing them at oh? Well, you just use that like kind of quote-unquote lower level uh primitive, which is what's in the the example for built-in.

A

I think let's I would be- I think it. We should probably take this as a sign that that needs some reworking, perhaps a bit at the very least, maybe just our approach to how we talk about it, but also probably the overall design.

A

um I would like to figure out what this person means by like can be used by cube builder, but I and I am a little bit sympathetic to the idea of wanting to apply the same. Like I don't care about manually, constructing the request response I just want.

A

uh I just want to have like kind of a similar interface. That's not a method.

A

um But I I'm not sure what we want to do there I have. I also will have to read through this pr.

C

So um so I talk with this person internally once and I think the main motivation to push this through was to actually get a client, which I highly suggested to not plot a client through uh or like a global client, um because the default implementation is on the types themselves, and so I think I think that's where this came from originally um so I'll say that, like in cluster api, like we're we're using the types as validation, defaulting, etc, like as it was suggested, it is pretty limiting.

C

Although uh I did like the fact that, like a client wasn't there because you don't getting potentially into a loop between resources, um so it has its pros and cons as long as it's documented, like I, don't have strong opinions to like push that out of the um of the from the types to like a different struct or something it will make it right. Now.

C

It's like a very clear, though, that, like, for example, for defaulting my books, like it targets this specific version of something of like a type um so in the new world like I guess it would be um not that explicit, but that's fine as well.

A

Yeah, I mean the- I think some of those are like kind of, as you said, that is a a little bit of a nice benefit. Is that, like the high level interface prevents you from doing things that are potentially bad and if you want to get, if you want to do stuff, like have a client in your web hook, you do end up having to drop down to the lower level stuff.

C

The other benefit that we saw it was that, like you, don't need any r back. If you don't need a client as well, which is kind of nice, because then your web books just become like static function in in some ways.

A

Yeah that that was kind of the idea to capture that that common case.

A

Yeah, I I think I would like to know at least because, like I don't know, at least for a lot of the cases that I've I've dealt with, and I think this maybe says something there's something wrong about our documentation or our api or something but a lot of the people I've talked to that have been like. Oh, I wanted to use the same interface on built-in types.

A

Just didn't know that the low-level interface existed.

A

A

Like it's not clear to me how much of a burden that is over over this right, like how much of a jump it is from returning, instead of returning error or nil returning, uh webhook.succeed or webhook.failed,.

A

uh Does that make sense.

B

Yeah yeah, I guess the main point is probably actually that we should document this, and I think that was also the point vince was trying to make before.

B

There are some cases where maybe you want the client but there's a good chance you're on your way to build yourself a foot gun. So you should be really sure you know what you're doing there so maybe making that ex that explicit in the docs, rather than implicit by making it hard to use a client would help you.

A

uh Yeah I I agree. We definitely should be more explicit in the in the book and in the docs and stuff too um all right, let's, let's follow up with this person offline as well or not offline, but like on on the issue as well um and write some of this down.

A

Anyone else have any thoughts.

C

So what are we doing for this particular pr like? um Do we want to push back or like take a step back and do a design, maybe in the next cycle and validate the use cases a little more.

A

Yeah I I'd like to just get a little bit more information because, like some of some of the stuff that the person has commented on, like some design that q builder can take advantage of is, is it's unclear to me what the user is trying to get at there.

C

Okay, yeah: let's do that and yeah. We can follow up.

A

Yeah, I I'd like to understand like um where, where the impression that cube gelder can't take advantage of the built-in web hooks comes from and like stuff like that,.

C

All right should we move on um father, you're muted, if you're talking, I don't know.

A

Oh no, I haven't, I was just giving people a moment to respond or anything but um right.

A

Looks like we have two more uh items that just got added, I think joe is on both of them. So do you want to start your shirt.

D

So the first one is just something we talked about last time where we were looking at adding a flag for zap time encoder, but we talked about basically like, let's not add the flag, but let's add the options on the struct to enable um setting encoder options.

D

um So we've updated that pr to do that and I think we're just basically interested in pushing that along and seeing if there's anything else left to do there. So I just want to kind of just. This is just a reminder, mainly.

D

But this one now, basically, like only has the options on the struct uh and then sets the defaults accordingly and uses the options and there's nothing to do with flags in this one. As far as I can remember,.

A

Okay uh looks like as of two days ago, I'm assigned so it should be on my list of things to look at so.

D

Okay, if there's nothing else there, then the next topic I have is about communities in general and specifically with 119, so we we started to look at updating some of our operators and stuff to start using 119s. We want to be ready for ga when it comes out. So of course, one of the first things we do is we uh try to bump our kubernetes dependencies and and all of a sudden, we see that control. Runtime. Has this incompatible blogger issue, um so there's a pr 10 11 to blogger, to support the new logger interface.

D

That's in mockertv 0.2, which fixes it such that things can start compiling in kubernetes 119., um but then, like. I have the question of like how exactly are we managing the versioning of all this like? If, if we merge something like this to master.

D

Does that like before 119 gas, are we then like? Are we able to still backport stuff to existing releases that don't have that change? I guess I'm curious. If we've done that kind of thing before.

A

Yeah, that's how we're doing that so, usually, usually, if we, um if we have existing releases, we will and we we start changing stuff on master. That's breaking we'll just use cherry picks to cherry pick to the stable branches.

A

Okay, so it does happen. Like cherry picks, do happen on occasion. um I think, for instance, um vince did a release recently of 0.5.

A

That has some cherry picks in it, um but yeah usually usually the way to do that is just you know, normal get cherry pick and then file a pr against the stable or the stable branch, so release dash and then whatever the stable right.

D

So the idea so with is, I know- we've talked about this before and it's a little cloudy. So it's the idea that basically, we release 0.6 supported kubernetes 118.. As soon as we want to effectively. We can merge a 119 pr and to master and then from then on. If we want something to be supported with 118 we're backboarding to 0.6, are we like kind of aligning minor versions to kubernetes versions.

A

uh We don't explicitly align minor versions to kubernetes versions, but, like version upgrades that are breaking right in terms or ver dependency upgrades that are breaking changes that surface something through our api end up being end up corresponding to major versions of controller runtime. So in this, in this case, like go logger, we have. We have part of this custom logger in our api um and so therefore it's a breaking change. We need to make this change.

D

Yeah, okay, so I guess, then, my final kind of wrap it up kind of question is like. Are we then, okay merging this logger fix on master such that users that are wanting to start trying out? 119 can compile.

A

Yeah, that's, I think, that's fine. Okay,.

B

I think that would be very good because there might be people that want to try it out and it might be helpful for both us and maybe even up some coordinators. If people start using this in their own development branches, and then we can get feedback rather than waiting until there's a final release.

D

Alfred, are you suggesting even merging the um the kate's bump, pr that I have that right now is pinned to like an rc version.

B

Of 119 yeah, basically, I think that would be a good idea.

D

I think yeah, so I guess that's another kind of question. If we do that and that effectively means uh we have to wait unless we're okay, making a tag on a pre-release version kubernetes. But I think that what that means is. We are effectively now for sure waiting until ga of kubernetes 119 before we cut a new release, tag for 0.7, which I think I'm personally okay with that. But that's maybe a question for everyone else.

C

I would, I would agree with that, especially because we have a lot of breaking changes and proposals that are like um trying to push through right so like um we could do a pre-release of attack like if we really wanted to, or I mean we could just tell folks, to use a commit if they wanted to try things out, which is probably fine. I guess.

D

That's what that's, what we're doing in some places an operator sdk and we're just like all right pin to the latest master. We know we have stuff on there that we need that we want to use. So if, if we merge in a release candidate 119, we can always just pull the latest commit on master.

C

Cool um before we merge breaking changes, um I'll take a look. If, like we should do a zero six release uh so that we don't have to backboard things, because it's like more work for for us, I guess and um yeah, maybe cut like a release on the current uh main branch and then yeah we can start merging in uh and opening also. I think we already have at least zero six branch yeah we do so. We should be okay.

D

If possible, I would like to get that uh encoder config option um pr on to a 0.6 release, if possible, yeah, which I think we had to tag that way. But um just if we're going to start merging other breaking changes, like you say, uh might be nice to see that merge before we merge other breaking changes. If we're going to do more releases on master first here about six, stop z.

C

Sounds good um honestly, I was just waiting for 119.00 to come before. We would open um the the master branch to breaking changes but, like I'm completely fine to kind of release and say uh this is now like on 119, it's a breaking change and we'll keep it rating before 070.

D

Cool that all sounds good to me.

A

All right does anyone else have anything that they would like to talk about today.

C

Yes, one last thing is: uh we need a release of controller tools elsewhere, right.

A

C

One I wanna yeah correctly: okay,.

E

C

I'm I'm pretty booked this week, but uh so apologies for that like, but it's on my agenda for next week, unless someone has more time to do it before then.

A

I think that sounds good.

E

Do we want to um lock the cube builder release on controller runtime and controller tools.

E

I mean there hasn't been much effort on on my part to to get the people to release out. I've been a little bit busy with the 1.0 sdk release, but um I think, aside from a docks change that there's pr and one or two other things that are in the uh v3 scaffolding milestone, I think we're pretty close to what to where we want to be for the people there 3-0 release.

E

um Like I mentioned, uh I think, last week there there might be a few issues that we want to take a look at and add to the 3-0 milestone.

E

um So I I have some time this week, so I can go through and make sure that those are added, but I'm certain that we're fairly close to being able to release that.

E

Does anybody have anything specific that should that I think, should go in the three-hour release.

F

G

I think the most important that I can remember that he I think it would be nice to be would be we upgrade the crg's ng yb hooks persons use the view one I think it is. It would be very nice.

A

uh Yeah, I I I think, that's a great point, camilla, it's a good time to do that.

E

Yeah and sully you- and I discussed a little while ago, uh some things that need to happen before that. I think that preserve unknown fuels being explicitly set to false is one of those things there's a pr open for that. uh I think the author needs to well. I guess we should also discuss like um what's happening with the v3 alpha plug-in, because all the changes that we want for 3-0 are going to be in that plug-in because b2 is frozen, so this preserve unknown fields. Ppr is going to go into the v3 plugin.

E

I would assume that the switch from v1 beta 1 to v1 by default scaffold in the file and the makefile is going to be in the v3 plugin as well.

E

D

Yeah, I think that was it is: is that a change that could be made to controller gen defaults, or is this something we only want to do in.

G

G

It's not impossible. Actually that doesn't say that I know maybe saw the hair knowledge more, but I believe that we need to change the templates because we have the webbing hooks as well. uh We we need to pick reach the every hooks version too. So they'll work hound to customize the make file. We do not attend to all scenarios.

E

What needs to be changed for workbooks.

G

Very soon the same, it's the same because these, uh if you look there, there has a puji, passionate guy trying to push a spec to ignore the defeat you know. Happy is the same. They were the webbing hooks resources, always up the bridge in the person and it's together with this energy.

E

Okay, is there an issue open for that, specifically.

G

E

Okay, well um that yeah, if that's the case, that needs to be changed as well.

F

Okay, thank you.

E

So it's yeah so ideally for uh in my opinion. um Ideally we should have uh the v3 plug-in on beta by the time that we are cutting the 3l release and make that the default.

E

I think in as far as I'm concerned, beta means bug fixes only um and I mean that's kind of ambiguous when it comes to plug-ins, but um I I think that's like stable enough that we can make it the default.

E

Otherwise we can just straight up, make three stable, but uh I I think it would be really nice to have um this new plugin, be the default in the three-hour release. Otherwise, there's not really too much that we're releasing in 3-0. You know.

G

The other thing that maybe would be nice discuss uh would be the binaries, uh because I believe that you, for now we are shipping the kubectl into the other, requiring bananas to to one to set up the invitations with the kobe builder binary. So if you, you will change that.

G

I believe that it may be nice for three as well, because it should be a break change, maybe because, when we remove, we need to remove the code, for the release has the scripture that he did the release and he built the binaries, and you put it together so for v2, these are required and we can no longer required for victory engineering. We can, when we will hem over the video we can remove, which is called for any.

E

G

E

Is that a predicting change, though, because that's that only has to do with what's packaged in the released hardball, not with the actual binary itself.

G

I think it's so because, for example, okay, we do a release and we don't ship cheese by nervous with kobe builder. Okay, uh when someone yeah, I think, if someone installed the old binary, they you'll be there yeah yeah, we need yeah. I think it's fine, don't you broke the video? It's true sorry for that thinking better, because we you should have the binary one.

E

So I I don't necessarily know if we have a solution for how we want to hand those binaries over to binaries over to users, but we do have the scripts in the main repo that users can just strip download those binaries pretty easily um it. I don't necessarily know if it makes sense to be packaging, those binaries with the cougar binary itself. I don't really have an opinion about that.

E

um If anybody else does, then I should definitely talk about that, but I I don't think that's a release. Blocker.

A

Yeah we did originally, we originally packaged them for convenience purposes, because it was convenient to be able to just get all the stuff you needed to start running your tests and stuff in a single tarball.

A

I know it has since caused some confusion around uh like how how you're supposed to like upgrade your test dependencies and like people, don't know about the existence of the alternate links of the of the of these separate test, binaries and stuff, like that.

A

But we should probably make sure we have a good flow for for how the users get the binaries and teaching the users. If, when we remove.

E

Right so you know, I believe you merged the script recently that will just download binaries that have fixed cube and lcd version um pretty easily, and I think at very least directing users to just like download that script from the release tag and run it, and then that will update their test. Binaries.

E

I think that's a reasonable flow.

G

I think we should jump structure that you know when you call you know making fire targets has a question for that as well. So if we could move it, which is before the release, it would be nice because today v3 waking, we could remove it from the release.

G

The binaries uh ng, don't ship anymore, with a complete builder one and the users will have a way to you know things you'll be obstructed for they. They don't need to know about that.

F

Unless they would you like to change things, I think well you're talking about.

E

A make rest uh make role in the uh cube builder repos make file.

G

Yeah, if you see the search, see the issue 686.

E

G

E

Personally, don't think it belongs in a makefile because, like so let's say you put it in the builder repos make file, then the user has to download the entire cue builder repo to run make.

E

Why not just have the script in a very obvious place and talk about it in the docs and be like pearl raw github user content at the.

G

E

G

The problem is, if you follow the kickstart and you don't have the binaries set up in a specific place when you run make a docker.

G

It is your fail, because you will call the make it test into the make it test. It requires that the inverters be configured with the binaries. So now we have the scripture, but in my opinion, users she should be transparent for the users. Do you know it's? I it's painful. They need to.

F

G

I think so why they need to care about it. You know the configuration, unless mainly for new users, uh I be so. My idea is like just the same steps that we can suggest in the documents abstraction in a making file targets. So when they run they, we don't need to change the quick start now. You know always working in the same way.

D

I'm wondering if I think these two things are not mutually exclusive, like we can have a doc. So, in my mind, like m test is like a controlled runtime kind of thing right, it's it's a controller library. You can use control, runtime and test without ever having touched q builder. So it makes total sense to me that we would have some dock and some like really simple script, that you could use as like a controller runtime user to just go and get the amp test binaries to make it easy.

D

But it also makes total sense to me that if you are using queue builder or operator sdk whatever that we can easily scaffold, we do this for customize already so like. Why are we doing it for customized, but not for these other binaries? So I would probably argue yeah like it doesn't hurt anything to add it to the makefile we've already kind of have that precedent for other binaries that are dependencies of running make targets. So what's one more so I would probably advocate for doing both.

G

I think I just.

D

Said would you like.

G

It my I just sorry the congestion. My idea is like I'm a new user. I don't know nothing. I want to follow. Follow up with the quick start. If you don't care about configurations at all, then if I would like to change the setup, I can go in a document and understand how to work, and you know, do the things as.

C

I wish just to add to that to that a little bit I had a question, so I was reading the the link pr in the script, but it seems like now. The script doesn't download pre-built binaries, but it builds them. Is that correct.

G

This is a apartment that she we have before mac uh infertility, data cooker nets uh service is not provided for for mac os, so we need to download the project into build locally. This is the same solution that we do now to generate. The could be builder binary, so was just she, you know moving through switch to support.

C

Yeah so arms, so um we have been using a lot the published, artifacts and, to be honest, like it's, it just seems like a step in the wrong direction to build locally all the time that you're running a test, the entire kubernetes distribution.

C

um So if it's possible to keep publishing the artifacts and use those like um I'm happy to mundane them, we use them pretty extensively, especially in ci as well. Yeah.

G

I love that. I, I really think it's a circuit download to the browser to build it locally, to have the binary if it's possible download it. You know it's like why not japan just doesn't have for mac for linux. Sorry.

A

Where, yes hold up sorry, I I may have missed something: where are we manually building kubernetes locally I've? I've linked.

C

You to this like so this I guess like it's only for darwin, so it's all on mac, but uh so I mean, if I have to do this every time I do make tests. Well, that's not going to be reasonable. Why did.

A

uh I think I missed something, but we like we definitely have a process for for having ci builds already of of of kubernetes of the kubernetes test artifacts for darwin. It's not. I think we should fix this script. I I agree. I don't know, I'm not sure what happened here, but uh that probably shouldn't have.

G

Yeah happened problem. The problem is that for mac os, we don't have the binaries available uh to download. So if you're looking the last link, that is putting the search when we do the build of the kobe builder be not binary to do the release.

G

We do all these steps, so the easy way was provides a scripture that you do the same and you do just once uh after you have the binaries. You don't know anymore, but I didn't like the solution as well, and I would love that we could have the binaries to download the binaries only from some place, I'll.

H

Interject that uh the kubernetes binaries for linux are published by like as as tars as part of uh building kubernetes the api server binary used by coupe builder and believe co-builder does a darwin, build of it and does publish it at some some place because it because the darwin tar of coup builder, like you, had the api server binary compiled for darwin in it. So I think what we need to do is change the script to by default.

H

um Pull the api server binary from wherever it is being hosted.

A

Yep it's currently, so I can give a quick overview of the process for people on here. uh We have a special branch called tools, release which vince has helpfully linked.

A

This branch, whenever it gets changed, automatically gets picked up by google cloud, builds as part of the q builder project. The configuration for the cloud build is listed in in the branch as well. It automatically gets run and then it gets published to a cloud storage bucket, which is linked. There's a there's, a like go.qbuilder.io link for it um in in the book uh which I can add, but you can also just it's under artifacts all cologne.

A

I will paste the link, but you can see you can go and browse these and we, basically, whenever that branch tools, release gets updated, we publish uh darwin binaries uh as well as uh linux, binaries.

C

Yeah, so we should definitely keep doing this, uh because this saves a lot of time and it's really yes,.

G

Definitely I agree. I agree. I didn't fortunately have in some place to download just the binary you know so to solve the problem for us easily and run in the mac.

G

We fill out chinese installed, the kobe builder binary. I did it, but she I will fix this cryptic and I think it is would be something nice do. You know we do before the three release. You know far down ship more together, the binaries with we haven't touched. They could be building one and the others. They know we are able to download each one.

C

Instead, yes, for what it's worth like, um we have the scripting cluster api that you just specify kubernetes version which has to match one that we have published in qbuilder and it downloads those uh binaries locally and sets everything up properly. It only downloads them if you can't find them as well and if the version matches.

C

So I can link to this. If you want to reuse it, but um it's pretty much already there and has been working fine. So um when we run make tests locally, for example, it would actually run to the script first set the entire environment and then uh like a log it out like it's setting the environment uh and yeah adapted a little bit like yeah.

G

It doesn't, I think, that you'll see that it is descriptive for the binary they could be builder by nato today. Can you see the exports.

C

G

Yes, so it means that you need to have in user local kobe builder, bing install it, otherwise it doesn't work.

C

We can change these paths uh like as as much as we want like. We create the temp directory anyway, so it doesn't so like this. This doesn't require you to have q builder it just under cube builder bin. Just I guess, like I don't know uh sorry. I think you wrote this a long time ago and I copied it, but uh yeah like this doesn't require cube builder. In fact, I don't. I don't have q builder running locally, but the make file that we have uh the test target.

A

Yeah, those are those are just. Those paths are just the default from a long time ago, when we started working on cue builder, that's kind of the default location where q builder looks for things is, is optical builder, bin yeah or user local cube builder bin, but we can yeah.

C

I think uh the one that I should actually has it creates a temp folder yeah. It's under attempt right now, but like the test one, then it just sources that it runs those two things vegetables and set up, and then it runs go test.

G

Yeah, but to the feta tools, you know they could be what they're being, and I don't want to have the kobe builder beam to have the other beings. You know.

C

It doesn't it doesn't it doesn't, it doesn't require or download the q builder binaries. It only uh has cube api server, cube, ctl and ncd.

C

This is just a name: it doesn't it just under a folder, but this folder does not contain you builder, because we create the folder here. So this could be anything yeah.

A

You could change that path to anything.

G

Yeah, no, yes,.

A

Why don't we move this? The rest of this, I think, we're. Why don't we move the rest of this conversation to an issue just so that we can capture this more concretely.

G

I think it is exactly enough anyway, yeah.

A

uh All right and I think, we're running low on time. So uh does anybody have any last minute things that they would like to talk about.

A

All right, if not, uh I will see you in a couple weeks and as always, you can find find me on slack if you need anything or any of the other approvers or anything all right, see y'all in a couple weeks.

D