Kubernetes SIG Cluster Lifecycle, 9 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cluster Lifecycle - Cluster API Provider Nested office hours 2021-03-09

Description

#sig-cluster-lifecycle
#capn
#capi

A

All right, good morning, everybody, uh this is the uh march 9th uh cap in office hours. um As always. This is recorded and posted to youtube. So don't say anything you wouldn't want to be broadcast to the whole world. uh We have a couple items on the agenda, so I think we can just get started, but actually before we do do, we have any new folks that haven't been here before I'd like to do a quick introduction.

A

And if you would just unmute- and you can start talking.

A

Cool all right uh as well, if anybody is, uh if you can go and add yourself to the attendees list, I'm going to drop the actual google doc for this in the zoom chat.

A

All right, so we can get started ciao. Do you want us? Do you want to talk through the two items that you added to that agenda.

B

Yeah, um okay, so I would like to discuss two issues. The first one is when I implement the nested etcd controller, we wanted to use some existing templates like when we create the etcd parts. We need to know like what should we um pass into the command line? Something like that. So I would like to know how can we refer to templates like? Should we put the just the pod templates of the etcd in our caption repository or how?

B

If layer, should we use any magnesium that allow user to specify uh what templates they wanted to use? um So I think we used to we.

B

We talked it talked this before and we decided to use the uh the common spec from the could be builder declarative patent, the repository, but I I I discussed with chris last week or two weeks two weeks ago I forgot, but uh chris you you mentioned that this feature is not stable right now, right, it's, they are going to change how they are going to use this common spec, like the version and channel to refer to the template right. So.

A

uh No, it is so from their side of things. It is, it is standardized. um Okay, it's more just it. So I think what uh that might have been my mistake on how I communicated that. So um what I was thinking was that we could go forward with what you did for the fcd implementation and then- and we could merge that, and then we can try to realign a little bit more with the declarative patterns as we go forward. um So basically don't implement anything specific to that.

A

Just have the manifest in there like you currently have in the scd implementation and then what we.

C

A

We can go and use their their uh file based or directory based structure to actually load those manifests in, and then we just populate them from our reconciler. um That's.

B

What I was thinking I see okay, so um could you remind me how they totally achieved this like? I noticed that there are two fields: one is the version. One is the channel version means. What's the version of the specific component like 1.2.3, the channel means if this release is stable or nightly, or something like that, and you just mentioned they use things like directory directorate to organize all this version in a channel. So does that mean we need to create directory inside our caption repository or we need to host it someplace else.

A

Yeah, so let me actually share my screen, I guess um and I'll take you on on a tour as I have used it and know about it. I'll just go through the examples that they already have. Let's see share my.

A

Screen builder declarative patterns cool all right, can you all see my screen? Do I need to make it big or anything.

D

Like that cool.

A

All right uh so, basically we're looking at uh the declarative patterns project for their guest guest book operator. I'm just going to go through this because it's already pre-built, so what they actually do in here, that's a little bit different is when you actually set up your controllers. Your controllers are written much more lightweight than what we have um in terms of the only thing it has to do.

A

Is it kicks off like a common reconciler and the common reconciler goes through and it can do object, transformations on any of the resources and you can pass it in in any functions that can go and change any or mutate any of the actual manifest the raw gamma manifest files for you. You can set owners on it. You can set labels so on and so forth. You can.

C

Preserve the name.

A

Spaces anyways, you can read through all those and there's a bunch of other functions that you can pass into this init for each component.

A

Then, if we go back uh how they actually structure that and how you actually point to it is the in the types you have like what you already know about the common spec and patch spec. So the patch spec allows you to override anything. You want kind of like a customized patch, and then the common spec defines this the version and channel now in version and channel how those actually happen are set up, is there's this channels directory at the base of every project and in essence these could be.

A

These are just loaded into a into the docker file.

C

Okay, I don't know if.

A

Long term these will be added in with any of like the new 116 um bin data kind of styles, but for now they're. Basically just added into the docker file and you'll notice that there's a file called stable and stable represents the channel stable, and this has a pointer to a version and so.

B

A

Is how it how it gets the manifest for each one so in ours, we'd actually have manifest name, etcd version, whatever the version is that we have in defined in our stable deployment uh and then, if we go back to channels you go into guestbook. So this is notice how how the naming.

C

A

Tree happens and then under here you have manifest.yamo, and so that's what actually points to it. But what's nice about the common spec is you're, you should be able to.

C

A

Those entirely if you want to as long as your are back still suffices for whatever the the changes are going to be.

B

So um I have a question so the the version here does that align with the version of the components or just the version of our controller like if we have a new version of etc. 3.5.0 then do we need to create a new version under maybe stable channel. For that.

A

That's a good question, I would say we should, for we should follow the versioning scheme of the components. So, if etcd is yeah 3.5.4, we make a new version and we make 3.5.5. We make another version, that's in there, so that you can.

C

A

Have those maintained um I do vince, there is some. There are some policies within within cluster api about versioning, or at least there was at one point I remember hearing um is that is that a conflict, because I what I remember from this is a long time ago, if, if I'm right um was each version of cluster api was specific to a version that was actually released, and so you would update the component tree for cluster api.

C

A

What versions were available to you to deploy? Am I misspeaking there.

A

If you are talking, you are muted and if, if you aren't there, that's fine, I can ask uh separately.

E

A

Yeah, sorry about that, so uh I was over there for one. Second, no worries. Can I I'll repeat my question, um so I remember from a while back- and I don't know if this is still the same policy but in in cappy at one point there, the policy was versions of the released kubernetes versions that you could create with them uh were associated with versions of the actual control plane components like the cappy components, um or is that not the? Is that not the case?

A

I might be completely wrong there, but I feel, like that's what I remember hearing at one point.

E

So, just because, like the version of the uh of which components.

A

So if you had, if you had kubernetes 119 and you were upgrading to 120, for example, you would typically have to update your cappy control controllers as well to support that.

E

That happened in the past because of breaking changes, uh but so alpha 3 is actually the widest release so far and we still keep supporting the new releases.

E

Okay, um it might happen that in the future we won't like you we're going to say, like hey, you have to update in the new version to get the new kubernetes version but like unless that's required, like I usually discourage that because yeah we want to support like a wide range of.

A

Case versions and it goes backwards as well, so, if you're on 118 and you had like, if you have down till one sec, is it basically like follows the version skew for it or is it um because the the the thing that I'm trying to figure out is? Is it okay if from a cappy perspective, or is this any breaking or weird implementation, if we support say, for instance, in our controller, we have versions for ncd that are 3.5.4 3.5.5, 3.5 point like and and up and down, and both ways on the version, sku or.

D

Should we be.

A

Versioning those separately and having releases for each one of them and not containing all of the backwards compatibility. No, I would I.

E

Would do the backward compatible way? That would be the uh the way that we we're doing that today. um So um actually also clarified like there is a minimum version for the management cluster, um which is 119. I want to say for alpha 4 and it's 116 for alpha 3.. um The the main version is dictated by a few factors, but the workload cluster. It's like much wider.

E

um We do support like a wide version, but we don't support any downgrade. So that's you can go back like from after you. Upgraded a cluster um there is. There is something we want to support is rollback to make sure that, like you, could you know roll back to the previous stable version, but we don't do that yet um so yeah backward compatible sounds like the way to go um and making sure that you can only go up. It's actually like a good limitation to put in place when you're starting up.

A

Okay, cool all right, so so that that that answers that then so we can maintain as many channels as many versions as we want under the channels, but we can specify what is our stable version. Okay, at the current time,.

B

Okay, yeah sounds good yeah, okay, so I will keep the current implementation and we will merge it first and later we will uh move forward. We will uh try to align with the implementation of the uh the declare declarative pattern. Yes,.

A

B

Sounds good that.

A

B

I okay, so um okay, let's talk about the second issue. I have so I discussed with you last week and we agree with that that the ncp controller are going to create the ca. The root ca for all the components and each nested components will component controller will be reached, then will be responsible, create the actual certification and certification based on root right.

B

But when I implement- because I forgot when I discuss with you- I forgot that when I like actually sign a certification, I I also need the the rsa key paired with the uh the root ca. So I think is that possible to store the rsa key inside the secret as well or do you think that is sensitive.

A

um I'm trying to think of.

B

If we store the key inside the secret as well, then anyone who can read a secret will have the right to generate a certificate by them by the name of the root ca. So I I'm not sure if that is securely enough.

A

um Let's see yeah that sounds slightly. Concerning um I mean I would I want to vince if you have, if you have background on on what q8m is doing there um yeah? How does that actually happen? Because, because, if cube adm is is not generating the certificates uh up front and it's a lot and it's passing it down to the actual machines to generate then what? How would that? Actually, how would that typically function.

E

uh So for control plane, uh you have to pass like all the casings and the keys to make a control, plane, initialize or join uh the for nodes. It's just the ca search of the cluster.

E

um Now like for secrets, he said like, um if you have read secrets like you, could use that certificate to gen like potentially do something malicious like are we talking about the management cluster or the workload cluster.

A

E

Be specific for the.

A

Workload, cluster because it would be a single per week per workload.

E

Cluster, uh well, you could have access to both. I guess if that makes sense so like I'm, so look for the workload cluster, like the secrets are like on disk. um So if you have someone that already has ssh in one of your machines, the security holes say somewhere else right um for our back.

E

um If, if you have like the management cluster like it should be a little bit more tied down like you wouldn't want to give out like access to the management trusted to everyone probably want to give out access to the workflow clusters to your users.

E

um So there is the concern that, like the secrets are stored, um you could use, I guess, an external ca, but that doesn't really solve the question like you could go around and like not like make it so that, like you, use external cas and things like that, but then you have to generate all those things that and management of those certificates will become that nightmare really quickly, um especially in you know, in capi, where everything is immutable, um you want to delete and recreate machines, and then you have to generate those certificates.

E

So so what is it? You're worried about: let's do another question which which attack of actor.

A

uh Ciao, do you have jeff.

B

Yeah so um so yeah like ben said, if we assume that the management cluster is cannot be accessed by an average user, then I think it can be secure if we it can be secure enough. If we just have this assumption and then we can store the both the ca and rsa key inside secret and the secret is just on the management cluster, so the workload cluster will not have the right to the user of the workload cluster will not have the right to access this secret, so that can be okay.

B

I guess, if we have this assumption.

E

Yeah- and you may also want to like lock down secret access of the queue system and space on the on the workload cluster so that they don't get the other certs.

B

Okay: okay, yeah yeah answered my question: yeah.

A

From that side of things, so that's really to generate so we get around this in in vc or in in what you did in vc manager by doing everything at the uh virtual cluster controller and generating all the search for everything. Now I think going back to our chats. uh I think we talked about doing you and I, on slack, were talking a little bit about maybe storing the actual client certs from the controller instead, um which would get around that.

A

So, if we basically had the ncp generate all of the key pairs, uh while it still had the ca and the rsa key, we wouldn't need to actually pass that down correct.

B

I mean, if did you did you just say? If the ncp controller generate all the roots include root and also the certificate everything yeah? That would be good, that'll be good, then I think, then you don't need to pass the ca or isa key into the secret. You can just store the client certificate and key inside the secret.

B

That will be enough for all these components to talk to each other, because etcd only need to talk to api server only need to talk to etc, and the controller manager only need to talk to api server yeah. So we only need to have two sets of client certificate and a key pair, etc and api server just to begin. Okay,.

A

Yeah then we need to. We need a way to be able to generate them for uh be all being able to off to cubelets or via an agent right as well.

B

Cubelets, be an agent um yeah, yeah, correct, yeah, right right if we wanted to have another like uh uh cubelets client certificate.

B

If my memory is correct and also yeah, I think we still need to have the ca store in a secret someplace, because when you start api server you still have to have the ca of the etcd. Then you can use it to yeah to check.

E

To set up the authority.

D

B

Yeah yeah right right.

A

um Okay, so what I ended up doing is uh this morning I went around and I just I went back and updated my ncp doc, uh so I had. I hadn't had a chance to poke you and say if you can go, give that a read, but um in essence what I ended up doing was writing it up as though we would do generation of the client certs as well and do based on your uh our chats. Sorry, everybody.

A

You weren't all a part of this, because we did this in in dms, but uh in essence, we were chatting about how we could potentially store the std client certs, uh based on the same name, name prefixing, that we use in capi. um So let me actually just share my screen and I can show what I what I ended up writing up.

A

um I did a little bit of a tbd on some of these things because I don't know all of the certs that we're going to need to generate and we can always mutate this later. Let me share my screen again.

A

So in theory, if we go down to search so basically what I ended up doing is uh calling out another another piece, which is the the uh key pairs for every single component. So we'd have cluster name uh dash component dash, client, cert and those are going to be the key pairs, basically in essence, generating very similar to what we're doing for for the service account, and then we have the so how that's all structured is all written out there.

A

So we'd have ncd as a prefix as a component name proxy as a component name, cubecontroller or controllermanager, sorry as a component name and then potentially cubelet, and then anything else. We might add later. We can add into this list as known components as for other pieces that can get deployed as pods. How does that sound?

A

That? Would that would make it so that you don't technically need we don't need to store the rsa key in that same place, but I mean we always need to be able to recall back to it to do uh refreshes on on any of these certificates anyways. So we do have to store it at the end. Yeah.

B

So so ncp controller will be responsible for creating all this secret and store all these things. Okay, yeah that.

A

Way, you don't even have to so you basically just get everything you need and just keep cuddle, apply it and then you're you're. In essence, a deployed component.

E

A

Does that seem reasonable.

B

Yes, that seems reasonable to me: yeah cool, all right. I think we missed the api server. Do we are we going to have a secret response corresponding to the api server.

A

Yeah um so right now I only have it as the proxy cert, um but I guess I can make another one yeah I'll add in another one that is the the api server as well. So let me I'll update that um I did a couple other updates to there to resolve everybody else's feedback on that dock. If you want to give that another read through as well.

E

One thing, though you might want to think about chris, is like I've. Seen like you, you don't have anything else apart from that api server, um how are you going to handle like certificate rotation and things like that when you want to.

A

Upgrade yeah, so if we do it at the ncp level, how do we upgrade an individual component or how do we? How do we yeah renew the search for an individual component.

E

Like maybe you want to use the pod name that you're creating as the suffix or a hash of that to because then you can use, you can create new ones. When you regenerate that part, I mean, or you roll out a new version, and things like that.

A

Okay, all right yeah, that's.

D

A good idea, um let me think about how we could do that since.

A

D

Yeah, why we don't use a certification manager to do all these.

A

Things because yeah, so we can't expect everybody, does everybody's gonna use cert manager for their deployment, and so we need to have a common path for people to be able to provide their own cert. um For how for how these control planes will get brought up.

D

Yeah but search is complicated, correct. You have a different thing so better to have a common place to to do that, so people can replace it if they need it. Instead, you put into each component and implement it.

A

Yeah, so the idea there is cert manager is an add-on that you can use and by having a common named secrets, you could overwrite those by saying in your certificate resource with cert manager. You specify the name of the component. You want it to generate a cert for, um and it will generate the secret into that it'll generate a secret and store it in that name that you specify in your certificate, and so you can override anything that's in this dock by using cert manager, but we need.

C

To have something.

A

Built in that that will at least manage it at a at a self-signed way. um Okay, so that we can deploy this without cert manager as well.

E

This way, a certain amount is required for copy, so.

A

I thought it was only uh it's required in v4, but not v3, right unless.

E

A

In all the versions.

E

Okay, because it's required for your builder and so.

A

Gotcha, so should we.

E

We actually install it for youtube interesting.

A

B

So I think if the certificate manager can generate all these things, then, and also can do the rotation like certificate rotation. I think it can be good option right, because if we do not use certificate manager, then the ncp controller have to be responsible for the whole life cycle of their certification. We have to do rotation update all this right.

A

Okay, all right, um so if we were to do that, if we were to basically make it so that it's a requirement on cert manager yeah, let me let me spin my wheels on that one a little bit, I'm trying to think if that would cause any issues um to require cert manager to do the generation for the search for the control plane.

A

Components, because I understand why it's therefore at least for um for the cappy control plane, but I'm trying to figure out why it would be there for, if, like cube, adm, is typically generating the search and managing the search for the workload clusters.

E

So you usually like a weekend. You start manager because qbn runs on the machine and it's like too early for that workflow foster to actually have those certs. um But in this case like given that you might well, I don't know how they could. You will have to figure out like attaching of nodes but from the control plane. You could leverage certain manager and yeah just create those crs and get the secret and push it down.

A

Okay- okay, all right! Yeah! Let me let me let me think through that then and see if I can update this doc for what that would be. I basically in my original writing of this uh wrote that you can use it.

A

You can supply your own search with this, but we're going to follow the same path that we have with the cube, adm control, plane and bootstrap provider that it does some of that generation for you if those aren't supplied or if those aren't generated for already, um but I guess that makes that would make that makes sense. All right, I'll, spend some cycles and update that all right and think through how that would work.

C

Sure I think uh student manager, definitely, if increase you, think it is reasonable. We can definitely go there, but my current environment is, I don't know how much time we are going to spend if we really do stuff. But if we want to make our marched milestone, this is okay.

C

We just do some mean simple, uh just leveraging what vc manager part of key management- very simple, there's, not a great story, but again our first release and we can put a manager in our the next is, I think, which is more reasonable, um because this is a blocker. I don't want to spend another two weeks to explore how to resolve this blocker.

A

No, that makes sense, um I think the biggest thing. The biggest thing that we need to solidify is what the names should be on those so that we don't have to change the names on the secrets, uh and I think those at least are are solid, because those follow the same format uh that we have in capi. So if we can follow those patterns, j use, what we have in vc manager to implement the entry, but then also be able to support overrides with cert manager, then we're okay.

A

I think that that makes sense, and then we can iterate on it. As long as we keep the a common naming format, we don't need to worry too much exactly.

C

So because you know upgrade, you know, key upgrade is a long long story. I mean you know what happens very frequently and not not near future right. So it's not blocker to me so the market is. We have to make sure all these you, so it's available, otherwise other components.

A

Can't come up yeah, okay, yeah that makes sense yeah. So then I'll I'll probably go through here and change a couple of the pieces in in the the dock about using the cube, idiom stuff uh and point more towards the vc manager uh code base, and then we can generate or we can leverage those use, the new names and then um yeah and move forward with that.

C

A

Cool all right does anybody else have anything that they want to bring up uh we're at the end of our agenda, for.

A

Now, at some point faye, if you get a chance, uh I would it would be super awesome still to do a demo of what you're doing with the experimental stuff. But that's separate from from uh this of.

C

Course, but not today, because I I I wish I can do that, I I wish I could do the you know uh kind of tutorial little tutorial this week, so sure we uh get everything done and my demo would go would kind of go through that tutorial.

C

Okay, one thing is: maybe we can take some time to discuss when he is we can describe. We are about the example controller for the crd, okay, yeah yeah, so so so my yeah. So I think I think I agree with what this is common generated. I I think that that controller can be as simple as generic as possible.

C

I I was even start thinking it's just it just has some document but but but you put, you know, actual coding in that, so um I think it is okay, um but I agree with most of what chris was mentioning say using the name like just fool.

D

Okay, I can change the name to full yeah yeah.

C

My little concept is you put a sample code like it did like all the petroleum dws uw thinker, um so so my worry is that uh are you going to have to make consistent every time we change rules so because this is not like you know, interface code or framework code, which is kind of pretty stable.

C

It's real kind of working piece. I'm not sure. Is this good enough to just put actual code there uh or just use kind of document describe what you are going to do? I don't know which one is better so.

D

I think the code is just example that so we can it's not to enforce people to use it, but I put there just example that how I use this your framework to generate another thinker, that if anyone wants to separate the syncer- or they want to combine with your existing thinker, that you can combine a big thinker, including the not optional uh component, but also this is cr which he defined separately. So this is how it works.

D

So when you change the infrastructure in the future, I think you evolve very quickly, but I still hope that people can can still make sure this kind of a some implementation which some company like us, we did uh can be compromised that the people, when changing your infrastructure code, not too dramatically that make this piece not workable, but this is not enforced. Just uh put there to make sure we have example and people can contribute on it and enhance it. The start point because just writing.

D

uh I already finished writing. So the code is, there is a shouldn't matter right for, for your entire.

C

So so so here's a I'm probably wrong, but just my initial thought I haven't fully think think through the other process. My my my little bar is, if you put it away like you, did you have a make file, so the assumption people is here. I need to make something and the make file should just work.

D

uh uh So I will make file it's separate there, because this example. So if you look, the structure is totally like a separate uh uh thinker. We build it. Build start root, from example, is not from the virtual cluster, so I will put a make file there so that people can build it and uh to to fill it, and if they don't have time they can enhance it in case I I made something wrong, but it's quite complicated.

D

That's uh I make everything work now, so it dynamically can introduce crds then on the cr can be synced and can be patrolled. So this is a maybe someone.

C

Yeah, I wonder my point: is you can do that away, but uh so we have two two ways to so so if we want to maintain that the problem is how to maintain that right. If we want to maintain that which means every time we do the change in the amazing curve repository I need to think about. Oh there is, there is an example there should I change that file as well. So that's that's kind of it's okay. If we all agree, we do that. That's fine, so, on the other hand, is rosie example.

C

Directory is maintained separately, so some somebody else like you probably need to think about every time we we make some change in the main repo, otherwise people reading the example will get a mistake or whatever if I fix the bug.

C

So if there is bug in the main call so who is going to point at the bug to that example, directly.

D

Yeah, so the way, if you look at the code currently in example except this uh api, the the full uh all I, the crd crd apis, which is generated- this is separately. But if you look at the core of this code, it's just one of the sinker components like underscore source. Crd noun is an underscore uh source score uh uh uh slash. uh We call it a project as a or full. So this is like one component very slim, very slim. There. Everything is a import from your infrastructure.

D

Nothing was uh actually there, so there are only actually three files which can be put as a component under slash resource.

C

Yeah, I I think I need to check. I thought you are doing some.

D

Copy, if you just do importing no yeah everything imported, so only so this you can consider it. It's just slash resource resource, slash component, this everything is there and nothing else from so. Actually, I can even check into the uh resource slash directory as a new component. A new sync command like a crd example, but we put example is that I don't want to interfere with your infrastructure, so everything is uh is just imported from your main infrastructure.

C

You have a checkered goal right. You have resource projected check., you have other controllers implementation.

D

Check yeah, yes, the checkered goal, so this checker controller and uh dws. This is three components that every component in your thinker need to have correct. Just like.

C

My point is so if we change something in those three functions, for example, fix some bug. We do some refactoring um in the main report. Not this example repo. Are we going to pause the roads change to this example?.

D

Yeah, if you do the bit, the idea is that, hopefully, if people do this, big change also should consider um this kind of uh if he agreed this example cr resource synchronization. He should also do one step to make sure that also compliant, because this is just as the exact same kind of a component as the normal thinker, which we use user use it. So I don't want you to make dramatic change in the infrastructure that so the people very hard to use your infrastructure in the long run.

D

So this is a one way to to make sure our infrastructure is generic enough, so everyone can benefit from that. uh So this is a is a good thing for the infrastructure.

A

Real quick, quick question for you fair you suggesting potentially taking this instead of doing it as as actual code and basically taking in essence these files and saying, if you wanted to implement something very similar, here's a markdown file in the docs directory that says these are the potential things that you would implement and what a dws should look like.

A

Like almost pseudo code like it should look like this, you should have a patroller that looks like this. You should have a uws if you wanted it to look like this, not actual implementation, so that we don't have to maintain implementations.

A

And then you can just point and say: oh, you could import this, this command file and it's going to automatically load everything for you. I see.

C

Yeah, my original story is just something like pseudocode kind of pseudoblocks, just he's playing what you should do. I didn't expect a comparable something that can be compiled to the binary. um I I I thought that would be so. You know tightly coupled with the real synchro implementation.

D

And everything yeah, but I think a good thing for you, the single infrastructure that the people do use crds and they do have dependence on your infrastructures so, like recently, some of the changes are so dramatic and uh of course it's a good way is not to break a lot of things, but they do have some complication on the crd, our component.

D

So in this thoughts I put this component in so at least when you change the infrastructure, you need to have some reference that oh people use this way, so I have to change, uh make sure it's still compliant. It's not so dramatically. Change. Only think of this 10 count the standard component in your sinker. I think this is good for the long run for the sinker infrastructures. Yeah.

C

I totally agree with you, but but the problem is, you know: even the synchro has been there for a year, but I would say still in very early stage, so we don't. We don't even have a formal, you know formal release date yet so, which means uh I mean at least for currently any code change or any recorded refectory. We were thinking we are, we shouldn't worry too much about. You know breaking kind of others, because we are not even released even even a beta version, so that that that's that's a reason.

C

At least at this moment I don't want to put very strict. You know enforcement on how we change the info code, because it's not a good implementation anyway, we need to change it right. So if we release something, for example, we formally release a one point, any release we have formally released. Now we need to think about. Oh any changes. We need to be very cautious about where that break others. So that's a that's a that's a reason, but I I agree with you once you know we we release it. It has to be stable.

C

I need to we need to think about the order. Oh oh need to, you know, break make it make all everything is backward compatible.

D

Yeah, if you can same thing for your experimental kind of directory, which you put these new things, which also do the thing, but I think you also depend quite some on the infrastructure code there I have to so yeah so same thing.

D

uh So this gives you some vision that, because this infrastructure is so nice that can serve a lot of purpose, I have some new ideas to make it to even grow on different angles, but uh with all this index either in the example is experimental is tightly not very tightly combined with your existing code, but at least loosely tied loosely coupled with your infrastructure code, which gives us at least some some uh some reference, at least for the people to to work this way.

D

Unless you don't really agree on the implementation I did with the crcr thinkers, then we can discuss everywhere if you agree that I think this is good example that we we use this way.

C

Yeah, I think sometimes I think, as a project getting bigger and bigger. I think people will face this kind of thing right if we kind of, for example, we have so because the example crd you can, you can treat it as another kind of experimental features which is can stand alone, which is stand alone, things based on the base, sinker infrastructure, but you do something else, um but if we agree that freedom of photos for the scheduler that I was doing so I never ask you know anybody.

C

You can change thinker whenever you want, you don't need to think about the experiment, I'm going to think experiment to matching the latest infrastructure. So I, unless so, I'm not asking everybody to change something to make sure the experimental can work. So, on the other hand there that, if, if you, if you think okay, uh you don't want to put extra, you know constrain or burden or anybody to change synchro code. So you are going to be responsible to sync any changes on the synchro to to the experiment, crd, I'm I'm totally fine with that.

C

D

Yeah, I agree so at least I have a point to just start discussing, because when, if I lose this code after several months, people change this infrastructure, I have no any base to talk start with, because it's very complicated things. Now, if the code is there at least, I can point out this some certain line that, with your new change, is make difference or make difference. So this way at least there is a reference point that I can start to talk and make sure the guy who check the infrastructure code can recycle.

D

Is this the best way or not? If he thinks best way, then I would do the change on my side. The example code make sure it's compliant, but at least the code there. We can have a start point to discuss why your infrastructure change is good or bad. So at least I have some reference. If I just put a document there, no code, uh it's really hard to put everything there, so you cannot even start to talk.

C

I kind of disagree with that in the sense that if you want to discuss whether some code was good or not, it shouldn't be happened.

C

A few weeks later, you should be having right away in the pr so that yeah.

D

Yeah pr, so I can point into the code. So, for example, you did a big pr to change that. So then I can point to this. uh You know review. I can tell him that, okay, if we want to compromise something like uh like in the cr sinker like in example, uh if you do a little bit change this way, that would be nicer or at the end he's not agree on that.

D

Then I would go back to change my code to comply with this new change, so there at least we have one run of the discussion happening, it's a so that people can be think double think if his new pr is good enough or has some enhancements.

D

So this way we have to can can start a discussion, otherwise uh it's hard, because when you do a pr I have known my code- and I also forgot this code. So maybe the change is very hard to to to to in the in the cr area that cannot be seen cannot be.

A

From a maintenance perspective, I think is really what what faye's getting at trying to maintain this going forward and making sure that we're keeping everything in sync in sync- and in essence I mean, if you look at a big project like kubernetes or anything like that, you don't really see. uh Example, sample controllers, for example, being in tree in here. um It does feel like this could be written as a relatively simple uh dock. That says, if you want to go through like tutorial like we have for the for the tenant dns.

A

If you want to go through this, these are the five steps that you have to do or 10 steps that you have to do to go. Do this, and here we can basically take everything that you did copy it into markdown, and then we don't break potentially build time. We don't build, we don't break any build life cycles as we get this all this code moved over into cap n um and we have to. We have to actually run it through ci processes, since we don't have any of that in in the multi-tenancy repo.

A

So I think it probably does make more sense from a maintenance perspective to make this a document instead of being example, code now way you and I can try and talk through what it would be like to do this in another repo.

A

I doubt we'd get something from the from the sig side of things, but potentially we can talk with our uh ospo and figure out if there's a way that we could publish this separately. Otherwise, we might be. We might maintain this internally, just as a here's sample code for for our folks, using it and then otherwise, point to uh otherwise point to a document that we wrote instead um so basically take take. This, take the three files that you have in essence convert those into into parts of a markdown file.

A

That says this is a patroller and a patroller does this, and this is how you would write one that looks very similar for a cr um and all of like a walk through, like you'd, get out of the book book.qbuilder.io, for example, to make it a little bit easier to follow along because right now, yes, the sample code makes sense and it's eaten and it's easy to go. Look at and walk through, but I think being able to explain it.

A

A little bit deeper is going to be much better, much more beneficial to to someone trying to pick these things up and build these art themselves versus having example code physical example code. That's compilable, um because they can always go and look at what the patrol or what the patrollers look like in any of the resources and any of the dws or uws in any of the other existing resources as more reference for compiled examples, um and those are always going to be more up-to-date than an example directory.

A

That's just in the in the repo does that seem to echo uh what you're looking for.

D

um True yeah so or you want me put into the experimental that I maintain.

A

It for it for you or no, I don't want to put it in the experimental. I want to put this in because it's not really an experimental feature. I think we should put this in a like a markdown file in the docs directory, so go and and write a write, a couple, a very short tutorial.

A

If you want to make a custom sinker, what do you do uh and walk through the process that you've done multiple times to build these sinkers before and phased on, and chow's done and jerry's done and everybody's like everybody's built these things, but now we should probably make it more concrete that says: okay, these are what the different components of a sinker are.

A

This is what um this is, what each of those different style sinkers are and how you would write them yourself if you wanted to do one for your custom cr and then you and I can talk and try and figure out if there's a way that we can house example code somewhere, that's not in tree, so not in this actual repo at all, um so that you, if you.

D

Both agree on that and then I move it this. uh I close this pr, so yeah.

A

You don't have to close this pr, I mean you can keep the pr open. We can just move. We can move this into into a a docs file instead uh and then and make those make those go files either referenced into or loaded into. No they'd have to be uh copied over into um a markdown file as like the the tutorial for how to do it.

A

That makes sense.

C

Yeah yeah yeah yeah. That's exactly what I'm saying so you can even put that somewhere called markdown file. That's okay, but just don't make something: that's comparable come viable. If so, if you make comparables as I think chris got a good point, so you need to make sure it doesn't break ci, so I so it can be annoyed if the project's getting bigger and bigger. So every time you you there's, some complex error happens in the example directory and and so yeah. That's my point.

C

I think I think especially from any people want to write sinker. I think they cannot be a dunk developer. They have to have some understanding about everything that has been working so just show them. You know what are example code and if you just tell them, please copy these three files and the dual sed replace uh part to crd, and it probably just will work.

A

Yeah something yeah, because then you could really talk about the the intricacies that you had to deal with there like keep the changes that you had to do, because you didn't have to make changes to the the like core infrastructure of the sinker. I noticed that, like you, had to make sure you had to export the the check, the meta, the meta fields that weren't already exported those were just those are typically unexported functions in there.

A

So, like keep all of the code changes that you had to do this to make this possible and then right, there's.

D

A

Place. That's a metaphor: I need to change, uh but it's optional. I can copy it. This is uh no.

D

No, I'm saying keep those.

A

In this in this pr, because I think those are good- I mean I assume that faye would agree like if we need to export functions so that we can do build these custom thing. These custom syncers and not have to copy a bunch of functions. Let's make sure and push those upstream so that or push those into the the course anchor keep those changes. It's just instead of having those three files that that you, that are an implementation of a foo sinker.

A

Let's just put those in a in a in a markdown file and says: hey we're going to take this sample controller, foo cr and we're going to show how you would you would write a dws and the uws and a patroller for this to do x or I mean to to sync resources for your custom components and then.

C

Just it's playing a thief kind of you copy something, and you make a thief. Just explain that beef and it's more valuable to me, because now we can look about we can come. We can think about potential recommendation of the chord structure to to make sure that the diff as small as possible, so that that's all our purpose right.

A

Yeah and just to just to show uh sorry, I'm going to share my screen for two seconds as well, just to kind of show what what um I think we mean here. The the perfect example of this is the tenant dns stuff.

A

So to do our our uh forked core dns, we didn't for core dns code base and put it in the repo and make it something that's compilable from here uh the it explains the problem state, what you're trying to do the solution and then exactly as face saying: okay, what would the diff be to actually make this happen? So in this instance, it's a little bit easier because you can just copy this and make and like execute the patch for this.

A

For us, it's probably going to be the example code that you already wrote in each one of those files that goes in here and then it will go through. Okay, once you've done this, you need to write a command file that imports these things and imports your new syncer and then from there you're good. um So in essence, taking those three four files and just putting them into markdown.

D

Okay, okay, I can do that yeah.

A

Cool all right uh we're getting close to the end of the hour. Is there anything else we wanted to bring up before we drop off? Otherwise we can see everybody next week and.

A

Fantastic everybody all right, see y'all next week have a great week.

B