Kubernetes SIG Cluster Lifecycle, 18 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cluster Lifecycle 2022-10-18

Description

https://docs.google.com/document/d/1Gmc7LyCIL_148a9Tft7pdhdee0NBHdOfHS1SAF0duI4/edit#

A

Yeah, oh, can I just click the quicker to the tradition. Like you know everybody. This is the sequester committee. Today's the 18th of October 2022. We have one uh agile item, which is the addition of a new provider for coaster API. uh More particularly, this is the Helm other provider yeah. Let me share my screen. It's probably so take it away.

B

Yeah I can give some updates, given that the discussion is ongoing on the cluster API Community since another time. So, as you know, uh cluster Dawn is a an opinionated space in Caster API. We have a a solution which says: cluster resource said that has been implemented more and more or less as a stopper Gap customers set is a way to install some arbitrary yaml when the Caster is started, and but yeah cluster of Donna are sorry.

B

Customer Source set is, is very limited, and so some time ago, uh Caster Pi Community start the a discussion. How can we solve this problem, especially given the upcoming immigration of CPI out of three, uh and so uh what they come out is basically an idea basic based on a few principles. The the first one is that in the community there are really great tools that do cluster that do a dominoes and like Elm, K, up uh or others, and so instead of Reinventing in cluster API, a way to.

B

We should Implement something in cluster API that basically orchestrate those tools, so we leverage on on those tools to actually package addons to actually install them to control this them are in sync, etc, etc. Cluster API, the customer API provider for n. It is just a an orchestrator that basically called when uh when a new cluster is uh is created or called M, when we change the definition of the provider or we chain or when the customer gets uh get upgraded.

B

So the idea is that not do not are inventory with the wheel just to coordinate what is in the community. uh uh Jonathan worked at this project. Microsoft is, as far as I know, he's already using it for their CI and.

B

And yeah this is one of the idea around and on the management there is in the community. uh In my opinion, it is interesting in an interesting project and I, really like the fact that we are not or invented a wheel and just leverage on what the community offered and motor specifically on what the user are already using. So we are kind of matching the user where they, where they are, and so in Caster API.

B

There is a a proposal that has been recently accepted: uh Jonathan demo and a working prototype is starting to get the requested from other folks to collaborate on this uh provider and basically is facing the problem. How can I make collaboration on this provider Streamlight and controller and and guaranteed like uh like we do in in kubernetes seek, and so this is why they come up with this idea.

C

What is uh so, what is the nature of the integration between Cappy and uh this, like? What ties are there.

B

The natural integration, so did this provider started like an experiment and, and the experiment was try to figure it out how to execute this orchestration. Okay- and uh please note that these are this provider has been implemented before in cluster provider, was life cycle, looks, etc, etc and So?

B

Currently, in this current implementation, but yeah, the document is also talk about what will be next iteration in this current implementation, basically, that this provider is just a controller that watch for the cluster and basically when something happened on the Clusters, it goes and call Elms to make something happen on the generative cluster is more or less the tldr of it.

C

So there's so it's just a cons. It's a consumer of cluster API right, so I, I I, think which that's that's I think we should just make sure that uh that we aren't locking out other options right like there are a lot of people that don't like help and that we, so if cluster API isn't making any calls to this, then it doesn't seem like it seems fine right. uh Just like it's a consumer of cluster API yeah.

B

Exactly now the the link between this project that cluster PDI is very weak. What is we plan in this written somewhere in the document is to have iteration and make this this controller to leverage on cluster API lifecycle looks but again it it is. uh It is pluggable you you it it eventually, someone could keep this provider clone it and make it use. I, don't know, okay up.

C

Yeah I mean the uh the place where I can see that we need a specific integration is to make sure that all the things that prepare a cluster have run before we Mark that cluster is whatever we want to call it like ready or finished, or you know good to go right so like, for example, don't allow user workloads onto the cluster until the authentication hooks have have been installed or until the author that the image scanning tools have been installed right so that that is I, think where we need it and I think when we go to build that, like, let's just make sure it uses generic mechanisms, but yeah then like if people want to I, know they're people that want to use Helm and I think we should allow them to do that like we shouldn't, stop them from doing that.

C

So I think this is fine as long as as long as we leave those options open and- and we sort of try to like look at other tools as well right, like um I, wouldn't call it the cluster API add-on orchestration proposal. I'd say like it's: a Helm integration for cluster API.

B

That's that's a third comment, uh yeah just to be the the picture, while working on these or more or less at the end of the work. We discovered that other projects like flukes and some integration between Caster, API and L, and so it it is a starting point. uh I have a it is one of the topic that I would like to bring up and start discussing, maybe at kubicon and then come back and in this in this meeting, but yeah it is an interesting space. I think of things are Spike spicing up.

B

uh Let's see I'll ship out, but uh I agree with you. We should we should not look in uh cluster API. We found a specific uh solution for our Don manager, but we should have a good extension point that everyone can. uh Okay.

A

Yeah, certain parts of the proposal can be rewarded to make it like a more. um Of course. This is for Jordan, but certain parts of the proposal can be rewarded to scope out Kiln as one of the options.

A

B

But but I think that is already this way, because it is an example of a concrete implementation of an ID of orchestrator, which is defined in the in the paragraph Above So. This proposal kind of Define.

B

This concept of orchestration, what the orchestration should do and then basically use Elm as a first implementation. So it's come more or less what what we did for the kubernet mean Contour pay provider, a booster provider or for AWS when we started with providers.

A

B

A

It has to start from somewhere, of course, uh so this.

C

Sorry I just want to say: I put it in chat, but uh the repo name. That's proposed actually I like a lot like it's. It's a very accurately, pronounced cluster API add-on provider, Dash Helm, uh so it's it's the atom provider and it's the helm, implementation and there could be others and I think that's the right way to frame it and that's I guess what Jonathan proposed so yeah I like that! That's that's! A good.

A

I, like that, I don't want to take the credit, but I think I insisted on being specific about it, but yeah. uh So uh the you say that Microsoft is already using this, but I'm wondering if you mentioned that the life cycle hooks integration is still like work in progress. So how? How are the usernetes if we still, maybe it doesn't work fully yet.

B

No, if you go up at the beginning of the document, basically we are talking about.

B

uh Maybe up again sorry.

B

Oh, maybe maybe it is ah it is the solution will likely require criteration. So this is what we figured out. The first one is the foundation, and it is the one describing this document where.

D

B

We we Aspire the kind of blind DCd of orchestration, uh uh this idea of another event, the the the wheel, etc, etc. The second one is that we will build a better orchestration using what we learned and uh and integrating with cluster life cycle looks.

B

Then we know that there are more, uh for instance, there is this idea of a don't dependencies when we can start really twist done if a one depends by another uh or the idea that that uh Justin just suggested that when the when it is the moment to allow user workload to get into the cluster, so you know we we kind of started going down the path we like the the results and yeah. uh So going back to your question on lubame, now that the orchestration is really simple, they're watching the cluster, no more notes.

A

So I guess the this is some of the struct, so we can see later, which is the home proxy. So.

D

A

Not a the responsibility of the Home Alone controller to observe, what's happening in the cluster and decide whether it should perform certain action and.

B

A

Later they might have to change that and integrate with the life cycle hooks.

B

Yeah at least part of the integration.

A

Let's see, but this is like the user interface like basically where people work what they want like Charles and yeah.

B

You you can, at least from my point of view, the ux that you see here is the.

B

It is really the first iteration about an idea, and now we have to gather feedback to understand exactly where to have to go. We have some idea, but uh the feedback per user is invaluable.

A

This is like a go-to completely remove because the resources, whilst this gain subtraction.

B

It it is, if I remember already, it is a a future work to reconsider.

B

If we want to keep customers who said if customers set could become another customer, don't provider, because at the end it solves the same problem. So for now, for now, I think it is too early.

B

There are still people using the cluster resource set and- and we are still having uh PRN proposal to improve them, but we know that there are some limitations that are are the two results, so uh I think that it is official goal to reconsider the. What will be about cluster resource that, if you scroll down just a few lines.

A

Yeah, they could actually be some sort of a raw, a question Source set provider that you, basically, you can install some Java with it and forget the blood circle hooks and maybe uninstall. This yellow, install.

B

A

That that would be really cool. uh I mean. Let me take you for Bridgeton for for this uh I. Oh one for this presentation. I want one question. I have is that this is more like the broader uh sick topic that we have a other project that unfortunately, I always have a conflict and I haven't joined the meeting, but I think uh Justin and me you probably enjoyed more often than me, but like the question, though, is like we have adults for cluster API uh I, don't know overall special interest group.

A

We last time we discussed. Maybe uh this we shouldn't create like a cube ADM operator, because cluster API may have this uh operator logic in it. Now we have this add-on provider uh logic coming in question API. The question, I guess for the umbrella series should be just pumped uh a lot of these complicated topics to Cluster API as the coaster management solution, and that eventually will integrate these uh features.

A

So maybe we should start um removing uh some of sorry, not removing, but maybe we should drop the adults or project and just claim that uh we cannot do it, uh for you know for cops to work for chaos, to work to for Cuban, to use the same add-on solution and it will cost the API to use a same amount of solution. I guess that's my investment.

C

I think this is a good step in in the in that discussion. So we should. We should establish that we have the notion of pluggable add-on providers in cluster API. uh Helm is the first word first or second, one cluster sets was that or cluster thing was the first one uh package sets I thought, but uh Helm is the second uh cluster add-on should try to integrate as the third, uh but that's on the cluster add-ons project and if they can't, then we should. uh We should revisit but yeah.

C

um So, in other words, like I, see this as an opportunity to to produce the the add-on operator version. So I personally, am not the biggest fan of Helm and I, but I think that the way to persuade people of that is to demonstrate that, like there's, a better option and so I should build that better option and integrate it right, and so that's why I think it should be open so that we can have different ways of managing things in the Target clusters.

A

Yeah I see what you've been just uh I. Remember the.

A

Other uh basically operator idea, so basically this could become eventually a like a cluster API add-on uh provider solution. So.

C

Yeah, the naming gets a little complicated I, don't know quite how it would work. But yes, like the the hooks, the same hooks same mechanism and yes uh exactly we should try to. We should try to make sure that it works for for cluster, API and and uh other scenarios as well.

A

Provider- operators- plural, maybe I, don't know they.

E

I was just going to say: I could immediately see a use case around this uh I think for pizza mentioned flux, but we use Argo CD heavily I could easily see um creating a similar provider around rocd um we've already kind of done. Some minimal work not directly in cluster API about just in some interval code that wraps it to do that.

E

um I can definitely see that following the same contract um and then you would get all of the all the support that it has already for Helm customize bear manifests, etc, etc.

E

So yeah very interesting.

A

Yeah definitely a lot more providers can be added, which immediately uh brings a concern that I have that we have, as a Specialties group way to manage some projects, and uh I can only imagine that if people want to uh want us to host all these uh adult providers in repositories, we export having so many projects. So I wonder: where should we draw the line? We have been pretty? You know not non-conservative when it comes down to adding Costa Rica infrastructure providers, but with other providers.

A

I worry that we can pretty much double the number of constantly PSO projects which it could be okay, but, for example, Zoom meetings uh are pretty much out of the question. I think at this point in calendars, uh I think it's romantic. Sorry, Zoom meetings are okay uh calendar. We have actually a limit we're hitting a limit on the calendar. We cannot have more so projects on the community calendar for some reason.

A

I think Bobby can explain more about this, but um I don't know too many super projects, but at the same time I'm wondering we should have a like a better folder structurally requested. Api is a signal in its own.

A

You know so rich.

B

Yeah, let me say the technical problems apart. I think that the problem is not having some projects, but the problem is the attention is that we are.

B

We should become. We should be better in uh ensuring that all these projects have a minimal standard in managing the community, and second, we should kind of do a better job as a Sig to make sure that what we are proposing is is part of, uh let me say, consistent story and, and we go in a direction. Maybe we experiment a different Solution.

B

That's fair, but uh let me say: I do expect that the role of the Sig in terms of a guidance or consult on and yeah somehow to be carried out in a different way, but we we can discuss with regards to setting a bar. What I've discussed with uh Jonathan and Cecile is that when Jonathan asked me, I basically told John hijanta. If this is a one-man project, it is hard to say we should get this into the into the Sig, but basically dense seal step.

B

It up and and told me that the Microsoft is interesting. So it is behind this project and so I easily see that this can have its own. uh It's some Community behind so I think that yeah we we are a place towards the project, but we are a place to grow community. So we can set the bar around minimal Community minimize set of people making a project.

A

And we have to have some sort of a bar around popularity of a particular solution. We have to evaluate if this will be interaction for a community to know. I think, like you say, Helm is probably going to grow uh something else.

A

Basically, we have to think about it. Yeah.

B

If I may add.

C

Sorry, how much at a very high bar so I'd actually say that if I was a betting man, I'd go the other way, but uh we can have that bet separately.

B

uh Another idea that I was thinking and sorry for not sharing before is that we can kind of tell new projects that we allow them to incubate for a certain time and then we re-evaluate and if after I don't know one year to a year, we see that the projects still have only one contributor or two contributors or it is uh uh one company show we have to discuss together. If the community is the right place for that project,.

C

Yeah I I would I would hope that in general we we don't require these things to be in kubernetes. Six like having the extensibility mechanism is, is important and so that, if you want to do it outside of of kubernetes six, uh you should not.

C

You know, have a horrific you shouldn't have to Fork cluster API to do so, and then, if there are multiple companies that want to collaborate on it, then we can talk about you know, should it be in kubernetes six, but like a lot of those Cloud providers, for example like cluster API providers, presumably if they don't have a lot of contributors could be under the the cloud provider themselves right and so, uh but I think it's fine, but I, don't I, don't think we should I, don't think we should I think we should generally ex I, think we should put in the hooks and expect that a lot of repos are going to be hosted elsewhere.

C

uh So, like I, don't think we should necessarily have to host all these meetings right off. The bat like it isn't the case that that I will immediately be asking for a repo for uh cluster API add-on provider, cluster add-ons or operators or whatever it is right. I would expect to host that an existing repo.

A

Yeah I think we can also, if at.

D

A

We can basically have fold the other related topics to be inside the main course return. I think it eventually happens to be too much for that group to manage. We can have an office hour specifically dedicated to others. Of course the API uh is doable, but I think we are pretty much hitting the limit on the calendar at this point, so we have to I think we have to remove one meeting to be able to add another one. It's like that um yeah.

A

So with that in mind like should we should we allow this uh Helm repository to be created now uh I, genuine, plus one but uh I, don't know I just uh I'm concerned that uh growing with too many adults and like for basically saying we have to set up a bar, somehow yeah, listen.

C

I'm I'm plus one because I feel like they've, done the work to to integrate and we need a reference implementation um if there was another one that we wanted as reference presentation, I think that that would be also okay, but yeah I wouldn't want to we're. Not we're not saying all of them were saying.

C

This is a couple of reference reference, implementations or one one or two reference, implementations and they've done the work to do this, and this is the expectation and it's not setting a precedent um and Helm is a very popular choice these days, so.

A

Yeah well, basically, this is not a you know a great example, but uh in the cubeadm uh kubernetes communication we had a bit of a conflict of interest. Eventually, where we had a pretty much a link uh where people can install uh cni, it was like a table of popular cnis. You know popular is clear, the subjective, so there's some people who came to us and asked us hey.

A

Why are you not having my cool, uh you know add-on to be part of this table and we said like we don't know who you are uh it's a new project? We don't want to add it. So it's not fair. They responded so potentially with adults.

A

I don't want us to be in a in a similar conflict of interest situation, where somebody is complaining that we're about creating a repository in kubernetes six for a particular adult uh that that's why it's I think it's pretty much important to try to create some sort of a bar or if, if it's too much for question API as a project, we could just we can make the decision here. I don't know uh just uh present this to us.

A

If nobody has heard of a particular adult solution, we uh we might say okay, this is we don't know what it is. We probably should not allow you to have this Repository by the law of another way. How would you do it God, damn.

D

Okay, can you hear me.

A

Oh slightly quiet.

A

Okay, uh well, I think we can press one. This.

A

D

Oh I, can you hear me now? No yeah? Oh, you can't. Oh okay, sorry uh yeah, I, I. Think I just had a question, so this is a reference implementation. uh Will the repo or you know the project? Will it be open to uh let's say so, adding support for another another tool apart from Helm uh the so the re, the reason that I asked this is.

D

uh You know I think when we you know when cluster API began right and when we uh we had the reference kubidium bootstrap provider, but it became the the de facto you know. Provider used. uh You know almost almost for you know, for for all clusters, regardless of infrastructure, I.

B

D

B

D

Are some um uh Talus and et cetera? There are? There are some exceptions, but uh we in, for example, the AWS infrastructure provider. We um have some issues, I think that we still need to work through, because uh we made assumptions that you know the bootstrap would always be done with kubadium, and so in this same way, I I worry a little bit that a reference implementation that you know just right. It's just a reference simulation.

D

This doesn't mean that you, you have to use Helm, but because it's there and it uses helm other you know there will be other assumptions made uh that that may cause problems for for something other than Helm, and so so it would be. If you know, if there are owners- or you know, people interested contribute, an integration, that's other than than Helm would would the project support that.

B

uh I cannot answer that about. Let me say what the project we support. I I think that we have learned a lot over time about integration and what we are saying is perfectly right, so, even if you start with the best thing that about integration, if there is no uh different people trying uh going through the same integration point, you you, it is hard to discover where they are weak or not generic enough. So, but this part, let me say, of a project mattering In, This, Moment, In, This Moment. The integration is really simple.

B

They are just watching cluster API. So there is there. I, don't see this risk now and also using lifecycle looks they they are using this. They, the idea, is to use the same. Other cycle looks that everyone else can use so I, don't see this risk but understand your concern. So I think that it is a good point to to discuss with them.

B

Honestly, I I think that they they uh at this stage they they have enough on their plate to solve uh instead of and they will be kind of wary to not scope, scraping to something that but yeah. It is up for discussion. But I got your point. It is a it is a reasonable concern, but at the same time being doing great extension Point takes time and we have to give them time to to get there.

A

Perhaps the health providers should be more clearly stated as hey. This is Alpha and uh don't rush to integrate into it, because we are at stage one.

C

Yeah and maybe another way to to make everyone happy, is uh unfortunately they're not here, so we can't communicate this verbally but like if we like basically say that we would not expect we, we are treating this provider as a as a first or reference implementation, and we would not expect the cluster API to to put in things that are not that are specific to this provider that are not. That would not be used by other uh add-on providers. In other words, they are not.

C

They are not special they're, just the first type thing, even if even if we never see a second one like we would expect a clean architectural separation.

A

Yeah I see the stated here that is working. This is work in progress. It may change at any time. Oh yeah uh for British, have you discussed with Jonathan or somebody else that potentially having this uh batteries included. That will provide a recite the copy report, similar to other things.

B

uh Honestly in cluster API, we already have uh three providers to Experiment three, three or four experimental features that don't framework the complexity there is already is already big and uh in my opinion this is better.

B

We can also eventually reconsider, like we did for uh Cuba mean provider that a certain point we we move the in three, but uh let me say, managing these complexities is getting difficult inside the copy repository, so I kind of prefer to leave them experiment freely.

B

This is also what happened and with the operator we started operator in copy it was ours to get a good. Let me say working balance between the two teams and uh at the end they move it to a different rifle.

B

A

It is a trade-off.

B

If we look back at kubernetes, they they add all in three and down and moving it out. uh Our Prime cons.

A

Yeah separate repository, so a lot of tips that clear disagree from this point uh that he prefers the monolithic model, I personally of repository management, I. Think I'm, not a big fan because uh you have like 200, plus issues and uh I prefer the separate trees for everything and then some sort of a modular Junction process puts everything together. I think this is how I prefer it, but yeah.

A

Maybe the best students provider to live in three will be something that is the success or, of course, the resources like a very raw um adult provider. But given the like owner files, medicine, which many issues I, think it's better to start out of three and then decide if you want to move it back.

C

Yeah it's at that point. It would be it's not a true reference implementation right, if we're being honest like it is, it is in that regard more the one that's doing the work and the first one like a a true reference. Implementation would be like a real stub and maybe in that world. Yes, we keep that one in in that true reference implementation entry, but in practice we have to build it.

C

This way we have to build a real implementation that that does the hooks and we try to make sure that the hooks are relevant to Future providers and if we need to move the home one out at some point in the future. Hopefully we don't, but it shouldn't change too much.

A

Yep, or maybe just the pure provider, has this excellent folder with like a hey by the way we have this detailed computation. This could be the reference provider, but yeah I think it deserves like a separate reposit.

D

A

Before when copy started, originally we had this uh infrastructure provider templates about adorable. If we have like a disciple, uh I, don't know, what's happened today we rewarded, but we had it for a long time until AWS when we started Azure started like everybody was using this, it was based on gcp, basically I think uh I could be wrong, but anyway we just use the gcp information. Do you remember.

B

I, don't remember: we had something to clone. When there was the V1 alpha one time, then it will answer to.

C

B

C

Of it, when we did, there was a big change. Was it was it? um Was it the move to crds? Did we have like the accurate I can't even remember the timeline? Yes,.

B

Etc, etc and we dropped the uh degradation API server Etc there was this bigger factor and basically, each provide a big common, a reference implementation. At the beginning, it was Kappa which was the the one moving a little bit faster than the other or or a gcp, then Kappa, and then now they are more or less all at the same pace.

A

All right, unless somebody else has uh questions or more comments from this I guess we can move through giving up first one on the tickets, um uh just questions: how is Vince about the this Vince has any comments about this proposal. Overall.

B

I I think that he's supposed to so he never show the to be against this. When we discuss this in cluster pi, let me say if I look at this from from a Sickle Cell, 30, I I, think that is the same idea of Justin uh we have to. We know that we are.

B

We are still not where we would like to be in this space and if there are people that would like to experiment uh and and do it in the right way, we should basically work on them.

B

From up one more point of view, we are using K up in our product and not Alma. So, but uh let me say if there's an essential point: it works well, we can do our new, our duties and and integration Downstream as well.

A

All right: well, uh please, uh please want this issue and selective I think we exhausted the topic. Thank you very much for being so. You pretty much presented the instead of the Jonathan. The future.

D

Maintenance I think. Actually, you are.

A

In the future, almost I think.

A

Okay um and that's all, we have I think uh any other topics for today.

A

All right, we have 15 minutes back. Thank you, everybody and see you again in two weeks like take care.