Kubernetes SIG Cluster Lifecycle, 31 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cluster Addons meeting: 2020-03-31

Description

Meeting notes: https://docs.google.com/document/d/10_tl_SXcFGb-2109QpcFVrdrfnVEuQ05MBrXtasB0vk/edit#heading=h.yugdr2ba2zfv

A

Great so we're recordings at the beginning of our cluster add-ons meeting, which happens every two weeks if you're joining us in the future from YouTube greetings fellow time travelers.

A

This is where we talk about making add-ons more consumable and making better patterns that are UX, but our packaging formats, as group is coming with a lot of terrific ideas that I'm excited about and I think we may have a shorter meaning today, because our agenda- it's not very populated, so she might be kind of nice, given current events, I'm just going to share my screen so that we can all see the meeting notes if anybody has decided to put any in feel free to add your name to the attendees list.

A

If you desire and do we have anyone new journey from us really I haven't seen you in a while. It's really good to see you, hello,.

B

I can sort of reintroduce myself I'm now at ISO valent working on cilium project, and there are a few few things about this group that are relevant to what we do. It's in him.

A

Yeah, it's sitting on this super exciting technology and looking forward to working with you in that context, more and then yeah everybody else is pretty regular here. So I guess I'm the Sicilian topic we can just kind of we don't really have an agenda or anything. So we can just write.

C

B

A

B

Yeah, maybe yeah I haven't I, haven't quite figured things out. I just thought that start joining these meetings again I could.

C

B

Few things that I need to start looking into, but there are the things that's button in my way, so the.

A

Major thing: that's most interesting to me from an add ons context with psyllium is I'm just gonna. Add some notes here.

A

Is that a cilium has a coup proxy replacement, that's built in.

B

A

Implementation yeah.

B

That's right, yeah and then.

A

Obviously, it's very configurable CNI similar you know to in terms of configurability and complexity, to things like calico and some of the OVS based solution so add-on configuring that add-on upfront and it's you know CNI is totally in the context of the cluster add-on, which is something that you know it's required.

A

You know in the bootstrapping process like needs life cycle that matches with the cluster at some level, but when I think about the coup, proxy or replacement feature, that's one of the biggest use cases empathizing with the community and with like third-party vendors and projects that are producing networking solutions that replace coup proxy and then having Covidien. Being this very supported upstream tool that you know is in practice endorsed. You know by cluster API, and the kind of thing is: there's no way to disable. Really the installation of good proxy right now, right.

B

And yeah, that's that's one thing. There's a data point, though the proxy replacement is only fully functional on newer kernels. So that's that's the other end of the stack that somehow needs to be brought into the picture. So that's something to think about a bit more as well right, yeah,.

A

And then I'll mention that COO browser as well has an IP, vs plus IP tables based one yeah.

D

Like a world where you have nodes running different versions- and you want to like make this to work together or you just want to like, not install, if you don't have the kernel, let's say this clusters.

B

I guess they could be different scenarios, but it's probably more common for for for for people to have same networking set up on all nodes.

D

Yeah, it's just curious if that was like a goal or just adding extra complexity, that doesn't make sense so.

E

Outside of the cube proxy kind of conundrum, if we're talking about an external system like cluster API, we also have other challenges when it comes to C&I providers as well, and that's there are certain API server arguments that may be required for a particular CNI or a particular set of firewall rules that you may need for one C&I versus another, and we get into this weird bootstrapping thing where your choice of C&I, which right now we handle it's like a downstream concern, actually impacts the bootstrapping of the cluster, because you have this codependency between these components.

E

So that's something that we're interested in trying to figure out how to solve from the cluster API side, and we would very much like it if a project like cluster addons was able to solve it in a way that we would be able to consume both at bootstrapping time, but also after the fact for lifecycle management of these add-ons for the clusters that were bootstrapping as well.

A

I'm really interested in well what your thoughts are Jason on how we could like what is the useful layer of abstraction and level of abstraction like for solving that coupling.

E

So I think there's kind of a couple of layers here and, and one of those layers needs to involve some type of metadata that would need to be published by the authors of said.

E

Add-Ons to you know have a way to express kind of these dependencies like conflicts with you know, cube proxy or replaces cube proxy, or something like that. In addition to kind of other things like you know requires you know, kubernetes be configured with a pod cider that matches this, or you know some way to expose this information in a way that we can consume at bootstrapping time from cluster api.

E

But then also, you know the downstream concern of you know cluster api itself, not managing these add-ons directly but managing them through. You know the cluster add ons project.

A

Yeah I think we.

C

We discussed these kind of things Lee and.

C

So there's kind of two levels of answer: I think one one is that we very much wanted to be able to to strip out the hard-coded stuff from places like cube, ADM and then so. That gives you a way to solve. The replaces cube proxy question because you just have like nothing or you know whatever the absolute minimum is and then you and then everything's an add on including cube proxy.

C

C

In my view, several steps up from there as you get more complexity is, is an operator. So you know to the extent that there are complicated things to do in managing the network add-on or some other kind of add-on.

C

Put that complexity in an operator and then that's the thing that you install through the the kind of yeah mph or an add-on, and then it figures it out. I.

C

Yeah I mean my gut feel is things like needs, a certain kind of cider, or something like that? He's we're never going to have generic usable abstraction for things like that. Yeah.

E

Any challenges as those chicken and egg type scenarios, because you know if we're looking at like you, know the bootstrapping scenario for cluster API and spinning up the initial. You know cluster. You know if we don't set it up with the right.

E

You know hot cider, for whatever C&I wants to be deployed. You don't even get to the point where you can deploy the add-on manager, because without C&I you can't actually deploy any application. That's not using you know, host ports, yeah.

C

I'm, pretty familiar with stalling, CNI and and thinking about so I believe I've had the issue open for about three and a half years that we need to be able to back to the cluster pod Network, so either firm from kubernetes, but in the in the context of installing and bringing up a cluster I mean I, I think I think that that can be a top-down thing. You know the person requesting a cluster can say: here's the here's, the cider that should be used, or probably a set of them for for ipv6 and so forth.

C

Here's the cider that should be used and it gets passed down and it's available at the point of add-on, installation and I, don't think passing things back up again and trying to negotiate and so on is ever gonna fly. This is just my my opinion. It's worth whatever you paid for it. The.

E

Only reason why I bring that up is because, in some cases you need to do that in other cases you need to set the you know you need to configure the allocate. You know you know, notes right or flags, and you know there there's different things for DIF.

E

He and I providers, especially if you look at flannel configuration versus something like psyllium or calico, and you know that the the chicken-and-egg is the difficult part right now, because the answer that we have right now is the onus is on the user to know the three different places that they need to set these things, so that you know they don't.

C

Worked in this area for five years now, so just pass it down pass the information down and the the networks will adapt. Well, maybe not flannel yeah.

A

As a mechanism that that does make sense, but yet there's certainly the UX consideration um when thinking about cluster api, kuba Dems component config the cops configuration you know, the clusters, api from eks cuddle and then combining that with some of these, like we've, proposed the add-on installation api which doesn't need to run from inside the cluster that could so from a context of bootstrapping right using the same API right to to get that ons into the cluster.

A

When there's no CNI is a thing that was the proposed integration with Cugat hem, but uh composing things like the EPS cuddle, cluster API or cluster API itself with the Covidien component. Config gives you the bare minimum like validated objects that would allow you to specify the same thing in multiple places that certainly been kind of my like intention or expectation of how the add-on installer would compose with tools like KU vidya.

A

You know so that you could, from the start, you know say: I'm not using ku proxy I'm using psyllium with these flags enabled I have machines that are of this kubernetes version. This operating system in this kernel I'm using Covidien to bootstrap the API server and scheduler with you know these arguments. My nodes are annotated in these ways, so there are route reflectors or whatever right and there's a lot of like the same value. You know being used in multiple places right or values that are related to each other or calculated from each other.

A

But then these objects have to be composed in a similar way. That deployments and ingress- and you know, service annotations with external DNS or something would need to be composed together and then oftentimes you're also passing. You know like in that example, a DNS. You know fully qualified DNS entry like into your application, so it can do routing or something and so like as somebody who's working with all of these API objects and composing them together.

A

The those fields are duplicated and it's on you right to make sure that it's done right, and that is difficult.

A

So then there are tools in the application space like helm which allow you to create abstractions, that kind of like Brian. You were saying this top-down like approach where you're now feeding that value all the way through these places.

A

Is there either tools in the community that we can use to provide good examples, and maybe even things that are experimentally, endorsed right to help people create clusters with complex networking options is probably the most interesting problem space to me. Oh.

E

I think the challenge there is is.

E

If we're trying to if the goal that we have here is to kind of solve the problem in a generic way, that is consumable by the vast majority of users and tools and- and you know, d-do trying to deduct the space that we're in if we start looking at templating tools and template, which is and and things like that, if we get to, if we tie ourselves to one solution in particular, we vastly limit the value to a whole other section of the community.

E

So if we tie our kind of you know, hitch our wagon to helm, for example, to solve these problems. For us, there's going to be a large part of the community that isn't going to be interested in using helm to solve those problems.

A

Yeah, that's certainly been at the like that valuing freedom and a level playing field and extensibility has been like the center of our conversation. Since you know, early last year from cluster Airlines I'm.

B

Not totally sure why we talking about templating and such things in these context be kind of theme. More of this thing that it seems like what we should be discussing on this topic is whether whether there should be a high-level capability like an idea of a high-level capability in the cluster say: core DNS has the provides the DNS capability. A CNI add-on, provides a general metric of ability and some CNI atoms provide a service load. Balancing capability and q.

B

Proxy is just a service labonz capability provider, or something of that like that I, don't know, what's the the the best language, but something that that would manifest itself, and you know in a fairly high level, hopefully simple enough API, where we could. You know, look at a cluster and say: okay, this cluster has a service load, balancing capability implemented by Q proxy and are we gonna go and get rid of Q proxy and install psyllium instead and whatever the whatever the networking capabilities provided would be? We could swap that out or work with it.

B

So that's like from Salim's perspective right and from the perspective of supporting a case where a cluster is running already, let's, let's now install psyllium for the purpose of testing psyllium right, just slow down. Let's answer the.

C

Question that you asked mark, oh.

B

Yeah, oh yeah, when.

C

We tried to do the simplest possible thing. We found a need to do this kind of IP address plumbing into the config, and so we look for a solution for that. So I have no argument with your desire to work in in higher level abstractions, but mm-hm.

C

We also need to be able to I think replicate what we have today. You you, what we thought should be possible is to take the hard-coded. You know hard-coded config, for core DNS and replace it with a soft coded add-on config for accordion. We thought that should be possible. Look. We thought that should be the simplest possible thing. No suddenly and immediately fell into this trap. You know you need to do an IP address, computation.

F

E

So one of the other challenges that we see from the cluster API side is also infrastructure requirements that may be required, especially in the CNI space. So, for example, you know if you use one CNI, you need to ensure that be excellent. Traffic is enabled between nodes or, if you use another one, you need to have this port enabled.

E

You know with this protocol to ensure that things work and you know trying to fit the right infrastructure requirements for the right. You know C&I provider in particular, but we also can look at other other things as well. If we consider things like CSI providers or Cloud Controller managers in scope as well, those could also introduce I am requirements between the infrastructure provider and the plug-in installed as well. And how do we express those requirements in a way so that we can ensure that you know those infrastructure requirements are met?

E

For you know, the required plugins to you know, stand up a functional cluster.

B

Yeah and well, however, the cluster api is much better position than cube admin right. Cluster api could actually do something about it and on the on the on the point regarding the ports, I would imagine it should be possible to actually read ports that, like a demon set uses, obviously that's a little bit indirect, but you probably look at the reports that are actually being declared in the daemon set or whatever the pauses are yeah.

E

But that also requires us to kind of introspect, the C&I provider and, and that sort of thing that's where I was get trying to get into the concept of kind of exposing metadata that we could consume. Rather instead, that could be in a more kind of version and supported format that we can more easily rely on, because you know otherwise.

E

We'd have to know whether you know this is this plug-in is deployed as a daemon set or a deployment or or whatever, and try to introspect across all of those various things which ties us into kind of doing certain things towards certain implementations or basically trying to figure out the world, sometimes generically, and so.

C

That it's a really really great point. We make Jason, and this makes me think, there's there's a kind of an N by M, matrix of things, cost or API level. Things that are capable of implementing like like I, am, for instance,.

C

Some formal rules I and there's the sort of one set of dimensions along that side and then there's the other set of dimensions about things that either care or do not care. You know like like a C and I plug in a CSI phone again whatever and.

C

So if I, if I, think about it's kind of a similar situation in in CNI world, where, where we said there we're actually not going to put all these things in the spec we're going to put them in a separate place which we call conventions and and basically it's a lot more flexible people come along and say it's um so back in back in your ok, someone can say: well, here's para maana, here's a kind of a CR D, that's going to tell me about I am requirements, and people on the consuming side can implement that people on the plug-in side can add it.

C

Let's call them add-ons. One can implement that.

C

There's gonna, there's gonna be what's a good word. You know kind of jostling in that space. This there's gonna be multiple people coming up with comparable requirements and they're, not quite exactly the same and but yeah I, guess I, guess I feel it could could shake out. You know, there's probably not thousands of these things. There's probably a couple of dozen and.

E

And from my perspective- and this is something that we're we're trying to enforce in the cluster API space, because we have kind of the core cluster API stuff, which defines kind of the interface- is that individual providers can implement for doing this sort of saying anywhere, where we can push kind of the definition of a lot of this stuff on to the folks providing the implementation where we can the better, because you know one, they know.

E

You know these things better than you know somebody working more generically in the space, so they they're more easily to be able to be authoritative over these things, but also you know just having a way that we can. You know through contracts, you know, consume that information in core cluster API means we're not kind of hard coding. You know as much on the course I'd and and I'd like to see something similar here, so that you know it's not on the add-ons project to kind of curate.

E

This information, where we can kind of defer it off to you, know the folks who are managing the individual kind of add-ons instead.

C

Yeah, let me just find a link for the conventions that you know. I was talking about.

C

It's not exact match but turn something comparable.

C

D

D

A

We looks like you're talking but gonna, be my bad yeah thanks.

A

This idea of exposing metadata for the add-on and relating that metadata to the add-on is, is interesting to me operator. Fermor has done some of that in relation to an operator there's this challenge that we hit. Whenever we talk about grouping these the actual things that make up the add-on together and we've, we keep running into this missing idea of packaging, and so, like tools have have done this to varying degrees. You know with git repos with charts, and we have the OCI packaging kept that like uses docker images at as transport.

A

Getting figuring out at an API level like what the thing this apply to is interesting enough add-on. Installer configuration also does this, so there there's, probably some minimum amount of introspection, that's necessary from something like cluster api, to dig into the add-on to at least find the metadata custom resource, or something that we create. Is that something that you were thinking like? Is that how you would think of it? Look, Jason or or anybody else else, Rek as.

E

Long as there's a standardized way, you know to be able to do that. I, don't think we're necessarily picky about what exactly the mechanism is.

E

You know and so to be clear, you know: is there a centralized component that is going to be managing all add-ons, or are we looking at individual components to manage each add-on in this model? Sorry, for my lack of familiarity from.

A

What we've proposed the central component is a library and an API that are just responsible for doing a top-down approach and storing a minimal amount of state about what is applied, and so that allows you to. You know. Whenever you invoke the add-on installation library, take the packaging format, that's specified and the reference to that package and then make sure that it's either applied or removed from the cluster and.

E

A

E

How about further lifecycle operations for that add-on? Our.

A

Answers or that, if is if the lifecycle is beyond a coop, CTL, apply or delete and a pre step of inflating the add-on from the package. Then the package should install an operator that that deals with the add-ons lifecycle in a declarative way.

E

Okay, I'm just thinking about things particularly more like cute proxy that are tied to you, know the particular version of kubernetes that you're running in the cluster, because, obviously you know those ones that are more dependent are going to be more of a challenge on errand to ensure lifecycle is up to date. On those then say some, that's more a little bit more disconnected like Cordy and s, for example, yeah.

A

Where coordinates can run inside of the cluster, you know like without much help, although it does mean some cluster details well,.

E

I'm thinking more along the lines of you know the version of core DNS, somewhat disconnected from the version of kubernetes, and you have a little bit more play there in you know version drift and worrying about you know: updating the life cycle of those components versus something like cube proxy, where you know you kind of have to make sure that you're in you know at least n minus one. You know for the related kubernetes cluster I.

C

Mentioned I think it should be possible to install keep rocks, are using the add-on mechanism. I'm not I, didn't mean, therefore, that's the way we should always do it.

C

You know that there may be things that are that are just too complicated, yeah, I guess, I'm I, don't know I'm trying to I'm always trying to get the the simplest thing that could possibly work out and in contact with the real world where we'll learn much more than we will just trying to think about it. So.

C

So yeah, where there are complications that are not strictly necessary. My my instinct is to kind of want.

A

Yeah I mean upgrading good proxy from a good bit. Am standpoint has been you know, not the most complicated thing and so like when the cluster you know goes through the upgrade like supporting a new application of the coup proxy daemon set, and then, if you want to do something more specific, like target nodes of a particular kubernetes, a version, that's a certainly within the realm of like a declarative, apply.

E

Yeah, it gets kind of complicated, though, because when we looked at it, we recently tackled this when we started dealing with upgrades with cluster API mm-hmm and the way that we're handling it we're not doing the cube ATM upgrade operation because we're trying to keep individual machine instances more. You know independent, so we're not trying to mutate them after we create them.

E

So we basically stand up a new, a new machine and cube ATM, join it to an existing cluster, and that new instance that's joined is the newer Cooper Nettie's version and what we found we had to do.

E

Is we basically had a reimplemented, what cube ATMs doing to upgrade cube proxy at the end of that control, plane management, but as we did that we realized that now, if we update that daemon set after we do the control plane now we're updating that daemon set for all the workers, then may not already be that aren't updated yet so updating the daemon set at the end of the control plane. You get into the situation to where you're running the newer version of cute proxy bendy cubelet that it is running on those worker nodes.

F

E

You know trying to properly handle that lifecycle, who probably should wait until all of the worker nodes for the cluster are also updated before we update that daemon set to ensure that, while it hasn't been an issue today to run a newer version in queue proxy against the cube, let it does break kind of the API guarantees that we have.

C

Missus that we didn't used to install cube proxy as a daemon, so who's, who used to be aesthetic manifest on them on the node. It's a dam and said now well, yeah.

A

Yeah I wonder when that change was made, because as long as I can remember, it has been so that that's probably since, like at least 111.

A

The so this makes me smile.

E

A little affine to defer the because cute proxies is like a weird edge case anyway. I.

A

Don't think that this is so different from a lot of updating workloads in kubernetes. There is I think a a missing link here. What I hear in this upgrade story is that a node local daemon, that's supposed to be versioned in relation to the version of the kubernetes components that are running on it. Mainly. The couplet is not being scheduled in that manner, and we have tools to constrain this, mainly I'm thinking of node, selector and its equivalents.

A

We do not have to have one coup proxy daemon set, and it may be a logical error to think about the upgrade in that manner, in the same way that it is a common logical error to reuse, config, maps and secrets inside of a deployments rolling update, very very common issue.

A

Is people try to reuse the same config map, but that actually breaks the semantics of using multiple replica sets that are supposed to point to different versions of the configuration and the same thing with secrets when you upgrade a TLS cert on a web server, you supply the old one and the new one right and in in the same way, if we're trying to support version skew multiple of I mean this is what API machinery and our versioning strategies are for.

A

Then we should support scheduling the appropriate versions of coop proxy to the appropriate node pools. Does that sound, crazy or logically false to anyone.

E

I mean I definitely agree, but you still have lifecycle management challenges around. You know cleaning up the no longer needed you know configs and Damon SATs, and you know creation of the new ones. You know in advance of when they're needed. There's.

A

No cost to having like a gaming set with zero nodes that apply to it.

A

And I think that that that kind of, like desire for cleanliness in a namespace or something is, is maybe you know something that we could look at. But it's certainly not a first order need well.

E

I'm thinking more along the lines of you know, if you do it, you know taking the Q proxy example. You would basically have to have a daemon set for every supported version of kubernetes down to the patch release. It.

A

Would yeah it would be the ones that your cluster has gone through. I. Don't think that that's that unreasonable then.

E

You also have other challenges too related to if somebody's providing kind of custom images and things like that, so there's still kind of this lifecycle management step around you know creating these, you know and I guess he could create them on demand. Somehow yeah.

A

I I think it just the problem that it solves is having multiple daemon sets that are version specific is.

A

It's in it's solving the issue of imperative state required during the upgrade I. Think. If you can solve that problem by using the scheduler, then you are back into declarative, bland and then providing custom images. Upfront cleaning up a namespace. When you know no machines of a particular version exist anymore, like these are things that can happen in the declarative, reconciliation, loop inside of cluster API or kuba TM, or any cluster management tool.

A

It's getting into cap territory, which makes me kind of sad a.

A

Yeah I'm not sure if I'm clearly expressing, hopefully you can see a little bit of where I'm coming from here but yeah. Just to like what Brian said about like reaching for the simplest thing, it would not be the first time. In fact, we almost have like a prior art, a pattern of upgrade strategies and bugs and miss applied.

A

You know logic inside of things like kuba diem, for you know getting the components to upgrade across a kubernetes version, being you know, kind of just enough to make it work, and we certainly could you know, as like future versions of the tool evolved. Like pick a different upgrade strategy, I mean that's I, think it's fine to question that.

A

To me, it's always made sense that you should have the ability to schedule different versions of coop proxy or constrain the scheduling of good proxy to a particular group of nodes.

E

Yeah I agree, and it's actually wanted like. If you look at like the upgrade Docs, that's one of the things that's recommended is to basically run a different daemon set for queue proxy for each of the versions that are part of the cluster and using the node selector. For that.

E

But that you know I'm not too worried necessarily about cube proxy, but when we start talking about the lifecycle, management of various CNI providers and other kind of add-ons, you know I want to make sure that whatever that you know upgrade process is, is something that you know a consumer such as cluster API doesn't have to know. You know all of the details about you know all of the various supported add-ons that we wish to kind of support.

E

You know that that we would have to codify logic, basically to manage a lifecycle of all these things. You know in a different way. You know anything we can do to kind of formalize that lifecycle management helps. You know that consumption of various you know add-ons that the community may produce yeah.

A

Certainly, the idea of removing something when all of the things that would require it to exist is interesting, but yeah I think as much as we can encourage a declarative lifecycle. We should attempt to, of course, the community to produce add-ons that provide solutions that are declarative like and.

E

And one of the examples that I, like one of the things that stands out to me, is core DNS right now and the fact that you know if you update from one version to another, you may have to update the version of the core file. That's being used to feed the config.

E

You know right now in cluster API, where we're. Actually you know we have that step where we're ven during the library from core DNS to you know, inspect the version that we're going from and to, and you know, converting the config if we need to and all the mechanics of swapping the config map that feeds into core DNS.

E

All that as part of the upgrade is all kind of hard-coded logic that we have wrapped around the the core, DNS migration library and, ideally, that's something that we would like to not even have to think about other than just saying. We want to update core DNS from this version to this version, as requested by the user to.

F

Me that sounds a lot like what the core DNS operators should be doing, if we're using an operator to manage it as an add-on.

F

Mm-Hmm I think that's a really good point. Actually, we've been talking about like packaging, add-ons and operators, and you know, we've been looking at the coop declarative kudos are the COO builder declarative pattern that takes set of yellow files and applies them, but that's often not sufficient to do a declarative, update between versions like I think it's the point that you're making it's a really important one, and maybe we should talk about that. Some more yeah.

A

The other bit here, that's interesting is the leakiness of API machinery in trying to manage things that do not use API machinery, so core DMS being configured by a core file, not having readily available tools that are integrated into kubernetes. You know to deal with the core file and its associated API required. It ends up. You know, being that, if you want to work with an upgrade, then you write a bunch of code and import libraries that you don't feel comfortable, importing and owning you know into Cappy and the same your into the same problems.

A

If you deal with you know a Tamil file for a load balancer or you know, somebody's I and I.

A

It's it's really interesting to see like in the helm space because they have yeah mol and the values merging semantics, like the solution in helm. For this, that kind of generically applies is to template the config file and then convert the config. You know into the values animal so that you can do merging and overrides and all that, and then somebody decides to maintain that. Hopefully, so it's much better if we can have.

A

I mean it's, it is a solution. You know to use an operator that owns that domain and then, basically, the operator is exposing the abstraction for you to convert your declaration into what you want done to your config and when, when that should happen, and all that I.

F

Think there's a problem: if you don't do something like that, like a core file or a reverse proxy tamil file. I think is the other example. Yes,.

A

F

You're not managing that via some other interface, that an operator translates for you or something along those lines.

F

You run into problems where there's an existing configuration and you're updating to the new version, and now someone has to make the decision of which parts of that configuration the user owns and which parts of the configuration the system owns and can automate it and like whether it should automatically update or migrate fields or sections or whether it should I'm give up and say, you've edited some stuff. That's I, don't know how to upgrade, and you need to resolve this manually. That kind of thing yeah.

A

This makes me think of like when you do an apt update, like D package like marks, the package is old, config right or even if you uninstall it like. If the config was not managed by the package, if the configure shops don't match right, then it says: oh, a residual config like left on the disk and uh yeah. That's it's really really tricky. I think that there is a lack of.

A

Practice of like composition, so our core file right now that we ship by default in cubed iam just has the single core file field. But when I've been working with some multi cluster networking examples kind of just on my own and what I found is, if you want, like multiple systems, collaborating on the core file to say, configure multi cluster DNA in a mash or something, then you it's best to break up the core file into different pieces and then like import them and have environment variable overrides and things.

A

So we we could do some of that to empower users to be able to. You know like set the clustered domain like without the upgrade tool having to know what the cluster domain is. That kind of thing.

F

I'm not sure I followed that last break. Can you give an example of what you mean yeah.

A

A

A

Hopefully, you guys can still see the screen, but it's basically in was it customized lib Cordy mess here the I'm using a get ops approach to override the core file that is produced by kuba diem in order to add an additional import statement, and then the extra zones environment variable is added after cluster local inside of the kubernetes plugin config, and you can break this up even more with like smaller imports. For you know the the error reload and you know health configuration right like that- could be imported from a different core file same thing.

A

For like the metrics thing, it could be imported from the different core file, and so this this is a very general. You know default configuration that does work for most people and then I've kind of decorated it with these additional points of extension, so that you can provide a different config right that is optional.

A

So here's a patch for the deployment config, which allows you to specify an optional core DNS configure inside of the config map right. So this contact map doesn't have to exist in order for core DNS to run, but if it does, when the core DNS pod is created, or actually this is managed by deployment. Yes, so that that would always work yeah.

A

When the config map is created, it gets mounted, you know in the conflict door and then core DNS will reload the config and same thing: I haven't added and oh yeah, accordion se and V. So a separate config map. You know to override the extra zones and.

A

You know you could add a system owned one and a user owned. One.

F

Right, I think the the pattern is a really good example, I think when we start talking about how can we say something? How can we preserve this property for any add-on? I think that's where it gets tricky, because you have this problem even with you know we're talking about a core file here, but you have this problem with Kubb resources as well.

F

How do I know if a field of a particular object is owned by some system component that will overwrite things or reset defaults versus what the user intent is indicating like there's there's some distinction based on whether a field is an aspect versus a status, you know. Typically, statuses system owned in spec is user intent. So it's not always true, especially when you start talking about multiple operators, interacting with each other, trying to share components share EP. Is you almost need, like an ownership model for subsections or fields of kubernetes api? It's so.

A

F

A develop rings.

A

But yes, huh I.

E

Think that brings to mind a particular pain point that we've seen with implementing stuff with CR DS and trying to persist across what we're calling a pivot. But anytime, you have to be able to survive a backup, a restore of a CRD based resource with status of resource, and that's you know. Sometimes your controller needs to be able to store information that still needs to be persisted across that backup/restore and the only place he can really do.

E

That today is in the spec and that gets into you know the the ownership model there, because anything in status needs to be completely recreated.

F

Yeah I think yeah. That's it's a problem.

E

But I noticed we're almost out of time and I wanted to bring up sadef cluster API proposal before we get before we completely run out of time. If that's okay, yeah totally so I know, there is some some potential overlap with cluster addons work with the with the proposal as it stands today, but we do also see some value independent of cluster add-ons to still be able to just basically take arbitrary, yamo and one-time apply it.

E

You know at the end of cluster creation, in some cases, there's going to be some things that aren't specifically add-ons or some things that you know don't have a full lifecycle management component to it and we still kind of want to see that experiment through, but I also want to see.

E

You know what we can do about getting some more formal integration with the add-ons project kind of specked out and how we can do that kind of in parallel with sedessa proposal and then over time we can look at how we do those things or merge them and and that sort of thing for a longer term.

E

But the fact that we have the experimental, API, group and flags now in cluster API means we can do a lot of this experimentation without having to worry about the core, closer API types and an experiment at a more rapid pace. With some of this stuff.

A

Someone closed the document that yeah so I think sounds like basically you're asking for some shared consensus on hey: let's, let's go through with this experiment as it stands, and then you're, declaring or you're just explicitly staying still interested in the long-term composition of addons work with post apply, yeah.

E

A

Yeah I'm, certainly not interested in in block anybody blocking anybody from making progress on, seeing how cluster API can create usable clusters and, what's the most appropriate way to or the the best way to communicate about how to put these things together. Like do you guys want to see like an enhancement proposal, do you want, like just comments on the post, apply stuff or a PR, so.

E

I think we have two different routes that we can take. We can either take you know a proposal based approach or we can take POC based approach and if we go the proposal based approach, I think we probably just need to get.

E

You know some time together between somebody from that's, that's familiar with the cluster API project with somebody who's. You know more familiar with the addons work and and see what we can do to kind of hash out that proposal or we can just go ahead and you know, go POC phase with it, which you know it I think it depends on whoever we get involved, both from the cluster API side, and then we add on side, I'm, fine, with either approach and I. Think you know.

E

Whichever way you know where it's best for the for the contributors, I think.

G

E

A

Mean I I'm, the only person who's contributed, actual code to the installed config thing, and it's really not that complicated, so I'd be happy to just write a POC that come that kind of fuses with sadef swert.

A

If that's all right with you sadef, I don't want to mutate your thing beyond recognition, so I.

G

Think yeah that works PLC. My is what I'd prefer to to see howhow. It combines together. Okay,.

A

Cool yeah I've, been wanting to play with the post, apply, configure I'm. Actually it's like the next next step on this multi cluster networking thing that I was working on I wanted to see what it would look like to use like post apply, config like a new one. You know just like try and upgrade a component. It's because I am my mental model of how that converges. When there's a cluster seems to think that there's some creative ways to use it so.

A

Yeah I know we're running out of time already we're over so but yeah. If you'd be fine to read a POC and you have the bandwidth to do so. I'm happy to submit something thanks for for bringing it up Jason and for both of you for joining.

E

A

I'll mark that action item is.

A

+3 for POC to cluster API post apply for add-on this integration, cool and then I still need to talk to Justin about cops.

A

Great does anyone else have anything that they need to bring up or a little over, so.

E

If you want I can meet up with you at some point later Lee and go through kind of the stuff that we're doing today around for DNS and queue proxy for lifecycle management. Will cluster API as well that.

A

Would be very helpful for me if you would so kindly take the time. Do you want to just send me an email or ping me on the slack yeah.

E

I'll reach out to you in slack and schedule some time. Ok,.

A

Thanks Jason cool: well thanks everyone for a wonderful discussion. Today, I thought we were gonna have a short meeting, but as usual everyone's just too dang smart, so I'll see you guys in in two weeks and even before, then thanks.

B

Bye bye, everyone.