GitLab It's time to Go!, 9 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: It's time to Go! Episode 1

Description

Informal chat about the Go programming language.

A

Okay, oh hello, everybody! This is the very first it's time to go meeting and we're having some technical difficulties with the calendar. Sorry about that, but should be fine now, so uh we are about to just look at the kubernetes repository um because parallel had some questions, um and I think I think maybe I can answer some of them and we decided to record it.

A

So what's the question.

B

So the problem statement is, uh I would like to talk about comparing two unstructured types and, uh like the tricky part, is that some of the fields in uh are uh defaulted by kubernetes. So if you submit an object in kubernetes api and then you get to get it back, you may get more than you actually initially submitted into the api, and uh I was thinking about writing a function uh and up streaming.

B

Hopefully that would uh compare to unstructured type in a way that it takes uh the thing that we get from the api and the thing we compare it to and checks if the first one is a supper super set of the second one. So if all the fields in the second one are the same in the first one, a modulo or something additional that we didn't find in the second one and uh yeah, so we just wanted to talk about it and how this can be implemented.

A

I think I can answer that. Okay, so, first of all um the thing we touched upon before is that kubernetes repository is a monorepo. So if you are looking at kubernetes, uh damn machinery repository, which is.

A

Shared hobra on the whole screen. You can see the screen.

B

C

A

This one yeah, if you look, this, is the source of repair machinery repository, but it even tells us where it comes from. It comes from kubernetes kubernetes, staging sources, comprehensive machinery, so staging sources kubernetes and here's the source for all these repositories.

A

So you make changes here in this in the staging directory and then the bot publishes it here see it's both publishing these things and pulling commits.

A

Yep, that's how it works at the moment. There are some repositories that are not uh staged like gtos, for example, there's a youtube. Surprise there, maybe a few more but uh generally. This is the thing also seeks all some repositories.

C

A

Different organization.

A

Something like that or single.

A

Or active energy- six, that's yeah, so they all sound like that kind, for example, and so on. You builder customize, but things that I used usually leave here, but not customized. Actually, anyway, the back to the actual question the thing about um structure, I think it leaves here.

A

And this is the interface for it and then there's implementation.

A

So it is yeah, it's a bit tricky to work with this as a developer, because uh in this repository these directories are siblings in in the vendor folder and they need to to these repositories to.

C

These directories.

A

So you have the same file under two parts in your editor.

C

A

Then you make changes there, but then they also here but also the editor- is not smart enough because it thinks these are two separate types.

A

One is vendor and one is on the staging and when you search for okay, where is this used? This will search for the right thing, because we are looking at it via the vendor directory.

A

But if we find the same thing via guys, one structured see it can even open it in the in in a separate tab, but it is, it is the same file, so it its store. um It stores the object, the courageous object as unstructured just one. You know, and it's actually wrong. It should be in 64..

A

She it doesn't allow me to edit it by this page, but uh by this time, but by this tab. I can like edit it and then here, if I refresh wow, where does this come from.

A

So the question is how to compare an object and the object that has went through defaulting. uh I'm not aware of an api to do this, um so what kubernetes does is you submit an object? Then it goes through mutating, the hooks then by the dating of hooks and then defaulting, and then after that it is, I don't know, persisted or created you know, so it's it can be an update, the patch uh trade and anything else. uh Your api server supports any other verb.

A

But if the object has been persisted and then you have this source of without defaults, there's actually no api for that. Some people like argo cd in etop's engine, they use the schema object from kubernetes sources, and that is what kubernetes uses internally to apply defaults.

A

So if you go to.

A

Api, no, not here the g apis. This.

A

Maybe this no now defaults here, maybe.

B

A

Ups, something where we definitely have because defaults.

A

So this is a handwritten logic that applies defaults to an apps v1 deployment, and then this function.

A

Why is it not referenced anywhere interesting.

B

But this won't help us with tabs that have defaulting web, but, unlike with things that have defaulting web web hooks, so there's a crd and somebody defined also like defaulting web hook.

A

Yes, exactly this is not going to help I'm just telling you what what people try to use so they use. Schemas and, like you see, this schema, is getting some information by defaulting like uh some defaulting functions so for demon set. This is the thing that you'll be called and if you default some fields and blah blah blah.

C

A

If not said, then sets the sets it to this way and so on and so forth. So this is handwritten and it leaves incubators it's it doesn't leave anywhere else. It doesn't even api machinery, it doesn't even client, go defaulting, leaves on the server so github's engine from argo cd imports, kubernetes as a library and uses this schema.

A

This is incorrect, because, if you are talking to a different version of cluster, then defaults may be different. So your logic that assumes that this will be the thing you know you, I will send it and it will default the object like that. This logic is simply wrong.

A

It is correct most of the time, but not always it will be wrong, and what your program will do well depends on your problems, but most likely something incorrect.

A

um Yes and it doesn't take into account webhooks and it doesn't take into account cds and I have created an issue about that in the gitobs engine.

A

A long time ago and yeah there's they, I don't think they have an answer for that also. I have struggled struggled with this problem um like four years ago, five years ago, when I was building a controller that applies.

A

Objects based on an object that contains those objects.

A

It's like a bundle, and this thing would uh read that bundle from the object from the api from kubernetes api uh and would do a topological sort um in taking into account the references between objects and uh apply them one by one waiting for the objects to become ready. Look if it's a deployment and something depends on that. It would create the deployment wait for it to reach the red state and then would proceed with the graph.

A

B

Interesting, uh can we stop here for a moment.

C

B

Said you did a political sword on kubernetes objects? How did you determine dependencies between the objects in the first place.

A

Well, in this case, we I in this case I relied on references between entries, in that I called it bundle, and it was a list of objects and.

A

This controller would analyze those objects and look for references and.

A

It because of references, you know the order. Basically, you user defines the order, not any sort of automation, program, just analyzes what the user defines.

B

This is interesting because it's like maybe a topic for like a maybe another topic, it's at some point, because we also had like a similar issue uh like uh setting up and then cleaning up multiple objects using a controller and how to uh handle dependencies between objects. So you cannot just remove everything you can. You need to remove things in a certain order, and this kind of like strongly relates to what you said, but maybe let's talk about it some other time.

B

B

So can we go back to unstructured? uh Yes, so if we go to the object definition.

B

Or unstructured.

B

So what I was thinking like uh first one- I think this can be done recursively, because it's basically like a map string interface and we could.

A

Yes, sorry, what can be done? Comparison compare comparison so.

B

Without defaulting sure yeah, so uh if we just uh so, uh maybe I will start from the beginning. uh First, one is the interface we want to use because we can compare unstructured objects uh uh with one to one approach, so we expect them to be exact, equal, even the default fields, or we can check.

B

We can compare them in the superset approach and to keep things simple. Let's focus on the superset approach now, having said that, uh like unstructured object is just like a map interface, so it would basically so but here's a catch like this interface can be a slice or another map string. And how do we compare slices so how what's our strategy for slices in the first approach with one-to-one matching?

B

This is simple: we basically go through slice, maybe we sort the slices, but we go through the slice and we expect all the immense totally be equal, but in case of the in the second case in case of the superset, how do we approach slices?

B

So if there's there are more elements in the slice should we assume that objects are not equal or should we just look for this uh elements that are defined in the uh second object, the, and we expect them to be the uh to be present in the first one that we got from the api uh so that we check if the slice has items that are in the second objects, and we ignore the ones that are not in the uh second.

B

So this is like the question how this could work? uh It's like yeah, so that doesn't make sense.

A

It's it does make sense.

C

A

It's not a technical difficulty right. It's a semantical question. How should the program behave and it should behave differently depending on what it is comparing, so some fields uh have semantics of.

A

If you update it, if it's a slice or an array in json, if you updated it, you like, if you send the patch command, for example, for the field, then you replace the whole thing, but some fields have a different semantics. If you patch it, you add to to the set- and this is captured in annotations for the fields.

A

Let me find an example.

A

I don't remember how it's called.

B

So I think I know what she was getting that the this is used in merge patches, like in the in the patching.

A

Strategy, yeah fight strategy, that's the thing I was referring to yeah, so strategy merge and how to merge like the type field in this element. This field serves as a key. So if an element with this type is there already, then it is replaced. But if it's not there, then it is added.

A

It's like a map, basically from key to value, and this is the key and values the whole object, and then it merges them by, as you would merge, two maps, basically either replacing or adding to the to the mapping, and then there are other strategies. Yes, you can search for them.

A

A

B

A

Yes, so the thing that you are describing the function to compare it would.

B

A

Look at the how they call tags and the tags look, provide strategy tag and look for this, but I'm not sure if this is available for crds for customer resources. But maybe it is. I think if you define a schema using open api, you can define this.

C

A

B

So there's another question: do we need it? uh Do we need to do it on the first step, or can we just like a first iteration, or can we just assume that if the lists aren't equal like one to one like we compare lists, we don't even sort them and we just compare them and we expect elements to be in the same order as we submitted them to api api server, and if there are any defaults, they are just added to the elements to the in the list in the in the slice.

B

Sorry, but order is not changed and existing fields are not changed. Can we make this assumption.

A

And it depends on what you want to do, so it depends if code with such an assumption is good enough for your for the problem you are solving. If, if it is good enough, then you can make an assumption if this will behave not uh not in the way that you want, then you can't make this assumption.

A

B

What do you want.

C

A

B

A

You have this problem: what what they're trying to do.

B

uh So I'm trying to determine.

B

So, in the controller we get a reconciliation, we we start the reconcil reconcile and.

B

Of the object that is responsible for creating objects like deployments, stateful sets uh ports and sorry deployments, uh ingress objects, services, etc, and this uh during this recon- and we want to make sure in this controller that all the definitions that are uh in the api server are exactly what is defined uh uh in the controller, so sorry reflect what is defined in the controller. So if, for example, user changes, the deployment controller reverts these changes if the deployment is owned by the controller and from the other side.

B

If, for example, somebody reconfigures the crd of the controller to make the change in the deployment that the controller detects that the objects differ and applies. The change and the problem here is that we could theoretically just issue a patch and just just make sure just just reapply the object and just let the kubernetes api sort this out. But this would cause too many api uh calls, uh because, whatever reconciliation, we would have to reapply all the objects. It's very simple, but requires al. That causes a lot of traffic.

B

So the idea is to use like a controller runtime cache that will uh from which we get the current object. Definition define and compare it with the object that we uh create. We determined to be the one that should be currently in use, basing on some crd sense and some other conditions, and if they differ, we only then issue a actual patch, a request to api server and uh like yeah, so does it make sense? Is it clear.

A

It does make sense, it is absolutely clear. This is what this thing does, but I have a question: why don't you use the server side, apply api.

B

uh Like I was using, but it's so to my best knowledge server side apply, uh still causes an api and can be counted, for example, in gke against api limits. Is that correct.

A

uh Why are you calling it so many times.

B

uh So either so on, every reconciliation uh we.

B

So we get only the reconciliation event, but we do not know exactly which object changed or maybe, uh if the configuration that we are using to generate the objects changed. So we just reapply everything to make sure.

A

B

Brutal and simple approach so.

A

Yeah, then you need to make it a bit smarter, so some some ideas you can.

A

Look at the generation of the objects so for the object from which you get the input you can in the controller. You can remember what was the last one. What was the last generation that you have reconciled?

A

Okay, actually you don't need to remember. Ideally you put it in the status observed generation and that's the typical pattern with observed generation. Then. The second thing you could do is to get that reverting behavior is do the same thing with objects. You look at observing at the generation of the object and if you have seen this object already- and it is okay- so you just remember object blah with generation.

A

10 is all good. This is what I have applied and I was the last one who wrote it basically who touched it so I don't need to touch it again. Basically, and then you just scan the objects, you don't issue right, you should you first look at your cache check, the all the objects and, if someone is some of them are wrong. You you patch them using server side apply and then you don't have the problem with defaulting and you don't have the problem with too many api calls.

B

You know a couple of follow-up questions, so that would assume that we also store the object definition that we use to apply so that we use to create a given generation of the object in the metadata so that we can compare once we reconcile the object so that we can compare if the data.

B

So okay bear with me uh because there are a couple of things here.

B

Let's assume that the deployment that we reconcile and we go for all the objects that are this controller is responsible for now.

B

We notice that the generation of the object is the same, but we do not know if, like the definition, that of the uh like, the configuration of the controller then uh changed in a way that requires this deployment object to be adjusted, and if that's the case, we need a way to tell if we need to reapply the object using the newly bit configuration.

B

So there are like two causes for updating this deployment. One is that the configuration of the controller like uh changed and the other is that user change this deployment.

B

The second case is easy because, just like you said we compared the generation. The first one would require storing the meta, like the deployment uh that the in the object definition that we use to create this object and then compare it during the next reconciliation. If they are the same because we won't be able, we won't be, then comparing the defaulted object, but the one that we used to create it.

B

Do I get it right here.

A

Yes, so, first of all for anyone who's watching um this is the generation thing it's incremented by kubernetes. Every time you touch or not, you anything touches. The object and touches means mutates, not reads. The reading is fine, but for writing. This is incremented.

A

Not necessarily plus one it can. I think it can go up by any number.

A

Basically, um maybe not, I don't think, doesn't matter so and then typically controllers look at the generation and then they also store the observed generation in the status, and this is that integer uh the degeneration of the object it acted upon when when it saw it, so that's where you store the thing that you know uh the generation of the thing you act upon and together with the rest of the status, so you can say all good for generation, then, and then next time you can, you are reconciling you're looking at the generation in that observed, generation generation, maybe still 10 and then observed generation, maybe still 10.

A

And then you say: okay, everything is good. If the status is okay, then maybe generation will be 11 and observe generation will be 10, and then you you can say that hey. Actually, I haven't seen this object or I haven't reconciled it successfully before because observed generation is less than generation. So something must have uh updated the object, since I last acted upon it.

B

Okay, now uh going back because I have more questions. Sorry, I'm.

B

Apart, like servers, you mentioned that server side apply and it's like, I know how it works, but how does it help us here specifically to server side apply?

B

So if we are just sending a patch up like an object as a patch, uh it basically just lands in the server and it's if it exists, uh it gets updated. If it doesn't exist, it gets created, plus the ownership of the fields is set. Is there anything else we gained by server side apply here.

A

I don't remember: okay, I think there there are some benefits. uh Yes, I think there are definitely some benefits. That's why it was created.

C

A

I'm not sure about.

A

Maintain managed by fields this set of things.

A

Managed fields: yes, so who who is the owner and who manages the particular field? I think this is only updated by apply service apply and then also g14 is taken care of by by that um there may be other benefits like saying the whole thing as a patch. I think it's not it's not working perfectly or correctly, even in some age cases,.

B

I think there was an issue, but a very great issue by uh garden made by gardener. Why do don't we use server side apply? uh I can try to find it and send it to the channel after the meeting and yeah. So maybe somebody will be interested because they outlined the edge cases of server side apply pretty nice.

A

B

Cool now uh another question.

B

So controllers should uh like the kubernetes way of controllers, is that they shouldn't hold a state really that they should reconcile the state of the api server. uh So- uh and I was wondering like because when we, if we are, we have our own crd. This is this is easy uh because uh we can store, we could store the observed generation of the object uh in the status we create, could create a field of the status, but in the status field, but in case of deployment.

B

It's like uh this is not the type we own. We would need to like redefine it to have like this observed applied generation so that the last generation that our controller seen on this object and applied it uh in order for it to not to store the state itself.

C

B

uh Would you agree.

A

I mean I agree with the problem statement, but yeah we shouldn't redefine anything of course, so I had this problem the same problem within in smith, and I think, if I remember correctly and that's what I would do today, I would have a a pair object field in the status of my object of my custom resource.

A

That's where I would record the generation of the thing I touched together with the observed generation of that original source object, so just at the end like make it, maybe not at the end like make a status uh patch, uh not patch.

A

You are probably using a sub resource for stairs, so send the status there and uh uh record everything you have uh touched so far and store whatever you want there. All your step state can go there. Also. I don't think it would be a problem if you store these things in in memory of your controller, and why?

A

Because a bit like uh not a problem, because it's just an optimization, if you lose this, nothing wrong, you know happens just more api calls, that's the kind of problem that we are solving here, uh but nothing will go wrong, but the downside is that um if you have multiple copies of your controller, uh they will do double work or triple work or you know same work, but uh so so again this is more api course again. This is just nothing will go wrong.

A

It is just more useless work or duplicate work uh so to overcome this uh well, either store it in the status or store it uh in memory and do leader election, but also maybe do leader election in any case, to avoid this problem with double work.

B

So uh definitely a leader election, and the problem is that we would have to uh like.

B

We do not really know how many objects and what kind of type will be uh under this controller supervision. So I think that storing the state internally will be the simplest approach and so that, basically like storing the style installing the latest generation applied so uh uh so yeah. I agree, and this is like a good idea.

B

So to summarize, instead of like playing with comparing a structure to type, you suggest that we go a different path, a simpler path that we store the generation of the object in uh that we last seen applied after applying in the controller and also in the annotations of the controller. We store the latest applied version of the objects so that we can compare them directly.

A

Annotations or status in the states.

B

But we cannot, we cannot do it for deployments or kubernetes types and we are. We need to supervise like the basic types too.

A

Yeah yeah, I was suggesting to store the generation of the deployment in the status of your source customer resource.

B

uh No, no, because this is uh we will be, we won't be able to determine up front which or how many objects, what kind of type of objects we will will be under.

A

Control supervision make a list, make a list, and the key would be the name and namespace, and the value would be your generation that you have observed. Something like that.

B

How big can status be.

B

Are there any limits.

A

I think the tcg value limit is definitely applicable and I think it's one megabyte or something like that or two megabytes, maybe but yeah. You will not reach this limit. I think I mean unless you're doing I don't know a thousand.

B

A

Or something like a.

B

Modern thousand.

A

B

So uh I think I'm processing this. uh So please bear with me so uh so you're suggesting uh using my own words, so you're suggesting to put the generation of the object and the latest applied.

B

Then json of the object.

A

Status, not the full json of the object, just that generation.

B

A

No, don't need to put the full object, so this is the thing.

A

So bundle is that definition of the bundle of objects and then there's the status, and then there is observed generation of that.

C

A

And then statuses of the resources and for each one you have the name and then conditions you.

C

A

A

B

So this is understood.

A

Generation 64 something like that, so you only need the generation for it. uh You don't need the full gamma of that object.

B

So this is understood- and I was just wondering so uh we want to avoid reapplying everything every time the configuration of the controller changes, because, let's assume that the controller manages 100 objects and we change the configuration only for one of the objects, so, instead of reapplying 100 of them, we just want to reapply only the configuration for this one given project object, and this requires us to compare the json that we used before with the json. We want to use now this way or another.

B

Would you agree with that?.

A

Okay, so I see there as as we discussed, there are two problems: one is user, changing things and the other one is configuration being changed and then you don't know what what's going on? I think uh applying 100 object is okay, because configuration changes like once a day or something like that.

A

So it's not a problem.

B

I I see, and we can do it- we could also do it iteratively, so we could go with the generation for now and if the configuration changes offered to occur occur too often, we can optimize further with storing the jsons and just trying to solve. The second problem is that correct.

A

Actually, you can just store hash in that status.

A

C

B

That's good, yes, that would help yes, yes, yes, dump of me that I haven't thought about it, yeah, but it's like. Actually we could even do it. We could do both, so we could.

B

So if we just store the hash, we can basically store both information observed generation and the hash of their latest applied configuration in the status of the master crd, and then this way we make the controller stateless and just have that when we just just go without uh any comparison of objects altogether, we just have a very simple comparison and now so now the last final question.

B

Sorry uh I I uh can the is there any tooling in kubernetes that allows uh that produces, like, like marshall and jason's, always produces the same uh order, for example, for slices or for keys, because I'm afraid that, like like you know, the order of key of keys in the map is undefined, and this can like uh screw the hashing.

A

uh Json standard standard libraries, json package, sorts, the keys for maps.

B

And you think that uh yeah, so if yeah so in the slides order of slices should be also preserved. Okay, so that solves the problem. Thank you very much. That was like uh super useful and uh I wanted to like share like I do not know if you, this is pretty late for you. So if you like, uh we can finish now and just talk about this next week.

B

So I wanted to like do a very short presentation of the golang quirk uh I found in the json I we talked last week, but maybe let's do it next week. Let's have something for the next week and.

A

Yeah, I think we have chatted for quite a while and let's do it next week, but before we go, I wanted to mention one more thing, because it is related.

A

So you will have a problem of removal or pruning of resources that have been deleted from your configuration right. How do you detect what has been removed and remove it from the cluster.

B

So I think this is solvable because I can first of all I can start with.

B

I can start with uh determining the source of truth configuration basing on the input to the controller on the user configuration, and this will give me the list of objects, so a group version kind or and name namespace or just name.

B

Now I can fetch the status and.

B

See if any object changed- and I can also like do a third step- that I can use the cache of the controller and issue the get and see if they get returns not found or not. So it's like even do you do you see where I'm getting here.

A

uh Yes, but that's that's, not gonna work, so you.

C

A

First of all, regarding the status there is this document.

A

A

Second and then moved it somewhere.

A

I found the fork of that object of that.

A

But yeah there is this api convention that we need somewhere.

A

A

Anyway, I can add a proper link to our notes, so.

A

Maybe it should.

C

A

So basically, the idea is that your program shouldn't rely on what is in the status.

A

Status is for optimization, so yeah.

C

A

Of your controller shouldn't yeah, I can't find it here, but it is somewhere so behavioral controller shouldn't correct. This shouldn't depend on status because status is just an output. It's it shouldn't be used as an input for your group, understood.

B

And this is interesting.

A

The second thing is, you cannot look at your informers at your cash because if you have just started them, you know.

A

The cash will be empty and you well, the cash will reflect what is in the cluster, so you cannot use the cache to give what is in the cluster and what you know about. Basically, so I think I suggest you to have a look at this project called clitus, which is what the club agent uses to do. Githubs, and this is what kpt uses for kpt live apply.

A

It's it's developed most by googlers uh and it stores the inventory of the name, kind and namespace and so on in in the configmap. So then you have the information persisted and then you can see what is the new state? What is the state in the cluster and what is the inventory and then you can give them and see what should be deleted.

B

uh I'm sorry I got lost here. So uh uh let's start uh please bear with me. So, let's start with informer, so you're saying that if the controller just started and informer is empty, if I'm using the controller, runtime client to fetch a client fan and operation object, uh it will go through informer, but if and if the object is not in the informer, it will return me 404, even though the in the api server. This object is indeed present or will it block until it fetches the information.

B

A

It will not block it will not find the object if the informant hasn't isn't synchronized. So typically, when you start your program, your controller, you wait for informative, synchronize, so to fetch all the objects from the cluster.

B

A

But isn't isn't it.

C

A

Automatic, yes, it is automatic, but uh it's not instant, it's not instantaneous. So if your controller is really quick, it will it can um try to read stuff from the informer before informer has synchronized with the cluster. Informer can be out of sync, it's a cache and also it doesn't guarantee that it is up to date, so your controller can be running. Then there may be network disconnect.

A

You can read from your informer still, but uh it's not up to date anymore, and even if there isn't no network problems, you it's still not up to date, even like, even if you are reading from an api one nanosecond after you have received the reply, you're already reading, it's already stale.

A

So if you make you're basing your decisions on a potentially updated information, that's because there are no transactions, no locks. It's a distributed system.

B

So a couple of things here, uh first of all, first of all, we just use informer as an optimization.

B

So if the conformer says that there is no object, even though there is uh the worst thing that will happen is just past issuing another kpi call. Second, one is that this is basically the kubernetes way, because we are reconciling state and we are having eventual consistency and not just like strict consistency. So this is like fine.

B

If we just do a couple of iterations uh third thing is that controller, hopefully won't be discarded too often, so we won't have this information very often, and fourth thing is that we do have going back to what you suggested with this project uh will have a look at it, and uh I I need to educate myself here uh my and uh I just want to use my own words so that you could check my own thinking in case when we know what the correct configuration should be and we have like a status that we can uh that we know that reflects what we was applied.

B

The last time we can prepare the list of objects.

B

We can determine if objects changed or not, and the only problem that stays is that the problem, if the object was deleted or not so we did. We part we segment like we. We shared the problems into like two three ten into two different buckets, detecting to change and detecting the removal, detecting removal we discussed- and this is uh clear, detecting or uh detecting of the removal.

B

uh Here we have this eventual consistency. So even if we informer is out of date, we can reapply and we can- uh and the worst thing that happens is that we will issue more api calls until informer catches up.

B

But this is not a bad thing and this yeah, so does it make sense? Did I miss anything here? It's like what do you think about it.

C

A

Well, what what I was saying about uh these problems with informers, they're, not blockers, they're, just things to keep in mind like that's, so it seems you wherever I understand this stuff, so uh cool. No, you yeah! You don't have this problem, then.

A

C

A

Not sure I I'm following the what you're describing with the approach, so I I just suggest you to have a look at this project and it will it it already. Basically does what you what you want, so your job would be to just determine when, when you need to rerun the applier from from this project, yeah and.

C

A

Can have your custom.

A

Inventory implement you can implement your own custom inventory and don't use a config map, use your own object, for example, or a separate object or whatever so, but but yeah it implements this graph, colonization application applying and so on.

B

uh I will and it's pretty late and I already got uh tons of information from you and um and that helps a lot and I would definitely educate myself with the project. So that would be all for me for today. Yeah.

C

B

A

Well, I hope that was useful, I'm more than happy to help, uh because I I went through this myself. I know how it was difficult and it's. It became easier now with the server side apply, but it wasn't available when I was trying to do this now. It is and also this library, on top of it, so just use the library but anyway uh happy to answer any questions of the chat next time.

B

And separately exactly, and I think we can uh stop recording now.

A

Okay, everybody bye.