Kubernetes SIG Cluster Lifecycle, 23 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cluster Lifecycle - Cluster API Provider Nested office hours 2021-02-23

Description

#sig-cluster-lifecycle
#capn
#capi

A

Good morning, everybody uh this is february 23rd. uh This is the cluster api provider nested office hours um talk about how the progress that we're doing on uh building out the cluster api implementation um for nesting control planes in in kubernetes clusters. um We have a pretty well, we have an empty agenda, so we might be able to end this unless somebody has things they want to bring up and talk about.

A

I don't have anything that I've added so.

B

um Yeah, I'm just curious, so it's uh uh I think, chelsea uh lcd, that controller is, is reviewed and checking. Why.

A

Yeah, I think, there's a couple changes on there. um I think there was feedback that I checked in on uh this morning and it looks like there's still some unchanged or un un uh finished working.

A

Yeah we were chatting a little bit uh last week about the the common spec uh channel changes. I think the big changes that are still in there were we still had some of the pki generation. In that controller, we can merge that I guess, and we can just pull it out in the next pr that change it, that that moves that over to the ncp, if we want to go forward.

B

Repeat that comments back stuff, so I didn't.

A

Yeah, so if you go check out the the cluster api, uh nested slack channel and then just scroll up, I mean a couple messages from thursday. We were talking about whether we wanted to move forward with the current setup or go fully into the q builder declarative patterns and use the version and channel fields that are defined in the common.

A

A

That makes sense. I can drop a link to what I'm talking about.

B

I thought we we probably will use the version right. The channel part is pretty flexible in my opinion, so it's kind of you can use that point to any ui how you want.

A

Yeah, um I think the the the question here was: do we want to basically go down the path of using all of the built-in reconcilers for from the qbuilder declarative pattern? So if you haven't looked at that the inner workings of how that project is set up it, basically it it writes the the reconcile function for you, so you don't even have to you, don't even have to implement it. You just supply it with that common spec and it goes and looks up what manifest is supposed to grab.

A

So in the current etsy dpr, we have everything hard coded as as go types to create the stateful set and so and all the components. Well, sorry it it decodes from yam from a yaml file, all in that controller, um and so if we went the full keybuilder declarative patterns route, it will uh it'll automatically create a lot of those resources for you, based on what's supplied in the channels uh url in essence or channels, pointer.

B

I suppose that the url will point to yammer file.

A

uh Yeah yeah, so that that points to a a yaml file or a custom. Well, it's all yaml, but it can be customized driven or just uh raw kubernetes manifests.

A

So I think the plan was to move forward with this, and then we can make change. We can iterate or we can. uh We could change the implementation as we go forward with the rest of the controllers, um but I I want to make sure um yeah I can ping chow or if you want a ping chow, uh we can check in on on those last changes and see. If that's that's been done, um the other one, actually, that that is good to call out uh if you can get.

A

If you have a chance or if anybody has a chance, can you just go through that other pr on uh the ncp, doc and nc and and the nested controller and nested control plane? Dock is kind of.

B

I give some comments before, but I think you address it and uh it looks it looks okay to me. I think I can. I can just add a comment and say: look. Look at me. I want somebody else other than me sounds good. I think you can just that's mortgage.

B

um Maybe let's just take this chance to how about just go through all the opening issues, and you see.

B

Because we still need to track some progress on the opening issues, especially on the developer in science, for.

A

Sure yeah, I think we were kind of waiting on the on the ncp dock so that we could start the the nested api server um here. I can share my screen and just show this.

B

Yeah, my point is something maybe can be done in parallel. So is that something that doesn't have to wait from the ncp.

B

You can just assume me, I mean the interface part, you can just assume something will happen and do the rest of thing.

A

Yeah, no, that makes yeah. I mean it's yeah. That makes perfect sense. um Okay! Well, that's also a good call out so judong uh vince koscheck. If any of you have a chance in, can go and give this a read through the nested control, plane and nested uh cluster proposal. That'd be fantastic, just get some extra feedback in there um and if it's good just give it a give it a thumbs up.

C

Yeah I already go through that. I don't see something uh we have discussed. I don't see anything wrong. I will give up.

B

Come up yeah for that comments. Dude make sure I mean people more than okay other than me have reviewed it. Okay,.

D

Yeah I'll write something okay, sure thanks I'll check later in the day, thanks.

A

Okay, cool and then yeah, so we still have a whip tag on this thing. On the the nested cd controller.

A

Cool uh moving over so the cap end goals and the high level cap and goals and the high level user stories.

A

Those are basically taken care of by the the the um the ncp and nc doc that I have for the proposal, so those will get closed out and that's technically also what the same the one above it is, which is the base architecture, those are just to gather people's um people's requirements in separate spaces for user stories and and the the goals uh so that those bottom three, uh four, five and six should get taken care of by that pr.

A

um There's an open port there's an open issue about updating the design resources so vince you had. You had mentioned that we had uh some fancy stuff within uh nest or within cappy to basically generate some of those. um I moved over and used that to generate the ma, the diagrams for the nc and ncp dock, so at least the make files in there now so that can get done at any time. I don't think it's a it's a pressing issue. um That's just version controlling those one thing, chris.

B

So um if, if your sap design is merged, so basically we have. We have made agreement that the pki stuff will be done in the central city controller, so not individual components that have taken care of their own pks. So the the the charles you know, component design needs to be uh needs to be changed because that doc says that each component maintains its own.

B

I think you better create a new issue, update.

B

Yep, it's so confusing.

E

A

A

A

Plan, what was that pr number? It's 29.

A

Yeah, I think, there's another call out that you would that you had commented in here um just to call this out oops, so you had added in that. We basically should we update this, since it still mentions the ca being created by each component yeah. So.

A

A

A

A

A

Cool all right, going back to.

A

That it sounds like we basically can yeah, so we've got 18 that that's going and updating the designed uh the the actual uh diagrams to use plant uml. Then we have the base scaffolding, which I believe that might be I mean we might have. No, that didn't actually get done because we don't have all of them in there. That's right, okay, cool all right, then we have create controller. So that's what what chad's been working on. um I started this work and then skipped off to go.

A

Do the ncp work um so now we have the so once we get the ncp work, we can get that the well. We can actually start that in parallel and we can basically create the controller for that, so I'm to actually make another. That is, create the controller for the nested control, nested cluster and nested control. Plane uh controllers I'll make two separate issues.

B

We'd, better design them so that you know people can know exactly what they are going to work.

B

So can he help more if, if there are left some issues opened you know assigned? I said you know you need to talk with chris.

B

A

B

A

Pretty much anything in here that that uh can be picked, it can be picked up as long as it's got. It's been documented in the design. Docs would be what I would say. So we so what we don't have there's. I think, there's three issues in here that we need to make, uh because we need to make one for the controller manager. We need to make one for the control plane and we need to make one for the cluster controller. So.

B

A

Are all separate, independent components, yeah.

B

I think I think I actually call him pick up one of them. I mean easy. I think if you guys pick up the api server, he can pick up the controller manager. It's pretty similar yeah.

A

um I'm just going to link out to this one and then I'll go I'll fill out the information on these. So I don't have to type it on here, uh but I'll do that all outside of this call not to waste all your.

E

A

Okay and then we have one other one, which is the nested controller.

A

B

Yeah well then, this one too tall, so he knows.

E

B

How about others? So, let's make sure the open issues has a designers.

A

Sounds good uh real, quick just going back through the just going back up to the top because we have or to the middle uh the admiral their admiralty yeah. This thing, um the feedback that we were getting from from k fox, um I think you you had you had.

B

Responded, I I I need to refresh my mind, so let me see what's the best problem.

A

Yeah, so the idea so so k fox has brought up if we could bring this into into um into cap n um to basically use use that multi-cluster controller at all or leverage any of the bits that they've done for scheduling for multi-cluster scheduling, similar to what you're implementing in the um I'm going to assign this to you and then I'll. Let you handle handle this because I think, because you're doing something similar already in in vc, so.

F

Just some more requests like we usually tend to ask, um either to integrate so like the other way around non-cncf projects and not use them.

A

F

A good call, um because there's a term of ownership that like it, has to be owned by cncf um to make sure that, like we don't make decisions for users that, like they wouldn't want.

F

I don't know if virtual cubelet is incidents yet, probably not.

A

It it is a cncf project.

E

A

A

I'm gonna yeah: how do you wanna, how should we handle this then, or what would your? What would your.

C

Suggestion be vince yeah, I'm really interesting about the virtual couplet, so I'll take a look as well and if I need help uh uh favor try that and I will take a look at this item.

C

F

Okay, I guess it's in sandbox still, so I'm like we'll definitely wait for incubating. First, usually um and yeah I mean like, if that's useful, for for you all like you, I guess you could start like looking at it.

A

No, I'm still going to send it. Okay, yeah cool yeah, I'm going to sign it to you faye. If you want to handle it. If you want to just triage that it sounds like we probably are just going to push this off then uh and just be like we're working on other things for right now and as as we get to virtual or as we get to uh v cubelet into incubating, we can. We can evaluate how it fits into this world.

B

Well, yeah, my intuition, that once you say what you call it you did it.

C

What'd you say I thought.

B

Once we were talking about the virtual cooperative, we will eat it, we will win because I mean all specs.

C

But the virtual equipment already had broad support from different clusters, so we should leverage on this community effort instead of creating totally new ones right.

B

Now what I'm saying that for the situation, any situation that the virtual company was trying to tackle in in our vc model, we should support it.

A

Oh okay, okay, true yeah, because it just should register itself as a cubelet, so it shouldn't be. It shouldn't, be anything special for us. It should just be them. Integrating in in, like vince was saying cool all right, uh so the other ones. We have an sdk nested api server. I'm just gonna sign this to myself.

A

Since I was working on it.

A

Let's see, we've got update the con nested component um yeah.

B

B

A

Yeah, I'm just waiting for that to for that to get merged uh before before I go and make some changes and then have to recycle through that um cool. So we have got the cluster and the controller, the yeah, the nested cluster and the nested control plane types um and we have the controller manager.

A

Let's lock it closed.

A

All right does anybody want to pick up uh control, plane or cluster types and.

A

Controllers, um if you've got cycles, uh if you don't that's fine.

C

uh I can take a look first and let you know after that: okay, all right, which one yeah we uh create cloud controller and that's controlled, ls. Okay, I will take a look first, see how much work and how can I evolve more on that.

A

Well, I'll add more into the issue after after this meeting. uh Just so, you don't have to see me type.

E

And then, if you're, if you're comfortable.

A

With it, then we can assign it to you it's mostly just following the design doc that I wrote that's in that's here: okay,.

A

Cool, um do we have more issues that we want to get filed in here? I know I've been doing I I know fay you've been doing a bunch of stuff in the in the vc land still and I'm I'm doing a couple more things, and I think wei is also doing a couple things in there. So we've let this sit for a hot. Second, um I'm not sure if there's other issues that we should that we should file as of right now um or if we can just move forward with these yeah.

A

I think we can move forward.

B

So I mean it's pretty straightforward in the end, I would so yeah if you really want to sell something. Maybe we have a milestone? Oh, when should we expect we have a working demo of all together.

A

uh Yeah, that's a good question.

A

I don't know, um and I'm not sure if uh yeah I'm not sure what we should be working back towards um or from.

B

No, no, I'm not saying that you know you can tell vc the milestone. I'm saying that this ncp right right.

A

Yeah you're trying to you're trying to figure out when we can set a milestone for when we should have something that we can actually play with and demo here from from the cap and land yeah.

B

And then I mean uh I suggest that we we discuss this and make some milestone. I mean it's just planning sure. Otherwise uh it is kind of hard to track the progress, and that makes sense.

B

If you need those, I mean we need to also think of how to move the vc stuff to this repo. So everything that cannot be done in parallel. I I think we have to die in sequential, because it's about the repo organization. How do we agree so that, because we may have two different crd, even groups mixed together, I mean maybe somehow in the later it has to be temporary, but that has to be done until you know we have some concrete ncp ready already.

A

For sure now that makes sense um real, quick vince uh from from the cappy side of things and managing milestones, and, and all of that um is there any. Is there any common practices that I should be following or we should be following things like the actual milestone that you're tracking against? Can it be something like just v001, or should it be like just mvp or is there any patterns that we should be following or is it loose the mvp? We did zero one.

F

Zero just because if it's one alpha one, that's what I would target um and then usually we create two milestones: one for the zero release and one for the x release. So everything in zero releases, release blocking and the dot x release is like gonna, be in a patch version later, but it it it doesn't. It cannot be breaking changes.

A

Gotcha, so it gives us a chance to basically have have these aren't these aren't in scope right now. We can work on this and and we can still have a yeah, something that we're tracking against and something we're tracking against next, in essence, um and then we have a next milestone.

F

Which is kind of like a catch-all for everything? That's like not like uh like now and like you need a proposal or something like that, um and then every like cycle. We kind of like reconciliate, like all the issues open to see like if we want to put everything in a new.

A

In a new release, okay, so we're gonna have a next one as well. So three, okay, uh I didn't set any dates, as you can probably already tell um on these.

A

So do I wanna work back towards uh or we want to work towards some specific milestone. uh Is there anything that you're trying to you're trying to hit time-wise, fey uh or.

B

Yeah yeah I mean I don't have concrete, I mean come and practice as soon as possible.

A

Sure, no, no, of course, of course um I mean we're looking at so we're like february 23rd. Do we want to try and get these controllers done and an mvp in place by the end of march end of q1 of of this year? um Do we think that's reasonable? That's reasonable, at least to me.

B

Okay, you can see work end of march.

B

Like I mean I can make it a little great aggressively.

B

I mean make this repo. You know runnable for for vc end of april, something like that.

A

So work, so uh we would get these controllers out done by the end of march and then uh end of april we'd set like the next milestone, which is which is integrating uh vc into it. That's what you're saying okay, I like that.

F

Is is that a 1.0 uh for the end of april, though.

A

I wouldn't consider it an uh a 1.0. uh I think we would still be which that would just be integrated. Well correct me if I'm wrong faye, but I would assume that would just be integrating what we have in vc, so that you can leverage these nested control planes with vc, but still not be a full dot like a full 1.0 release. I think we'd still need some cycles on that.

B

um uh From one point: what's the car here criteria for 1.0, so I mean I I I actually have never hit any project that I was working on to reaching one point or we manipulate just zero points. I mean one two three is kind of. uh I don't know. I have some other criteria on other projects, one 1.0, but I don't know for capping. What does 1.0 mean? It has.

F

Because everything has 50.

F

for now, um but yeah like so the the zero one. Zero release, usually like it's expected to be something. That's that works um like it would be the first release and um if you need multiple steps during like the next couple of months like you could also um release like beta tags, so pre-release versions uh in github release.

F

B

I think I think yeah there are a couple of things right so so, since this repo is about the campaign, so um if you look at the theme it has to be I I I as long as ikea works, I mean you reach a milestone. um This report doesn't necessarily because of the naming issues I mean, although it has combined with vc, but that this repo, I don't think it's 100, representative vc, uh although we may represent this repo as we see so, um I can see that you know for we.

B

If we bring vc stuff in, you can just name it.

B

0.1.234, that's, okay, it has to be 0.2, um it's my opinion. So if you think you know, once we bring vc stuff, you make it as 0.2. um That's also fine, but I'm assuming every you know. Bumping of the major release number like 0.2 means you have some uh innovation or innovation on the kpn stuff. This is my understanding.

A

Yeah, I would agree.

E

A

Since adding vc into this, isn't really changing any of the actual isn't changing any of the the yeah.

D

A

End function and functionality, then we can probably just do it as a dot one, and we can always. I mean, although I guess what what you're getting at vince is the. If we do, if we do a dot one release, what do we like? I guess it.

A

What depends there is how we define what is a runnable state if you can deploy a control plane within a cluster, even if it doesn't work using all the vc bits that still, that still is that still is a 1.0 release of the cap, end controllers being able to deploy, and then we drop the vc stuff.

A

On top of that so yeah, I think that I think that makes sense and we could just do like a 1.1 release that adds the extra functionality and if we really, if we decide at that point, then we want to pivot over to do a dot two release. Then we could do that instead,.

B

Cool yeah, so yeah, that's my understanding because of naming so the dot two has to be something new for the kpi architecture, either architectural functionality. um But all the you know, arguments you know additional stuff, even for the vc stuff. We can add this small minor version number. If you track to.

F

For progress, other infrastructure providers like we did a release once you could create a kubernetes cluster, so that's kind of like the the baseline.

A

Yeah, that's perfect, then that's that's! That would be what what fits into our 1.0 it's just you won't be able to you won't be able to. uh You won't, be able to schedule any pods from it in the nested cluster. Just yet, but that'll come quick that yeah cool all right, um so I'm not going to set a milestone. Well, actually so I'll set this 1.x release uh as that april. Actually, no I'm going to leave that timeline. Blank for right now and we can just put stuff into this um I'll.

A

Add this as integrating vc with cap n and then so we'll. We won't set a timeline because we'll kind of wait until we get the that uh 1.0 in place and then we'll move uh we'll set a timeline once we have that fully um closed.

B

Out yeah, but I think we should have set the timeline for 1.0 somehow right.

A

Yeah I've got it uh due march, 31st.

A

Cool, so that gives us a little over a month to get a couple more controllers written and get this thing ironed out. Yeah I mean I.

B

Mean setting a goal is much better because people sound pressure, absolutely absolutely.

A

I mean, if you never what's, there's the uh one of those programmer, um not virtues, but but uh rules that are basically like there's just it's going to be a never-ending work. If we don't have an actual timeline or a release that we're working towards so all right um do we have any other housekeeping items that we want to do we want to handle while we're here, um I guess we can go through and throw some of these into the into those milestones.

A

A

So we basically.

A

Have I believe that works right.

A

Did I do that wrong? Vince, potentially milestone.

F

uh You need to install the mouse and plug in.

E

Oh darn, okay cool! Well, we can do that. Well, here I'll, add an issue.

B

Okay, that's okay! I think every issue.

B

A

All right so then, everything is basically representing that for right now and then we'll get the and uh we can get the milestone, plug-in added, and then we can actually use those commands cool. That sounds good. I like this, so we have an actual plan of plan to move forward on a little more concrete.

A

Anybody have anything else that they want to bring up uh agenda still yeah.

A

Otherwise we can get back 30-ish minutes.

D

You want to discuss the sinker stuff uh if you want, or.

A

Sure uh do you want to do like the open pr's I mean, I mean the pr that you are working on right yeah. We can talk a little bit more about that.

A

Yeah your most your most recent feedback was basically the flow that I'm missing um so yeah. There's. uh Let me let me give you a background background. I want, I think you.

A

I think you fully understand what I'm what I'm going to be implementing, but in essence I have another pull request that I haven't finished, uh which is adding in a web hook that basically just handles, creates and creates uh go through and and following this same exact flow um just goes and adds the adoptable annotation when it creates the resource and expects that the that, when the syncer picks it up, I mean I don't walk through the code, but when the syncer picks it up that it goes and checks to see. If that annotation exists.

A

um Now the flow that you're talking about that isn't covered, we create the the service. We change the service spec vservice back then v service is created. What.

B

Am I saying that they do so web hook when you create the server, I mean you have time difference yep and if the petrol kicking the p service is going to be deleted. So that's the whole concept.

A

So I believe maybe I didn't actually.

C

Oh, you said the vcsync will delete the p service because of some kind of great condition, correct.

B

Yeah your current code, unless you change.

A

The code this so this actually stops that from happening.

A

So I check to see if it's an adoptable service, so we go through here. Let me actually open up this, this full file. So I I did add that after you added that feedback, uh there is still there is still a workflow in here. That is. That is bad. uh In my opinion, which is the well there's, there's two parts that are that are potentially conflicting.

A

So once we get the once, we get the list of services for them from the patroller we go through and we check to see if it should be deleted and if it's not an adoptable service, we added, we add the should delete. But if it's an adoptable service we we don't add that now the one place where that breaks down is if the admission life cycle in the tenant control, plane say the tenant went and added in uh something that validated to see.

A

I don't know that they set a key and it failed that that check, but it it succeeded, the admissions or the mutating.

A

We would have created the service in the super cluster admissions at the tenant control plan will fail, meaning we still have the orphaned service in the super cluster, but it looks like it's adoptable, and so it just sits. There that's a case that I definitely haven't handled yet, um and I'm still trying to figure out exactly how how to handle properly yeah.

B

So there are quite some you know if you think about this more so the corner cases are quite, there are some common cases eventually right so and- and I I I think there is- you still need to either change the code or whatever, but again so if somebody delete the p service, so what we are going to in current quality, I think we have created another piece service which is completely wrong.

B

So it's okay, I mean, but you have to stop this behavior, I'm okay, you come up with saying, uh just don't create it, leave it as and let people to figure out how to resolve it I mean, or uh if p service is deleted, you need to delete the result resurface because you need to notify the tenant.

B

um Both are fine, but you cannot leave the current implementation, which is which is yeah with another p service, which is apparently wrong.

A

Yeah, that makes sense. I wonder if there's so, the the area there that I'm still trying to navigate on how I would how I would implement is if I go back and I delete the v service.

A

What what downstream could that affect if a controller was was looking for those um because what? Basically, in essence, what what could happen in their in their tenant control plan that they wouldn't expect? If that were to be the case, um and I've been trying to figure out if there's a way that we could do it otherwise, where, if the p service is created or is deleted in the super cluster, we still have some allocation of that.

A

But I don't think that's possible, um specifically the the cluster ip, because that's the only thing that we're trying to deal with there is like making sure that the cluster ip stays the same, um and it's almost like.

A

If it was already, it would just get reallocated to something else, or it could potentially get reallocated, because this like what I was thinking there is, if we, if there was a flow where, if it's an adoptable service and the p service is gone, we go and actually try and recreate with the cluster ip that was originally assigned to it. But that could have been reallocated that that's impossible.

B

You have to do the reverse, reverse order, so the if the p services is gone, which means that class ip is gone. You shouldn't make that so.

A

Yeah because there's no.

B

Pool so I mean yeah you, people can argue that this might not happen in practice, but I'd like to know so. No better things will eventually happen.

A

Sure, no I'm sure it will. I'm sure it will like just an accidental delete on on a service and in the in the super cluster would just tank that that cluster from actually operating prop properly. So maybe what what should be done, then, is we should so? Are you thinking that I could implement in the um a deletion check within the super cluster check to see if that virtual cluster exists or the virtual cluster service exists?.

B

A

I mean I could, but that would be that's just more more um more work for the for the super cluster to then try to triage back and forth.

B

I mean you can add a finalizer if you want.

A

Oh that's an interesting idea. I hadn't even thought about that yeah, okay, interesting, so we could basically have an uh have if it's been. If it's been created by the tenant control plane. uh Through this adopting phase, I mean you.

B

Should write something else, not the not in incorporate this in the sinker, but you can should have some other other more. A new controller. Just kind of this kind of p service sure make sure that one take care of how to once you remove the finalizer.

A

Okay, that's an interesting idea, yeah that seems like that would actually solve that problem, because then we did it. We in essence wouldn't be able to create it, wouldn't be able to delete that from the super cluster. The only thing that we'd, still we wouldn't be handling is like. If the entire control the super control plane went out and was like, was unrecoverable from ncd yeah.

E

And at that point.

A

We'd recreating the state of the cluster, no matter what so it doesn't matter, I guess so you just need to think about. You know if you add.

B

A finite error, it's kind of safe for you just think about how to you know, do the cc w collection once the intel vc is gone. You need to remove the this can be done separately in another controller. Just watch the vccr kind of thing.

A

The one the only problem there is, if that doesn't catch, uh if they actually delete it in the vc. So there's there's the phase there, where I kind of do have to handle that in the sinker. um Potentially, I can still put this under the same feature flag then, because, if you delete like if I wait and wait on a controller to check to see, I guess I can have an outside controller. That's listening for the services in the in the tenant control plane as well, because.

B

Like if you deleted.

A

It from the tenant control plane and it had a finalizer, and I didn't remove the finalizer, because I still had the vc around it would still the the super.

D

B

A

Wouldn't get deleted another problem that.

B

I was, I was mentioning the pr which is, um I think, uh let me speak so the the the it is possible that people delete the these service and the creative service with the same name immediately. So the wrapper, who can clear the old one.

A

Yeah, but I had uh yeah that's a good point. I had originally basically set it up to do. Finder create in essence, so it'd go and if it found one that was already existing it could it could leverage that and just take it over and re-adopt it. But there's probably some issues there from a security standpoint where, if you adopted a service that was associated with other well, no, it would just get re-reconciled at the end of the day.

C

uh Chris, uh do you think it's possible that you get rid of current vc thinker service part and create a total new one? With your web hook, combined with the super cluster service creation, we got the mix with the current vc syncer service synchronization.

C

We always have kind of a loop uh either you create that one, uh the vcsync or recreate or redelete this kind of thing. So why not just to get rid of the current vc thinker service, synchronization method? Totally, then everything goes through your uh super classic plus web book for the service.

C

This will be very clean for you and for all the service say. Do you think that possible.

B

That's solved some for the problem, but that doesn't solve the problem. I was imagining so yeah.

A

We still have to handle a lot of cases there. We we have to re-implement a lot of what vc's already done to do that.

B

I mean I mean you still have the problem right. If p services is deleted, although thinker won't create a new one. So what's the reaction? In your case I mean even you have an individual thinker.

C

Yeah, then you delete the virtual cluster service and because the the p is deleted, then you have to read the v. So then, then the user has to recreate what new one through the webhook, because as of today, every piece deleted the the virtual have to follow.

A

What, if what if it's, not a user-driven action.

C

A

C

If you have a root privilege and can control the super cluster user has to bear with that. This is a by design. You cannot uh compete with admin in the supercluster.

C

That's correct: you cannot never fight the I mean in the super class and if super class has the privilege to delete anything, the virtual has to follow. You cannot force it or not force it. This is the this. Is our.

B

C

Day, one correct.

B

But model I mean the source of truth is v. It's kind of p cannot delete it.

C

Yeah, but by design you cannot but someone in the privilege mode if they won't clean up the super cluster. We have to bear with this. uh No consequence. I.

B

Think it will create recreate everything, because synchro is also running at the root.

B

Okay thinker is also rude, so you are saying the problem is two rules are competing each other? It's not. You know root competing with talent, because the thinker, if you think itself, is the root, it has a full community to operate the the supercluster.

B

So if you delete basically like we are in the same privilege, you delete my things. I need a creator again.

C

Okay, so this is uh this- is the kind of.

B

This is different. It's not saying that I mean we. We create a hole for the work for the tenant to use the virtual resource, the super resource. No, not that way. It's like it's like proxy is, you know the thinker is proxy everybody and now everybody as a root and because it's a very strict way because we control we carefully control everything in a synchro to make sure there's no abuse.

A

Yeah, so if we so so, your your flow, your suggestion that you had uh where basically pretty much everything that's implemented here, stays the main addition that we would add, is in the checks on a deletion at a supercluster.

A

We go and delete the I mean in essence delete and recreate the service against the control against the virtual clusters, control plane from what the object was currently in there. Just basically remove the cluster ip and let it re-allocate itself because then it just goes through the admission life cycle again, so sinker gets a request. Sorry uh request comes in to the tenant control plan it for a new service. It goes and creates in the super cluster life goes on.

A

Somebody goes into the super cluster or something happens in the super cluster and that that service gets deleted. The sinker notices, the deletion on the super cluster side of things deletes uh grabs the object from from the tenant control plane. The v service deletes the v service and then recreates it with that same exact object without a cluster ip assigned to it and without resource versions and all of the other, the metadata. In essence, it would go through the admissions lifecycle and create in the p service.

A

Again it's just a weird hop there, but it would go and reallocate a new cluster ip and it would be routable at that point.

B

A

A little bit weird.

B

But yeah really limitation, but functionality wise may be okay, but.

C

Yeah, this is why I, I was a little bit uh have another option with uh chris that I use nat to net the virtual cluster service using the super cluster ip. So I just do a net on the port side so that we can translate locally to the virtual cluster service to the real ip, which is a allocated by the super cluster.

B

That's that's fine. I mean if you can change anything in the kobe proxy side. That's you have tons of ways to resolve this problem.

C

No, no, not groups, I'm I'm. I was using a sidecar like like east iowa, but so just do a little net.

B

The whole network stack in whatever ways you want that car could be proxy, and this problem is gone.

C

Yeah, so you you guys using this approach right in your internal and you couldn't practice it are you using poop, frosty, okay,.

A

Yeah, you guys have some special sauce special sauce there. That makes this a little bit easier to work with than than I think what we're working with you can manipulate narrow stack. So it's pretty easy.

B

But I mean I'm assuming solution, like don't change that part and make it work.

A

In essence, yeah part part of that is one of the reasons.

C

Why yeah the crew proceed? Stuff has a little limitation because not every user use a group process. Some user is style. Some user use, a different network stack for the kubernetes, so just changing cupracy is uh only one solution, so we would like to have a more generic approach than that.

B

Looks looks better than cookie proxy. In my opinion, yes, or even you need a container if you, if you really want to do that.

B

But I don't have a strong opinion if you, if you have a good prototyping, that's that's nice. I don't have strong objection for that, but uh people will always have their opinion of having another sidecar containers.

C

Yeah, this is what we did yeah a little bit, but.

B

uh But that sound car has two I mean celica may be reasonable, better yeah, but.

C

Because this would only change the kernel of the local park.

B

I mean you, you probably have to put it in. You need contamination.

C

Yeah there are some in your container. I think yeah.

B

You have timing issue, we do hit that, so you have to make sure the rules inject before the real container starts.

C

Yeah, this is the still yeah. We still have a timing, a little bit time issue. I think they're no guarantee because there is a using the synchronization to monitor the service creation, all these things, and it cannot be real time well,.

B

That's the reason you probably have to put in you need a container, so you need to kind of need to do something. You need to wait. I'm here, you know the whole entire rule is up is correctly, but it's a bit tricky because chris, you know we do that because we we leverage, cutter containers. So then you can do that so way. If you want to do that, which means you have a root privilege, you you, your port, your container can change the host networks iv table so yeah. It may not be true.

B

You I mean multi-tenant environments. This doesn't sounds right.

C

But I I kind of don't need a fully privileged. I don't need a three kind of privilege level, because you need to change the kernel that net net net table so that you have to have a three limit, the privilege, but you cannot do it with a normal user privilege to do that for sure.

B

It depends, but I I I I can I I can see some limitations there, because um I think I think people are even talk about get rid of the part. You know entire privilege about operating the narrow stacks.

B

um If you joined the multi-tenancy, I mean working group, they were talking about the the the reducing the um capability, the system, the the policy for the capability. I think you to get rid of a lot of probability of operating on the networking stack if you open and that's okay, but uh um I mean I mean it's very specific in scenario specific, so um I agree you know um chris's intention to try to resolve it in a kind of don't change that that part that's hard to make yeah.

C

Yeah, I also agree on that, so this is why we just explore a little bit other directions not go further. Yeah.

B

But I think you can try. I mean if you don't have that limitation on changing the whole narrow ip table. You can try with each container so.

C

You can yeah, I already did that already did that it works, but uh we didn't just don't want to go that direction, because we need a privilege, change and and also chris seems, a more kind of a clean version. So.

B

Yeah, I I think yeah, uh I think, for the first first version just make sure everything works and kind of resolve these corner cases just try to change as much as possible. um At least people can give you a try to see what's going on. This is a hard problem. I mean we, we can point out a short card, but with some additional condition to resolve this problem. But if you don't have that short card, I mean for sure.

A

Yeah, there's some definitely there's some weird corner cases here for sure to resolve. um I I I super appreciate your help like navigating these things and and calling them out that, because I I had missed that virtual cluster one and I'm surprised that it hasn't bit us yet, at least in my testing uh that it's mostly just like that timing issue.

A

um Okay, so I can get I I'll get the I'll get the other change in here and feature flag it as well, just like everything else, that's in here, um so that it doesn't affect anything that you're doing and then um cool. That makes sense. uh How's the scheduler stuff going just out of curiosity to check in, and we only have seven minutes left, yeah.

B

Yeah, I think we can. uh We can have a working demo this week, actually.

C

Awesome, so is this your code in the experimental uh in a motor tenancy code? Oh this part, okay, um awesome.

B

Yeah, it's kind of we I put a lot of efforts on, I mean most.

D

90 percent of my.

B

Work was you know, maintaining the cash, the steady scheduling part is pretty simple, but at least you can for the simple use I mean very common user. I can. It can cover 80 of use case. In my opinion, yes, um let's create a namespace given, and the quota will be spread to multiple supercluster and the thinker. Each supervisor will have a thinker just sync. The particular parts have the writing.

B

Last name annotation.

A

B

That's fantastic.

A

Yeah I mean if you have a working demo, uh I would love to check that out next week or if you, if you're willing to to to present on that.

C

Interesting to see it yeah.

B

Yeah yeah, I mean walking demo in the sense that I I would. uh I think that odin probably is one or two changes left. So I assume we have a looking at working levels this week I I mean, and next we definitely we should have a working demo, but I I I prepared zero document, so some document first uh basically yeah.

C

You want to see some of your explanations about the code and uh how, with your basic ideas,.

B

Yeah yeah, so the most challenging products, not nobody, addresses the marty. Crossover is the scheduling part this scheduler, I mean compared with kubernetes schedule this schedule algorithm wise, is much much much simpler, but it is much much difficult on maintaining the cash sure.

C

It means the the cash of each cluster.

B

Schedule cache I need to catch the castle state. I need to catch the name space state. I need to catch the ammo. Okay, because think about this kubernetes need to only watch one one api server. I need you watch, you know m plus and api servers.

B

Any one of them can be done and I need to make sure my scheduler is is not broken due to any of the cluster is done, so everything part is very simple. The tough part is the cashback okay.

A

Cool yeah I'm excited to see that uh whenever you're ready just make sure you add it to the agenda, so we can actually have it like scheduled out. um You may not even take I mean, take multiple slots and my.

B

Stores may not enough because there are a lot of stuff okay um but anyway, so yeah. I have some progress on that uh hope. We hope I can show you give me some feedback.

A

Very cool awesome all right folks, anybody have anything left they want to bring up in the last four minutes.

A

All right sounds good. Well, we will see you all next week and have a great rest of the week. Bye.

C