Kubernetes WG Multi Tenancy, 14 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 20200714 Kubernetes Working Group for Multi-Tenancy

Description

- Special announcement from Kubernetes Steering Committee

- Discussing of Arktos, part 2

A

um Okay, hi everybody welcome to our regularly scheduled multi-tenancy working group. Today we have pretty packed agenda. So lucky is here from the steering committee to give us an update on some changes, I think to how they are overseeing the various six and working groups, and then we are going to have a curated conversation about CDs and multi-tenancy and then I think we have the team on from our last meeting to take any questions about their multi tenants, kubernetes solution so lucky. If you'd like to take it away. Wonderful.

B

Thank You Tasha, hello, everybody and working group multi-tenancy. My name is Lachlan Evensen and I, representing kubernetes tiering. Today. This conversation is mainly for the leads of this working group, but, however, everybody will be involved in this conversation. So, as you can see on the agenda, there was a mail sent out about a week ago, introducing the concept of annual reports and it's steering's goal that as kubernetes is getting larger and larger that we want to create scalable ways to have reporting process.

B

So we've introduced this concept of annual reports because previously all working groups and seeds were required to actually go and provide quarterly updates at the community meeting, so we're trying to figure out new ways to make this process more scalable and, as such, we've introduced this concept of annual reports. So this is a way for the community to see how working groups and things are going and steering also to have a finger on the pulse of where we can help assist these different teams and processes within the kubernetes community.

B

So I think the the main point I wanted to get across today is. There is a link there to the mailing list. If you take a look at that link, you can go down to the bottom and you'll actually see the process as defined of what we're trying to do. So all this to say is: there's a template that you need to go through and we're asking each working group to actually take a lead on getting that done within the next 30 to 60 days and we're just starting with working groups.

B

So I just wanted to initiate the conversation with this working group. The other thing is I'm, actually the liaison, as on behalf of steering to this working group, so I'm here to help you through the process. So I don't want to take too much time in your meeting, because you do have a packed agenda, but I just want to introduce that concept and who needs to be involved in that annual reports process. We're asking the leads of the working group, obviously, but we don't want to limit it to that.

B

Anybody as part of this working group can be involved in the creation of that annual report, and the idea is that we give that back out to the community. So they can see what this working group is doing, how things are functioning and there can be a mark in time for how things are progressing.

B

If I may I added one more agenda item and then I'll give you back the rest of time, we've also introducing we're going to send out a mail in an effort to make more take a more active role in creating an inclusive community. We're actually asking the leads to attend the Linux Foundation unconscious bias, training who are asking every lead in the kubernetes community, whether there are technical, leads or leads of things like working groups or SIG's to participate in that so I've added a link there.

B

If you have already taken it, please feel free to go. There I know it's Sanjeev and Tasha who lead this working group. Please go and attend that because we want to make sure that we're creating for them and educating folks on how to use an inclusive language to all their community members. So with that I'll take a moment to answer any questions. But I see this as an ongoing conversation and if you have anything, one need anything from steering, feel free to ping me and at me I'm at Lockheed, 83 and kubernetes slack.

B

But thank you very much for the time there any questions. If.

A

Anyone has any questions, definitely pipe up. I took the Linux BIOS training a couple a while ago, and it's actually pretty fun and at the time they gave you a free jacket for completing it, but I'm not sure if they're gonna scale that yeah.

B

Absolutely so I did it recently, as well as part of being on steering it's a fantastic. It takes about 30 minutes, it's completely online and digital, and even you know it's a course designed around, even if you're speaking at events like come on how to use inclusive language and even confront your own unconscious bias, oh I wouldn't limit it to anybody's welcome to take it. It's free we're just requiring that the leads of the communities must take it within 30 days.

B

If there are no further questions, I'll pass it back to you Tasha. Thank you for the time and thanks for having me yeah.

A

Thanks thanks for your come in cool, okay, yeah, and if anyone has any further questions, feel free to ping me or lucky, yeah,.

B

A

So our next agenda item was going to be talking about multi-tenancy and CR DS and let's see where.

A

Did we have our discussion leader on the call today I'm, not sure if I see him, because if not, we can jump to the next topic.

A

Okay, um so did anyone want to kick off that conversation because I think was Rodolfo, but I don't see Rodolfo on the call right now? So if you want, we can just talk about arc dos or I know a bunch of other people were chatting about multi-tenancy NCR DS on the mailing list. So if anyone wants to kind of jump in with some thoughts, we can do that as well. I.

C

Mean is Rodolfo available. I was like I can try to ping him in the sieve. If he's there yeah.

C

You might be on the other link, given that a couple of people have had some problems with it: okay,.

A

I can help him but I know.

C

What I was more.

D

Yeah, whatever thing.

C

A

Yeah, it doesn't look like he's online, but I pinged him. So if I hear back, uh you know we can always switch cool. So, let's see the only other topic that we had was. If people had questions about our presentation last week for about arc dos and I know that we, we didn't actually finish the arc dos, the entire arctos deck. So if people have any questions or the arctos team wants to finish their presentation, we've got time. If you, if that's something you'd like to do.

E

Yeah I think this is joining I. Think last time we finished up the attack pretty much out attack and but we didn't have enough time to discuss a nozzle details today. It sounds like we have some time. If you guys have any question regarding the mod 7c occurs. I will we have.

F

Yeah shouting I think that will be useful. If you don't mind just to sort of just refresh us, could you share the first couple? Could you share the first two.

E

Are the price from.

F

Your deck and then we yeah.

E

And someone given me the permission, I'll I'm disabled for sure yeah.

F

So Daniel is asking on the chat. Could anyone do overview of the proposed multi-tenancy architecture project so Daniel? There isn't like a single single multi-tenancy architecture. We basically we've got a few different projects that are sort of supported by this group, and you know the wikis have more details on these. So at any of there isn't it's it's not just one unified, multi-tenancy architecture, there's two or three different projects.

F

Or you wanted specifically you're, not dose okay, so I think that's what shouting is gonna say right now: yeah sorry.

E

I I sent a link to the tack in the in the mail list. So if you guys interested feel free to check out the stack it online.

E

What slides do you guys want to do.

F

Ten minutes as a quick refresher for those that might have been not on the last call.

E

Sure yeah I can kind of quickly walk through some key ideas in this. In this and.

A

Just as a note, the entire recorded presentation from last week is on YouTube too, so you can check out our youtube channel and watch the entire in-depth presentation for anyone who missed it.

E

Okay, thank you yeah, so I work quickly, walk through like here. The key pointing in the actor multi-tenancy basically are closely to see if it's a project open source project derived from from kubernetes and the way I did some car design enhancements.

E

One of them is the multi-tenancy.

E

Our multi-tenancy model is a virtual class model with a provides strong isolation among talents, but the difference than the previous of multi-turn. What you class seen in what coryza in our implementation? It's a long one control plane. We do not a deploy like separated, control, plane for each talent and for for the talents for each talent, it's pretty transparent. They are still using existing kubernetes api like under kubernetes. It was to interact to share the this one physical cluster and they can also manage their talent, uses the using the existing api's.

E

For example. If a talent admin wants to do some authorization teamwork, he can still use this cluster or customer binding, Aurora, binding or quota, or all these existing familiar ideas. For sorry all this existing familiar API objects, but all these API objects is the transparently limited at towards a telescope.

E

So if for our talent, admin creates a class, the rule that a classic rule only applies to that talent, it won't impact the other talents on this fiscal cast.

E

So the the the fort and keep design principle in our design refers to providers is serum isolation and this autonomy, which means talented me, can't manage its own talent without going to class timing every time and also the backward-compatible all the api, the back of the compatible. The last one is a manageability. We provide some across calendar like a CRT sharing mechanism mechanism to provide manage village. This is the overview.

E

The the most of the fundamental concepts is a yes is a talent space. We basically we introduce a talent space in a cache server, so each talent on that hill and space and the in that silence base. He can have all the existing resource hierarchy.

E

We also have a system telling the space and the unity system talent, space, there's a new object, type called a talent whenever you create a talent, object, the tenant controller kicks in and initialize a tenant space for that talent and the internet Helen space.

E

We initialized with this super talented, like a default namespace or cuba system, namespace or any other key for the objects, and this talent space will look like just like the who I pass over to that talent.

E

Since we added is a tender.

D

Just one quick question: so if you want one mapping between the tenant and the space I'm.

E

Sorry I didn't get the question: can your people yeah.

D

Is there a one-to-one mapping between the tenant and the space? Is there any easier, a one-to-one mapping yeah? You know each tenant corresponds to a single space. Yes,.

E

Well, this talent is telling the object it's a one-to-one making to the actual tenant space when you created this object, telling the controller watch late and they initialize the tenant space and creates all the default objects in the tenant space. When this object is deleted, this talent, space, we're all telling space will also be deleted.

D

Okay, thank you.

E

Since we introduced talent space, the internal UI, our object object that UIL has been changed. If we expose this UI out directly into customers, it will break other like water compatibility right. So we introduced a new feature called it's called shorter path. The idea is that we customer can still use the same relaxi URL 2002 access a resource, but we were transparently resolved it and change it to the internal full path. So here you can see. There are two talents. There are two user fun from two different talents.

E

They are using same URL to access object, button there, each of them cattle their own copy. It's it's kind of totally transparent, I, isolated.

E

And the further access control we extended the current authentication magazine mechanism. We still support other authenticators, it's just that for each Authenticator we add a new field called the talent in the in the corresponding fire order talking so after the authentication. After the same step, we have now four fire incoming requests. We have the pending information in addition to the original username and a group information for the authorization. We also support our existing authorizers, but actually there isn't a much change with each tester. Add several rules to enforce the talent.

E

Isolation for now system tend a system telling the user can access any space under a regular calendar user can only access its own space.

E

That's the that's the kind of the talent, isolation rule that are we added into the authorization, and after this step, it's all the original authorization for it, the authorized checks. The checks is like a beggar back configuration to do the original authorization check inside the space. So it's not a big change. It's just some extra chain extra check-in in front of the current authorization check. I.

G

Have a quick question: just John if you can go back one slide.

G

In terms of auditing, because some customer would require that we provide you know, yes, how do you have you seen any objection to this mapping internal.

E

G

E

There pro preserve all the capability for auditing, because before we convert the converter for pass, we have all the original information like where the records to come from. What's a user identity, a credential associated with the request, what's the original, because the UI are all the information is still there, it didn't diminish and.

G

Other things are still available, then. Yes,.

E

It's available, yes, okay, thank.

G

H

He's yawning Gemma had a you know, so, I think for a lot of these things, you're describing to insert, for example, like the tenant information and the metadata etc.

H

Do you have some information on how exactly like, in terms of the kubernetes modifications underneath I think the main question I have is: are these done as custom resources and extensions to kubernetes, or are these modifications in some of the core kubernetes components itself.

E

This is a very good question in the beginning of the project. We also thought about implementing the who our solution using some D capital approaches, like a say, a DS, but after some investigating, we find it's very difficult to provide a complete completed, a strong isolation to be the only the IDS. Let me give you some examples.

E

Let's say a way in order to toast about this backwater to my village, we have two changes in the point in the 20 resolution codes in that castle, leather heart is very difficult to implement as a CID, you basically another changes, the internal logic with the ID and also for for some convenience. So we want to have this talent.

E

Have this talent a field as a part of metadata, so it's kind of more consistent than for any time for you to recognize which object belong to which the tenant space right- and this is also very difficult to be achieved through the CID. Of course, we can say we use annotation something, but on this were kind of from design perspective. It looks like a not very adoption.

E

H

E

Honestly, with this solution, I think the wrong change. Either there are some entry or kernel code changes, I suspect that is heavy, a kind of difficult discussion. If we want for upstream.

H

Right and I think that's the so obviously what you've achieved is fantastic right. This gives like from a user experience perspective. This is everything you would expect from a multi-tenant system.

E

And it's also our kind of building multi-tenancy mechanism right. Ladies native woman, at you mechanism right.

H

So but the main question, of course, which I think which is interesting to discuss, is like how are you thinking of this is just a custom kubernetes offering or I is the plan to try and propose some of this to make it into upstream changes or what are your thoughts.

E

First, the Archos itself is not an a long project. We will continue our cause to add the new features and offer to some customers, for which it requires multi-talent capable operating system and in meanwhile, as I mentioned earlier, I, don't I'd be happy to work with them monitoring the working group to up screen before. If it's the community is willing to accept these kind of changes, but I guess this would be a kind of long-term effort right.

F

Have you have you presented this to Sagat, no.

E

Not yet because, because I walk in career he's a cross sake, cross, stick entry point right, so I'd like to start with money, tenancy working group.

C

E

C

It makes sense for us to talk about whether we should move on to the cigs. With this, the Russian I.

F

Think we're disagreeing would be in the past. Whenever the API machinery group came in, they seemed to indicate that if new project was initiated with an EPA machinery to do all of this, it would have to go through sort of API machinery, yeah yeah.

E

Yeah I guess so: yeah.

F

Have you had at least informal discussions with them outside like if, if they would be willing to collaborate.

E

Or I haven't studied yet because I will be the young prize a little project at the self and though I just started reaching out to you guys.

F

Now this is very cool stuff, it's exactly what multi-tenancy theoretically should be yeah.

E

I think if we I imagine if we design companies from beginning, we should have this right. It should be like a like this away. Yeah.

C

Yes, there any reason why you called the top-level division, tenants and not spaces, and the reason I ask this- is that that any different definitions of the word tenancy and you can have some tenants and and producers and consumers and assess context, and so the term tenant I think has often been caused. A lot of confusion.

E

C

If why call it tense about spaces, yeah.

E

I also briefly mentioned this in my last talk. It was totally written historical, ok, well, I I would change it to space to make it clear, because we wanted to clear differentiation between the between the subject user subject and the resource object. If we call it a penalty, it's computing, but at this time of historical couldn't, read it so. Yes, it.

C

Could be that could be improved, yeah.

E

If we will read our design, Docs USA in the updated, the design, doc use the space term, but in the code is they'll, call talent and.

C

The other question that I had is a month so I seen you you've deployed this internally. Yes,.

E

For hearing me, Alec, Eames and.

C

Sorry I apologize if I asked this two weeks ago, but my memories I'll write down the answer this time. But what is the? What are the main reasons that you see people wanting to use a space? Is it is it to create their own CR DS, and if so, why are they? Are they developing? You see ideas or they just installing operators, and then the other question I had is: if it's not seer DS.

C

What is it because I think that the the the main hurdle that we would have in trying to upstream this is making the kid there's a strong need that can't be met in some other way, o.

E

Space is actually a very generic mechanism- yarmulkes I it's not only for cid. Let's, let's say you have two teams in the company and this year a physical cost, but each team want to create their namespaces and objects. They don't want to worry about this potential, naming conflicts or other.

C

If I can just clarify I understand what the features are, basically anything that's currently cluster scoped is a space booked, and so it can be duplicated, so not just your DS, but also main spaces. We everything and less patrols everything so I understand what can be. What can now be divided by ten, but I'm asking is, in your experience, what have been the main drivers for the people wanting to use this feature. So now what can be done? What is done? Yes, I I.

E

See in first, our current deployment is a pretty limited. It's only sharing, it's only shared among several small teams. We haven't had any large-scale deployments in the real production, yet so, okay, based on this limited experience, my observation is the the things the driving force. Is this clean, viscosity and management?

E

They don't want to see anything any anything resource managed by other teams getting into the edge they have their view, so they want to have a kind of a only about the resource objects, the everything else deployed by their team. That's our current a used case and.

C

I tell you that that includes what that implies to me, because today you get that within the main space with that implies to me is that they are deploying either each team's deploying either multiple namespaces or multiple non-native. Spaced objects is that.

E

Correct yes, each atima creates their own namespace and multiple namespace. They don't want to be limited into long namespace, because each team have they are multiple different objects. Sorry, multiple projects, multiple users, hey.

I

Shawny can I add a few words here. Yes, actually agent agent, if you're looking to our design down, you can see they're actors, it's a project. We are working, try to make kubernetes to be available and big the public forum with very good scalability. So once we think about popping across public the public platform, then we would like to have a strong isolation between different tenants because different tenants. There are no chests this one scenario that we are considering so.

C

This is a so in a zero trust environment who is running the cluster? Is it a platform team within one company, or is this supposed to be like a kind of kubernetes as a service provider it can bother provider yeah.

I

It can be to two companies like right, so there is no chest between two companies and actually the one company should not know the existence of another companies using this platform. That's that's the zero chance scenario that we are considering for this strong Martin Tennessee.

C

Okay and so the that make sense, and then in that is so virtual clusters, so just to compare and contrast, the difference is that virtual clusters is going to use that's somewhat more resources. Is that correct to run the additional copies of the API server? Is that the main difference between the two, because it sounds like they're kind of addressing a similar problem space.

E

So you have you're comparing this to the to the other. Moreover, to class the project right is yeah.

C

E

That Faye and I think the Wolves difference is this resource consumption. There are differences with with this design. We are able to do some across calendar, sharing, correlation, etc.

C

Okay, so that's okay, so there is so one scenario is zero trust? Zero sharing? Yes, sorry: what was your name? The other person who spoke up me.

E

I'm funny I own.

C

uh Somebody else just.

I

C

So, okay, so that so in one scenario, it is basically the this is kubernetes as a service. The different tenants have absolutely no visibility into each other, and so that would compare closely to something like virtual clusters, but with lower resource utilization.

C

In the other scenario, you want to allow a some amount of sharing between the two, and so, in that case, it's more likely that this isn't really kubernetes as a service. This is probably kubernetes being run within a single organization, but the different teams just don't want to deal with each other unless and what and they wanted to be, an opt-in that they deal with each other, so they were multiple name faces. They want possibly. So it's not like mainly multiple namespaces was the key driver.

I

Not just martin namespace, actually, all the resources for a tenant, it's totally isolated from another one, so namespace, of course it's an important type like part c rd, as you mentioned just now, everything like even persist like a pv pv c. Then you just name it yeah.

E

Yeah, because there are also some long limbs with the resources.

E

Notes in the first, depending on different deployment stage in the for public cloud, are a kind of propriety collocation our nodes object as you're, creating the system space right.

C

So most people.

E

C

Permission to view what's on anode, because if they did, they would be able to see or sorry I guess that if you look what's on and know, you'll be filtered into your own space, so you'll only see pods from your own space. No, we.

E

Haven't done that apart, yet for now a regular talent, a regular talent, can only see the see the noodle name from like some pod objects properties, but he cannot actually carries a I've shown. No. The object, I see okay, but I mean some way to see some requests from a scenario that the ajik meeting scenario. Then they probably want to enable some attendant notes. That's something we haven't. We haven't even and we don't have detail yet, but we see some feedbacks from teams shouting.

F

I I'm sure you said this last time, but it forgotten can the individual spaces in dependency and ice or the self to share it. Yes,.

E

They're exactly how in dependency eyes, so basically, each tenant space can have their own sphere. I'm! Sorry see you miss the eye or ceoddi, see and.

F

E

See you in the hi okay, I've, seen hi seeing eye I see eyes. Now it's not a space stuff. It's uh it's a network network, so.

F

It'll be a single shared CNI provider.

E

It depends we support. We introduce a new network object, as I mentioned here. I didn't uh yeah I think we.

F

Actually, ni provider is a common one is because.

E

We introduced a new object network to encapsulate to abstract, abstract, isolated, a network seeing eyes associated with the type of this net object. Let's say if you create a network, excuse me, object, type metal object and the type is like the tablets of the PC. Then you need to give higher at RSVP see accordingly, if we, the network, hyping a neutron, dines effect by OpenStack Neutron, and you need to deploy a neutron seeing hyah cutting. So it's it's associated with within this network type. Instead I was a workspace workspace I.

E

Just sorry, instead of space, space are just like workloads to the network.

D

Each network that routing will.

E

Change as I mentioned, the public clouds may riser is a very important scenario we started with, so you are in our design. Those networks are totally isolated. You cannot cross network traffic unless you use a public public, a Yankee I think that's also. The form of current will be seen in public cloud. Unless you elaborate piece appearing, you cannot do intervie busy communication.

E

F

You mostly borrow this from this kind of architecture, from essentially from OpenStack multi-currency in projects no.

E

No yeah I don't see much similarity between this and OpenStack amore Tennessee, okay, I.

J

Have some questions two questions? The first is about sharing the I really like the design, but is there any concerns about sort of like a noisy, neighbor type problem? Yeah.

B

If there's just an example at one object, increment as.

J

I've been working with recently is a priority class object, and so it's cluster wipe and influences for the scheduling of pods, and you can imagine a scenario where one tenant is making a pod in his space with. Maybe they can extremely high priority class. That's above the priority of the you know, coupe system pods, and then, if the note that those pauses are running on, you know turns over the scheduling.

B

J

That tenants, pod to potentially come ahead of the system. Pods, is an example of a scenario. Yes,.

E

That's our kind of feature to plan the plan efficient, like this talent level, using performance, isolation and a narrator limiting tour to isolate the performance performance of our multiple multi talents to prevent any calendar to abusing from abusing this shared a control, plane and also for scheduler. We have a tenancy, fair, tenancy fairness, scheduling plan. By the way we don't have time to work out that part. Yet.

J

Okay, but there's no way for a tenant in his face to influence sort of the the master components in any way. Once.

E

The way implement of this appellant, a level patella, the performance, isolation.

J

Okay, cool thanks and the other question I have is like in this in this system. You're envisioning the tenant as a user. But what? If we? What? If you think about it as an operator, is the time and I want to installed ten operators so I have I want you operator, have its own tendency model?

J

Is that something that is entirely orthogonal things to project or I by.

E

Operator sorry means sies, it's the kind of humans are the CID operate, prefer the operations there are controllers, I'm, not sure. If this this slides answer your question, we we basically we supported each tenant to install their own CRT and operate independently, so it'll work, tenènte and society operator. You don't want to impact others, but we also support us here. The Saudis.

C

Right but general, what do you mean when, when he said like different operators, have different tenancy models? Do you mean like different instances of the same operator, but in smaller scopes, or do you mean operators that could somehow conflict with each other or just like completely orthogonal operators, yeah.

J

I mean like the first thing you said, for example, like some operators want to watch all namespaces. Some operators want to watch one namespace like conceptually this thing. This is a picture of users installing operators in their space, which makes sense- and you know, if you're working on a system that is installing operators for a user on behalf of the users. Then you think about the operators themselves, as potentially being.

J

E

Yeah in the left side that you can see, let's say three talents that each install different, the CID operator. This is the operator you can just rerun the existing operator binary and no could change. And when then, in your example, I said operator a wants to list our namespace operator P only list some other some partial namespace, but as they are totally independent, there's a see different namespace they're, not interacting.

C

Operators correct yes, oh.

C

Yeah yeah. How often do you see this case where a lot of different tenants with in the same cluster want to run different copies of this different versions, perhaps of the same operator? And what is the driver behind that? So.

J

One of the drivers is actually relevant to the cross plane presentation they had where people are running their. These operators are designed to do different things on cloud providers outside of the cluster and they kind of want to run multiple versions of the same operator for different teams like different I am accounts or whatever for each operator. That's that's a good classic scenario where it's like I want multiple copies at the same thing for that. For that reason, so.

C

That's that's one model, and why do they want different versions? Just like the teams all installed it independently and the central administrator hasn't bother to take over the administrating on their own schedule? Something like that. Well,.

J

I think it's like from what I understand like the way. A lot of these like operators like AWS and GCP ones, are like they.

J

They have I, think I think the permissions are scoped to one like one team or one user, so like if you're in a multi-time, C environment, everyone can share the same AWS operator. For example. Thank.

C

You Oh different versions there, but they're different instances. For example. Yes,.

J

Yes, there, for instance different versions. Well, you know people upgrade them, which is a different process, but that's also something that some people want to move faster or whatever than others other some.

B

Teams want to have.

J

The latest others don't.

C

Ok, so yeah you could you could if it were rewritten properly and if they all agreed to use the same version, they could all share the same CRD and then all run in their own main spaces, with their own service accounts with different permissions. Correct. But then upgrading.

J

C

J

Think that example is like this. The system's charity, like we have here like quite nicely where it is essentially the same operator like so.

E

That's how some not great author than.

J

To be reinvented,.

C

Yes, I would agree with that I guess so what I met my so that my thought about this proposal and I think I may have mentioned this last week? Is that there's a lot of complexity and it results in a very large change and, for example, like the attack surface of the API server, I think or sorry not there's kind of like an implicit promise that those spaces are fully isolated, a promise.

C

That's never really been there within the ever before, and so that's going to be challenging and it does affect the behavior of a very, very large number of components. So, there's going to be a lot of complexity if we want to upstream this, and so what I'm trying to understand- and this is behind all of my questions- is: what's the magnitude of the gains that we're gonna get as a result of this?

C

Are we if it's going to be well, some operators who would previously have had to think about how one C or D is shared between multiple instances. They no longer need to think about that. That might not be a that might not be very motivating. Given that, for example, the authors of every cluster-level component in kubernetes now needs to think about this concept, so you've shifted a lot of that work to a group of different people who might not be willing to accept it.

C

So that's really what I'm getting out behind all of these questions, I'm asking like? How would we, if we don't have spaces? How would we do all of these things today? So I can understand like what's the relative game that we're gonna get from introducing. That is these new features, I.

J

Just want to say that that's and that question makes a lot of sense, and one thing is just that you have operators that depend on out their operators and that's what that's what it probably gets more complex because of you know. A lot of operators want certain require certain API or C or DS, that other operators are providing, and so that's, where you get a lot of the complexities like you get a jumbled web of these basically and then upgrading that it becomes much more complicated.

C

What's the complication up in upgrading it because, in theory, when you update a CR ID, it's usually supposed to be backwards, compatible, I, guess in practice, do you see that that doesn't happen, or is it something different? Yes,.

J

I mean like, if you update this unity and you're not doing it safely, you can basically break the CRA cluster in theory. Yes, they're like they can be backwards compatible, but a lot of operated authors well like the best reading operators are about Chris but backwards compatible in theory, but in practice.

J

E

B

E

I want to quickly add something in for regarding the coaching: what was the driver for this kind of change? Right you are wishing. The CID is just a small part. Our vision is that we want to deliver multiple organizations to safely and easily share the same physical cost, and this can be. There can be different scenarios once an error is a cloud of provider can provide the virtual communities class the service to to uncross to the public cloud, the users that's for public class scenario.

E

The Nelson error is a private cloud within one company you have different teams and that they don't want to deploy different of physical kubernetes cluster and operate maintained. There's just one whose share a larger, critical crust first to to share the illogical uber result poor instead of aesthetically partitioning the resource and the second one was have just the wrong kind of one group to maintain to operate the infrastructure instead of each team, how to maintain and operate of their own infrastructure.

E

That's our kind of rubbish, see I did only small as a small part of this pseudo evolve.

C

Yeah, which gets into the conversation we were going to have with.

K

Ensuring this is me um yeah, so the number the first question is a you were mentioning that there were some cross cannon, the collaboration that can.

B

K

In this model, I mean.

G

K

Further explained just give us a case what is the case a tutor and once you collaborate one.

E

Of the cases that we already discussed these, we want to label this shared the CID right. So it's common to have some like a system-level CID results. For example, now some storage like a rook or some networking solution, they are also implemented lgrd and for all the payments on our class sim cast there they're, actually very likely they all needed CID. So the first first we use the case in says. We want to share CID among talents for some special systems, the IDS and the second that we are currently thinking about.

E

You, label optional, cross talent, access, let's say talent, a you created a steal across the rural binding in object, but in the class the rural binding you specify, some results are available to user from another talent. It's like a friend a class in C++.

E

So this is across optional. Cross talent, results, sharing.

K

Okay, so yeah yeah system, CRT, is kind of so in current design. It is visible by all tenants or yes, yes,.

E

It's kind of you first system class, the operators first that deploys its ard in system space and tested after its tested and no and working fine. It can apply a tag. Special annotation and designed well kind of be published. I mean pop I, publish I mean it's a virtual. In fact, it will appear. You call the tenant space immediately. Okay,.

K

So you still kind of you are kind of copy because you.

E

Know it's no no copy, because what a way to actually is when you query our tenant space, we combine the resource under this tenant space and also the SIS the resource in system. Space means that the specialization some kind of dynamic occur and merging. You know we don't copy the sir okay.

K

Got it? Okay? So that's good! So another question is so if you are saying that each 10 and can not okay I only see no name, but another object that may cause problems were scheduling right because I understand you are still using just one schedule: it's not pretend scalable right. Yes,.

E

Scheduler sharing here so.

K

Then you, if people wants to like, do the finit here like, for instance, if this one me GPU and Odin the nose with special label that has deep you enabled if the people doesn't see load object. How that? How would that work or yeah.

E

That's something we're helping thinking about it along.

E

Currently, we are willing forwarded to one solution: data, it's kind of the operator, the class the operator publishes it some away unknown or support. Ida know the selectors. You can use butter, regular user, you don't you, don't see the extra knows. It's like you visited a cluster class, the class of website, our documentation. You tell you what kind of labels like a GPU we just mentioned are supported on this cluster. This you can use in your pod pod another select, but you don't need to carry the action. Node objects.

K

E

The thought is that the infrastructure is maintainer buzzer operators and in especially probably class in era. It's probably won't be a wannabe, its folder to the in the user. I see.

K

So then, otherwise I think that may be a little bit difficult, which is the web hook so for web hook. There's no way you do tenon aware web hook in my opinion, or you guys really did that we.

E

Already have some investigation on that part, but unfortunately, I I'd only recall the details, because I I didn't have time to go through that part detail. But if we are interested can send email to me, I I will include upon who investigate that part. Yeah.

K

Because because we who is on the critical, is a hard creation, I guess over that first- that is the cost of object. Second release. I, don't see, it is very easy to say owning a pride of web hook selectively based on the tenants, but you.

E

Mean the sorry you mean the web hook of the also ready you.

K

And these allow the tarentines the webhooks say so say that is this loud. But if you really want to open it, that would be a little bit difficult in my opinion, because unless you security and change that culpa.

E

I think we investigate this and some other for also reducing and but I'm, not sure if I covered all you talked about I think that we can fall out of life. The council at the more I'm looking more into that apart.

F

Shining going back to governance, how do you see this going forward? Let's say that the upstream communities community decides to not accept this. Will you be moving autos along to new kubernetes versions independently or your your your own, a slightly older version of kubernetes? Currently, yes,.

E

Yes, yes, we we haven't decided when we can operate alleged machine to Casa. As you see, we made some a kind of entry changes. You know what tell sometimes first other wait, but octo ctcf definitely will be a kind of move forward less than a long project and the way regardless kubernetes community, except these are not in fact, are closely. The self has a much broader vision then, and the multi-talent itself.

E

We are working on different different areas, either it's kind of targeted for a larger scale, cloudiness conscious, supporting both Williams and the containers, and also to some intelligent scheduling among cloud and edge. So it's kind of pick up a larger part than the multi-tenancy.

H

So the multi-tenancy extensions, though, are those restricted to the api server, or are there the community's components you have to update as well? We.

E

Have to update some can we have, though, some update some other components like the scheduler accumulator controllers, but all the changes are actually very similar. It's a more like copy-paste changes, because you for all this civt at least watch. We just need to add a new parameter, the talent and we already changed the like 15 controllers. Basically, one person can change. I want two controllers, maybe three controller a week: it's very fast and the client go.

E

We need to make some changes, but we are based on the wrong I sing along 15 RC Wersching, so the clinical stir done. The supporters is explicit. The concept parameter with the new version of clinical. It already has I think it already. Has these contacts parameter. It would be much easier faster to change. The content it should be mature should be much reduced. If we have that I explicit the context permit.

K

So have you thought about you know? How did you do real upgrading because I see this really challenging if you want to operate a kind of standard kubernetes to this are closed cluster? Yes, yes, oh sorry, I mean who to change it.

K

Just I already have a commands cross right. Okay,.

E

K

Rocked those clusters well I, feel.

E

That is really challenging, because if we want yes, any kind of this live migration or upgrading, it is changing right here. The good thing is, our changes are transparent to the client side, so that I think I need to work. Give us some kind of advantage. Oh yeah.

K

I worry more about, like all the metadata like EDC D. So haha yes, yes, yeah yeah, that is my biggest worry I'll tree I.

F

Think you wouldn't upgrade an existing cluster to this. You would upgrade all the you would port all the apps over and they should work transparently. You there isn't a use case to take an existing companies cluster and transform it into this one, but you could have all the applications on an existing cluster pretty much transparently move over to this kind of a cluster.

F

K

That's really the more important thing that yeah, but that's the assumption that people you know would accept the option that recreate everything but I would say cases I see a lot of cases. People just say: don't change anything I pot, don't you and make my pot down you all things transparently. Otherwise, don't do it. We would yeah.

E

Favor we'd only require I mean if we suppose this live migration, we don't require customer to recreate an you see a lot.

K

Of you know, preparation I mean the simple see. If you ask every people recreate, then nothing needs to be done, but otherwise.

E

No, no, no, no, no faith! That's what I'm trying to a clarify! Let's say: if we do some line migration, all the stuff in the current ID city will be treated as a system space. We actually, we can have like an optional space keepers, so it's kind of compatible. You don't need to recreate anything after the migration, it's basically it's a cluster with the only one space that is telling the space. Now you can create a new talents.

I

Here, I, like Edison words here, cuz I'm, the one who make change the edct data here, so first I Payton, you don't need to worry like older, like the old formats of their etcd data. We still can comprehend it. If we see it still format. We know it's something belong to the system space and this linen you later like when, with the tenants information inserted in the EDC D key, we can also understand it so here what I would I say all the EDC D data it's backwards compatible. So so it's okay.

E

So basically, the the who are the YTD, the OTT City- will be just the system. Space yeah.

I

Yeah yeah cuz, like we can sync single system space as the as the code, clap kubernetes cluster and then the Mari tenancy, where I'm not a tenancy I, just had some news basis from different new tenants. So all the data like you, actually there is no minor migration. Just the met there and and many our actors knows it.

F

Yeah I mean eventually, you wouldn't want an app running in the system main space. But that's if you want to take a cluster without individually putting the apps, that's what you have to do, but eventually you want the apps to run yes.

E

G

Hi, this is on two again: I have a question on future directions: okay,.

E

G

You sure earlier not that one, the the arctos one you.

E

Means the other slice yeah.

G

You mentioned unified, container and VM orchestration I'm curious to hear, if possible, some of your thoughts on.

E

Some, oh sorry, I didn't get lost party. Some are what.

G

You're thinking in that area, oh, we.

E

Already implemented, it's a I probably can I know it's another related to that more tenancy topic, but I can quickly learn two minutes to explain its basically, we currently in cloud data, centers, more and more containers are running on bare metal, so we have to operate accurate, oxidation stack. Why is the kubernetes based for containers?

E

The other uses, the older like the VMware or the OpenStack for VMs, so our our reaching here they want to expand the kubernetes to have a single expression stack to handle both containers and VMs and in the way we did use different. Is that the current uh some approaches based I, don't based approaches? We.

E

What you extended, Zapata definition, you can see. This is original part of definition. You can have contained multiple containers. Now we have three aims and we, because we extended this most fundamental object apart and we can. We can reuse everything. On top of that, like a replica set deployment, jobs and a schedule, it's a one API. Why I wrong scheduling wrong special stack? We already we already implement part of the phone lines, but because VM lifecycle is a complicated I, it's not only started stopping a rebooting VM. You have like this attention.

E

Attention attach of, or names attached, Aniak dynamic, reconfiguration live migration. We are still working on this advanced events and the stated configuration well I can also share this slide. Slinky because interesting kind of being a broader sense for Otto's.

G

Great. Thank you.

E

Sorry accessible, we are also actively looking for external partners and for any user cases. So if we are interested in project feel free to contact me and we can work together. Thank you. You.

F

See, customers more in the service for a space you? What are your initial customers? I mean typical use case for your.

E

F

E

Targeting and we are targeting the user case or the scenario that is a cloud or provide essentially logic, large-scale cloud pride. Well, because we look at the current open source of projects, there isn't actually our open source project qualified for large-scale and Monte Monte, Monte tenancy and this unified vm container infrastructure for large-scale cloud providers.

E

So that's why we decided to work on this open so to have this open source project yeah very nice.

F

E

Think we are running out of time I. Maybe we can continue discussing in the next meeting or in the email.

F

Yeah I think so I think yeah.

A

I think that sounds good. You know. I would ask that if people have any questions, maybe hit up the mailing list, and you know if there's any reading material for arctos maybe send that over the mailing list. So we don't have to.

C

You give the same the same presentation.

A

Three times but yeah, thank you and you know I think that this is really raising some good questions from everyone. So it's it's very thought-provoking and thank you for sharing. Thank.

E

You for your time.