Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kcp: Towards 1,000,000 Clusters, Name^WWorkspaced CRDs - Stefan Schimanski, Red Hat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kcp: Towards 1,000,000 Clusters, Name^WWorkspaced CRDs - Stefan Schimanski, Red Hat

Speakers: Stefan Schimanski
In 2014, namespaces were added to Kubernetes. Many tried to implement multi-tenancy on-top, with limited success. What if namespaces are just the wrong tool, and we better invest into cluster-like isolation called workspaces, built deeply into the apiserver. The kcp project explors Kubernetes - with logical cluster support to implement workspaces - with ability to scale horizontally via sharding, towards 1,000,000 clusters - with novel API service models disrupting CRDs. In contrast to other projects like vcluster or OpenClusterManager, kcp challenges years old decisions in Kubernetes by going deep into API-Machinery and apiserver. Strategically, we reduce the size of clusters to those of namespaces, and by that open up the space between workspaces for innovation, while within a workspace kcp is just Kubernetes. Outline: 1. from namespaces to workspaces 2. APIExport and APIBindings, identity based security 3. scaling up kcp to 1,000,000 workspaces.

A

All right so I I hope I can be hurt. Yes, so when I come everybody to I think the last talk today in this track. So I'm happy that so many people are here interested in a project. A couple of people um work on for I know one and a half years or something kcp.

A

um It's complicated to understand. There are many views. You will see many views on this thing, which is called kcp and everybody will find their own definition, their own use of that and which way to inspire you to think about how you can use it and how it's different to the tools we have had in the past, especially complementing kubernetes, as we know it so about myself.

A

I'm Peter, schumanski I'm, looking at redhead I, have been working long long time years on CID, so I was implemented of many of the foundational features and cids and cids are everywhere in kcp, so everything we we see here, kcp cids are behind that.

A

It all started with an experiment or with a yeah, with a prototype, 2014 kubernetes added namespaces to kubernetes. Before that everything was basically cluster scoped. Nowadays, everybody takes namespaces for granted.

A

There was discussion around whether they needed like whether this namespaces are necessary and, yes, the community concluded. Yes, we want them and they were added and the isolation we get from those places is pretty weak. Everybody knows that, like you, cannot run multi-tenant workloads with high explanation via namespaces in namespaces. Our namespace is because they separate names. It was never meant for anything more than that, so the experiment we did was what, if we did a different kind of partitioning of kubernetes, what if namespace is provided more isolation than just names?

A

It's the most prominent example. Everybody in our cids are not namespaced right. So if we want namespace crds, we get something like a type space, bad name, maybe but not namespaces. So the experiment is: let's do more isolation than just names.

A

If we do that, we get something we call it workspace and the workspace is a type space in this sense that crds are independent, but basically everything else like Discovery, open API. All that is independent, even namespaces are independent. If you have two workspaces, you will have two default namespaces on both sides, so the result of that is basically a workspace is like a cluster. The user thinks it's a cluster but expect by the same kubernetes API server.

A

If you want, we put a Target, so we built that, like in one API server partitioning into workspaces, so you can run it and you can have hundreds of workspaces. Every workspace is basically as cheap as the namespace, so as cheap and as quick. You can create new workspaces in milliseconds and they don't cost because there's the same control claim behind it.

A

We put another goal, and this is one million on one hcd, one API server. You cannot have a million name spaces. Everybody knows that. So we put another goal here, which is far beyond what you can have in one API server just to drive our exploration.

A

Obviously we need something like many API servers horizontally scaling, so that was a goal's idea as a prototype and the goal of course was clusters become uninteresting. Like you, you, your cluster, is your workspace. You don't think about that. The other. There are other people on this cluster if you are talking about namespaces, usually you're, not admin in the cluster, so you think about. There are other people on this cluster.

A

But if you have a workspace where you can install crds and everything else and your admin, you forget that there is actually a cluster or an API server behind all right. So if we do that- and we did that those questions come I mean if you talk to people about kcp, is this a better kubernetes? Will we replace kubernetes with that? And now we have kcp and it's so much better?

A

Can I insert operators in a namespace? Finally, and if KSP is its own project, the question comes: can we build that into Cube itself like? Can we do something like namespace V2 and forget them forget about the questions for today? Those are basically not the questions we are asking. We try to um step back, so we have seen touring machines in one way or another in the studies of computer science or mathematics, so their compute models. This is a physical touring machine, it's a machine which can compute it's very theoretical, but it can compute everything.

A

Basically, you can compute in any programming language and there's this this band, this thing with the characters and it computes. It operates like a small CPU and in computer science. You ask yourself: what can this thing compute? Can it compute the square function? Can you compute exponential functions and it turns out the thing can compute everything, and then you ask yourself for what is it really suitable, which things you can compute on this thing in a elegant way, and let's get into this mood of thinking, we built a machine.

A

You know one machine which is kubernetes API server, you can build controllers. Kcp is built on on kubernetes, which means you can build controllers, but we will in extend this thing. We don't stop at the workspace. We will add dimensions in a second. We built something. We call it for today the kcp machine, and we ask ourselves: how can we use that we're not building the better kubernetes?

A

If you look into the into the website of kubernetes, it's saying basically, it's a production grade container orchestrator kcp as as a core project is nothing has nothing to do with containers, nothing to do with orchestration of containers, so it uses a technology of kubernetes, but we try to generalize it so forget about containers for today for the next hour. It's not about containers for this talk. Basically, what you get are those workspaces, here's one, it's pretty empty, there's nothing inside there are cids.

A

There are a couple of things like conflict maps, Secrets airbag resource quota, all those generic things, but nothing about workloads, just not there. So it's a genetic API server and it has cids.

A

All right and oops, when we do that, we want to add three dimensions. So we'll talk about three extensions now, so we start with this empty API server and we extend it the First, Dimension I add we have seen it already. It's a the red arrow here, the red Dimension One Million. We want many of those things. Every single workspace is like a small kubernetes cluster, so you can write controllers against it. If you have LCD behind so you have all the resources within semantics.

A

You know from kubernetes, so basically it's it's kubernetes at its core, but it doesn't have the workload apis.

A

The second dimension is: if I have a million workspaces, what are Services between them? How can controller operate on one million workspaces, so we were thinking how can I program a controller which is basically multi-workspace aware?

A

How can I do something like I get objects? Everybody knows list and watch in kubernetes, so you have informers at the programming model of kubernetes. What is an Informer against many workspaces and how does an operator do something with many workspaces?

A

That's the second dimension, and if you do that, if you think about that and find a solution and I will show how it looks like later on, you can build multi-tenant services, nothing. You can really do in kubernetes. So these not across a million or something like that.

A

A third dimension is, we add, basically locality over the whole planet. So you have workspaces across the planet. Basically, in every region, for example of the planet, every workspace in itself again is like kubernetes has a guarantees of hcd for consistency across workspaces. You don't have that, especially you don't have that across regions across regions, you need a different model and I put those blue boxes here at the bottom. They are not consistent as or in the sense of entity, for example, they're eventually consistent.

A

So if you need Global State, you have to somehow get that in an eventual consistent way.

A

All right, so we have three dimensions: horizontal scaling, API servers provide a provider Persona, so this is controllers which are against multi-workspaces. If you do that, you get the service provider Persona into the system and Planet scale, so it's really distributed over the planet and the question is again think of a compute model. What can you build with such reconstruction and that's a challenge for us today?

A

So if we translate that imagine you have one one million generic workspaces, cheap and quick as namespaces, so in in one milliseconds, you can get another one spread over the planet and if you do the math 1000 object per workspace. You get a billion objects. Something like that and I have some examples here. What you could build with that who has built end-to-end tests of controllers, I? Guess a couple of people here, so controller runtime controller tools, there's something like an end test right.

A

It starts an API server and you can do your your control, Loop testing. If you have such a thing, which has space for one million, forget about an API server, just use such a thing and Point your controller against one of the Clusters one of the workspaces and do stuff until it's finished and then delete the workspace again again. 100 milliseconds, you have a new one.

A

Super cheap super, easy, that's one one use case: if you have a million workspaces you might think about how do they relate one obvious thing is Arc hierarchies, so you could maybe model a company hierarchy in workspaces, so every workspace has one parent, and this means you get a tree like this yellow company and it's a green company, it's at the bottom there. So you can build something like that and we will see later on.

A

So we have done that and we have built a user experience around around that you know in kcp it's not necessary, like it's an optional thing, but depending on the use case, this is very helpful.

A

Clusters means they are isolated in a sense they have their own airbox system, so they are their own whole whole bindings, completely independent, and this means you can basically start as a cluster admin in every workspace and it's independent from the permissions of the next one of the of the neighbor. Basically, so you can start with a model where everybody is cluster admin, again they're much smaller than clusters, so sharing is not a problem.

A

If you have many of those workspaces right, we share clusters and kubernetes today, because there's overheads like to get a new cluster. It's expensive and just takes lots of time. If it's so fast and quick and cheap to get one, everybody can be cluster admin again wooden operators. I talked about that so service providers for those workspaces. It's an obvious example see these are independent, so you can have different versions in different workspaces.

A

Green is V1 V2 alpha one and one V1 and would V2 we have here. So everybody can choose. Basically, Which CID version is desired.

A

If you do that, you can also think okay multi-10 operators, one can be one dot, two three. So an old version and people stick to it: the yellow people- and there are those Cutting Edge people. They want the newest version and they can run in the same environment in parallel.

A

So you can build such a setup and keep Services running side by side.

A

A generic control plane is super interesting if you don't talk about workloads. Cross plan is an example. Cross-Bit is a project which talks about non-workload objects, and this is a very good example, very good use case how you can use a generic CID based API server, because train doesn't need deployments. It doesn't need things about networking about ingues, it's just unused by cosplaying, so use something like sap and run plus plane on it, and the overhead for cosplaying is much much less than on kubernetes, because they can share infrastructure.

A

Think about use cases where yeah in the iot space in the edge space, you have a million objects and they somehow need a consistent workspace, a consistent control plane.

A

You could use that to connect your fridges to kcp, for example, and last but not least, here comes I mean kubernetes comes back into the picture.

A

If you have many clusters, you might think about use cases where you sync policies or where you think actually workloads or you have the source of tools which operator should be installed and as the last thing I will have another slide in a second. You might share apis between the kcp on the left and clusters on the right side and with that you get something like API as a service based on crds.

A

So if you want to understand more about the last use case, there's another talk tomorrow and uh birds are feeder on Friday about the idea to have basically services on the kcp side, which are multi-tenant just by the system by definition, and to connect many many tenant clusters, kubernetes clusters to kcp, and you get a model which implements a software as a service based on kubernetes cids, supernative No, Rest API, which is non-standard, but with normal cids and yeah. Those two possibilities to hear about that.

A

All right, so I showed some examples. I I bet the audience here has many more.

A

We start with the fresh kubernetes without the workload apis, every cluster is generic and isolated and we add stuff and the challenge for you is to think about what could you build with that by adding by re-adding the things you want and to have many many of those workspaces all right, so um zooming in a bit into API sharing so I said a service provider can Define apis and can export them to many workspaces and build an operator operating then on the tenants.

A

So here in this example, that's the actual API about that to get a bit more concrete, so you have a API resource schema. Imagine this looks like a crd, so everybody who knows cids, you have the names here, the current list kind and so on. That's basically a CID spec, nothing else! You define that in the system in kcp.

A

If you do that, nothing happens, it's not that kcp serves anything. It's just a definition of a resource schema and then we have built something called an API export. So we can take those schemas name them here in the in the list, in the slice, create an export of that and make your API resource schema as an API available. Basically, in public sets to the kcp to possibly one million workspaces, everybody can bind to your export and use your controllers.

A

Basically, your semantic, you define for your API, so up to a million API bindings can exist and they point in some way. So we use a path here in this hierarchy in the interporter type. They point to the service provider provider, to your export. Basically, and in the moment the user in his workspace creates this object.

A

The the API in this example, its third manager, is available in that workspace. Nothing is running in the workspace. You are the one on the right side who operates to up who operates those objects, so you will see every object: the user like certificate in the case of search manager, every object, the user creates. You will see in your controllers and you will do whatever is necessary to implement the semantics and again one two one million. That's the idea all right, and this feels like software as a service right software as a service.

A

You have this Persona, this one person. Basically, we export something and many many consumers and you come back to I mean if you do, that you have software as a service, at least in kcp, and this is from another talk around I did some time ago um in kubernetes we have build something which doesn't support SARS. This export thing doesn't exist in kubernetes right. You can have cids, you can have operators, but there's no way to get an API from somewhere into your kubernetes. So we have built something like that.

A

We have something like Services, we have cids custom types, but there's something missing, and this is basically this thing and there's a CLI for that. We have built to bind and Export to your local workspace all right. So um one million workspaces, a billion objects, cheap and fast think about what you can do.

A

um This is a website we have created to to bring to to say basically what what kcp is and the main the main points here I want to to highlight. Maybe this is the most important slide for understanding. This is not about replacing kubernetes it's using kubernetes at its core, like we use it. Basically, it's a complete API server minus the workload apis is in kcp. It's used in kcp everything in a workspace. So this is a threat rule set in stone.

A

Everything in the workspace is kubernetes, so it's every API we offer is conformant to kubernetes very important. Every workspace is like that: tooling, just works. If you use Q cutter, it just works in the workspace and the apis we build are just just conformant. They are based on cids, you don't see crds, but in the background cids are at work all the time, so we complement kubernetes. This is most important yeah and we live in Breeze, kubernetes and the community.

A

So we are all part of the community of kubernetes all right, so isolation, 2 workspaces is a big thing. We have seen that um can be many admins in every workspace. Another art admin and many devices can be connected to that. So acceleration through workspaces. We have seen up to a million massively multi-tenant API services, so you can build controllers operating on many tenants.

A

Users can consume those Services natively via a bind command and there's nothing running in workspaces. You don't see how things are operated, but they are available and the apis just work, as you would expect think about. This whole thing is horizontally scaled and you have workspaces over the whole world. Basically, in every region you can have, you can have workspaces, but it's from it's forming, basically a mesh. So it's one system and you will see how the ux looks like in a second.

A

It's one system, one installation, it's not like kubernetes clusters, which are in this region in this region in this region, and you have to decide it's it's all in one system. Is this a white Garden? That's a important question, so we build bindings right. We have exports and bindings, it's nice, it's in kcp, but what does it mean for my kubernetes cluster and there's a project? We have also started to write multi-tenant operators in kcp and then use something like Cube bind. Kubind is an open source, Community project.

A

We have started in it's available in GitHub, so you can search for that, and this brings back everything you built in kcp large scale like if you're a service provider you ngf thousands of of customers. You can use kubernetes apis built in those workspaces and bring them back to kubernetes clusters and qbind is doing exactly that. So, if you look here at the bottom, it's small, maybe you will find again a cube cattle bind command but again here it's not against an API export in case AP, but it's against https, URL and tomorrow.

A

There's a talk about that. To see it in more detail how this works. So no white, Garden kcp, is a back end. It's a system to build multi-tenant Services! That's one view of kcp. So show me the code. What does kcp s code? Everything was Theory now. So let's get this into more concrete context. So here's a repository, Casey, PDF kcp on GitHub and yeah. You can check it out. You can. You can run kcp. So that's that's cool clone that into the to the local directory, and it's all in one.

A

If you started with kcp start it's it's not like kubernetes with different components. You can just say case. We start and it's running. If you do that, there's an admin Cube config being created in dot kcp. If you set your Cube config variable environment will be able to that file. Cube cattle will work. It's kubernetes you in a workspace. So two two commands later it's running and you are in in the first workspace.

A

So we built this command w s which stands for workspace and it's abbreviation. You could also write Cube calculate kcp workspace, but because this is very long and you type it often, we aggravated that to just ws and this operates more or less like CD change, deer in in Unix, so you can say Q Capital, WS Dot, and it tells you un root. Hood is the first workspace. There's nothing else in the system.

A

It's pretty fresh, fresh LCD, just one workspace called Hood and you can create another another one, so I create one which is called cubecon and it's created. It takes I, know 100 milliseconds to do that. It's ready and we can enter it. So let's enter it, I said it's like CD, so let's say Cube cuddle, WS kubecon and we are in in the background your workspace is put into the group config. So there's a different URL pointing to the new workspace and you have a fresh kubernetes like cluster.

A

It's empty, it's generic, but you can add everything you like. Why are API bindings, for example, or any crd you like and remember, orkaya Keys was the example. What this is implementing is exactly that. We are now at this position in the hierarchy, so we started somewhere at the at the top here and we moved one level down.

A

That's our cubecon workspace and we are there and we can go back like I said be in a different workspace, and this is very much like every workspace is like a directory in Unix same idea and again this is just one way to see and use. Kcp kcp is much more generic. You can.

A

You can run kcp at least we're working on that without any hierarchy, but this is just one way to to use it and for many use cases it's pretty nice to have that otherwise, and every workspace again is like a kubernetes cluster without apis.

A

Of course, you have to add apis. We will see it in a second how this works, so you can see API resources. What is there config map Secrets events, namespaces resource code, everything which is generic service accounts, airbag and so on. So this is what you get when you check it out and start kcp start you get a cluster with one workspace and you can create any any number you like.

A

So to sum up, kcp, as a repository gives you that gives you a generic control, plane, So kubernetes based on kubernetes, but we have removed disabled reports. It gives you workspaces so isolation in one in basically in one chart um it gives you a command to move around. It gives you export and bindings, and it adds sharding we're working on charting to run that to run that in multi. Multiple shards across the globe was eventually a consistent, Global State.

A

You can imagine why I'm talking about global State, one Global State, you see here on the slide like the exports, must be Global right. If you export something to everybody in this mesh of workspaces on the planet, you have to to to replicate your export everywhere. That's one example. We have more examples of that, but API export is one of those things, so you will serve everybody in this mesh of workspaces and that's the fourth uh Point here. That's charting.

A

All right, so we have empty workspaces and you can imagine what you can build on. On top of that we have some things. Some projects, some site project space, is basically some Services. We are working on and one is to re-ed compute. So we have removed compute cool for many uh for many use. Cases like if you talk about Cosplay and you don't want compute, but compute is kind of interesting and one topic we have tried what we're working on is building something called transparent multi-cluster. So it's a compute service. It uses the apis.

A

That Cube also provides the deployments and services and similar things, so you can apply as a usual apis. You know, but you get something like a basically an elastic compute service based on kubernetes apis.

A

The inside here is I mean this is just one service very important. There can be many many others and what we have done here- and this is a pattern- I think um people like in this area. So we have re-edited deployments. We have re-added services, but we have not reused the scheduler of kubernetes so to build elastic compute. You want a different scheduler. Obviously so we run workloads not in kcp.

A

We run them on real kubernetes clusters, but we are hiding the Clusters from the user, so you can add more and more clusters as capacity basically and the user's workloads. The deployments in the example here will be scheduled on to those capacity clusters, so we have replaced something of cube. So this thing, the the construction we have built here in kcp by removing things, gives us lots of flexibility to re-ed them like we can re-add deployments, but do another kind of scheduling.

A

If we take that to the next level, there are different like different kinds of compute. You can imagine so many people might know v-cluster. For example, can you build recluster on kcp I bet you can again you get all the sharing of workspaces.

A

So it's an interesting topic, maybe to talk about things like V cluster, where you put kcp on top of one cluster of one openshift or one kubernetes and reuse the compute below, but could give everybody a workspace but share infrastructure, light API server, another example you want deployments, but you want to schedule to Edge devices.

A

Edge devices are I mean if you have many Edge devices like tens of thousands or Millions. Even you will not have a million nodes in a cluster right. This doesn't work so the schedule of kubernetes, maybe it scales up to 5000, but not to twenty thousand or a million, so maybe for H. You want to do the same patterns. The same thing you want to get back deployments and other kinds of yeah workloads, basically workload like resources, but you want a specialized API for the scheduling for the placement of workloads.

A

So there are people in the community who think about that and next week say they start something like I know. Maybe it's a sick which is coming up. We will see but to talk about apis placement apis, which we use, Cube compute objects, but build another placement mechanism, and there are many more of those things, and this is a pattern that you can basically reconstruct three compose things. We know and you're much more flexible than in kubernetes all right. This is a website. So take a look. um We love collaboration.

A

Everything here is open, so it's kcpdev, slash kcp on GitHub, um it's an open source project, Community Driven. We have weekly meetings on Tuesdays, 5, PM, European time so uh six hours before what is it 11 or something um we have a YouTube channel. If you want to watch um discussions or Community meetings, it's all there um if you Google group. So if you want to, if you prefer emails, that's fine and um the most important thing we have a slack channel, so we are always busy basically doing work hours.

A

So there's always somebody if you want to talk to us, explore kcp or even help us contribute. That channel is the main place for communication.

A

All right. That's all I wanted to show and I think we hopefully have some time for questions um before we start questions. um We have T-shirts here in the front. If somebody's interested in t-shirts um Andy is here, NDS co-leading um kcp, um he just he was the the two T-shirts I think so come here and yeah all right. So time for questions.

B

Great presentation, thanks Stefan um two questions: uh is there any relation to uh hierarchical namespaces uh with kcp.

A

No technically not um namespaces, so we are also asking. This is a similar question. Can we just do namespaces V2 I mentioned that right, there's something similar and namespaces. We want there an API and it it is limited, as it is right sure. So we cannot change anything. um Workspaces have a hierarchy. If you use this model with this model, you get a higher key, but the namespace is inside are flat again because their names, basically one the pattern. You see, people don't care about namespaces anymore, so just stay in default. That's it.

B

So I mean, would it be possible or thinkable to have the workspace, be a core Concept in kubernetes API.

A

um Yeah, it's feasible, certainly the API server. We are using it's a cube, API server with modifications, it adds partitioning of the key space and yes, technically, this is feasible. It's not done at the moment. So we add, we use Cube API server and API extension API server, and we add exactly that.

C

Okay, you touched on the idea of sinking at global scale and an eventually consistent manner. What's the mechanism for that, what would you be using to have them talk to one another.

A

um At some point, we're just exploring something pretty simple: it's a distributed cache, so the objects we know we want to share, we want to replicate every chart, will proof them into a cache which is globally replicated and the most interesting bit here is the question: what is a compute model in such a world like how does a controller work on eventually consistent, Ary data?

A

So, as a model we came up with which we are implementing is basically a controller which is aware of that right, it's a different model, a little different from kubernetes extended. Basically, it has a local Informer, so it has a local memory cache from the local chart. So it sees everything every workspace, local and if it doesn't find something there, it has a second informal which is fed by this distributed. Cache, that's a compute model. We came up with um this works very fine with those things like API exports or in in compute.

A

We have sync targets, for example, things which are pretty static in a sense, um I would eventually consistency and Nothing Stops us from adding apis, for example, for consensus. So we, if I mean sometimes you want that right, you need consensus, which is summer Global and there could be an API which is based on a different mechanism. Not this distributed cache, but something else which gives you consensus. Those things can be built in, they can be even Primitives.

C

A

Not I mean it's an open domain, basically, what we need and what we build there.

D

uh Thank you, Stefan for the presentation you mentioned earlier, like that the target is to be able to create a workspace and make it as cheap as nice places. Can you tell us more on how workspaces are implemented like is there.

A

D

Api server, it's process a dedicated etcd. How does it work and.

A

So um we add something: I mean you know it: how containers work in in the Linux kernel right. um There are some kind of context and there's a namespace name passed around in the Linux kernel. We have basically the same model here, so we pass around in the context, something we call a logical cluster.

A

It's part of the API server URL, the the client talks to so we derive it from the UI From the Path and pass it around and in the storage layer of kubernetes. We turn that into a different prefix in at CDE. So that's a technical trick to do that, and there are some some things. I don't go into details here, but you want efficient lists cross tenants, but on the same resource. So you have to put the logic cluster name at the right position in the key, and we can talk about that in detail.

A

But that's idea! So that's why it's efficient! That's why you can have inform us cross workspaces.

E

ah Thanks Stefan great talk, uh I've done some questions, but I'll limit it to one. For now uh you mentioned that with kcp you envision, like developers having a single API where they can still deploy things like Cube CTL applied chef deployment.yaml, but the actual workload runs on one of the underlying workspace clusters.

A

Yeah in this model of transparent multi-class, it's one service I swear that it's one service built on top. It's not necessarily the compute service. You can have many other variants that one works in a way that every kubernetes cluster, which is capacity for the service, has a singer agent which connects to kcp. It watches its tenants, so it watches deployments and it pulls them down and there's some transformation happening to make them work on the kubernetes and then they are executed in kubernetes and the status is soon back.

A

So in the workspace you see deployment dot status, whatever you are used to in deployments.

E

I see okay, so syncing all the status back to the kcp server for that deployment.

A

That's a model of the service yeah I see yeah.

E

F

We're a little bit out of time. Is it okay? If we keep going.

F

Keep going uh thanks for the great talk: um are there already client, libraries or documentation or guidance on how to write a multi-workspace controller.

A

Yeah yeah, so uh so Andy is an expert here. If you want to come after the talk, um he knows all about that we have a extension of controller runtime clients and we have a generator which is similar to client go which adds scoping to informers and everything. So you can. You can watch everything on a chart, but you can also scope it down to one workspace. So it's very client go like I think it's even derived from that I'm, not sure but NES expert, so client, libraries thanks.

F

A

Lot good question.

D

We're gonna do one more.

D

G

G

Kcp subject to all the scalability limits of kubernetes, so when it comes to hcd as far as number of objects, you store, or you know, the way, eventual consistency or replication, Works isn't kcp, subject to all the scalability limits of hcd. The same way you.

A

Mentioned scheduler hcd is um I mean if you scale out, we have multiple LCDs, we even researched, it's not really ready and we don't use it at the moment. Research to use something like cockroach as a basis to scale even more, but the point is- and this is it's an intentional decision- workspaces are smallish clusters, so they are usually not the size of a 5, 000 old, kubernetes cluster.

A

So, of course, for one workspace, you have the scale limit of NCD, obviously because this always lives on one LCD or the whole episode machinery limits. What you can do in one workspace, but because workspaces are this consistency domain, so the resource versions only make sense with in one workspace.

A

You can scale them up like you can have many many workspaces or many many charts and every controller really cares about one partition of that. That's the way how we get much higher scale.

G

I see so you're, focusing only on a subset of objects like workspaces and through the export apis, you're selectively, exporting them to other clusters and that way, you're managing scale. So.

A

The objects always live in one workspace and the workspace has it must fit into one LCD. That's that's the limit. The export is cost workspace, that's completely independent, but the objects live in. One workspace.

F

Thank you so much.

A

F

A