KCP KCP Design Discussions, 24 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Logical cluster deep dive

Description

A discussion around how logical clusters are implemented in kcp and how we might approach enabling priority & fairness per logical cluster (i.e. per workspace)

A

All righty: let's talk about how kcp and kubernetes support logical clusters or workspaces.

A

So um just to recap what we talked about briefly off: recording kcp embeds, a fork of kubernetes that has been modified to be logical, cluster aware, the fork is never intended to be externally visible to anyone trying to code clients and controllers against kcp.

A

It is only for internals of kcp to function so existing clients or existing controllers that are able to talk to kubernetes or openshift or anything that looks like kubernetes or openshift from an api perspective will continue to function. Discovery exists, so everything you could do in client go. You can do against a kcp workspace.

A

Some resources may not exist so, like kcp doesn't have pods out of the box. So if you have a controller, that's trying to manipulate pods- and you point it at kcp- it's probably not going to work, but if you're just working with namespaces and custom resource instances, all of those mechanics will function if you're trying to write a controller or a client that is able to talk to multiple or across multiple logical clusters.

A

That requires some additional coding, but we're trying to provide some helper libraries to make that easy.

A

So we have a document that we haven't merged yet, but we will try and get it merged soon. That shows how we have modified ncd to be multi-logical cluster aware so, what's.

B

A

To look at is, uh if you look at the normal current storage mechanism or storage pattern, everything's under registry subdivided by group resource uh and then in in kubernetes or openshift. There is no cluster, it's just group resource and optional namespace and then a name and for for kcp.

A

This we add a cluster segment in between the resource and either the namespace or name for all of our built-in types. So that's for namespaces secrets, services or sorry service accounts that sort of thing anything that is a custom resource instance will have registry group resource and then either an identity or the fixed string, custom resources for the purpose of this discussion.

A

What those mean is not relevant, so I'm going to skip over it and then just like we saw down here, we do have a cluster segment that precedes an optional namespace and the name- and this is how we introduce multiple logical clusters into an xcd store. That's back in kcp and then all of the code in the kcp fork of kubernetes, which is in kcp dev cooper.

C

A

um So everything in this fork has been modified to support the multiple logical clusters that um are visible in the city storage patterns. Here, let me pause there any any questions on any of this.

D

uh Yeah so for the the cluster uh segment that there can be a hierarchy there right.

A

No, uh the hierarchy is contained wholly within this segment, so it could be something like um root: colon red hat, colon engineering, okay,.

D

A

Colon to separate in that hierarchy, so we are um guaranteed that there's only ever one level of nesting when it comes to specifying the cluster. Just because we we.

C

A

That hierarchy within a single stream.

B

Okay, good question.

A

All right, so how do we do all of this? So uh basically, if you go into our fork, which is what I have open on the screen here anywhere- that you see a kcp dev logical cluster import is somewhere that we're doing something with logical clusters.

A

um So let me find I'm just going to be showing some examples here. I think um you.

C

A

Grep for this on your own and then, if there's individual questions that you have totally open and happy to answer those questions, but just as some examples, I'm inside of staging source, case.io, api server, package, storage, etcd3 store.go.

A

So this is where we're interacting directly with etd at the storage layer, and so one key thing that we do is there's a decode function that is invoked when getting from met cd after creating an object and it's been stored in xcd and when doing conditional deletes and guaranteed updates.

A

So if you look at get, for example, get will determine the logical cluster name from the context. This is something that kcp is responsible for setting in a couple different places um why my font size went down. But um if we trace like you'll, see that there's a whole bunch of places where this with cluster function is invoked. And so you can go look in the places here and they're, also in kcp.

A

A

Let me pull over kcp hold on a second.

A

B

A

Actually closed it, that was weird.

C

C

Oh, that's what I was looking for: command shift, okay,.

A

Ctrl shift, so if I go in and look for.

A

Okay, nevermind I'll come back to that later, um but suffice it to say there is a cluster name, a logical cluster name that has been added to the context. I have to go back and double check where it got added, but in this particular code flow we are retrieving a key from ncd.

A

Once we have retrieved the data we pass it to decode decode will do some decoding and then you'll see. We have a kcp comment here. That says we need to apply the cluster name to the decoded object, as the name is not persisted in storage.

A

This is key because we are not storing anything about the logical cluster that the data in ncd is associated with in ncd.

A

So what we do is we know the cluster name from the context, mainly because, when you're doing when you make a request to kubernetes the request pattern that we have looks like uh one of these, so this is an actual url path that kcp would receive from a client.

A

It would be slash clusters and then you would have a cluster name and then the remainder of the request would be a standard request that you would make to kubernetes or openshift so slash apis. Some group some version some resource. It could be slash api, v1, slash secrets, so anything you can do in terms of metrics health, liveness, probes, apis, api requests. All of these have a prefix with the cluster there.

A

So when you're, looking at this, that you know that the cluster name generally came from the url and we actually store the cluster name as an annotation and that annotation is kcp.dev cluster.

A

So um once we are decoding data, that's coming out of ncb, we set that and then that is available to anywhere in the code base that needs to work with it and the logical cluster package or repository. That's in kcp allows you to take any basically anything that has a get annotations method on it, which is all kubernetes objects and you can call logicalcluster.from and that will return a logical cluster name. That's been decoded from those annotations, and so what you'll see um is logicalcluster.com.

A

So you'll see in a lot of places here the namespace controller, for example- or this is the um yeah. The namespace controller is using the logical cluster for a namespace to figure out what logical cluster we need to delete things.

A

The um there's a whole lot in here so uh custom resources, for example, we'll take a look at the crd, get its logical cluster and use that as part of establishing crds. So there's a whole bunch of code in kubernetes in our fork. Where we are, you know, injecting or doing some logic that involves the logical cluster.

B

A

There, what questions might you have.

A

Oh you're on you, jamie.

D

Yeah, sorry about that, so um my question about the um is about the the annotation. uh So in that case, annotation is part of the the object data right, yeah that's stored in at cd.

A

No, it's not stored in ncd. It's only available on the wire when it's being decoded.

D

B

D

um So so logical cluster is not really a cluster. It's um a way to distinguish, um or- or I have some isolation in the objects being stored in the storage.

D

Is that the fair description, yeah.

A

So a logical cluster is meant to be like a real cluster, where you have cluster scoped resources, like name spaces or uh cluster role, cluster roles and cluster role bindings.

A

You also have name space scope, resources like configmaps, and all of that is what you'd expect to find in a normal kubernetes cluster, but in kcp, because we want to have the appearance of isolated clusters within a single api that you can access.

A

Each isolated cluster is what we're calling a logical cluster. So uh one advantage to this is if you and I are both sharing kcp and I have a logical cluster and you have a logical cluster. We can each create the exact same crd in terms of group version and resource, but my crd could look 100 different from your crd and they will not collide with each other.

A

I can't do any crud operations on crs from your logical cluster for that crd. Unless you give me access to it and vice versa, you can't do that with mine and so it from that perspective, it is like n clusters in one process.

B

D

So um one question I have is um when, when there's a so I know that a workspace is mapped to a logical cluster. So each time you create a workspace, you you have a new logical cluster in in the kcp.

D

A

Right yeah, so every workspace has a logical cluster. Not every logical cluster has a workspace and the the logical clusters that don't have workspace are reserved for internal kcp system use. um So there are privileged kcp admin level. Clients that can go do things with these logical clusters that don't have workspaces, but you generally don't need to know much about that or worry about that.

A

So yes, like a workspace, is a logical cluster. There are more.

C

A

You can do with a workspace that there's like extra features that are in there above and beyond just saying it's a division of.

D

Ncd, so, from a user's point of view, um a user doesn't really see logical cluster right when, when the user is inter interacting with the api server is through the whichever workspace that he is in.

A

Yeah um now you can switch workspaces right.

D

But that also means switching a logical cluster right because there's a one-to-one mapping.

A

Yes and um controllers do have the ability to see data across many logical clusters in like they have the ability to do um what we call wild card lists and watches.

A

So, if you make a request that looks like this, you will get instances of the group version and resource that come from lots of different logical clusters or workspaces.

A

There are some special requirements for uh what you must do to make this work, but they're they're, straightforward, and so you can, um if you're, trying to write a controller.

A

uh Let's say that you're, I don't know you're doing cert manager, because that's our canonical example, so cert manager, today only operates against things in a single cluster makes sense that that's what clusters are they're standalone, cert manager could be updated so that you could install one copy of it and it could handle certificate requests in any logical cluster across all of them. um If you don't rewrite or modify cert manager or any other controller, then you have to have one copy installed per workspace.

A

Looking at that workspace, or I should say you need to run one copy looking at each workspace, it can live wherever.

A

But the cool thing is about the wild cards is that it will allow you to reduce the number of instances that you have to run while being able to support consumers or users who span multiple workspaces.

D

And how do you configure the um the actual workspaces that can be visible to that controller?.

A

You see, we have something called an api export, so this is a another concept or a different topic where you, the the mentality or mental idea here is, I am an api provider. I want to own my apis and I want to own the implementations of those apis and I want to share them with people.

A

So to do that, I would create something that looks very much like a crd, but it's called an api resource schema. I create an api export that points at one or more api resource schemas.

A

It just references, my name and then once I have that api export defined, then I am able to look at its status and there's a special url in there that I would consume and use, and if I access that url, I have the ability to do a wildcard list and watch and- and I can do- um I can do individual crud operations per logical cluster or per workspace. So uh if I know that you have a certificate signing request in your workspace abc123, I would have a url that looks like clusters.

A

Abc123 certificate sign in requests and I'd be able to do stuff with that.

A

um Let me s, I don't see if I have an example.

A

B

C

A

Export, actually, I might hold on.

C

A

Let me pull this off screen and see.

C

A

Server running so, let's see.

C

A

A second okay: here we go so I have my local kcp running and you I'm in the root workspace, which is our top level workspace. And so, if I get um we'll look at tenancy, what you'll see down here at the bottom.

A

Is this url that shows up? So, if I do this.

A

So I can put in that my url is services, api export root and then the name of the export. You know this is the url that I got and then I do clusters star, and this will give me discovery for everything that's in here and so I could say, um cluster workspaces, whoops.

C

C

Miner might not work.

A

Okay, um I'm oh. I.

C

A

Any I must not have any findings on this all right, I'm not going to try and make this work on the fly, but basically um this is what you would do. Is you would you would go through the virtual workspace url for the api export and then you'd be able to see these things and operate on them? So if I wanted to do something on a single um single one, I could be. You know, root abc one two three, and if this particular thing existed, I could I could get it. I could create it.

A

I could list it, etc.

A

So um I know in terms of like priority and fairness.

C

Trying to remember the name of the.

A

A

I looked at this code once before.

C

Actually, I think I had an issue for it.

C

C

Yeah here it is okay,.

A

So this function in uh staging kate's, api server package registry, generic registry store.go. This starts the observer that periodically tracks object, counts per resource name, and this is um created when.

A

Where is this so inside of complete with options which sets up storage for a particular resource? It does call this start observing count, and the problem with this is that this basically is monitoring.

A

uh This gets invoked once per api type, so like once for secrets once for config maps once for widgets, whatever you're doing, and it's going to do an object count based on the prefix and the prefix that this thing is doing is based on the cluster.

A

But the cluster that it's putting in is a wild card and there's some um some hackery that we did so if, if you're doing a star, which is the wild card request, then the prefix ends up just being uh an etcd registry group resource that ends up.

A

That's the prefix, and so this ends up counting all instances of whatever your resource is across all logical clusters, because if you count where this is the prefix, it's going to include cluster one cluster, two plus three, uh and so the idea that I had in my head for how to attack this problem was that while we can start observing for that global prefix, you kind of need something that is watching for workspaces and when a workspace is created, you start a track or start one of these observers for each workspace.

A

But we probably don't really want to include or can't really include any kcp code like cluster workspaces as an import in here. So we need to find some way to be able to.

A

Like from within kcp, get kubernetes to start these observers based on events like a workspace being added, but make it generic enough so that we have the ability to do that without importing the cluster workspace types into kubernetes.

A

If that makes sense,.

D

um Yeah, I'm trying to understand so you're you're, giving this as an example when, uh when you need to.

D

uh You're saying um if you do want to distinguish the the workspace, uh this would not be the way, because it's using a wildcard to uh to to replace the cluster name in the prefix.

A

So yes, and no like the biggest problem, is this is called once per api type.

A

But so, if I create a thousand workspaces that has no bearing on this function being invoked because it's invoked when the api type is added to the api server. So that's when the built-in types, like secrets are registered. It's whenever a crd is created. um It's going to do that uh per crd, and especially with the api export bit that I was showing a few minutes ago when I create an api export for certificate, signing requests.

A

That does turn into a crd, but it's only just one. It's like one crd that represents certificate, signing requests that are going to live in a whole bunch of different workspaces.

A

So if, if we want priority and fairness to use, object, counts and those object counts need to be segmented per logical cluster per workspace, then we have to do something that can start counters and track counters per logical cluster and resource, as opposed to just per resource.

B

B

A

um I apologize I I need to end this a little bit early, I'm starting to get like a floater in my vision, so um I kind of want to go and lie down. um Okay, but please continue to ask questions in slack and um when I'm not having something flashing. In my vision, I'm happy to chat more.

D

Sure sure yeah I need some time to digest. uh Whatever you just said and um look more into the code, and I might have some questions about how you know an incoming request is dispatched or does it really make any difference if there it? The request is to crd within a workspace or or it doesn't really make a difference.

D

um The when I say make a difference I mean: is it the same hand handler chain, that's servicing that or yeah um yeah. So there's no like um individual handler chain per workspace right. It's all served by the same handler chain, but there might be different actions depending on the workspace or logical cluster. There.

A

Is an individual hand or there is a shared handler chain for pretty much all requests.

A

um Resource quota is running a variety of go routines per workspace, so we like that's just an example of where we have multiple controller instances and multiple admission instances where it's one per workspace, we'll do the same thing for garbage collection when we implement it other things like the tokens controller rather than running one token controller per logical cluster. We run one token controller and it's handling across logical clusters.

A

So, depending on what part of the code base you're in things may be different, but in terms of request handler, it is one handler that's doing everything for incoming api requests.

B

D

Okay, yeah I'll I'll, um try to understand the code a bit more and then I'll come back with some more questions. Thank you for um this session.

A

Okay, you're welcome um and please reach out in the future, uh happy to provide more details.

D

All right, thank you. Oh good, one I'll see ya. Thank you. Take care, bye,.