Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Towards Something Better Than CRDs In a Post-Operator World - Stefan Schimanski, Red Hat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Towards Something Better Than CRDs In a Post-Operator World - Stefan Schimanski, Red Hat

Speakers: Stefan Schimanski
CustomResourceDefinitions are driving the extension ecosystem around Kubernetes. This talk is about the search for the next step, a successor for CRDs in a post-operator world where service providers use CRDs as first-class API for the services they are building and offering to tenants. CRDs as we know them are installed in customer clusters, usually together with operators or controllers. With that they are under control of the users: - users can tweak the CRDs. - users are the ones updating and controlling the operators with all the complexity and pitfalls updating operators and APIs can have. This situation is not a good fit for today's problems, and it's mostly an artifact of how CRDs and their life-cycle were conceived years ago as a tool to add in-cluster concepts. This talk is about lifting CRDs up to be a first-class verhicle for APIs provided and consumed by different parties, without the operator-glue, in different clusters, standardized, securely and federated.

A

All right so welcome to the last talk. I think before lunch about cids everything is I. Guess was about crds those who have seen Joe before it was about cids this as well. I'm Stefan, schumanski I'm, one of the initial authors heavily involved in cids, so lots of the things you see today, defaulting structural schemas. Blame me for that. I was all in there um today. We want to talk about something which is CID based, but could be a next step for the ecosystem to use them.

A

So this is how it started a few weeks ago. I realized that so it's so hard still to get something like a my secret claim. Api into my cluster. There are so many providers providing MySQL, but if I want to have one I have to go to some websites or some cloud provider UI to create databases. It's still not native right. I cannot say cue cattle create MySQL without installing anything into my cluster. So what I want to have? And that's a submission?

A

That's basically the the doing where we want to go to as an ecosystem. I don't want operators to do that. Like I want to consume a MySQL from anybody who offers it in some service, I want to have it native CID, based in my kubernetes, but I don't want to run anything I, don't want to run an operator just to get a software as a service MySQL and for eight years we have built systems like operators or for eight years, but in kubernetes. Basically, we don't support software as a service.

A

This is basically is a rent here. This started as a Twitter tweet um very I talk at the time and many people answered I guess some of them are here.

A

There is no such support in kubernetes, while the whole ecosystem around us builds software as a service. So can we change that and six years we have spent in building operators and I'm, focusing here on operators which don't operate like operators which don't run local pots?

A

This last class is totally um it's a good use case like if you run pods, and you want to run your on-premise in your cluster database. That's fine, but if it's a software as a service I, don't want operators I think we as an ecosystem, do something which is not really helpful, basically creates so much complexity when I have to maintain operators for software as a service. Basically we we have building or we are building that right.

A

Everybody knows those pictures, many adapters between different worlds, different standards, different um API variants, and but we are in Cube here so we're in kubecon and we love Cube apis. So we need something else, and that was basically the result of this Twitter and we need to build something which gives us software service natively in Cube, that's the topic of today.

A

So the future. Let's imagine I want a database, so I chose not MySQL here. I chose mongodb and trademarks and everything so mongodb. For today we want mangoes mango DBS. So what if we had this command in Cube cuddle? Cube cuddle bind against a URL at mango dbcom. That's this big mongodb service provider. He implements they implement this protocol, whatever it is. In the background, imagine this exists.

A

What happens? um I'm redirected to my browser by that I'm developer? So I want a super, easy experience, I'm a GitHub, so I click on authentication at GitHub, I say yes, mongodb should see my my identity. So let's click I'm here I want the mongodb CID. Basically the resource which can claim a mongodb from a service provider.

A

Imagine there could be more I mean there are some companies who have hundreds of different types of objects? Mongodb is really single. Focus I just have mongodb I bind to it. I click on bind I'm asked about additional permissions, so mongodb wants to create one secret where they put a token to to connect the database and they want to create a service. In my cluster, where I can connect to to connect to the endpoint of the mongodb and I say yes, they can do it and we are here.

A

So we have a Cid in the system. My kubernetes knows about mongodbs, nothing installed, one command. Few clicks done, that's not going to be claimed so get money. Business empty, but I have one here so for my porch mongodb database, very simple, nice API I apply that I check. There's nothing running here. So just infrastructure stuff- you have in your in your cluster, there's no mongodb Port, because the service operator, the service provider, runs that for me, a few seconds later, it's running, there's something provided by the provider and I'm connected can look into it.

A

There's a status everything we know from normal native cids, it's running type ready, is to looks great I have my service, so I gave the permission to get a server a service just this service, nothing else. It's there. I can connect my application to it. I can check. There's an API binding, so Cube cattle bind creates a binding there's a URL. It's connected ages, 11 minutes, there's a heartbeat between my cluster and the provider, so everybody both sides basically knows the status of the connection.

A

That's it that's the future super simple extension mechanism for cube. I, don't want operators for that. We can do better so to look a bit into details. What have we seen? Api is consumed in the cube cluster without without running anything in that cluster.

A

We have seen more, we have seen API bindings, I checked the API binding the status of that, and this is a sketch of the API. This will become much more Concrete in a later point in the talk here. It's just a sketch, a binding on my side and an export on the provider side. On the right side.

A

You have the provider Persona service provider as a database provider mongodb as a company they export the API and I can bind to it so I just say the URL is that one I want those resources to remember the selection of The Binding, so the resource I want mongodb. In that case and there's a binding security I don't want to give anybody access to my cluster I think we share this, usually that service providers shouldn't talk to my API server.

A

So in this case, what I want the service partner needs ways to see: The Binding, that's the first thing, maybe access some more things like token secrets and service, but if there's other stuff in my cluster, like other service accounts, deployments, conflict maps, anything which is not connected to mongodb, it's a service provider, foot or even C and, of course, not access.

A

There's more than that everybody knows those screens like on the mobile applications can ask for permissions, so they can ask. um Can I read your GPS position? Can I read your contacts whatever everybody knows that and similar things for oros based authentication? So if you you have seen that earlier on the screen, so I gave the permission to see my identity to mongodb similar thing and the similar thing we could imagine in this model, like a service provider, could claim access claim access to the mongodb object. This is pretty natural right.

A

That is nothing special I would have to agree with if I buy into mongodb, of course, service provider for see my mongodb objects, but we also saw services and secrets, but very very slim, down like we use a minimal set of access, and we call that permission claim. This pattern could be something in such a model where you can claim access beyond the actual resource you're exporting.

A

There are more variants of that. Who has that in the basement? Guess many people know that right fuses in the basement? There's this thing: um you cannot access it I mean technically you can. You can open the anti-temper seal with any kind of tool and access it, but next time the electrician comes into your house, so they won't be happy right. So similar thing, um we could protect access to certain things.

A

One example here is: there's a mongodb object: it has a status right, the user should not touch status, I mean if it does it's your LCD, so you can do it technically, but we can do measures Implement measures like admission, for example, which just stops that I mean in the usual case. Again it's like an anti-temper seal. You can open it, but there's a reconciler and the service provider will see what you have done. So there are consequences for the contract with your service provider.

A

There are more things you could place: objects into the cluster which the user maybe shouldn't be able to update like it is resource quarter again, it's not super high security, it's more like to prevent bad user experience if I only have permission to create five mango DBS at the provider level. A resource quota locally with the number five as maximum makes sense right. So you get a natural, Cube native message. Oh this cannot be created because quota.

A

Those things can be built in this model. So what we have seen- and there are of course, more patterns, more ideas and you can be creative um to to think about that. We have seen exports and bindings. We have seen permission claims. We have seen those inverse things which act like this anti-temper seal, so think about it. The rent from from August it was I think there is no stars in kubernetes, but we can build it right. This future is feasible.

A

This was August 17th and well. We took this serious and just sat down and built it right. So there's Q bind it's a project in GitHub, go to it, it implements The Binding, it implements a connector which does a connection between your cluster and the other side. There are no permission claims yet so everything you saw about permission, claims and inverse provision claims it's not there.

A

We know how to build it because we have done similar things in kcp, so we we have explored this domain, so we have ideas how this could look like as an API. But this is the next step you bind for. Pure resources exists today and I will showcase it in a second and after the talk you will have a chance to do it yourself, all right, so it's coupon GitHub coupon.

A

um If you go to the to the repository, you find those three components. Basically, there's a q cattle bind plugin.

A

There is a vendor neutral component, which does all the logic so oops, oh it's hot, so there's a connector which is a vendor neutral agent on the cluster, but super minimal. It just swings the objects back and forth and there's an example. Backend implemented. You can run this example backend on any Cube cluster. It uses namespace oxidation for tenants, so you can connect many clusters and many namespaces in those clusters and create mongodbs, for example, very important. This is an example.

A

There's a protocol mostly based around based on apis, so we can look at the apis and you will see what the protocol actually is. Lots of the back end is example: code, like everything about authentication authorization is just an example. If you want to integrate that with your service provider codes architecture, whatever you have already like, if you're a mongodb and you have something like authorization- and you have user databases and everything there's no need to replace that just take this example back and adapt it to your needs- support the protocol. That's it.

A

That's the cue cutter plugin, so it's bind so essentially two parts to that. The first one is basically what we have seen here. So you pass the URL.

A

This is nice for the developer, workflow right. Everything is done by a via click in the browser very easy, but eventually you want to use it in production in some sense, so you want to use it, for example, in Argo, CD and github's tools whatever and there's a second part.

A

Basically, when you call the First Command, the second part is implicit, but you can call it explicitly, so you can basically run the bind as a buff, but as a dry run mode and outputs yaml gives you a request, object, and you can then take this request, object persisted it. Has everything describing what you want to bind to and then um yeah use that as a basis for automation, for example- and it looks like that so again, the components I just showed tube cutter binders.

A

Here the connector is running on your cluster here on the left and the back end back end is running. Here is basically a set of controllers operating on those API objects here, and it uses normal Cube namespace isolation. So it's a dotted line because obviously um it's also namespaces right.

A

Everybody here in the room knows what namespace can do, what they can't do and you can run any kind of operator to do the work on those objects and I will show in a second how to use the mongodb community Operator, just as it is offered in GitHub and just build a service. We will see that in a second.

A

Okay, so let's build mongodb in real in August. It was all around nothing else, but we can do it in Rio, so I found that mongodb, let's Implement mangodb in Yeah by using a mongodb operator.

A

So that's a repository I installed this thing in my cluster and I found out that's pretty hard to use actually there's an API, obviously to create mongodb Community objects. They define basically databases, they're namespaced and the first thing I realized. Oh, you have to create those other seven objects manually here on the right side. So there are two service accounts, two holes, two whole bindings and I. Don't want to give this to users right it's far too complicate it. I don't want that. It must be simpler.

A

That's a term I like to use here this mess, which we just have seen on the slide. It does not matter if it's in a warehouse right. Warehouses are ugly, but they are super optimized.

A

So let's try to do Warehouse style Computing in Cube for software as a service, so messy things you have to maintain them, but the customer will never see the mess. Pretty cool change to the operator model we have seen I think so. This is what I want to show my users I want to have this. This API um replica account, so it's ha or it's not ha and maybe a version. That's it and I want to map that to mongodb community, so class plan to rescue I thought preparing this talk.

A

I wanted to show mongodb directly tricky ugly, so cross plane can do this. Templating of those I mean of the main object, plus those seven objects. I also need right. That's what I did so I I want to use cross plane to make that simple.

A

Okay, that's architecture- and here you see the power of community course balance Community project I use it, as is we have built Q bind and both of them together gives you a service you as an I.T provider in the company, for example, you can build that right, there's no line of code, it's all defined by a cube, Cube resources.

A

So what we? What do we see? It's? A top in green. You see everything about, Cube bind so here's a binding, here's the export and it exports an scid. So that's a mongodb crd, reflecting basically or implementing specifying the schema, also example um API. We have just seen that one, that's a mongodb, CID and cosplaying adds some stuff for templating, think of cross plain just as a templating engine, but built in Cube resources.

A

So what we do we Define something I cannot see it here, yeah, that's a composite resource definition, that's a term in cosplaying, and out of that comes the crd we want to expose and we Implement that via a composition, that's a term in cross-plane. Basically, a template template for resources in Cube, so it looks like that lots of yaml I just give you a glimpse on that. I mean look on cosplay if you haven't seen that pretty cool project.

A

Basically what we have on the left side here is this composite resource definition. It's like CID, very, very similar, some other stuff inside. But basically you see um the name of the CID which we want to expose that's the mongodb and on the right side you see the templating. So basically, this is a API definition here on the right side. It's implementation, you can imagine you have more than one implementation, you as a service provider for mongodb in the company.

A

You run your developer workloads, maybe in some software as a service instance of that, because that's fine, it's quick, it's cheap, but your production workloads. You want to have on your database clusters locally in your data center, because it's cheaper in large scale. You can do that like you, can implement the API on the left side via multiple compositions, pretty powerful concept anyway.

A

What you see here on the right side is basically a list of a list of resources here at the bottom, you see the first one, which is a mongodb object itself and there are holds oil bindings service accounts, all the things we have seen in the slide before and out of that, if you apply the cosplay Machinery, you create those objects in the cluster and you have cost pin running. Of course you get a normal CID and Cube bind will just pick it up and Export that let's do that.

A

So what I've done here I have a cluster winding, obviously um very simple: cluster, no.

A

It's upbound running here abound is a crossbar implementation, as mongodb is operator running and not much more. I have a certain manager because I have a web front end here. I have decks running for authentication and I have my example back into one thing: that's it basically! So, let's see so I bind this command bind to mongodbde exports, I open the URL.

A

It's a wireless works. I was pre-authenticated because I clicked on the GitHub link before so. That's why you don't see GitHub I bind done, I have a CID, and this is real. It's not fake, so I can see how the connection goes.

A

Radius 2, so it's not ready. Well, it's not ready! Oh it's caddy, all right! So and now we look on our mongodb example like this one. We apply it mongodb demo and now something on the right side should happen. Just I mean you don't have to read it just that the resources created cross plane is templating. Now this stuff out into the cluster, creates a namespace there's a namespace for my cluster.

A

So here you see cluster w h something minus default because I'm in the default namespace here on the left, so I get the namespace here on the right and everything is created and eventually you see the port running. Actually, it's already has has happened so there's a demo, zero Port, it's a non-ha mongodb, it's running and I can also check status of that thing.

A

Where is it I? Don't know if it is too anyway, so Setters also isn't I. Have my mongodb super powerful? No code, not a single line of code.

A

Yes, architecture I'm happy to talk about it later, all right, kcp I'm from the kcp project. um We have explored that a little bit in a different context. Everything you have seen here is on Cube. Kcp, basically, is the bigger version of that. If you're a mongodb like the company mongodb- and you want to build that- and you have 20 000 customers cube is heavy. Namespaces are maybe not enough for isolation. Kcp gives you other isolation and efficiency in operating all of that. So there are three isolation between workspaces and everything is very lightweight.

A

That's why kcp plays a role here. That's just a sketch I'm happy to talk about details just showing it very quickly here. So this is a data model behind it. You can see the first part of the talk under this URL. If you want to share it, just do it: Cube cattle minus bind IO and it's an open source project. We have a channel in in the cube. Slack Cube bind come to the project. We love collaboration. This is really just a start.

A

If you're interested in this topic- and you want to shape it, you want to get your features in there join us there tomorrow, there's a there's another meeting. We have a room for birds of feather it's in the schedule. You will find it. So if you want to talk about read, details have questions about code and API come there. We are there to talk about it.

A

Okay, we are nearly at the end. um Any good talk needs a task and a t-shirt to take home. I, don't have T-shirts here, but we have SARS. So we have built a SAS service for you to use so I've uploaded the slides go to the scatcom website and you don't have to make a photo or anything and remember anything. There's an API called t-shirt.k8cio.

A

And you can bind to it in your cluster before doing that, of course install the bind command. We have a true index, so user use cool as a plugin installer to get it and apply your t-shirts claim, and when you do that check the status and you will see a code in the booth we had to pick it up.

A

B

Any any questions, questions.

C

uh This might be a newbie question, but how would you deal with the oauth or oauth2 flow for the authentication in a more kind of controlled environment that a user might not be? You know yeah.

A

So um the export API- actually that's a binding provider API, which allows different methods to authenticate, so the provider can say I support, oos, I support token flow I support those other things password whatever so be prepared for that. Yes,.

A

I didn't talk about that. The CID is really owned by the service provider. So you don't have to care. The service provider sees the cids. It seizes status of the CID reflected into the system, so they can see when it's I know storage versions. For example. They see it right, so you don't have to care. Of course you can touch it, but then the reconciler will take it over.

A

A

Okay, so is this something to merge into Upstream?

A

There's nothing forked in cubes, this is purely built. On top whether Cube cattle bind becomes a thing which Q pretor, just ships- I, don't know I know some colleagues are here who are involved in cube cutter talk to them red t-shirts. Here, um maybe it's not necessary technically.

D

So so this puts uh the onus on the service provider now for maintaining crds.

A

D

And that's no longer a problem for the operator. Yes,.

A

D

Gives it back to the yeah people best.

A

I would say make that decision. We fixed the personas I think this was always one for a long time. Yes,.

B

How can the developer then consume on mongodb like I, said some metadata or anything else? Oh.

A

This is part of the cids which are exported. I mean um those are an example. It's a claim and a claim which in this example it creates a service and a secret. So you would Mount this the secret into the application and you get a service like a DNS name to connect to, but this is really part of The Binding, not of The Binding mechanism. It's part of the modeling which apis you export, so the sales provider has to build something which fits your use case.

E

One more at the back.

F

Hey I think this was really cool thanks. Thanks for showing us um I was wondering what are the remaining crds, that what are the remaining crds that are still on a cluster uh on the client side. So.

A

My my mission is to get clusters as empty as possible and there's another threat around that give people admin permissions again often the ways that people don't ultimate in their clusters is because there's so much stuff running as well. So if we take out those operators which are operated by another team, we empower the people again who own the Clusters. So of course, again, I said it in the beginning, there are operators which run locally and they make total sense. I think we, just as the ecosystem, have built too many operators which are just glue.

A

You remember this adapter chain in the beginning. We have done too much of that.

B

Is there a question over there? Sorry.

A

E

Thank you, I saw the the diagram and I was trying to make sense of it quickly. Over on the service provider's side, you had some I think red lines, delimiting what you called cluster.

A

Yeah this one, maybe that's a hard isolation and I had another one yeah somewhere I, don't remember. Yeah.

E

This one is there, um so is it the case that uh the modeling of this is that for each cluster in which there's a consumer, maybe.

A

I chose a data structure thing, that's, maybe better and just very quickly people are they want to go to lunch anyway. So there's a cluster namespace, so one per cluster and there are more namespaces like it's the one at the bottom here which are per namespace on that cluster.

A

Here we in the example backend we just prefix um I mean it's a lower name space with the class Dynamics based name, but this is totally up to the implementation of the backend. We use namespace acceleration here, because that's the only thing in Cube. We have right.

E

Right, so the service provider would have to think through how many clusters and.

A

Of course, you can also Imagine you use this model in Cube. You don't use kcp kcp, of course, is much more advanced, but you can even have multiple service clusters like a database team can have 17 class uh clusters running, and this back end will Point by capacity balancing in some sense will point the one cluster on this on this service cluster and the other one to another and distributes a load.

E

G

Thank you for the talk. My question regards to the um you. You can specify what about permissions. You you want, you need, and it I think it's pretty common, that a service is created and you have to read or list the.

H

G

Definitions and write to the status, so is there any default setting for this yeah.

A

So actually airbag does not play a bit. Airbag doesn't play a big role here so as a model about permission claims in the beginning. The idea is that this agent thing this connector. This basically offers just a view necessary, and this is not airbag right. It's much more powerful, a more, not powerful, let's say more specialized for this use case of a service provider. If you just offer The View into the cluster that is necessary, it's not airbag on the cluster. You can use airbag as well to limit access. Yes,.

G

But my question was: if I need this powerful mechanism or isn't it? Isn't it standard that I just want to create a service for accessing the service and to read the custom recess definition and up to the update the status sub resource.

A

I mean, then you need something in the cluster. The service provider has no access to your cluster. That's very important. It's always call out like the connector connects to the service provider and there's no way back. Everything is built in this protocol. I can talk about details later on. If you like, um so it's it's made in a way it's secure. It's calling out, you can control airbag yourself, but you have this additional mechanism to protect uh against. Okay, I think what providers can do.

H

So um I I understand the like the sinking of the spec and the status and the crdcr, but I was a little confused about um how the controller is working. So maybe can you go back to that other picture.

A

H

The service provider side is like looking at the resources on the service provider side yeah.

A

H

A mongodb operator like creates deployments and and other resources.

B

Right to bring up mongodb and so.

H

I, how I I was confused about how the controller that's running on the service provider side? Does that part.

A

Yeah, it's creating it's unmodified resources. It's a stock, open source provider like it says Community operator for mongodb unmodified. It will seize a mongodb community object, that's an API as they create it, sees that and does its job. It does not seem special.

H

But like, how does it create the mongodb like deployments and.

A

Other resources on the clients? No, no! It's not on the client cluster, it's software as a service. It's running on the service provider, cluster! It's nothing in the client cluster! One.

H

A

Think about permission claims to create a deployment. One could think about those models as well. It's much more complicated. This is also possible in theory, with a model, but the variable position is about software as a service I.

H

See so when the client went to like use, mongodb like it just connects using the mongodb, client or yeah.

A

The normal user, 27 000 17 or something.

H

Right, okay, got it got it. That makes sense thanks.

B

Probably time for one more question, then yeah.

I

So how does uh credential or generate like a secret some something that have to happen on the customers, uh the consumer side, cluster, in order for the application to conduct to in order to consume the service that provisioned in a service provider? So is that which component actually put a piece of information? Lots.

A

About this permission system you implement, like maybe their service accounts around right, you, maybe you as an admin you buy into the API, but this application using that I will see, for example, use a token or something. This is completely part of API modeling. What you're exporting that's not solved, but you bind, we don't dictate any authentication model at all for the services we export. That's completely part of the apis.

A

You export, if you have something like service accounts and service account objects, for example, export them maximum available to the user and just gives them a nice API to give the right identity to bees.

A

So it's API modeling, which must happen there of the service provider.

I

So I I can't really get that so in terms of a mongodb. So what is the credential? Actually, in the end, we provision.

A

I

Service, then, how do you connect it? You need a piece.

A

There's a secret traded like in the beginning, I've shown with the permission claims there's a secret in your name in your.

I

Area so, which component actually made that happen? Because.

A

That's the service provider. The service provider will get access to sync, a secret to create a secret on your site, but just this one secret of a special name, nothing else.

I

That happens on the client. I mean the consumer side cluster right. So the connector does that or the.

A

Connector just sinks: it will make sure that the secret the provider has created gets into the.

I

Consumer cluster, okay, so that's the connector, so the connector sinks. Yes, the secrets.

A

I

Yes, the service.

A

Provides into the consumer side, yes and the connector is open source. When they're neutral everybody can see what it does right, that's powerful! You can it's a community component, everybody can review and audit and everything. So it's safe, like it's a secret, really lands on your side and it's safe. It cannot do anything else and creating the secret.

A

All right, thank you time for lunch.