Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Implementing Anti-patterns: Kubernetes Cross-namespace Resource Ownership - Tom Coufal, Red Hat

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Implementing Anti-patterns: Kubernetes Cross-namespace Resource Ownership - Tom Coufal, Red Hat

Kubernetes is a very open system allowing developers a great extent of freedom. However it still follows rules, design principles and sets well-described boundaries. It constrains developers to well understood cans and cants. Prohibiting cross-namespace ownership of resources is one of such rules. A namespace scoped resource can't be an owner to a resource in a different namespace. Yet, there always is that one use case that we can't solve in any other way than by smashing through those walls of rules. We will explore why such a use case makes sense. And what challenges rigid namespace isolation brings to integration between Kubernetes native services. We will discover how one can construct a solution to this problem. Without re-implementing or replacing default core services - like the native garbage collection mechanisms.

A

Hello, can you hear me good um hello? Everybody welcome to implementing antipatterns aka implementing cross namespace resource ownership in kubernetes. uh I hope I won't disappoint you uh with this talk. I know I will disappoint you with it being an anti-pattern and implementing it in kubernetes and I hope you're not too disappointed by that. So what we're gonna do today, um I'm gonna talk a bit about some kubernetes concepts um that we usually use and we usually see like ownerships dependents, name, spaces.

A

Garbage collection and the rules that goes with those concepts, then we're gonna, try to exploit and implement and do some forbidden stuff that kubernetes disallow in the documentation and in the implementation as well, and we're gonna try to do so without smashing through the walls of kubernetes implementation. We're gonna bend the rules, so it fits our uh our very anti-patternish wrong use case that I'm gonna in the meantime try to justify to you. uh So you see that we really need to do something like that.

A

So first things first, uh who am I uh my name- is tom tofal. I work as a software engineer at red hat at the office of the cto in emerging technologies. What do we do? We research various topics, we experiment with managed services, open services and things like that, and one of our key area of focus is analytics data science, uh oriented workloads and uh things like that. So that's where this use case originated from and I'm gonna talk to that a bit later. So what I'm gonna be talking about? There's a very nice coincidence.

A

uh Yesterday there was this book signing event at the redhead booth. I didn't know about that before beforehand, so I've added the slide in the last minute about kubernetes pattern patterns book uh by colleagues of mine at redhead. Honestly, I haven't read this book.

A

If those offers are here, I'm sorry, I'm sure it's a very nice book uh and since I haven't tried it, uh I'm gonna be doing anti-patterns, so we're gonna shoot the deck and uh we're gonna smash through it. No, what just happened.

A

Good, so how we're gonna do it? This is the agenda as usual. First we're gonna start with introducing some concepts to you uh going through the most basic stuff for most of you, maybe, but also going to some some more advanced things. uh So hopefully uh it's not just a recap, and we will get some some new connections between those concepts.

A

Then I'm gonna provoke you with the use case um and then we're gonna explore only one solution, because we don't have time for more and really uh we've come up with this one solution, so we implemented it and we use that one solution.

A

We can save that for the later q, a or other discussions uh with beer, or something why this implementation may be wrong or how to do it properly.

A

Then, as a fourth point, we're going to implement it, we're going to see how much effort is it to implement something like this and last but not least, if the time permits we're going to do a live demo with our project that is running somewhere in the cloud. So hopefully, networks will be kind with us and it will work so first concept that I would like to introduce to. You is hopefully not very new to you how many of you here intentionally uses kubernetes namespaces on their clusters.

A

Great everybody knows what the namespace is awesome, so just to recap what kubernetes documentation says about namespaces: it serves for isolating groups of resources and then the recommendation talks about namespace-based, coping and cluster-wise resources. So that gives us some information that there are two distinct types of resources in kubernetes, that we have something namespace scope and then we have something that is shared across the whole cluster.

A

This is very important uh thing to remember that we can use two types of resources and there are a lot of things that comes with with those different types: there's security and arbeit constraints that comes with cluster wide resources, that out of blue. You have access to all uh namespaces in a cluster.

A

On the other hand, if I have a namespace based resource, I can have a resource with the same name in a different namespace, and now I have with two people talking about the same resource with the same resource name in two different namespaces. They are talking about two different stuff.

A

So that's also an interesting topic or interesting point to remember that we have two two things uh happening there when to use namespaces. Well, usually in environments with many users or distinct workloads that I want to separate and isolate right, common knowledge um so far, so good, so another point here is that each kubernetes resource, if it's name space scope, is, can live only in one namespace, so we can't have a resource spanning multiple namespaces.

A

It can be either cluster scoped or a single namespace good and last but not least, what kubernetes documentation talk uh says about namespaces is something called resource quota. This is a very interesting and important concept with namespaces that comes with namespaces, so resource quota.

A

Again, I'm quoting the documentation. This is something I've learned from implementing this thing: read the documentation, that's what we usually don't do when we're hacking stuff and putting things together. So that's why I'm quoting on the second slide, I'm again, quoting kubernetes documentation so resource quota.

A

What does it serve? What purpose does it serve? Well, we want to ensure that each distinct user group each distant namespace in cluster gets its fair share of resources that we can't have a single namespace, consuming all resources in the cluster eating it all and there's no resources left for another team. So we can set some boundaries upper boundaries.

A

What's the maximum amount of resources, one team one namespace can consume.

A

So that brings us to multi-tenancy.

A

I don't know about about your use cases of kubernetes. Our use case is mainly running long lasting clusters, with huge footprint, tens of nodes, even better, even even more and hosting multiple teams that we want to keep isolated on those clusters.

A

So we have user team, a and team of users named b and many others, and each of them have access to multiple namespaces or can be one can be more. It can be tense um based on their need, and we want to isolate those. So we do that through namespaces. We do that through resource quota we did through some are back on top of it.

A

Switching gears owners and dependents one last one last topic to introduce to you before we jump into into the real stuff, so owners and dependents also a very well-known kubernetes concept right, the most obvious chain of owners and dependents that everybody uses is a deployment replica asset port. So deployment owns replica asset replicas. It owns a port if I delete deployment replicas that gets deleted, pods get deleted very, very common, very common workflow, and this is something that is used across kubernetes in many various formats and ways.

A

So this is how it looks on on the api side of things. On the manifest side of things, you usually don't define a manifest for replica asset. You usually don't define a manifest for pod, so these are usually very hidden from the end user perspective. They create a deployment manifest and are happy with it. Everything else is orchestrated by kubernetes for them, but these only references are still present in those yaml files. Yaml manifests json manifests in the cluster once deployed, so I can take a look into a replica set and see.

A

There's a metadata field called own references that is referencing. The deployment- and it contains identifier, information- it contains unique id kind name, api version, but also two other flags that are used for uh for additional control additional stuff. On top of it. So there's a flag setting a controller setting which resource actually controls the reso the dependent, because you can have multiple owners for a single resource, so you want to avoid them fighting over this resource.

A

So you have this control flag that uh it can be set only in a single own reference. Then you have a flag for block owner deletion. So this is telling the garbage collector how to behave when this resource is being deleted or when the owner is being deleted and how to how to handle those situations.

A

Then we have another concept that is kind of working hand-in-hand with owners and dependents. That's finalizers, so you don't have to specify an owner or a dependent in finalizers, but finalizer needs to be clever enough to determine these relations.

A

Usually you see that in persistent volumes and persistent volume claims where you can actually start deleting a persistent volume claim or persistent volume.

A

Even if it's mounted to a port, the port, the persistent volume per system volume claim will still stay in terminating state until the port gets deleted, and this is ensured by the finalizer, as you can see in the manifest for persistent volume, claim, there's no, no owner definition, no owner reference that this port owns this persistent volume claim, but still the relation is maintained.

A

So when it comes to regular owner owner references and ownership of resources in kubernetes, we can have these things right. This metrics I can have a cluster scoped owner again going back to namespaces and those resources, though two distinct types, that I've talked before, that I can have cluster scopes and names for scope resources, so I can have an owner which is a cluster-scoped resource, and I can own a cluster-scoped dependent again comparing apples to apples good. That's that's normal thing to do what I can also do from a cluster scoped resource.

A

I can own a namespace scope resource which we see as like a small subsets, narrowly scoped resource than the clusterwise one, okay good then I can I can have. I can't have a namespace scoped owner owning something on the cluster scope, that kind of breaks through these isolations right. That's also a very obvious thing to think to have, and then I can have in the top right corner. I can have both of them name space scope.

A

Well, can I.

A

It you can, but they need to live in the same name. Space.

A

Kubernetes will allow you to define the ownership reference to a resource living in a different namespace. It will allow it it will the api the admission control will accept it and then the the kubernetes controller will come in and start uh working on it. So what will actually happen?

A

Are these issues? What's going on with uh my resources? What's going on what's happening? I have I have a resource that is owned by something, and this resource is just out of blue deleted by the controller. What happened? Well, what happened is the controller goes into my resource that is owned by something goes through. The own references looks through the owner references and sees.

A

I have this namespace scoped own reference, and it doesn't exist.

A

Ok, there's out of sudden there's no owner and I'm a dependent resource, I'm a dependency of something I'm generated from something, so I'm ready to be garbage collected, I'm ready to be released and just delete this resource, so you define across namespace ownership, everything's great until there's time for the garbage collection to kick in and it just deletes your resource without any warning, because it feels that this is a legit thing to do. So.

A

If I go just really quick back here to those own references, you see no namespace field in it using name kind, uid and flags, so it really doesn't uh convey the information that there's something to be happening with namespaces.

A

So we get those issues um what's happening with my resource, my body getting deleted and also articles about. These are the lessons I've learned uh and by skimming uh kubernetes documentation, and these are the things I really shouldn't be doing. So beware: okay, since version 120 of kubernetes, we get this nice notice in the kubernetes recommendation.

A

Telling us don't do this. It must exist in the same namespace. Otherwise the garbage collection will kick in beware.

A

So this is what we're going to implement. Is this it's disallowed by design and we're going to try to bend kubernetes? Why? Well?

A

Okay, we have some extra stuff on the right that should be there. Well, we have this operator, that's what you see on the top and we have a cluster clusterscope operator, a controller, that's working on the clusterscope level, normal thing everybody does that right and we have namespaces.

A

We have users in different namespaces, so we deploy something some custom resource into various namespaces. This is called in our project. This is called meteor shower and this resource is holding some specific generic configuration for that particle instance of meteor shower deployment again. Imagine it as a kafka cluster custom resource or something like that.

A

We can generate additional resources from this meteor shower within the same name space. So there's a meteor resource which is part of meteor, shower it's owned by a meteor shower, and this is a very normal relationship again deployment replica set pot, similar relationship here I have a meteor shower a meteor derived from it and some workloads pods generated from it in the same name, space.

A

That is, that is fine. That is good. That is normal thing to do. Well, then, I have a problem that I want to solve, because for some reason I want to generate resources in different name spaces. I want this operator to be able to integrate with external services in different namespaces.

A

Well, why? In our scenario, this is a data science project that needs to integrate with legacy additional applications deployed to different namespaces, and I want those namespaces with resource quota. I want it self-contained and isolated, so we have one namespace where we are running: build pipelines for uh data science repositories- and this is this- should have different quota than deployment of some data science workload.

A

It should live in a different namespace and I also want to support multitenancy. I want to support multiple instances, because I have multiple teams on my cluster, so I can't have my shower instance cluster scoped, because I want more instances of instances of it that would solve the issue right. I would have cluster scope owner of all the resources.

A

Fine, I can't have that. Well, what do I do? What do I do now? I can't just set the ownership reference to that other namespace. That would just delete my resources. Why would I want to search ownership references?

A

Well, this is simple. We don't want to leave garbage. We don't want to leave mass after ourselves, so if user leaves and deletes their workload deletes their meteor, I want to be able to garbage collect from other namespaces. I don't want to leave images and image streams which we know in openshift left in those other namespaces.

A

So I want to be a good steward, a good guy and clean the cluster after my usage. So what do I do? Well, the solution we came up with is to provide a resource called meteor, comma, which is a mere shadow of my meteor in those other namespaces.

A

Then I can define the ownership reference as a normal thing within a single namespace, so meteor, comma in namespace, f, owns all all the parts in namespace f, and this is a normal, normal regulation.

A

Now, instead of syncing ownership references across multiple namespaces, I just need to ensure that I have a meteor and meteor commas side by side in the name space and I need to ensure that they both exist at the same time and I need to synchronize their creation, their deletion and passing of the ownership references fairly straightforward solution.

A

But it's an anti-pattern too.

A

This is the api or the resource definitions that we came with, came up with just custom resources, meteor and comma resources, and it defines couple of fields in it. So first thing is an owner reference in meteor.

A

That's the standard thing right. This is owned from a shower called parasites in the same name space, it's an ownership reference. So if I delete my meteor shower, I can be sure that this meteor gets deleted as well. So it's not left behind.

A

The other thing here is a finalizer, and this finalizer is here to ensure that if I'm deleting a meteor, I will also query or schedule for deletion. All the commas.

A

Then, in status I track all my commas in all the namespaces, all the external namespaces as something like an ownership reference which is understood only by our custom controller, but it's there and it has additional fear for for a namespace.

A

So um this way I know that there's this additional resource in the other namespace it's pointing to the comma, it has the same uid. It has the namespace name and kind and stuff like that.

A

So in my comma I also need to set a finalizer to ensure that the comma is not deleted until somebody deletes the meteor, because I don't want all the resources in the other name, space just disappear. uh If somebody's doing stuff in that namespace, I want to wait until they intentionally recur and just for good measure. I also track the owner in here in coma just for redundancy and to know where to look at if I'm doing something with the coma.

A

So we have this pattern of all referencing everywhere and uh so far so far it seems good. But how do I implement it? What do I do with it? Well, if you think of think about it, it's not that easy. It's not an easy problem to solve.

A

uh If you think of creating resources for you through google, apply or something like that and manually, creating all references and and getting those new ids and everything right.

A

It's it's not easy thing to do and if you do it wrong, the resources get deleted because they don't exist. So the cube controller is very strict about that. What we can do, what we can use is our goal line binding: let's go for the golang sdk and have fun with it and try to implement it through queue builder and operate it through operator sdk.

A

This is this was our solution, or this is our solution, how we approached it and it turned out fairly simple for such a complex problem. So first the synchronized, the finalizer. What do we do in the finalizer?

A

Well, it turns out that this is just a few lines of code. So if I have a, if I don't have a deletion timestamp, the resource is not being deleted, so I need to ensure the finalizer is properly set on those resources standard thing to do. I need to add it to the manifest. I need to update it, update the resource good and if the resource is being deleted.

A

I just query all right. I just schedule all the commas to be deleted as well and then, if they are deleted successfully, I just remove the finalizer, so the resource can be deleted. Well, what do I do when I want to delete all those commas again, very simple, very simple, single for loop.

A

I just list. I just walk all the commas that I have in my status and just schedule them to be deleted.

A

Simple well, when it becomes more complex, is actually synchronizing the creation, but our operator acts watches the cr creation and acts upon it. What is meteor being created? So we just when we are creating meteor. We just list all the external service services. We need to integrate it and look for them and then it's all the same.

A

It's just do. I have a namespace for that external service. I do okay, so I get that namespace do I have access to the namespace? If I do, I can create comma in there, so I create a comma in there and I just do some on the right. I just do some extra magic with only references and setting up those names generate online references and set them for my tracking in the status field, which is again fairly simple so now.

A

This is all the implementation that I had to do. I've implemented the creation of all the commas across all the namespaces and I've handled all the deletion phases. This was all the code. This was all the implementation.

A

Well now I need to deploy and test and run it. So we use this community called the bridge. First, where you can get for community workloads, you can get free infrastructure and deploy their workloads in there and uh operate it and collaborate on operations and on sre related stuff, and things like that. So we use it as our uh as our uh go to go to community for for uh kubernetes and openshift and stuff like that, uh and uh we deployed it there.

A

So, let's take a look at it.

A

This is our meteor instance. It provides you a way how to get. This is unrelated to the dog, but this is the use case being explained. So we automate away from a data science type of kids, repository to images being deployed to jupiter, hub and other other other ways. So we get some meteors in here that are being created and if you like, fill in this form pass the url that what happens, there's a meteor resource being created on a cluster.

A

So we can take a look. What actually happened in the cluster.

A

I have a meteor resource, as you can see, managed by perseids it's owned from a shower resource. I can go into the manifest and I can see that there's an own reference for per se. It's shower and there's some spec and has some commas in in its status. So it's integrating with some other name spaces. So I can query uh for well yeah. I can query for meteor and I can query for comma.

A

So I can see I have quite a few meteors in this namespace, and this is my meteor that we've been just looking at and I can take a look at a different namespace at the other namespace that I'm integrating with as the external service. Now I don't have any meteors here, but I have commas and they are named the same, so I can just for consistency. They are named the same, so comma really does nothing as a resource.

A

It's a it's a blank shadow of the meteor that we can use to own the additional generated resources and garbage collect them afterwards. So if I take a look here now into builds image streams in this namespace, I can query for this name.

A

There's an image stream, which is name which is owned by this coma, so this image stream, which is a internal openshift resource to hold images in their internal openshift image registry that is internal to the cluster. It has some tags, it has everything, but what's important here is there's an own reference that we've added to this local coma in that local namespace.

A

What we also have for this meteor in the other namespace we have built pipelines.

A

A

That are run against this meteor against this meteor spec. So again, it's owned managed by this meteor. So if I delete my meteor, which we can do.

A

If I delete my meteor, what should happen is that I'm going to duplicate this, so we can see it.

A

A

All these pipeline rounds and all these images, this image stream should be deleted and, as you can see, they are living in a different name space. So if I delete this meteor.

A

The controller yes, the controller should delete those pipelines and delete the image stream and just to be sure, we can take a look at this namespace and get those commas.

A

Yeah, I shouldn't be having it here: yeah, it's not there. The comma gets deleted as well, so.

A

It worked, it did its job and it was few lines of code. So if you have a complex use case, if you have something that is hard to solve well, if you think about it and go into q builder operator sdk and really try to implement it through the golang sdk, it's really easy to do to bend the kubernetes rules and to implement it's implement something that is really really disallowed and really advised not to do in 20 lines of code.

A

Thank you very much. If you have any questions now's the time.