Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tweezering Kubernetes Resources: Operating on Operators - Kevin Ward, ControlPlane

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tweezering Kubernetes Resources: Operating on Operators - Kevin Ward, ControlPlane

Operators have become prevalent for the automation of repeatable cluster operations, replacing engineers in the Kubernetes configuration process. Although removing human error from the equation solves repeatability issues, Operators are often highly privileged with namespace or cluster-wide access to change resources. A compromised operator allows an attacker to deploy custom workloads very discreetly, and a rogue resource could go completely undetected. This talk asks and shows “what’s the worst that could happen?” to Operators by: - showing you how to threat model core Operator functionality - demonstrating how an Operator-based attacker can modify resources and gain persistence - how to securely appraise and test third-party Operators before trusting them - what to look out for during a code review or security related events.

A

Welcome to my talk today this is tweezing kubernetes resources operating on operators. So what we're going to talk about today, um primarily kubernetes operator security, um I'm going to briefly go over a little bit about kubernetes operators. Talk about security, I'm also going to talk about what does the operator introduce into kubernetes? um How can that be abused by an attacker and also?

A

How can I now review that, from a security perspective to see whether there's any kind of risks associated with it and then, lastly, um how can I detect operator abuse so I'm kevin ward, I'm from control plane, I'm a senior security engineer. I've got over a decade of experience, working in the security domain, doing all sorts of things from threat modeling to security architecture.

A

I have also do pen testing, where I can. My mantra is I like to harden by day, because primarily I'm working on securing stuff and then by night I like to sit in the dark and hack stuff and see how I can exploit it.

A

So essentially, kubernetes operator is an extension of the kubernetes api, with some operational knowledge that operational knowledge is provided by custom resource definitions which extend the capabilities of kubernetes beyond its initial primitives.

A

There's a controller that reconciles those resources for you um utilizing all the magic that kubernetes has. These are primarily used or often a use case, for it is stateful resolution, so you might have a database that you wish to update. You may want to drain the data out of it. You may want to rehydrate it. This is usually a very good pattern to try and keeping it in sync with your cluster.

A

There are some operator tools that exist out there, primarily through the operator framework, which offers several little things that you can do. The operator sdk provides a scaffold which automatically generates code for you, and you can bootstrap a project really nice and fast. So it's super useful for developers.

A

The operator lifecycle manager, essentially installs, updates and manages your operator for your cluster.

A

It does so via an operator bundle which consists of a cluster service version, which contains everything that an operator needs pretty much all bundled up by a little bit of custom resource definitions.

A

Interestingly enough, the actual olm, it consists of two operators in itself, so there's an it's an olm operator and there's also a catalog operator last on the list is operator hub, which is essentially a just a community hub where you can submit and share your operators.

A

So what does an operator introduce? Well, we've talked about custom resource definitions and what they do, but they can basically be anything that you would want them to be for to run your cluster to provide the operational knowledge.

A

They also also introduces a custom controller, which essentially is just the container that gets deployed normally in a dedicated namespace, with a dedicated service account, but this isn't mandatory.

A

So you can leverage existing resources if you wish to which is kind of a bit scary and we'll kind of go through that a little bit later and then, alongside all of that, depending on whatever you would like to do with your operator and how it's going to work, it could also introduce further kubernetes resources and resolve those, but also extend further outside your cluster and actually start configuring cloud resources, which is quite scary, stuff.

A

Lastly, on the list, it usually introduces some logging and metrics as well just to give you some observability.

A

So what what can go wrong well from a key threats. Perspective service account permissions. Now. Operators are usually trying to do administrative tasks um and to do so, as you probably know, they're highly permissive or certainly have highly permissive service accounts. If the operator gets compromised, you can leverage those permissions to pretty much do quite a lot of stuff getting close to cluster admin from a from a deployment of the actual controller itself.

A

These can be a privileged container as well um just so they can access and reach re reach the resources that they need to, um and also they're just as susceptible as any other container or image to vulnerabilities or dependencies that would reside within it.

A

But I think the biggest thing from a workload perspective in kubernetes is the scope of an operator, because not only it could just be name space bound to a specific name space that you would want to resolve your resources of how you want to do your work or it can go completely cluster-wide or multi-cluster if you really want it to which again extends those permissions, but even further still it does like it can be externally bound. So it can start resolving those kind of resources.

A

Any of those permission sets if the operator gets compromised can really really take a attacker to the next level. So one of the things that I worked on was to try and understand the the attack pass through. So I decided to do a mitra attack-like matrix for operators, and this is now live on our repo. If you want to have a look at it, I've provide provided all the verbage assigned to it. I think significantly.

A

There is very little kind of access initial access for an operator, it's not normal for an operator to be kind of like public facing so normally to try and actually get into an operator. You would try and compromise the image either in the repository, or you would compromise some credentials, whether that be some crowd cloud credentials that you've managed to get or whether that's the cube config file that might reside on a developer's laptop.

A

Alongside some of those threats, I've also included stuff around olm, so olem has a mechanism to automatically install uh an operator as part of its updates. Now, if you don't have to validate that, so it could be anything so if you're, not in control of like the line of which that gets installed, you could just install something malicious without validation.

A

You could also manipulate the catalog also to point to an operator image repository. That's not supposed to be so. There are kind of little things that you need to kind of bear in mind when you're looking at it. The items listed in bold are essentially the uh the cloud resources so stuff that is external uh scoped.

A

So if you're doing a kind of risk analysis- and you want to look through this for an attack path, um if you're, if your operator isn't externally bound or accessing those resources, you can basically remove those off the threat matrix. So you can kind of get a better view of what your, what your risks are.

A

So, what's like a common path, look like well adversary could still some credentials and attain cluster access from there. You could enumerate the pods, maybe he notices that there's a an operator in there. um You could potentially exec if you, if you could and then maybe from there, you want to start enumerating what the service account can do. Maybe the name space is already in if it's in a dedicated one. Maybe you want to do it within the cube system?

A

Namespace, um basically from there, you can leverage the clusterrol bindings if it has those um to deploy another malicious container into so, for instance, in this instance cube system, and then you can start bit by bit start taking over other resources with like a privileged container granted, there's a lot of ifs and buts and other controls that you can apply within there.

A

So a more kind of stealthy attack path that you could take is actually replacing some of the codes for a operator within the image and inside, like a registry, so that could be an internal registry with a disgruntled employer. It could be like a supply chain type of attack.

A

Now the functionality of the operator will maybe remain the same, but what they could do, then, is install like a malicious car and that sidecar will then be deployed out alongside other containers and then intercept a load of the requests that come through to gather potentially sensitive information.

A

You can then, potentially then extra trade that data to an adversely controlled cloud account.

A

There are several cves you can kind of find if you look for them around operators. I think significantly they're, not always just about the operator, though, so there are things that are deployed alongside the operator that affect it and how it's done. You can see that with, for instance, the capsule proxy.

A

Essentially, the vulnerability is within there because the proxy is running as cluster admin. It gave you access then further on to the operator and so forth, but yeah. It just gives you an idea that operators themselves are also vulnerable, they're, not immune to this.

A

So how bad is it? Well, I decided that I would download and have a look at every single operator that exists on operator hub and assess it for what I would consider key threats. So what are the service accounts that have been deployed? um What sorry, what the service cam permissions? What deployed security contacts? Have they got applied and also look at potentially sensitive cluster role bindings? You know like cluster admin and then, whether they're deploying in a separate name space.

A

Now this is for security context. Red is bad, so red is basically saying for that specific security context that has not been applied to an operator um yeah, it's pretty damning and I'll go through the statistics of the overall. How many have not been set the purple lines? Well, the blue lines are obviously that they are set, but the purple lines are the actually the opposite of what you might want. So that is, if you have privileged escalation false you would you would be sending it to true.

A

That is quite scary, but at the same point sometimes operators are doing some quite permissive things to do so, for instance, it might be accessing and setting up a container storage interface, for instance,.

A

So next up on cluster roll um yeah, there were a lot that had cluster role permissions. um You can see that there were a few that had, I would say, cluster admin or cluster admin equivalent. That is, I have access to every single resource and every single uh kubernetes api.

A

Alongside that, surprisingly, there's a about 10 across the board would have bind or escalate or impersonate.

A

So the breakdown of this essentially is like 90. Had a dedicated, namespace, really cool, that's excellent. The only issue is that 84 had cluster roles, um so it kind of almost defeats the point of having that dedicated namespace, somewhat um 64 percent uh had access to secrets and 70 were able to execute into pods 58. Only 58 um did not use 58 didn't use security context and only only 10 of the the ones that were listed on there were dropping linux capabilities.

A

For what it's worth um the roles, I had a quick look at some of the roles as well that that were ones that only had roles defined for them. 95 of them had access to secrets and 72 percent of them could exec into pods.

A

So how do we start securing an operator.

A

Well, the cncf working group for operators has produced a really lovely paper. If none of you have read it, I would highly recommend it. It goes through everything. You would pretty much need to know all about operators. It goes through the breakdown of what an operator is. It talks through control, loops talks through the actual controllers and custom controllers and what they mean. It also talks about common patterns and operate the operator frameworks and, obviously, security as well to some some degree.

A

um Google cloud have also got an excellent blog post about all the best practices that you would follow for uh for operators.

A

That's kind of things like use, a single controller per application if you can use declarative apis and then also give some advice about how to set up logging monitoring for those applications.

A

But essentially what does the the cncs say for security.

A

Well, the the first and quite big tribe in there is about transparency and documentation. When you're developing an operator, it's essential that you, you define exactly what the thing is doing and what you're trying to achieve. In that sense, you can then start breaking down what permission sets you need, whether it needs to be privileged or how the thing needs to run, whether it needs to be externally soaked or so forth.

A

So that's a very important thing for for for an operator to do beyond that start, as I just stated start defining that scope. So they talk about cluster-wide, external and namespace, so cluster-wide you have access to all resources that you need external cloud resources and then name space.

A

They try to. They basically say restrict our back permissions as much as possible and and certainly for cluster roles, only use, if absolutely necessary. Now, we've seen from operator hub, which doesn't contain just all the operators on the planet that that is, that is not true and quite a lot of being set to to cluster role permissions.

A

The other thing to bear in mind, if you think, if your operator does have access to external resources, there will be some cloud im associated with it and you're going to need to review that as well.

A

um They state that you should leverage se linux app armor. Second profiles. I think this is quite well known across the industry and as always, scan for vulnerabilities and consider supply chain security. um I'm not going to go anywhere near that there's about a dozen of talks, probably this year about supply chain security from prevention.

A

From a prevention strategy perspective, um I decided to start playing with and trying to build like an operator one of the things that was quite surprising to me, based on some of the research that I've just just done, was that for 1.18.1 um there is a the the open sdk operator. Sdk gives you two security contacts out the box as part of the manifest. So if you see a developer who's removed that you kind of have to kind of question, why somewhat um it may be bound to like the way that the thing works.

A

But I think you should really question that my advice is just be explicit, so be very careful and be very explicit about the verbs you want to use the resources you want to access and the apis. You want to access so try not to do star star star permissions. Try not to do core api star star.

A

That is just to be quite lazy.

A

The only issue is that an operator might actually require that as part of its work, but I do question that somewhat especially when you get into around our back api, because that's where, if you buy a star, then you're starting to define the escalate privileges, so last kind of bit of advice is start to work with the developers to scope out your operator start restricting the operator to the name spaces that it's supposed to have, but not only the sorry, the name space is supposed to be deployed to, but also restrict it to the what it can watch as well.

A

Just slightly different review the cluster role permissions and make sure they're not overly permissive and then also um for do the same for cloud iom and the same for to make sure that a role is defined. If you only got a namespace operator, so wouldn't it be nice if we could do this in a kind of automated fashion, um well, control plane have made a static analyzer for operator manifests. This is based off a lot of the kind of risks and threat modelling that I've been doing primarily at the moment.

A

It's only focused on security context, cluster role, permissions and initial name, space use and when I say initial name space just to be clear, that's default and cube system of whether it's just been deployed into there.

A

It's demo time right, so, let's bring so oh that is very zoomed.

A

Can everyone read that okay, so this is this is a bad robot and within bad robot you'll find all the rule sets and the rationale behind the rule sets in there. um So straight out, the back you'll start seeing things like whether it's being deployed or using like default namespaces, whether it's got a security context set where the security contexts are actually configured the other way around.

A

uh And these are probably like the primary three things I would recommend potentially blocking a build on um so runs this cluster admin nah. You should be using a dedicated cluster cluster role and defining all the permissions for it.

A

Cluster role with full permissions is just basically the same, so make sure well kind of listen con isn't, but you do need to make sure you kind of block that as well and ask the developer whoever's building this to redo it again and I believe, the same for the core api resources just because they're so sensitive beyond that. I look for things like it has access to both cluster roles and cluster or bindings in terms of entirety secrets we can zec into pods what else we got in here.

A

Removing events, custom resource is a really interesting one. You would imagine that it probably needs custom resources, but you should be declaring the custom resources that it should have access to not every single custom resource across the entire cluster. I think that's a bit scary for me, so let's just do a quick demo.

A

If I can bring my terminal up, it's pretty simple. um You just do bad robot scan and then I've actually taken uh three manifests that are based on manifests from operator hub. I haven't normalized who they are, because I don't want to cause problems with people and start having fingers pointed at me.

A

But if we look at, for instance, demo one wow, that is quite big, maybe we need some color as well.

A

Right so starting from the top we've got some scoring here where basically says that no security context has been set, we have we've got impersonate defined within the cluster role. We have a full access to admission um secrets and the school kind of designates kind of what I believe is the severity to it. So some things around like events and modifying logs are kind of low.

A

So let's try the next one.

A

What have we got here ah so in this one? um It's actually found that it's been deployed into cube system.

A

um We have also got some stuff up here: yep cube system, no security context set and yep we've got secrets, uh access to secrets and cluster roles and cluster world bindings.

A

So for the final one I don't want to do that. What do we got bear in mind? These are based off actual operator hub ones. I just want to just point that out we have now got no security context. Set is running with cluster admin. I think somewhere yeah it's on the default namespace and it's got star rule cluster role. So pretty scary.

A

So what about other threats, because bad robot can only go so far right? So what it's not doing? Is it's not looking at malicious kind of operator code, I.e code, that's been modified by a adversary, somewhat and kind of hiding it within there.

A

It's not it's not looking at, certainly the repository of which an operator lives in if it's public and whether it's been overtaken um it's not looking at whether binaries that have been included in it's non-minimal, so things like shell, like pretty much, I I be questioning whether there should be a shell in it at all, but there are some that do.

A

What else is well, can you think about um whether, like um whether you want to draw like an operator internally and then maybe it's modified like local and then just deployed? You need to come some. Maybe some checks to consider around that to maybe make it more permissive. So you have some initial trust because it's come from like a reputable source, but then someone in between reconfigured it so what's kind of getting to the point is we need to operate pipeline?

A

um I wouldn't say that this is like the de facto, because there's so many different things now you can do in a pipeline, including s-bombs and so forth, but the idea is that provide some provide some developers. Some are back guidance.

A

You should review the operator code itself to just make sure it's doing what it's supposed to be doing.

A

Maybe then look at using something like bad robot to scan the manifest make sure you do some vulnerability scanning, there's loads of tools that can do that against your container and then finally, really validate those that container in a test environment to make sure it's doing what you expect it to do.

A

So what have we got for a detection strategy now to me the way I view it is that an operator is essentially an automated runbook.

A

So as part of that, you know exactly what processes and procedures it's going to run, they're going to generate some events and as part of those events, you can have like a set load of logs and events to look for when you get those deviations away from what that operator is supposed to do. That's when you might want to investigate and flag now, that's not perfect, because the operator itself might be updated.

A

So you need to make sure that when the operator is updated, the functionality hasn't deviated too much that you get load of false positives.

A

It also could just be doing bad stuff anyway, so you might not detect it. So, for instance, you know deploying resources that are intentionally vulnerable or it deploys misconfigured resources as well, so making them, maybe public accessible or something on those lines.

A

So for the um for the common attack path um that I talked about earlier, um I considered: what would a cloud provider get you out the box? Would it be able to detect those different things that are occurring now, the kubernetes api event logs? Are your friend at this point and essentially, when I did an initial exec, you can see that it found that I was doing that event nicely down there, but when I pulled down qctl um and put it in the bin directory, um it didn't detect it.

A

I just want to add as well, by the way it's running as root, and this is an actual operator that I pulled um off operator hub and deployed. uh It's not detecting me doing a. Can I, on the on the actual kubernetes cube system namespace, so that's not detected and, as you can see, it's found some quite permissive settings cluster wide, but it will detect me downloading and trying to deploy uh another malicious container.

A

um As will it then know when I've actually tried to pivot to it as well. So there are a few gaps, so we probably want to think about maybe having some enhancements to our logs. Now, unfortunately, there's you know there are some stuff that the clouds provide. um I have not seen too much from aws uh guard duty does kind of limited stuff.

A

um Gcp have now uh enabled something called container threat detection, and that will let you know when uh it will alert on when a binary is added or been executed, when a malicious script's been executed or there's been a reverse shell.

A

Recently, microsoft have released defender for containers. I'll be honest, I haven't had chance to play with it quite yet, but it boasts abnormal kubernetes service, account, operation, detection and command within the container running with high privileges and also detects suspicious file downloads.

A

Beyond all of that, we obviously know that there are third-party solutions that can help us along the way, such as cystic aquasec and obviously twist lock or formerly triscot. So what does the future lie for operators?

A

So when you try and build an operator and upload it up into operator hub, the the operator goes through essentially a score card, and this shouldn't be mixed up with your uh open ssf school card. It's primarily just acceptance testing just to make sure the operator just works.

A

There is a section where you can write some custom tests and kind of what I would like to see is some just notional security tests run alongside that before you can make entry into the operator hub just so that by default, some really bad settings are not set.

A

um The operator white paper talks about work being done around dynamic access for operators, and that is essentially privilege, escalation, sorry elevating privileges uh to perform sensitive operations, so you can imagine they want to maybe set up the custom resource that they need to, but when they're performing the reconciliation and watching it might er lower those permission sets to just do that function.

A

When you have that type of control, you can then start applying a policy engine where you can then see what it's, what permission sets it's trying to do and whether it's trying to access other resources that it's not supposed to, but one of the things I quite like the sound of um which is a very old security technique, which is on anomaly-based detection. The operators are just basically doing this run book that I've talked about.

A

It would be really nice if we could just build and train a detection engine to determine whether it's doing the things that it's supposed to do the caveat is you need to know that all the full, the functionality of that operator and any deviations for it, and then that goes for the updates for it as well.

A

So, basically, in conclusion, operators can do just as much damage as any other kubernetes resource. You should really, when you're building an operator define the core functionality of that operator. So you make sure that that operator is only doing the things that it's supposed to do and you can restrict it down to the stuff that you would do.

A

You should review the operator for scope and permissions just to make sure it's bound to what it's supposed to have, and I personally think you should block deployments for default operator permissions as I've kind of indicated, make sure you apply your security limits modules where possible and profile your operator with logs and metrics, and alert on any deviations.

A

A

I realize I'm way within time, um so if anyone's got any questions, they want to ask. If anyone wants to talk to me or ask anything, please do.

A

B

Okay, um so my question is: do you think that the security issues within operators that are introduced upon building your operator are due to um operate a framework such as um like, basically, the people uses like templating to build their own operator? So, for example, last year I built a crossplane provider which works similar to an operator, and we built that operator based on that, like that provider based on the existing providers, and it was very difficult to understand what was going on. Where are things supposed to be?

B

How does it work, so we just made it work based on existing kind of templates. um Do you think that's one of the main issues, and do you think that other people working on that on kind of.

A

Yeah, I mean it's a very good question. My feelings are that the operator framework is is really good because it actually does provide some level of guidance for you. Does it do everything out the box? I mean when you bootstrap from the operator sdk. There is no service account there and there's no kind of guidance for that. um Potentially there maybe should be some more um that to help people along the way of like what to know to do.

A

um Maybe that that's kind of the next thing, but I I don't want to kind of force the it's so essentially what the operator framework is doing is excellent and they're doing it to a level that enables people to just download and start building operators. If you start being a little bit more, uh have a bit more opinion over those operators itself, you can start restricting people from what beyond what they're trying to achieve.

A

So it's a hard balance between how much do you actually restrict someone doing to them them feeling that they can't achieve what they've got to do, because they've got fulfill a certain criteria. So it's a hard balance, but um I think more, maybe more guidance around service accounts as needed. I'm not that's! That's my best way I can kind of describe it and and how operators should be built. um There's a lot of scaffolding that the open, sdk kind of gives you. So it's it's good.

A

In that sense, it gives you kind of what you need, but from a security perspective, you really need to think about the design of like what it's going to do and maybe even apply some threat modeling to kind of understand like what. If what? If, um if I start applying these permissions, what can basically go wrong and that's usually the best scenario to kind of start, weeding out the details.

C

Hey yeah great talk, um I've got a couple of questions actually one of them there you mentioned about uh dropping permissions and I think that's a really interesting concept. You know of any any ways that that kind of can be done today, because I don't think that's something simple. You either need another operator with more permissions to manage a set of operators and then you've got like an nginx style work process thing I see you smiling and looking at.

A

Yeah now it really is excellent. It's the kind of thing I think we need to start moving towards and maybe not even be beyond even operators, but um the the white paper is quite vague about it, and I tried to do a load of research. It was like well what is this research going on and I couldn't find anything. uh It's obviously something going behind the scenes, potentially with the operator framework or something they're looking at um but yeah.

A

It is going to be challenging to do because you need essentially an entire. I think an entire system to kind of then start managing all of that, and then you probably want to run some policy against that as well just to validate and make sure it's just doing what it's supposed to do. So it's really it's really tricky, but I I I'll be honest with you. I did some really like it was. This was just based off the white paper.

A

They were saying that work is being done and I took I mean you know I they can't publish that without having some kind of ratification. I would I would assume I would assume, um but yeah that I it definitely is for me kind of like next stages of like what we can do and starting to try and tighten these things down. Yeah.

C

And then my second one is the the analysis that you're doing there and like actually looking at you know the far-reaching permissions that some have and stuff. I think that's super valuable. I wonder I think, when it comes to people looking at something like operator hub or anywhere else, the visibility of that kind of information is lacking um and not specifically pointing at operator hub. Here I mean universally.

C

um We it's not particularly clear to an end user. What permissions you're granting people do you know of anything where, like either work on going or tooling? I guess bad robot is one bit of tooling for it, but making this a bit more prominent, saying like highlighting the risks to people as they go to install things or before they do.

A

Yeah I um so one of the things that we talked. We were having a conundrum over where whether we want to assess the permission sets of something that's deployed, because that's actual against something that's in a manifest that is, could be um the the issue that we have is that we have some clients who just actually can't even deploy without having some level of security assessment done. So that's why we kind of went to the manifest kind of side of it. um There is more stuff being done in this field.

A

I know that um I think security are going to put some more guidelines around what the our back permissions and specifically ones that are really kind of dangerous to use but yeah. For me, it goes quite similar to anything in kubernetes only that operators are hugely permissive and they can then do so much damage yeah.

C

Yeah exactly thank you all right.

A

D

Hey uh so, at the end of the day, an operator is really just another controller. So all the concerns you have here are: aren't they equally applicable to the core kubernetes controllers themselves? uh Do you have any insight into how locked down those are? Aren't.

A

uh I can't say that I investigated too heavily into the existing controllers. um I know that they're locked down a lot more than what these you're open just to configure this controller to do whatever you want. So that's like that's the more like, whilst I would say, kubernetes controller is bound to kubernetes primitives and what it needs to reconcile and work on.

A

You are right in saying that it's just another controller, but it's another controller you can you put in and you set the configuration for and you have and all the image that goes into it and things like that.

A

You, especially from a cloud provider perspective you just that just automatically underneath the hood right, they can take care of your control, plane and all the components within it whilst like with this you're, deploying it into a namespace within your cluster, and that is then usually accessible, and can you can do some basically some naughty stuff with it.

D

Okay, cool thanks, no worries.

E

Thanks for that talk, um you mentioned a lot about the permissions that operators have, and the example you pointed out was specifically reading, writing secrets and a lot of namespaces and and that I agree. That sounds very scary in the beginning, but if you think about an operator like a populator that provides a postgres as a managed, application will probably generate credentials for that database for you. So it's usable and store that in a secret and also needs to be able to read it.

E

So it's kind of also like a regular use case right and you kind of have to give the operator permissions in a lot of namespaces, because you don't know where it's going to be used right. If you have a multi-tenant platform, you can't just list the three namespaces ahead of time. You will always be like adding stuff to it.

E

So I'm wondering have you maybe considered that this is potentially an area where we need to rethink a little bit about the rbx system kubernetes itself, so it has a concept of identity like in the linux file system. I can write too many directories, but that doesn't mean just because I can write a file in a directory. I can read everyone's else's files. You can just read the stuff that maybe I have created.

E

So I'm wondering like. Is this maybe a point where we need to kind of extend the rbx system a little bit to be more granular, because right now there aren't a lot of options. You can like say you have our back on a particular object by name, but the secret would probably have some auto-generated name. So you can't, like you know, predict that, so there aren't really a lot of options for developers really to lock it down as much as we would really like to have.

A

Yeah, I I definitely would think so, um there's always more stuff. We can do to help, but there's always the the balance between. Do we make it too granular and really hard, or do we and and then it becomes really unwieldy for a developer to to actually configure or an engineer to configure.

A

So it's it's a fine balance. Basically, I take your point about secrets is why the score is not that high, because I understand that there are use cases for that, but there are risks associated, especially if you're going across cluster wide. Some of the patterns that they that you can put out is just have essentially a core operator manager to manage. Then multiple controllers across your cluster that focus on specific things that they're going to run and they can run maybe against specific namespaces as well.

A

I haven't kind of gone into like how I would then do. I guess, toxic combinations of seeing if this has access to that, and that has access to that those two together. Can I then do something: that's kind of a little bit beyond what I've. What I've gone through. I've mainly focused on that operator that is controlling the the first thing that's deployed and building out.

A

So there are different patterns that you can use to potentially break down operator permissions and what it's focused on, um but I don't know from an our back perspective like how we could go a little, maybe a little bit further with doing that.

A

Potentially you want to maybe focus more on roles and then start having multiple roles that are then pointing rather than I have a cluster role, and then it can just do all these different things.

A

Any more questions, I think, that's it. Thank you. Everyone.