Ceph Ceph Virtual 2022, 15 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Data Security and Storage Hardening In Rook and Ceph

Description

Presented by: Federico Lucifredi & Ana mcTaggart

Data Security and Storage Hardening In Rook and Ceph

We explore the security model exposed by Rook with Ceph, the leading software-defined storage platform of the Open Source world. Digging increasingly deeper in the stack, we examine hardening options for Ceph storage appropriate for a variety of threat profiles. Options include defining a threat model, limiting the blast radius of an attack by implementing separate security zones, the use of encryption at rest and in-flight and FIPS 140-2 validated ciphers, hardened builds and default configuration, as well as user access controls and key management. Data retention and secure deletion are also addressed. The very process of containerization creates additional security benefits with lightweight separation of domains. Rook makes the process of applying hardening options easier, as this becomes a matter of simply modifying a .yaml file with the appropriate security context upon creation, making it a snap to apply the standard hardening options of Ceph to a container-based storage system.

A

All righty, so hello, self-community nice, to join you virtually in this new event and thanks for to Mike for making this all happen, um our marketing manager won't let.

B

A

Away from the idea that we have to introduce ourselves so very briefly: I had the privilege of spending my entire career in free and open source software um I'm, the product management director for Obsession as a platform I'm red hat and soon in IBM. uh Previously it was Ubuntu Server pm at canonical. If you remember 1404, that was my favorite accomplishment there and if you go back the decade, uh the systems management car that cars hired everyone at Susan.

A

uh There are a few things here um on the list that I worked on. um I was the maintainer of Mount I, suppose I, technically still am also. We consider Collins version preferred these days, so I don't have that much to do, um but it's still there on your Mac. If you type man man, you find me, um Shameless plug I have a book by O'reilly and that's why you see me there surrounded by clouds in my author portrait and um well I. Think that's plenty enough for me. So Anna.

C

Hacker I use they them pronouns and I work on security at red hat and soon to be ibf um I'm. Currently, an incident response at red hat and I'll be working on security for all of for all of our storage products, such as cluster odf and sap. When we moved to IBM I did grad school at UCSC in uh formal methods and also Storage security. I did my undergrad at UMass Amherst and in my free time, I love to hike with my dog and garden.

A

And there would be a third speaker which is Mike Hackett, who unfortunately, is super busy keeping holding before it for our support team, so it could be with us today, but he is. uh He is an equal contributor to this work. So um thanks thanks to him for all that he has put in now we're not going to do the introduction to what Seth is in the session.

A

If you don't know what Chef is you can ping me later and I'll tell you but I, don't think that's necessary and so um same for Rook they're great. You know, you know everything already so.

A

um Security practices harden a specific point of the infrastructure, cherry picking practices without the model of the track and the attacker is not a viable strategy.

A

The joke usually goes. If you want to protect from all possible tracks, you should turn the computer off where it can concrete drop it at the bottom of the ocean, shoot it in the sun and things like that. In other words, absolute security is not something that's usable.

A

It needs to be defined in the context of an actual track model. Are you facing script, kiddies or Gru, or some dry that privileged inside their scenario? Some of these want to steal your data. Others want to crypto, lock you and hold you for ransom, others, yet maybe just satisfied with the disrupting your service.

A

So um oh look at that. It is reminding me appropriating a security presentation, I guess.

A

A

Excuse me one second: where he's a pending uptake that needs to be installed.

A

All right back here are the slides back up.

A

Yes, hopefully, yes, um so some of them may want to actually cause damage. Others may be satisfied with the disruption not need to actually steal your data, so the the model of who you're facing is essential so that you can pick from the vast collection of hard-earning practices you can put in place, which ones make sense for you and your users.

A

So let's Dive, Right In, uh the public security zone is an entirely untrusted area of the cloud uh here, we're looking at the network. First, uh it could be the internet as a whole or just networks external to your crafter. That.

C

A

Authority over data Transmissions Crossing, this Zone should make use of encryption note that the public Zone as I just defined it does not include the storage cluster front end. The chef public underscore Network, which defines the storage front end and properly belongs in what is called the storage access zone. So maybe not the most ideal terminology there, but that's what it is.

A

uh The SAS client Zone refers to networks accessing test clients like the object, Gateway, the soft file system or block storage. Deaf clients are not always excluded from the public security Zone, for instance, it's possible to expose the object gateways as 3 or Swift API and the public security Zone.

A

Actually, that would be a very common scenario. In fact, next the storage access zone is instead, an internal Network, providing such clients with access to the storage cluster itself.

A

Finally, uh the cluster Zone refers to the most internal Network, providing storage nodes with connectivity for replication, heartbeat backfill and recover.

A

um This Zone includes the surf clusters. Back-End Network called the cluster Network itself.

A

Operators often run clear text traffic in the cluster Zone, relying on the physical separation of VLAN separation uh from all other network traffic, um and here is a good example. This would not be a valid choice if your attract Model includes an adversarial privileged Insider, um whereas it's perfectly fine. If you're a threat model is for script kidneys. So um again, you need to choose what you're doing, based on what your thought model is.

A

These four zones are separately, mapped or or combined, depending on the use case, and the threat model in use.

A

Now, components spanning the boundary of two security zones with different Thrust out or authentication requirements must be carefully configured. These are natural weak points in network architecture, as could always be configured to meet the requirements of the higher level security zone of the two.

A

Obviously misconfiguration here is a problem, but these are also natural attack Targets in terms of places where people look for exploits.

A

In most cases, however, your focus should be on the security controls due to the likelihood of attacks to try to escalate from from one zone to the next.

A

Operators should consider exceeding Zone requirements at integration points which, for a storage product is often easier to accomplish than for a compute product.

A

For example, the cluster security Zone can be isolated from other zones easily because there is no reason for it to connect to other zones.

A

Conversely, an object Gateway in the client security Zone will need to access the cluster security Zone's monitors uh on Port, seven, six, seven, eight, nine and osds on full range of ports and will likely expose its S3 API to the public security Zone um exemplifying the the Zone Crossing scenario. That I was just describing.

A

What happened here.

A

Oh sorry did I turn off notifications.

A

Maybe that's what I did wrong.

A

A

Okay, so sorry about that to learn getting your interruptions on the presentation now: um okay and that's it for the network, part and I. Think- and this is where Anna took us over.

C

Yeah so going on to sort of what we do in product Security, in addition to the design of theft, storage. Let's talk a little bit about what product security at red hat does.

C

Product security at Red Hat poll is a secure development life cycle with the goal of reducing risk and improving security for both staff and Rook. We suggest improvements we pen test. We manifest all of our dependencies, even the really obscure ones in an in some container somewhere. We review vulnerabilities, we track weaknesses and exploits.

C

We approve a router releases, checking details. We review all new releases, checking new libraries for cpes checking old ones too, uh and we've recently somewhat recently within the past year. This we split it into incident response and security architect roles in part to ensure good Geographic coverage and preventing any lapses covering all stages of our software, because I don't want to work at two in the morning and nobody does and nobody's so sharpest. But so one of these branches is called incident response. This is the branch that I currently work in.

C

We respond to incidents, so we can release fixes in a timely manner. We investigate all vulnerabilities that are reported to us just in case some really minor. One is in fact pretty serious. We work with Upstream we work with engineering. We want to make sure that all flaws are fixed.

C

We also try to coordinate all Publications to prevent disclosing a flaw before engineering is aware, and in some cases, we'll even wait to publish until we have a fix ready. This all depends on whether or not we're actually able to keep it under Embargo. Of course, if somebody publishes something to Twitter, we can't exactly follow this process, but when we get a report to us, we we really try to coordinate to minimize the amount of time that significant security vulnerabilities are public without us having it fixed.

C

We also regularly check for exploits that are discovered well after a vulnerability is published, because that can affect the severity of an exploit of a vulnerability and in very exciting news. We recently open sourced our incident response plan that covers our process for responding to incidents. We do this to help industry overall and really include the open source ethos within incident response on the other side of things prior to release our security architect Works, to improve stuff by working on the design of it.

C

We manifest all dependencies, we have a list from engineering and then we also automatically do this with tools, as I said earlier, to make sure that we don't even miss the most obscure container pulling something as part of a build somewhere.

C

That may not seem obvious, but could still be exploitable in a really theoretical scenario, so we're really doing a good job, manifesting everything and, of course, that's required, but we're doing a really we're actually getting everything.

C

um We also, as I said, before, carefully review all major releases, our new libraries for cpe's vulnerabilities. We review all exceptions to security policy in some cases will approve them and in other cases we'll say: hey engineering. You actually have to go fix this and we work with the compliance team to validate stuff to our security standards um and any policies that are that we're implementing.

C

And in addition, we've worked to improve cryptographic, algorithms used and stuff. We Implement those in ways such as the improvements to messenger too, and we work on other design choices like the key storage improvements and rgw and then moving on to some of how we do encryption.

C

Let's talk about encryption and key management for yourself server side operators overwhelmingly choose to encrypt data at rest, using the Linux, unified, Key, System, better known key setup, better known as Lux, and all data and metadata of a soft storage cluster can be secured. I'll give it one moment.

C

Federico, let me know if you'd rather have me put the slides up.

C

C

Yeah, so with data at rest, we're encrypting it using locks. All data and metadata of a soft storage cluster can be secured using a variety of dmcrypt configurations. Almost all of red Hat's customers choose too not a requirement, but most people are doing it and it makes a lot of sense too.

C

We are also enabling security best practices by locating our monitor damage our mons on separate hosts from the osds. This basically means that the drive or host is physically separated from its decryption Keys. You need both drives to get the keys and the data and this interest to anti-affinity of the keys and the data they encrypt. Obviously,.

C

Object, stored, Gateway. In addition, our rgw has additional capabilities. This includes encryption at ingestion time, the use of per user Keys, as opposed to per Drive Keys We, allow key rotation with tools like Vault. We have support for AWS, SSE KMS and more. We even have a Department of Defense Cipher certified under Pips 140-2 as a standard that can be used when they're supplied by a Rel, inappropriate version. As you can see, we just have a wide variety of options for Key Management on the slides.

C

All these are great options and they're all supported just to make it as easy as possible for the user to use proper key rotation and management.

C

And now that we've talked about encryption at rest, we got to talk about encryption and Transit. Network communication can be secured by turning on the CEF protocol. Encryption in the in the messenger version 2.1 protocol introduces Nautilus. This is the one that we worked with product Security on and you know we definitely have an option to have the Legacy clear text protocol.

C

Fine, it's secure enough. The networks where the cluster is using the stuff protocol are usually physically or logically isolated from access, but uh so for compatibility reasons you want to have the clear text protocol as the default. At the same time, we now have an encrypted internal protocol and again all data at rest is encrypted, irrespective of access protocol.

C

In nearly all cases, all of our customers are choosing to use those methods and for data in transit, we're using fips 140-2 certified ciphers as an option and again we we offer that option just with compatibility and overheads why we wouldn't traditionally encrypt the back end protocols, but in most cases the performance impact is actually really insignificant for a properly designed cluster. Our latency is overshadowed by the network communication as long as our CPU overhead is accounted for and we have some best practices on this slide.

C

uh For example, you can use TLS security from your object gateway to your S3 clients. You can use your you know we'll go into this, but TLS termination at H.A proxy is a special case and we always incur encourage Network hygiene at individual mode so going on to the next. uh So going on to a little bit more here. Looking at specific protocols, our S3 service is usually secured between rgw and the S3 client, with TLS on Port 443.

C

um We do allow plan HTTP on Port 80, just in case somebody needs it for some public data. It depends on the nature your data being served confidential versus public.

C

And again, just like I mentioned TLS termination at aha proxies a special case where the link between h, a proxy and rgw is in clear text. So you have to keep in mind security zones just like Federico was talking about earlier, but provided it's located in the appropriate security Zone. We have no concern and again just like I was mentioning earlier standard Network practices like firewalling individual nodes to only exposed a cleared list of Port Supply. Let's not have all of our ports open to the public.

C

Let's only have a clear glass, so best practices apply. We make it easy to have those and also have additional security practices on top of them, and now talking about Rook, specifically uh Brooke can use custom resource definitions crds to encode many of the settings that we want.

C

So, for example, that's just one more moment.

C

Also, with security, it's very important that it be accessible for the users and if it's frustrating the users they're less likely to implement the practices such as updating their computer, if it is constantly asking them to do it in a presentation.

C

Okay, so Rook specifics, Rook can use custom resource definitions since to encode many of these settings. uh For example, we configure trust certificates for rgwritos Gateway web server.

C

um Rip supports at risk data encryption, as we discussed earlier within within flight stuff protocol encryption in 1.9, kubernetes user permission systems applies to persistent volumes, so permissions quotas, all that comes from kubernetes with nothing group needs to do. Brooke supports e-management systems as well in our DSi driver allowing individual volumes to be encrypted with their own key. This limits the scope per key. This all ensures that we can follow best practices easily, such as key rotation, revocation and limiting the scope from each key.

C

This limits the scope of unencrypted traffic and it limits the scope of each key, which is a fast security practice going on to the control plane.

C

On the next slide,.

C

We'll be talking about the control plane and as popular as by ansible SSH is used by Seth admin, sub ansible.

A

C

Day, one tools to provide a secure command path for install and upgrade options as part of post management.

C

We do this so that the dashboard is usually not exposed to the world, and yet it also needs to be reachable by the operator's workstation to be of use.

C

The manager supports the whole infrastructure and needs to be reachable on the storage access Zone. Hence why we're using SSH? So you can see this is how we're implementing it with a sort of different ports, and you know where exactly we're using SSH Keys um again with the management dashboard. For example, the dashboard access Zone uh may be tailored to suit your local threat model uh with with SSH.

C

You know you can definitely Association using a user, and then you can change to root access if needed, provided it's secured with SSH Keys, there's all sorts of different practices that you can do here now going on to our next slide.

C

We're going to talk about identity and access.

C

So sap uses shared secret Keys, which prevents our clusters from Banning the middle attacks by default uh for Authentication.

C

Now this is all authentication and authorization are on by default if the user isn't supplied you're going to just give them the client.admin as the user and restrict accordingly. Now you still need some good practices. uh For example, you may only want to Grant keyring, read and write permissions for the current user and root with client admin user restricted to only you definitely don't want all users to be root as per best practices, because that just gives them way too much permission.

C

So now, let's talk about W briefly with rgw and identity and access. We support the key and Secret model of AWS S3 and the equivalent model for openstack Swift.

C

And again, the administrator's key and secret should be treated with appropriate respect. You should use administrative roles very sparingly to reduce your risk profile and again your rgw's users. Data is stored in the soft pools, as we can see for user Authentication.

C

They should be secured, as we previously discussed. Regarding data at rest, we can couple with oidc providers such as keycloak, backed with your organizational's identity provider, for an even a better sense of granular roles or identity. Access for attribute access, um really we're trying to make this as accessible as possible for different options and allow our users to have as many different choices that are pretty secure going on to the next slide.

C

I'll finish this up with identity and access and then I'll hand it over to federicio. After this, but with identity and access, we support ldap and active directory users. We do recommend secure ldap for identity Services. We also again, we support openstack Keystone to authenticate our object. Gateway users in general. We just have a lot of options for identity and access that are pretty secure and when the slides are back up, I'm going to hand it over to federicio to finish us up on the auditing slide.

C

A

My bad uh other thing is fairly self-explicative. This is the lag format that we use. um You should uh aggregate your logs into a central location. If you can, if you have a central um log management system that is highly recommended, you probably also want to purge your logs on uh on a regular schedule. There is no sense leaving them around forever, just in case something um something is getting leaked there, that we're not aware of.

A

And then data retention, so this is actually a fairly juicy topic. There is a lot here, so that's data retention, behaviors generally um users don't go and access rados directly. So we look at the at the higher layers here in RBD, rgw and to some degree the file system. We have what you could consider intentional data retention. You can go and delete objects, but, for example, RBD has this new bit of functionality called the trash bin, where it essentially gives you an undo button where you can restore a volume that you've deleted this?

A

That space was not needed or it's a policy to to purge it after a certain amount of time did not kick in yet uh on. Rgw buckets are, of course, versioned. It's part of the design of an object store, so that's a natural property.

A

um Obviously, if you want to manage data retention policies, you need to be aware of these behaviors, and then there are places where you intentionally increase the data retention like setting up rgw4 uh write one semantics, as a financial customer may do, or setting up a multi-factor authentication on rgw to to protect against. Let's say a ransomware attack to make it more difficult to go and delete data, so um these are things at at the sef's level, um but once you have deleted the data well, it may not be easy to recover.

A

An expert could still go and piece together. The bits from the drive or or from rados um maybe could not recover any an entire file. But um from the point of view of data sanity, sanitizing um overwriting files in self is not going to do anything.

A

Obviously, your overwrites are going to go someplace else than when and then where the original data was stored, so for a secure deletion generally, a best practice is to encrypt the drives, which you should be doing anyway, um and to sanitize the Drives By, forgetting the key that they were encrypted with. That is much faster. It is compatible with Hardware return policies. You don't have to um you.

A

Don't have to explain to your heart disk vendor that you shredded the drives for security reasons um and you're returning them uh competing for a new drive as part of the warranty process that they may not work too well, and um so it's very fast. It's compatible with with um uh RMA processes, and you should be encrypting the drives anyway, so you should have the ability to do that to begin with in terms of hardening the binaries.

A

These are highly vendor dependent um and these are Red Hats choices. Of course, we run with SC Linux by default, which probably surprises no one, since uh in in our company, SL Linux is sort of a religion.

A

We have options to support uh fips 142 out of Rel, so you can install Rel with um with ciphers that have been certified by the Department of Defense and uh and Seth will just make use of those and then, in terms of heart and binaries, you have on the left.

A

You have on the left here the the options that we use when we build the safe binaries and on the right, you have the options that are generally provided by the operating system, environment and kernel, um like the general mix of aslr hardening strategies or limiting system calls that you can access and um position independent, executables and the like. All of these are part of um Raul's default. Configuration I believe that they're also part of um ubuntu's default configuration. So our most common os's are covered.

A

um Of course, if you use another distribution, you should investigate what what they do, because these are essentially um three additional robustness for your binaries and they make it a lot harder to exploit um to exploit um vulnerabilities that we may not be aware of stands in the way of crafting exploits essentially.

A

Okay, so that's um the end of the formal presentation before we go to any questions, we have a few links for you in terms of uh resources that may have additional interesting bits: um iranios not from aqueous security, has a nice um explanation of managing secrets in kubernetes and how the kubernetes secret mechanism works and how it offloads to actually secure locations, because the kubernetes secret itself is just um is just running a hash on the on the security.

A

It's not it's not actually secure, but when you're storing it into some other things like um Azure, Corporal or some other external location, then that mechanism actually actually protects your secrets. So every Everything varies is expanded by running uh hacking, kubernetes chapter six by um Michael, housenbluster, former colleague and.

B

Andrew Martin.

A

uh Covers storage extensively in chapter six, and it's a very recent book, so it's it's um up to date or nearly up to date, uh the storage, our actual product um has a full volume of data security and hardening. We sort of worked you through the main takeaways of that, but um it is, it exists in written form.

A

um uh The kubernetes documentation has a nice tutorial on encrypting secret data at rest, sort of expanding on what we're talking about at the OS level for the osds here, and the last link explains uh that um alphabet soups, that I heard on the left slide of all the hardening options.

A

If you are not um kernel or glibc junkie, and you probably have no idea what those are or do- and there are very few places where they are documented, so um the GCC documentation is a good place to start and goes and details or what the actual options do.

A

Okay, so with that um we are done, um we actually have some. Thank you as an acknowledgment for all the people that contributed to uh to prepare the presentation and um that's it.

D

All right, thank you.

D

Hey, do you have some questions here.

D

The first one I'll go to here is a U.S of Dos of stuff services and scope.

A

C

A

Didn't catch that I'm going on.

C

I can say so with denial of service attacks for services.

C

um We generally do if we get a report of a vulnerability that could result in a Dos, we'll generally send it over to engineering just like we would any other vulnerability trying to prevent and fix it. um Of course, this is a very broad category of attacks, so, for example, saying like oh, it's certified against EOS.

C

Well, what happens if somebody doses your internet provider at the ISP level and stuff that we're not so much preventing against, but, generally speaking, if it's an attack towards a vulnerability affecting staff, we will file it and fix it, just like any other, including for denial of service attacks.

D

And it another question: I have: are the performance simply implications of encrypted osds, known with regard to CPU usage and I O latency.

A

Yes, um there is a study that um Ben England did I think a year and a half ago, um I, don't know if it's uh being released outside of red hat I would have to look um essentially.

A

uh There is obviously an overhead to encryption uh in flights and you can offset it with the appropriate sizing of the CPUs. So if you're going to encrypt all your network traffic, you account for it at the design in the design phase with a little bit of additional CPU. The fact that stuff is networked.

A

Storage means that the latency is generally shadowed by Network latency and the encryption overhead is not going to affect latency much, but obviously you need to size the cluster so that that it has the higher power to uh to do this in the customers that where we see encryption and flight in news this is generally a regulatory requirement. So it's not not something where there is a choice. Their regulator is saying.

A

Even if the network traffic is on a separate Network, we don't care, we want it encrypted, and so they they just do it and they add a few more cores. But that's that's the extent of the effort. So it's it's not significant impact at rest. um Encryption is essentially completely overshadowed by Network, latency and, and it's something that at this point is the fault: I stopped tracking uh the prevalence of encryption at rest in 2015 when more customers than not were using encryption at rest. Now, it's it must be in the 90 plus uh group.

A

It's uh it's effectively an exception when, when there is no interest in the trust, succeedingly right.

D

Okay, thank you.

D

I think, that's all the questions we got in chat.

D

Anybody else have questions. They want to speak up on real, fast, hey.

E

Frederico, thanks for the presentation, hey Hannah, thanks for your presentation, um I had a question about audit laws. Hey um it I was recently troubleshooting an issue and I was surprised. uh It was kind of a discovery for me. um So uh when you run the CLI commands, uh you know like Steph status or any other, like you know, read commands um from the CLI. Those are all those all get logged in the audit log.

E

However, um I was surprised to not see like if I, um if a volume gets created like an RBD volume, gets created or do or destroyed, that type of activity doesn't get logged in the audit log. Is that something? My knowledge is only up until Nautilus, so I apologize, if that's changed, um but is that? Is that something that you guys are? Is it is that is that configurable or um are there thoughts to extending the audit logs, like someone deletes an RBD volume and then comes and complains to me that they lost data.

A

E

Don't I, don't I can't tell them like what happened.

A

I um my first question would be um whether there is a difference there between the uh log levels. There was an effort. um It's been several years back where a number of things were pushed into higher log levels, because our logs were enormously chatty. They were developer friendly, logs, not operator friendly logs. So probably the first thing that I would check is maybe it's being logged at the higher level, but it seems um it seems wrong that at whatever level you're operating at that is not being logged.

A

So um can you can you send me a quick note to remember uh and I'll I'll track this down? Thank you.

B

Yeah Federico, I I, don't think this is logged uh because in general uh the uh the well it's not logged in the in the audit log, at least right. So because the the audit log is uh mostly uh limited to just kind of cluster-wide uh activity, and that means mostly things that concern Raiders itself and not the higher level applications such as a Blog device or the file system.

B

So when you remove an RBD image on the radius level, uh what that means is a bunch of objects get removed so, like that's, that's not considered to be that big of a deal like as compared to sound one, for example, creating a new key or uh you know, reconfiguring the entire pool. uh So things of that nature.

B

uh So yeah there is uh it's not a question of log levels uh that just does not go to the to the cluster uh to the audit log.

A

Understood, thank you Elia, okay, so I guess we need to figure out if there is another log where this should go to and how.

E

Yeah I mean I can see why you wouldn't want to log the creation and deletion of each radius object in an RBD pool. I get that but um yeah. No, it's interesting. Okay! No! But thanks thanks for answering the question.

B

Yeah, that's definitely something we can. uh We could look, I mean uh there is no reason uh it is just uh it just. It would need to be uh designed uh to concern, not just uh let's say, RBD image, creations and deletions but kind of across stuff, because, like I said right now, the idea with the audit log is that it's, you know fairly low level, and if we wanted to extend that to these higher level, uh you know application specific and I keep using the word application. But it's you know not the user application.

B

These are just things on top of radius. uh If we wanted to extend it to that this, you know we would need a comprehensive design. uh You know just logging, for example, creation and deletion. Events for IBD images uh that that should be fine from the perspective of you know not overloading the log and not overloading the cluster, because you know there's a natural limit on the number of RBD images you can create just because at some point and you're going to run out of um uh well.

B

At least, if those images are used, if they're all empty, then you could potentially create uh millions and millions of them. But uh if those images are real, then there is natural limits so um and there's no there's no fundamental reason to not log that uh to the audit log. uh But uh you know that's, that's that's not as kind of not a short-term fix. uh This would need some design work and uh to have a comprehensive kind of story across the entire project.

A

An application Raiders application, Level log if.

A

And one that doesn't get swamped too easily we're tracking every rgw object, solution. I think that could be interesting.