Cloud Native Computing Foundation Cloud Native Security Conference North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Data Security and Storage Hardening in Rook and Ceph - Federico Lucifredi, Red Hat

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Data Security and Storage Hardening in Rook and Ceph- Federico Lucifredi, Red Hat

This talk will be presented by Federico Lucifredi, but features his collaborative work with Ana McTaggart (Red Hat) and Michael Hackett (Red Hat).

We explore the security model exposed by Rook with Ceph, the leading software-defined storage platform of the Open Source world. Digging increasingly deeper in the stack, we examine hardening options for Ceph storage appropriate for a variety of threat profiles. Options include defining a threat model, limiting the blast radius of an attack by implementing separate security zones, the use of encryption at rest and in-flight and FIPS 140-2 validated ciphers, hardened builds and default configuration, as well as user access controls and key management. Data retention and secure deletion are also addressed. The very process of containerization creates additional security benefits with lightweight separation of domains. Rook makes the process of applying hardening options easier, as this becomes a matter of simply modifying a .yaml file with the appropriate security context upon creation, making it a snap to apply the standard hardening options of Ceph to a container-based storage system.

A

Good evening cloud native security conference, it's an honor to be joining you for this first instance of this new event. I'm joining you from boston.

A

This is red, hat's famed westward office, although you can't see much of it- and this is just a small conference room that we turned into a virtual event recording booth, but um with that and without further ado, let's get going now my marketing manager- wouldn't let me um get away from the concept that I need to introduce myself so here it is very briefly had the privilege of spending my entire career in free and open source software.

A

uh I was the product management director for trite hat for the last seven years. I previously was the ubuntu server pm at canonical.

A

If you remember 1404, that was my favorite release there and if you go back the decade, I was the systems management tsar at susa, uh the dreaded systems, management, czar, actually some things I worked on on the slide, a bunch of different products, and uh I was the maintainer of man for about 10 years.

A

Technically, I still am, I suppose, but I'm letting uh colin take the lead with with this version. It seems like the right thing to do um so. uh One last thing: I have a book out on aws operations by o'reilly. That's why you see me surrounded by clouds and that's plenty enough. I guess now rook for those of you that are not in storage.

A

Rook helps reduce the operational burden, storage team faces, creating horizontally scaled self-healing clusters and stuff and ceph is complemented by rook, making them increasingly self-managing.

A

Even self-scaling, you could say rock enables have to deploy kubernetes with on kubernetes with ease enabling the benefits of containerization storage is just another workload running on top of the compute plane or on bare metal as an external entity for petabyte scale, independently managed storage clusters serving different kubernetes clusters and possibly other workloads. So as usual, we can make things abundantly complicated.

A

The combination of the two is a quite sweet storage solution in general and obviously perfect for kubernetes, which is what it is designed for. Now. Security practices harden a specific point on the infrastructure, cherry picking practices without the model of the threat and the attacker carrying it out is not a viable strategy. The joke usually goes. If you want to protect from all possible threats, you need to turn the computer off, bury it in concrete drop it at the bottom of the ocean. In other words, absolute security is not usable and probably unattainable.

A

It needs to be defined in the context of a threat model. Are you facing script kitties or the gru or the dreaded privileged insider? That we should all be worrying about? These are very, very different scenarios. That's the starting point for any security consideration.

A

Now, let's dive in on once we have the profile. What are we going to tighten and what is what can be tightened?

A

Starting from the network? We can break the network in security zones and the public security zone is the easiest to understand it's an entirely untrusted area of the cloud. It could be the internet as a whole or you could conceptualize it as just networks external to your cluster, that you have no authority over data transmissions crossing this zone should make use of encryption.

A

Note that the public zone, as I just defined it, does not include the storage cluster front end. What in ceph is called the public underscore network which defines something different. The storage front end, which properly belongs in the storage access zone.

A

The self-client zone refers to the network networks accessing ceph clients like the object, gateway, the ceph file system or block storage. Ceph clients are not always excluded from the public security zone. For instance, it is possible to expose the object gateways as 3 and swift api in the public security zone.

A

The storage access zone instead is an internal network providing safe clients with access to the storage cluster itself.

A

Finally, the cluster zone refers to the most internal network, providing storage nodes with connectivity for replication, heartbeat backfill and recovery tasks. This zone includes the ceph cluster's backend network, called the cluster network itself.

A

Operators often run clear text traffic in the cluster zone, relying on the physical separation or vlan logical separation of the network from all other traffic.

A

This would not be a valid choice if your threat model includes adversarial privileged insiders, for example, but it is a perfectly fine assumption for other models, and this is a good example of what I was just saying earlier.

A

These four zones are separately mapped or combined, depending on the use case, and the threat model in use now components spanning the boundary of two security zones with different trust or authentication requirements must be carefully configured.

A

These are natural weak points in network architecture and should always be configured to meet the requirements of the highest thrust level of the zones connected. In many cases, the security controls should be a primary concern due to the likelihood of attack. At this point, operators should consider exceeding zone requirements at integration points which, for a storage product is often easier to accomplish than for other types of products.

A

For example, the cluster security zone can be isolated from other zones easily because there is no reason for it to connect to other zones.

A

Conversely, an object gateway in the client security zone will need to access the cluster security zones, monitors port 6789, the osds, all of them on port 6800 to 73 7300, and will likely expose its s3 api to the public security zone, port, 80 and 443.

A

Let's move on to encryption, server-side operators overwhelmingly choose to encrypt data at rest, using the linux unified, key setup mechanism, better known by all of us. As lux all, data and metadata of a cef storage cluster can be secured using a variety of dmcrypt configurations and almost all of red hat customers choose to use this facility.

A

A security best practice is to locate, monitor demons. The mons on separate hosts from the storage, daemons osd's, ensuring anti-affinity of the keys and the data that they encrypt. This results in physically.

A

Removing this results in protecting against the scenario where physically stealing a machine or removing the drives does not carry the keys with the with the part that was stolen.

A

The object, store gateway has additional capabilities, including encryption at ingestion time, the use of per user keys, as opposed to per drive keys their rotation with tools like vault support for amazon, aws, ssc ems and many more things.

A

One thing that I'm going to call out is the department of defense ciphers certified under fips 140-2.

A

These can be used throughout ceph, as supplied by rel in appropriate versions.

A

Network communication can be secured by turning on ceph protocol encryption in the messenger v 2.1 protocol introduced with the upstream gnarliest release and updated sense.

A

Here, clear text is preponderant because the networks where the cluster uses the sef protocol are usually the trusted ones and they're, usually physically or logically, isolated from access compatibility and overheads are the primary reason why back-end protocols are not encrypted by default, but in most use cases the performance impact is negligible.

A

For a properly designed cluster latency is shadowed by the network communication, provided that you size the cpu, the cpu core count to account for the fact that you're going to encrypt everything. So it's going to cost you a little bit in hardware, but you can do it without without seeing a performance impact for most workloads.

A

Looking at more specific protocols, the s3 service is usually secured between rgw and ds3 client with tls and port 443.

A

Although plain http on port 80 is obviously also an option, depending on the nature of the data served confidential versus public. There is plenty of scenarios where rgw may be serving just web pages. Tls termination at an h. A proxy endpoint is a special case where the link between h, a proxy and rgw itself is in the clear text and needs to be located in the appropriate security zone.

A

Standard network practices like firewalling individual nodes to only expose the clear list of ports apply all the usual network hygiene things that you would normally do and let's look at rook specific things for a second rook can use crds to encode. Many of these settings, like configuring trust certificates for rgw's web server.

A

The kubernetes secrets mechanism is not really secured by default, so that's another thing to account for base64 is easily reverted and needs to be hardened with encryption at rest and access policies to containers using secrets.

A

Base64 is not encryption. Base64 is just encoding, it can be revo reverted with one command.

A

Rook supports at rest data encryption, as we discussed earlier same thing, with in-flight self-protocol encryption not supported yet, but coming soon.

A

You should in any event be using the cloud's, extremely flexible, software-defined networking to segregate unencrypted traffic um to private networks really on on a public cloud environment. You have no excuse not to be doing things that way you can go wild with creating zones, kubernetes user permission system applies to pvs, so permissions quotas and all the rest. All of that comes from kubernetes, with nothing for rook. To do here, which is great, more interesting, perhaps, is kms support in the csi driver.

A

This allows individual volumes to be encrypted with their own key, something that, for example, in block storage. We would like to do, but we cannot do without rook at this point not yet we're working on it, saf, adm, sensible and other deployment and day one tools use ssh to provide a secure command path for install and upgrade operations.

A

This is common these days it is the control channel popularized by ansible for host management. The dashboard is usually not exposed to the world, but it needs to be reachable by the operator's workstation to be of use.

A

Lastly, the manager daemon supports the whole infrastructure and needs to be reachable on the storage access.

A

Zone, a few of keys protects the cluster from man in the middle attacks by default. A good practice here is to grant keyring, read and write permissions only for the current user and root with client admin user restricted to root. Only rgw predictably supports the key and secret model of aws s3 and the equivalent model for openstack swift.

A

The administrator's key and secrets should be treated with the appropriate respect, use administrative users sparingly to reduce risk profile, as you would do anywhere else.

A

Rgw users data is stored in ceph pools, which should be secured, as we discussed previously. Regarding data at rest processes, ldap, active directory and keystone identity vaults are all supported.

A

Operator actions against the cluster are logged and should be periodically reviewed, as well as aggregated to your log management system. If you have one- and you should have one once, data is deleted from a seph cluster, it generally cannot be recovered for practical use. There are exceptions, however. Rbd has a new facility called trash bin, where dynamic use of spare pool capacity can be used to retain deleted images until that spare capacity is needed or until a certain number of days is elapsed similar.

A

Similarly on rgw versioning of object, store, buckets may result in deleted objects preserved as part of version history until they are deleted by a policy or by the administrator.

A

Wherever user data retention is a concern. Configure your data storage pools accordingly.

A

Additionally, individual data blocks that used to constitute an object, file or volume are typically still present on the persistent storage system until they are overwritten by use.

A

You cannot overwrite a sf cluster just by writing a lot of data to it. It's um it's not going to work so secure deletion is another question that we often see, and the easiest way to sanitize media is to do the right thing and use osd encryption.

A

And when you need to sanitize the media, you forget the key and that's the most effective way to to be able to discard uh drives safely without overwriting, with a separate device or de-gaussing or shredding or anything of that sort, which is usually not possible when you have to return them under warranty.

A

Hardening options are highly vendor dependent. The following are some of red hat's choices. Other seph distributions will of course, vary we ship with selinux on by default in enforcing mode.

A

Of course, you know that sc linux is sort of a religion at red hat, so that's hardly surprising and we can make use of fips 140-2 certified ciphers, as supplied by rel, when configured to do so.

A

Hardening of binaries is also of interest because of what I told you about demons sitting at the boundaries between security zones and being natural targets for attacks to cross those boundaries.

A

I believe red hat self storage, binaries use sec, comp, pi and several, if not all, of the aslr options, but I didn't manage to catch my build master to confirm before recording this, so I'll have to defer this to the q a session if you are interested in any event, these are worth exploring because it makes exploiting of overflows or that kind of binary attacks all all that much harder, and on that note, that's what we have for today.

A

I hope that I've, given you some useful information, I'm very interested in hearing what your questions are and um remember. Speakers are pavlovian devices. So, if you like the talk, let us know that you did.

A

If you wanted more of something, tell us what is missing what what you'd like to hear- and I guess that's it for today. Thank you.