Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Enhancing Kubernetes with the Security Profiles Operator - Colleen Murphy & Sascha Grunert, Red Hat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Enhancing Kubernetes with the Security Profiles Operator - Colleen Murphy & Sascha Grunert, Red Hat

Kubernetes provides ways for container workloads to leverage Linux security features like seccomp, Apparmor, and SELinux, technologies that allow applications to be bound by security profiles that prevent unexpected and malicious behavior. But crafting and deploying these profiles is a manual process that requires administrators to operate directly on the underlying host and end-users to have knowledge of the security configurations of the hosts.

The Security Profiles Operator is an out-of-tree Kubernetes enhancement that provides cloud-native APIs to manage these profiles.

In this session, Colleen and Sascha will discuss how the Security Profiles Operator has evolved. They demonstrate how the project empowers workload security by making seccomp profiles easier to use inside of Kubernetes. Besides that, they will speak about the future of the project, how it may integrate into Kubernetes and what it means to combine profile-based security features managed from one source of truth.

A

Hello and welcome to our presentation on enhancing kubernetes with the security profiles operator, my name's colleen, I'm a contributor to the security profiles operator and I'm a software engineer at souza, and it's a pleasure to be here with you today.

B

Hey friends, I'm sasha one of the maintainers of the security profess operator, and it's a pleasure to be here today and I work for redhead.

B

What will we cover in this talk? First of all, we would like to speak about how to enhance default kubernetes workload security. So we would like to speak about which goals we would have in mind and how to achieve that in such a rapidly evolving project like kubernetes.

B

Secondly, we would like to lay out the current state of the security profiles operator, so we will cover the operator features in detail and how they can be used to make workloads more secure, and then we will see the operator in action. So this is time for demoing the operator, while keeping common use cases in mind and after that, last but not least, we will lay out our plan for the future of the project.

B

So, for example, we will discuss our long-term goals and we'll leave some room for some q a session afterwards how to enhance default workload. Security in kubernetes kubernetes does not provide strong security defaults out of the box, so this means that kubernetes has to be designed for usability over security. So, for example, if we decide to restrict the discourse of a workload, then we can have to keep in mind that this may break applications, while it makes others more secure other than that security. Related decisions have to be considered by the system.

B

Administrators, the site, reliability engineers, as well as application developers and those different roles, make it hard for the community project to decide which position may be the most important one and which should be considered as the default, the security context of kubernetes or it's a security configuration that will be applied to a pod or a container. So this native api field makes the usage of the security feature comfortable, because the api server can validate the fields directly by applying them to the workload.

B

Some security related features like app or more are not graduated to the security context, yet they are still using the annotation based syntax to apply settings to workloads which is not obvious to use and which is kind of error-prone, because containers have the reference they have to be referenced by.

A

B

And second, the discretion to the security context in kubernetes 1.19. So the feature is now fully integrated into the security context and will not support annotations anymore from kubernetes 1.23.

B

Our first goal is to make features easier usable by providing native fields. For example, if you look at the second profile of the security context, then we have a native type which can be localhost runtime default or unconfined.

B

If it's localhost, then we have to specify a relative path, and this makes it easier for the users, because the absolute path has not to be taken into account anymore, and if we compare to the former annotation based syntax, then we can see that it's way easier to read and that we don't have to actually reference the container name anymore, because we have the security profile field as part of the container or as part of the complete pot.

B

One drawback and limitation of that is that we don't have a localhost profile validation, so the relative path is still error prone and there is no solution about how to distribute second profiles to all nodes. Our second goal is to increase the user. Adoption of the feature so how to provide stronger defaults for second kubernetes, can we make the container runtime default profile the need default for all workloads or how to distribute second profiles across thousands of nodes? So can administrators get a mechanism to easily distribute those second profiles?

B

And the last question is how to create that composer for my application. So is it really necessary to collect all syscalls by hand, or is there a way to record profiles? So what do we want to do? Let's write a kubernetes enhancement proposal. uh No features can be iterated quickly out of the kubernetes source code 3., so the development cycles can be shorter. We can enrich faster testing and feature. Graduation is even easier.

B

So generally it's less effort for the community, as well as for the project maintainers to do it out of three and kubern86 can sponsor such a project, so this provides community infrastructure access as well. This provides full ci testing in brow, image, promotion and distribution and repository access management.

B

This is exactly the point where the security profiles operator initially is a comb operator was born. So please visit six case.io security. Professor operator and the project is sponsored by signal and lives in the kubernetes six namespace, which is kinda nice, because then we can leverage brow.

B

As already mentioned, the project already turns one year in april, so we already have two releases and plan for the next one containing many new features and enhancements, and the idea is to cover common use cases around security profiles, so around saccom c linux and app armor, and all that bundled into kubernetes.

B

So we have a list of user stories and you can give them a look and those contain the three mentioned: perspectives of the administrators, the site, reliability, engineering and the application developers.

B

So we have a huge roadmap, but the project at all is still very tightly scoped. So right now we still focus on feature completeness for second and say linux, and how does it work in detail? So let me pass over to colleen to answer that question for you.

A

All right, thank you, sasha, so I'm going to discuss a little bit about how the security profiles operator works and talk about the current features that we support in it. So the operator is driven by a few different custom resource definitions that we can kind of break into three general categories. The first is the profile crds. These are the setcomp profile and se linux policy crds, and these are resources that define the actual security profile that will get installed on the underlying linux host.

A

Then we have what I'm calling composition crds. These are the profile binding and profile recording crds and those are resources that will help application developers to build and use security profiles, and then, finally, we have more operational crds. These are resources that help cluster administrators with the running of the operator itself.

A

So going into a little more detail on each of these, we have first the security profiles themselves. These are kind of the cornerstone of the operator we started out with setcomp and setcomp restricts a process to executing only a finite list of allowed system calls defined in a sepcom profile on the host, and so the setcom profile crd provides a kubernetes native interface to create and validate these setcomp profiles.

A

This will get managed on the backend by a daemon set that converts the setcomp profile resource into a json configuration file that gets picked up by the cubelet and used to launch these containers and then more recently, we added support for se. Linux and seolinux defines more complex access, controls for processes and files via policy rules and object labels, and so the se linux policy crd, provides again a kubernetes native interface for creating and managing these sc linux policies.

A

This will also get managed by a daemon set that drops these sc linux policies into a directory on the host and then there's a daemon that will watch this directory and invoke seo linux tools to install these policies on the host, and then we have some more advanced resources that help support application developers.

A

We have the profile binding resource which creates a binding between a workload image and a profile resource, and so with this, there's no longer any need to change the yaml for a pod to set a security context to bind it to one of these profiles.

A

Instead, there will be a web hook that will inject a reference to a profile into a pod security context when you create the pod, and so this is especially useful if you're running some kind of workload that, for example, if you installed an application from an open source, helm chart, you would no longer need to fork or edit the helm chart. You can just create this profile binding and the pod and the profile will automatically be linked and then another exciting feature is the profile recording resource.

A

This is for setcomp and it generates a set comp profile for a running workload, and so with this, there's no more need to write these complex profiles out by hand, no more guess and check of running the application and checking the audit logs. Instead, you just create this profile recording resource and it automatically creates a set comp profile. Kubernetes resource for you that can then already be used by your application.

A

This does depend on using cryo as your container runtime, and it depends on the most recent version of cryo.

A

And then, finally, we have resources that help support cluster administrators running the operator. The security profiles operator daemon resource is kind of a configuration point for controllers for the controller that runs the daemon on the nodes. So it lets you toggle different configurations or set node tolerations for the daemon set, and then the security profile node status resource is a read-only node level view of the status of installed poly profiles.

A

So since profiles are cluster level resource resources, this gives insight into the availability of a particular profile on any given node in your cluster and then. Finally, it's also worth mentioning a couple other features in the operator. We have this login richer feature which supplements the audit logs with annotations. That help give context to audit events and lets you trace events back to the kubernetes workloads that generated them. So that's extremely useful, and then we have this profile stacking feature and what this is is kind of an inheritance model for setcom profiles.

A

So it allows you to compose application specific profiles on top of standardized base. So, for example, if you wanted to create a base profile for all the system, calls that run c would need to launch a container.

A

Then you could also create that as a base profile and then create a slim down profile with just a short list of system calls that your application specifically needs that wouldn't be necessarily being needed by other containers and with this profile stacking feature. These can be composed into one single profile, and so with that, I can break into a demonstration of some of these awesome features.

A

So the first thing we can look at is the deployment of the operator itself.

A

The operator web hook depends on start manager, so that needs to be installed first as a prerequisite, and then it's a simple matter of applying the operator manifests to your cluster that will create all your custom resource definitions, set up your rbac and run the main deployment, and the deployment is what will spawn the daemon set. That does the actual leg work of managing the profiles on the nodes in your cluster.

A

With that done, the first thing we can look at is profile reconciliation or how a profile resource in kubernetes actually gets rendered to disk on on the nodes in your cluster.

A

If we take a look at this example, setcom profile resource this gives us a list of system calls that an application should be permitted to use.

A

We can deploy apply that resource in kubernetes and then, if we go and examine it, that will give us a relative path where the profile is installed on the nodes in the cluster, and we can even go and examine a node and see that the profile has been rendered into a json profile under the cubelet, where the cubelet can find it and use it, and then for a pod to make use of this decom profile.

A

We need to define a security context for it with a reference to that relative path to the profile under under the cubelet.

A

If we go and create that pod, the pod should be created successfully, but if the pod goes and tries to use a system, call that wasn't expected and wasn't explicitly permitted by the setcomp profile. The host operating system will forbid it, and if we decide that this is an expected system, call that our application should be allowed to make. We can edit the profile.

A

Add that system call to the allow list and reapply the profile and restart the pod, and then the pod will be able to use a system called successfully.

A

Since we're managing these resources as higher level resources, it allows us to add logic that kind of enforces relationships between profiles and workloads. So, for example, if I tried to delete a profile, it will not actually delete the profile from kubernetes.

A

While it's still in use, it will mark it as deleting but leave it existing, and it won't actually delete it from kubernetes until all of the workloads using it have been removed, and once that happens, um it will get removed from kubernetes and it will also get cleaned up on the back end, because the kubernetes controllers give us the tools to be able to clean up on the back end when something is deleted. uh When a resource is deleted on the in the kubernetes api.

A

And then another feature that we can look at is profile stacking, and so what this allows us to do is create a setcomp profile to use as a base, and that could define a limited list of system calls that we think that any container should be allowed to use and then define a much more refined, slim down profile, that's specific to just our application, and so this gives us a much smaller easier to use easier to read profile.

A

And then, if we create a pod that references just that child or create both profiles, the base profile and the child profile, we can see that when it gets rendered, the composite profile contains all the system calls from both sets of profiles. The base profile and the child profile, including this application specific system call.

A

And so, um if a pod wants to use that child profile, it will be able to do so successfully.

A

And then the final thing that we can look at here is profile binding, and so what this allows us to do is define a binding resource that links a profile resource like a sitcom profile or in the future, an sd linux profile to a image tag, and then we can define a pod that no longer needs to explicitly set a security context.

A

We can just have a basic pod spec defined here.

A

So if we go and recreate the original set comp profile and then create that binding resource and go and create the pod, then if we go and examine the pod, we can see that the statcom profile has been injected into this um into this pod once it was created, and so this is really useful for running, like external workloads, for example, on an open source helm chart where you don't want to modify the helm chart. um You just want to use your custom setcom profile.

A

This allows that to happen really easily, and so with that I will hand it back to sasha who will demonstrate profile recordings.

B

In this statement, I would like to show you how we can record that compromise with the app with the security profiles operator, so to be able to run this demo, we would require cryo 121, which may not be released yet, but the latest cryomaster should be shifted sufficient at all. If we look at the output of pseudocrystal version on my local machine, then we can see that we run cryo in the latest 121 development version.

B

It is also necessary that we set up the oci second bpf project on the local machine, and this is an oci pre-stored hook, which locally compiles the ppf module attaches it to the workload and then traces its calls. If a word load gets deleted, then the syscalls will be put into a file which can be picked up by the operator. We have to configure cryo that it points to the actual hook, and for that we can look at the hooksteer configuration option which points to the oci second ppf hook in my local machine.

B

So everything should be set up what's next now we have to create a profile recording, which is the crd and a simple profile. Recording may look like this, so we create a profile recording with the name demo right now. We can only record second profiles and we choose a pod selector which matches our labels app equals demo and all those workloads which match those put selectors should be recorded.

B

So then, let's create the profile, recording and put it into our cluster. If we now look at all our profile recordings available in our local namespace, then we can see that the profile recording is up and running, and we can also verify that the pod selector still matches.

B

So, let's use a new pod to create some syscalls recordings.

B

So for that we just run a simple part which gets immediately removed after it has been run and we rename it mypot and we add our labels app equals demo, and then we create a directory test and let's see what happens now, the pod gets deleted, which is fine and now we should have a second profile available locally in disk. So, let's look at our second profiles in our clusters.

B

Here we go our demo. My pod second profile has been successfully reconciled on every node.

B

If we now look at the second profile in more detail, then we can also find our mkdir syscall in it beside other scores, which are necessary to actually run any workload in our cluster. Now this is the least amount of this course necessary to run a single workload in a kubernetes cluster. On my local configuration.

B

We can also see that we get some events, so the second profile successfully got synchronized to disk.

B

Now, let's delete the second profile again and recording single pods is nice, but recording multiple containers within a single bot is even better. So, let's create a pot, so we also add the labels to our pot, and then we run two containers in it: one engine x and one radius container.

B

So this should work at all and if we create the pot and then we wait for it to be ready, then we can look that the recording is now in progress. So we should now test our application, so we can, for example, load balance, the nginx or the lattice container, and then we can actually see that all ciscos or all code bar paths of that container application gets triggered now for test purposes. Let's delete the pod again now we finally got two additional profiles instead installed instead of just one.

B

Let's look at our second profiles within our cluster and then we can see that we have our demo redis and demo engine x profile successfully installed into two different second profiles.

B

So let's look at those profiles to ensure that they are actually different, and here we go. We can see both this calls are actually not the same for both second profiles, so the recording succeeded at all.

B

Now, let's delete those profiles again.

B

We can finally bring this to the next level because the operator is even able to record whole deployments if we, for example, create a deployment which looks like this- that we have two replicas of our engine x and redis containers and still have applied the same labels at epicos demo.

B

Then we can create this deployment and wait for it to be available within the cluster.

B

Now, if we remove the deployment from our cluster again, then we can look at four profiles indeed, so we got a profile for each replica, so they may differ because different load balancers may reach different replicas of the deployment, but we are now able to actually assemble our second profiles by just stuffing them and looking at our workloads. So this is our main strategy: how to create second profiles for actual real-life workloads- and this was my demo for recording sitcom profiles with the help of the security profiles operator.

B

What do we plan for the future of the project? Well, there are many things planned, but a few of them are just here. So the first thing we would like to do is we would like to introduce a note status for profile, reconciliation to avoid data races, so we are working on a fully transferring node status that determine if a profile has been reconciled to this gonad. So this helps us to avoid data races and make the mobile or implementation more robust, especially for large scale deployments further.

B

We would like to simplify the deployment in favor of automatism, so the webhook deployment should be part of a single all-in-one deployment file. We would like to add custom deployment logic to keep the rollout of the operator in sync. We are also considering to remove the assortment manager dependency completely.

B

We would like to work on audit logs for the score recording, so we already have the log file retro enricher implemented for the operator, and this is able to collect kernel messages directly, and we also plan to enhance this log enrichment mechanism to be able to record profiles directly from this data.

B

We would also like to add full features, say linux and armor support so right now the security profiles operator is targeting to also enable the linux policies too, and up almost report will come in the future releases as well and the last point, but not the least one is metrics.

B

Our plan is to provide a matrix endpoint for the operator to export security related information to prometheus. This gives cluster operators more details about the actual security policies and what is going on in that cluster. So we would like to love to see your contribution if you're interested and if you want to get started working with any of those topics, and with that I would like to thank you all for listening this talk, and now we are happy to answer any questions see ya.