Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing Kubernetes Applications by Crafting Custom Seccomp Profiles - Sascha Grunert, Red Hat

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Securing Kubernetes Applications by Crafting Custom Seccomp Profiles - Sascha Grunert, Red Hat

Applying seccomp profiles to Kubernetes workloads is one of the most efficient ways in securing containers. The profiles have to be created with care and need to be maintained over the complete lifecycle of the application. This manual effort causes that many applications either stick to the runtime default profile or turn the feature off at all. In this talk, Sascha will demonstrate how to create a custom seccomp profile for a specific containerized application. It will cover the basic techniques of collecting the required syscalls by hand, and also advanced ways of utilizing eBPF and automatic audit log tracing. The session will also discuss the drawbacks of relying on automations. In the end, Sascha will show how to create multi architecture profiles and utilizes in-cluster enhancements like the Security Profiles Operator to create an application specific profile. Join this talk to learn more about seccomp in Kubernetes and how to secure your applications!

A

Hey everyone welcome to my talk about securing kubernetes applications by crafting custom. Second profiles: I'm sasha and it's a pleasure to be here today.

A

What do we want to see in this talk? So? The first thing I would like to tell you about is a brief history about setcoming kubernetes, so we reflect the development progress of second in kubernetes and we will see what the current state of second kubernetes is. After that we will craft a customs account profile by hand. For this we will use a real world example, and we will use two methods like tracing the logs and running recording second profiles by using ebpf.

A

After that, we will speak about how we can automate away those manual efforts, so how we, how could it be possible to integrate our recording into a ci cd system, for example, and how we can get rid of all those manual steps in between and after that, we will speak about the bright future of a per default, more secure kubernetes. So how can we make the kubernetes more secure by default by using second profiles, for example?

A

Second, is a cisco interceptor feature for the linux kernel? So it really works like this. So you want to do a this call on your application and then you can decide which, which action do you want to take with this syscall? So, for example, you have a different list of actions available. You can, for example, say that you will only want to lock this action. You want to arrow out, or you want to allow this cisco, for example, and this can boost the application security by limiting the list of allowed syscalls.

A

So you can maintain a list of allowed cisco's. You can also maintain a list of blocked syscalls, and you can also find greatly define what the error code, for example, should be in case your disallows. This call- and this has been added to kubernetes a long time ago, so we also have a default security profile.

A

This has been defined by the container runtimes, but kubernetes requires that such a profile exist in every container on time like container d, cryo and docker second was going to ga, which means generally available since kubernetes 1.90. So we now can consider this feature as stable since quite some releases, and it also supports most linux kernel versions.

A

There are some constraints, or there are some environments where second may be not supported. So we also have to take this into consideration that we have, for example, a decent architecture or linux distribution. It does not support secop at all, but in general.

A

This second fields are usable by a dative field in the security context or a deprecated annotation. This can be done by the container or by the pot. So we can apply profiles to parts which then inherit to containers, or we can specify those profiles on a container level and overall goal is to remove the annotation support in 1.25.

A

I also have to mention that all workloads run unconfined by default, which means that saccomp is disabled for them. We have a special feature introduced in one of the recent kubernetes versions, which is called staccom default, and this allows us to change that behavior, for example, by using the default profile for all workloads on the specified node. I have to mention that this feature is alpha for now, so it's not enabled on every node per default, and one drawback is that default profiles may differ between container and time.

A

So it may be possible that if we have clusters with mixed container runtimes, then it maybe may be overall behavior may be different between one node and the other.

A

Those custom profiles can be defined as json files, so there is two main issues with that. So, first of all, they have to be distributed to all nodes to be available for the whole cluster, and the container runtimes have to apply them from disk. So we don't have an automated way in kubernetes to distribute those profiles to each node and then load them from disk.

A

How can we craft custom second profiles by hand? Our overall goal is now to understand how can second profiles work and which possibilities do they allow and then how my application behaves and which this calls it executes during that time, and we have to collect a list of this calls which are required to pay it allowed.

A

Additionally, so we have to take the whole cluster separate setup into account and to not create two restrict drift profiles, which means that different architectures may require differences, scores and also different architectures allow a different differences calls and the workload configuration has an influence on the executed syscalls as well, which means that having a workload configured differently, for example, by setting some options to some configuration files or something like this may also lead to executing different code paths. And this will automatically lead in different syscalls, which has a direct influence on the zaccom profile.

A

The example project we would like to choose today is cube object proxy, which is basically just an http proxy, but it can perform our back authorization against the kubernetes api, and this allows us to restrict requests to the api at all in network, uh isolated environments, and it has been developed initially to protect prometheus metrics and points. So it's possible to add an additional additional layer of security to usually plainly exposed prometheus metrics, and this is a single container deployment which simplifies.

A

This is called tracing at all having an application which consists of multiple containers or even multiple deployments, like a microservice architecture, would require us to create second profiles for all single deployment and all single containers, and this would be probably too much for this demo here I'll link the project here, so you can check this out if you want to learn more about cubarbic proxy, so how to actually record those scores. The first method we can use is tracing the con locks.

A

So this requires audit d, or this look to be installed and configured on the system, which is something which may be possible across all distributions. I mean there are many distributions, like plain ubuntu, which only have syslog, and there are also other distributions which ship order d in a good configured way by default.

A

Nevertheless, we have to consider that there is a rate limit not only for other d, but also for this lock, and we can set the two sis colors here to disable the print k rate limit and the rate limit burst. Otherwise, we will probably miss some scores during the logging of extensive applications.

A

We mainly rely on one single linux. Kernel function to record all this calls, and this is audit. Second, you can see this function here directly from the latest version of the linux kernel, and the only thing it does is creating the log string containing a bunch of information like the cisco itself and the architecture and then printing that to the kendall buffer.

A

But this comes with a bunch of limitations, so, for example, this course will only get locked if they got requested for logging, which means per default allowed and also blocked. Those calls won't be locked at all. So generally, logging has a high performance impact. I tested it with a bunch of applications and it really slows down the overall application because it blocks every time we have to lock a line, and there is a special setcomp action available for logging.

A

So I already mentioned that there are multiple actions available and creating a second profile on disk, which is a bunch of json for our profiling. Applications is as simple as this line, so we just have to specify a default action, and this action is second back to log, so which is the actual value here.

A

We also have to double check if logging or lock the keyboard lock is part of proxy's kernel set comp actions lock. Otherwise we would not lock anything at all.

A

Now we have to put this profile into the default location for the cubelet, to look for second profiles, which is smartlab cubelet. Second, we name our file log.json and for the example of cube orbic proxy, we'll change the security context of the second secretary context of the deployment to specify the second profile localhost and then point to the relative path, which is log.json.

A

Now we can finally run our demo application and trace the bar log audit audit log in my system, so we have to be aware that those files also rotate at a common size. It also depends on the configuration of audit d and and order d itself can also be configured to have a rate limit.

A

We can double check that by running suited order, ctl minus s which prints out the configuration for that and then the the question would be. How can we link the workload which is currently running to the actual output of the audit logs? So the only thing we can do is we can just yeah. We could also.

A

We have a process name, for example, but process names are not unique on the system. We have to choose the unique process: identifiers of the pid and with chris ctl we could look for our container on the local machine like crystalps and then look for cube object proxy. Then we get a container id and by using the container id, we can use cryctl, inspect and then grab for the info pit, and this gives us the process id of the workload of the container.

A

And when we have that, then we can obtain a list of syscalls. So, for example, if we run pseudocad wirelog audit or the d-log and then look for the type setcom, because we have noted the multiple types available, but we only are interested in the typeset comp.

A

Then we additionally prep for the pid, and then we have an automatically resolution of the syscall name, which is written in uppercase here and if we sort those uniquely and also now remove the new lines here that I can print it out, then we will see all this calls available or required for running the cube, our back proxy binary.

A

So, for example, we can see we need bind, we need clone and yeah. We also need socket, for example, because it will actually create a socket and a bunch of other ciscos like listen and that's interesting, but we only have started. The application for now means that this list of ciscos reflects only the solder of the application, not its actual usage. So it's really important to gather this course for all available code paths.

A

So what we now do is we create the client we created example, client, um and this will then connect to the cube orbit proxy, we'll try to gather the metrics and then we can see if we do it do it again. Collecting this is called that a bunch of ciscos have been added to our list, for example, connect and get peer name so means that actually using the application will trigger the code paths, and this is crucial for this overall approach.

A

So what we can do now with this list of syscalls, is that we create an allow list of possible syscalls for this deployment. So we use a default action of arrowing out.

A

We can also specify the arrow return code if we want to and the arrow no error code like which is e-perm in this case, but we could also choose enosis, for example, and then we specified a list of ciscos which are allowed so disallowing everything and then having something like an allowed list is always the most secure approach rather than going the other way around, and then we add those syscall names to our list and after that we can use this profile as a new second profile on disk, and this is something I would like to demonstrate.

A

You know so, first of all, let's double check that my system actually is able to lock anything by using order d and yes, so the second actions, locked uh skill process, kill, threat, um trap, aeronaut things like that, and also the most important part is the lock here now. What we can do now is we can create our profile, which is called log json, and this profile has to be copied into the wallet dublin second directory.

A

Otherwise it wouldn't work at all. So if we now double check our deployment, then we have to ensure that the security context of the container for the cube outback proxy specifies a localhost profile which is called log.json, and this is already the case. We can apply it and then we can wait for it up and running, which is already the case now and now it already should lock some this calls into our log audit audit.log.

A

So if we now retrieve the container id by using, create ctlps and export it as like this, then we can also export the process id by using crash. Ctr inspect like this, and the process id should be now available. Like this, but what we now have to do is we can just grab our audit locks. We can do it like this, because we have multiple order blocks and then we have to grab for the type second additionally grab for the process id, and we can also double check like this.

A

That there are, those entries are unique, and here we are. This is a list of this course. We have right now and we can also. We also have to actually use the client. So we as a client example available for kubernetes proxy, which creates a job, and this job will run to completion like this.

A

So we can just see that it actually works, so the authorization itself against cubic proxy has been worked, and if we now run it again, then we see that we have a bunch of moses calls and those two scores can be the base for our actual second profile.

A

So let's modify this log json profile and use pc5, for example the example we have in our demonstration.

A

So this profile now contains 25 syscalls and if we copy it and if we copy it into our profile.json, for example, and then modify the deployment to point to this profile, dot json, then we can verify that it actually works. Then reapply the deployment again, and we can see that it is now up and running by using our recorded profile.

A

So, let's add some thoughts on this overall approach, so creating profiles via deluxe can be slow, especially when we consider using it in the cicd based automation. So if we consider having a huge test suite of end to end tests, which already takes a bunch of hours to run, then we probably will further slow it down by using the audit logging and all nodes have to be pre-configured to not rate limit those logs and especially gathering all application code.

A

Paths is really the hardest part here, so we have to ensure that we test that that we test all use cases accordingly, and there is another way of doing this by utilizing eppf.

A

So ebpf is a technology which allows us to run code inside of the kernel by loading it dynamically, and this supports a bunch of trace points. For example, we have this rows. This calls enter trace point which gets executed for every syscall on the whole system so and even before, we want to do that syscall. So this provides us some basic mitigation point for the syscall and but we have to be aware, that is always runs in global scope of the whole system.

A

So we have to correlate the information of the process to the container in the same way as we have to do it with the audit logging, we can use tools like ppf trays, which already allow us to collect the required data. So bpf trace provides its own abstraction language on top of ebpf and, for example, we can run bpf trace and then select the trace point rows.

A

This calls this enter and then we already pre-filter for the application name cube, object proxy and what we then print is we print a pid, and this is called id and if we run an application like this in parallel or even before the cube opec proxy has been started, then we can start the cube object proxy later on, and if this has been done, if the global proxy is teared down and we have run all our tests, then we can grab like the pid into the output which provides us.

A

Then a list of this call numbers like here, so there is no automatic resolution of the syscall name. We only get the number on the local system.

A

This is reason because those syscall numbers can change from system configuration to system configuration, namely the architecture of the system. Now we have to convert those syscall numbers back into the actual name which can be done by the au syscall binary, which is part of audit d. So if we dump all ciscos, then we see that we have the number correlated to the actual name of the syscall, so that gives us standard profile, and then we can apply this profile to the workbook itself.

A

It is possible to create setcom profiles manually without having to write an ebbf application from scratch by using this approach, um but it would be also possible to create an own eppf application. For example, there is the golang binding lib ppf.go.

A

Then we could correlate the information of the process id to the container. We are using by using the c group path on the local machine and then collect the data directly at hog, but this sounds complicated. Doesn't it so? My thoughts on this overall approach are creating. Profiles would not affect the system. Performance in the same way is logging, because ebpf is really fast.

A

We would still have to overcome, so we are in the kernel space and we would still have to overcome the performance drawback when moving the data like the syscalls and the process ids back into the user space. So this has a performance impact, but not in the same way as the logging. Yes and all nodes have to be reconfig configured to contain either the custom, uppf application or the dependent tools. So we can't just use plain kubernetes.

A

We have to use ppf trays or compile the ebbf applications and ensure that the eppf application runs on the node and things like that um and gathering all application code path is still extremely hard, but there must be a better way in doing this right, and this is actually the case.

A

For example, the security profiles operator is an operator which focuses only on security profiles for second c linux and app armor, and this provides an automation around lock and ebpf based profile, recording it automatically traces, the logs at the right time and that extracts the data means.

A

If I run a workload then I can pre-define that I want to record this workload and then the security profiles operator will take care of tracing the logs and gathering the data, for example, correlating the process id to the workload and it can also leverage ebpf to recode those profiles, for example, for performance reasons, for example, and this also automatically correlates the workload to the underlying process, and it creates second profile. So if a recording has been done, namely a workload has been removed, then it will automatically record a second profile based on that workload.

A

This second profile will be represented as a second crd, so it's even easier to handle it, and if we have a second crd, then it automatically reconciles all those profiles to all nodes. So we can have a workflow where we have a professory a profile and then automatically reuses it within a cluster without relying on a single node cluster. So the distribution of the profiles will be taken automatically, which is really great.

A

How does the eppf recording and the security profiles operator work in detail? So, first of all, we have to specify a custom resource definition which defines that we want to record a workload. This will happen by using a standard selector and if the profile recording crd access and we create our workload matching that selector, then the security profiles operator will use a web hook to add a recording annotation, and this is the only job of the profile recorder.

A

So adding this annotation will indicate to the bpf recorder, vr grpc, that we now want to start a recording and this loads automatically loads in libpf application into the kernel which doesn't require recompiling the application itself, because it uses ebpf query, which means compile ones, run everywhere to start the application locally, so it loads the eppf program, which also uses this is entertracepoint and this program records.

A

This is calls for every pit on the local machine and it throws an event on the pid which which will be used by tracking the mount namespace, because the mount namespace is usually something which is really stable across a container, and these events will be used by the event processor. In the ebpf recorder.

A

It tries to get the container id for the process id by using the local c group, and then it tries to find the container id in the cluster, and if this has been found and it looks for the profile recording annotation and then it starts tracking the profile for the pid and the amount namespace or this is the overall loop. And then, if we stop the workload, then the profile recorder will actually collect the syscalls from the bpf recorder and the ppf recorder will take care of automatically unloading itself.

A

So the bbf program itself runs only during the life cycle of a single recording or yeah. Multiple recordings are probably all possible as well.

A

I would like to demonstrate that to you. So if you look into my current kubernetes cluster, then we can see that the security provides operators up and running and that we also have salt manager deployed, which is a direct dependency if we don't have any other static effect provider available within the cluster.

A

The first thing we have to do because it's disabled by default, is to enable the ppf recorder and to do this, we just have to patch the security profiles operator, daemon configuration which is called spod and over the spec that we enable the ppf recorder and set this to true.

A

So if we did this, then we can see that the security profiles operator will take care of rolling out itself again with the new configuration and after a couple of seconds, the spod instance should be up and running again.

A

If we now look into the into the logs of the ppf record or container, then we can see that it does some sort of self check before it actually starts. So it gets the ppf. It does a ppf load, unload self-test, it loads the ppf module and it tries to attach the trace point and then it if this are all is successfully, then it unloads the module afterwards itself- and this is great because this already provides us a feedback if the ppf module is working as intended.

A

So the security profiles operator ships a default example for recording statcom profiles by using bpf, and if we look into this example, then we can see that we have a special kind profile recording available for the security profiles operator, and we can give this a name which is test recording in our case. So this will be later on the name of the recording itself or be part of the name of the recording itself, and what we want to record are second profiles. So for the recorder bpf right now, we only can record second profiles.

A

The lux recorder would also support recording the linux profiles. By the way we use a pod selector which matches our labels app equals alpine, and this is the overall indicator for the recorder to match any part which contains that label will be recorded.

A

So let's apply it.

A

And then we can also double check if it's available.

A

Yeah our test recording is available here and what we can do now is we can run a workload which is using an alpine image and contains the label app equals alpine.

A

And if the container is up and running, then we could usually do some tests or execute some ciscos. For example, we can create a test directory, test1 and also test2, and if we exit this container again, then it will be automatically removed and after a couple of seconds, the security profiles operator, ppf recorder should also return the the profile for the track. Pids, you can see the compass h command and the mount name space related to it and also see that all sub commands, for example, mkd as a sub command of the s8 shell.

A

If I run it within it and it also tracks the amount in space of it and that's equal and if the result has been returned, then the ppf recorder cleans up itself and stops the module itself for security purposes.

A

So now the second profile should be available right. So if we look for our second profiles, then we can see that the test recording alpha profile has been installed and is available on all nodes within this cluster, and we can also look what the actual install path is. So the operator itself creates a namespace operator and then it uses the namespace which is default in this case, and then it creates the json file for the second profile.

A

And if we look at the profile itself, then we can see that it is an allow list, as I, as we already did, for the log recording. For example, we have a default action which is arrowing. We also have the local architecture available here and the list of allowed this course, which is only the list of really necessarily allowed ciscos, and you can also can see that we have mkd and mkdir ad here available within this profile.

A

So general thoughts on this approach, so we get mostly get rid of all manual collection obstacles, so we don't have to pre-configure the local node. We just have to deploy the security profiles operator and that's it, but gathering all application code is still the hardest part here. The integration into a cicd workflow would allow us updating, statcom profiles with the application lifecycle.

A

So we have to consider that second profiles change when application code changes, and this could be integrated into a whole cicd workflow for collecting the profiles and then distributing them as well by upgrading, because the security profiles operator can also reconcile those profiles on each node, you can also take care of updating them. On the other side. The security profiles operator could also be used in production to distribute the profiles using the crt.

A

Let's speak about the bright future of a per default, more secure kubernetes, so the second default feature should be graduated to beta and kubernetes 1-25, but this means that also it won't be enabled by default, because since kubernetes 1.24, those beta features are not enabled per default anymore. So creator, aging, the feature to stable, would gain us a security boost in kubernetes.

A

um The plan is also to make it api aware so that we have actually representation of the sake of the use decom profile, not only for the cubelet, but also for the end user and then handling those upgrade paths is probably the most complex part. So we were thinking about upgrading only or enabling the feature only for new workloads and existing workloads won't be touched at all, but, for example, downgrading again would be then even harder than upgrading.

A

You could help us um by making kubernetes music by default, for example, by using the custom second profiles or by using the security profiles operator or just sticking to runtime default for all your applications and other other than that. It would be really great if you would try out the second default feature in kubernetes and provide some additional feedback about how it behaves. If you enable second default and everything works out of the box, then you can also choose runtime default for your application as well, and that's it. Thank you for listening to this talk.

A

um I really appreciate your feedback and let's have a chat about this.