Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CRI-O: Deep Diving into the Security - Sascha Grunert, SUSE & Daniel Walsh, Red Hat

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

CRI-O: Deep Diving into the Security - Sascha Grunert, SUSE & Daniel Walsh, Red Hat

Container Runtime security is one of the most important aspects of the Kubernetes ecosystem. Runtimes have to ensure that all security constraints provided by the end-users are met on the system, as well as provide strong defaults for less experienced users. CRI-O is a container runtime that prides itself in its focus on security and safe defaults. In this talk Dan Walsh and Sascha Grunert will dive deep into CRI-O’s security principles. They will present common container workload securing practices and demonstrate how the container runtime will apply those to the target system. The talk will cover best practices in SELinux, AppArmor, seccomp, Linux capabilities and namespace isolation techniques which make Kubernetes based end-user applications more secure. At the end of the talk, they will also cover a general overview of the current status of container runtime security.

https://sched.co/ZexY

A

Welcome everybody to our cryo security, deep dive, I'm sasha, I'm one of the hosts today and I'm a maintainer of cryo and one of the presenters today and I'm happy to serve for you as well, and I also have dan walsh on my uh on the presentation, who is also very well known, for contributions to container run times like cryo and also some other container runtimes like potman scorpio builder and many more.

A

So what are we going to talk about today? So, first of all, I would like to talk a little bit about the basics of container security, so.

A

So generally, we have to talk a little bit about how we want to have container security in terms of actual kubernetes, and things like that. After that, um I would like to talk a little bit about how we secure container images and, for example, container signing in container encryption, um and after that we can speak about actual container security features like linux capabilities, segcomp, selinux and also app armor.

A

And after that we will give a little look out into the next namespace changes in cryo, which are yeah. One of the upcoming topics in container runtime development.

A

So basic container security principles. um First of all, yeah. We have to choose the right amount of isolation when it comes to securing a container workload, and this can be very hard because we have to deal with different kind of layers of container security like we have security in run times like run c, and we have also security in terms for cryo, for example, and on top of that, we also have to speak about container security in conjunction with kubernetes.

A

So we have to speak about features like how to utilize seconds linux and even app armor to make our workloads more secure by default. So this basically means that container workloads have to run with the least possible privileges. So, but this doesn't only apply to container to running containers. It also applies to container images, so they have to be secured as well, and the container content, for example, for example, um the default execution parameters of the container images can be reduced to reduce the actual attack surface from a container runtime.

A

So distributing container images should also happen in a secure fashion, so we have to deal with container encryption and also container signing, but what is actually a privileged container? So we have, I found multiple definitions of across different projects like kubernetes and docker, so privileged containers have most of root capabilities on the host machine, so, for example, if we speak about kubernetes, then the pod sandbox shares the network pit and the ipc namespaces of the host a running privileged containers also disables features like app armor, zek, comp and linux capabilities at all.

A

So you have to be aware that if we run containers in privileged mode, then we can't use features like app armor or that comb in the same in the same manner.

A

Privileged containers also mount critical paths into the container. For example, the host devices and kubernetes, for example, um supports privileged containers by the pod spec, so we can specify a pod dot, spec and containers privilege security context, and then we can set privilege to true.

A

If we set just one container in the overall workload to true, then the sandbox will also run in privileged mode. This is root caused by because the pot spec doesn't have the privileged field and the security context at all.

A

So what is the least privileged container? We want to run so restricting pot and container security context to a bare minimum is actually a good approach, so we don't want to share the host pit or ipc namespaces, which are also off by default, so we are good to go.

A

Then we would like to run the workload as non-root user and in a non-privileged group, so this can be done by, for example, if you build a container image, then we can specify all that default users that we don't have a need to actually run the container workload as root user.

A

We also want to drop unnecessary capabilities from the container and in the best case we would drop all capabilities from the workload, but that that is not possible in every case, because usually capabilities span um and some syscalls, which can be more frank, granular adapted by, for example, like features like setcomp.

A

We also may want to disable the privilege escalation to forbid that higher privileges can be accessed by the parent browsers. This is on by default and I can just recommend to turn it off and then we can also apply a zaccom profile. The most container runtimes come with a default.com profile, but if we we are able to actually specify a custom segment profile for our workload, then we can drop even more capabilities or can also drop even more.

A

This calls we can also consider um adding app and app armor or the linux profile to the to the actual workload, for example this kind of overlaps with setcomp. But if we speak about app armor, then we are able to yeah limit access to their different paths inside of the container as well, and on top of that, we can also enable root. Only root file systems read only root file systems from a pod level.

A

It's just a great security enhancement as well and yeah on and on. I must say that reducing the attack surface of a single workload is more or less a try and error approach, so we can not be sure in every case that the workload still works as intended. If we just drop some capabilities, for example.

A

So, but there is more than just securing the workload, so we have different kubernetes features. On top of that which should be taken into consideration. So, for example, we can use service accounts, cluster roles or names based roles and cluster rule bindings or namespace rule bindings to strip down the possibilities the application has per default.

A

We also have to consider port security policies, which should go away in future kubernetes releases, but we don't have an adequate alternative for it, so we can consider using port security policies, and we can also apply network policies to the whole application, which makes the application all in all more secure.

A

Besides that, we also have some other security hardening possibilities on a note level in kubernetes, for example, container image signing features um and like enabled image, signing cryo and enabled image. Encryption in cryo will reduce that possible attack surface when it comes to man-in-the-middle attacks, for example, then the container images can yeah can can be encrypted as well, and we can also access limited access to lcd and, for example, we can also encrypt secrets at rest level, and there are many more features which are a little bit out of scope of this talk for now.

A

So let's focus a little bit on the cryo perspective from the cryo container on time perspective, and so the first thing so container images signing and encryption in cryo works out of the box. We added some. The new configuration option, which can be specified via the decryption keys path, is an option in which defaults to etc, cryo keys, and if we put our keys into that directory, then cryo supports oci image encryption. This encrypted images are usually visible via the media type plus encrypted and pulling designed.

A

Images is also possible in cryo, so we have the container configuration in, etc containers and we can also enforce signing container images via portal policy.json.

A

um We just let me share my screen. I prepared a little demo for you.

A

So here we go. What we see is that we have a single node kubernetes cluster and we have a running running cryo on that window. This is my demo and then we have a running kubernetes stack, so everyone cube ctl get bots. Then we can see that we have just running cube dns here.

A

So let's take a look: how container image designing works? If we look into our registries directory on my local machine, then we can see that we have a default file which specifies a default sig store for staging. So.

B

A

Stick store will be used. Signature storage will be used when we actually design those images, so they will be put in there, and we also have two other files where we can specify container registries and signature storage locations which are served from an https endpoint and, for example, for registry.opensuse.org.

A

We have specified a signature, storage, radio, c opensuse.org six store to enforce a policy like that. We can write a policy json file. For example. We can specify that in that json file, where our transport docker is specified, and then we enforce on registry.opensuse.org, add gpg and encrypt that key and the key path is can be distributed, for example by the distribution itself, and in our case it's the public key from registry.opensuse.org and yeah. If we now pull a container image from registry.opensuse.org, then we can have a look.

A

If should succeed at all in a couple of seconds.

A

And we can also.

A

Have a look into the cryologs and, for example, here we can see that cryo automatically tries to access the signature, storage and also down automatically downloads the signatures for the container images layer, and this usually works in a couple of seconds, and now we are done. We downloaded the image and yeah. If we, for example, now consider removing that image again and yeah just for demonstration purposes, we move the gpg key to a different location, and then we try to pull the container image again.

A

Then it should fail, because the policy.json would like to enforce the I copied it one. Second, I have to move it. That's not correct.

A

Otherwise the file wouldn't move. So if we go into edc containers gpg, then we can see that we just have as a backup image. Let me just remove the image again and then we pull it. Then we usually should get an error.

A

Hopefully this.

A

A

Yes, all right, so now we get an error message that none of the signatures were accepted, so the regular gpg keys or try. It doesn't work at all because it's not existing, but even if the gpg key is not valid or not valid for our signed image, then we would get a different or a similar error message that it's not possible to download the image at all. Now, let me just move it. I can do it later on.

A

So then, let's talk a little bit about running, secure workloads with cryo and secomp, so setcomp graduated graduated to general availability in kubernetes 1.19, which should not well. It should be released in the next couple of days. If we speak about kubecon europe, and previously this will be would be handled via pod annotations and from 1 19 on we have a dedicated field on the pod security context or on the security context on the container level.

A

This means that we have now a security context, then zaccom profile. Then we can specify a type like runtime default or localhost or unconfined, and if we specify localhost, then we can also set the localhost profile to a local profile. So those local profiles will be looked up in the cubelet into the cubelet directory, which should be usually wallet, cubelet, it's the cubelet root and then slash secomp. So if we put the profile into that directory, then we can usually reference them directly via myprofilename.json.

A

I can demo that for you as well- and we are now on the same cluster so as already shown- let's just have a look at our zek comp example, one. So what we have here, we have deployment nginx deployment.

A

Then we specify our security context and our zaccom profile in our case runtime default, so cryo ships a default con second profile, which is included in the cryo binary and but also with some distributions distributed via a file called an etc container. Second json. Well, this is a pretty pretty good setcom profile, but you can see. For example, we have some. This calls which are allowed by default and not every application may need all those. This calls to be executed in the in a normal way, but let's try it out.

A

Just let's create our nginx deployment and the containers are now in container, creating state and now they are running and if we run cry ctlps to get an idea about which containers are running. So we have both here and everyone try ctl inspect, and then we can specify the container id. Then we can see that the whole.

A

Then we have a reference to the whole second profile directly in the inspect, because usually the container runtime, like container run times like cryo, pass down the second profile to the underlying container runtime, which is run c in that case and then run c applies the second profile.

A

So that's this one.

A

So this works pretty cool and let's just delete our deployment again, because usually we can't change the socket profile on the fly. So we have to would have to recreate the workload.

A

And if we now look into our second example, then I just did more or less the same. But now we add the security context to our container. So we specify now a second profile from localhostpath and we choose our demo.json now demo.json is located in.

A

Walletcubelet.Comdemo.Json and it's basically the same the same as the default as the default profile, but it just forbids rmd on the arm. There is just a syscall to remove a directory, so it shouldn't be harmful for our nginx application, but let's just try it out.

A

We just recreate the workload.

A

Wait for the containers to be created and yeah, then, let's just exec into our one of our nginx pots,.

A

And here we are and now yeah I'm root user inside.

A

This container, but I usually should be able to, for example, create direct a directory test, but I'm not shouldn't be able to remove the directory again. Yeah and.

B

A

It says that the operation is not permitted, because the second profile enforces that and yeah.

A

If we speak about second at all, then it's not that easy to write those profiles at by hand, but there's a little nice fancy project called the ocis.com bpf hook, which can be used or which is just the general oci hook, which is supported by cryo at all, and this oci hook can be used to trace this calls and generate second profiles out of the box.

A

So I can really recommend to play around with it and it works also very well with podman and for example, if you run it with potman, then you can specify the annotation io.containers.tracescall and then some input, it's not necessary to specify input path, but it's necessary to specify an output path, and then you will get the second profile generated into that path.

A

So, let's what other options do we have to actually secure our workloads? So the next thing I would like to talk to you about is app armor, so app armor is still a better feature in kubernetes, but we also try to also consider to move it to ga very soon, maybe in 120 or 121, and right now we have to handle via pod annotations.

A

So we have to specify an annotation at the port level, which is called container.app.armor.security.beta, and then we have to reference the container name and the profile ref, and this profile ref can be run time default. It can be also localhost profile, name like we do in seccomp, and then we can also set it to unconfined, which disables the feature at all.

A

So I just would like to give you a short demo about that as well, because I'm running on a system which supports app armor and let me just.

A

Delete my second workload and app armor works a little bit different than zaccom, so app armor profiles are usually stored in memory and to I created a profile called container engine x and etc, app rmod and every profile in etc. App on modi will be automatically loaded if I boot up the system because app only starts automatically so this profile looks like this and we can't or we don't only have to can limit network uh restrictions to the to the actual workload.

A

We can also specify a root root level permissions and can yeah also, uh for example, drop capabilities down here. So we have a lot of flexibility in that profile, but it follows a dedicated syntax, so we also have to test it or create it manually.

A

There is also something like an app armor recorder, but I don't want to talk about this at that point, so this profile seems pretty pretty well um now, let's just check if app armor is running, so we can run our aaa status and then we look for nginx and then we there we can see that our nginx profile is successfully loaded.

A

So if we now look at our app armor demo example, then we can see that we have the annotation here we specify container app armor security and our container is named engine x, minus the ctr and that's the container down here and we use the localhost nginx profile.

A

So let me now apply that profile, edit workload, sorry, and if we now, we can see that it's running again. Let me just.

A

Check the containers running and if we inspect that container.

A

And then we can see that the app armor profile engine x has been loaded successfully. So this is pretty nice and usually we can also go into the let's just exact into our engine x container again.

A

And if we look, we take another look at the profile, then we can see that we can't access yeah, slash spin there, so I'm just rooting to that in that container. So I can, I usually have root level access. I can create directories like test and I can't go into bin and create their directories like test yeah and it says permission delight and usually the audit log and do it does give us an indicator.

A

uh Audit log give us an indicator about what actually happened on the system and we can there we can see. For example, there is an audit log for nginx and we try to create yeah our root level directory test and this actually works all right. So that's it from my demo for app armor and now I would like to hand over to dan who speaks about capabilities.

B

Okay, great job sasha, um so basically this part of the talk, I'm just you know, uh sasha in the beginning of the talk talked about.

B

We really recommend that everybody run containers without being root inside of the containers, um and you know for the most part, I believe that almost all workloads should be able to run without being root and that's probably the most secure way, you're able to run it, but in certain work certain workloads people need to be run as root or need to have multiple uids inside of a container and need some of the root capabilities.

B

So the idea of linux capabilities is it divided the power of root, so sort of everybody understood this root as being all-powerful and many many years ago, the linux root capabilities. Linux kernel basically broke down root into a whole series of different capabilities.

B

So the uh um the interesting thing about containers- and this this was sort of developed by docker a long time ago. Is they figured out that you could run almost all workloads uh by default with um you know, just a small group of the capabilities, so the power route was broken into 37 different capabilities, but actually docker sort of standardized on 14 capabilities that it ran for containers, and we sort of you know latched into that, and most container engines now have settled on those 14 capabilities.

B

um The the problem with those 14 capabilities is um nobody knows what they are right. So it's sort of you know you you, even even I who deal with this a lot. I couldn't probably answer uh tests and get all 14 right. um So uh what what I have done is I've actually looked and so here that here are the 14 capabilities that are allowed by default um in most container engines. But when I sit down look at these, I look, you know what is the history of some of these?

B

You know why why these on so, um for instance, this one is audit right and audit right his ability to write to the auditing subsystem. So sasha showed you, you know looking at app armor errors in an audit log. So this the auto right allows you to modify parts of the uh auditing system to be able to manipulate the auditing system.

B

So why would we have that on by default in containers, and it turns out that when containers were first becoming popular, most people thought of them as as virtual machines and what they wanted to do? Was they wanted to run ssh daemon inside of them?

B

But when they'd launch a container with ssh demon running inside of it will blow up because the ssh demon, once someone tried to log into a system, would basically write a record that said dan walsh logged into the system, and that would be blocked because it didn't have audit right so that got turned on by defaults for all all containers and of course almost nobody runs this as hdmi inside of containers now so, but we still have that on by default.

B

The next one I want to look at is make node so make node is a capability that allows you to create device nodes inside of containers. So you know, but again you know when we look at cryo and kubernetes we're looking at running containers in production, and you really don't want to allow containers to create random device nodes.

B

What we want is the container engine to create the device nodes and not allow them to just create random device nodes on their own, but we allow this by default and the reason we did that is because originally people were doing lots of installs and builds and and certain packages, I believe from ubuntu. You know actually created device nodes during the installation procedure, um and so really you know when we're running containers of production, we're not building container images, we're not installing rpms we're you know. So why do we need make node on by default?

B

The next one I want to talk about is net raw and that rar is a capability that allows you to create any type of ip packet, so tcp icmp, any type of ip packet can be created and then put out on the on the vpn and the problem with this. Is it really gives an incredible amount of power and and has led to some escapes where people were able to break out of the virtual private network by forming packets in certain way um and and the real reason for net raw being on by default?

B

Is that so people could actually execute the ping command? So when I execute a ping command, I created an icmp packet and and put it over. So when I went into a container I'd want to create that. So again, it's it's an incredibly powerful capability that really we don't need on by default, um especially for containers in production and and the linux kernel has other ways of running, um allowing you to create icmp packets.

B

So you can continue to ping, um so we actually, you know, enable that by default in cryo, instead of net raw and then the last one is sister root and systrood is a capability that allows you to create a route inside of a container. But you know a container is really kind of a super powerful to root. So why you know most people aren't creating roots inside of containers. So why do we allow that by default?

B

um So so when um in cryo, what we've done is we've looked at this original list and we basically said: let's, let's get rid of some of these ones that are less less necessary and and we're able to run almost all of our containers without these four capabilities. So we can drop down to you know just basically nine or ten capabilities that we run by default uh for our containers when you're running it as root um the next thing um and obviously- and you know if capabilities are fully available inside of kubernetes.

B

So if you do find a container that needs additional capabilities or something that's not allowed by default, um then you can, you know, add those capabilities through your uh your kubernetes yaml file. So you know we can override those. If, if you need them, but really you know at base level, we want to run everything as as tightly as possible from a container point of view.

B

The next thing I'm going to talk about is sc linux and and sc linux and app armor are two ways of doing: what's: called mandatory access, control or control on systems, one difference between the way sc linux works and the way app arma works is that app armor is controlling mainly what's going on inside of the container and what selinux does is it wraps?

B

The external part of the container, but the really great thing about sc linux is that just about every type of file system breakout when the process is able to break out of the container and get to the host or attack. Other containers has been blocked by sc linux out of the box, um so quickly people, you know, tend not to really understand what sc linux is and a few years ago I wrote a coloring book on that and I'll just go through it real, quick and try to sort of explain.

B

What's going on with sc linux and uh think of every you know, sc linux, labeling system, every everything on the system has a label. So let's say there was a cat process and a dog process, and so we have a process type, a type of a cat and a type of a dog, and then I have food.

B

So every object every you know, directory file system. Everything on the system is, is labeled, um you know with you know, a label, and basically we have label is cat food and we label it. One is dog food and then we write rules into the lace kernel that basically says allow a cat process to eat cat food and allow a dog process to eat dog food, and the basic idea here is that everything that isn't allowed is denied, and this is really exactly what an esee linux rule looks like when ridden to the container.

B

So now, if the cat tries to eat the cat food, it's allowed to eat and if the dog tries to eat the dog food, it's allowed to eat that, but when the dog tries to eat the cat food, it's blocked.

B

So, from a container point of view, this means that when I'm inside of my container I can you know, we basically launched the process as being the container type and we launched the content inside the containers being container files, and we write a rule that says the container can read and write container files if the container process broke out and tried to read the home directory or try to read any other directory on the system.

B

If that content is labeled as container content or container files, then it will be blocked by default and that's how essie linux is blocking. All the you know, potential escapes of file system escapes and in containers, um so those are sort of the hard-coded built-in uh tools that we talked about your setcomp um capabilities, se, linux and app armor, and some of the other features that sasha talked well.

B

One of the things we're really trying to finally accomplish is actually to get username space, so I mentioned earlier that you know people want to be able to run route inside of a container or they want to be able to run multiple uids inside of a container, and so we have a pull request right now into cryo that hopefully will get merged soon and and will become a feature to allow you to specify a um a an attribute that allows you to basically enable usernamespace.

B

So what happens? If you need root inside of a container, you know and what happens if you need more than one uid and some use cases where this becomes important? Is things like building container images, so most people that do builds inside of kubernetes right now need to have root or need to have multiple uids inside of the container, so they end up having to run privileged containers or they end up having to do things like leak, the docker socket or some other container engine socket in and basically break break out of confinement.

B

Another thing would be: you know, they're needing to launch multiple processes. Inside of the same uid, so by enabling username space username space allows us to map root and other uids inside of a container as non-root on the host and and what we've done in in trio now with the latest.

B

Pull request is to allow us to basically run different containers, different workloads that require root or multiple uids, to be able to run as root inside of the container, but if they break out they're treated as like non-root on the rest of the system, and we've enabled that from the top you know from the highest level um you know in in cryo all the way down to container storage and with handling some of the re-labels and images. So this is a really cool feature. That's going to be coming very soon um to cryosystems.

B

So at this point you know we're racing to get this talk done, and at this point we could open up for any questions.

A

Hello, everybody so here we are live and then color and first of all, I would like to apologize for not uploading the slides to sketch. But I did share the link in the slack channel and you can access them via that.

A

Okay, then we have a couple of questions left.

B

So one of the uh questions earlier was uh talking about basically talking about an exploit loud because of cape uh cap. Make node- um and I you know said in the talk cap- make node should really be turned off by default. Is there's almost no reason for that to be on, and if you need additional capabilities for your containers, then you can specify that in the kubernetes gmo file.

B

Another question about what is the relationship between cryo and podman, so podman is a container engine for running sort of traditional. What people think of as the darker workloads cryo is a is basically a demon process for running kubernetes workloads.

B

They both share the underlying mechanisms for pulling images and storing images, so they share a lot in common, but they both can run totally independent of each.

B

B

B

So the question on is: there's a question on se: linux is there a new seo linux syntax for selectors configuration to distinguish between in container object processes and things outside of the container, so in in from an essay linux point of view, we'd like to run all the contents inside of the container is a single label.

B

We believe that this is the way microservices should be written, so we treat all content inside the container as one label and then, if they break out, they treat it as a as a container and therefore they're only allowed to read, container, read and write container content.

B

Each container has a separate what we call mcs label, but basically it's a unique identifiers for specific containers. But right now we.

A

B

Multiple labels inside of containers, this.

A

B

Difference between app armor and sc, linux and app armor is is actually providing security or separation inside of the container, whereas the linux is providing separation outside of the container, so they work a little bit differently. In that case,.

A

One question is: um what is the relationship between docker and cryo? So does it work as the same as an as it does in potmen, yeah, cryo and potman share common source code and dedicated libraries, so all security patterns which apply to potman should also apply to cryo, and there is not that big relationship to docker anymore, but, for example, if we look at a default second profile, then we can just say that historically, it has been inherited by docker. Yes, that's it's a historical relationship.

B

Yeah, the main comparison between docker and and cryo is around container engine container images, so um grio and darker both support, uh what's called oci images, and um so, if the input so anything that exists at any container registry, you know dr io quay dot. Io uh auto factory um is supported because there's a standard around one of the container images and also the way we launch containers. We also use a standard called the oci runtime specification.

B

I am both using right now run c for launching containers. So the way you the containers on the input and the containers on you know when you execute them, are both run the same exact way between docker and cryo. So therefore they can support all of the same content and podman as well. We've just been notified that we're running out of time. Now we will be available on the slack channel to answer any other questions that people have or if we didn't get to a question here, just ask it on the slack channel.

B

So, thanks for listening and uh goodbye happy to see you there.