Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: On the Security of Copying To and From Live Containers - Ariel Zelivansky & Yuval Avrahami

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

On the Security of Copying To and From Live Containers - Ariel Zelivansky & Yuval Avrahami, Palo Alto Networks

Nowadays mature container platforms (such as Docker, Kubernetes and LXD) provide users a way to extract files from a running container. There are several different design approaches for implementing such a copy feature. In this talk, Yuval and Ariel will present the ups and downs of the different implementations with a focus on security and possible vulnerabilities. Throughout the presentation, different vulnerabilities that affected the major container engines will be reviewed. A live proof of concept of a vulnerability in the Docker copy command will be presented.

https://sched.co/Uabe

A

So welcome to attack. We have very happy to be here at sunny, San, Diego or not so sunny today, but yeah, so we're here to talk about the security of copping out and inside two containers. There have been in the past few years, many vulnerabilities in copy implementations for many different container engines and we find it's.

A

It might be a great opportunity to come here and discuss the different vulnerabilities that we saw and what we can do for the future, and so myself, I'm I get the reference key, and this is your Bolivar ami hi guys. We work for Palo, Alto Networks and previously at a small company called twist lock with a we have a security research team where we try to find bugs in the cloud native environment and runtimes and programs that are used in cloud native environments. So we have a bunch of TVs.

A

We found we disclose everything, write some articles about what we find basically trying to make cloud native most secure for everybody and food that the last we that we've looked into existing vulnerabilities and some what we've found we saw that CP is actually some place where there have been too many vulnerabilities for something that is supposed to be very simple.

A

So with that the gender for today we're going to start with some background about both continents and the CP command, how it is used, who uses it and and what's the purpose of it basically and then we'll just dig into some of the vulnerabilities we've seen that we think are most important and interesting for us to learn, inform and we look at portman, docker and finally kubernetes, which had some of the worst sipping abilities in it and we'll end with some takeaways. Basically, what we think may be the best approach to implementing CP from scratch.

A

Now that we've learned so much from different vulnerabilities.

A

B

Can you guys hear me yep? So, let's start by talking about the containers. So, as most of you probably know, containers are restricted processes suited to a separate file system and but I think the best way to really understand what it means is to see how you start the container. So, let's see how docker starts the container Dockers uses a LAN C, which is the industry standard tool for all the low-level stuff.

B

I have actually run in the container and it spawns the helper process called run C in it and what can seen it does is start building the container around itself. So it enables several a isolation features supported in the kernel that you probably heard about, like namespaces cgroups, which routes to the image file system, which for Dockers, is that this path and does all sorts of idle things and then, once once it enabled all of those restrictions.

B

It is isolated enough from the host to be considered a container and then the final thing that runs in it does is exact the binary that you asked for. So if you can dock here on Ubuntu sleep, it will exact sleep and the result is exactly what you asked for a container running the binary that you asked for in the file system of the image that you specified. So those are like containers in 60 seconds or less.

A

So most of you in this room have probably heard the term container engine or container runtime, and the difference between this two is not always very clear. So, let's posit that for a second and explain distinguish the difference between them. So container runtime is basically everything you've all just described. It's the. It is the program that actually uses the kernel to create containers. You can run containers with it or execute processes inside containers with it, and but it doesn't give you all the juice that we actually like about containers it doesn't.

A

Let you manage images, it doesn't let you connect to a remote repository or basically just manage the containers running on your system and see what is running and and run complex comment upon these containers that you have. So this is container engines with many different implementations for container engines, including docker, CIO and Portman.

A

They give us all the like comments that I was to build images and so copy from inventing containers. So for that reason we have so many different implementations for the copy comment, because it's implemented at the engine level and not at the runtime level. Each engine manager it implements this comment differently, so you have a sip comment for docker, for Padma and for lxd, and there is all different implementations we'll take a different approach for how to actually make the copy operation and kubernetes in itself also implements the copy command by itself.

A

Even though it is not the container, and so what is the copy command? It's probably what you imagine it to be copy files form container to your computer or formula builder to a container or you can even copy from different running containers, and the syntax is similar to the UNIX CP command that you may be familiar with.

A

So it's not all that complicated and the use case is actually more common that then, we've originally fast, because it is used by many developers and DevOps full production environments to either debug the containers and see what's going on with them, deliver some files into them. It's using some scripts to manage containers, we've seen copy actually being used in the world, and so it is a real. It is relevant. As for that, you can see that it has been implemented in all.

A

Basically, almost all the containers engines have an implementation for the CP command, so you can guess if it is useful and people are making use of this comment so having vulnerabilities, it is actually highly relevant and the past few years and we've seen a surge in vulnerabilities being disclosed and some information being disclosed about them, and but we even saw some CP vulnerabilities being found back as when containers even started at around 2014. When dr was implementing this command.

A

There were some issues, some security bugs and that you can find and see that they have had issues that have could have been exploited. So it is not a new concept, but in the last few years many more people have taken interesting to finding this kind of vulnerabilities, including our team, and with that, let's just dig into the vulnerabilities and see the actual implementations issues. So, let's start with Bodmin.

B

So that's the pullman CP command. It's pretty simple! That's how you copy a host file to a pair for the container via ABC and let's see how the department actually implements this. So the first way importnant does is build the container path, but from the hosts view. So it takes the container mount point on the host and adds it the path that you asked for in the container and then it does something that is quite similar to a normal unix CP command using that path. So that's pretty straightforward. What could go wrong here?

B

Well same links, so this is the first CD we'll talk about today and the problem was that symlinks improvement will resolve under the host hood. So let's look at an example. Let's say we have an attacker control container, which has a symlink inside it called fake bill and then a padma user does something like Bodmin CP host file to a path that contains this a symlink. What will happen is the semen could be resolved under the host a would and the file will end up actually in the host file system and not inside the container.

B

So that's been a it. Isn't that easy to exploit, because you need to know where the user tries to copy to, but you don't want to have a command where a files could pop up on the content on the host file system. So that's not good and let's see how doe can deal with it. So now we are talking about copying in Indo cow. So how Daka did it to deal with this problem? It will first resolve the container path under the container.

B

Would only then it will take that results path and add it to the container mount point on the host and finally, it will copy so the same example: we have a symlink inside the container and we try to copy a some file to a path that contains this same link now using docker. So the first window does it resolves the path so we resolved it, and only then we added to the container mount point on the host and finally, we copied.

B

So that seems to solve the problem, because the ceilings are resolved before the copy occurs, but it actually didn't a and that will just do a second CV, a we're in dokie. We could have a symlink exchange attack, race attack, a and let's that, with an example. So let's say we have a completely normal CP command copy, a normal file to form the host to a normal director with some deal on the container. So what will happen dr. will try to resolve the passing the container it's a normal directory in normal path.

B

So it stays the same. Then it will add it to the container mount point and finally, it will copy. So that seems good, but the problem was that if an attack hell managed to take one of the components in in the paths in the container and switch it to a symlink between stage two and three we'll have a race attack. Well, the file will still end up on the host file system. So that's quite annoying and you can see that symlinks caused a lot of problems when copying to and from containers.

B

But what can we do? What other solutions are there to deliver that? So one solution that was used is to sort of partially enter the container, and what that means is that the container engine will fork and run a helper binary that help and process will partially enter the container, which normally meant root into the container file system, and then it will read all of the files that you requested in the container while being suited to the container.

B

So all symlinks are resolved under the container root and you have no ceilings, they can point to the host. So, let's see how doe can implement this solution. Now we are talking about copying out so the docker demon folks and run a helper process called local town. Local, tell then true to the container. It tells the requested files inside the container, and then it passed back the resulted tar file to dr. Damon and there are no siblings issues because it access the files were been routed to the container cool.

B

So this seems like a very good solution. What could go wrong here, some stuff? Well, you are partially entered in the container and that what that means is that the helper process is sort of creating a bridge between the container and the host, so doc, he'll tell the helper process is partially inside the container right, it's routed to it, but in some aspects it is still inside the host means it is in the host, namespaces, cgroups and so on.

B

So it could create a bridge between the container and the host and that's exactly what happened in the worst CP CV in all container engines. So far and one of the wall civvies in Boca a up until now. Well, he will have a full host compromised whenever you copied out files out of the container. The attacker didn't know didn't need to know when you were copying on from where you're copying, whenever you copied out of an attacker control container, it will gain, would code execution on the horse, so that's bad, and so how? What happened?

B

So, as we've said, docket our routes to the container right and Duncan is written in go specifically. That version was in a thousand one point eleven, and that version has some feature or you might call it a bag. Well, some packages, some go packages that had embedded Segoe C code inside them. Will dynamically load shared libraries at runtime? So what would happen is docket are would shoot to the container file system and then will dynamically load. Several libraries form the container file system. Those were specifically several eben assess, Lebanese.

B

So to clarify you have a process which is own, which is only treated not completely isolated, which a loads an execute code which is controlled by the container. That is a host compromised. So the attack scenario for a these vulnerabilities, the Doka user, that copies files out of the container, which is either run in a malicious image where we have a malicious Leoben, assess labelling. Oh, if an attacker compromised a running container and replaced the Libin SS libraries. So if that, let's see POC for this vulnerability, can you guys see that yeah cool?

B

What I did is create an image with my own version of the Leoben SS library and what this label it does is once it is loaded to a process, it executes a binary at path breakout, that's all! So, when dr. tau, we load this library, it will execute the break a break out a binary, h, path, break out, / break out in the container. So, if that's not so clear, wait a few k second and we'll be able clearly. So let's run the image.

B

Cool so we have the container running and now, let's copy some files from the container. This is by the way, the vulnerable, a docker belgium. So, let's copy out some logs, for example,.

B

Up, oh I, call it sorry about that pull em. So what I did with dead break with the breakout binary here he just sleeps. That's all it does so we can see what's going on inside the with the copying. So let's look at the process tree.

B

Cool, so we can see that the dog is demon actually spawned located our helper binary and because it loaded our malicious library form the container it executed, the break of binary and the breaker burning, as I said, only do early. Does it sleep so we can see the process tree. So, let's inspect a bit closely or more closely the dr. Tao.

B

So it is process PID.

B

Cool and if we look at the hood, we can see that it is indeed suited to the container like we expect, and we can also see if we, that is credentials, that it is privileged. So this bit mask means full privileges as a container as a comparison inside the container.

B

You have less privilege, as some bits are opened off, so we can see that now that we better understand what's going on, let's learn all we need to just kill the sleeping process.

B

So now that we understand what's going on, let's run another image and in this image what I did is just replace the Blake code binary and what it does is mount the host file system to the container at past hosts at past host FS. So let's see that in action.

B

Cool and then when we copy out any files, so, for example, let's copy out bin Bash doesn't really matter.

B

So, let's see if a breakout, binary Rend and we can see that it did in a new directory call those the face was created. And if we look inside, we can see the entire host file system and the container had full access to it. So we can do something like.

B

Then, if you look at the host, you can see that the container can actually do whatever it wants.

B

Cool, so that's pretty bad.

B

So how did they fix this? So why did they do what they do? Is they they alter the init function of the docket out process so that we, it will call just randomly random functions from the problematic go packages to force it to load the liabilities, the Libin SS libraries in the init phase before shooting that they could add to the container.

B

So now, though, they will be loaded from the hosts file system and an aside from that I'm pretty sure that in a current versions of go, they fix this issue, so go doesn't no longer lows but those loads, a certain libraries at runtime. So we can see that even this solution of partially enter the container could have bad implications and can cause really severe vulnerabilities. So let's have a look at another solution.

A

So we've mentioned during the copy from the actual host and also partially in the container, is problematic, and another implementation possibility is actually fully in turn in the container the same way, you would do a docker exec or a cube Cattell exec.

A

You could just get to all the namespaces of the container and simply not be able to see anything from the host, and that way the previous owner ability and other vulnerabilities we've seen would not be possible because you're simply namespace inside the container and all the KML protections are affecting you and when what could go wrong with this implementation. Well, now your helper binary is inside the container and anyone coming inside the container, with more capabilities or same capabilities as yourself can control you. So that means any process.

A

Basically inside the container can actually deal with what this binary you have does and its output and interact with it, and that means attackers with the same privileges. You could actually affect what you get outside the container form inside it. So, instead of getting out code files, you were expecting you might get something else, and that was actually a problem with kubernetes.

A

So let's take a step back and look at the kubernetes implementation and see how it was done in communities so cuba, catalyst CP, is the command and forms documentation and may be a little bit small. But what it says is you have to you need the tile file, the top binary inside the container. You want to copy form. So if you want to copy files from a container, it must have the tile binary inside it.

A

Otherwise the operation would just fail, and this implies heavily- that the cube cattle command simply execute the top file form inside your container, and this was actually the implementation to copy files out or inside the container. It uses the container tab from from the being table user bin tag, whatever it would have a file on, and it would archive the file, the user requested and unpack them back at the host, and that leads to the main question of what.

A

If an attacker cannot change or control data that cube cattle is running, and so basically it could change the output of that style, binary process and deliver something to cube cattle that may break it or simply not give the user what they wanted or worst of all.

A

If there is some issue with the implementation of cube cattle itself in the way it pulses the output of this stuff, then the attacker might be able to exploit it from inside the container, and this was some CV that was found back in 2018 and by some researcher it tried to exploit cube cattle and he found out that there was a simple path reversal inside its implementation. So what is the festival cell?

A

Basically at our file, is just a list of files and directories, and if we have some file, which his name has some dot dot slash characters inside it and when cube cattle, would take this out this file and try to create it on the host file system.

A

It would follow this full path and by following it means it would simply go back, and instead of copying to some remote deal that the user requested to copy to it would go back to vote and then copied the file to slash temp directory, and that means that you, an attack, a form inside the container, could copy files he controlled, whatever files created to any place on the user system, which is also civil society bed- and this was fixed by sanitizing the path sending the filenames form inside the TAS, so anytime, cube kata would see dot, dot, slash, it would simply eliminate it, and this one our ability would not be able to be exploited again.

A

But, as you can imagine, this fix was pretty specific. It dealt only with the issue of path to versa, so if you might have had any other issues with the implementation of cube cattle, this wouldn't have fix them. They are still using the Taliban nari form inside the container, and it is still untrusted.

A

We simply cannot trust any file that is running any binary form inside the container because it can be controlled by an attacker, and so this led me one ear about one year after seeing this vulnerability and I realized that they are still you, kubernetes is still using tab form inside containers and I wanted to take a look at the implementation and they sled me to find some issue with symlinks, and so, instead of doing a regular path, divest and what I found was I could use the Bullock links to get whatever file.

A

I want for my control tile to the users system and regulate the previous issue and how? How did I establish that so I would in in my top binary it would create. It was always bring back the same time which had a file with a header that was a symbolic link to some critical path from the system that I wanted to override, and the second file in that town would be some file that is in some allegedly directory.

A

There is no real directory called seam on on this tar file, but there was a symbolic link and cube cattle would simply go to that to that path. Symbolic link create this symbolic, this file, this critical path and copy malicious file to the full, critical path that I intended to get the file to, and the same previous issue was related. Then I could exploit this to override any file. I would want in the user system which could be, and if you cutter was run as what could be really critical files, achieving code, execution and whatnot.

A

So I did his closest issue to the kubernetes and OpenShift security team, and they were really responsive and helpful in solving this really quickly, and there was a patch for this and then advisory I did in my original advisory that I sent suggest to redesign this feature and basically not use at a binary form inside the container and I. Let it realize it was actually something that has been suggested by kubernetes developers in some github issue and it the Western ongoing discussion on fixing this, but nobody has actually took the glove to to change.

A

The implementation, so this was an issue for a while and moving on some months later, the C&C of security audit you may have heard of that was invited to find bugs in the kubernetes implementation actually realized. The fix to this previous vulnerability was insufficient and they found some games with which they could actually be achieved after the symlink escape, and you can see the number of jump from 6 to 9. Another vulnerability was found a few months after that with the same piece of code.

A

Similar section is simply not it's not easy to implement and there was yet another way to escape the restriction and copy files to the host. So they were both free series for this issue and I.

A

Hopefully now the implementation is much more secure, but again the old idea is that using something we don't transform inside the container is something we should really avoid and I'm, not the only one that has thought about this, and there was a kubernetes enhancement proposal to actually I didn't, implement the CP command in communities or actually get rid of it, and there is currently a known, huge, ongoing discussion. What should be the future of the CP command? If you have some useful insights, you might want to join in I.

A

Think some of the users participating in this discussion actually sitting in this room so yeah. So we don't know what is the right thing to do with kubernetes with your cattle, but we do have some insights looking at all these vulnerabilities to what we think may be good to do. What is the baseline?

A

You want to follow when implementing a CIPA comment from scratch now with any container engine, so the first thing is actually reading the container processes before coping. So you see group says some feature called sigil freezer. You could simply use like most container engines already used, see groups, so you could. You could use that to actually post the containers before doing the CP command.

A

Then you'd avoid only seemly and any race issues like the docker, a vulnerability that we've discussed earlier and we do think you should enter in the container doing it with the current feature of the kernel providers. Doing this from the host is still not so very safe, so you need to do all the coping form inside the container to avoid symlink issues, but you have to do it cautiously. So what this cautiously mean it means get inside the container.

A

Only with the truth only change the mouth namespace, but you don't need to change everything else to prevent someone from inside the container from interacting with you. This is actually what lxd does and use nothing from the container.

A

That means either on purpose like using the tab binary form inside it, but also making sure it doesn't use any libraries from inside the container by accident or simply making sure it is completely statically linked and doesn't call anything and some attacker might have put inside the container, and with that, let's also mention some of the something that are being implemented in the kernel right now to actually solve these issues and allow us to do something even much better and stronger than that. So.

B

There is actually a new Cisco that is been built exactly to tackle this problem and a bunch of other problems that are related also to container so it's called open at 2a and what it does it opens files in directory a while restricting restricting the path resolution so that you can open a file with multiple with multiple flags, but one that interests us I, look up beneath and look up in wood and what this face.

B

What's this Cisco allows us to do, is I can be a host process and I can say: I want to open this file inside the container, but I want all pass resolution in the kernel to be restricted to the hood of the container. So basically ceilings can go outside of the container. So that's a really cool idea, and with that we can go back to the simple implementation.

B

We saw the start of pod when, when we just open, we just try to access the file from the host but use this new Cisco, and we can avoid all of the complex, complex solution with like entering in the container that really caused a lot of trouble. So once that implemented, we believe that in the future there will be a much safer way to implement the city command and we hope that it will be implemented soon. We don't do it in say a bunch of other smart guy a and that's it. Thank you guys.