Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Where Are Your Images Running? Stop Worrying and Start Encrypting! - Brandon Lum & Harshal Patil

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Where Are Your Images Running? Stop Worrying and Start Encrypting! - Brandon Lum & Harshal Patil, IBM

Container image signing has made it possible for cluster operators to cryptographically verify that images are trusted. However, the same cannot be said of whether the nodes that run these images are trusted to view and use the contents of these images. This can be particularly important if compliance requires knowledge of WHERE a container image is running. In this talk, we will show how DevOps, trust bootstrapping, and key management, in conjunction with container image encryption can achieve geofencing of execution. We will demo the new encrypted container images worker node model in containerd and cri-o working in such a context. In addition, we will explore how to bootstrap node trust, from simple setups to advanced key distribution using HW Root of Trust/TPM technologies like Keylime. At the end, one should be able to “create an image only be usable by clusters in the EU region”.

https://sched.co/Zepc

A

Hi, my name is brandon and I'm here today with my co-speaker harsho and we're going to talk to you about a problem that would have been easy to answer 30 years ago, which is where's my workload running and the answer would be simple: it'll be yeah, you know it's a room, it's in the room on the seventh floor or something, but today, when asked about where my workload is, we say the cloud with virtualization, containerization and layers of abstraction coming into place.

A

The answer to this is not as straightforward anymore, whereas my workload after all, the compute has to live somewhere, both logically and physically, in a country in a room in a server in bare metal.

A

But why does this matter.

A

There are several reasons why this would matter. First, there is, if our container contains confidential code trade secrets, for example, top secret trading or military algorithms. We wouldn't want our code to be accidentally run in data centers. We don't own, you know also even in someone's basement, for example.

A

The second is for compliance reasons and highly regulated industries like financial, federal and healthcare. For these industries they go even a step further. They not only require the container workload to be tied to a physical location, but they also want to ensure that the underlying technology stack of the machine is attested and approved.

A

And finally, the last is for things such as export control or to enforce other legal regulations. So this is exactly the problem we're going to talk about today. How do I tie compute execution of my container to a specific location, and this is also called execution geofencing and by the end of the talk today, we're going to learn how to do this with containers.

A

Our claim is container image. Encryption plus key management equals execution, drill, fencing, and we will show this so to start exploring that we're going to first look at one of the underlying technologies, which is container image. Encryption.

A

So what is container image encryption? So this is an effort that started almost two years ago. It basically allows the ability to encrypt a container image. The process of this is fairly simple. On the left hand, side during the build stage, we add an encrypt step and during the run time we add a decrypt step, so between build and runtime the container remains encrypted.

A

This means that the contents of the image remains confidential over the network or even in the registry.

A

So, in a nutshell, this technology enables us to encrypt a container image, so that is only decryptable by a particular key x, and this means that we can retain the confidentiality of the image, even if, for example, the registry is hacked.

A

So this is a feature that's available today across various parts of the ecosystem, like builder scorpio, container d, cryo and the docker registry, and we'll be showing some of this in the demos today,.

A

So before we go ahead, let's go through a quick um asymmetric, crypto primer to see how this works, so in asymmetric, cryptography such as rsa we use keepass made out of public portion, which is not secret and can be published and a private portion which is kept secret.

A

We can encrypt a piece of data over here on the left, with a public key to get the encrypted cipher text in red, and then the cipher text can then be decrypted by the private key which only the owner has to obtain the plain text.

A

So now, let's go to example of encrypting image, so over here we have address, and we say we want to encrypt an image for alice. So let's go to the flow here. So what else first has to do is to generate an rsa key pair and then publish the public portion of a key.

A

So given the key, we can then go ahead and take our container image and encrypt it with a public key. We then take this encrypted image and push it to the registry, like docker hub.

A

Then find finally alice is able to pull the encrypted image and because only she has access to the private key and no one else. Only. She is able to successfully decrypt the image and run the container.

A

So let's go ahead and take a quick look at what the demo of this looks like.

B

A

Let's take a look at a demo on how we can encrypt an image for others. um So, on the left hand side here we have lss workstation, which is a brown terminal, and the first step of it, as we saw earlier, is generating the keys.

A

So we can do this by using the open, ssl command and we're generating our rsc keypad.

A

So over here we have a private and public key as we saw earlier, and what alice has to do now is she has to make a key available so that we can encrypt an image for her.

A

So now we have the public key and we're going to take this public key and we're going to simulate a different workstation and we are going to create an encrypted image so that only address can decrypt it with a private key.

A

So on the right here, I'm gonna copy and as this public key and I'm gonna paste it in here and I'm gonna build an image for alice. So this is a really simple image. uh It's a busy box that just cuts out. This is an encrypted image for alice and we are gonna use. The builder tool to build this image so builder, but is kind of like docker, build it, takes the docker file and builds an image and we are going to push this image to docker hub.

A

So what we are going to do is we're going to say, build up push and we're going to add additional flag called encryption, key to specify that we want to encrypt this so that it's only decryptable by azure's private key.

A

So we say build a push encryption key and we have the path in docker hub.

A

And we can see that it's now successfully uploaded to docker hub.

A

So now that this is done, let us see where the address can then download and decrypt this image.

A

So I'm just going to clear up the cache by just removing all the images currently in builder, and so what we're going to try now is we're going to say we're going to pull this image from docker hub.

A

All right, so you say build.pull with the image we just created and we're going to see that it's going to try and pull the image, but it's having some trouble with it, and so in a moment we're going to see here that it has an error that says missing private keys needed for decryption now, because the image was encrypted, we needed to provide a decryption key so that it can be decrypted.

A

So in this case we have to add additional flag here to pass in as decrypt the private key. We have decryption key flag and then we say add this private key.pm and what this is going to do now is it's going to successfully download and decrypt the image we can then go ahead on addison's workstation. We can run the image by using portman, which is kind of like docker run, and we can see that it is running the encrypted image that we have created.

A

All right so in the demo we just saw how we can create an encrypted image for others and how she was able to run it on her workstation.

A

So next question adam says, is how do I take this image and run it on my kubernetes cluster? So let's take a look at that, so how encrypted container images work with kubernetes cluster is, should they contain a runtime such as cryo and container d?

A

These runtimes are responsible for putting decrypting and unpacking the image, and in order for the runtimes to be able to decrypt the image they like address need to have access to the key material.

A

So today this can be done by passing it to the file system and, for example, in cryo. The default directory for this is etsy cryo keys.

A

So, to help distribute the key to the container runtimes more easily. We will make use of a key syncing operator that we've written this is open source and is also available on operator hub.

A

What the key syncing operator does is allows the ability to take a kubernetes secret as a key and distribute it to the container runtimes, so others can easily create a decryption key by creating a secret with the specific type equals to key.

A

So, let's take a look at this now we're going to show how the address can take the encrypted image from earlier and run it on the kubernetes cluster. So the setup here we're using is very simple. It's a mini, cube setup with the container runtime cryo, and what we're going to do first is to install the operator that we talked about which will synchronize the keys.

A

So this operator is called the encryption keysync operator and you can get it in this repository here and so we're going to install it using helm by first creating the name space for it and then we're going to use the helm install and so once we've successfully installed the operator here, we'll be given some instructions.

A

But before we do anything, let's first try and run the image that we created earlier.

A

So we're going to create a part here and we're going to see that it errored. So let's take a look about why it failed to run.

A

So if we go all the way down, we can see that the error message why I could not unpack the image was because it says missing private key needed for decryption.

A

Now this is expected because the image is encrypted and we have not yet provided the key to the container runtimes to decrypt.

A

So let's do just that. So from the instructions in the operator, we see that we can install a key by creating a secret with type key and we're going to going to do exactly that.

A

We're going to use this command and we're going to point it to the private key that we created earlier, which is in addition directory because we're decrypting we are using the private key and we're going to just name the secret atlas, key cool.

A

So once the secret has been created, let's now try and recreate the pod.

A

So we are going to run the encrypted workload.

B

A

Here and we can see that is successfully running and if we observe the port we can see. That is indeed running the image that we have created before.

A

Awesome so, let's recap what we've kind of seen in the past few demos.

A

So we started off by encrypting a container image that was only decryptable by alice's private key in the first demo and in the first demo we saw that because atlas had a private key accessible on her local workstation, she was able to decrypt it and therefore the container workload that was encrypted was only decryptable on addresses workstation, because the key was only available there.

A

So, in the second example, we went ahead and others used operator to make her private key accessible to the nodes in the kubernetes cluster. So in the middle here we see that edison's private key is not accessible on both hobox station and kubernetes cluster.

A

So this meant that the container workload was now decryptable and executable on both her workstation and addison's kubernetes cluster.

A

So if we take a step back and in this scenarios, we see that you know alice has been this person that has been generating the keys she's been making them available to different entities.

A

um What else is essentially doing is manual key management right she's acting as a vault, but just a person, that's generating keys and putting it in the correct locations.

A

So if we generalize this right uh as his key mentioned, what does key management do for us? So key management says that a certain key x, for example, in this case x, is alice's. Private key is only accessible by a set of entities e where e in this case is addison's, workstation and as kubernetes cluster.

A

So we take the definition of container image encryption that we got from earlier. We get that container image. Encryption allows us to encrypt a container workload, so that is only decryptable by a particular key x.

A

Key management says, I can say, a key x is only accessible by entities e, and so, if we put this together, we can get the statement that says the container workload is only decryptable on the particular entities e, and this is exactly geofencing, because if entities e may be a set of workers in a particular location, that would be able to allow us to do a fancy execution awesome. So now that we have this capability, harsha will show you how we can use this enterprise scenario.

B

Thanks manden so uh now extending on what brandon said that you have a key management and you have image encryption.

B

So, let's, let's get to the actual scenario where I have like a u.s cluster here in red and then a cluster in european union, and I want to make sure that the container images that I am creating are only able to run on specific uh cluster.

B

For example, if I create a container image that I want to be able to able to make sure that it only runs on european union cluster and nowhere else, so what what we can do here is we can use image encryption and the key management in combination such that the the images will be able to run on uh appropriate cluster.

B

So to elaborate the point further, if you, if you create ima, if you encrypt the image and use the proper key management where cluster gets authenticated, the the keys are only provisioned to the cluster if it gets properly authenticated uh with the key management.

B

uh Only the in this case, you can see the cluster in eu cluster, uh the! U cluster will get the keys and the us cluster will not be able to get the required keys.

B

Let's jump into demo to see this in live in action.

B

Welcome to the demo in this demo we'll see how you can use image encryption to do uh jio fencing for your container images. I have two clusters here: one running in european union and another running in us.

B

The one running in european union is using container d as a crm time, while one running in united states is using kaio as his cri run time. Both cryo and continuity support image decryption.

B

So, let's try to build some sample workload.

B

So I have a simple docker file here. I just want to make sure that this sample workload only runs on the cluster that's running inside european union. So let's build this using build now, once it's built, you can use the same builder command to encrypt this image using a public key once it's encrypted. It can only be decrypted using this private key. So it's very important that you copy this private key to the worker nodes.

B

No, where you will run this image, you can manually copy this private key or you can use this operator here which will copy it safely to the worker nodes and it will place it in a right location where cleo or contender d can find those keys.

B

So now that we have built it, let's push it and while pushing, as I mentioned, we'll use this public key to encrypt the.

B

B

Done good so now that we have built it encrypted and uploaded, let's try to run this on our cluster now this has found. So I have a sample demand set here which will use the image that we just created here.

B

So let's go ahead and.

B

B

B

B

It's running right, that's good! So now uh now that we have, we are able to download the image and uh decrypt it and run it successfully. Let's see what happens if, if the cluster that doesn't have this required, private keys tries to run this container image so on the us cluster I have this. I can run the same workload which refers to the same image that we just built here in european for the european union cluster. Let's see if we try to run it here, what.

B

B

And you're getting an error: let's see what this error.

B

B

So the error says that none of the private keys could be used for decryption. It means that this cluster is not able to use this image because it doesn't have the required private keys and this. This is how we are able to prevent the cluster running in us uh from using the image that was built for running the uh for from running the cluster that is running inside european union.

B

So, let's try to do it other way around. So we have a docker file here.

B

And this is just a sample code. We want to make sure it's only able to run inside the cluster which is running in us, let's build using builder and once it's built, we gonna encrypt it using this public key, and this is the corresponding private key.

B

I have already copied this private key on the worker nodes of this cluster so that we can successfully decrypt it now that I built it. Let's encrypt it and push it to registry.

B

Let me go ahead and delete delete the deployment that we created sometime back not from here.

B

And now that we have built uh an encrypted and pushed image for us cluster, let's try to use it on this cluster. So I have this ml. You can refer the sim we just build. This image keeps it l, create minus f.

B

Get parts it's creating.

B

And almost there, yes and it's running so now we can do the same thing that we did earlier with the image that we built uh built for european indian cluster. What happens if the image that you just built for us cluster, if you try to use it on the cluster, that's running in a european union?

B

Let's see that okay, it parsed. Let me.

A

B

um Cat usa, so this is the image that we built here.

B

So let's go ahead and deploy it.

B

Get parts get parts it's creating.

B

And it gives the same error that we saw here when you try to use the container image which, where the the required keys, are not present on a cluster. That concludes my demo. Thank you.

A

All right, thanks archer, so before we end of the talk I'd like to address uh one aspect of key mention that we talked about earlier, we said that key management allows the enforcement that key x is only accessible by particular entities e.

A

However, today, how do we ensure that those entities can access those keys and we today rely a lot on secure process or reliance on the trusted administrator? However, this introduces a lot of human error and I think that this is an area that we can do a little bit better.

A

So in research today we are exploring how to do this, um to do something even more, which is to tie it to a highway route of trust like a dpm and even go ahead and have the ability to attach the firmware all the way up to the operating system level.

A

So this way, not only do you have tie into a particular set of hardware, you also are able to know what is exactly running in the underlying stack of your container.

A

So there are a couple open source technologies that do this today, there's key line which is developed, but mainly by red hat, and then we have intel data center, secure libraries, also known as ice echo, which is a set of libraries that allow you to perform attestation and get attestation reports back.

A

So, in addition, there's a there's been a publication by this, and this is something that we collaborated on with them, and this article talks about how you use these type of technologies, together with the container platform to bootstrap trust, all the way from the hardware route of truss to the containers.

A

So this is a piece of world. That's pretty interesting in itself. It kind of is a topic in itself, so we don't have time to really go in depth, but this is something that we are constantly looking at when we'd like more people to engage on this topic.

A

So to summarize, uh we talked about um the question of where are my continuous running. We talked about why and sub-scenarios can be important and how we can achieve this through encrypted container images as well as key management.

A

We showed you how to use encrypted container images on various components and parts of the ecosystem like container the aquarium, builder and docker distribution.

A

And lastly, we touched a little bit on talking about highly regulated industries and high assurance.

A

uh How do we kind of get to the next step in being able to bootstrap trust into our container platform, and this is something that um we want to continuously work on, and we would like to be engaged with this, so we have a repository here that we we've created so that people can join into the discussion. So please do feel free to come and engage with us.

A

So that's all we had for today. um Thank you very much, a quick shout out also to stefan phil, who did a lot of work during the initial stages of the future and also with the other collaborators from oci container, the cryo and the containers repository.

A

So with that, I'd like to open up the questions and also there are links to various components that we talked about in the slides if you can download them. Thank.

A

B

Oh well, uh can you hear me? I think I have sorry I wasn't mute, uh so must images be decrypted in order for static vulnerability scanners? There's a question from casper came in uh casper kramer uh about. uh If you need to use the scanning tools like trivia or claire, do you need to decrypt? And yes I mean uh so the way it would work? Is you you can when you encrypt the image you can have a multiple recipients.

B

So let's say I encrypt the image that I need to run on my cluster and then I can add the a private key that I will associate with the clear, for example. So when the when you know your build pipeline, you will be able to that clear instance will be able to decrypt the image. uh Do your scanning and uh no, it can go ahead with that. So that way, uh you don't have to disclose this image to everyone except your instance where you're running the scanning.

B

So that's question from casper. I think you have another one more question now.

B

Are the manifest and config files of the image, also equipped with a question from alexandro? I know right now, the uh the first implementation we have layers of the images that are encrypted, not the manifest itself: hey brandon, okay,.

A

Cool, I'm I'm sorry. I had some technical difficulties, um so I just want to kind of add a little bit more to what hashtag says about the image scanning. So one way that he mentioned was you have a key so something that we're looking at is um you can encrypt so there's a question in the q a as well on.

A

Do I need to have different private keys to be able to? um Can I use different private keys for different machines? If I want to or can does the image is an image only tied to a single private key and the answer is you can use different private keys? uh You can encrypt an image for multiple using multiple public keys, and this is kind of like an approach that we were thinking about in terms of how one will use this for scanning service.

A

So, for example, a service like trivia or cloud or something would expose a public key, and if you trust the service, you will encrypt your image both for your hardware as well as for the for the scanning services that you trust. um That is one way to do it. There are a couple other approaches to this, so what um the image encryption is also done on a layer level.

A

uh So what you could do is, for example, if your confidential code is only a small portion of the entire image right, because we serve the operating system because of the middleware. You could also leave your middle middleware operating system layers decrypted, because there's no uh confidential material that and only encrypt your topmost layer.

A

um So that's another way of doing it um and, if you're, using things like openscap or things that can scan locally.

A

Another thing that we recommend you know if, if you're already in a high assurance scenario- and you don't trust anyone, including external providers, you want to do your own scanning in-house. um You could do. You could do this by doing the scanning before sending.

B

A

Uploading the images to the registry.

B

Yep yeah, we have a question for philip village about what is the benefit of using asymmetric keys here instead of symmetric keys, so the symmetric keys are we do use symmetric keys to uh encrypt the layers? We just wrap them in a sigmetric keys so that you can have recipients for that. uh So brandon. You want to add in that.

A

um No, I think you said it pretty well, um we use both of them. We didn't go to the spec over here, but if you, if you go to the we have a couple links in the last slide, uh some of them and if you want to download the presentation, there's a couple links there. The spec will contain more detail about how we use both symmetric for the data and this we're using asymmetric jets for the uh key wrapping and um key delivery.

B

Now we know uh it's available in docker hub uh there's a question whether it's it's available, azure container registry- I haven't checked. uh If it's available, I will assume I.

A

Continue, I know it's available on ibm container registry.

B

Yeah, it's yeah, ibm continuously, docker has supports it and as soon as any other provider will upgrade to the latest uh version of the registry distribution, uh they will get the support because the code is upstream anyway,.

A

I think we tested this with registry if you're using docker distribution. It's we were testing this as 2.7.1, so I think anything above 2.7 should be fine.

B

I'm just going with questions.

A

Okay, um there's a question for alejandro about other manufacturing conflict files also encrypted um so right now only the layers are being encrypted. um We had a lot of conversations about encrypting the conflict as well. um It seems like there are some technical challenges in terms of how a lot of the tooling is written, where the code paths are not um are not kind of similar to the layers. So we started off with layers conflict. Is conflicts are kind of from the roadmap manifests we weren't.

A

We couldn't really figure out the use case for manifest conflict. We definitely think it should be encrypted. um What we are, what the next step for this is we're actually proposing this encryption scheme at the ocr artifacts level. uh So what this would mean would be. You could encrypt any um oci media type right, so you could have. uh If you had a manifest, you could have layers and then, um if there are new oci media types, it would just be a suffix.

A

So if you look at the media type of the spec, um if for the layers you have like tar.gzip and what we're doing is we're just adding a plus encrypted suffix and.

B

A

What this is just tells it that this blob is encrypted.

B

That kind of answers the question ron chitaro has: is there any signature details in image manifest that can be read by admission controller? You know when you, if you can fetch the image, and if you read the layers, the layers will be tagged as not as these. It will be tagged and then enc will be at the end of it.

A

Yeah and- and I um I would say that this will be used in tandem with any signing solutions like doctor content, trust overhead, simple signing um just because we are only the encryptions- are really focused on the integrity of the um the granular level. Blobs like the layers, but in terms of like the entire manifest and the provenance. We're not really looking at that.

A

So I would say both can be used.

B

Together, I'm just going on more questions.

A

B

Okay looks like we had time.

A

um Oh yeah yeah it's inside well time um I'll. I guess we'll be on psych. If there are any more questions and then we can go from there.

B