Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 4 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Establishing Image Provenance and Security in Kubernetes - Adrian Mouat, Container Solutions

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Establishing Image Provenance and Security in Kubernetes - Adrian Mouat, Container Solutions (Any Skill Level)

Take any container running in your Kubernetes cluster. What can you say about it and with what level of certainty? Do you know where it came from? Could an attacker have modified it? Is it up-to-date? Can you identify the exact revision of the code that the image was built from? This talk will look at what guarantees Kubernetes gives you out-of-the-box, and what you can do to establish a trustworthy and reliable workflow for deploying and updating images. Topics and tooling covered will include: - building images in a repeatable manner with BuildKit or Bazel - distributing images through registries - verifying provenance with secure hashes as well as Notary/TUF

About Adrian Mouat
Adrian Mouat is Chief Scientist at Container Solutions, a cloud-native consultancy and Kubernetes Certified Service Provider. Adrian is a member of the Docker Captains program and the author of "Using Docker", published by O’Reilly Media.

He is currently researching image management and distribution on clusters. As part of this research, he has released ImageWolf, a PoC that uses bittorrent to distribute images across a Kubernetes cluster and Trow, a cluster-first image registry.

Adrian is a regular conference speaker and trainer. He spoke at several events last year, including DockerCon and DockerConEU, CraftConf, TuringFest and GOTO Amsterdam.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Okay, well, you need to get kinda, it started on time and she's looking receipts, there's still a few at the front. Yeah I mean remote from container solutions and I hate long introductions. So we'll get started.

A

Whenever you buy food from a supermarket, you can look at the packaging and you can actually tell quite a lot about the product. You know you can tell the ingredients you can tell where it came from its country of origin. You can tell when it was packaged. You can tell you know how long it's good foreign stuff there's a bunch of seats down the front. Here you looking for a seat and either it's a meat product or a farm product.

A

You can quite often even tell the farm that it came from, um and this is a picture of a cow, Halen cow and from a farm in your Edinburgh and you'll note that it's got just work, you'll notice, it's got tags in its ears and those tags actually relate back to an animal passport.

A

So also livestock was in Europe and has an animal passport, so we can track its movement and throughout Europe and beyond, and all this sort of stuff you know, requires a lot of work and time and money, but we decided that this is worthwhile doing so that we know what goes into our food chain. We knew that it's safe to eat nobody's tampered with it that it is what it says is, and this talk is kind of out. Can we say the same thing about this software? That's running inside our data centers.

A

So for a given executable it's running. Can we tell what it is? Can we tell where it came from? Can we tell if it's up to date and if we can answer these sort of questions to at least a reasonable level of confidence, then we open ourselves up to quite a lot of problems.

A

Not just could execute very something, malicious or was a backdoor or perhaps just out of date and vulnerable, but also are we in compliance for the license for that executable and beyond that, just not have been able to answer these questions makes it difficult to maintain and debug when things go wrong in production, because it's now difficult to go from the executable to the source code. It was built from so in. This presentation also discussed this in terms of Cuba, Nettie's and containers, but we can ask similar questions.

A

You know about the Hulk, the host operating system and your developer laptops and your IDE s and even gains a level of mic record if you want to get paranoid, so I've come up with an incredibly catchy acronym for this. It's before lunch, so I'm going to skip over one word. But um basically it's know. What's going on in your cluster. Okay know what is going on in your cluster.

A

Basically, what I mean by this, as just doing like docker, PS or coops et al, get pods, you should be able to easily tell what a running container is as I'm is a nginx. Is it Redis? Is it my application code where it came from so who is responsible for that image, getting deployed into production, how it was created? So what was a docker file that was used to build image?

A

What was a bell context, the source code and also whether or not it's up to date or, if there's a newer version with potentially where security fixes that I should be running instead and for all these questions, it would also be nice if we could prove these answers, that's and verify that it is what we think it is that's running and that nobody could have tampered with it, for instance, but one funny thing is: kubernetes actually does not always work with us here, especially if you're a new user there's a there's, a few things you may need to understand about the way kubernetes works before we can even start answering these questions and I've got a short demo to show you an example of sort of.

A

Firstly, problems with kubernetes yeah. This works I'm, not quite brave enough to do a live demo, so it's in ASCII, cinema I'm afraid, but basically we of a simple web app. The web app is written in rust, because rust is great.

A

We've got a docker file here, that's going to build this web app and, in this case we're going to build a version. All this web app does is serve pizza and we're going to set the pizza to be quite through from aji I'm going to build this we're going to end up an image that just says: Quattro formaggi pizza, not that I've called it a Moute pizza today. So a more it's my user name on a docker hub, as you probably guessed so I can push it up later. I've run it locally.

A

Just to show you what it does.

A

Yeah, so it just returns: Quattro formaggi, pizza, okay, so the next thing we're going to do is push this version of the image to the docker hub.

A

Note you get back this digest, so this is really quite helpful and we'll come back to this later, but basically that's a content based hash of that image. So whenever you see that they Jess, you know what can only ever refer to this image.

A

Now we're going to try and run this image on our kubernetes cluster, so for this demo, I got I just use, google kubernetes engine or whatever it's called and we're using version 1.96.

A

For some reason, my local clients, out of David Mahoney I, missed a three node cluster so about its basic as you can get, and there's no poison in the default namespace at least there's actually no pods and in any other namespaces apart from the system, stuff.

A

Okay, so I've also got a simple kubernetes Yongle file, and this is just going to deploy this web app. It's going to deploy this pizza image and we're asking for three replicas and we're going to load bounce over the top of them in a round robin fashion. So that's going to so idea. Is you know our load balancer well rain! Dropping across these three replicas.

A

Then keep CTL three and we've deployed that service and now a spade dis bit up so because I've asked for a load, bouncer yeah dad she takes a second before Google can grab an external IP for me, which is why you see pending, but because I sped this up, it happens instantly which is nice and then we're going to curl this endpoint and what I expect to see back.

A

It's my Quattro formaggi pizza, but that's actually not what happens, and in this case we've got Hawaiian back and that's kind of confusing, because when the Hoover sent did push that image up to the docker hub and the bushland docker hub is the correct version. But that's not what's running in production, I and I think this is kind of confusing. To be honest, am I explaining why it's happening the second?

A

A lot of you probably already know yeah and we actually end up getting manga written for seasons as well, and so what's happened about to go a bit over top corner, so what's happened? Well, we can investigate the venom pods by doing get pod special llamó, which gives us all the IP and all the round towards, and it's a whole bunch of information there, including information on the images that are running.

A

But we can see that the old images that were run an errand and a mult pizzas a day image. So it looks right there. But then, if we ask for the image ID, we can sort of see what's gone wrong and I actually have three different versions of the image running, which isn't what I want or expected.

A

Now. The reason that that's happened is cause. The default pool policy and kubernetes is, if not present now I didn't write that in the llamo. That's just the default and what's happened is that no ones that have deployed to already had and all version of that image there and so they've used the old version. Grassman go to the docker hub and pull a new one and in fact, by accident it turned out all three nodes for a different version.

A

So a vendor that was three different versions of that image: room which I'm going to argue in most cases, isn't what you want and some cases it probably is.

A

And they actually go as far as saying like the default semantics here are both confusing and dangerous for new users, because it's not really obvious why this happens, you can fix it and I'll go into a site a minute. It explains how you fix this and there's different ways. Actually what you can fix this and the easiest option is just to change the default pool quality. So in that communities Yunel file, I showed you I can set image, pool policy to always, and what that will do is when them DeNooyer goes to run an image.

A

It will first do a docker pull of the image, so it should get the newest version and now would have solve the problem in this case, and you can actually make that the default by using always pull images, admission controller. So the mission controller runs and every time you submit your mo file to the kubernetes control plane, it will automatically set that the image policy to always pool, but that actually doesn't really fix the problem entirely I mean you it's an easy case, considerest.

A

What if I, do a docker push and at the same time as kubernetes, is deploying a new instance? I could very well end up with one lord pulling a newer version of the image and other nodes, so it still hasn't actually solved the problem and I actually think docker swarm. Would there's a lot better here and what docker swarm would does its when an image is submitted by name to the control plane.

A

It resolves it to a digest so that shower hash that we saw before and then it sends that digest to the nodes, to run the containers and by you're absolutely sure that all the notes will end up running the same version of the image which I think is much nicer and also it's a version of the image. At the time it was submitted to the computer control point I think he probably could you know, implement a solution that does the same thing.

A

In kubernetes, using an admission control I did intend to write that I just didn't have time.

A

But a few of you have probably pointed out that the real problem here isn't so much that the pool policy is wrong. It's that I'm using images image tags wrong in the community's view of the world. You see the reason this happens is Kira. Nerys really views image tags as immutable. So if I give an image a tag that should never change it, you only really ever fur to that single image and that's just the kubernetes model of the world.

A

I'm not fair enough, and it has a lot of good semantics. As in you wouldn't have this problem and for that reason a lot of people use things like get jars and build IDs and their tags to make some unique.

A

But that doesn't that's kind of incongruent, with the way that we use docker right when I like using docker, looked in my development machine I'm, doing docker build tag test or whatever and I'm doing that multiple times. So the test image is constantly referring to something different. So there's a you know as a jump here between what docker, how doc have used tags and how Cuba Nettie's use tags and how we're first taught to use image tags and how Cuba Nettie's uses it. But the problem is both.

A

These models are useful, I mean I think we definitely need some idea of immutable tags, but we don't really have that. We just have conventions at the minute.

A

The other thing is, if you consider an image like radius, colon 3 on the docker hub that refers to the latest version of a version 3 of Redis and that changes over time as we get updates to raters and I kind of you know you like that model, because I can do dock a pool of Redis colon 3 and automatically get the newest evil version, but that's again a mutable tag.

A

So what I would like to see is if we made if new to and we had some concept of immutable tags, but we also have a separate concept, for you know types that can change over time. I called it flows here and also I know openshift, there's some similar image streams, but I tried to get to the bottoms out the other day and it's a little bit confusing. So I would like to actually see something a little bit simpler than image streams.

A

I might call the flow is the effort, it's only a suggestion, and what is it so going back to my questions at the start of the session? How do we know.

A

How do we know what a container is we do like docker, PS or Coupe CT I'll get pods, and this really comes back to the same problem with naming things appropriately and actually one problem with using digests under the way that swarm mode works. Is that you end up seeing that the char hash will you take docker PS and that's a bit frustrating because it's much less meaningful. It's not really human readable, but you can get around these problems by using labels.

A

You can attach labels another metadata to your doctor images, so you can write more information on exactly what they are for. Your talk, I would suggest using something like semantic versioning or the get hash that was built from a build ID for from CI CD or something, but you can also put that into metadata on the image labels.

A

And, if you go to this, if you go to the OCI image spec, there is a section called annotations and in the annotation section the Archie define some standard metadata and couldn't get repo and the get harsh and the build date and stuff that you can put on your images. So I thoroughly recommend that how was it created, and so this really comes back to and can we recreate an image? That's been in production.

A

Okay, so can we find a docker file for it and can we find the source that it was built from I'm Google do a lot of work in this area and they really push there reproducibility and he read the release engineering section for by the way there is some seats at the front. If you want to come down, feel free, it's a few pursuit of art yeah. If you look at the the release engineering section of Google s, our ebook, they talk about reproducibility a lot and Google take it.

A

You know very seriously to the extent that they can reproduce like an after-the-fact in a binary way so like if you've got an executable, Vernon they're able to go back to their source code and recreate that exact same binary to the bit level and I know that, on the face of it, like docker, sounds like a really good way to create reproducible builds of software but in reality is and there's actually a few problems.

A

If I run docker build now and then you go in wanaka build later there's quite a few areas on which the built images can diverge, and these are the three main ones. So the first one is like the fron line. So if I did something like from Debian, then I could very well have a different version of Debian to you and also that's going to change over time.

A

I can pin it down more by giving like an exact version like Debian, :, queasy and I can actually use an absolutely exact version by using Debian at and then a digest like that, this char hash, which will only ever refer to one specific image, the other places, if, like you, apt-get some software or um install some software in your docker file that can also change over time. So you may well want to specify you know, package numbers in your dock or files, and the third place is very similar if you download any software.

A

So if you do curl or double you get inside your docket is that can also change over time? Also, I'd be we need curl W get stuff. You are opening yourself to like somebody tampering with that downloaded file in transit or at rest, so I thoroughly, recommend you do stuff like use, checksums or check to GPG signature, etc.

A

But even if you do all those things you don't get to a level of binary reproducibility, so Google have this hermetic build thing where they can recreate the exact same binaries for executables so like. If you have built a binary in C, they could recreate that same binary and that's not a trivial problem because it doesn't just mean you've got this exact same version, the source code. It also means you need to have the exact same version of the compiler. It was first built with I, don't know all the exact same Flags, etc.

A

So it's a bit more complicated than it appears at first, and it turns out.

A

If you use docker build, you cannot get to a level of binary reproducibility and the reason is there's just some stuff that sort of randomly well not randomly, but changes between builds and the primary thing is like file times failed time stamps currently in darker, there's no way to say: I want all the time stamps to match like the git checkout or to be the epoch, or something like that, which would be very nice, and this will also metadata within the image such as the build container IDs and they created on time stamps that will change.

A

But for most people this probably doesn't matter for most people. You just want to get some level of guarantees over here.

A

If you do want to get to bane of reproducibility and Google, actually open source, they're sort of build software we've just internally called blaze, I think and the open source is basil and that can be used to to build like Java jars and T stuff, but it can also be used to build docker images and I. Believe if you use basil, you can get perfectly binary.

A

Reproducible docker images, but I haven't verified that myself, I did want to go into like I talked about like a build stuff and build kit and things here, but I don't have the time so I'm gonna, move on I'm afraid um is the image up-to-date or should have even a newer version, and is there a newer version available for it for, like a containers, I have one in production. It's actually weirdly hard to answer this question and I haven't actually found a good solution that I like I, would like.

A

You know why can't I like ask my registry, give it a digest and say you know what tax this is related to and things like that, and is there a new version of that tag or something, but as far as I know, there's no way you can do that, but we can solve similar problems.

A

And one thing to note, though: it's a bit of tension here we want things to be easy to update so that I can easily update for Dutchman if it's a security fix that comes out, but that's actually in a lot of attention with reproducible builds because, if I have like from Deputy and in a big digest jar hash I can't just do docker pull the Debian to update that image. I have to go and get the new digest the new version of Debian and put that into the docker file.

A

So there's a tension between reproducibility and stability and keeping stuff up to date. I know there's no solution for that. You just have to decide as an organization either explicitly or implicitly where you stand on keeping things up to date. This versus and reproducibility.

A

One thing you can do, though, is the verse. A bunch of software for doing vulnerability. Scans, sees can't scan images and find out, if there's any sort of non-renewable --'tis in them, and there's a free and open source tool called clear from core OS again I read huh that'll. Do that and you can embed that into your CI CD platform, and then you can say things like if this image has more than two critical vulnerabilities, don't deploy it to production, there's also aqua and I believe there's a free version.

A

They were that you can use, and it's also a new vector and twist lock. So there's a lot of companies in that space.

A

On a similar kind of thing to this, the leaf works of a tool called flux and flux flux. Does it tries to keep your kubernetes cluster in sync, was like a git repository? So if you have like image IDs in your git repository, it will constantly keep them the cluster up-to-date with that.

A

But again that doesn't really solve the exact problem. I'm talking about now so I'm afraid I, don't really have a solution at the minute.

A

But going beyond that, how can you prove that an image is what you think it is? um I was a bit sad, I didn't get to see the talk earlier, so I was trying to prepare for this talk, but I believe they kind of talked about this. This problem, as well Angra fears, mainly it's going to come down to digests. So one easy thing you can do is if you have a CI CD platform and you build your images and then test them.

A

And then, if you do in continuous deployment you deploy them to production or you can deploy them to staging. If you, when you do like a docker push to the registry and you take the digest, and you then put that they chest into the cuneta channel file or however you're going to deploy and then you're absolutely guaranteed that what gets deployed to production will match what you built. Otherwise, you know things could have changed over time, etc, or somebody or potentially an attacker could have tampered with something etc.

A

But in a lot of cases the sledge s are just unwieldy and hard to use like it's a big long shower half, that's just not very human friendly if you're using third party, stuff, particular or you're, providing images to other parties.

A

Even within the same organization you can use like keep GPG to sign, stuff and I believe core OS. You should do a lot of stuff for that as well. But now that's notary, I'm not really gives you a really good solution for saying images and being able to verify that an image that you have did come from who you think it came from, and you can quite easily run that in your local cluster and also it was built by docker.

A

But now it's part of the CN CF, which is pretty cool, so I think we'll hear a lot more about notary in the future.

A

And here's a problem that a lot of people talk to me about, or I've heard a few times. It's people, you know run a kubernetes cluster want to be able to say in my cluster I only want to run images from my registry. I.

A

Don't want to see like images from the the docker hub or kita IO running, because they you know I only want to have in just my registry, so I get a chance to get them and to me that sounds like a reasonable and thing to say, but it's actually surprisingly awkward to implement you end up like looking at networking rules and things I think you could write an admission control to do this, but I haven't seen one that exists so far. Does anybody else like want to like do that? Is there anybody?

A

Okay? Well, that's good to say and.

A

So looking at the future, a lot of this comes down to, like community strength, really comes from its distributed architecture. Okay, so it's got a very distributed, independent parts and makeup given Ares and they can be swapped in and out, which is fantastic because it makes it like scalable and reliable and also we can do cool things like I.

A

Believe people are working on couplets that interface with cloud providers like you know and like Fargate and whatever the Amazon thing or since amazon thing in a zoo or container instances and that's made possible because of the distributed independent architecture of kubernetes. But it also creates some problems because it becomes difficult to like get a view of the whole picture when things can change at every point and to control things as it flows through there.

A

So I think what we'll start, seeing it's more and more like holistic solutions that join together various different powers of tools and so on, and in fact we already really are seeing that.

A

So if you look at openshift or docker, Enterprise Edition as far as I can tell it's not an Enterprise Edition logo, but so I'll just use a docker one, but I am talking specifically about Enterprise Edition here, which you know can manage kubernetes clusters and but both these products actually embed a docker registry that you can run that runs in your cluster and have controls over images as they flow through from bells into tests and into production. So we already are seeing more holistic solutions and I think that trend will continue.

A

I should also say, like those are both quite piñon, eating about how you work with an images etc, but I think we'll see more tool in in general, as I say that we would like to seem that the talk before and go fierce and in total, but I didn't have time, but I suggest actually had worst out of liquor. I believe it's about securing the supply chain, so very similar topic and but I container solutions and we're working on something called trout. So you go to the trout that are your website.

A

You can read more about it and it's all about managing images within the kubernetes cluster. So it's the core of it is a registry, but it's also going to tie into like kubernetes features and tightly integrate with it. So you can control the flow from ingestion through to distributing images to production servers, and we have just started work on this. So we'd really like to get any feedbacks.

A

If you go to the trailer I or web page you'll see a sign up form and there you can either sign up for early access, but you can also provide us with some feedback and watch features you'd like and so on. Ok, small plug over and so just to wrap up.

A

The big thing is know: what's going on in your cluster, but to do that you're going to need to understand kubernetes and it's sort of model of the world you're going to want to think holistically so from the beginning of the chain. When you build in your images all the way to when the run in the production think about the provenance, how can you prove that an image is what you think it is reproducibility? Can you recreate the image, so you can debug it and what tools available things like Angra fires and untried, the I/o?

A

Ok, thank you very much.

A

In the slides, there's some references, if you want to have a look at any of this, do we have time for questions, feel free to go I think it's lunchtime now so feel free to run away. There's any question so.

A

A

Okay, so apparently there is an image controller for controlling registry. You pull from and you can like, if you tweet, that out neat I'll retweet it any other questions or people keen to get some lunch.

A

So other than references, maybe ma'am, you can't download these slides as well. I've uploaded them already, and if you want to ask me any questions, I will be on the container solutions booth. So you want to find out more about try. Oh I just want to talk about any of this I'll be on the container solutions, booth, which is next to the Cystic stand.

A

Okay, thank you.