Cloud Native Computing Foundation Cloud Native Security Conference North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Not-So-Fantastic Leaks, and Where to Find Them In Containers- Alex Goodman, Anchore

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Not-So-Fantastic Leaks, and Where to Find Them In Containers- Alex Goodman, Anchore

Building images can be surprisingly difficult, particularly if you need to use packages or applications that are not open and publically available. It’s all too easy to end up with access tokens, credentials, or build artifacts left behind in non-obvious parts of an image. Once you have an image how certain are you that you’ve cleaned up properly and that it doesn’t contain any secrets? Does it have any vulnerable software packages? Is your base image hiding information or unexpected content from you? This talk will show you common pitfalls that lead to information being hidden within an image (either wittingly or unwittingly) and how you can be sure there are no lurking surprises in your image before you publish it. I’ll show how to automate these practices in a Tekton pipeline that both builds your image and acts as a quality gate for publication, because no-one wants to be the person with access keys sitting out in a registry.

A

Well, hi folks, how's it going. My name is alex goodman, I'm a senior software engineer at encore and I'm here to talk about problems that you might find in containers and how they get hidden. So so in this image here we have all the layers for this image and the layer that's selected right now is best most representative of the application payload and it's it's 13 megs in size, but the whole image is 500, megs and so in another example here, so the the application payload is about.

A

You know 230 megs, but the the image is 3.2 gigs. The idea is that you're shipping a lot more than your application, especially now that we bought into uh container technologies. There's a lot more service area that you need to defend and there's a lot more places that stuff can hide. So what are the kinds of things that can get hidden?

A

Well, you have the you know the typical you know, secrets, etc, as well as unintended assets such as, like you have development, payloads or any blobs that may be sitting around or known vulnerabilities for your os packages or any of the language packages that you're bringing in. So what are the common ways that the stuff gets hidden? So these are the top three ways. Ish that you'll find most of the time is folks in their docker files going off and removing something with a remove command, but it doesn't actually remove anything.

A

It's just adding a wide-out file, and so, if you were to push that image, you still have that the payload that's being pushed up to your registry, so not good.

A

uh Also you have, uh if you're just you know in bulk fashion, adding you know everything that you can in your docker context and you don't have the proper docker ignore file in place. uh You may be adding something that you just didn't intend whatsoever and the last one maybe you're just installing packages- and this is probably everyone. So when you install a package, you know you may have a vast set of dependencies. You know that are also coming along for the ride. So do you you know, are you? Are you up to date?

A

Do you have all the patches et cetera? So how do we surface these hidden problems? Well, the best way is to try to describe what is in your image. So the best way to do this is to generate an and then take that description and analyze it for either known problems or known expectations, or at least an audit trail left behind for incident response.

A

So sift is a tool used to generate s-bombs. It generates s-bombs in spdx, 2.2, cyclone dx 1.2. It knows how to generate s-bombs for container images for file systems.

A

It knows how to catalog a wide variety of ecosystems and also beyond just packages, and it outputs in both, like a you know, table summary as well as like rich json output, so it'll go on forever. So in that regard um yeah. So when it comes for looking for problems, we have gripe.

A

So gripe is a lightweight vulnerability scanner. It knows how to look for vulnerabilities for in a list of packages, whether that comes from a container image, a file system or if you have an s-bom which is really useful, because you don't need to bring all those bytes with you, the entire image that you're scanning, you really just need the s-bomb and that's a lot faster to scan, and this supports you know several different ecosystems for scanning again for matching vulnerabilities, so yeah. So now we have these two tools.

A

We want to tie this together into a workflow, and so the idea is that you build your image and you generate your s-bomb.

A

Then you use that s-bomb to run quality gates against it, so vulnerabilities secrets, etcetera whatever it is, that is important to your organization and then, after all, that's done. You publish your image and your s-bomb a lot of people couple step one and three together they build and push their image and then they run tests against it, which is something you don't want to do, because if you have something like secrets, then you're going to have to spend all day scrubbing that from your registry.

A

So so what we're going to do is use kaneko and sift for building an image and generating an s bomb gripe and a little bit of scripting for quality gates and scopio and cosine for publishing our image and our s-bomb.

A

We're going to tie this all together using tekton and kubernetes native ci solution, so yeah all right, it's demo time all right, so we have an application called count goober. uh It is an application. You know, given a number in a sentence, it knows how to extract it and replace it with other numbers very useful. uh So now we have this very useful application. We want to build- and you know, validate the image.

A

So here we have a tactile pipeline, made up of a set of tasks here for building and validating this image, and so I've already got this loaded up locally, so we and already run in a techton run. So sorry in a tech time pipeline run.

A

So here's what this pipeline looks like. So we're fetching our repo we're prepping some assets and we're building our image. When we build this image with canaco, it stays local, we're not pushing it anywhere.

A

Then, with sift, we are generating our s-bomb, and this has spawn where you get. We get an s-bomb.json output at the very end here, which has all the information that we need and we you know, we show a nice summary uh to show uh yep. It looks like our account. Goober app is indeed there version010 and once we get to the quality section of our pipeline, it looks like we have a vulnerability, scan and we're using the s bom as input and we've.

A

uh We will fail if there is a severity that is of higher greater and it looks like that's: what's happened, so quality gets failed here and yeah. It looks like nltk. Has a few high high severity vulnerabilities that we need to remediate. So, okay, that's one problem.

A

It looks like there's a secret here too and notice that we're using our s-bomb as input, um and if I look at the details, yeah it looks like assets. Config e and v has got a generic api key and we don't actually have the key value here in the logs, because again we don't have to scrub stuff. So you know no values are here: okay, so let's go and start remediating stuff.

A

So in our application we said nltk needs to get updated. So it looks like if I were to head down to set three four four. So.

A

All right so we're updating an ltk and our other problem was. We had a secret laying around an assets, config yep. We have an api key and don't worry. This is a fake one. You can take as many pictures as you want um and yeah. So, let's go to uh what the uh the culprit of this is looks like we're prepping assets. We have this config env. That is capturing all of our environment variables. We probably shouldn't be doing that so um and that's not in our git repo, that's just in the pipeline.

A

So um so I commit- and I push this so now- I've remediated all my problems. I think so I head back to our pipeline. I kick off another run and just for time, I've already done that run uh so yeah green. All around looks like we've built our image generated our s-bomb. Our s-bomb shows that nltk has been updated. Indeed, past our vulnerability scan and we have passed our secrets quality and we published our image. So it's been pushed to our registry and we also have published out the s-bom.

A

It's also sitting in our registry, so.

A

And I've already pulled down this image, but it looks like indeed it is working a-okay and if I wanted to check out the s-bomb, I can always do use cosine to pull down and take a look at specifically what the s-bom contents are. So, in this case lots of json so yeah that uh that's all I have let's see thanks um yeah, if you're coming to find me. If you want to talk about sift about gripe, I'm happy to talk about anything whatsoever, we'll be at tom's. Watchbar tonight for incor hosted a happy hour.

A