Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Digging Into Your App's Container Image Layers for Sneaky Vulnerabilities - Pablo Galego, VMware

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Digging Into Your App's Container Image Layers for Sneaky Vulnerabilities - Pablo Galego, VMware

Mitigating vulnerabilities in container images is, most of the time, a straight-forward task: update the base image, use a newer version of Node or Java, bump the patch version of a project dependency, etc. However, all useful pieces of software are complex and vulnerability scanning tools fall short on explaining why they are flagging some edge-cases. This session walks you through mitigating critical vulnerabilities in popular container images like Java-based ones, from the obvious to the sneaky ones, and how to leverage layer explorer tools to narrow the search field for the latter. It is meant to be a hands-on session, first we will use Aqua’s Trivy scanner to analyze an image generated for a Spring Boot app and then wagoodman's dive to explore in which layer we are introducing a version of a library with critical vulnerabilities, while Maven seems to tell us otherwise.

A

Hey so thank you for being here. This is my first q com. I've been to several amazing talks where the speaker said that it was their first time speaking in public after kobit. This is my first time in speaking in public period. So let's see how this goes. Thank you. Thank you.

A

Thank you to the cncf. um You clearly don't know what you're doing.

A

So a bit about me: well, we are going to see how you can explore the layers of your container images uh to find some sneaky vulnerabilities and previously we'll go through some more common mitigations um very quickly about me, I'm pablo I'm a software engineer at vmware I joined like one year ago. I started working on containers and helm charts and now I'm more on the java microservices world.

A

By the way I I've also studied law before computer engineering. So if someone has a problem here in spain, talk to me the story behind this talk um was in november of last year, when I moved from the team maintaining the vietnamese open source catalogue for those that don't know, bitnami offers a huge library of containers and help charts built, tested and published by an internal pipeline.

A

I'm now part of the team, that's developing the product that will supersede this internal pipeline and offer many of its features as a service.

A

So in this new team I was tasked to establish a vulnerability baseline for the java microservices under development.

A

We needed a way to tell if, under a new circumstance, we were worsening our security posture and a baseline is what allows you to to make that judgment.

A

So we had a very good starting point. Almost a lab environment. We had very a very limited number of micro services built with up-to-date technologies.

A

Most of them used build packs, but, as you may know, build packs require your app to fit into a set of requirements, and when that can happen, the classic dockerfile approach is unavoidable.

A

So we also had already in place a vulnerability scanning facing ci running aquas tv, but it was just informative and non-blocking all of that to achieve a sound goal to have no creative, no fixable critical vulnerabilities in production.

A

Why do I think this entry level talk might be relevant because, as you know, containers are an excellent tool to ease the deployment and to its deployment and development, but they also bring their unique challenges when it comes to securing them to people that never had before a similar responsibility.

A

If you are working on a team, that's similar to mine developers are now responsible for the patch management of the os base image, and also this shift happened. While containers are marketed as boxes that just work. So why would you even want to look into them right?

A

um Finally, many of the open source vulnerability scanning tools are noisy by default, have noisy outputs by default and requires some tweaking to give you the useful information on what you can actually act.

A

So I'm going to go through the steps of fixing the vulnerabilities in a javas prim boot maven project to show that that's a task that can be done by anyone with a minimum knowledge of containers and even going one extra mile is not that difficult. Once you remove part of the of the magic of containers, so let's go quickly removing that magic, so containers, as you know, container images are sets of layers we'll see in a bit.

A

What are those layers- and you may remember that there was this common phrase of every instruction in a docker file, generates a new layer but with bilkit that's no longer true.

A

Now the layers are cached and if no changes are detected, the cache layer is used so effectively only run copy and add instructions create a new layer in your image.

A

If you run a docker build with this simple image with this simple docker file, sorry you'll see that the really the only layers that are created are the ones that are highlighted highlighted here and the other ones are just skipped.

A

Layers are just folders and configuration files if you run that command you'll see an output similar to this. I've highlighted in green, yellow and orange the layers from this docker file, and you find in one the mini dev file structure, and it is just a stripped-down version of debian maintained by vietnamese on the other, the empty file and, on the last one, the cool binary with the ca certificates unpacked.

A

So I've set up a playground for this talk that are two java projects, simulating a plugin system and a container library that, in this case, just holds the image for building the demo service.

A

Okay, so let's see live okay, I hope everyone in the back can see. This is a strange layout.

A

So the builder image, the demo service, uses multi-stage fields multi-stage fields, help us implement the builder pattern in which we separate what in which we divide the process of bringing source code to production into two images. The first image with all the build chain tools and the other one just holds the binary generated or the byte code- to have a slimmer image.

A

So here we'll just run a docker, build and call it local, maven, column, maven and so on.

A

So in the demo service, this is a one of the the skeleton of one of the micro services that I had to look into. I kept the original structure, but no code, so I have the api core infra some ports and adapters pattern and in the booth module the boot module is the one that holds uh it, acts as a glue of the project, because it has, it has a dependency on all the other modules and also the the application.

A

Entry point so in in here is where we are using the dockerfile maven plugin to create the image of the app and the configuration settings are very simple: the repository is going to be the artifact id. The tag is going to be the version of the project and then, as builddarks, we are sending the jar and some other environment variables.

A

So if you are curious, the this is the docker file.

A

Here we have the builder image, some magic to install helm and here the real image that is going to be deployed, so we'll use the maven rubber and the typical clean install command, and hopefully the network is going to be okay, okay, so, while this is building, we can go now to the process of refining the 3d output, we'll start by running a naive, 3b image and use yeah and use the image generated here.

A

Sorry for those in the back, because this is not going to be using very well, but trust me nothing to worry about just 200 vulnerabilities in a container image.

A

So this is a not not the output that we want, because we want just the critical ones so for that, 3b has a flag that we can use called a severity. So we can do dash dash severity and set it to critical.

A

And if you notice, 3b, the output of three was now instantaneous, because 3b cached the result from the previous run.

A

So this is nicer, but we still have here some vulnerabilities that are not relevant, because if they, if a package doesn't have a fixed version, we are not going to go to to develop the fix for that package in the upstream debian.

A

So 3b has this nice flag called and fixed that will give us the actionable items that we need. So with this with this, we are good to go.

A

If you are okay with it, I'm going to start by with the base image vulnerabilities. This is an open, ssl, critical vulnerability. We will have to go here and I have bad news.

A

I think the only way to to know in which image this vulnerability is resolved is just manually checking if a new revision of the base image or if a newer version doesn't have this vulnerability. So hopefully running again, install will generate an image that doesn't have this vulnerability.

A

Okay yeah so now 3v is detecting that this is a different image from before. So it's going to take just a little bit.

A

Yeah, this is done, so we can go now with this uh very odd runtime dependency.

A

This was helm was one of the reasons why we couldn't use build packs for for this service, among with other, more rare, runtime dependencies, again more bad news to know in which version of helm this one, this version of container d is bumped. I had to go to the helm, repo searching the issues and see in which pull request the they bumped the container, the version.

A

So again, I know for a fact that in the 381 version of helm, this will this vulnerability was mitigated. So with that revealing, we can see if that's really out.

A

I can't believe everything's working.

A

Okay, 3b again is going to take a bit and we are left just with the vulnerabilities.

A

It's debatable if mitigating the the compile time vulnerabilities.

A

Well, if scanning at the container image level for compile-time vulnerabilities, is the most efficient way. 3B in fact has another option: that's not image, that's file system that will look into the palm.xml instead of the jars.

A

But if you ask me, this is just a good last mile verification, so somebody may know which cvs this is the spring for shell rc and fortunately,.

A

We have this very well organized and in in the original service we had only two. We only had to bump this maven property, so we could bring the the latest springboot version.

A

You may have noticed that I'm using java 16, because that's the version that we had in november- that's deprecated and.

A

We now use java 17, so okay, so now we should be you. We should have uh get rid of spring for shell and the one that's left is this xstream library.

A

So if you know maven, if we are using this extra stream in our project, we will we could just look for it, but we may we may be bringing it through spring.

A

So maven has this plugin called the dependency plugin with a with the goal called tree, and we can see if a extreme is here and it's not again, if you are, if you ever worked, if you ever had the fortune of or this grace of working with maven, you also know the uh pla, the help plugging and the effective point goal, that's going to merge all the different poems of your build and that's what a maven effectively uses to to run the build.

A

So if that's not here,.

A

If that, if that that's not here, we can know for sure that, in this project, this extreme library is not because maybe it's our source of truth, but in some place of the container image. There's this extreme jar because 3b is flying it.

A

So what we can do with 3b is, instead of using the table format that just prints the most important bits of information, there's an extra flag that we can use to unleash all the information that 3b collects. So with this json flag, I'm going to assume it yeah with this json flag.

A

We can find here this xstream library and it says that we are bringing it from the hello plugin, that's in the plugins folder. So with this information we can go here. Look at the docker file, slash plugins probably is going to be in here and we are bringing something from the builder image to this. Second one.

A

By looking at the name of the folders, we can be pretty sure from which of these instructions. We are bringing that hello plugin, but trust me. The original docker file wasn't so simple, so we can be sure by doing something like uh what we did.

A

What of what we did in in the previous slides, something like a docker safe to a tarp package and then looking to into the the contents for this hash this one, the one that starts with four a b a but there's.

A

There's a more convenient way by using dive, so dive is a layer explorer tool that will give us a ton of information about our image.

A

So probably yeah, I have to zoom out a bit sorry for those in the back yeah.

A

So that's a lot of information and well I love it, but if you maybe you can read it in the right, it says current layer contents.

A

I usually toggle the unmodified because having all the files from the base image clutters uh the screen a bit, and this way we can have, we can see only the the files added or the files that have been modified.

A

So in the left we have the layer explorer the layer details of the one selected and the general image details.

A

If someone here has good eyes here, we are leaking build time secrets because we are using the arc instruction in a way. That's not meant to be used. Arcs are just to be used for environment variables that you want available at build time, but not runtime, but we came here for another thing. We came here for a knowing in which of the layers we are bringing this hello, plugin and obviously in the 120 byte layer shouldn't be so.

A

We are now sure that it is in this line of the docker file that we are bringing the plugin with vulnerabilities. So we can go and be sure here.

A

Yeah here it is so basically what what I had to do in real life is call the maintainers of the plugin they've. They bumped the extreme vulnerability and then in the demo service we'll have to use a newer version, so 3v doesn't flag it.

A

So conclusions of this talk, you shouldn't be afraid of going one extra mile seeing the contents of your images because they are just files and folders. Many vulnerability. Scanning tools need some fine tuning to provide a useful output and, as you saw, mitigating the reported vulnerabilities is very often an easy task. You just have to tune the scan, set, sensible expectations and get the job done.

A

So thank you. Mathias.

A

If you have any questions, there's a mic in the center of the room.

B

uh I have a question: do you have trivia implemented in your pipeline.

A

uh Yeah we we had um a step in ci that just used the 3b cli.

C

Hi and thanks for your talk, thank you. Is there a reason why you had the helm binary in the final layer of the docker image, because usually I think you don't need it at all.

A

Yeah, that was a tactical temporal thing we are now we are. We now move that from the orchestrator service and we had it in the uh just in the execution. I know how to explain it, but yeah. It was a temporal thing now now this orchestrator service uses spill packs, it's just sending the what we want to execute to a tecton cluster and in there it's where we have a helm and the other runtime dependencies.

A

B

Hi, this might be a bit off topic. uh I was just wondering: why do you use debian images as your base yeah.

A

uh That that was also temporal. uh We um yeah- I should have asked why, to the developers uh they used this slim version of debian, but for me it was strange also that they used the the open jdk instead of the gre. So I know uh I don't have an answer, a clear answer for that.

B

If I had to guess it's probably for size using the debian slim pack, yeah yeah, we make use of a lot of the bitnami containers and I'm responsible for fixing the vulnerabilities and the way I deal with 99 of them. Is I replace the base image with ubuntu yeah.

B