Cloud Native Computing Foundation Cloud Native SecurityCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Verifiable eBPF Traces for Supply Chain Artifacts with Witness and Tetragon - Cole Kennedy

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Verifiable eBPF Traces for Supply Chain Artifacts with Witness and Tetragon - Cole Kennedy, TestifySec

Until now, validating the build environment and detecting tampered tooling in a build has been very difficult. This talk will show how Cillium Tetragon and Witness integration simplifies this process for developers and security engineers. Witness is a framework for supply chain security that implements the in-toto specification. It has a modular design, easily extendable for various attestors, backends, and key providers (including SPIFFE/SPIRE). This talk will show an attestation plugin that programs Cillum Tetragon to provide detailed eBPF traces of a build step. Additionally, we will create a build policy that verifies the trace and blocks the execution of workload compiled by a malicious compiler when the compiled workload is executed.

A

Hey y'all hope you had a good first day here in Detroit we're having a lot of fun here, we're going to probably I'm, probably not going to be too long, um but, like I said, my name is Cole Kennedy I'm, the founder CEO of testify SEC, um my co-found over there Mikhail he. uh He gave a talk earlier on archivist. If you didn't see that I'll be showing up a little bit in this in this talk, so I encourage you to go back to the the feed and go ahead and take a look at.

A

That might add some context, so at testify, Sac we're building a platform that enables Enterprise to verify the Integrity compliance and trustworthiness of their software um and we're doing all this uh with open source software with a thin layer, proprietary software on top and really what the problem we see is that ever since we started creating software, we hand that software off to uh production right, there's really no way to tell what was in that software, where it came from what was in it?

A

What tools are used to build it right, and so it's just a lot of magic in security and compliance. Engineers and administrators are really struggling with this, especially with a lot of the new regulations are coming out right. You know, there's a new ndaa, that's about to get signed into law that says no software can have vulnerabilities right and there's ssdf, that's going to require you to certify all the steps of your supply chain and make attestations about that.

A

If you're, a small company that tries to sell software to the federal government, this is going to be really really tough for you right. It takes a lot of manual labor to produce these reports and we're really trying to help these companies with this type of a burden through Automation in attestation.

A

So what I'm presenting on today is verifiable at the station's uh of ebpf traces right with witness, we can do a lot of different types of attestations in in this tester is uh experimental, it's not in the main branch yet, um but why do we want to do it? Well, we want to find hidden cves if we look at the log4 Shell or the heart bleed attack those artifacts that were vulnerable to those attacks. There was no way to tell that they're vulnerable, just by looking at the artifact itself later on.

A

Some tools did come out that kind of helped. With this, let you kind of sift through the things, but when a cves just announce like these, what tools do you have as a Security administrator to find out where you're affected in your inventory?

A

um There was nothing. Now we have a little bit more. uh You know we got s-bombs and we have some other tools that can help out with this. But really it's very difficult number two. We want to thwart malicious actors. uh Solarwinds had a really bad time. um I think we all remember what happened with them right. Their build system was completely compromised, um so they were shipping off signed artifacts that their customers trusted that had malicious code inserted into it.

A

What happened was their build system was compromised to the point where there was an agent running on their build system that every time a compiler kitchen action kicked off, it looked for a specific file and replaced it with its own version. Therefore, therefore, injecting that Trojan and there's really no way by looking at that, artifact that you can tell that that happened right if there is a paper Reflections on trusting trust written a long time ago.

A

Right so we've known about this for a long time, we just haven't done anything until recently and then finally, automated uh we want to automate pipeline compliance right. One of the biggest burdens we have is not necessarily that making sure everything's secure but making sure the people that care that it's secure have the information that they need to report up to Executives that say, hey yeah. This is good or your customers right when a salesperson is engaging with their customers.

A

The first thing that a high compliance high risk organization is going to do is ask you to make attestations about your software, and this is really tough to do. If you're a small company just trying to ship features um so at testify Sac, what we want to do is three things. We want to count for all the build materials everything that goes into a build. We want to account for that. We want to account for all the build processes.

A

If you do a static analysis test on your on your pipeline, we want to have that information, be let you deliver that to your customers right. If you do a go, build right, we want to know which compiler you use to do that. Go build with that way. If that compiler has a cve that that's going to compromise your entire organization, we can revoke privileges for that workload and then, finally, right we want to make sure that nothing was tampered with like what happened in solarwinds.

A

We want to ensure that the build materials that we expect to go in the build actually do go into that build so I'm, going to step back for a second I'm going to talk about witness witness is our open source project we started about a year ago, the Genesis was actually at kubecon uh n. A last year, Mikhail and I have been working really really hard on this, and some software around it as well.

A

What witness does is it implements the in total specifications or parts of the internal specification and allows software producer producers to make and verify attestations about the software that they produce. It has Integrations to projects open source projects such as Sig store Inspire, to allow you to sign these attestations without having to worry about keys. In addition, we do have a platform that we're developing that will also provide this capability. We have Integrations into GitHub and gitlab.

A

Currently we have a GitHub action, that's experimental that allows you to do this seamlessly and we make it easy for it makes it easy to produce verifiable evidence for software builds. If, if you want to verify that that saroff came from your uh some from your static analysis tool, that's what witness does if you want to verify that the s-bomb was produced that build time from the uh the code in your repository, that's what witness does right and finally, we support both containerized and non-containerized workloads.

A

We understand that not every uh software artifact is going to be delivered in the container. There's the vast majority of software, whether you're, embedded or virtual machines, does not exist in a container in the future. Maybe it will, but today it doesn't. So what does it support? Well, what witness does is allow you to make these attesters to pretty much do whatever you want, they're, just little pieces of code that describe information about a system. Currently we have a testers for for GitHub.

A

That's those are currently a PR for that gitlab AWS, gcp, openssf scorecard s-bomb through the open, S spdx generator sarif oci, so you can bind it to actual container images and then materials what went into that um build product? What came out of that build and then command run how'd that build happen and that command run is what we're going to be focusing on today.

A

So the other part of this equation is archivist right. You have these attestations. You need somewhere to store them. Mikhail spoke about archivists earlier today at length, so I'm not going to go in detail, but in general it's an untrusted store. What do I mean by untrusted well I mean that all the data in archivist is individually signed. So even if the database is completely 100 compromised, if you don't trust the certificate Authority on that piece of data, you as a client are not required to trust it right.

A

What it does that indexes attestation into a graph database that allows you to follow that allows you to follow the nodes and edges of your supply chain to find the evidence that you need and then finally, it provides a graphql API into all this attestation data to make querying very uh efficient and flexible.

A

So right now, in our release, Branch, we do include support for tracing within witness. We currently use P Trace to collect traces. So if you enable your Trace functionality doing it while you're doing a witness run, what happens? Is we attach a p Trace to uh to that process and collect all the assist? Calls from that? So that way, if, when, uh when that process goes out, the compiler process goes out to go, touch a file or read a file. We know about that. So we stop that process. We hash that file and record it.

A

So we have and then we allow that process to open it and read it and the compiler can do what it wants with it. This is great and it works really. Well, it's synchronous. That means we stop the process. We stop the world and we record what's happening before we let it go on, but this also does have some downsides right.

A

um Hashing takes time, when you add a p Trace to a process that six times so it really does slow down, builds quite significantly and there's also some other issues with threading as well. Actual impact and real CI systems, usually they're smaller machines in many cases, but sometimes it may double or even triple build times.

A

So, currently experimentally, we we have implemented ebpf support in witness using tetragram, so what witness does is because it's actually executing your build process. We have that full context of what that build process is so we know what the PID is right. We know what uh what we also know what materials it should be touching right. We add that context to it. So then we send that information to the tetragon agent with the fine traces.

A

So we say: okay, let's, let's Trace everything that touches our working directory, where our source code is, if anything changes in there right, we'll know about it. um Let's.

A

If anything changes in there we'll know about it right and so we're able to um we're able to record a very accurate bill of materials for everything that happened in addition to any network calls. So we're able to prove things like hey. Did this this CI process make a network call nope, it's it's a hermetic build. We can say that did any of the files change. Well, the build processor happened that we didn't expect it to nope, so it wasn't tampered with right.

A

But there are definitely some trade-offs between using BPF and P Trace. Well, BPF, that's asynchronous! So what we're having to do with BPF? Is we get that grpc? Call that says? Okay, this file was touched, then we go hash it well.

A

um Our compilation process has already read that in by the time we've hashed it right with P Trace write that synchronous. We get to stop the world and actually see what happens so with BPF. We do miss some things. We do miss some file, hash changes. It is in our testing, it's pretty accurate and we do get enough to determine if it was tampered or not, but again, Ace being asynchronous. It just has some some trade-offs BPF.

A

It can monitor your entire system right with P Trace right we're limited to only monitoring that build process, but also we don't require a root with betrays right. We can run P Trace traces in GitHub actions on a shared Runner. We can't do that with BPF right, because we require system level access to install BPO um with p-trace.

A

The coding part of it is a little bit tougher and more prone to error with BPF the abstractions that tetragon has made have made it very, very easy to implement traces very, very quickly, um depending on what your needs are.

A

Our current P Trace implementation- we're not looking at any network, calls that's a future addition that we're going to make our current implementation with BPF right. We get all those Network calls. We know exactly where your build process called out. So if your build process called like called out to some malicious server, we can make some correlations with that and then there's that.

A

Okay, we shouldn't trust that, because it contacted hacked hack, chinaserver.com whatever, instead of uploading information or downloading information, like I said before the BPF agent, you need wrote on this right, so it makes deployment much more difficult if you're a small, medium Enterprise and have a smaller security staff, you may not have the ability to install a bunch of software all over your systems, because you have to maintain that with P Trace right.

A

We just dash dash trace and it just works because we control that process that we're calling BPF. That requires a modern kernel and then P Trace right. We have compatibility I, looked it up right, Unix V6 in 1976 is where that was introduced. So I, don't think we have compatibility back that far, but p-trace does so I'm gonna go and I'm going to do a demo. I got some code up on GitHub. You can go follow along if you want hit that QR code.

A

A

All right, so this demo.

A

Is non-deterministic, so you may not go that well, we'll see.

A

So the first thing we're going to do: okay, I'll, show you what this file looks like.

A

Oh okay, so the first first thing we're going to do we're actually going to attack solar, burst type tampering right, so I recreated a I I built a red team tool. It's on our GitHub repository that emulates a solarwinds attack, but in a Linux environment and using go. What we do is we're looking for any invocation of the go binary and when we see that go boundary we attach to it and we look for it trying to open a file called main.go.

A

If we see it open a file.maine.go, we we inject some malicious code into it. We let the compiler start back up again and then once the compiler is done, we replace the original code back with it. So nobody knows knows the difference. This is. This is very similar to all the similar to all the solar winds attacked.

A

um So let's go to the demo.

A

All right, so the first thing we're going to do here is we're going to actually start tetragon and then we're going to start our solar sploit tool and then we're going to run witness. Oh, so we're going to name a step, we're going to Output an attestation watch prefix right here. This is for the BPF right. This is we're going to watch all the files in in our build folder um tetragon address. That's a grpc address for the tetragon agent right here we are providing a key witness. Does support keyless.

A

So if you're running this in a build system, you do not need to provide a key. You can configure it to use oidc credentials um and then we're going to just run our Command and we're going to Output a file called hacked and then on step. Four we're actually going to show that at the station and if everything goes right, you should see the hashes changing all right.

A

So it needs root.

A

So now that exploit is running. It's looking for any indication of of go here, I'm going to send.

A

Let's do a different screen.

A

All right, following the Target now we're sending the um stuff to tetragram we're setting the trace.

A

All right, you can see a bunch of stuff happen when I switched over the screen, so we have in a debug mode. So we're getting all like the different events that happen, um and you can see here that this exploit looks like it worked. So, let's see.

A

All right, so what we're looking for is file hashes right.

A

All right, so you can see the file hash we started out with is um starts with the zero. A zero Alpha go down. Go down, go down! Oh look at that. It changed process two three, eight, nine one: zero change that file hash from zero Alpha to a six nine right, that's probably not good! That's our main dot go! Nothing should change that file. So we know from this Trace that this build has been tampered with. So we should probably do some alerting on that and do some mitigation.

A

Right and because witness includes a policy engine, you can actually do that with a witness. You can write a Regal policy that evaluates this to make sure this type of stuff doesn't happen and and make sure this. This policy is applied to your build pipeline or integrate witness into something like an admission controller, to make sure this stuff never gets in your kubernetes cluster.

A

So let's go look at the policy all right well time for demo, two.

A

All right, so here's another use case of this technology. um Let's say your compiler was compromised right. You had a malicious actor that changed your compiler or it could just be a cve, that's really bad! That makes anything compiled by that compiler vulnerable to a a remote rce right.

A

So what we want to do is find everywhere every single artifact in our system, every single workload that use this compiler.

A

It's really difficult to do this right now, you're, most likely just gonna, have to rebuild all your software and redeploy it again and if you're working anywhere that has Legacy software, we know that this isn't always possible.

A

Right so you can see we still have tetragon running there.

A

We're gonna stop this though we don't need our exploit running anymore. Oh I want to show you something too. Just to show you is, was actually hacked yeah, your code is hacked, so that's that's what it injected in there um all right. So let's look at this demo next.

A

So the first thing we're going to do start. Tetragon we've already started tetragon. So then we're going to build with witness and then with the experimental BPF tracing, then we're going to create a policy. I'll show you that policy and we're going to sign that policy. One important thing about Witnesses, it's different than a little. Some of the other policy engines you may know about is that we require policies to actually be signed. That way. Not anyone can just put a policy in there like a break class policy right.

A

You would need to sign that break class policy in order for that to be accepted.

A

The number four we're going to verify and we're going to make sure that that malicious compiler was not used in this build in this case. Yes, it was used in this build, so we should get a deny poly deny on this.

A

A

All right we're setting those traces based upon the PID of this process. We get those debug messages.

A

All right so we're actually using our online public uh version of archivist here, so we stored it there. So hopefully the Wi-Fi works. When we do this verify.

A

So we're going to sign that policy. Actually, let's go take a look at that policy. Real, quick, too I just want to show you what it looks like.

A

All right, so what we're doing is we have this attestation right here? This witness command run attestation.

A

And then right here we have actually no Regal policy in it. Uh-Oh I may have booked this demo we'll see.

A

A

And we're going to verify it right, so this succeeded because we actually didn't have a policy on it. Let me let me pull that code out of GitHub I. Have it on Old command. I'm! Sorry about this.

A

All right so we'll take this Rigo policy right here.

A

A

Looks right, doesn't it I think so.

A

All right so now we we created a new unsigned policy. So let's go ahead and sign that.

A

All right cross your fingers right and now that failed, because we had that hash in there of that compiler. We didn't want to use anymore. So now we've taken something that takes a lot of time. If you talk to anybody about how long it took them to mitigate against something like heart, bleed or log4j right, if they have the policy engine like this, and they have these attestations about their software right, we can really lower that time and that's what we're trying to do here.

A

um So let me go back to my slide. Deck.

A

That's it and uh if anyone have any questions, what are your questions.

A

Yeah, so witness is just a binary that requires no root privileges at all. So what we did is we actually have a git lab Runner that we integrated with um so it took pulled down the gitlab Runner code and I found the part where it creates the shell command that that gitlab actually injects into that container. We just instrumented that with witness that works great.

A

So now, if you're, using that gitlab Runner every single time, you run a command developers, don't even need to worry about it right, just if they're, using that gitlab Runner you're, creating some attestations right for GitHub, we did something a little bit different, we're it's it's open source right now, but still experimental. So we have a git.

A

We have a witness, run action that that just takes some commands and then it'll run whatever's in there right and wrap it with witness Jenkins something similar I think we're looking at maybe making a Jenkins share library or integrating with the agent for Jenkins. But that's probably going to happen within the next couple months or so.

A

That your file change the thing you want to express yeah, so I am not the best at Rigo, um but what you're doing is you'd Loop through all those sha songs right and anything. That's in your working directory is your code that shouldn't be changing right, shouldn't change and so I would create a loop that went over and verify that all those Josh slums didn't change.

A

All right any more questions.

A

All right, if you do have any more questions, you think about anything else, I'm going to be around all week until Friday I'll be at a bunch of events. uh You see me in the hallways, otherwise hit me up on Twitter. Dms are open. Linkedin hit me up on messages there um or you can hit a contact us on our website testifystect.com, but we love open source.

A

So if anyone wants to collaborate, got some use cases for us, we'd love to work with you hit us up on GitHub and let us know how we can improve this, make it work for your organization. Thank you.