Cloud Native Computing Foundation Cloud Native SecurityCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: See It to Believe It: Bringing Observability to Otherwise ... Parth Patel, Kusari & Shripad Nadgowda

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

See It to Believe It: Bringing Observability to Otherwise Opaque Container Builds - Parth Patel, Kusari & Shripad Nadgowda, Intel

Container build is arguably one of the most security sensitive operations in the whole application supply chain spectrum, which has largely remained opaque to date. It is typically implemented as a multi-stage process in the Continuous Integration (CI) pipeline that includes cloning the source code, resolving and downloading dependencies, compiling and packaging applications and finally publishing the built artifacts. To establish trust in the final built artifact, it is not sufficient to ensure security guarantees around just the built artifact, but it is critical to provide provenance and integrity assurance for every action in the pipeline that went into building that artifact. While tools, such as Tekton Chains, provide visibility into the steps that were performed and components that were used during the build process, we are still missing the lower level syscalls that were made. In this presentation, Parth and Shripad will present an open framework using tetragon to bring out-of-band runtime visibility and provide automated attestation for tekton based CI pipeline.

A

Awesome good afternoon, everyone so today we're going to talk about runtime attestations. So that's you know how we bring observability into otherwise opaque container builds. So a quick introduction, um I'm parth, so I'm a co-founder at kosari and kusari is a supply chain. Security. Startup and our focus is you know, building tools that help developers create secure artifacts, but at the same time answer those hard questions of exactly what are they ingesting previously to that?

A

I was a Solutions architect that worked in regulated Industries, as well as commercial and I'm, also a maintainer on various open source projects such as intuitive attestations in Toto, golang, Fresca and guac I'm going to hand it off to my colleague treepod to continue the presentation.

B

Thanks thanks Bob hi everyone good afternoon. My name is I'm A Cloud software architect at Intel I'm, currently leading a few core services at Intel developer, Cloud that we announced last month and security is one of them and I'm also motivated to address and go on the Journey of addressing security challenges in the open source communities.

B

So talking about security right, so almost every discussion we have leads to software supply, chain security and for good reasons, and yes, we will be talking about that in this particular talk.

B

So, let's start in some familiar uh place right. This is a the attack. Factor, that's come from salsa I. Think most of you have seen it many times. It is peculiarity with it. It shows the vulnerability all along the way from starting from developer when they are building their code and committing to the source repository to the way how we are building it, how we are ingesting the dependencies and how we are packaging, it all the tools, all the processes.

B

They are vulnerable for compromises, which is a very uh risky thing, and then how do we start addressing these challenges? So there is another good characterization that I see again coming from salsa is we need to think of our defense shield into three different dimensions right? It consists of how do we secure our code?

B

How do we secure our dependencies and how do we secure our build and there are different material levels in which how we can address these challenges in this particular talk, we are going to focus on the build aspect of this particular defense Shield, and it's not like. We are starting from scratch. So if you see in last few years last couple of years, we have come a long way right again, going back to the same threat model that we just saw.

B

We have added a lot of security controls along the along various Dimensions like when the developer is committing their changes. They can sign their commits at code repository. We have multi-factor authentication, secure configurations enabled for build. We have the desired state of formatting and reproducible build when we ingest dependencies. We have vulnerability scanner and the some maintainability maintainability checks through scorecard that we are doing.

B

We are pipeline signing security, so we are basically marching on this journey of improving our security and now, let's talk about the the Hermetic field, and the reports will build so build aspect of this, so this is again desired properties and we are not there yet right. So we need uh with this is a journey that we need to basically go together now, zooming into the.

B

Why secret, why security or build is important, so we'll basically zoom into the what build process essentially comprise of so, if you think of a typical typical build process, it starts with some source code right source code is input to it, and the source code might be what source could we write like python or Java code? We have some dependency manifest like requirement.txt for package log.json, and then we have some build recipe which could be a Docker file, and then this is input to our build process where we have multiple phases where this build happens.

B

So first thing is: we we set up the build environment. We set up our base images. We install our build tools, then, for any unresolved dependencies. The the dependencies which are not pinned in the Manifest those gets resolved to their correct versions, the dependencies their Registries get resolved in lot of cases. A lot of organizations have their private package Registries like artifactory, so this this gets resolved at runtime at during the build process, and then finally, these dependencies are downloaded. They will compile this of our application and we package them.

B

So there are so when we talk about build right, it comprises of whole spectrum and things can go bad in any of this uh different phases of this build. So what things we are trying to address here right. So when this build is happening, uh we want to know that what build tools are being used right. We don't have visibility into it because most of these phases that I mentioned they are Dynamic and they are non-deterministic.

B

We cannot determine what the package will get resolved to at build time where the package registry or my package registry mirror, will get resolved to so these are all Dynamic and they might change based on build to build right, based on the input to the input configuration that we provide.

B

So we need to know we need to have this visibility like when the build is happening. All these stages are happening. What build tools are being used, uh whether my building environment has been compromised again, we don't want to go in detail like, but the solar wind right. We are all heard it many times last couple of years, so we don't want to pick on them, but yeah. That is essentially what we are trying to address here like if we our building environment, are compromised. Then we need some visibility.

B

If we can uh someone something that can signal us right uh where my dependencies are getting resolved from, are there any intermediary artifact getting produced and what are those right and again, we talk about hermetic bill, which is the uh which is a Holy Grail that we want to achieve. But if you are building it and it one of the condition or the one of the requirement is it doesn't, it should not pull any dependency from internet right. So how do we ensure that when the build is happening?

B

How do we ensure or who can attest that this particular build is hermetic? So those are the challenges that we are trying to address in this particular work. So what we try to do right. So what was our approach? So we said. Okay, we have some build servers. We have this build process which is running on some build server, which can be kubernetes worker nodes which can be genetic energy and so GitHub Runner.

B

What about we put this monitor in between our build actions, build process and our build server, and what this monitor essentially need to do. It will collect the trace. It will collect the trace of all the network connections that are happening. It will collect the trace of all the file operations, any credit operations that are happening and the process executions.

B

What processes are getting executed, what inputs were there and how this project exited, and we don't want to collect this information just for the sake of it right as if we don't have enough data already in our supply chain uh idea. Essentially, our objective essential is to to generate some semantic understanding of this data when we collect the trace, then make this stress understand this straight path.

B

This stress and create some attestation that we can make some statement about this build process that this build process is hermetic, or this build process made this ah connections or this it modified this set of files. It executed this list of processes and then we can build some policies and we can make some decisions out of it right. So our idea behind this monitor is to create that registration which, which can be actionable. We can take some actions on top of it.

B

So how do we started when we started the design? We had different choices and as I think yesterday. Cole was also mentioning about this, so we started with the the common the estrus piece dress, where we actually started tracing this uh wrap our build process, build actions into these tools and we started tracing them. We call it in band because now we are essentially, we are the parent process that are invoking all these build actions, but then we suddenly find right with this approach.

B

We need to change all our existing build tools, build actions to wrap them under this uh protocol.

B

It was hard to nice and then this this tools, if you run the P dress, it collects the information about a process right and we need to add a lot of decorations around it like this process belong to this part. This part belong to this Tech, this stacked on task or these actions, and it belonged to this pipeline that all external metadata collection and mapping that was very hard. Then we say okay, what about? We do something out of band, so we said: okay, we start with a BPF which will run as the agent.

B

It will collect this stress, all the trades that we need and give us. This uh information collect this information transparently. Right again, the challenge was the same. The contextual information gathering was a challenge. We couldn't get that pretty easily. There was a lack of policy support right. We couldn't customize what information need to be collected, what we need to turn on and off, so we couldn't do that. Then uh last clip on this project was an Australian. It was like a new kid on the blog that everyone was talking about, and we said, okay, why?

B

Not? Okay, let's explore this, what this tetragon is and then we suddenly find okay. This is actually the right tool for the that we we are. We can leverage and we can build on this. But again, this whole solution is not based on tetragon. It can be applied with any other tracing tool and as the path will go in detail, uh we are basically talking about one particular implementation here. Right and the good thing about tetragon is, it is kubernetes is aware: we need to collect the trace.

B

It already has the additional metadata where it can. We can associate this context and then it has automated policy control right. So we said okay, this is something that we can go with and then once we monitor the data once we collect the trace, uh our next objective was: how do we format this data? What do we do with it right? So we said: okay, we need some attestation format and for this our options was very limited. It was actually a single option. We said okay, we know the in total is the right approach.

B

We will go and create an intertor station, and then we also proposed a new predicate type for runtime registration. This is something some work that we have been doing.

B

uh So this is the approach that we, the design choices and the approach that we are taking and, uh if you think of as I mentioned right, so this runtime monitor is happening when your your pipeline is running. So if you, if you want to see where this particular runtime registration is happening during the scope of your pipeline, so this this particular picture, it shows the life cycle of a pipeline right. You start with the composition. We compose our pipeline from a lot of Open Source component. We install our setup our pipeline.

B

We, the pipeline, gets triggered by some events, uh GitHub events or manual trigger, and then it is executing- and this is where the runtime registration, Monitor and attestation is happening right and the the questions that we are trying to answer is how the artifact is built right. What is happening when the artifact is getting built now there are other Solutions uh that are creating the salsa registrations for build like taked on chains or some GitHub actions that you have probably heard of those happens after the pipeline completions right.

B

So once the pipeline finishes those registrations get created which tells you what task actually ran for that particular your artifact or your image, for instance, that get built the runtime registration is happening. One step one stage before this, uh this Saul's attestation that are getting generated and again, as you will see the how these two technologies merge together and to talk more I, will invite past to talk more about it.

A

Trip I was saying: is we wanted to use tecton pipelines so for the general audience, if you're not aware of what clecton pipelines is? Is that it's a CI, CD Cloud native solution? It builds everything in terms of containers and it's very easy, because it's uh it builds everything in a femoral container that can be destroyed and isolated. It also has the ability, with tecton chains, to create attestations right. It creates all statistications signs, all satisations, that you can push into oci and um in other directions.

A

So we started off with this tool in mind, because it's because right we're going to be working with tetragon tetragon is kubernetes aware. Tecton is based on kubernetes. Kubernetes is built on kubernetes. It does everything within pods and containers. So you can utilize those two technologies together to create our runtime attestation, so, like sharepa was saying, tetragon is a tool that we chose in this case. It's an ebpf solution right because it's very it observes uh using the PDF technology.

A

um We don't want to limit to. We don't want to fix two tetragon right, so we want to make the runtime attestation very generic, but in our proof of concept we chose tetragon because of the benefits that it provides out of the box at the same time, right because it's kubernetes aware it can see the pods, you can see the processes running within the Pods. At the same time, right it can see, file access, network connections right. You can see a lot of those things that we do want to collect.

A

So we don't want to introduce another controller right. The kubernetes space is already full of controllers. We don't want to introduce another one. So what we decided is that, let's, let's just have the tecton change controller, be the one that creates that attestation for us right, tetragon is going to collect all the logs for us, communicate that over to techton chains, and it's going to create that uh sign at a station for the runtime station and then push that into any kind of direction. Right. Tecton chains already has the ability to create attestations.

A

It already can do science right, you can already signed attestations, push to oci or push to a document database. It has that ability. So why not utilize all those capabilities instead of starting from scratch? So that's what we decided to do so, let's get into a real world example of this.

A

So real, quick starting off I want to show off that this is a simple kubernetes cluster on you can see. All that's running is the tecton chains, like I mentioned before the tecton pipeline controller, which has two components: the controller as well as a web hook and then tetragon itself right. Those are the only pieces that are running I'm, not we're not introducing anything extra. Besides, what's already coming out of the box with tetragon or tetragon, as well as tekton.

A

So what I'm going to be showing off here in this pipeline basically, is that it's a multi-step Pipeline and it's going to go through and it's going to clone down the repo it's going to build the artifact, it's going to scan it it's using using gripe, it's going to create an s-bomb and so forth. It's going to do all these different steps and at the same time, what's happening is that tetragon, in the background, is actually capturing all the processes that are running for each one of these tasks.

A

If you remember before I said that tetragon is kubernetes, aware right, it can see specific processes within a task. Each one of these tasks are running individually in each each one of the individual pots, so they can collect all the processes that are running and we can go and actually analyze them in the future kind of thing. So it helps us oops, sorry about that.

A

um So in the future, it helps us go uh see exactly what's happening. What happened in each one of his individual tests and make policies based off of it? So you can see it just finished. um All the all. The different steps in this pipeline completed so now, let's take a look at the actual result.

A

So this is the build step right, the build task that actually ran for this specific Pipeline, and you can see the main point is what I want. What I want the audience to focus on? Is these four things here the pipeline or I'm? Sorry, the payload is the salsa attestation that tecton chains already generates right. So that's that's something: that's not that we're generating that sector on chains doing that, as well as a signature down here.

A

The new thing that we are adding is these two pieces right here, which is the runtime payload as well as a runtime signature like I, said: uh pecton chains has the ability to write tetragon is collecting this information. Tecton chains will take that information and create the attestation for us, and it's appending that to this uh Into The annotation here in this task run as a label. You can pull this down and actually view it. So we can in this case right it's it's actually getting pushed into the actual kubernetes object.

A

We can push this into oci or some other object in the future. If you wanted to and because it's signed right, we can go back and use cosine to go verify. Yes, this is signed by my private key and that is valid.

A

So taking a look at the attestation itself that it generated, so what I did is that I cut this down for size right because it generates a lot of data and I'll talk about that. You know why what we want to do about that, but for now kind of talking through what this, what the actual annotation actually looks like right. So, like srupa said, we started with the in total attestation format, so you can see that the subject is up here.

A

What artifact that it built right the pipeline built and then down here, is a specific right starting from down here. The predicate type is a specific at the runtime attestation itself. So the first thing we see up here is the monitor, ID and like I was saying before is that we wanted to make sure that we made this at a station very generic, so we can choose different monitors right. It does not have to be touch or gone. It can be.

A

You know, P Trace, s-trace, some other tool in the future kind of thing and then down here. Is this specific thing? That's actually building this artifact right building the statistation so in this case right those tecton chains, but again it could be something else. It could be GitHub, GitHub actions or gitlab or something else coming down line, and then this is the actual task run object that we are actually looking at that that actually collected these logs for down. Here you see tracing policies. So what this kind of does is out of the box.

A

Tetragon has some capabilities right, but if you wanted to, if you want to include more like, for example, if you wanted to include network connections right, if you wanted to trace network connections or if you wanted to check file access and such you have to give it a tracing policy that says yes enable this and look at this information and capture this information. So what this is doing is okay.

A

We also want to know exactly how the tool itself right, tetragon or some other tool is configured and how is it capturing that information or what was it supposed to capture so, for example, right if I don't see this tracing policy, if I don't see this TCP connect policy here right, if I don't see this one, then I know that okay, it didn't collect any network connection right. It didn't collect any kind of network process logs for me, so that can't say for certain that is hermetic, because now it's not collecting anyways.

A

But if I see that this tracing policy existed, there's a TCP connect policy right and if it did not collect any kind of network logs, then I can say for for certain that it was a hermetic build. So that's why we want to capture this tracing policies so down here. If I scroll down, is the actual processes themselves right, so we kind of broke them out into event types.

A

If there were file access types exit exit entering um you know, what kind of binary was it being used at the time and what kind of arguments are we taking and if there were privileged calls themselves? So, if I keep scrolling down, you can see an actual example of, because I had that tape. The the network policy active right.

A

You can see, there's a TCP, send message here and it did actually send message to some kind of public IP address right so right away, I know that this process was not hermetic and I can't make that claim anymore.

A

Foreign, so what do we do with all this information like I was saying, like it's information overload I had to cut down the size of that attestation because it was pretty massive right, so it seems like okay, it's going to give us a lot of information back, but it's collecting information for information sake like we need to do something with this information right so because they're, one time, attestations and logs themselves are pretty massive right. They can end up being 500 megabytes. They can be.

A

You know they might contain confidential information that maybe companies don't want to distribute or they're even hard to distribute because they're, so massive right, they're, 500 megabytes or something. So what do you do instead? So we're? Actually in the talk uh in the in Toto attestations group we're actually in the talks of maybe using you know the software supply chain attribute Integrity, otherwise known as sky and kind of having, and you know, for example in here you can see that there's an attribute that says it's hermetic right so saying like Okay. This artifact was built.

A

This subject right here, oops. Oh sorry about that, this artifact here was built and is built in a hermetic process and and the evidence that says so. Is this the runtime attestation that we generated that's stored somewhere else right that could be stored in some kind of other public data database and that you could go access if you wanted to manually check, uh but what what we do instead, let's say, for example, in the future, is that tecton chains right it generates right.

A

It goes through the runtime logs to check to see if there are any kind of TCP connect, calls or anything being made. um It checks to see that the tracing policy was actually active. Yes, it's supposed to collect, Network logs I didn't see any kind of network logs. Then I can be for certain that it is a hermetic build. So I can create this smaller attestation and use this instead and distribute this out with my artifact and push it into oci.

A

And in the future, we're thinking about incorporating this into guac right grock is graph for understanding, artifact composition. So you can see from this diagram. It's you can see the artifact in the middle, which is that orange piece you can see. There are two attestations. You can see, there's two lines pointing inward Square stations, whereas the salt satisstation, as well as the runtime adaptation, as you can see it was built by a monitor to buy so what guac we're trying to do with is basically having all this kind of information right.

A

Like I said we have this information now. What do we do with this information? So now we want to check you know, for example, I want to have a policy that says: I only want to make sure that I I run any kind of artifact that was built hermetically in my production environment. That I can go query for this kind of information, and then it can. It gives us gives me back with confidence saying like yes, I saw, there's an annotation associated with it.

A

There's a runtime attestation associated with it and I can I can now run this in my production environment because it meets all those policies and I'm not going to go too much into detail, but you can take a look. My colleagues are going to give a talk about this on Thursday uh there's a QR code you can scan, but this talk is going to be on Thursday. If you want to learn more about this, so the road forward for this is that we definitely want to make that.

A

So we have a predicate proposal in place right now in Toto, in total attestations repo, it's currently in draft, but we want to make it we want to make it more generic. So, at this point, I think we're looking for Community feedback involvement from others to see exactly okay.

A

What other information do we want to capture, and does it Encompass everything that we need so we're working on that piece for the specific proof of concept that we're working on there's the one of the things I was working on is getting the tetragon policies so that tracing policy that I showed earlier. That was a quick, quick code change that I did, but basically we want a formal PR. That's going to be put in that kind of gives you more information in a structured format of exactly what configuration tetragon is set up with.

A

So that's that that's that proposal, as well as formalizing the change right so change so I showed you the change integration with tetragon push that informally into tecton chains, so that people can start are utilizing it and then lastly, write the guac integration. So we can use this information to create policies and then use that for governance.

A

So a quick recap right, so we want to have visibility into the build, so we wanted to use. You know we wanted to integrate with existing tools. We don't want to introduce anything new, so we can use tetragon tecton chains to create the signed, runtime attestation that we can use and then take that information, and so that we can generate this. uh The attribute Integrity right, that's that sky at a station that we can use to check to see.

A

Yes, if this is hermetic, because the runtime out of session said so- and here is the evidence and here's the evidence for the runtime- and you can go query it manually if you want to so definitely get involved, we have our our own slack channel in the open, open, ssf, runtime, attestations and you know, here's our Twitter handle so definitely contact us. If you have any questions or if you have any comments or if you want to collaborate more. Thank you.

A

Any questions thank.

C

You very much anyone have any questions one over there, I'm running.

A

And real, quick before I forget, we do have T-shirts. So if you guys want to pick up a cool, kusari t-shirt, see or see my colleague in the back and we'll get you get you that great.

D

Yeah great talk very interesting stuff um I I, like the Hermetic, build part of it and there's also a lot of other ways that people have talked about doing things with build systems to try to protect them better, like, for instance, reproducible builds or having some kind of Hardware root of trust that you use on your build system and stuff like this and I'm. Just wondering if you've thought about. Would that use the same mechanism that you're planning uh that you're proposing here or would you have separate mechanisms for them for some reason.

A

Yeah so I think for a reproducer bill builds like a sheep. I was saying we definitely use this mechanism right because we can again go through and validate that. Yes, all the same processes ran for two separate bills if they were running um to validate that they were producible, so there are definitely more use cases for this right. In this case, we chose stormtic because I think it makes a lot it's easy to connect with and easy to understand, but definitely there are definitely more use cases that we want to explore in the future.

A

C

Any other questions right over here.

E

So there was a presentation yesterday on testify: SEC archivist. Yes, so how does this compare and contrast with what.

A

So Cole and uh Mikhail are actually working with us, so they were just on a call with us last week, um so they are also going to take this. um So we are going to like we're working together on this. In terms of we want to formalize this attestation format right, the predicate type so that they can also produce the same type, so we're we're going to work in collaboration. So that's what I was saying.

A

So we want to make this a standard, runtime attestation, that people can use in other contexts like he's using it using. um You know, he's actually using tetragon in this case right, but you can and he's using another his own witness tool right, so it doesn't have to be chains and tetragon. It could be something else, and this and everyone kind of creates a runtime attestation that everybody can benefit from.

C

Okay, anyone else going once going twice great well. Thank you very.

D

Much thank you. It was great talk.