Cloud Native Computing Foundation TAG Security / Supply Chain WG, 2 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF Security TAG Supply Chain WG 2021-09-02

Description

CNCF Security TAG Supply Chain WG 2021-09-02

A

B

I can't hear you, can you hear me.

C

B

I hear andres, I don't hear you ava.

B

ah Fancy meeting you here andres likewise.

D

I'm currently driving, and this.

C

Bluetooth, carplay, doesn't let me access this room. Lets me.

D

Access all the other apps.

C

D

Gonna go on mute because my phone is about to dive out and put it in the cradle. But.

C

Okay, yeah: what.

B

Go ahead. No, I was just saying, as I'm sure, you're aware, I'm causing no trouble whatsoever.

D

C

Well aware, what.

B

Brings you here today get bombs.

D

You had been working on that, yes,.

D

Yeah, I I remember looking at the ride up and some of the work you had on your personal github.

C

ah Yes, it's been a while.

B

I don't think that that actually the personal github is quite a great state yet, but the write-up is getting better sweet.

D

Are you going to be sharing some of that yep.

C

C

E

E

Nothing still nothing.

C

I hear you now.

E

Okay, I'm using a backup microphone that is not very good quality.

E

E

One more test.

C

B

I hear you but you're very you sound, very tinny how about now slightly better but faint.

E

That better, oh, it's still very.

B

Much enormously better.

E

Okay, great, no idea why it lost all of the settings, but here we are.

C

D

Eva so you're scheduled to present in the interest of time and punctuality. Why don't you get started now that your av is perfect.

E

E

Let's see yeah, I guess brandon is not able to make it today.

D

And michael should be joining any minute: okay,.

C

You mean brandon mitchell, isn't he here he's traveling no random.

E

And there's michael, I see him just popped up in the in the chat list um cool. I think enough. Folks are here. I think uh I'll be doing a little bit of the talking today, but I think ed's going to also do a chunk of the presenting we've been poking at this idea of get bombs.

E

I might have mentioned it to a couple of you in the past.

E

I don't know I'm still letting my morning, coffee settle in, um I'm gonna drop a couple links in chat here.

E

I think ed will run through a presentation, and I've got a um sort of the beginnings of a white paper on this and what we're really looking for is feedback and collaborators who wants to work on this with us who wants to come tinker and hack on it um and I'll probably have a bunch of comments both of us might at the end of this, of who else, we've socialized it with um just yesterday, I met with some of the oci folks uh about getting the compatibility of this idea with containers seems to fit pretty well so ed, I'm gonna.

E

B

Sure so um we're gonna be talking today about a git bomb which is sort of an attempt to get to a verifiable artifactory. um It's particularly useful for enabling launch time security scanning, but it can be used for a whole variety of things. I wanted to briefly stop and call out um you know ava black and frederick house, who are collaborators that I have in this particular endeavor, um so really quickly. How many of you folks, like the unix model, quick, show of hands? You know they basically build a tool. That does one thing.

B

Well, that can be you know, strung together with other tools: okay, good, fantastic you're, gonna, love. This, then because it does exactly one thing. um So this is a security crowd, so I'm sure you're familiar with the colonial pipeline hack. That happened recently in the resulting uh cyber security executive order in the united states that came down that was primarily focused on two things.

B

One was zero trust networking, uh the other one was s-bombs or software bill of materials, um and so when this sort of landed, s-bombs became a hot topic again and usually when you look at sort of what people are doing with software bills and materials, they're building some sort of a tree where they've got some artifact that they're talking about and then a whole bunch of metadata about the artifact and artifacts can be things like executables, dotto files, source code files, class files, you know, containers, they come in all kinds of shapes and flavors and they typically have all kinds of different ways they get identified um and then metadata is stuff like okay.

B

What vendor did you get this from? What's the release version? What's the contact information give me licensing, information, copyright, information, um etc, etc, and they usually get built out sort of this way in a tree, and there are a few competing standards in this area: suid spdx, cyclin, dx, I'm mostly familiar with spdx.

B

I used to work with those folks way back in the early days, uh which is kind of how I got this thinking about this again, and so, when thinking about this, like this, this kind of a thing when you've got a tree with metadata. It gets a little bit of mess, get a little bit messy for a variety of reasons, and so often when faced with hard problems, I'll, try and take inspiration from very simple systems that are powerful, which is sort of how I got thinking about get again.

B

So how many folks are familiar with git internals like what's actually going on inside, get okay? How many folks are aware that git is actually an object, data, an object, store masquerading as an scm, okay, good um and, and so basically git says. Okay, I've got objects and it's got a very simple object: format, there's some kind of a type, a space the size as an ascii string base, 10 of the content, a null character, and then it's content now some interesting things about this.

B

When you look at the different types, you've got blobs, which are usually thought of as files, but they're really just byte arrays. You get trees, which sort of represent the file system structure of things and commits which we're very familiar with, because that's usually what we're manipulating.

B

um And then you could take this git object, the header and the contents, and you apply sha-1 to it and you get a git ref and the git ref gives you a almost certain to be unique identifier for the array of bytes. Now we all know that hashes in principle can collide all hashes can, in principle collide.

B

um However, um it was recently noted by github that they have yet to see a single git ref collision anywhere in their entire system, which makes me feel very comfortable about the unlikeliness of collision.

B

um Also, probably some of you are probably jumping up and down screaming right now in your heads about shawwan and the shattered folks and the fact that they claim to have broken it. um And yes, they have in fact demonstrated that in a pdf in which you have infinite flexibility to introduce junk data, you can in fact very expensively break sha-1, but there's a lot more structure to this than what you get in their example.

B

That makes it enormously harder to actually break git ref and I'm aware of no actual demonstrations of breaking git rough or anybody even getting close at this point. But then that result is you get this 40 character, byte string of which we're usually familiar with the first seven or 20 bytes? Frankly, and of course, when you look at blobs, as I said, that's just a bite array now the interesting thing to really keep in mind here is it's the content of the files. This does not include file, name or path information.

B

It does not include mode information, it does not include any metadata, it's just the contents, so any file anywhere with the same contents will have the same blob object now. This is really important when you're building an artifactory, because when you're building an artifact tree where that file happens to be located is metadata about it. It's not actually a fundamental thing about what gets built into it.

B

If I build something in my home directory totally different sets of file names, then you building your home directory, but we're building the same source code to the same executable.

B

Then it doesn't actually matter that we were in different home directories when we built it in terms of all the characteristics around vulnerability, information, licensing information and everything else that you actually care about with an spom, um and you know so. Basically, this is a super important and useful characteristic of get lobs.

B

So here's sort of the first very simple realization, please. No none of these realizations are particularly smart, they're just blindingly.

B

Obviously, once you realize them, every artifact is a blob right, because every artifact in a system, whether it's an executable, a container a source code file, an object file whatever it is, every artifact object, you're dealing with is an array of bytes, so you can assign a git ref to every artifact by just treating it like a blob and in particular this is incredibly useful, because when you're talking about software, the leaf artifacts in every artifact tree are your source code files and the vast majority of those in the universe are already being stored in git, which means we have already freaking indexed most of the leaf nodes in the universe.

B

Using this identifier.

B

That was the first realization. The second realization is, you got to separate the metadata from the artifactory, and the reason this is crucial is that metadata can vary independent of the artifact tree I mentioned before. For example, there are people who want to put contact information into the metadata for the artifact tree. Well, when you've got companies that are providing support for things, I may send you exactly the same artifact from cisco that you got from yoyodyne, but we may have entirely different contact information.

B

So, if you've put the metadata in line in the artifact tree, we're claiming that we're giving you a different set of bytes, even though they're exactly the same set of bytes just because you pick up the phone to somebody different, and so you want to pull the metadata out of the artifact tree and then, of course, you can have the metadata reference back into the artifactory, but we'll get to that subsequently.

B

So, just to sort of give you an example of what I mean by artifact trees, because pictures help a lot of people right. This is sort of a familiar one from the sea world. Right, you get a bunch of dot c files; they pull in a bunch of header files, those get compiled, 2.0 files, which are another node in the tree. Those get linked together into an executable so like this is sort of the hello world of. How do you think about this? The c case?

B

Now, of course, you can get more complicated. You can say well what, if I've got static, libraries that I'm linking in right? Well, that's actually not so hard, because, again c and h, files get linked, auto files in the case of the static library, those get linked into a dot a file. When you build your executable from a static library, you link the dot a file into your executable. It's still a tree, and you still have these artifacts that are all bytes.

B

One that gets really fun is running executables because oftentimes, at least in the sea world, when you have an executable um you're, not actually really executable by yourself. You've got a link in shared shared libraries, but again these are just things with their own artifact tree, and so you can actually reason about the running executable tree independent of the executable, because, for example, it may be that this executable is vulnerable with particular versions of a shared library, but not with others, and so in terms of reasoning about vulnerabilities at runtime.

B

You actually care about the running x, um running artifact trade.

B

This also works for other languages go, does something very similar with autofiles and they files um java, albeit slightly boring, compiles java files to class files and rolls them up into jars.

B

Python is extremely boring in this regard, because py files get compiled to pyc files, and then they all get dynamically linked at runtimes. You you in python you're always dealing with the runtime artifact tree, and I can literally go on and on like every single example, I've looked at has something that looks like this that can fit into this particular approach.

B

So the the first part of the proposal here is to say, when building an artifact tree simply use the get ref as the artifact id when building your tree so separate. The metadata use the git riff as the artifact id.

B

Then the natural question you should be asking yourself is: okay great. We can identify the nodes in this graph. How do we identify the parent-child relationships right? Because that also matters a lot and so what I basically suggested for that is you very simple document that we call the git bomb.

B

The get bomb document describes the direct parent child relationship between an artifact and it's immediate children, so, for example, for artifact two, it has two immediate children, artifact four and artifact five, and so you just have a simple document that says blob space, the get ref of artifact four.

B

You know a new line, blob space, the get ref of artifact five, a new line. You keep going down the line and, of course you want everyone to come to the same conclusion. So you lexically order this, but it turns out to be really easy to lexically order this and arrive at the same answer any time, you're compiling four and five into two.

B

And, of course, you can do the same thing exactly with artifact three to get its get bomb file for artifact three and then, where things get fun is artifact one, because artifact one has children artifact two and three, but they also have children. So how do you capture the treeness of this and the answer is and for those of you who play with merkel trees? None of this is rocket science.

B

You simply say: look if you're an artifact that has um children who themselves have children, then you start the same way. You know basically blob space in the get rid of the artifact. That's your immediate child, but then, if your immediate child has itself a get bomb, you simply say bomb space and the get ref of the get bomb of that particular child.

B

So now these git bump files now describe the tree. You've got a very, very simple format for how to deal with these and because we're lexically ordering so we're going to get this the same way. Every time I can tell looking at this particular bomb. You know I I can basically tell if you give me the collection of the git bomb files that I've actually got a tree. It's the same method with merkle trees, that's used by git itself to make sure that you actually are getting the code that you expected from a particular commit.

B

Then you could have all of your metadata reference either the artifacts by their git ref or if they need to reference the parent child relationships by the get ref of the git bomb for that parent-child relationship right. So, if you want to say, okay, artifact 7 is a source code file that is the source of a known cve.

B

You can reference that as metadata. If you want to say, artifact 6 is actually licensed as gplv3. You can reference that as metadata, um and this way you can actually have all kinds of independent reasoning about metadata, um both in s-bom formats, but even also in other places, because the artifact tree gives you a skeleton to put all of that stuff on by the way. Stop me at any point. Folks. If you have questions, I love questions.

B

Okay, so the really cool part about this is the fact that metadata references into the tree is that it turns out. This means that git bomb is compatible with the other s bomb approaches already in the world, because all of them allow some form of external identifier that you can use to reference things and, and so don't think of get bomb as a competitor to the existing s bomber approaches, because we're not they're doing way more stuff than we're doing. Remember we love the unix model.

B

All we're doing is building an artifact tree with very high precision, um we're actually complementary to all of these approaches, as well as a bunch of other things you could choose to do in the future.

B

So this is where we get really fun, because now that we have a git ref for the git bomb for a particular artifact, we can ask ourselves it's small, it's literally 20 bytes in size. Where can we stick it where we can always find it and, of course the? Where do you stick it? Where you can always find it? Well, of course you stick it in the artifact.

B

So when you're dealing with elf files, which are things like executables, shared libraries and data files, just embed the get bomb identifier for the immediate children into an elf section named bomb- is someone trying to ask a question? I I'm not hearing, but I'm seeing some folks saying things.

E

Nope just chatting in the chat, I don't see any hands up.

B

Okay, that's cool, so l files. This is fairly easy and it's actually also cheap. It's sort of going to cost you about 89, bytes per executable, and while the iot people there do exist, people from 89 bytes is a stretch. That's not most of us ar files um they're. Basically things like that. A files um also, I think it's devs- are also done as dot a file ar files. You can simply embed the git bomb identifier into an archive entry, name, dot. Bom.

B

You can play the same game with general artifact um files, your general archive files, um the same thing can be done with annotations. In java, in pyc files, the python community could choose to adopt an understore underscore bomb that gets computed by the python compiler um we're looking at container images, there's actually space in the manifest for annotations.

B

You could call the annotation.bomb and stick the git bomb ref in there. So far every place I've actually looked for a. Where can I stuff a metadata into an artifact? I care about the only one I found that I can't figure out. A good way to do is the old style x86 boot sectors.

B

Those things are super tiny and have no well-defined internal structure that I can see so other than that every place I've looked.

B

This is a very doable thing and then, of course, the question is who embeds that get ref into the artifact and of course the natural answer is let your compilers and linkers do it, and the reason this is the natural answer is particularly if you've ever had the misfortune of trying to figure this stuff out from c your compilers and linkers are the people who authoritatively know what child artifacts are being built into the new artifact because they're writing the new artifact, um and so that becomes a very easy method and it also becomes a very transparent method.

B

Because then, if I sit down- and I write a program in c and I haul out a git bomb enabled lvm and I build it, everything is get bombed right. So if this moves its way into the compiler linker chain and a variety of people have expressed interest in doing so, then you sort of have this coming through with great fidelity, relatively free and.

E

I'll jump in there and say: oh please, a lot of the discussions. I've had folks like well, why not just parse the logs ed's point is right. You know you have the compiler linker.

E

It is the authoritative reference and also the amount of effort it would take to convince every um software developer out there, a project out there to change their ci system or process or remember, to run an extra step after their build is exponentially larger than the number of compilers out there that we could go work with typically compiler folks are also much more concerned about security and doing this. Well, it's just a much easier surface area to go. Get this work done!

E

That's enormously! It's one of the reasons I'm I'm focusing at that at that intersection.

B

Yeah, the other thing that I found- I I at one point was quite close to a group of folks who were basically looking at out looking after the build for a piece of software that had about 3000 developers working on it, so industrial grade software and that's where I basically learned that you can't trust anyone, but the compiler linker, because they were highly motivated to extract this information um and a whole bunch of other information about the builds in a reliable way.

B

And you know if you started talking to them, you got an endless assertion of corner cases where, if you wanted the compiler a linker for this, it broke down and broke down badly, um and then I sort of hum a few bars about how you might you know one of the ways you might propagate this forward through the tool chain so that you can move the breadcrumbs forward and collect them at the end of building an artifact. But I'm not going to spend too much time on that right now.

B

um The nice thing to realize, though, is that if you have an artifact say I get an artifact from a vendor.

B

That's been signed by the vendors, so they I know that's the artifact I got from the vendor, then I can look in that artifact and I can see the get bomb ref in that l section of that executable, for example, and then, if I'm given something that purports to be the get bomb, I can know specifically whether that's actually the get bomb or not because tampering with the git bomb would change to get rough and therefore tell me that I'm not getting the information. I thought I should get.

B

So distributing git bombs also becomes super easy. You can literally just do it by simple static pattern. You know, example: dot com, slash bomb, slash, ref, the get ref of the artifact could give you the you know the git bomb file that you're looking for um or they get. You know the you could do it by um you know you could ask for a fully resolved version with all the get bombs of all the children. You could also ask for metadata about a particular git ref, so it all becomes fairly referenceable in a static pattern.

B

The sort of thing you could build into cli commands or other usability to tell your customer stuff about it.

B

Now this is sort of the thing I think folks on this call are going to be most interesting and most interested in. So how many folks already see very clearly what this does for vulnerability tracing like? Is that obvious to everybody?

B

E

Question better question is: is it not obvious to anyone if so speak up now, so we can help answer that.

F

Yes, please, so let me guess: you're going to scan for.

C

Known get drafts to known artifacts with vulnerabilities, so.

E

Imagine at container launch time opa being able to express a policy that says cross-reference the get bomb of this container image against. You know all the all the hashes of all the things in it against this list of known vulnerable packages and if you find any matches block the launch of that container,.

B

Let me take it actually, but a step further down screw packages right. Someone has determined that there is a well-loved open source project that provides a crypto implementation and in food.c of that crypto implementation there is a vulnerability.

B

Unfortunately, food.c has been copied everywhere across the known universe, because it's bsd licensed it's a single file. It works. Nobody really has to deal with it. Much imagine being able to now see that food.c has been built into things in the artifact tree of that thing. That ava was just describing that's enormously more powerful than just packages.

F

um I I think I would have a question just about how uh willing the uh the the folks, who are sort of managing the vulnerability databases to include some of these things that I think one of the big issues uh we've seen is that you know um implementing any sort of change like that. You know, like often I think cves are mostly just focused around packages and also not really including a lot of metadata where sometimes it's like. Actually, this package doesn't include this vulnerability.

F

Just this one flavor of that package is actually including that vulnerability.

B

C

B

Yeah good good.

E

Before we run right.

E

Today, today, the vulnerability databases have no reason to include that data, because no one could functionally use it. Some, I would say some private parties may be including that in their scanning tools, but it's not there on mitre. If this were widely available.

E

We walk before we run.

B

I mean- and please also note- and I do want to distinguish this strongly from scanning tools, because scanning tools are usually trying to rummage through data and figure things out, and if you actually use scanning tools in a lot of these environments, you often get a lot of false positives that are difficult to sort through. It's very easy for them to miss things here, we're talking about providing a high fidelity, high precision picture of the artifact tree, uh and it's not so much of a matter of scanning.

B

It's literally a matter of just reading the git ref and pulling the thread, which is a far stronger position than scanning.

E

And far faster to execute at launch time.

B

Yeah, the other one that's actually kind of nice about this is in talking to folks who are interested in being able to disclaim vulnerabilities. um You can attach the disclaimers to artifact refs as well. So if take my apocryphal food.c, that has a bunch of you know really useful crypto implementations in it, but it turns out that it's one function in particular that's the source of the vulnerability.

B

If I'm a vendor, who happens to include that which I can immediately see because I've got my get ref my git bomb tree and I realize my customers are going to be coming to me and saying hey. I think you might have this vulnerability. I can attach a disclaimer say to the artifact id of my executable.

B

That simply says. We know we have this file, it's vulnerable. If you call this function, we have looked, and we do not call that function and therefore we disclaim actually having this vulnerability simply because we we are building this violet and that gets to be a huge operational issue if you're, actually a vendor who ships software, because how many folks actually ship software that industrial scale on this call.

B

So one of the things that happens, if you ship software in industrial scale, is you have thousands and thousands of queries from your customers saying.

B

I think you might have vulnerability your cvex for a variety, whatever reason and the thing is even if you've actually been really on top of your game- and you know you don't the means by which you communicate this to your customer is incredibly expensive because it usually turns into support calls simply being able to mechanize this, not only in terms of how you publish the information, but in a way that your consumers can mechanize their consumption of. It is a huge win for everybody.

B

So, as I mentioned, there are other uses, not just vulnerability stuff. I think ava mentioned assassination with opa. I love that use case. A lot um there's also a lot of interesting stuff that can be done around forensics.

B

um So if you maybe haven't gotten to ss station, you might be logging out for forensics, which would allow you to go back and know exactly what it was that was compromised.

B

um I've got people that I'm dealing with internally, who are very interested in repeatable and verifiable, builds, and so getting that high fidelity artifact tree picture um is the huge win for them on repeatable builds because the first question about reputable builds is why exactly do you believe? That's the same, build.

B

So uh questions discussion, um feedback.

F

Now this looks really cool, um definitely uh interested in understanding a lot more about this. I know uh there's like several projects out there that are starting to do use, merkle, trees to start to sort of either build s-bombs or better trace the understanding of of the supply chain um it. uh Actually, I guess my first question is: is there any sort of um have you written any code for this sort of thing yet, or is it mostly conceptual right now.

B

um So I I've started hacking a bit on code. I've just become deeply delusioned with the fact that goes. Elf. uh Parser is only a person, not a writer, so I'm fixing that um I've started writing a little bit of toy code and I've got several folks who are looking at various compiler implementations. I've had folks have taken a look at lvm and gcc. It appears to be eminently doable there and there's some interest in upstreaming in that direction.

E

Looking at rust and dot net c sharp as well, there's already similar work uh as far as generating the bomb in the compiler itself for net. um So there's definitely interest. We don't have. I don't have running code either right now, but all in the code for this needs to end up in a bunch of different compilers.

E

So I think the first, the first piece of code to write is just going to be an example of building these um the git bomb documents and the hashing in this format working out the api for it.

E

C

E

With some python to do that, but like the language doesn't matter.

F

Well, yeah, I don't know if um this tool might be useful, but I did post it in chat. There is uh the the folks from nix wrote a thing called patch elf, um which is a tool that they use to sort of deterministically link.

F

um You know, since, since they are trying to take the hard path of literally compiling everything from scratch and making sure that they can deterministically sort of link, everything to just their specific shared objects and libraries, um but it could prove to be uh interesting, so I just figured I'd throw that out there. I I know very little about it um outside of uh that they use it.

B

Yeah, so that that that's actually fascinating because it sort of gives you um it it gives it gives a starting point to sort of because you you drew it into.

B

I know much more about elves than I intended to know um because at some point I had somebody who was mumbling darkly about sizes of things, so I went through and actually figured out the size count uh on what it would take to insert into elf binaries. um So yeah.

E

uh Sanjay and steve in oci yesterday about getting this added or how can be supported in container images and registries seems pretty easy, especially with with his work on uh orca, sorry or how.

D

Are you, how are you folks thinking about it in relation to the reference architecture? Are you looking to incorporate it once there's a code for it available as something to reference something you'd like to add on to the next edition downline.

A

If you put as an annotation directly in the container image you built, I don't think you need to worry about all that other stuff. It's just directly in that image. You're working with.

E

B

Yeah you, the the the the real goal here, is essentially to get to a point of thoughtless ubiquity, um because engineers are lazy, it's what we pay them for.

A

Yeah, the only comment I had on the annotation was just just to add a namespace in there like an io.cncf or something like that that that way, you don't worry about other people stepping on it, but that that part's, easy enough to me.

E

Yeah we I had a question yesterday like is there an iana? You know reference for this type for this not yet um yeah yeah.

C

Right now, where.

E

To where to home the work of building the code, I think that'll also determine determine what that domain reference. Is, it probably won't be cncf, but who knows.

A

You can also ping um oci and ask them if they want to add a name under their namespace and they would document it and that that point you're now verified so multiple ways to go down that path.

E

B

Yeah- and this is actually super helpful, because the thing is um one of the nice things about talking to people at this particular stage of the process. Is you get a lot of good feedback that you can incorporate into things going forward um because we do wind up with better stuff together than a part?

B

The other thing that's going to be super interesting here is that how this gets embedded into a particular artifact is going to be a little bit bespoke to the kind of artifact you're talking about right. I'm not going to stick an elf section into a container image because it doesn't have l sections um and the what constitutes an artifactory is also going to be. A little bit spoke to languages.

B

You know, python is not going to have got h files, for example, um but part of the goal here was to have a single single way of doing this. That is so freaking simple that everyone can use it um productively, sort of back to the unix model. When I commented at the beginning, you know one thing that one thing does something super super well simply and links with other things. That's the whole goal here: it's ambitiously unambitious. What we're doing.

D

Geneva, how are you thinking about this in relation to the reference architecture.

E

Reference architecture this group has been working on is trying to well you've discussed a lot of different things in in those meetings, including how do you track work down the supply chain?

E

This seems to answer a chunk of that in a really concise way, in terms of how how to write about it in a reference architecture document not sure yet. um But I'd love to I'd love help getting this added to the build tools and added to the paper talking about how it can be used in uh whether it's you know container image, registry or container build chain.

C

D

Or that is something to be built in order to complement that right, even like it's not generally available today, but kind of like direct people like you should strive for building like this reference tree or get bomb.

E

I mean ideally, what I'm looking for is someone to folks in this call who are interested enough in this approach to help add it to the build tools.

A

So I've got two questions coming off of this um one is we talked about having the linker throw this in, like the elf header stuff, like that, are we in any way verifying that, or are we just assuming that we're now trusting the linker to do the right thing.

E

So you don't trust your compiler, your linker.

B

Yeah, I I think.

E

That question was resolved in 1984.

B

That you should trust your compiler uh for those of you who, I think it was pike who wrote the paper about that.

A

um Reflections on trusting trust.

E

Exactly so, there's a actually a counter to that. um I link it in the hackmd that you all have access to um diverse double compiling the phd thesis from david wheeler.

E

It's a demonstration of how to prove a binary is trustable, even if you don't trust the compiler that compiled it. There's no discussion around that topic, but I think, for the sake of this work, let's assume the compiler is trustworthy and handle that separately.

A

Yeah as not so much, I think that we're going to get a malicious compiler breaks everything, but more that if the compiler is fed, maybe bad data or somebody just influence a broken version of this or somebody just reruns the same build with stale data every single time. It forgets to update this one field: there's no hard link in.

E

There that requires is there, so so this is not s-bomb data in, in the sense of all of that, metadata of you know the file path or who's doing the build. What system it's on? That is the kind of thing that's often captured in a more robust s-bomb. That's not captured in a git bomb. Git bomb is just the artifacts that go in and come out.

E

So there is nothing that can be stale about this, like if you've got an old header file uh or an olds.so that you're linking in well, your binary is going to have that in it. It's it's not stale metadata. It's a stale, build.

B

Yeah, the the point about steel metadata actually gets to um some one of the reasons why separating metadata from the artifact tree is a really important thing, because you could issue a corrected s-bomb that points to the same artifactory and say: oh, wait. Crap, we forgot to update this metadata um without invalidating your artifact tree at all. Right, you can say: oh wait.

B

We we forgot to update uh some, you know licensing information or some vulnerability, information or some other piece of metadata about it, because the thing that we're actually going through and validating in git bomb is the artifact tree. The metadata is declarations about the artifacts and the relationships in the tree and it's entirely possible to not only update those without invalidating the tree.

B

It's also possible that different vendors may give you slightly different metadata, say, for example, in the case of the contacts example, I talked about where, if it just so happens that you know hp and red hat, are both shipping you, the same binary but they've got different contact information for who to reach out to if you've got a problem with it.

B

That's perfectly valid because the artifact trees are the same if the byte array is the same um and the metadata can be different if it's coming from different suppliers, and this allows us to reason about certain kinds of things that are crucially important to separate.

B

um There are two kinds of data that I've seen people co-mingle on this one is commingling um the identity of an artifact, in other words, what it is um with its location, in other words where it is um because if I have the same artifact in two different places, it's still the same artifact, the other one that I see, people co-mingle is commingling an artifact identity with the artifact's providence, meaning if I got it from hp versus getting it from red hat.

B

Those are different providences, but providence is metadata about a thing right, which you identify as a thing, and it actually becomes super important and to be able to distinguish that. I don't know if that helped at all. I may be getting too metaphysical.

A

No, it goes a little bit into my second question and I do want to be careful. I'm not. I don't want to call anybody's baby ugly, so I'm not doing that at all. It's oh.

E

C

I'm just looking.

A

Yeah I'm just looking at where the borders are and where we need to potentially add security or where this covers us, and so I'm just kind of drawing those lines. So the hash we're going to get out of this. It's just a single hash. It's not the whole tree itself, and so you need that pointer. Going back to you know some gif on database of some kind out there that says. Okay, this hash you've got represents this entire tree of data.

A

How do you envision distributing that? You know kind of going to exactly what you're saying right there is that.

C

It's not really.

B

That that's so part of it is that we're going to discover better practice as we go along, so you don't want to make too much into the initial notion right. um So when I'm talking about distributing git bombs, I expect to see best practices emerge, but I don't think that when specifying git bomb you want to be pedantic about what that should be. So take this slide as a suggestion more than anything, but because the things you're talking about are entirely um because the things you're talking about are basically entirely identifiable, but they get refs.

B

You could literally just put up a static sub domain uh if you're a vendor where people can go grab this information, you could also see people scraping those and putting together more centralized databases for you to go run against you could easily imagine um you know somebody like github uh collecting data around things like this, for their releases, for example, and publishing that as part of your release, um all this stuff becomes doable.

B

um We're sort of in an interesting place regardless with regard to this, which is a whole lot of stuff, is mechanically possible. We're going to have a whole lot of fun figuring out. What's wise.

A

Yeah, I guess where I'm going with that, and maybe it's just that there needs to be that single source out there, like a github. That's doing this, for us is that if I've got a single binary, it may have come from hp. Do I know to go to the hp, git bomb server, to look this up versus someone else? Maybe they repackage the binary from someone else, and so do. I know which server to go ping, that on.

B

Yeah, no, absolutely and there's some interesting stuff in there. um Git bomb itself is not explicitly trying to solve the distribution problem. We're trying to make solving it easy, if that makes sense by having bite-sized chunks that better in standard formats and like I said because your question is in fact a really good question. um I sort of hum a few bars on the slide about distributing git bombs, sort of as a solution exists humming, um but I'm by no means saying that this particular distribution solution is the optimal one.

B

I think will be interesting to see what sort of settles out with it. um The other thing is the git bombs will tend to be very small and light because they're a very fixed structure of a very small size. So I also don't expect them to be super expensive in terms of carrying that stuff around.

B

um Oh, and by the way, do your comment about calling people's babies baby ugly, hey man, we're dealing with security here right, all you're, really pointing out is that maybe you shouldn't be dangling the baby out the window.

A

Definitely agree on that.

F

Yeah, I think, there's another thing that I think has been brought up a couple of times in these discussions, which is um when it comes to a lot of these things.

F

You know, there's inevitably going to be areas where you know, um if you're talking about open source right, it's going to be easy to audit, like the build of open source, can be easy to audit that yes you're generating valid sorts of things, but I think some folks have kind of also brought up that there is the potential for saying hey if some of these things start to get encoded in um I'm trying to think of of a good term, because I know standard is overloaded and yeah yeah but, like I don't want to it's a simple.

C

F

Yes, yes, but, but I'm saying, like you can potentially sort of um require vendors to uh have audits that somebody, a third party, is saying yep. We believe your build system is generating reasonable s bombs here that we are signing off on that and then um be able to. It would be useful to better uh to understand how, like some of those out of band attestations could also be potentially um coming in there. Where somebody third party has come in and said, I am certifying that vendor x is doing the right thing.

E

So having vendors provide attestations, they have inspected something they've audited, something they approve of something yes, absolutely as gets at the beginning. Following the unix model, git bomb is trying to do one thing and do it well and not solve that problem. There are other proposals for how to distribute, attestations and audits that could usually reference a git bomb and say: yes, we've audited this git bomb at this time by this vendor. On this platform it is trustable- and here is the window of time for which this um certificate is valid.

E

All of that can happen and connect to this reference to this, but is not part of this.

B

Yeah, it makes.

E

B

I mean it's interesting because there's a lot of creative things, you can do around the skeleton because again get bombs not doing very much. It's sort of um doing one simple thing and your idea actually is close to one that I that it occurred to me, which is how many folks have actually been involved with acquiring a software-based company like the due diligence process.

B

So one of the things that almost always happens is you go get a third party auditor um who to whom the the company you're looking to acquire sends their code for all kinds of hygienic examination.

B

Now you don't get it as the potential acquirer, but the third party that's going to do the hygienic examination say for things like open source contamination. They get the code and there's there's. I think one dominant vendor in this space that I'm aware of there are probably others, but you could actually accentuate that process by saying look: okay, you've been you know you, the company, that I'm trying looking to acquire. You keep publishing executables and they have git bomb reps, so we're going to give those git bomb refs to the source code.

B

Auditor and now we can actually check to see if you're, actually giving the source code auditor the code that you really built into um the executables you've been shipping.

C

Can you can you repeat that sorry, I think I didn't quite get it if you can just go over it once more.

B

Oh so typically you'll take if I'm a company x that is looking to be acquired by big corp right and I will send my big corp will say: I want to make sure I'm not buying something. That's so gpl contaminated. I can't use it right, so big corp will turn to auditor x, will send their code to auditor, and auditor will go through the code and produce some kind of report about open source usage and possible contamination.

B

This is because you know little company x doesn't want to simply give their code to big core they're legitimate concerns with that. um If this becomes common practice, little company x has been selling their product that has get on refs in it, at which point the auditors can look at the get bomb rest in the executable that were being shipped historically by company x and say: okay, the source code you gave us doesn't match those get refs those git bomb trees.

B

So that's a problem and it makes the source code auditing process more reliable, potentially.

B

I was just sort of riffing on.

E

The previous idea for vendors of open source, pro or vendors, who are taking open source and wrapping it in a service agreement, makes a downstream consumer of that uh have better tools to be able to audit what they're receiving and say. Hey, you told me, you were you, know rolling, kubernetes together in this way, but uh the hashes for the thing you sent me don't match the git bomb. For the thing you said it was.

C

B

So other questions, ideas, um other stuff.

E

I guess I'll I'll pull andre's question back a little bit. um The the content of this meeting stream is focused a lot around producing, hopefully a white paper or reference architecture for the cncf supply chain work. um How would we best integrate the what ed and I just shared today into that work stream in the timeline you all have given that we don't have running code for this right now?

E

Does someone want to you know, help knock out a poc with us in the next couple weeks, so we can fit something into that paper or just keep it all theoretical.

F

So I'd be interested to hear obviously more thoughts from our side. I think it would be right now at least cutting it short, because we are trying to get a good deal of stuff out for kubecon in october.

F

um With that said, I think some of the concepts here are definitely things we might cite in sort of the future work. That's um you know that that we think is coming out and I'd be interested with. uh You know some of the folks who are very focused on the notary and sick store stuff right, like some of that, like how do you see all of this kind of coming together and and and where do you see that synthesis um I'd be interested in hearing more thoughts from from other folks? On that.

E

And that's just a clarification. I believe uh notary v2 already supports everything we need for this.

A

Yep, it would so as soon as you put that annotation inside of your image notary would sign that image's digest, which would have that in there. So it's just one more layer of the merkle tree for you.

E

So, let's I would, I would say, let's get that in if, if at all possible, um let's get it in, at least as an example uh to say: hey here's how you could embed a um get bomb metadata into your your artifact annotations, your container annotation.

B

Yeah, so the the the final question I want to ask is um because every time I I talked to a new group of folks about git bomb and I think, davis had the same experience. We discover another set of folks that we should be talking to about gitbang.

B

So who else should we actually be talking to about kepon.

D

Well, tying it back to the paper. I think the papers can give you reach to a broader audience. We do have dedicated assignees for it for the different areas we can help you identify who those are. I don't know if it'd be prudent at this point in time for to say, hey, we're, opening up the doc to you. Try to slot this in here or wedge it in or that, as michael pointed out, it we're cutting it a little bit close.

D

I do think we can give it consideration to say, like hey, there's active work around get bomb. This is something to pay attention that would be complementary to s-box out there, but it's in early stages of development.

B

Yeah and that's fair- I mean I, I have a basic principle that I live by, um which is no unnatural acts. I try and avoid uh asking encouraging or even having people mistakingly commit unnatural acts on my behalf and your basic statement, which reads true to me sort of says: look we're not going to go, commit a natural acts in this process around git bomb, and I am a million percent behind that.

A

C

A

D

A

I was gonna change directions slightly.

D

That's okay! I was just gonna joke that. Well ed, you've known me for a number of years, and you know that I'm a very irrational being who does commit a natural acts. I I.

B

Know this very well he's approached.

C

Me brandon go go for it.

A

C

A

I wanted to make sure we got that in um you're, asking who else in the one name that came to mind for me was like the open ssf. So if you haven't already pinged them, they might be worth chatting with.

E

I am trying to get on the open, ssf tac calendar right now already spoken to ryan about, so that is next on our on our tour as it were.

A

I feel like ava, you've got the whole document that says everybody we should be talking to anyway. So just keep running through that yeah.

E

It's such a long list, though I mean people, keep adding more to it every week. I turn around there's new contributions. Thank you by the way um to those of you here, who've been adding to it, but oh yeah.

C

D

I know ed likes hit lists to make that.

A

E

Like I'd start off with 8th wheeler.

D

There's tracy miranda like all the folks that are on the program committee for supply chain security. Con would be great people to talk to that are not, and this caller part of this group, but part of related efforts and the industry.

B

So is this the one that happened at bars in austin recently that I saw stephen augustus tweeting about.

C

No, this is the upcoming official.

C

Cool um sorry, one quick request in terms it's a bit housekeeping. Almost um I was looking at the uh the no for the meeting, and one thing that's lacking. Maybe for newcomers is at the top sort of a clear indication of what the current goals of the working group um is, because you can land in the document and sort of try and catch up on all the notes, but it would just be, I think, convene to have a bit at the top, like you know, we're working on getting this document out.

C

You know here's where you can contribute or somehow that uh I was looking through it earlier and couldn't quite find something.

E

Yeah at the moment, contributing to this is talking to me or ed, or somebody else involved. We have not set up um like a separate organization for it.

C

Yeah, sorry, sorry, my point was more general to the whole working group not specifically get bomb, though it would be also interesting for git bomb. Obviously- um and I I guess, you'll reach that point in time- it seems like it's, you know quite new. Also it's really cool. So thanks a lot for presenting um but yeah. My my point was more about the the working group in general.

F

Yep good point um yeah. I think we have some of that.

D

Actually, you think you could give you could take a stab at that.

C

Yeah, okay cool! If.

D

You could write like something really short and sweet that that'd be awesome.

C

We'll do sorry.

D

Michael didn't mean to cut you off.

F

Oh yeah, no, no problem yeah, I was just gonna literally say the same thing yeah we at some point. I know we did have something in there. It might have gotten lost in like gotten pushed further down the page where it should be. Definitely at the top.

C

All right, no as well I'll, have a I'll, try and looks better through the document and if I can't find any, if I can't find it I'll just uh you know ping some of you but yeah no worries I'll. Do it then.

C

D

You yeah good catch.

D

We have five minutes left. um Do we want to wrap with git bumps and talk a little bit about administrative things and timelines? How we're tracking.

D

You have if you could share the the link to the slides I'll post a recording later today and I'll make sure to include the slides there and we'll promote it to our channels. Thank you very much great presentation. So michael yeah, as you were, saying, we're trying to target kubecon for a published draft open for comments.

D

How how how do you feel retracting there is it? Is it feasible, how's everyone feeling they're doing.

F

I definitely have thoughts, but I want to kind of let other folks talk about it, but I definitely have thoughts around what areas or we have gaps in.

F

Anyone have any thoughts as well.

D

Axel brandon priya.

A

uh Just looking back the document, knowing that I have some updates in there too, so um I'll get it as fast as I can for my part, but yeah don't wait up for me. I guess it's. The key detail.

A

Yeah, I also updated a few couple of sections.

C

In the document, I think it's we probably need to have internal review to basically put them together and filter them.

F

But from my end, um just to kind of highlight.

D

F

F

C

Oh yeah, I was just I mean my only update is that I did, I think, a little bit of editing in the document, but um nothing too major. I think I still have a couple action items there that I need to finish addressing, so I can work on that this week.

F

Cool um from from my end, yeah, I think things are- are really starting to come together. um The only part- and this is uh also, I think, the weakest part from my perspective just like from sorry, my understanding is just around- um I think some of that sort of like how do we distribute identities and yaya, and how does that all sort of look inside this um inside the big picture?

F

um I think that's the the only thing that I think is like maybe I think, still a little um complicated, because I I I might not just be following it, but um that that sort of thing is, I think, one of the things there, but I I also think that there's been a couple of updates, since I last was able to check.

F

Anybody have any thoughts around any sort of big gaps that they think that, like are missing here that like, if we were to say, hey, you know other than outside, of like filling in some details. Is there anything that they think we can't possibly have be securing a supply chain? If we're not doing this thing or if we're not talking about this thing, is there anything that folks think is still sort of like a huge gap.

F

Hoping silence is a good sign there um it. If that's the case, uh do we think that in general the scope is now pretty decent um that we haven't kind of like you know, um because, because assuming all those things, I think one of the other things that we probably want to start working on is just talking a little bit about what the reference architecture actually gives you right like assuming you follow this reference architecture, here's the sorts of things that you could expect us minimum.

F

You know that uh that that helps lowering that these are the sorts of risks, that the reference architecture will help lower these the sorts of attacks that this helps mitigate.

D

You're deliberately not using the word guarantees.

F

Well, I I think that there are definitely things that, like at the end of the day, I don't want to say guarantee, because when it comes to any of these things, right, like I think, as we sort of talked about right, if, if, if somewhere down the line, even if you did all these things and it turned out- uh you know somebody compromised intel chips, yeah you're, probably still uh you know, you can't necessarily uh trust that. So I think the thing here is like you know, one is temper.

F

Expectations like we might be able to say this is a guarantee within this sort of general framework, but I don't think we can. Specifically. You know, uh I think, when it comes to you know most security things very few people are guaranteeing stuff they're, saying we're minimizing these sorts of risks. Where you know that kind of thing.

D

F

C

Going straight further.

B

I don't know I.

C

Literally, don't believe it would be a good idea too.

C

D

What would be a good idea to thread model this yeah? That's fine like let me let me switch off my bluetooth here, because I think that's causing a lot of latency and jitter sounds like we're arriving at a place where we have a rough shape of the architecture.

D

It would be like sensible to thread model it to come out with like hey here. Are the here are the possible vectors and, like here's, the probability of like risk.

D

Now, talking about unnatural things that, like the way the way you asked us to contribute to git bomb, if you and eva have cycles, we're happy to share this to like help us review it and help us identify anything outstanding and like for you to poke calls and and the architecture itself more than the writing actually.

D

Would that be something you'd be interested to take on ed or eva.

B

I apologize, I missed the ask.

E

So did I read until you said that so.

D

So we we're like at this point where well, we think we're we're covered on like the minimum viable architecture for a secure supply chain. At least we think so we want to test that assumption, so we want to float a very early rough draft that we have for you to poke holes at, and the other thing we want to do next is like well, we want to talk about the security of this architecture as a whole, which is what michael was talking about like talk about talk about risk.

D

So if you guys happen to have time and like want to pitch in and be part of, this effort, like, I think those are two areas where, where we could use help.

B

Yeah, well, I absolutely cannot guarantee time. I have. I definitely have interest, and I also have history um because I was building s bombs in 07 at industrial scale, um in order to be able to track a number of considerations, and I was involved in the very early days of speediac. So I understand a lot of the history of that stuff and I also understand really really well how these things actually break down and the places where they're actually real and where the theater and there's a lot more theater than anyone needs to talk about.

D

Yeah we we want to try to stay away from theatrics for as much as we can.

B

Yeah I mean I'm dramatic, but but usually not about security.

E

I am I'm happy to review stuff. I think this is all well in my time is the challenge, but not interest or domain work. So yeah add me: I will I'll do it again.

E

When you add me, is it okay for me to then share to some of the other folks in microsoft, who are closer to the cloud native projects, whereas my view is a little bit more broad on open source supply chains.

D

I think so, but I'll defer to michael, like how.

C

D

How focused versus like hey, open it up, it's gonna, be more feedback for more people. How much are we willing to take given the the short.

E

Runway, I can also, I can also funnel it through me right. I can just once you share it with me. I can share it with them, collect feedback and funnel it back if that works or not at all, if you prefer.

D

Yeah yeah, I think I think that would work the best. What do you think mike.

F

Yeah, I was thinking maybe starting off funneling through uh you, ava, um where it's like. If there are specific people who are like oh yeah, I have a ton on that. Then we can add them in, but I think to start off like not sort of send it out a huge blast. Unless there's like specific people who who have a lot to sort of contribute, no.

E

Not planning on doing a blast.

C

D

One one other thing on just like housekeeping of the dock: we're trying to keep comments to a minimum uh suggestions are preferable.

D

Okay, if you have, if you have trouble with something the way it is expressed or you think it's like there's not like high fidelity or like technicality to it and you you can take a step at like expressing that better or more accurately uh just go for it so kind of like the question or the commentary doesn't arise in first place.

E

Okay, so suggestions preferred to comments. Is this the only meeting discussing the doc and is it's the right slack channel uh to you know, go back and forth in questions on the dock? If I or ed have any.

F

There is a so specifically for that we we can introduce, we can invite you to the the um the ref architect arc writers, room.

E

F

Yeah, so that's like, like very specifically about you, know different. You know comments about like hey. Does this make sense in the doc yeah yeah um general supply chain, stuff or keeping in the other room? So I can invite you both.

E

Cool. Thank you.

E

Okay, I really should drop for another meeting. Thank you all for having us.

D

For a doctor's appointment, yeah thanks axel.

C

Take care everybody bye-bye yep later bye.

C