CHAOSS Risk Working Group, 13 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CHAOSS Risk Working Group April 13 2023

Description

Minutes from this meeting can be found here: https://docs.google.com/document/d/1iqIMpLBwuKSnE0BbQTgbsb9Im87IoN7IUzukochClCw/edit?pli=1#

A

Okay, um this is the group meeting for April 13th 23 2023, not 2013. I'm a decade behind and um yeah uh go ahead and start with the the announcements of.

B

You yeah, no, it is it is that uh it's that we finally have a desktop Dev API, um so I'm I was pretty happy to see this launch, because this was like one of the first things that I heard is feedback from the community, but like it's a really cool UI. But like can you just give me an API um and I passed along comments from this group from Dwayne from folks that I spoke with and they did it.

B

So that always makes me happy when clearly they listened um and offer that up to the community, so um I'm excited to see how people use it and see where see how helpful it is, and if hopefully it means that there are others that are going through and trying to aggregate this information on their own because they don't need to anymore. It's already. There.

A

Yeah so like I I actually went through, and this is part of the note that I posted in the risk, Channel and slack and um yeah I was pretty pretty pumped I could say like it.

A

um One of the things it does is. It shows the CD Easter packages and it Crea there's a clear navigation from the package manager to the GitHub repository or other repository, which hasn't really existed anywhere in a clean and easy to use way before so like, and it runs. Os open, ssf scorecard um as well like I, was pretty excited to see that like. If you hit like um this, you you can see like it's.

A

The scorecard data for react, which is awesome, um I, don't know how often it's run but like it's one thing, auger does right now is we run the scorecard now we could just call this API instead um I also like it's enumeration of the dependencies in a project and the versions. That's also a very we all are also manually scans for that now and doesn't need to anymore, and you can get really detailed information on specific package versions, um whether it's the default I think it was. Was it this one or this one I.

B

Guess my only question to you, Sean in terms of using this API versus what you've already built at auger is I. Do know that this data set is limited to a specific set of package managers. So.

C

If you, if yours.

B

Is more flexible and that it would Encompass things beyond that fixed set of information, because I think that would be the only concern I would have and that, because this is available it's great to have something. But it's still not technically everything.

B

A

B

I think it's impossible to actually have everything, because everything is growing and there's probably some mathematical equation that talks about how much we can Aggregate and in constantly growing set of information. I, don't know, David I feel, like you might know such things, um but I feel like that's impossible, but it just I, don't know well.

A

It does seem like that these advisory, one thing I want to point out- is these advisory keys. So if there's a cve open against a project, you get that information as well. um In.

C

Case in cases where you actually can tell that they match that's one of the problems. Challenges of the cves is that you're gonna have a cve but not be able to connect what program it relates to. Oh.

A

And and that uh that's important.

C

That is important. um It is a it's a well-known problem with the cve I mean to be fair. Historically, the cves, the Assumption was that humans would read them and everyone, because there's only a few dozen a year, um yeah that that failed, um and so there are options for recording information for mechanical mapping.

C

But um you know a lot of people, don't use them. uh Some people use something called a CPE identifier, but most projects don't have a CPE identifier, so uh open ssf has a project called osv which actually does tries to connect between vulnerability reports like cves and act specific projects so that you can mechanically map them.

A

Well, what what I was just very timely and interesting to me is, is that uh Matt and I are actually having a conversation about libraries.io yep um recently and like why people do or don't use it, and uh he turned me on, and so I did a little bit of looking around and he I think you've mentioned this before uh David. That's the nvd database.

C

Marshall vulnerability database yeah.

A

And so the nvd database also has an API. Yes,.

C

We're about to change it! Well, actually, they've already changed it they're about to deprecate the old one.

A

Okay, yeah, and so it's like I, was actually playing with this um yeah API yesterday, but I, don't I didn't actually open up.

A

C

Right I mean lots of people, look at the nvd and you know: um hey I'm glad it's there. um My concern is with the nvd has always been this. You know you know, please. We need clear, mechanical mapping from a vulnerability report to the software. That's vulnerable uh every time. If it's not there, I don't care, please throw it away.

C

Tell me when you're ready to to be to tell me about a vulnerability there's something in there is a vulnerability, but I don't know where it is, uh doesn't make any sense with uh you know when in the scale of today's world, so it's and the sad thing is that they're doing almost all the work except this little piece, it's it's you're. So.

A

Close well and that's that you know the piece that was missing from libraries.iour, which was incredibly cumbersome to get to was that relationship between the specific package and the directory that that package exists in in a repository, because many repositories contain multiple different distributed packages, and so what the depths.dev1 does is it actually um uh gives you that I think.

C

Yeah, what we want to do is the other way. Here's a package where is its source Repository, because a single Source uh repo can show up in many many different package repositories, but you should have a one-way link from a given package given reap a given package report. Repository back to a specific Source Repository.

A

Yeah so yeah, that's not in the nvd database, is what you're saying.

C

I'm, sorry, well, okay! Well, it's actually broader than that. uh If you get a cve, what you get is a text description of the program. That's vulnerable. You know. Now what you might get is a CPE ID.

C

Now a CPE ID is unique, and that is a useful identifier in the case, for something has a CPE, but cpes were created back when well, you know how many cpes could there be there'd, be one for Microsoft, Office there'd be one for you know, I, don't know Word Perfect or whatever is an important program at the time, okay, but the question was that you were only going to track a few hundred, so maybe a thousand, but just this what's happened is with many things is scale yo.

C

As soon as you say, wait a minute open source software programs can have and libraries can have vulnerabilities and there's literally millions of them. uh Cpe doesn't scale. Smooths are hashed to particular version, so they don't work either.

C

um A big fan of package IDs for this purpose, they're, not perfect. They got problems but they're. The closest we've got because then they can say things like in Maven. This specific name has that vulnerability.

C

ah Now we're talking. Okay,.

A

C

Maven this particular package name, this particular version uh they are working on version ranges that the big weakness of um package ID right now, but at least they've got a way to uniquely identify a package, a compiled package and not just a source repo, but just be able to do.

A

Yeah yeah I, so I haven't navig, so I haven't gone to depths.em to see if these advisory keys are related to the cpes or the ceids in any way and I don't know so.

C

You'd have to hunt I mean very, very few open source software projects have cpes, uh openssl does I know um some big ones. Do.

A

That's probably not in dubstep Dev, because open SS, what's openssl package managed by probably a bunch of them.

C

Well, right, I mean pretty much every system. uh You know you know so you what you're going to have is fedora's repos and rel's repos and debian's repos and ubuntu's repos, so yeah, so open SSL is going to be packaged in a lot of different repositories. It's true for most system software, actually.

A

Okay yeah, so they get to open SSL for and through npm.

C

No they're probably going to get oh okay. This is the npm module openssl, which is not the open, SSL. Most people. Think of you. Look at the version. Number yeah, okay, zero! One! Zero is not the open SSL, you probably mean is.

A

It two they have all the versions listed as two points.

C

I don't know these are careful. These are the npm module and open ssls. That's not the open SSL. Most people talk about interesting.

A

C

A

Can be confusing.

C

Well, that's why you need something like package ID, because package ID does is a package URLs, I'm, sorry, I didn't say: package IDs by Min package, URLs Now package URLs are as their URLs that refer uniquely to a particular package, and so you can distinguish, do you mean the npm one? Do you mean the Fedora one? What are we talking about.

A

C

Ssl that you're familiar with is written in C. It's not going to be all by itself in npm right, that's a JavaScript now I would. The odds are excellent, that the npm module called openssl is a shim that then calls the actual open SSL and that matters, because it's quite possible for a say, a JavaScript interface to have a vulnerability and not the underlying program. Yeah, okay, and so you need to be able to distinguish between those cases.

A

C

uh So much fun.

A

Yeah and the vulnerabilities database, though it exists and has a base URL you kind of have to there doesn't seem to like you have to know the list of cpes. You have to know right. Some piece of information.

C

um Right, you have to know what the CPE on is before you can search for it I mean that that stands to reason. Yeah.

C

We do ask for CPE IDs for the officer for cpes the best practices badge if they have it very. Very few people fill that in foreign.

A

Okay, so I have uh when I just asked for low vulnerabilities and limited my search more I got I got some descriptions.

A

But I don't even know how I would navigate this to map it, as you said, to projects so.

A

Use your S Pen mail.

A

I just I, don't know, I, don't even know how to references. Maybe there's a so there's a cert advisory reference. There's a right so.

C

I I, but there's no reference to.

A

Where the software lives.

C

Right right well, but but the thing is remember this was designed originally for all software, proprietary or or you know, open source or closed source. So most closed, Source folks, they're not going to release their code so that didn't make that wouldn't have made any sense to them. They want to know what program that you know. What is the executable that has the vulnerability we, we would call a package.

C

A lot of them aren't downloadable in that sense, okay, so their solution historically has been cpes and the problem is that they have not scaled well.

C

Well, yeah you're, not talking about depths.dev you're talking about the cpes of.

A

Yeah yeah and under a different uh having.

C

Well- and you know, I would say it's not that the nvd doesn't scale: um okay, okay, you know I, would say yeah nvd of some use it. It reports, vulnerabilities, uh known vulnerabilities.

C

However, it doesn't have a good way to match to map. I would say a good automated way up automated way to map vulnerabilities.

C

To packages or repo or or Sports repos yeah, okay they've, they use cpes to map.

C

But cpes are not scaling. Well, that's.

A

The problem there all right and what is the, what does that mean that cpes are not scaling well.

C

Basically, you have to, they have to be hand assigned.

C

Yeah, um the naming Authority for cpes I think uh there used to be somebody else, um but uh you know so imagine that congratulations, here's the job every time you've got a vulnerability. You need to assign a CPE to whatever it is: that's vulnerable, yeah, there's several million, oh, and by the way you have no money.

C

So uh what I tell people over and over again um it's you know, because I used to work with the government is you have to remember. Nist is not a money. Source, yes, does a money sink. Someone has to pay them to do stuff. That's not necessarily a bad thing, and it has some good people and they've got some good support people.

C

But when you say hey, this should do X, you're, probably right that they're gonna immediately turn around and say great where's the money, and usually the answer is well. Won't you take that out of hide and they'll say: I have no hard life yeah.

A

Okay, well, what's what's really interesting about, like both of these things, depths.gov to me is even though it's not complete, or um it does cover some very large package managers and gives a lot of useful information. It seems to imply that it will provide vulnerability, information um in some places, but it's it's not I haven't looked at it close enough to know. If that's clear, um this nvd database seems hard to connect directly to a piece of software and, as you mentioned Dave, it doesn't.

B

A

There's a lot of Open Source software. Actually that has cpes um Sophia. uh What I guess I'll I'm just curious if to expand the available data in the depths.gov API? Might there be some I guess I might ask someday? If the code for creating that database would be open sourced, then it could be expanded.

A

Beyond, its current limitation, like if Dev, doesn't want to do everything itself. I bet there'd, be somebody who'd, be willing to.

B

Yep yeah we've we've brought that up with them. Yeah.

C

B

That was the second most interesting thing to bring up from when I talk to folks about it, mostly because it's kind of listed as an open source support tool, but not actually open sourced, um in the sense that it is a closed model. It's not a collaborative model, and so I think that is it's worth raising again. I've brought it up before um I'm, just kind of curious to hear. If that's, if that has any place in the road map,.

A

B

A

Think the thing that they've done that is um really powerful is um getting to these um I think it's this one.

A

A

I think it's maybe this one.

A

Geez I'm I'm sucking wind, guessing which oh.

C

By the way, yeah I I, I, I, I I- don't want. My goal is not to derail anything but specific to that problem. It's a well-known big problem. uh I'm many other people actually signed a petition and I just included a petition in the notes. Here.

A

The which petition is that this.

C

Is petition to say: hey, cpes aren't working, please add pearls to every software. Every cve that reports a software vulnerability, a CVS are used for Hardware.

C

Also so you you know, and we actually talk about what to do in those cases too, but they it's a signed petition by folks, analytics foundation and owasp and lots and lots of other folks, okay, so they're aware so they're aware of the problem, you know, I mean we've got, you know spdx and red uh and uh various you know ION channel and Oracle, and you know oasp and Linux foundation and the Mayo Clinic among the gram attacked a whole bunch of people.

C

Saying hey, you gotta fix this, but uh uh they haven't to my novels. They haven't really fixed it. Yet they're hoping to change the metadata format to allow easier inclusion, but they're not saying we're going to require the information everybody needs in which cases I still don't care.

C

Oh look: I use a database that can't be used yeah because I'm frustrated when you're so close a small process change could really make a difference. Yeah.

A

One thing I'm just noticing is like for a package like numpy. You have to make sure you're using the right dubs.gov directory, because it's by package, man there's a package manager path.

C

A

That's interesting, and so this.

C

By the way in general, naming is hard, but in particular identifiers are hard.

A

Okay, so I clicked on the PDF link, because that, though, that's not the one, that's the um call to action. Is it this one.

C

Whoops I, just um it's.

A

A p it's a PDF yeah.

C

That's it. Yes, that's the that's the that's the petition. You look on the bottom. You'll see a whole bunch of names. That's how you can know it's a petition. Okay,.

A

C

Scroll to the bottom.

A

If I see and I see the.

B

Comments on that, or is it.

C

Kind of no that's not it's done, it is what it is. Yeah I mean you you could I mean you can talk to nist and other folks. I mean at any time they're not dead, but that particular petition. You know it was written making. Specifically.

C

Basically the problem is we can't map anything. We've got this big vulnerability database that can't be used because it can't be automated, and everybody agrees that naming is hard uh and we propose very you know. Yes, we identified the we admit the problem and we specifically recommend pearls and we have some suggestions and for Hardware we suggested G tins and gmns. That probably doesn't matter to you, but basically you know instead of noodling on the problem forever and having nothing useful.

C

You know please take these Solutions they're, not perfect, but they're, good, partial Solutions, and it's not clear to me that you're ever going to find a better solution so take what's it take what exists? Please.

A

Yeah it you know, I, don't know how I had some other things that Sophia and I had talked about talking about here, but I.

A

It doesn't I mean it's classic that one part of the government isn't talking to another part of the government, because if we want to secure the software supply chain darn, this seems like a very useful way to concretely work in that direction. Yeah.

C

That's what we thought too.

B

Well, I'm just kind of curious in terms of like. If the petition is closed, then, if there's something I feel like they're, potentially more folks, that would agree and either promote and or continue the conversation. Is there any Avenue for that, or is this just sort of? If we know the people that are interested in changing this, we know where to find them kind of thing.

C

I, don't if you can think of another way to cause action. Please do it. You don't need my permission uh that this this particular petition I mean our cosigner of it. uh This particular petition was born out of a frustration that we keep telling the gov U.S government, specifically the US government, the US government that you're doing a lot of stuff. It's almost useful, if only you would do this one thing, it would be useful, but you won't do the thing that makes it useful plea. Okay and part of the argument. Was it's really hard?

C

Okay, that's why they create the CPE process. If you identify things all right, I got that it's hard. This is the best available solution. It's not perfect use it anyway, and so you have a whole bunch of people. They know us wrote this letter, you signed it, you send it in they're, aware of it. Nothing's changed and in the end only government employees get to make these the decision of what the government does.

C

So the government has not chosen to act on this. Currently I am frustrated. Perhaps you can tell yeah it's coming through.

A

B

I have thoughts David, but I I got a recorded call.

C

Okay, okay, I mean.

A

C

A

Recording yeah.

C

No, no! It's! Okay, because I'm starting.

A

It all right, I started recording again so yeah, so I'm I'm thinking with you know, Sophia and I had discussed actually building a risk, metrics model and what it would be.

A

um What would be the first one and uh then the second thing is: are there some things that open ssf? What is what might it be the right way to collaborate with openssf to um enable some of the things that they're doing to also be formalized as metrics or work that they do could be referred to and constructed within a metric and then somehow connected to the tool set that we're using so, for example, okay,.

C

So so I can quickly respond because I'm deeply involved in all those things. I mean I'm at almost every openness up meeting I'm, certainly at every dashboard meeting, almost every scorecards meeting I lead the best practice badge so I'm, I, uh yeah I, know you're super involved, I I'm all involved in this stuff and I'm here too.

C

So uh so I have opinions about this, and that doesn't mean you have to agree with me, but uh let me try at least how at least I've been operating um the um obviously the scorecard has tried to identify some very simple, simple uh heuristics. They scale from zero to ten and measure that okay, the dashboard is a broader effort to take metrics of different kinds and present them to users to help them figure out. What's the risk level things like hey, do you do you have recent contributions?

C

That's a hard thing to scale from zero to ten. But if your broader question isn't just hey, I want to look at this. This project, real, quick but I'm, trying to make a decision about whether or not I should use it.

C

uh The the long-term goal is that dashboard will be the uh the point where that data is tucked in where it brings in store cars and the crux's badge and lots of other data to help people make decisions. Okay. Now, how does that relate to this group and Chaos in general? Okay, uh my view is that chaos in general, especially this risk working group, is basically does deep Dives to help identify some potentially useful metric. Okay, you you, our drill in try to figure out exactly how to measure it. Where are the problems with measurements?

C

How can we try to counter some of those problems? No matter what you do, any metric can be gained um and there's always the risk of in particular, if you only have one metric of of of driving the project to the metric instead of what's actually important, but you can make better and worse metrics, and so the goal at least hope is chaos. This group, in particular, has a better metric. The dashboard app folks are going to grab whatever metrics they can find uh absolutely I've told them.

C

I I've pointed them cast a number of times, I believe they are planning to reuse. Some cast metrics um uh I, don't right now they aren't using some of the tools like auger and so on, but I think they are thinking about it uh right now, they're just trying to come up with some basic evolutionary prototype that they can use, um and basically what they're planning to do is we're gonna, we're gonna, Define a whole bunch of metrics, implement it. Try it out, repeat, repeat many times: I.

A

Said I feel, like you've, showed us an early version of that English word yes before, yes,.

C

I have um so they basically they're the the dashboard there's a dashboard project, I guess uh where basically they're taking that as their first step and they're, uh the plan is they're going to start for various reasons. They want to start over Okay and so they're re-implementing a whole new uh dashboard, but they're going to use the current prototype as basically the first version they had was kind of a throwaway prototype. Okay, we learned some stuff. Now, let's make an evolutionary prototype and so on. Oh yeah. If you go to metrics.open ssf.org.

C

Hopefully it's still up org yeah, well, okay, I think they're. uh Their certificate expired, yep, yeah go ahead, accept the risk.

A

I'm trying to sometimes my browser doesn't let me do.

C

There's a lot you.

A

Do that, okay, that are insecure, it is spinning, oh I, think it's I think it's allowing me to go, but the little X is not moving.

A

Yeah I think that there is, there might be a sometimes uh at the server level they block HTTP requests, so it could even be their server. Oh there we.

C

Go there, we go yeah, um yeah, I, think I think it it goes. It I think it shuts down after being up for a while.

C

There's that it says https interesting, oh yeah, it's just an old cert. That's the problem! Yeah! We need to turn well yeah. The problem is we're not really being serious about updating this, because it's about to go away yeah, so this version was basically a graffada instance, all the work's actually being done by okay.

C

So you know basically it's showing some metrics and then the goal is to make it easier and the the fundamental problem that feedback we got from this was that was kind of hard to figure out, and you know what's good with you know, I mean the green is good and red is bad. But oh.

A

Okay, yeah I, remember.

C

So they're they're trying to redo it in a cleaner, nicer way and then go from there. Yeah.

A

Yeah we have, we certainly have the our the chaos tooling has most of that available as well. So there's a way to collaborate there we are out of time and I have to teach a class in nine, eight, nine and a half minutes. So okay um take care, I, think I think next time. Hopefully we can all still be here and we can work on a metric model together and um if, if there's when is the dashboard meeting for openssf.

C

uh Okay, so hold up here. um Let's see here today is the 13th. uh The next one is the 21st at uh noon: Eastern, Time, it's: okay, okay, it's called sigrisk or screen two, so noon Friday at next Friday.

C

A

All right thanks, everybody.

B

You're welcome there.

A

All right take care and congrats.

B

Oh, it's not me, but yeah.