OpenSSF Scorecards Biweekly, 20 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Scorecards Biweekly Sync (April 20, 2023)

Description

https://github.com/ossf/scorecard#connect-with-the-scorecards-community

A

A

Good we'll give folks just another minute to join and then we'll we'll go ahead and get started. Thank you.

A

All right we're about two minutes in so why don't we go ahead and get started? uh My name is Brian Russell I am product manager on Google's, open source security, team I work with a few different projects in the open ssf, including scorecards I'll, be facilitating today uh and I just want to. Thank you all for coming.

A

uh If you haven't attended one of these meetings before there is a document attached to the meeting, invite which will link you to the notes that you're seeing on screen and has a list of who's attending, it's your first time attending. Please add your name to the attendees and, if you've been before, hope that your name's there go ahead and just make sure it's highlighted so that we can see you've attended um before we get into the main part of the agenda.

A

Is there anyone here who hasn't attended before? Could you just introduce yourself if you say what brought you to the meeting today.

B

I'm new uh Justin, Abrams and I want to talk about sbob, scorecard inclusion possibilities.

A

Fantastic welcome Justin.

A

I think I recognize everyone else, but if there's anyone else go ahead and introduce yourself right now all right. Well, let's uh go ahead and just jump into the agenda uh say if there are any topics you don't see on here that you think should be added I. Think our agenda's light enough. Today you could just jump in quick to the dock and just add a topic towards the end, but.

C

A

Ahead and jump into the first one, which is All-Star not sure exactly who had it but Jeff I'm guessing it's you yeah.

D

So just project updates here um I we had an issue updating to the latest scorecard.

D

uh It was using out more memory than than we it used to and it turns out it's um the transport we're passing in is for was mostly for authentication, but it's also for um we also have caching attached to it, and it was an old change. But scorecard is now using that transport for tarball downloads, which is good so that it can download uh tarballs for private repositories, but it also means that our cash is overflowing.

D

um So I need to figure out a way to um pass a uh an authentication into scorecard when calling it as a library, but not have it cache the tarball or maybe not have a cash at all anyways.

D

um So that's kind of the what's been going on in the project, some other um some other, not many other updates there, but uh that's where we what we've been working on lately, so just a just a an update here, not a discussion topic.

A

Oh thanks Jeff for the update, um then on the scorecard side. If there's any announcements that you think should be in here, go ahead and just you can.

C

Either add them to the.

A

Dock or, if you want to say them right now, great too otherwise, I think we can go ahead and jump into our other issues and I'm guessing Justin. This is this. Is yours.

B

That's mine, so I have a utility called s-bomb scorecard and, as the name implies it scores. S-Pops um I think that it it may make sense to include that within the broader security scorecard um I've like there's an issue about it, which is linked to the document it is, there was some interest uh and I'm here to figure out what needs to happen to push that forward.

A

Foreign I guess: do you just have a sense of of where things are at right now? Are there any significant concerns that um you know would be helpful just to discuss in this meeting.

B

um I, don't I don't have specific concerns, it's mostly a how do I get from here to the destination. um What do you want for me uh to make? Make that happen? I'm happy to contribute code or whatever, just I, need to know what what you'd like.

A

So I'll just say: we've definitely taken in contributions before uh I guess. A general rule code is appreciated.

A

um I'm wondering, though, for folks that have emerged in Things Are there specific places, you'd recommend to get started thinking their dispenser or Rock Off any recommendations just for Justin.

E

Yeah I'm checking out the issue um right now: um I I think the how something would be included uh would determine sort of where um so I'm just gonna flick. The issue real quick.

A

I mean I would say already a good start just having an issue open. Usually a lot of our discussion is just done on GitHub um I think you know generally when issues are open, we'd, look or just a general criteria of you know. Does this really help Advance, scorecards and, and you know, does it change anything fundamental about it? Based on what you've said it does it doesn't sound like uh you know it's it's a major change to scorecard itself. It's extending what it does and basically making it work better with s bomb.

B

Yep, that's right and a username Lawrence Simon uh suggested that I attend this meeting to discuss to propose the idea.

A

F

Oh go ahead, I think it was in this call. Last last time we had this call I believe it was. Somebody brought up another tool that kind of had some similar overlap here and we were gonna see if we could get people together to kind of see if there's opportunities for collaboration or if there was not significant overlap, it might be in the the meeting notice. The minutes from last meeting but um I forget who the gentleman was he'd been at a conference and kind of got, um might have been hip check.

E

You're, probably talking about David wheeler right, yes, I was yep.

A

Let's say a lot of times: David does attend this meeting, um couldn't make it today. I think that um the best way to move it forward is just um I. Think two things one for this meeting.

A

You could just maybe share a little bit more Justin about um who you anticipate using the feature and what you know kind of core problem. It solves I, think sure it's helpful just for the whole group.

B

Yes, so the there's a lot of push to generate s, fonts and I- think that's well founded, um but it turns out that the quality of s-bombs is questionable um and I have a large number of s-bombs that can kind of back that up.

B

um Some are not even spec compliant some don't list the version of the package being used. Some don't say like what hat like. What's the identifier for this piece of software like how do I know that this is what it is, um and so the intent of the s-bomb scorecard is simply to assign a grade to an s-bomb and say yeah. This does actually have the relevant pieces of information needed to have actionable insights like if there's a security, vulnerability. I would be able to tie that back to this piece of software.

B

At this version, Etc.

A

If I could just ask again, how does it, how does it kind of do that such that it can make that determination, sure.

B

um It's ingests a flat file of an s-bomb and looks at the elements of it and it will say things like 30 of your packages that you've listed here have Associated versions.

B

That's not great or you don't have any package URLs here, so we don't know how to actually Trace that back to any sort of known identifier of a repository um or we don't know how this s-bomb was made.

B

So there's a there's, a handful of different things that it looks for, but it's all based on the context provided within the s-bomb itself.

F

I would definitely, second, that that we see a lot of of issues around s bomb quality I think it would be helpful to be able to have a project that you know scores at and obviously that's useful standalone um it'll be interesting to see exactly what your vision for integrating that into the overall score is.

E

You linked um I think you said that for your two cents, uh you should put the s-bomb along the artifact and not in Source control, so I'm curious. What role or I guess how you would include it with scorecard.

B

um So today's scorecard is entirely based on what's inversion control. Is that right? It.

E

Runs on a a GitHub or soon a gitlab repository, so it's very Source control based and configuration based.

B

um So it should be able to like, if I had access to a GitHub, API I could look at the releases that have been made for a given project and see if there are s-bombs associated or you know, stations like those sorts of artifacts that are included along with it um and I think that that is still within the realm of give me your GitHub URL or your gitlab URL, and being able to infer it from that I. Don't know if that's additional scope creep for the project. That is not welcome.

E

um um So I I think based on what you just said. It definitely fits better with sort of the release check which we have a check for, um and not necessarily the binary artifact stage, which is the issue that was linked in the agenda.

E

um But yes, uh I'm, trying to remember I think we just checked that there's a publishing workflow, um although I, might be confusing with Packaging, but it I think having an issue to discuss or just changing the issue. To mention the release. Workflow would fit better with sort of the scorecard frame so that that sounds reasonable. For me,.

A

Any other thoughts from anyone else on that uh does that make sense what Spencer's suggesting.

G

This is a nitpicky thing, but okay, we got a project. How will you find the actual s-bombs I guess just looking at the file name? Extensions is that kind of just walking through the files or assuming they're at the top level or I mean that's a low-level question, but we do need an answer for it.

B

Yeah I think that um there are I'm very happy to build in uh functionality to say: is this an s-bomb, yes or no within the context of the s-bomb scorecard, and then we could just um try files and see if they are in fact s-bombs that are known to the to the project.

E

I can include um a link, but uh Google has a utility called osv scanner and they have they had this sort of like how do we determine what an s-bomb is and there's you know some naming conventions that people may or may not follow um and I'll link the issue and put it in the agenda.

A

Yeah, thanks for adding that Spencer, so I would say, um I think on the issue itself.

A

There's maybe a few things just to clarify to add to it basically feedback here, one is, let's maybe adjust where exactly uh this would, which check this would exactly be linked to and then kind of just describe the general method that you're planning to find s-bombs and Metal.

A

You know also be I, think a natural place to discuss spilling in things like the osv, tooling I'm, seeing if there's some something to be used there.

A

um Otherwise, you know I I, think what we could do is clarify a few of those things on the issue and then either we'll close it out on the issue or we can bring it back to the meeting. If there's there's a lot of back and forth on it.

B

Okay, so I'll I'll make those two adjustments to the the ticket and then the the assumption is that people will engage on that ticket and direct me forward on how to do something else.

A

Yeah, that that should be uh pretty good path forward.

A

Thank you again just for for bringing it to the group. I think I think that's an exciting extension um other topics that folks want to bring up, or also last chance to chime in on this one.

G

Yeah, a quick first chime in which is I'm, excited, I'm, sure there'll be complications when you actually try to get do things because there always are. But uh I think this looks very promising.

G

And I I do have a totally different topic that I wanted to talk about real, quick uh uh last minute proposal on the agenda, so I understand, if you don't want it, but.

A

I'd say we're doing good on time. uh We've just got a few announcements at the end and uh we definitely have time to take on another topic.

G

Okay, this one's really just more of a heads up um and if you don't mind, I'd like to share my screen real quickly here sure uh so, let's see here, let's see if I can manage to push the right button. I can push buttons. Let's see if I can push the right button.

G

All right, uh this horrible thing pretends to be my desktop, uh so basically, uh probe and I are going to be uh giving a brief overview on both the opennesses have best practices badge and scorecard and uh we're a little bit. Both of us are kind of too many things trying to do too many things, and so we're right now trying to wrap up the uh the scorecard presentation.

G

um This is not the final. This is the starter of it though, um but I just wanted to talk about some of the things that we intend to talk about and basically see. If there's you know anything that, oh my gosh, you messed it. You missed something important, or at least in particular, missed something important or got something wrong: I'm not going to go through details, because I presume that all of us have some idea. What scorecard is, but hey, there's scorecard automatically scores OSS with a bunch of checks.

G

0-10, um you know listing some of the sample checks. Is that I'm I'm playing you know, and uh you know, uh and it gives an aggregate score.

G

Some of this is duplicative and we really need to pin this out, but just you know right now. This is the hey. This is the starting point and then we're going to clean it up.

G

um I intend to talk briefly about sonotypes analysis. I. Think that's really interesting. So um you know I mean that's really. Sometimes analysis, not your own, but I think it's helpful to know that in particular- and you know that some of them- you know that hey it looked good overall and that those four in particular were interesting. I did by the way, find it very, very interesting, I think nobody was surprised that code, review and Branch protection were important.

G

I think a number of people were surprised that checked in binaries and pinning dependencies were so predictive, but they are so. Let's give credit for that.

G

um I also want to so we we know we need to do cleanups, but you know that's kind of where we're going. um I want to talk a little bit about the uh upcoming scorecard work.

G

um If yeah I I think we ought to mentioned that get lab progress is ongoing, Milwaukee's been doing it as far as I know. That's not a secret, so I think we should. You know both both that it's happening and who's working it um uh improving Automation in general, with better support for different kinds of pipelines.

G

uh You know better tool. Detection I know that you know that's that Amazon is funding that work. um I know, there's been talk about potential new metrics, um you know you know empty and maybe maybe I should even mention the s-bomb. uh You know s-bomb scorecards as something that's a potential new metric and some various cleanups all right. And finally, this is my pitch about how to explain these two projects.

G

Okay, um my pitch is that best practices badge and scorecard both work together can work together. They take very different approaches to it um scorecard I mean the big advantages of it. Is that projects don't need to do anything? You can just measure them and you can get quick results on any project. That's a real big plus. That contrasts pretty. Obviously the badge where it requires best practices out requires project participation, Journal about 20, 30 minutes.

G

There is some automation, but a lot of it is you know it's still a form fill in uh what are the negatives though, um and I don't think these are killing it's just I'm trying to do a contrast. The big challenge with scorecard is the challenge for all tools, which is you've, got false. Pauses, false negatives, I think that's just maybe maybe I should even say. You know, like all tools right here.

G

You know it's prone to false positives and false negatives.

G

Like that, okay yo, uh you know so you know detecting tools, detecting the CI system. You know the detecting a file is not you know. The presence of a file does not mean that something useful is happening. Necessarily it's a good sign.

G

um Currently, it always does. Github gitlab is coming and going report on things that are automatable best practices badge. It works with any Forge um and it can report on what's important, with the caveat that that requires you, an analysis which is the the challenge. So the theory here is they should work together like chocolate and peanut butter, okay, so thoughts, comments, I I think this I'm trying to be fair to both to uh both sides, because I think each of them has their pluses and minuses.

G

Because of the approaches that they've taken that they've taken seem plausible.

A

I'll say personally: I I think this is a really nice way of laying them both out um I'm wondering just in this last comparison. If you were to say you know, scorecards is automated best practices badge is manual if that would just immediately kind of get in people's heads. uh This is this is the significant.

G

Part yeah: this is part of a larger brief, where it talks a lot about it, and the problem is if I just say it's manual. It's not actually true, uh because there's something like a quarter or a third of them that actually are automated. uh The the difference is, if there's no Automation in most cases you can override, unless the automation determines that the human claim is false, so there's automation to figure out the defaults, but to deal with the false, positive and false negatives. In most cases, humans can override.

G

So it's a simple way to describe it, but it's not actually quite true, it's more subtle, but it does require project participation and that I think, is the difference.

A

Yeah I would I mean I would agree with that. um I think that the mix of the two is is just I think, but maybe didn't come through for me right away that there is some automated in some manual for best friends right.

G

And and I didn't want to show that part because limited time I mean if you want to talk with me offline happy to do that um and uh sorry that we're not quite as far along on the actual slides but uh we're we're getting there. But we're not very worried about this, because I I think you know. Scorecards are known quantity, but we want to make sure that people who aren't aware of Scar card will will become aware and start using. It.

A

We definitely all want that uh when, when it comes to again who the audience is, this is I think you said for open ssf day yeah great.

G

So, okay, that that that's all I just wanted to give you know, even though um we're not quite as far along as I, want as we wanted to be I wanted to share that now, so that uh if there's some something wrong and really is, is there something important that I missed and we need to clean up the slides and eliminate the dupes? But is there something that was really important that we didn't include.

A

I'm watching for hands here, I I, don't see any yet does anyone else have any other thoughts.

B

I would say the one one thing that having gone through the best practices badge, knowing that there's levels is important because real effort to do level one and then you're like oh wait, there's more okay,.

G

Great yeah yeah now and to be fair for the best practices badge. We have silver. We have gold, we encourage projects to get them, but the reality is we've been from a uh this is not really a scorecards relevant, but what the heck? You asked the question um for for badging we've been focusing on our messages focusing on getting people the passing level we do for the ones that are important.

G

We want them to go beyond, but the theory right now is that, if you're, not even at the passing level, you're probably the biggest risk- and you know for the projects that are used everywhere and really security critical- absolutely we want to go higher, uh kernels and so on. But you know if you don't, if you don't have a way to report vulnerabilities and you don't have any tools and that sort of thing um we're more worried foreign. So that's where we've been focusing on our messaging anyway,.

A

C

A

Other feedback um I think before we've got another presentation announcement that Spencer's adding right now all right fencer. If you want to just give a quick note, then.

E

Oh uh yeah, just that David reminded me that this is a great form to get feedback on, uh for something like a talk. So Davina and I are giving a brief talk on uh six door and how uh scorecard action uses it. When we sort of crowdsource some scorecard scores that feed into the API.

E

um So David jokes, that you know his slides aren't done, but ours are even less finalized um but I'm. Looking at uh two weeks from now, and it's still before the conference um so I might bring that forward in next week or next meeting.

A

I think that'd be great and unless there's any other topics to add, uh we should talk about that next meeting and see if I can get a volunteer for a facilitator unless we already have one uh nope, so this would be May 3rd, I believe sorry, May, 4th um Jeff is that. Are you.

D

Volunteering, no no I had a question. Okay. If it was an open Agenda. Let me find the yeah.

A

It's it's back to open.

D

Okay, um I was looking at the the readme's or any um generalized guidance for pulling in scorecardita um I know we have the API, the rest API and there is bigquery I, don't know if that's available, publicly or and then there's also depths.dev um I think it'd be nice to have like a comparison between the three and, if you're, building an integration that needs to pull data like which one is um the one you should use um or if anybody has anything to say about that right now, I'd be appreciated.

A

I would agree uh that the comparison would be good. You know when each of these became available they built on something that wasn't there before so I think bigquery was initially made publicly available. An API was built on top of that to make it easier to access and since that time, depths.dev is a large consumer of that data. It started making a public API available for themselves as of just a few days ago.

A

So I don't know if anyone has a strong feeling that they would like to to be part of that evaluation. uh Otherwise, I will see if I can round up some participants because I agree. That's an important thing to do.

A

D

A

I'll be uh I'll, be asking around for some help.

D

Yeah I was looking at some of the rest apis on the uh on the readme I. Don't know if any of the others are um but yeah um right now for uh the guac project, we're pulling in depths.dev um and yeah I didn't know if it would make make sense to do one or the any of the other ones.

A

I'll give the disclaimer Dev is a Google project, so um it's it's not in the openssf I I can speak just as being a little bit familiar with that project. My understanding is, their API is open for use and basically what you're using it for now would appear to fall within the terms of what is acceptable to use it for um I, don't know if anyone else has other thoughts on that I think. A lot of this will come down too, to kind of just throw some cons of each one.

E

Okay, well, I will add that Dev does filter some data out. So if you want the complete picture, uh definitely stick with bigquery or the rest API pretty.

G

Cool, that's news to me: what does it filter out.

E

um There's a couple checks that were previously enabled in the Quran job and are still being tweaked under our new GitHub approach, where uh the data were not, uh you know it changes a lot week to week and it may differ from what projects previously had so just for all scorecard works on figuring it out. Debs.Dev is hiding some of those values.

E

A

I think to be clear: that's they're intending that part to be temporary in the long term. They surface all.

E

Correct correct, okay,.

A

G

This is just while they are retooling. Some things.

D

Yeah, and are there any plans to like merge the two, like I'd hate, to build a integration on the rest API? If uh tomorrow, you say, I, just pull it from back to that day,.

A

So, at least, to my knowledge there, there is no any plan to do that. uh I I think doing an evaluation of all of the ways to access. The data, though, is is a start towards. Does that even make sense? Is that something we should be talking about, um so yeah I will no doubt be maybe twisting some arms seeing if I can get some get some help to build out that comparison.

D

All right thanks a lot Brian.

A

And Ian I apologize, I think your hand up is you've had a hand up for a while uh go ahead.

C

No worries it was uh I'm happy to facilitate the next meeting so irrelevant to the discussion.

A

Fantastic, that was the that was the hand. I was looking for all along.

A

So any other last topics or comments about previous things before I have. We would dismiss.

C

Quite a I do have one minor one I think it's kind of it's a small note. um You know my employer. We talk a lot to gitlab, so we had another meeting uh last week where we talked to the chief product officer and some other folks, and so we made our desire to get them involved here. Very clear um and I know. Azine and ragoff are on it. There's a public channel that they created specifically to support this activity.

C

So if you need to reach out directly to like the lab leadership, um his name is Sam white. He is the manager product manager in front of all their government stuff, so he's very much aware of this and create that slash data specifically to help this team out. So if there's anything, you need from GitHub I know you're already talking to them a little bit, but this would be another really good way to reach out and make any needs known.

A

That's a really positive development for the project, and thank you just for making us all aware that we can. We can have that channel.

A

All right uh last call for any other topics.

G

It's not a different topic, it's a different topic, but on the whole depths.dev versus if somebody has a good way of summarizing how they compare. That would be very useful for a slide that I might have to make soon.

G

So, if somebody has thoughts on that between now and uh next meeting, that would be awesome.

A

C

A

I will say: uh I'm guessing I will not have that by next meeting, so I very much if you, if you have this already or could compile quick thoughts, you know between now and then it sounds like uh both David and all of the audience. Members in that presentation would uh would appreciate it.

A

All right, um well, I, I, think we've kind of covered everything at this point um very, very, very last call uh for for any last topics. We've got a facilitator for next time.

A

All right going once going twice, see you all in a couple of weeks. Thank you all.

E