OpenSSF Scorecards Biweekly, 4 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Scorecards Biweekly Sync (May 4, 2023)

Description

https://github.com/ossf/scorecard#connect-with-the-scorecards-community

A

B

A

B

One more minute.

A

uh In the meantime, if you can please uh either ungray your name on the attendee list, uh so we track it was attending.

A

All right, let's get started so hello. My name is Ian Dunbar Hall I will be your facilitator for this. uh This meeting, um if you haven't already, uh do please uh I, just ungrate your name, so we know who attended uh reach out to me. If you don't have permissions out of the stock and out there for you.

A

um So let's start out with introductions uh anyone new who has not attended this meeting before want to speak up and give an idea of who you are and what you're working.

A

All right looks like everyone's been here before all right. First on the agenda is Project updates. Is there anything from All-Stars or around the scorecard he's a per either group want to speak up.

A

Yeah, no more certain things: okay,.

A

All right, I have a feeling: it's gonna, be a short meeting. uh All right. We do have one open issue. We want to talk about code review check, uh so it looks like the discussion about using AI to review code and helping projects with a single uh hoping projects with single maintainers uh yeah go ahead and pick up the front side board.

B

Yeah, hey everyone, I added that topic, and you know it's a um uh it's a brand new area that I wanted to discuss with the community, and you know, use this forum to approach the topic.

B

um As you know, in scorecard we have a check that evaluates whether code reviews are being performed or not, and uh and the recommendation right now for code review checks, if you, if you get a low score, is to uh find you know additional people who can review your code, and this has been a bit problematic for projects where you know you have single maintainers. So I was wondering you know because of the latest advancements in Ai, and you know like uh uh now.

B

There are so many Dev toolsions that rely on AI, uh whether we we should consider the use of AI. uh You know, as per in in the code review process uh and sort of provide a way for single maintainers. To sort of you know like get a decent score on this check without finding another human and I know. This is a broad topic. So I was curious about the community. Thinks about this approach. What are the pros and cons?

B

And if there is any way you know we can incorporate or improve scorecard uh check, based on based on that.

C

It looks like many of the core maintainers are on the call today, but uh I think that's a pretty interesting. Take I I was recently looking at um something that semgrip did, which is sort of similar where they were looking at using their static code, analysis on GitHub PRS and then sending that off to chat gbt to like determine false positives or propose patches, for um you know, issues that were found by some grip and and I do think.

C

That could eventually become certainly a better than nothing check right like maybe it's not a full the same as a human review. uh I think the big concern there is one of the intents of that check is to prevent someone from putting malicious code into the code base that a good actor looked at it and said you know this.

C

Isn't somebody trying to put malicious code in and do we really have a good feel that AI is solving that I guess and you know making sure that there isn't someone that is intentionally corrupting the code base.

B

Yeah I think that that that's a good point and- and the idea here is that you know if you can actually figure out all these concerns, and you know there's a way to address these concerns. I think this would be a great vehicle to help maintainers, especially the you know, the ones that actually manage a project by themselves. So since we don't have core maintenance on the call, I'll also go ahead and create an issue to sort of discuss this offline.

A

Yeah we were going to call earlier today with some folks from cncf and I know that they are. It sounds like they're going to come out with a position on on this topic.

A

um I think there's some fundamental concerns about IP and if uh llms are going to introduce, you know GPL code and the things that that's going to be a problem, um so I think I mean I, guess it's kind of outside the scope of security right and providing security guidance, but um I I could I could perceive in some programs or some projects trying to say like we do not want to see code from this if it becomes a a license issue.

B

Yeah- and in this case the recommendation is to not use AI to author code, it is to review uh you know the code being produced by human users.

A

That's a good point. Well.

B

I think the action you.

A

Said you know ticking and creating an issue, for it is probably the best course of action and we can see from there uh since David just called uh dialed in he might have some uh some opinions on this too um I'd be interested in real house issue.

D

uh The whole uh AI generating code stuff.

B

Right, yeah and the the use case here is to not use AI for generating code, but to review code so that you know we can provide a way for projects with single maintainers to be able to like score uh decently. On this check right now, you know like they, because there is no other human user, they get a score of zero and the idea is: can we use AI to actually get something better than zero? It may not be ten, but something in between.

D

Okay, we're not talking about changing scorecard itself for changing talking about projects they're trying to implement and get uh better scores on scorecards right.

B

Yeah- and this is specifically for the code review check where we look at uh you know like pass changes and make sure that those changes are reviewed by a human.

D

Okay, yeah there's two different issues: using AI to review code and using AI to create code I, don't know how in the world scorecard could detect the use of AI to create code I realize it's not exactly what the text says here. um There are some efforts to try to detect it.

D

um I think that's a losing battle and I. Don't I I I mean I I I, don't even think we should try to go there um as far as using AI to review code.

D

I, don't think scorecards should prevent it.

D

um The question here is: does that count as a review of code I would think the answer should be no, um unless we've got good evidence that the AI is very, very good at reviewing gold.

D

um There are currently I mean we don't already don't accept code review from other kinds of tools if you run a static, analyzer and a fuzzer. Those are obviously highly encouraged. Behaviors, but I would view running. I would review Running an AI as another kind of static analysis tool, and should you use a static analyzer? Yes?

D

Is that a human review? No?

D

And so maybe we need to clarify that, but um now so far the reviews that I've seen is that the ARs are terrible at finding vulnerabilities um they're nowhere near as good as at least now careful here, because uh oops I I. The problem here is I'm speaking about AI, stuff and I. Think we all know don't blink. There's a change.

D

I I think there's arguments to be made. That 2023 was the end of the Industrial Revolution and the beginning of the AI Revolution. uh We will ask me in 50 years, uh if I'm a robin, um but um but right now we don't have very good evidence that they're any good at it right now. The evidence is that they're, remarkably better than they used to be at generating code, but they're still terrible at reviewing and finding vulnerabilities in it. So until that changes I think we should dream just like any other static analysis tool.

D

I'm thinking out loud with the answer, but I think the answer should be so you're watching me think through the process have I convinced anyone or not.

C

I think one of the one of the interesting things that I brought up before you got on was there has been some I, think sort of interesting work in taking a static code, analysis tool and, having it find say, vulnerabilities or problems in the code and then sending those to chat, gbt or another AI tool to a weed out false positives.

C

It's like does this appear to be a false, positive or not based on a large right right amount of data or to if there's a if there is a positive, then to propose a fix for that um as part of the merge request and I think there's some fairly interesting work. There yeah.

D

That's not new people working on that five years ago, I actually saw now, as I said. You have to be careful about. You know. What's current at least five years ago, they weren't very good at it, I mean they. They were a little better, they weren't great, but hey this year. Things have gotten a lot better, so I'm totally willing to believe that things have gotten better but again, I, don't think we're. We can make a claim, that's anywhere near as good as human review.

D

So until that point I think you know using AI to to help check problems absolutely, but I would just stick it in the category of static review, as opposed to human static code review and not human review.

A

D

A

Had your hand raise for a little while, oh.

D

E

Yeah um I I think I share the same concern as um we said that Keith that I think the original intention of this check was that uh people fundamentally didn't want to depend on single maintainer projects because they didn't want to be uh vulnerable to the whim of a single person, uh and you know maybe they're vulnerable to a collaboration of two people, but uh but not a single person, um and that's that's like fundamentally at odds with single maintainer projects.

E

People who host them and say you should trust me, but um I- think that's kind of where scorecards is right. Now is it's, you know trying to satisfy the desires of you, know: project producers and consumers right and the consumers say. Maybe this is something I don't want to trust.

E

um I think this kind of highlights that um consumers of scorecard or scorecard scores should be able to customize their their consumption. So if I do trust single maintainer projects, maybe I could get scorecard scores via an existing database. You know with my own filter on it or something like that.

E

um To show that um I, don't I, don't worry about the code review score.

E

But yeah I think I think we need. You know it's again. It's not about you know whether or not whether or not the AI is good or bad I. Think isn't you know. Is this a concern, but maybe not the most fundamental concern with the people that originally thought of this score, which is you know, don't make somebody let somebody who uh just does something again on a whim without any any collaboration, uh publish a package that then impacts. The text me.

D

You know what uh I think, uh Jeff and I uh came with this with different approaches, but I think we end up with the same answer.

A

D

You sure you're, not just two different AIS, just checking that that joke will get less funny over time, enjoy.

A

A

B

Right well, I think getting.

A

An issue out there sounds like a good first start, um since us I think most of the core maintainers are not on. The call today would be good to get them away in Jeff I. Think that sounds like a great idea and David. Thank you for your thoughts, um any other items before we we go to the summary.

D

I do have a request here, which is since we've. It sounds like we're more or less agreeing. Is there a change? Do we need to add some clarifying text to this Criterion that says, hey AI doesn't count I, think that would be perfectly appropriate. A lot of people are asking about Ai, and if this is something that we are widely agree to, I think mentioning it in some detail. Tech somewhere is a good idea.

D

um I could even draft that text. If you'd like.

C

David would have possibly another compromise be to allow that to upscore a different area, so you know again, if we're looking at.

A

C

You you made it analogous to to SAS checks, which I think is probably appropriate, but maybe they're more advanced SAS checks and to allow that to positively influence the score, but not to replace the intent of human review. Check.

D

I I think we were only talking about human review check. We.

C

Were yeah yeah so.

D

So I I would focus there. I mean, for the other, I mean right now: we're not really counting quality of tools, so I.

A

D

um If we ever go how's this, if we go towards evaluating the quality of tools, then I think yeah. We would need to go there, but not until then, but then, if we do that and that's plausible then yeah.

A

All right last item on the agenda is to figure out who's going to facilitate next time. Do we have any volunteers.

A

If there is no volunteers, we'll we'll shift, the conversation over to stock then see what we can find something there.

D

What's the facilitator transfer all about just nobody's here to run the meeting or.

A

So for every upcoming meeting we try to figure out who's going to be running ah okay, so um there's no volunteers. This time then we'll.

C

A

Try on slack for someone I.

C

Can take it, this is Caroline Lee, perfect.

C

No problem awesome.

A

All right, we'll call good thanks. Everyone.

B