OpenSSF Security Tooling WG, 4 May 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Security Tooling WG Meeting (May 4th, 2021)

Description

Meeting Minutes here: https://docs.google.com/document/d/1DoB7zgtLsP-JGF77ASkHV7UMofTE2wseniexaa6Q4M8/edit?usp=sharing

Jonathan Leitschuh and Jonathan Schneider talked about what they've done with automated pull requests to fix security issues.

A

Stuff that uh tied into uh the stuff that I was doing- and so um I think this is your first me- this is the first time participating in the open, ssf meetings right. It is yeah welcome.

B

C

Welcome jonathan, you want to tell us a bit about yourself.

D

Sure yeah I'm uh right now. I've found myself in as a co-founder of a company uh dealing with the automated source code, refactoring um style, preserving type of wear sort of stuff. So it's kind of right up this alley. It's uh we used to do like framework, migrations and vulnerability, patches, dependency, changes and so forth.

D

um I worked with jonathan a bit last year on one of these that he'll demonstrate a little later.

C

Do you mind putting a link in uh the chat I I would love to just go poke around and see what you're doing absolutely will do like? No, I'm not going to. Let you inform yourself.

C

D

We get into uh jonathan's uh presentation.

E

Anybody have any opens.

E

uh No big open, but just uh uh those of you uh yeah there was an open, ssf town hall yesterday and I think it went well and the metrics the security threats. Folks have some interesting, metrics things. So there we go.

B

All right shall I go for.

C

It only delayed two weeks.

A

Yeah around that's, okay,.

A

So all right, so I gave this talk previously at a conference called get up nova, which was a private, get up stars only event, so, um but the titleist talk is called creating thousands of pull requests to fix the uh open social security, voter religious scale um hi. My name is jonatha, I'm a software security uh engineer at and security researcher, I'm currently at gradle. um Why do I do this sort of thing? uh I have adhd.

A

I dig into holes and I go like just I like my focus can get yeah and- and I I get carried away with security vulnerabilities and and that's how I find things. This is a tale of two widespread security vulnerabilities.

A

um Let's start we're gonna start with the first one, which is a jvm ecosystem, um vulnerability, which is the use of http to download your dependencies either in maven palm files uh or um gradle build files, um but this yeah the java ecosystem before 2020 the the january 1st 2020 a huge number of projects for using http to resolve their dependencies. The other vulnerability was a a vulnerability to code uh code, generator called jhipster.

A

um Where there were one point, uh there were 15 000 repositories that got across github with this generated code in it that was had this vulnerability, and I found the vulnerability in the code. Generator grab the code generator fixed, but there were still all these generated projects that that were vulnerable.

A

um So, let's start with the jvm ecosystem used of hdp. um This is a you know, classic man in the middle attack of your supply chain components, um the build two major build tools. Java are for the java ecosystem or maven gradle. um Neither of them do any sort of artifact verification um by default. So basically the only verification that you have that you're actually resolving the components you asked for is tls right, like you know, guarantee that that nobody's monkeying with your connection between uh the server and you.

A

um Unfortunately, I found that um there were a wide number of repositories all over the system that were not not using https um back in uh 208 snow 2014. Just after the snowden revelations uh max vietman wrote up an article called how to take over the computer of any java scala or groovy developer.

A

um That kind of called out sonar type, which is the biggest artifacts or host in the java system, for not for only offering https if you paid a donation fee, um and so uh he published that article and uh called them out and said, hey like you know, you're, basically leaving uh this insecure for everybody and he wrote uh dilite, I think, is the product that was basically a thing that um backdoor jars files, and so this is his demo of you know downloading a jar file backdooring.

A

While it was in flight and then dropping in this, like you know, uh swing component that popped up saying, you didn't serve your jars with ssl, um so um of course cats. um So I you know in 2019 started. I found this bit of code in my repository, I'm like where, where did this bit of code? Come from? What I found is. I had actually copied this code from uh one of my dependencies, their source code, I'm like huh okay, so I reported to them and then I'm like.

A

I wonder where else this is and then I started looking and it just kept growing as this problem, like I just found in more and more repositories across github.

A

um Let's look at some quick examples of where this vulnerability can exist in gradle files. You can have it in your build script. Can you see my mouse yeah? I can see it okay, so uh you can have it in your build script block. This scope is dependencies that get resolved into the gradle build, which is like the thing it's the it's.

A

um These are like plugins for gradle to change gradle. um This is your repositories block for actually dependencies to your compiler and are loaded on the fastpass for tests. So this is not the build, but this is just like the application that you're building that you're going to produce either your like android application or your war or whatever, and then this is where you have uploading of artifacts right.

A

So this is like I'm publishing a library um and I'm publishing over hp and, in this case, you're also leaking credentials because you're done you're uploading it with credentials in plain text.

A

So these are the three plain main places: you'd find it in gradle in maven, you have the repositories block, um and this is you know dependencies of like where again, the compiler and loaded on the class pass for tests. This is distribution. Management which is used for publishing. Dependencies also has credentials, and this is repo plugin repositories. This gets loaded into ma. These are dependencies that get downloaded and run in maven as as plug-ins.

E

Yes, yeah, you mentioned the uh the password I mean. Not only are they getting set in the clear but they're being checked in and stored in repos? Aren't they.

A

uh No, those are usually in settings files, but you usually so for if you're reading these things in okay, um for here, usually the user, repo username, reaper password, those are usually read in from environment variables or properties, and like that um you I mean it's not on it's, not not to say that people don't just drop code in there, but most often it's fed in from one of those external sources. Okay, so we're moving vulnerable, the jet brains, the close foundation, gradle red hat kotlin jeff spring.

A

You know all these projects, uh oracle, the national security agency, linkedin stripe. All of these projects have this vulnerability in their open source projects. um uh I made a bit of bug mounting money from a couple of these companies. A lot of them didn't pay me a thing, and I you know it was a large large project, so I reached out to sauna type, which is the biggest house in the mavenik system and said hey.

A

This is a vulnerability I'm finding everywhere what's going on um and they reported back after a month of looking at their data that 25 of the sonar type maven central downloads were still using http um as of april 2019., um and I'm like great, let's fix this, so I've pushed forward a proposal um yeah. How can we fix this? So I push forward a proposal um on or around january 15 2020.

A

um All I contacted all these artifact servers and said: hey, let's fix this together, let's shut off the use of http like not a redirect, but like full, stop like you will get a 500 or whatever right like you, won't get a 302 or 301 redirect like it. Just you know, doesn't work and uh all of these different artifact servers that are the biggest ones in the supply chain for the javascript, all just full carte blanc said no stop like stopping requests that support http, um and this actually worked.

A

um uh You know I saw some like very confused stack overflow posts on the day that things shut down but yeah. um Yes, broken software broken software everywhere. So what about everybody else? So these are the biggest right. These are the biggest artifact servers right, maybe central j center spring right, but there are other people that are relying upon artifact servers that are like you know. My company or um uh jenkins has their own artifact server. There's like a bunch of different companies that, like or eclipse, foundation right eclipse foundation.

A

I couldn't get to to drop hdb support. They still haven't dropped, http sport they do a redirect now, which is whatever um so enter codeql, so github bought uh semi, which is now codeql, and when I saw it, I'm like I previously been doing this research with, like github fuzzy searches, huge pain in the keystore like github search functionality.

A

Not all that good. um It's good enough for finding specific things. If you know the right query, but like it's very fuzzy and it also only returns, the first thousand results um you and it's really hard to do like yeah anyways. So um when gita bought semmle, I was like okay, I can use this.

A

So I wrote this code qr query which looks for I can't do gradle, but I can do maven because they when, when codeswell runs it slurps up the code and also xml files, and um I can write this code coil query that finds these um these dependencies.

A

Our finds, finds this repositories block and sees that you're using http or ftp in that in that block and flags it for you and for this uh query they sent me a or they paid me 2, 300 bounty um for just for writing this query, as well as some associated documentation um and now, if you go to lgtm.com and look for https or sftp url um right the failure to use this.

A

If you look for this query, you can just see pages and pages and pages of this vulnerability on in open source software across github, um but this still left thousands of projects vulnerable. So we've gotten, like you, know, shut it off at the source, fix it. You know or detect it, but how do we actually fix this? So I said: okay, like I've. I've done you know. I I've written this code to a query. Let's use it to like actually fix all these vulnerabilities right, like they're, open source, kill them in fire.

A

So what is um uh what are we looking to do right? It's a simple diff, we're looking to make a change from http to https, um and um so it's a you know a fairly simple change like you know, and- and I ended up for this one- I ended up. Writing a pretty.

A

You know complex, regular expression to do this, but once I had the regular expression working, I'm like okay, like now, let's, let's, let's see if I can write a bot to do both pull request generation and so the github security lab, has a bug, bug slayer program and I figured okay. They had a program where, if you fix so the first one was write, a query um uh submit a query and we'll pay you money for it right. They have a that was the all for one one for all.

A

On top of that, you can use that same program and submit two maintainers hey. You have this vulnerability and get a cv sign, and if you use a code query to find those vulnerabilities, they'll pay, you uh they'll pay you at two thousand five hundred dollars maximum for four vulnerabilities.

A

I figured there were 17 000 repositories with a vulnerability, so you do a little bit of math. It's like you know a million dollars worth of like bug monthly payout, potentially, um which is not not not actually didn't, actually happen, but you know I figured math right. um So how did I build this thing? I used um the hub cli um python, pi github and pi yeah pi github.

A

I used async io to do um like a lot of things synchronously um in python and then one very annoying regular expression and a lot of bouncing off the uh rate limiting api for github um and yeah generated thousands of pull requests to fix this one. I think I generated 1 596 pull requests for this particular project and github took paid me to that.

A

They said this is worth two thousand dollars, but because of the number of vulnerabilities you found here are the number of cases you fixed: they paid, they doubled the amount up to four thousand dollars, just for the number of number of cases that I fixed with this pull request: generator um okay, so the second case was a a code generator uh a vulnerability in a project called jhipster, which is a code generator that generates um uh uh your base like spring mvc application and all like the bolt nuts and bolts of like getting you just like this base application up- um and I found this vulnerability- um uh uh this was the use, so the vulnerability was the use of insecure, render number generation in a security sensitive context.

A

So how did I find this? I ended up. I was looking into a while ago the use of random string utils, which is an apache commons lang, 3 library, um to generate random strings in a security sense of context right so like in passwords in activation keys in reset keys right. So when the reset key is in this case right like that, um when you do password reset right, I need to reset my account.

A

um You will you, you say: hey give me a token and that that token, right that you get from the server needs to be randomly generated. If it's not securely randomly generated. When that ends up in your email, you can say ah I can, then.

A

If you get like one or two of those numbers, you can then compute what the next number will be for one of those password resets, and so when you know you can then send a bunch of password reset emails to a bunch of other people and know what those tokens will be and then you can do the reset on their behalf, without ever seeing their email um and so random string.

A

Utils from apache commons line, three uses an insecure uh c, uh insecure, random regenerator on generator under the hood, and I found this code and I'm like you know, I kept finding the same code everywhere and I'm like what what is going on here. I eventually found out it was coming from a code generator um right. So this is the sensitive one right right generate reset key.

A

This is the reset key for uh resetting somebody's account um and when you search for uh this, this class right utility class for generating random strings, which is indicative of this, um uh this jhipster code, it came back with 14 000 results. um Unfortunately, github search limitations there's no way. There's uh it only returns. The first 1000 results, even with page nation and there's no known way to slice the data with additional search parameters to get you more information, um so special thanks to chris gavin and arthur bars.

A

I actually just talked to github and said: hey. I have this limitation. Can you get me a bigger list? Apparently there's some hard-coded limit of a thousand results and they actually like changed the code of github and like were able to run a query and like gave me this data, um and so that gave me 9 000 repositories, a list of right, 9000 repositories, um and I imported these 9000 repositories into lgm.com.

A

To then run my query against and find all the vulnerable instances. So how will we rewrite the code? So this is where we enter with jonathan. So um I was looking for a solution. Our one of the things that I've been playing with is okay, like the first time I did this, I used a regular expression, but we're actually changing code, this time right. This is actual.

A

Like you know, and and and even though it was generated by a code generator, it may have different formatting and if it's different formatting it's going to have a much more difficult time getting accepted by the maintainers. So how do I, how do I, and also like I, don't want to write a regular expression for this thing?

A

So how do I do this in like a sane way, and thankfully I've spoken to jonathan, he worked at gradle before we had a lot of conversations about this and I'm like um uh how do we do this together in a constructive way? So I'll? Let you take it away for a demo of rewrite uh so before we bring it back to me but yeah. um Let me unshare my screen and you can take over john.

D

Sure yeah thanks, um let's see just as a little bit of background uh rewrite it posted a link to this github repository earlier, but it was a technology that originated at netflix. I worked there on a central engineering tool scene.

D

There were about 700 engineers and two of us trying to get 700 engineers to do something big, like uh the big like cultural imperative. There was freedom and responsibility uh which meant the product engineers could do whatever they wanted. um As a central team, you could not put any gate. Employees could break their bills, couldn't anything like that right, so they want to use something different. It was up to them.

D

That means that there's 700 engineers there's you know 1 000, 100 different styles, and you know it's just like you know it's all over the place. um So we worked on building a lot of reporting tools originally trying to surface issues to help them. You know understand where we were they needed to make changes, but um you know we're engineers and uh you know unless it's gonna like break we're, not gonna do anything.

D

So they had a internal logging library that had been. That was basically like sl4j.

A

D

Java logging library they wanted everybody to use esl for j deprecated, the internal one. Six years later, there's still 5 million references to this thing. um So it's just one of these kind of things like it's, never gonna go anywhere, you're, never ever gonna be able to root it out.

D

um So that's where the the sort of genesis of this this uh this automated refactoring technology, that also did a lot of work to preserve style um rewrite is an abstract syntax tree based manipulation, um but, unlike most abstracts and text trees, the actual formatting of the code and type attribution is maintained inside of that ast, and it's also made to be a cyclic which most asts are not to make it something that you can actually serialize out and operate in mass offline.

D

So today we're able to you know, operate on millions of source files. You know- and you know, in a horizontally scalable way, uh performing different sorts of transformations.

D

Like john, that says, I have a sort of brief uh video here that shows some of these technologies and what I'm going to show is actually a view of running rewrite recipes from a nascent sass that that that my company creates, but all the rewrite recipes themselves are open. Source apache license can be applied to your build plugins and other sorts of workflows as well. Cool just gives it an easy way to to demonstrate it.

D

So with no further ado, um we um can start somewhere simple, like um just trying to find types find where particular types are being used. Jonathan was looking for uses of random string details earlier. Can you full screen yeah sure.

D

What, if you wanted to find an actual.

A

Actually that actually, it dropped when you full screen yeah. Let me see.

D

I don't know: what's the right, no doing new.

B

D

Yeah yeah. I think this is about what we're going to be able to do here. um Yeah. This is theater mode, so it's kind of like full screen you're, not.

E

You're not right now for sure to drop the share.

D

B

D

Going to go we're going to go with the like uh theater mode and I think that's the best. We're going to be able to do.

F

D

Okay, um so can you.

A

Share your screen.

D

Back awesome, cool, yeah yeah. um So what if I wanted to find a particular method pattern um you know say, like uh I, you know in this case I was looking for a particular occurrence of of that random string utils, um because the abstract syntax tree that we're operating on is is type attributed, we're able to actually accurately identify the um calls like in this case like make dirs, but we know that this receiver type is a type java.io.file um and that's essential to not get a.

D

You know, sort of over reporting situation into an overreporting situation. Where we're you know eagerly matching on just the name and not the not the type. So we actually, in addition to code, we have abstract syntax trees for things like maven.

D

Actually, so you know, we have an abstract syntax tree for xml for yaml for properties, but we can layer on top of those additional models of what that xml means or what that yaml means in this case, here's an example of actually looking for the junit, which is a unit testing library in the java ecosystem, on the compiled class path. So, in other words, this testing library is going to be shipped with main code and ran in production, which is, I think, not intended.

D

I think, there's commonly vulnerabilities and unit testing libraries that you know that would make them sort of like an undesirable thing to have on your runtime class path. And here you see, you know neo4j, which is a common graph database in the java ecosystem, leaks the junit onto the runtime class path of any of its users, and so it's kind of whoops right test dependencies don't belong in the app.

D

So what if we were to actually go and say excluded, so we're going to take the same group and artifact and we're going to say I don't ever want to see this in the compile scope again. um You know I can go, and I can see that it, you know, adds this exclusion block to the palm part of being style.

D

Preserving means that it detects that this file actually used tab indentation, and so it indents it over to the right and the in the way that a human would if a human were inserting this and we take all these little building blocks right like find methods and change methods and find types and all these different things, and uh and we can build them into higher and higher order things.

D

So in this case we're doing like a whole testing framework migration- um and you see where you know the the combined effect of this is yeah- we're changing uh imports. uh You know automatically in a sort of like uh in a way that respects the import style of the original project.

D

I got trying to get back here that the uh able to strip modifiers off you know you know change annotations, uh insert lambda expressions, you know all sorts of types of different things um and that's kind of that's kind of where we're at so yeah head on over to rewrite we'd love, to see your feedback on that and and sort of what we'd like to see for security.

D

Researchers, I think, is a world where um you know you don't just disclose a vulnerability and make it known, but you also are in a place where you develop the remediation that helps fix the issue, and so that's the sort of like uh ecosystem that we're trying to build one where you provide help to those engineers that are affected by the vulnerability, as opposed to just you know, adding to a work pile which you know is kind of ever increasing.

C

Absolutely because uh you know, while it's very easy to point at a problem, uh uh you know giving them a solution, is uh much more palatable.

D

It is yeah yeah, so I'll turn it back over to you, jonathan thanks for your.

B

D

So this is um this: is, can you see my screen again.

A

Yep yes, coming up. This is the uh the diff of the change that um rewrite did as an example. um So you can see it's adding a new import, adding new code in the right place, changing the uh this new or generating this new method and then changing all the calls to that old, vulnerable method to use this new method- and you know again, syntax, preserving all that stuff and um uh the result of this was um uh for the jvm, so results of my polarquest generation.

A

um I haven't uh checked the newest numbers these were from uh when I one was no no 2020, it was like no november or december of 2020. um So for the jv ecosystem use of http, I generated 1 596 pull requests. As of november, there had been 422 merged and uh 147 closed. A lot of those closed ones are um uh were closed because um they were merged, but like the maintainer merged self cloak like merge themselves and then close them um uh and then for the j hipster one. It had been relatively recent.

A

So even uh it had, you know the first one I did it and it had been. You know, 11 months, the second one I time this up. I did it in september and then I was presenting this in november, and so there were 3 880, pull requests and and 56 merged, um so it hadn't been very long. um Also, a lot of the hipster ones were like one-off projects. I did get a couple of like you know.

A

I was basically going for like wide because I knew there was going to be some some legitimate projects in the noise, but it was really hard to identify the legitimate projects. A lot of the j hipster ones were one-off projects, though, like the people just generated it tested out, jhipster and stuff like that, um but I did you know get some of those like. You know, projects that were actually using it and that actually cared um so there's some lessons learned from bulk pull request.

A

Generation, um lesson learned number one sign off on all generated commits um this will significantly increase the likelihood that prs will be accepted, um because most people have a cla requirement, and this you can also just sign off and that you know that uh that solves a lot of those issues um have fun with it live stream it, um which is what I did. um Yes, so uh I live stream. When I do when I do pull floor, press generation, I live stream. It I invite people to come in.

A

I you know, I have fun with it like I, you know things break live. You know my computer overheats just yeah um create some sort of save state um because I my bot failed and so knowing, where it failed and then being able to pick up and not regenerate pull across that already been generated. That's a really good thing to have um uh yup. um There exists a max writes per second api limit on github.

A

That's like very hard to find um quote if you're, making a large number of post patch put or delete requests for a single user or client id wait at least one second between each request. This is the biggest rate limiting the late rate limiter that I have for generating pull requests, um yeah uh so yeah. I have I've run headlong into this and it's annoying, but you know what github that'll prevent spam.

A

So yeah, um one of the things that I learned that I did not know kernel task on mask mac exists as a software hack to the cpu overheating problem. uh It's kernel test using uh 500 of your cpu to prevent it from catching fire. Yes, I actually learned this uh running obs and my bulk full aircraft generation at the same time makes a hot cpu um don't fork. 1 000, uh 1 500 projects against your personal account. um uh That was my account. They fixed that um yeah.

A

So uh for, but, like my account, my if you went to jlight github.com j lights, you most of the time, I would get the angry unicorn. If this page is taking too long to load, um they have. They have fixed a lot of these things and my page now loads much quicker and I don't get the angry unicorn most of the time, but I learned not to fork things against my personal account because of this just.

C

Most of the time, huh yeah.

A

E

What's the recommended alternative by the way I.

A

Will get there live streaming and running this bot on the same machine will cause cpu overheating yup. I actually use an ice pack underneath my machine in order to keep my cpu cool um forking. Thousands of repos with hub will cause name collisions. This is the this is to answer your question so um because you're forking, thousands of repos, if you have hub, doesn't like if you hit the fork button in github, it'll automatically rename it. If you have name collisions when you use the hub cli command, it doesn't rename collisions.

A

um So how do you solve that problem solution? Rename the repository after creation? ah That adds another right. This requires a second right operation for every repository, so I didn't do that. What I did do was create 35 bulk security generation projects or organizations against github. So I have 35 organizations and I failed over subsequently to each organization to generate the fork against, um and so yes, this is the number of organizations that I have assigned to my account.

A

I have 35 organizations that were just the subsequent failover of this won't fork, because there's a name collision all right. Try! The next organization in line and so yeah um yup uh that answers your question. It's terrible, but it works. um One of the things that I learned is do not click enable all on dependable alerts or depend uh depend about security updates after you've done this to your account, because you will end up with a boatload of spam against your notification. Feed yeah, my notification fee died.

A

uh Don't do anything of this with your personal account um yeah. I will do it again on my personal account. I don't advise it but, like I, you know yeah. So um that's that's! That's my my uh my slide deck um on what is.

D

The alternative to.

A

Your personal account like create some some other account. Do it through it from a like, do it from an independent I I will probably do again do this again against my personal account, because I like the identification of like this, was me and also like I get the notifications and stuff like that, but, like um so github has also so hear me. um There's there's been um yeah, so um this this project's been actually run more than just those two times.

A

um uh Github actually used my bot to fix cve 2020 this one right here um uh that was found. uh uh It was a an array overflow in um a it was an r hostname overflow um that had a cert advisory, and so they generated 1885 pull requests using a github bot account using my my polar press generator that code's not in this repo, um but they you know they have utilized my bot in the past to also fix security. Vulnerabilities like this as well, so um you know github's also.

A

I mean I don't know how often they're gonna use it, but they haven't. You know taken advantage of this technology as well so um and each project that I have that I do this thing. I just throw another file in there and it's you know it's. The fix is specific, but the engine underneath it is generic to like how do you generate pull requests and, like you know here, add an add, a method for the hook for all right. This repo exists. What change do you need to make to it so yeah yep?

A

um So that is that is me any questions so yeah.

E

C

Was gonna say so so how much work? What went.

E

B

C

You know generating the the file uh that the engine uses each time, because I mean I'm wondering okay here: here's a brand new cv, the e that comes out tomorrow, for example, against something critical and open ssl. uh uh You know how would you go about addressing that.

A

So the actual work to um add another like so what I use for rules are, um I I've used uh lgm.com and they have a json output which they they hand me. I write the query they say hey. Can you give me all these results and they'll hand me this massive json file with here all of the repositories and the files that we found this vulnerability in right? So that gives me the repo name and the file right, the file impacted and potentially even the line, but I'm I I didn't deal with individual lines.

A

I just said: okay, this file is vulnerable, let's fix wherever it is in the file, because I might be forking it in the future. Slightly past, when the code 12 query was run against the repository and the repository might have changed um so the first step is just identifying the files. Now that I have the files from all of that, my engine takes all those files and then you have a generic method, which is here's, this file that exists change it and so, for you know, I guess it's from there jonathan.

A

How long did it take you to write the the uh the um the rewrite rule for the that j hipster thing.

D

It was quick for us and I mean it's gotten even easier since then. So I mean I don't know 20 30 minutes.

A

And then he threw uh awesome part of it was that, like I didn't, run it inside of this engine. I actually, he just said: here's an end he's a gcp, endpoint post, the file to this endpoint or post. The contents of this file to end point it'll respond with changed files, so I I just like shot it up to gcp and then he gave it back to me and it came in a fixed state and I just wrote it to the file system. So I didn't you know it kind.

D

A

Google cloud functions too. Basically, I mean not like broke but like.

D

Definitely saw the limitation of that too yeah.

A

Yeah, it's like you know, oh, I need like 50 of these instances because clearly the load is like through the roof and then it would crash and, like you know, not crash yeah, the main problem there.

D

Is that there's.

A

D

On the server was uh you know so that you know you just uh if the repo happens to fit in memory you're, okay, and if it doesn't you're, not okay, yeah yeah yeah. So I think we did it on a per file basis, which was pretty wild, but yeah yeah.

C

uh With the the rewrite efforts uh on that particular bug, uh uh did you end up breaking anybody.

A

I don't I mean I can't imagine.

D

A

No, I mean, I don't think that I mean again it's just it's not it's, not I'm not committing, so you kind of have to approve it as a pull request right. You know if I I'm not, um but I think that, like everything compiled, I didn't see anybody complaining about. Oh, this doesn't compile.

E

Yeah, but that that actually was more or less my question, you know, there's that phrase uh you know to errors. Human to automate errors requires a computer. uh You know how do you test this so that, before you start generating a million pull requests, you have a fairly good confidence that the whole thing is actually generating.

E

I mean I understand in special cases it might not work. That's true for humans, but I'd like most of those bull requests to be correct. How how could you gain confidence in that.

A

I tested against one or two repos and then I just like go I'll. Let it rip this spot jack. You know, but yeah it's it's um I I so I run the. I run the body of the pull request, like the message by xavier at github to make sure that, like I'm, not gonna get flagged for um like abuse right. So I've got a lot of contact inside of github to make sure that, like hey, I'm gonna. Do this right like don't like.

A

Don't don't shoot me down because, like I'm doing this in good faith right, and so they know ahead of time, their abuse team knows ahead of time that I'm doing this, um and I don't know how did you test the like the change? Actually, like I mean this was a. It was a very stamped.

D

Show you actually it's just easier to see um the like, there's a pretty elaborate testing harness for the actual rewrite recipes themselves. So here's like a unit test. Actually that demonstrates a change, the a test for change, method, name and you get to specify the recipe the before text. This is what's kind of wild, even in the ide. It's syntax highlights the little unit test that you have running here and then you can.

D

You can verify after text here and you can dump libraries onto this, and so it's actually pretty easy to build up a little testing suite. But just looks like before after code you.

E

Had a bunch of unit tests in along with the transformation itself.

D

Yeah yeah, this transformation was fairly straightforward. I think um the more complicated ones like when we do junit four to five or a certain. You know like uh junior search as a sergey or something like that, we'll just you know you know test it out. Just like you would any other main code.

C

uh Daniel you have a question.

G

Yeah, I was just gonna say that you could use the presence of continuous integration configuration in the repositories to go. You know two percent sample of the repositories that have got ci.

E

Submit the pr's and then watch.

G

The check data that comes back.

E

I like that, unfortunately,.

A

So I just had a conversation with the github team about this. They are, they just put a check in for github actions. That means that if you're the first time contributing to, if you have action or to a repository that uses github actions, a maintainer has to explicitly sign off on the first. um The first change because they've been dealing with so many issues.

D

A

um uh Crypto miners, so they've they've made even not auto running um for first-time contributors.

E

But but nevertheless it uh still would be, you know you the odds, go up if there's a problem. One of those repos is more likely to detect that yeah.

G

E

Send it to say the other 98 yeah.

G

I wouldn't be a one-time thing yeah, it would be a case of like okay. I think I've got something that I want to to do this for submit to a small percentage. Wait. You know perhaps several days for enough people to have clicked that button, that you can be confident that things are generally passing and that no one has said hey. This change is crazy back to you yeah and then go wide.

A

I had one so the big push, the one of the biggest pushbacks that I got against the the http downloaded dependencies maven, one that I got was. um uh I was not checking to see if the server that I was changing, the url to actually supported https um and people were, I mean that was one of the biggest things that I got pushed back on and my response to most of them were that's. I just fixed the code right.

A

That has a vulnerability, but you need to go and contact the maintainers that server, because the server clearly is not you know the server not supporting https is. Is your vulnerability, so talk to your main, like this is consider this the impetus to go talk to the maintainer of whatever dependency server you're using to say this. You know you need to be supporting https.

E

And you could have included that in the pull request itself. Where you, you know, oh look, that's yeah.

A

E

Work- and you know- hey you're- also going to need to fix your server.

A

Yeah, I also support, so I say that I support this in my in my. I have a very long long description of the pull request and I say if you want to be, if you want to opt out of uh pull both pull request generation like this drop, a dot, github dot, gh dash, robots.txt file into your repository and, like I won't generate, and so I don't have support for robots.txt.

A

In my repository I just checked to see if any repositories on github have that file. Yet before I run the bulk plug generation- and I have yet to see anybody drop that file in any of their repositories, so I'm gonna.

C

Go drop it in mine right now, yeah.

A

D

It yeah you're right yeah, one one thing that hasn't been said about both pr generation here, which is interesting, I think, is that, um unlike when you do this sort of manually, when you do an effort like this manually in open pr, it's like it could be an exhaustive amount of work to go actually open. It all up open all those pr's, and then you think how many people are gonna, actually action, these things in a reasonable period of time.

D

When you do bulk pr generation, you can basically just force push over the branch repeatedly, so you know you can keep. You can basically keep the pr current over.

A

D

You know as it moves forward and even and in some cases detect when the diff becomes null and just close, the pr yeah. So there is an advantage to you know bulk pr generation that doesn't exist. When you do things manually, you sort.

F

Of have to time.

D

F

I can see this actually being used by some organizations or um projects um just for doing non-security fixes as well. I'm thinking you know, zap, we updated um log for j. We've got 35 repos, it's a pain to be able to.

D

Generate a whole load of.

F

um Upgrades like this would be brilliant.

D

Yeah I mean we just talked to somebody this morning. They said that they had estimated 1497 days of engineering time to move from j unit four to five and and now it's it's five days.

D

um You know and that's like uh it's that's that's kind of the thesis right is it like I mean like maintainability is destiny and code. um It used to be optional. I think when I go back to like working for an insurance company 20 years ago, we had like a lot of on-prem software that like ran on websphere and could still look very legitimately, be running today.

D

But now, when you develop software, it's all integrated with third-party libraries with apis and so forth, and if you don't keep it current, it just ceases to function. So at this point, in 2021 code at rest ceases to function um and yeah. It's you know so now, you're on the hook for that maintenance. You know whether you want to or not.

A

Yeah one thing I think I I would like to do next with this is: I would love to integrate, I'm working on some automated disclosure infrastructure, and so I'm so. The vulnerabilities that I was looking at mostly were.

A

Like you know, I realized that jay hipster was like one-off projects, uh a lot of those, um but I'm looking at some more vulnerabilities that are kind of more widespread in like components that are not tests right but like are actually in core components of like the java, like code flow and I'd like to automate the process around. Like you, I I sent you this pull request.

A

You merge it um and then I respond. Thank you for merging.

A

This will automatically have a cv number assigned to it um unless you say stop like unless you comment like in all caps stop and so, um and so that would allow like the automated disclosure of these kinds of vulnerabilities um that have been automatically fixed to also have an automatic, cve issuance process, coupled on top of that, that's a bunch of infrastructure that I have not thought about how I build, because you need to do monitoring- and this is currently run as a one-off event.

A

But um it's one of those thoughts that I've been kicking around my hand is like how do we? How do I find this vulnerability fix it and then also like deal with like the potential thousand cvs that need to get generated from that vulnerability, um so yeah.

F

There are dangers there as well, because there's a pattern someone can search for then the bad guys can search for it as well- and you know if you're talking about software, that people have got deployed somewhere, then they're not going to actually update everything immediately and let's go okay. Let's look to see what these guys are doing. Oh they fix these things. How can we abuse them before they get? You know all the fixes roll out.

A

So one of the things that I'm thinking about in that area is um this is like real. It's really awkward, because uh again, a circular researcher but you're finding vulnerabilities this kind of scale. um One of the things that I am leaning towards potentially doing in the future is um tying this into github sponsors.

A

So if you, if, if you sponsor me on github sponsors, I'm more than willing to privately disclose things to you like ahead of time, I will also potentially disclose things privately to like the apache foundation and stuff like that, but um in general writ large, like if you know, unless somebody's sponsoring me specifically to to do private disclosure right, I will just do the bulk pull request generation, because I just don't have a time right, there's not enough time in a day and like yeah, but like I'm willing to do it.

A

If there's, if there's you know financial money in in in that interview, yeah in that interaction, so.

C

So a lot of this is tied to uh uh lgtm and codeql.

D

C

Are you know, came from semel which is now bought has been bought by github? uh Do you see any changes coming to that technology that you know from the github acquisition that might help or hinder what? What you've done.

A

I'm concerned about the direction of um github's integration of codeql. I'm sorry, I'm sorry gray, because gray's is a manager at github. So he's hearing me say this: I'm concerned about the direction of of um of of coach well being like conglomerated into github, because currently the best way for me to do my open source security research is um so in lgtm.com.

A

You can import each project that you want to run queries against one by one by one and in order like so in order to import the thousands of projects that I do to find vulnerabilities at scale. I have written a bunch of scripts that use the api for the user interface, which is on an internal api to bulk import like the entire apache organization, into my account, so that I can run queries against it. Right github who works at github like people github can run their queries against all projects but like for us.

A

We kind of have to hack it together and I'm seeing that lgtm.com is, is not um the direction that this coql thing is going in terms of the future of the product, and I see the stuff, like the code scanning results that github offers per repository where things are headed, um but that information and those code ql queries and that all that stuff is, is specific to the repository and is not in a place, because what lgtm does is they? They run? They compile the code, they save the database and then they have a.

A

They have a version of that database that they store on their servers. And then you can run your query against thousands of them um the direction that's moving towards github the I have not yet seen a story that replaces that workflow for security researchers for doing security, research as things migrate from lgtm, which seems to be less and less of a priority for future fixes and security, improvement, improvements and everything's moving towards towards moving to github.com.

A

So um I'm that's concerning, but I've been assured by a lot of people that it's a priority, but I don't there's no stories that have been communicated around how that's going to play out cool. I can call.

H

It a bit if you want cool, it's basically a github brought semolin in 2019 and what we bought was mainly a variant analysis tool, which is what jonathan is talking about when he talks about kind of hacking together the query himself: you do that when you're like looking for variants of sense of vulnerability, it wasn't really a static analysis uh or like a classic sas tool. It wasn't something that people ran in their ci sites did a little bit, but but not so much.

H

The first thing that github did with it is, take it and say like okay. Well, we want to take the goodness of this and have it as something that works out of the box as a more like standard static analysis tool. So a lot of investment went into the code scanning side, which is running on every commit every core request: servicing results there not as much went into integrating functionality around variance.

D

H

Into github, but that's kind of just the phasing thing we're reaching. I mean we're not reaching the end of the static analysis part, but, like we've done the big heavy lifting on getting a static analysis, integration into github, we built out the flow for other static analysis tools to submit code scanning as well.

H

The next big thing on that teams list is building out the query console from lgtm and building that into github and using the um databases that open source either generates because it's running static, analysis anyway or because they've been pre-run by github, making those available in a query console we're putting together those plans. Now it's not entirely clear how it's going to run. We know it's going to run on the info on the actions infrastructure under the hood to do the scans. It's not certain!

H

Yet whether or not we'll use code spaces and have it as like, a basically online vs code experience or we'll build a dedicated ui for it, but it is coming. It's just not been the priority over the like first 18 months of like semolin.

C

Great, but you know, github doesn't have infinite resources to get this done now.

H

Who would have thought it?

H

C

uh I totally understand uh thanks thanks for for speaking up about it uh cool, so this has been very interesting. Thank you, uh uh jonathan and jonathan. I really appreciate it uh uh jonathan schneider, if you uh feel up to it, please feel free to to come in and join these meetings in the future. We're trying to figure out uh the best ways of helping everybody and uh rewrite sounds like an interesting tool. So yeah.

A

C

A

Yeah, thank you gray, and everybody at github has like made this work possible to begin with, like you know like I, I'm just taking advantage of a bunch of apis and like pulling a bunch of things together but, like you know all this infrastructure, you know I'm standing on the shoulder of giants so, like you know, well we're just trying to help yeah other people do cool right. That's our job yeah and thank you jonathan for like just like hey. I have this project.

A

Can we do this together and you're like yeah, like totally so yeah like just awesome? Thank you.

D

So much for just like jumping.

A

On there and going for it, yep.

C

Always excellent, um so do folks have any opens or or anything else in the last few minutes or is everybody.

E

Like, first of all, I am intrigued by by this. I think we it would be wise for us to think about what I mean. What is important enough and widespread enough to approach this way. I proposed in the notes, a thing about aura. um We just don't. I think we run out of time, so I would propose picking that up next time so that they have a chance to talk as opposed to just hey. You got 60 seconds, say everything.

C

Yep definitely makes sense.

E

Are there any particular tasks? I mean you don't have to know right now, but uh what other things might be widespread enough and worth fixing this way? uh Jonathan, you may already have have some things in the queue or thinking about some things. uh Jonathan l.

A

I have um I have a thing um I tend to focus primarily on java stuff. I can do sibo plus, if somebody writes the rule for doing the c modifications jonathan. How does your stuff handle c and c plus plus currently, because I know there's a lot of stuff there? Don't yeah.

D

You really have to build.

A

Each language, yeah yeah everybody a lot of c languages vendor their stuff. That's where a lot of vulnerabilities tend to, like you know, propagate a lot more than the java system.

A

um So if somebody has, you know, basically, if somebody writes the rule and like has either like a code that I run some code that I run or like an endpoint that I can hit with code like as soon as that exists, I can run and like a list of projects that are impacted with those two two components. I can you know, run this and do another live stream and have a blast with it. Right, like you know, I mean yeah yeah, although.

E

You know what, if it's straight up vendored, you may not need the sophistication of rewrite. I mean seriously. Here's the line, a diff works. We're done. You may not need the sophistication to fix. uh You know you copied that code ten years ago and things have changed since then.

D

Sure one of the biggest I think I mean this- is at a different scale. I guess, but I mean I'm thinking about things like if you've heard of spring boot, like that project, one, the one dot x line was alive for seven years and then the then you know they've released 2x 2x lines going to live for two and a half three years, and then it's going to be two years and it's going to be.

D

You know one and a half years and like that's a pretty tough migration, you know so you know a lot of people say: oh it'll take us two months three months. You know we can't do that every year you know like, and if you, if you fail to do it, then you know the next time. There's a cv disclosed on you know spring boot, 1x have fun you're on your own right and that's those are the things I think that are that. Are you know you gotta you gotta move forward right. It's not optional anymore.

E

Yeah, although I do think that there it's a two-way street, I mean, if you're changing the apis.

D

E

Much as a library at certain point, I have to say, shame on you stop writing code. It's.

D

True, you don't belong in the.

E

Industry, it's true, thank you. Yeah you're, just creating a vulnerability, and you know you're going to do that.

A

Well kind of I also think that there's a valid I mean if, if, if, if it's enough of a problem, right like there needs to be financial incentives to say this, is you know this? Is you can't be doing this right? Like you know the reason that cobalt, like people keep updating cobalt is like, because you know like there is so much financial incentive behind don't break anything, but we want to run the latest version of cobalt right like that. That's the financial incentives that exist right. There is not.

A

There are fewer financial incentives to don't break apis because, like they're being driven crazy by like old decisions that are holding them back right, yeah.

D

I think I think you look to an organization, and I don't necessarily admire this, but you look like a google say they do the monorepo thing, and why do they do the monorepo thing they do the monorepo thing so that they can they can move. You know.

A

D

Can move forward when you make a change? You know it it propagates across the code base. You know that has consequences to the organization's. Culture leads to heavy code review culture which does slow things down a little bit, but it comes with the significant benefit of you know, keeping things moving forward.

D

I tend to look at the world as, like um one big, you know repository of source code and I think my view of like uh being able to automate source code transformation is that you know it's like it's like an eventually consistent monorepo. It's eventually consistent with with in this in the sense that people you know merge, basically to fix, and the world I want to see is- is basically one worldwide, eventually consistent monorepo.

D

um That would unshackle us from like. You know, whoops. I made this api decision five years ago, but you know I'm now faced with this sort of like impossible decision, whether to maintain it for another three years or you know, hurt my my user base or you know, that's so um it'd be nice to innovate, about that cost.

E

Right, I I think you make a fair point. I think the problem is right now today, uh the the word eventual has an incredibly long latency. Yes,.

D

It's long tail right. It's long.

E

Tail yeah right.

D

E

Guess I I'm kind of agreeing with you things we can do to make uh things easier and I do think there are sometimes ways you can make things easier in other ways. Yeah.

D

For sure, don't.

E

Necessarily necessitate that, but but yeah if we can make it easier to move forward, then great yeah.

C

Okay, well, we are out of time. uh Thank you. Everybody. uh A great discussion today really appreciate it.

B

Thank you all yeah. Thank you. Thanks.

F

E

F