GitHub Security Lab, 22 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jonathan Leitschuh - Evicting HTTP from the JVM ecosystem supply chain

Description

Jonathan Leitschuh is a software engineer and security researcher for Gradle Inc.. Best known for the July 2019 Zoom 0-Day Vulnerability, he has also championed an industry-wide initiative to formally decommission the support of HTTP in favor of HTTPS, by major artifact servers in the JVM ecosystem. It's as part of this initiative that he ended up generating hundreds of pull requests in hundreds of open source projects.
Get involved with the GitHub Security Lab here: https://securitylab.github.com/get-involved

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
Google+: http://google.com/+github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

Hi so um yeah, so this talk is about um evicting htp from the jvm ecosystem uh supply chain, um also known as how I ended up generating 1 1596 pull requests to fix an ecosystem, system-wide security, vulnerability um hi. So uh my name is jonathan laichu, I'm a security, software, security, engineer and security researcher, I'm currently at gradle inc, um and you can find me on twitter at uh jlychu and also on github um same way.

A

uh So one of the questions that I get asked a lot is like how like why do you end up doing the research you're doing um ivd? I get distracted, uh yeah very much highway to hey look a squirrel, so um this is kind of how this project fell out too. um So, let's start with, why is https important in the um in supply chain right?

A

So um if you don't have https in place, if you're not using an sl secure connection to download your artifacts, you really have no guarantee that you're actually getting the artifact that you are asking for, um and uh um so there's a long history here of um of all of our supply chains in a lot of our ecosystems being built on http based technology and then realizing.

A

Oh hey: um we need to upgrade https uh and support https, um and uh so, if you're, if you are um downloading these artifacts over http, somebody can sit in the middle and and inject their own code into your into your artifact supply chain, um and this uh was actually um originally um disclosed uh by uh max speedsman um with a project called uh delente.

A

uh I have a hard time saying this word um dileton and what he disclosed was that um there was uh originally um maven central, which is the biggest artifact server used to supply artifacts uh to the java ecosystem, um only offered artifacts over http, and they offered um each gps if you spit spent uh a ten.

A

If you offered a ten uh ten dollar uh donation to the apache foundation- um and he published this article called how to take over the computer of any java or closure or scala developer, which called out sonar type for this practice, and because of this, um and as part of this max built a proof concept called uh dilettante.

A

That is an http proxy that injects jvm by code into jars downloaded over ssl, um and so this is actually his proof of concept where he injected code into jota time. And then, when you run a simple like hello world code, it actually pops a um uh a swing. Ui component that says you didn't use ssl to serve your jars um through the because of this proxy and his article because of the blowback um forced sonotype to offer https uh by default, and this was only back in 2014..

A

um So uh I was um looking at my code at some point uh in my company. I was working out of the time and I saw this. I was like there's a security vulnerability here. I have no idea how I got this here um and I ended up tracking this down and I ended up looking to see where I had ended up with this little bit of code and it's because I copied and pasted this url from the uh official source code of the project that I was relying upon.

A

So this is ktor is a kotlin based web server framework um and I had copied their url out of their source code into my build because it's just easier to copy and paste urls uh than trying to figure out exactly what it is, and this got checked into source control, and I was like huh I wonder where else I can find this um throughout the course of my research, some of the other. So this is in a gradle build file. um Here are some other places.

A

You can find this vulnerability in gradle build files, um so, in the build script logic build scripts uh are scoped to the entire project. um Repositories are used just for just for your code. That you're so so build script is, is is for um uh the code that is used. It's like your dev dependencies and npm there. These are dependencies that you resolve um to build your software and gradle repositories.

A

Block is used um here for um the dependencies that you use in your like production code and tests, and then this logic, where you see upload archives, is where you end up um declaring how you're uploading your dependency or your artifacts. When you publish when you publish things so, if you're publishing over http you've also, like generally leaked your credentials over a plain text, connection as well in maven, this is the same sort of logic that you'll see if it's insecure, um a repositories block, because I'm even as xml based, so that this is.

A

Where you'll see this sort of thing. um Let me just kill my.

A

Notifications, um so uh when I started looking for this, um I was okay, who else is vulnerable to this um and what I ended up finding throughout the course of this research. Just looking for these two kind of examples of code was, this vulnerability was very widespread.

A

It impacted a lot of very popular projects, and so here are some examples. The eclipse foundation gradle was vulnerable red hat the kotlin project spring jenkins, um the groovy project, the apache software foundation, elasticsearch jeff jetbrains. um I found these same. These vulnerabilities all these projects code. um I also found it in oracle software. The national security agency projects linkedin stripe, um just the list goes on. uh Portswigger also had this vulnerability in one of their projects, and uh I ended up doing some research on like. Where does this like?

A

Where what what repositories were these like? What dependency repositories were these vulnerabilities, mostly appearing from so I did? Github search is fuzzy at best, um so you don't get this. These numbers are not 100 inaccurate, but I did a search for these urls in xml in these various gradle and maven files, and the results that I got back were um when I was doing this research approximately, like 40 000.

A

You know you can see the numbers here um of of how many uses of these urls I saw in maven builds and how many uses these of these urls I saw in gradle builds, um and so this was clearly a very widespread issue across across github and open source projects, and so I started thinking okay like how do we fix this problem right? So it's a it's a very large wide scale problem.

A

I ended up publishing an article um in uh uh june of last year. uh I don't know, that's actually accurate. um I published an article called um want to uh want to take over the java ecosystem. All you need is a man in the middle, and this is published on june 10th of 2019, um and this was part of my.

A

It was like the kickoff of like disclosing to the you know the public, how widespread this true vulnerability was- um and I reached out at this point to um sonar type, which is the company that runs maven central, which was the first artifact host in the java ecosystem to open to have offer free, open, uh artifact hosting service for the ecosystem, and um they came back with this interesting result.

A

After a month of the research, they said, 25 of sonotype maven central downloads are still using http, um and this was back um when I disclosed my article.

A

um So you know a significant percentage of the jvm ecosystem was vulnerable to this supply chain, mana middle attack, um and so, as part of this understanding of like how big the scope of this problem was. I said: okay, let's fix this like like. Let's, and so I pioneered a uh initiative uh talking to a bunch of the artifact hosts in the java ecosystem, saying, let's fix this by killing support for http for all of you and so um on january. 15 2020.

A

um All of these different artifact hosts, uh collectively on on or around january 15th 2020 stepped in together and said, we're going to totally drop support for http um and only support https. So, instead of redirect it's not even a redirect, because a redirect from http to https actually hides the vulnerability, um it still exists because you can as an attacker, you can keep the connection downgraded.

A

um So all of these artifact hosts agreed and followed through with um just completely dropping support for http. And yes, there was a lot of broken stock software and a lot of comments on stack overflow. um Thankfully, the world is not on fire. I I do remember having a couple of issues reading some stories about some things that had broken because of this, um but on the whole you know it seems like the you know, industry has moved forward and and succeeded in yeah um so enter codeql.

A

um So the github security lab project was a really really cool um addition to uh this. This problem, because it meant that um with github rolling this out, we could. um I could write a query um that would find this vulnerability in maven pom files, and I was really cool.

A

It was really awesome to to find out that they were actually parsing these xml files as part of the codeql project, and so I was able to write this very simple query that finds uh the repository block or the snapshot snapshot, repository block or the plugin repository block in these xml files that are the palm files and look for http or ftp usage.

A

Very simple query and alert on that, and for this very simple query as a part of the all for one one for all: uh github security lab bug monty program.

A

um When I submitted this query to them, um they were kind enough to offer me a 2, 300 bounty for this, um including the documentation and all of that, um and so once I had this query, um I asked some of this uh this the team at um uh github- and I said: can you run this query against all of the projects that you have results for, and can you give me that data back and kindly enough? They did um so what I was able to do with that?

A

Sorry, I'm catching up with my slides on my notes um is so from this. um They were able to this. The this results are now generated and presented in the lgtm ui. You can go and look at all the alerts that are generated, but I needed this cohesively into a single data set um and I figured okay.

A

If I have all of the results of all of the projects that lgtm has indexed, um I can take that information and I can generate pull requests to fix this vulnerability, um and so I you know I figured you know it's open source kill vulnerability, you know, kill these vulnerabilities fire, uh the other part of it that was kind of fun was um you know I did a little bit math and I figured for at the time the bugs player vulnerability uh offered for, like at four vulnerabilities, about two thousand five hundred dollars.

A

I did the math and we got seventeen one thousand seven hundred repositories of this vulnerability do math, potentially no, that's probably not gonna happen, but you know I figured I'd. You know, give it a shot. So um so how did I end up doing this? So um I combined um the github hub cli um with the pi github um and some async I o, and a lot of bouncing off the rate lifting rate limiting api um and also I live streamed it. um And so my the the project that I used is this project.

A

I don't have it any of it in the slides, but it's uh called bulk uh security, pull request generator um and it's under my username jaylichu, and I used it to basically go through and check out every single one of these uh projects that were vulnerable. I pulled the code down to my local machine.

A

I patched it using a regular expression. I found out very quickly that you can't use xml parsers to fix xml code for this number of files, because the problem with most xml parsers is that when you write the file back out, they all reformat it into whatever the format of the xml file.

A

Formatter is, or whatever the xml file format printer chooses to use and that would have created massive diffs across every single pull press that I wanted to create, and I only wanted to change the specific lines that were vulnerable, and so um uh what I did was I I just wrote a regular expression that went and found that um that one line that was using http instead of https and just changed it to an s and changed all those files and um committed it uh and then created a fork and pushed it and ran into the git right right, limiting api, and I think this took like eight hours in the first pass in another eight hours, and so it's like 16 hours of cpu time for my one computer, because I kept hitting that rate limiting api um and at the end it was complete.

A

uh I generated 1596, not 197, I say 197 because I actually had one of my own example projects in there that I used as a demo. So you know it was actually one or fewer than what I put there um uh and currently 179 of those pull requests had had been merged at the time. um If you look yeah rip my github notification feed after that, um because I did this under my under my uh personal account- I didn't create a secondary bot account for this.

A

It's actually all bound to my github account um and also rip my uh repositories list, because now I have a ton of forks that I don't need, um but yeah. So currently, this is a snapshot that I just took before this talk. There's still 1055 open, pull requests um and 504 closed, um so that means about. 96 of them have been renamed um to be something that and I'm guessing that those additional have been closed.

A

So we're almost at uh 600 pull requests having been merged or closed um for this fix um so yeah, and um thanks to like the github security lab team, um they were real. They were kind enough to actually offer me a 4 000 bounty for this because of uh going above and beyond what they even expected for the scope of uh the um uh the the pro or the program that I was submitting to um so yes, that's that's the story of uh my generating all these pull requests. um uh Yeah.

A

Are there any questions that have come up in the feed that I can answer quickly? um It doesn't seem like I have there's any there um yeah. So I think that's me.

A

B

That was amazing and I think yes, people will hang out in the in the twitch start to to uh to talk with you, I'm pretty sure.

A

B

It's amazing: we really hope to see more stories like this right uh of actually fixing open source security at scale. This is really what we, what we're looking for right so.

A

And you guys yeah, you guys at github reuse the the bulks pr generator that I used to fix a different vulnerability. What was the other vulnerability that you guys used to fix with this? Well, it was some c plus c vulnerability. That was.

B

Yeah, uh I will try to find it back and uh put it on the chat. I I remember.

A

B

We we went to blog post about this.

A

It was very cool to see that that github had taken my code now black blob, it's really cool to see the github taken the code that I'd written and then used it again for something else. So I have a plan. One of my plans is to use this also for gradle builds. The problem that currently exists is there's no way with code. Ql to query. Gradle builds to find these exact vulnerability locations.

A

um So uh if I can find this data in some other way, it's very much my plan in the future to do this same sort of work for gradle builds so yeah.

B