GitHub Security Lab, 22 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hauke Lübbers - A short history of package dependency compromises

Description

Hauke Lübbers is a Software Engineering Manager at CSIS Security Group who thinks constantly about all of the software packages that everyone depends on. He presents a history of malicious open source software (OSS) packages and package compromises, diving into two specific cases for key lessons.
Get involved with the GitHub Security Lab here: https://securitylab.github.com/get-involved

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
Google+: http://google.com/+github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

uh So welcome everyone to my quick lightning talk um for me. This story started in summer of 2019. It was a very different time and I had some ideas on how to potentially detect some malicious packages in public, open source repositories and to test this theory. I started to collecting some data about past incidents about prior research and then eventually also about countermeasures taken by the open source package repositories, and so I ended up with this huge document and I thought it would be really a shame to let that go to waste.

A

So I uh decided to write a blog post about it that turned out to be way longer than expected and a really slow writer. So I only ended up covering the time frame between 2011, which is when I found the first incident and 2017, because otherwise it would get too long and release early and release. Often, if you're wondering who's talking. That's me: I'm hauke lubas, I'm a software engineering manager at csi security group.

A

We do remote forensics, phishing, detection and takedown and in general threat intelligence mainly for financial institutions and larger companies, and I'm also a volunteer lifeboat helm, and I do all of that here in beautiful copenhagen, denmark, um starting with some definitions. What is package dependency compromise? How do I get it? How do I avoid it?

A

um Basically, we all are relying on a lot of packages that we download from open source package repositories like, for example, mpn or pi, pi or rubygems, for example, for the ruby ecosystem and we're managing these dependencies through application or language package managers that help us keep track, keep them updated and yeah stuff, like that, also installing them, of course, um but if you download a malicious package, you might potentially be compromised through that, and that is then a subset of the larger software supply chain compromise, um which could also uh be done through other means, for example, by compromising a compiler or something some other thing that we use to in the end produce the products that we develop.

A

What I won't cover is stuff like operating system or otherwise system package managers like, for example, apt on linux yeah.

A

So it's only about application package managers looking at the timeline starting from 2011 to 2017, with a color coded in in proof of concepts, there were actually a lot of proof of concepts or incidents that we thought initially might be actually malicious attacks, but then later turned out to be proof of concepts by some researchers, some countermeasures and lots of research and publications and in the end we actually see in 2017 the two first malicious incidents that I could find, starting with the first one here in 2011 that was researched by benjamin lee smith.

A

um He had this great series of talks called hacking with gems that he would uh present at ruby conferences.

A

He would most often write packages that, as part of the setup, the setup hooks send a little ping to a server controlled by him, with, for example, the username and the hostname of the machine, and because he would present this at conferences. You would also print the names of packages that you would write. That actually did show the schedule of the current conference and he would print that on business cards and distribute them and tables at the conference.

A

So people would install them because they're all ruby enthusiasts, of course, and and then he would collect their host names and stuff and, of course, later had a much more fun talk um when uh when he would present and could look the people into the eye and say um watch out what you actually install, but because the the ruby community is very focused on ruby on rails. Of course like. If you have, if you write some web application in ruby, it's probably ruby on rails.

A

So that actually also gives the opportunity to directly integrate in that framework and, for example, extract post information that is sent by http post, like, for example, logging information and send that or forward it also to a malicious server, just because that probably covers most ruby web applications um yeah, but uh looking at actually the first counter measure. That was one of the first countermeasures that I could find that was employed by the community.

A

um That was just the registration of the uh the name requestist on uh on the python package index by a developer. Who noticed? Oh, I actually mistyped that, um thank god, nothing was was downloaded here or executed, and then he just registered this name, and I checked in january of 2020, and it was still downloaded 1200 times just in that month and still four uh repositories on github are listing it as a dependency.

A

Even though, when you install it, it just warns you that you shouldn't do this and points you to the actual library uh going further to the pizza party. That was the first proof of concept of a warm or of a package that would spread itself um that happened in 2015 and was a poc by chris contellini.

A

He just published it on github and didn't actually publish the package to npm because that might be too risky.

A

But what this was would do is it would check if the current user is also the developer or has publishing rights to another npm package, and it would then first open a kind of annoying youtube video in the in the foreground and, in the background, create minor versions of the packages that the victim had publishing access to, and it would then publish these minor versions, minor versions, because then many people might upgrade automatically and this these new versions would then include exactly the same spreading code, also in their setup hooks and yeah.

A

So the circle would continue.

A

Another incident that I put in here that isn't directly related to security, but maybe you already uh saw there the infamous left pad incident on npm um for reasons that we won't go into a developer, decided to delete 273 packages that he developed and maintained, and that impacted a lot of very popular packages, either directly or indirectly through other dependencies that they had that then, in turn dependent on left path, which was just a single line.

A

Single line dependency.

A

um Thankfully- and here we also see how the community works together, this package name was re-registered by a good actor within 10 minutes, because they noticed that their continuous integration processes would fail, and I mean they failed over all over the world, so they re-registered that so that nobody malicious could take that one over and full functionality was restored within two and a half hours, and I included this because it shows also how fragile these ecosystems can be and yeah that was in march of 2016.

A

going further. Also, um these are actually incidents that I want to go into a bit more detail on, because I have my own theories about them. This was fade. Zero, a security researcher from china, and he noticed when he tried to just execute some python code that it failed with this warning: import error, no module named smb.smb, you just want to create a summer connection and then notice. Oh, this smb thing doesn't seem to be a standard library. I have to install it. So what is the first thing you do?

A

Pip install smb, maybe, but it turns out that that actually isn't the name of the package that was wanted on on the python package index, but that was called pi smb instead of smb.

A

So the takeaway here is that the name of the package on a package manager doesn't necessarily have anything to do with the name of the modules that that package then exports.

A

Yes, so after he noticed that he started uploading, 23 typos, quoting packages of the same type, so packages that either imitated other packages that might be expected to be there um like, smb or proxy and stuff like that, and he generated uh more than 2 000 downloads of many of these within 24 hours in the setup hook of these packages, he he put some code that would send the username hostname ip, of course, some other information actually to a public github repository and would create issues there for each of these, like infected victims, which of course, isn't great from a privacy perspective and after some discussion he also deleted these these packages again and then wrote a really nice blog post about it.

A

That, unfortunately, is in mandarin, but it's it really goes into depth on a lot of these things and also shows other options that a user could have to be confused about something and how you could exploit that.

A

And the interesting thing is that one and a half months later we had the cross and incident on npm, which uh is on a different platform. It's no longer pi pi, but npm, but it had some striking similarities to the to the research proof of concepts that fate zero made. That led me to believe that this one was probably heavily inspired by that blog post.

A

um Some of the similarities are, for example, if you take a look at the names that fate zero used there on the left side and the type of squatting names that these malicious actors used. um You see, there's a lot of overlap, and especially uh tk inter is a weird, um a weird package to impersonate on npm, because that is very much a python library to to interface with the toolkit gui library.

A

So it doesn't really make any sense to uh to try to impersonate that on npm and yeah. That is just one more thing. That leads me to believe that that is heavily inspired.

A

um The this cross and incident was then discovered by a twitter user or a person who then informed the author of the actual cross and library, which is named cross-env on npm and yeah. He informed him and then quickly, the malicious packages were taken down.

A

These packages were actually malicious because they would send the environment variables to to a server that is also connected with chinese cybercrime and, of course, in the environment variables there might be tokens or other secrets that should not see the the light of day yeah. So that was the first actually malicious incident that I could find, and here, of course, we need collaboration between between languages or between repositories between the borders to to fight this.

A

Some short summary, uh the vectors that I saw for package imitation is mainly typo squatting, um so yeah writing something slightly wrong, but also a lot of standard library squatting. Sometimes you can register the standard libraries that are part of a language. You can also register them on the open source package repositories, or you at least could in the past. I think that's fixed everywhere now then package takeovers, which would be, for example, because a developer is compromised or a token is published somewhere they're.

A

Of course, multi-factor authentication helps, but it's not perfect, as we see in in the financial industry too, um and then of course scanning for publicized tokens, but also simple stuff, like always sending an email notification, if you or if a package that you are the owner of, um gets a new version in most cases, you might just be annoyed and be like yeah. I I I know that I just published a new version, but when you get that email and you didn't publish a new version, then you know that something is wrong.

A

Yeah then we have the vectors of re-registration of a popular package, name after it was deleted by its owner, um can fix that with cooldown down timers and stuff, like that or manual review, but also simple, simply taking over maintainership by package by just offering that- and that is something that didn't fit on my timeline because it's after 2017, but it happened that a bad actor just went to uh an open source maintainer.

A

These are stressed people, it's a pretty thankless job, sometimes so, if somebody offers help, they gladly accept that, maybe if they can show some nice prs and yeah, that of course, is hard to defend against.

A

They are the only thing that helps is really to share indicators of compromise and identify the infrastructure used by bad actors so that they cannot at least reuse that and make their life a bit harder.

A

In summary, attacks are happening. Collaboration across language boundaries is needed. Why the? Why aren't there more attacks? I ask myself. That might be a very developer-centric question, because for me it's it's very obvious to how to exploit that and stuff like that, but for advanced actors. It might be too much publicity because it's you, you have the audit trail.

A

Basically, as soon as you're discovered, everyone can see what you did and for other, maybe a bit less advanced actors, there's still other methods that still work like fishing or, where um maybe buying a browser plug-in stuff like that. um That still might be easier than than this.

A

In the end, I would close with a quote from sam boyer from his post, so you want to write a package manager, and that is package management, is that is at least as much about people what they know, what they do and what they can reasonably be responsible for, as it is about source code and other things a computer can figure out, and that is also very true for the security aspect of it.

A

So, thanks for tuning in and I'll be in the chat room for questions.