OpenSSF OpenSSF Day Europe 2022, 28 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Developer Security Essentials - Liran Tal

Description

Developer Security Essentials - Liran Tal

---
Open source software is pervasive in data centers, consumer devices, and applications. Securing open source supply chains requires a combination of automated tooling, best practices, education, and collaboration.

Join the growing list of organizations supporting the advancement of securing open source technology and funding the development and adoption of OpenSSF initiatives. https://openssf.org/

A

All right, thank you, everybody for your patience. We are now joined by lyron Tao, from sneak going to give us an exciting conversation about node.js security. After that we're going to have Kate Stewart talk to us all about s-bombs and spdx. So whoever had that question earlier received that for Kate and then we're going to have an executive fireside chat, I'll kind of give you some ideas about how the foundation is having an impact on Enterprises and around the book. So with that, take it away later. Thank.

B

You thank you all right, it's cool, so this is going to be more about the foundation and communities. Aspect of you know: node.js JavaScript, but other in general. That openssf is uh is a very large contributor too so I figured I would open this uh with an interesting uh game to kind of like capture your uh your, uh your attention, I I, assume the uh the reference doesn't escape. A few people here uh who have been around the block, and so I will ask uh who wants to play a game?

B

Does anyone want to play a game? Okay, a bunch of people unexpected uh but sure go ahead. Come over! Let's play again! Yes, let's play a game.

B

A

B

Should read the instructions and play along.

A

ah You are a patch of the dog. Your mission is to avoid vulnerable package versions, collect superpowers along my journey. Oh man, I will absolutely press space. This.

B

Is a vulnerable package, you should do something about.

A

It can I restart, go ahead all right. One more time now, I see what's going on here.

A

B

Killed by a code injection in that's JS,.

A

B

One third: try: let's go.

A

B

A

B

Who can do better? Who thinks they can do better come on? Come.

A

On yeah come on.

A

Best live demo ever.

B

Go ahead, click restart with a yes and go ahead and give it a try. Oh.

A

Nothing happened here, nice, okay, this.

B

Is the vulnerability page, but you could move with the arrow keys move with Eric.

A

Who is usually helps can I try again. Yes,.

B

B

Okay, thank you, cross-site scripting in reactors Vijay. Thank you so much uh hila. Do you want to try as well? Okay, we have had a backup plan here as well. So let's, uh let's see what Tila had a bit of practice, but let's see how it works.

A

B

Wait I'm I'm nervous.

A

A

B

Okay, I see you're like on it. You are very determined to pass.

A

I'm very competitive, okay, okay,.

B

There we go okay, Jason to see us here as well. That's amazing! Thank you. So much.

B

Congrats, everyone, and to all of you, I, think it kind of shows us uh how we need best practices and instructions, and all of these experiences like how do I go around it and if you wanted to play also, we can do it at the break and you can try to like and see how much points that you score.

B

So uh my name is developer Advocate at sneak I enjoy building all of those like education and content material for developers trying to really help them. uh You know scale up and level up all security before joining sneaker was a developer myself, so kind of like I know. What's going on I understand, you know everyone's pains, uh like you know, everyone here and I'm, hoping that I can try and do you know my best through some of these working working activities and these work streams like the open ssf to really try and engage them.

B

So why we're here, I think you know we're all here kind of like for the same thing right like supply, chain security, open source security, trying to do the best things right like open, source, uh sigs and things like that. It's happening uh through different ecosystems and driven communities trying to really do it.

B

um You could see, for example, you know why why very much we're all here right, because those uh you know those kind of uh you know, Dentures, are really out of this academic research paper, noting that the average npm install is probably going to pull in an implicit trust for you for your developers to basically put that trust on, like 79 other dirt package, uh maintainers uh in 39 other maintainers in general, as well.

B

There's other reasons why we're here also because of the fact that about three uh 300 3, 000, maintainer email addresses have been kind of like associated with expired accounts and domains, and that means that this other research that was actually just in 2022 um released showed us how we can you know, abuse the ecosystem itself to an extent right, so perhaps maintainers even need best practices, not just developers and I'll touch upon this. You know further on um there we go so while we're here.

B

Most of most of this is about best practices and the open, ssf working group. For you know, packet manager best practices is one of them when I say one of them. That's essentially because there are a lot and I I.

B

Think I heard someone here when I was like doing the hallway track thing saying you know there are like so many working groups Etc, um it's it's so many that this one specifically isn't even on the in the web page you like have to go and specifically look it up, uh which I sometimes do when I try to do it, and so this is it and there's like a lot of effort going on into this by a bunch of really talented people from different.

B

uh You know work streams and companies really trying to make a difference here, and so one of these package manager, best practices and I want to like celebrate one of them specifically. First of all, the fact that we have been you know launching, yes punching version, one, it's not easy to launch with open source with all of the collaboration happening, and you know trying to reach consensus on what is the best practice that actually matters and what is not and so on. uh But we have launched one and I think that's pretty amazing.

B

That happened earlier uh in September earlier, probably like last week, first of September we have announced this there's, like amazing, amazing work, which I really want to take this stage, and just thank you know miles Jordan, Jeff, eres, Laurent, Randall, everyone who have contributed to the open source. You know pull request kind of new ideas. uh You know changes, modifications everything through that.

B

It's been amazing uh turn out by all of those people really trying to kick this off and I think this is also something that is going to Kickstart other work streams for other package manager, best practices which we'll touch upon as well. So what what's in there? What's? What are best practices for developers? Well, one of them is CI configuration and uh you know, makes sense right developers configure CI, they use it get action or something else. That's pretty cool and there's like a bunch of uh this is a small expert from the actual.

B

uh You know best practice itself and you could see. I've specifically highlighted the way that this community works right. We have actually been trying to reference. Other work, streams and other products that are adjacent to the open source to the package manager uh working group right uh We've, you know open open, ssf scorecard is one of them. There are others uh which we'll see in a sec, but this is important.

B

Dependency management.

B

Sounds very, very uh mundane right, not anymore. It's kind of how I feel before I go in to that npm install press return, key and hope that it works and doesn't introduce any malicious packages. Type of squatting attacks vulnerable packages whatever this is something that's really really needs a lot of attention by developers.

B

Okay, this is the supply chain security around this and everything, and so, when I speak to developers, they kind of feel the same right when I show this, you know this meme on on Tech references, I I get everyone's the same kind of like nutting experience like they.

B

They understand. This is an issue, and so we touch upon in the best practices part. For you know a bunch of stuff I will not go into details of every one of them, but, like type of squatting has been a real problem, so you know tell people you know be careful what you're typing you know. Why is this an issue and so on uh Json stream versus Json stream? Does anyone know what this refers to different things?

B

Interestingly, at one point in time for the npm registry, it was case sensitive, so you could publish packages of potentially the same name but they're like okay sensitive. So if you do npm install for any of those, you would actually be pulling in two different packages. Of course, at this point both of them still exist, because if you remove one of them, you break the internet exactly so many dependents on one of them. So it's a problem at this point, but of course we can at least not allow you to publish more.

B

uh You know case insensitive issue, so that's not an issue anymore. At this point, um it's a good question. Actually I haven't checked, which one is is the one that's deprecated or not so I would need to check in on this nice, but we can go to depths.dev.

B

You see how to connect to that one two, uh you know celebrating, first of all, another open, not an open source project, but a really good resource for open source developers, which Google has made available for us to be able to actually like learn more about packages and I, really like the idea that it connects with a bunch of other things. Like you know, the scorecards are already inherently integrated in in here and actually shows you, for example, for a specific repository. Also, what's uh you know?

B

What's the reasoning for a specific score that you get so it's not like the overall actually have a breakdown of this. This is a really nice way of just gaining information, a bit more into this and I. Think one other aspect of gaining this information is, you know, imagine the following issue. Let's say you want to remediate one of those Json stream uh package dependency issues, whether that's like malicious or you know, as a security station, you want to see who's dependent on it.

B

Doing that today is a bit problematic, but because devstative Dev and a bunch of other resources, but this one it makes it really easy. You could actually have a reverse dependency lookup or what we call dependence. So, if I look at node IPC, which is a reference to a later slide, but you know some people here might know about it from prior uh incidents, you could see, for example, who's dependent on node, IPC, directly or indirectly, and through that alert them, for example, to upgrade move to a different package and so on and so on.

B

So these resources are. This is like one of the reasons why this kind of resources are really important to be part of our best practices.

B

Others are around release and private packages, so, for example, for release you might, you know, need to make developers aware that they need to use 2fa. 2Fa is really important for a whole lot of time with the npm ecosystem. It was a bit hard to enable to fa because there were no other way to automate package releases. Okay.

B

So if you enable the two effect authentication as a maintainer- and you wanted your CI to complete test, publish the package, that would be a problem like it would literally prompt you for 2fa on the CI, which of course doesn't make sense. But since then, since a lot of time actually for now, since for years, we have uh automation tokens which you can enable to fa, but then have automation tokens. So we could, you know, discuss the trade-offs of you know whether this is you know, better security, or you know slightly.

B

You know a bit a bit of a trade-off there or not, but still to a phase like super important, and not a lot of developers have enabled that. So this is part of the release process. We also touch about private packages, which has been an issue specifically with regards to dependency, confusion, attacks which started with uh you know, last year's report from Alex Bergeron.

B

If we talk about it in his blog- uh and you know the the bug Bounty program and the whole thing of the disclosure that they went out- and you know shared about, what's the dependency confusion, why does it happen and so on?

B

Of course, the actual problem to this date is that we're still seeing many many incidents relating to dependency, confusion every you know Vlog that I see on like a second Monday is going to be probably about dependency, confusion targeting packages from like Azure and from like different other other kind of like bigger Enterprises that you're trying to get into. So this is an important aspect and we've specifically went and provided uh instructions on what it is and how to overcome it.

B

The other reason why I really like what happened with the npm best practices in this in this uh in this working group is for me the following, like if you read this and again, this is like a small excerpt of the whole thing: it's if you're, not from like the npm or the JavaScript ecosystem. It really helps educating you how this system works. Okay, how this community Works? What's expected? I recently, like last month or so dubbed into first time into Ruby development to you know, extend a security research.

B

I was working on and I needed to like understand how Ruby manages dependencies and sub-dependencies and manifests, and how does it all of this works. Okay. This was a bit hard, even with the documentation existing to an extent where, if something like this were apparent, it would really help me because it tells you, for example, no. The Manifest trial does not list transitive dependencies.

B

This is like a really good primer for anyone, devops security practitioners and so on who's, like not tied directly day to day to this specific language ecosystem, to really get a glimpse of like what to expect- and this is what I really like about it. We even uh touched on like specific type of projects you could have like standalone, realize libraries, application projects I wholeheartedly recommend going through this.

B

It's not a long document, but it's super helpful and practical I will attach next on two items which are not entirely part of the best practices, and so I don't want to like represent any other people's opinion within the working group, because people have different opinions and I have my own, which is the following. One is preventative measures, okay, so so far you know you would expect best practice to give you ideas of how to prevent an issue.

B

An example of that is the following: another research, another academic research I, will share here again also very early from this year. Called weak links in the npm ecosystems showed us one specific data point: that's like super important to understand about what's happening with malicious packages in the ecosystem. Npm, as a package manager provides any of the maintainers on a tree of dependencies, the ability to go ahead and specify post install instructions. So when you do npm install XYZ I have to idea what that package, don't actually install XYZ.

B

But if you do that, Dutch package can run arbitrary codes as the user you're running this on. Okay, whatever it wants, and so if it can do that, that's a really bad source, for example, malicious packages. When, when you install a package, they actually read your environment variables and other you know other things that they can do very hardly, and so this data point is like super important in the sense of hey who uses and install scripts right.

B

Is this like 98 of the ecosystem, or is this two percent of the of the ecosystem and based on that, we could potentially figure out if we want to tell no, if we want to have the 80 20 kind of of win here to say we want to protect. You know everyone else on, potentially the trade-off that if you install something that needs to to legitimately install you know a browser or something else as part of the install process, it might break like you need to understand what you're doing so.

B

This is an interesting data point, and so just to emphasize this again here is a real package name, one three, three seven qq-js uploaded in 2019 uh install when instance when it's installed it collects actually your environment variables, which of course no one saves API keys in your environment variables right. Why would we do that? We all work very, very well with security and and KMS and whatever, um but it took two weeks to discover this and take it down and who knows how many people have installed it right?

B

It does it very easily if I'm able to trick you into installing this package. All it does, is you know post install hook. It has this submits this to the registry, and then once it's there, you know and install it has the instruction to run this command. Whatever is in that code is, is going to get executed.

B

This is how it works, and so we should really be able to like do things like npm install minus minus, ignore script right and I didn't add it either, because we haven't really added that as part of the best practice, but I think that's kind of like a question that is, uh is getting uh sometimes uh re-upped to see if we can actually add it or not, but that's one of it. uh The other case is reactive measures. Let's talk about one example that happened in the ecosystem.

B

Earlier this year, node APC, it's a very popular downloaded by millions uh kind of npm package. uh You know got sabotaged in a way. This is part of the of the code that was malicious that had got added into it.

B

um You can see, there's like a buffer from interesting, that's base64. We can decode. It looks like there's like Gill location, API, key hard-coded in the npm package. Okay, more uh based, 64, encoded strings, let's decode them dot, slash, dot, dot, slash dot, slash, looks like it's doing something on the file system. Well, it's actually deleting files on the ecosystem. Every time that you run it and it has a timer, this is a package that has been sabotaged by its maintainer, for you know reasons we won't get into here now.

B

But essentially, this is the impact of what this has when you, you know, have those packages now, one of those one of those versions. For example that was released, 916, okay, it was released within a day and within a day it also got more than 25 000 downloads. Okay, this is almost 30k within a day. That's how fast, JavaScript and CI and everything else kind of like gets.

B

The latest right goes to the bleeding edge and gets that so one of those reactive measures that I've been kind of referring to is telling the ecosystem, because when this happened, a lot of developers and teams were kind of like rushing well, which version do I have I, have the malicious version or not, and if I have it, what happens because I I can just always upgrade what happens if it's a transitive dependency now I am dependent on maintainers on the tree to get the latest for the whole trick.

B

So this has been kind of like an issue. That's been going on for for a while, and there has there. There were a few ways to mitigate it. Recently. Override is one of them so, for example, telling developers and teaching them about the ability to actually prevent and specifically kind of, like kind of like hard code, a specific, uh a pin down dependency version of this dependency that you actually want to get that you know is safe from this issue is important, and developers might not even understand like that.

B

This is possible to just add to their package Json, and this is a way to to like override everything else and you specify even for transitive dependencies, which version you want to get best. Practices for other ecosystems and I'll run through a few of those, because I believe a bunch of those ideas in the next slides are going to be stuff. That I have ideas for and I'm happy to, like. You know, collaborate with others, but let's say so: we've got npm. Hopefully we can have people being interested in other ecosystems and help us.

B

You know bring these to other ecosystems. Have this document there what about Community tooling? So there are a lot of developers out there who are part of the community part of like openjs foundation and stuff. Like that they're building security tools, should we have a place, that's kind of like I'm thinking of like a radar or, like the you know, like the cncf landscape thing kind of like drawing a map of you know.

B

These are the tools that you could be using because they might not be able to know even what is exit, what exists for them to be able to use that Howard best practice as awareness we created an amazing document. This is kind of like how developers see it the way that I feel right. What do we want best practice says: when do we want it now? When will we use them? Never because like who has time and like a best practice and I forget about it and whatever? Okay?

B

So how do we make them like know about it like this is like literally the you know, the awareness problem is is really like the you know, the 50 of the pr of the issue here like when developers know about an issue and I'm not talking about the hype of like hey, like I, don't know like uh some magazine says it or you know, blog post.

B

That says, there's been a malicious package or something- and uh we should uh you know, ignore it I'm just saying we really need to tell them how to use it, and so some ideas again here that have like scribbled, you know, should we have maybe Hands-On workshops should we have demo based on those repositories that kind of like take on those um best practices actually take them through. You know, step by step. You know hey. This is a challenge then solved it with one of those best practices. Here's another challenge in malicious package.

B

How do you solve it? We should have like a learning path and give them badges or something like that like readily or Linux Foundation, get them something to actually be proud of, and show off again ideas happy to hear everything else that you guys will have what about collaboration with other foundations. For example, you know I'm, you know part of like this open, uh JS Foundation, which used to be the note says node.js Foundation before and we are maybe duplicating work right, because we also had like this internal discussion. Do we need best practices?

B

Should we threat model, node.js watch it exactly. Should we do, and you can see like there's, there are a lot of really smart people are working in some silos, like collaboration, would be super important to make sure that everyone are on board and part of the story, and this resonates further on there's like a lot of issues that we've worked on, and maybe the open ssf can actually learn from other practices.

B

Like I was part of this node Foundation a few years back when we were doing hacker one program to basically get all the report for the npm ecosystems, millions of packages reported to us and we as volunteer to try as them looked at. What's going on, contacted maintainers, reproduced it worked to fix, issued a CV Etc seems like really really really hard work. That was ended up getting a kind of like a uh pushed over to like sneak, because that was the company.

B

That's able to like um actually work on this full time, and so this was, for example, kind of like let go from the openjs foundation. Perhaps there are other things that we should work on together.

B

How about the maintainers, because right now to an extent the best practice is mostly touch developers right, I'm, saying mostly kind of like this, because there's like about uh hosting private packages and releasing private packages, which is somewhat maybe maintainers but there's more things. I will give you some ideas recently, for example, what counts as a vulnerability. If I'm a maintainer and someone reaches out to me and says, hey, you know, I found a vulnerability. Well, is it a vulnerability or not?

B

I have I'm saying that ex that uh giving that example from like recent recent recent experience, I was actually going to disclose the vulnerability that I found and then I was. uh You know. Having second thought, you know, Consulting with some people, I opened a DM with some people that I trust from this. uh You know the same Community I was like hey here's, a gist of what I found. Let's talk about it, do you feel that it's a vulnerability or is it not right?

B

Maybe we need to structurize this whole thing so that we kind of like are able to also help uh security researchers right, not just the maintenance himself to connect everyone together and do that responsible disclosure for maintainers like how do you handle reports? What should you expect notifying users? Should you release security, patches, minor version or a major version? What's the difference? Maybe maintainers don't understand this fully and how it impacts everyone to just get up to this up to date on there.

B

So there's like a bunch of things here that would be really interesting for maintainers to actually understand and be part of this, so uh this is all of it. Please get involved, there's a bunch and a ton of stuff that we could really do really looking forward to working with everyone. So thank you so much foreign.