Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Securing Shopify's Software Supply Chain - Shane Lawrence, Shopify

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Securing Shopify's Software Supply Chain - Shane Lawrence, Staff Infrastructure Security Engineer, Shopify

Recent compromises of Codecov and Solar Winds have put a spotlight on software supply chain attacks, but this focus has led to new innovations for solving an old problem. In this talk, we'll discuss lessons that Shopify has learned in protecting millions of businesses and demonstrate these techniques using open source software. We'll look at how traditional defensive techniques can be applied in the cloud, how voucher and grafeas implementations can give you control over the software that runs in your clusters, and how the SLSA framework can guide you toward establishing trust in your software. We'll also look at how Falco can be used to detect malicious behaviour or indicators that your supply chain has been compromised. Attendees can expect to learn how to apply specific techniques for mitigating supply chain attacks.

A

Good morning really excited to be here, thank you all for coming, I'm going to talk a little bit about securing software supply chains and I'm not the only person who's talking about this lately. It seems to be an incredibly popular topic.

A

If any of you were here for the security conference on monday, there were several talks about it, and it's been coming up in the news more and more there's a lot of buzz around this, and it's not surprising, given how many attacks we've seen in the wild so rather than try to go over all the things that you can see in another talk or look up yourself, I'm just going to give you a little bit of information very briefly about some of the challenges that my colleagues at shopify have run into with this.

A

Some of the lessons that we've learned and why I think this is so important, of course, I'm most familiar with shopify's use case, but we all share operating systems, languages, libraries and here in particular, a container orchestrator.

A

So this is not a zero-sum game. This is a collaborative effort. I think the best time for all of this focus would have been 20 years ago before we built most of our infrastructure on a tangled web of hidden dependencies and unmeasured risk, but the second best time is right.

A

Now now it's been a long time since I got to come to kubecon in person, but when I was preparing for this talk, I was thinking back to my very first kubecon in 2017, and I knew that if I wanted to give a talk on container images, I was going to need a lot of images of containers.

A

So I'd like to take the simple task of how I would find those images and use it to show a software engineering process. It's not the waterfall with it. Don't worry so, just like anything else that I might try to do when solving a problem. I'm going to go out, I'm going to look to see if somebody has already solved this problem for me and offered the solution for free, so I plug in my ub key. I fire up my work laptop and I get ready to yolo click.

A

Some risky links, let's filter for only top quality, safe containers, and if this isn't quite what we need, we can scroll down. Oh this looks better: it's a magic container image. Toolbar, let's give it a whirl okay, we can just download it there's not even a charge for it. It's got great reviews, there's no way that those could be fake.

A

Now, if it says it needs admin privileges, that's probably fine right. It makes sense. It needs to administer all of those sweet container images. So what do you think? Would you install this? Would you download this? Have we gone too far? Already it looks suspicious right it has these impossible claims. It has click bait. It's got a very questionable presentation.

A

I think this looks suspicious because it looks like other things. I've seen before that turned out to be malicious. So, let's rule out the toolbar. Let's forget about the container images example for a little bit. Let's look at other sources. We might use to try to get software.

A

Here's one npm install fix error. This might look safe, this particular package fixer. It was a trojan. It was caught by researchers before it did a lot of damage, but you definitely wouldn't have wanted to download that when it was first reported, so what about npm in general? No offense, 10 pm, it's better! You know it's not the toolbar, it's almost always safe, but we can't really be sure that your trusted npm package hasn't been replaced by something. That's just a little less obviously grotesque than my toolbar.

A

How about this one? This is a really common pattern, a lot of applications that I like recommend that you install or run them this way. I call this pattern curl pipe bash. You pull whatever might be at that website right now and immediately pass it to a command interpreter.

A

Now this might seem like it's. Okay, you might have even checked the contents of that site, but it could change at any time and it's even possible if the attacker controls the server to change what's being returned depending on whether or not it's being processed by a shell. So this information would allow the attacker to come back. If you use this particular path, the infamous code cov bash uploader, they would exfiltrate all of your environment variables. They could attack your systems or even impersonate you using the secrets that were stored in your build environment.

A

If we ignore the bash uploader and just look at npm, you can see that there are still problems with this. If there is an upstream compromise, then there's no way for you to change it as soon as you run this code on your production systems, you're compromised, so let's move on to go get and it actually doesn't need to be go get you could be using import at the top of your go file or, if you're, using another language. This could be an include statement or something else altogether.

A

The point is you're, taking somebody else's code and you're running it in your systems. There's a reason that this is the last one, because we're going from the absolute worst, the toolbar, to something that's just potentially kind of a little bit bad. Now. This particular path is a common misspelling of a more popular package during the time that this attack was happening.

A

If you had used this, it would have exfiltrated a bunch of your data to somebody else, but even if you pinned to a hash you've entered every dependency, you could have been compromised at the full first pull or any subsequent upgrade.

A

So we have a broader problem of trust in our dependencies, whatever the methods we're using here when we import these, they become part of our software supply chain. We said the toolbar was suspicious because it looked like other things, we'd seen before that were malicious, but all of these look like they could be suspicious to me.

A

So, let's zoom out a little bit, we hear a lot about supply chains. Nowadays, I know it's why my washing machine was three months late and there are no new cars to drive off of a lot in my city, but when a manufacturing supply chain is disrupted, the manufacturer can't just go back and decide to eliminate the supply chain and build every single part in a single factory somewhere they need raw materials and specialized labor. That may not exist in every location.

A

It might be tempting for us to simply reject outside dependencies and say that we're going to build all of our software ourselves, but it's not economical and it's not even safer. For instance, if you try to create your own cryptographic, algorithms- and you are not a cryptographer you're, almost certainly going to make the problem worse as well. Communities like ours are built on sharing ideas and learning from each other supply chains are so fragile that disruptions even happen by accident. uh There is an infinite infamous case called left pad. You may have heard of it.

A

There's no reason to believe the developer actually wanted to hurt anyone or cause real problems. They just wanted to take their ball and go home. They deleted among some other packages, 17 lines of code in a package called left pad. It caused 200 failures per minute until npm restored it and broke many many packages that had no idea. They were even using this when supply chains are so fragile. It's no wonder that they're a target for attack a few years ago.

A

How many of you thought that you would be defending against nation state attacks by hackers supported by governments?

A

Not too many, but that's what solarwinds was. It's widely believed to have had state-sponsored backing and it was one of the most prolific attacks we've ever seen. It even got attention from the white house and got an executive order, encouraging us all to do better and in 2020 how many of you really cared about the minutia of exactly how your log messages get passed from your application to the console.

A

I actually really did, but that's just me most people don't most ceos and csos don't, but somebody was paying close enough attention to this to realize that there was a remote code, execution, vulnerability and a ubiquitous dependency. That did exactly that. So many businesses were impacted that the cost of mitigation and resolution is completely incalculable.

A

You might even think this is a global vulnerability, but the ingenuity helicopter on mars was also running log4j. So this is actually now an interplanetary problem.

A

Congratulations: there are significant risks in the complex systems that power nearly every facet of our lives, from our finances to our utilities, to how we shop and play the risk is an inherent result of using other people's software and it can be managed, but our ability to do so today is limited by a poor understanding of how to measure the aggregate risk that comes from this system based on its individual components. In fact, right now it can be challenging to even identify what those components are when it comes to physical supply chains.

A

Manufacturers have experience evaluating this risk. We can't just stop using third-party code, but we can, and we must learn how to use it safely.

A

So, even if you're, not in security, if you're a developer or engineer or sysadmin, if you work with vendors or if you support any of those people, then you can have an influence too on securing supply chains.

A

So I'm going to very quickly go through a few of the things that shopify is working on in this space. I hope that it'll start a conversation. You need to start with secure development. You need to know who your developers are. You need to be able to trust that the people who are saying they're committing code really are who they say they are, and you need to have effective reviews to make sure that they're doing what they say.

A

They will need to automate your builds and something that was new for us is codifying the build steps we're using in toto it's been very successful for us and it gives you the ability to document a recipe of what is going to be built and how it's going to be built now. Entota won't stop you from doing something. That's wildly foolish even, but it will tell you that the build happened, exactly how you said it would you can do vulnerability scans?

A

There's lots out there gripe's a good one, we're using trivi I'd love to tell you exactly how to do. Reproducible builds, but I don't know please somebody tell me the illustrious s-bomb her software bill of materials. uh I also wanted to tell you which the best format is, but we can't figure it out. We're using cyclone dx right now. Spdx is also very popular, and the only reason we really favored cyclone is because we're already using trivi for vulnerability scanning. It also generates s-bombs and it generates them more easily in cyclone.

A

So we're using that cosine, though this is working really well, you need a way of guaranteeing your sbom wax. Seals are so three centuries ago, but cosine from the open, ssf, standardizes signatures and attestations. It's essentially solved the problem of storage and signing for us. uh This is misleading at the end here, there's not just two challenges. Those are just two that I want to highlight. One is that non-packaged software doesn't get detected by most of these generators. We're working on an internal project called hansel that may solve this.

A

I hope I can tell you more about that next time and we're still trying to come up with processes for vulnerability. Triage we've got all of this information in a new source that we didn't have before and with some kind of other events and detections.

A

We have years of experience coming up with heuristics allow listing things deciding which things need attention from a human and which can just be ignored every time by the computer, so coming up with ways of identifying deep vulnerabilities and dependencies of dependencies that we may not even be affected by is a problem for us that we're hoping to solve with better processes.

A

Now, once we have trust in our build system, we need to have a secure deployment, so we're using our cloud providers binary authorization to enforce this, but we also have an open source project called voucher.

A

It already gives attestations for build provenance and ci checks, and what we're adding now is an extension of this to attest that there is an s-bom attached from the builder and then, when we pair this with binary authorization, we can see that the voucher attestation is there and refuse to deploy anything into our production environment if it does not have an s bomb.

A

We also take guidance from salsa, that is the supply chain levels for software artifacts. I think the c is silent and invisible in that acronym. Now there are a lot of other people here in this room who know more than I do about this. So talk to them. But our opinion of this is that it's an excellent guideline, especially the first three steps. The fourth one seems to be a little bit vague and hand wavy right now, so I'd love to get more clarification about that or hear opinions on what others are doing there.

A

So we have a great tool kit, but it doesn't matter if you have that tool kit, if you can't reason about the risk, if you look at my magic toolbar, you can see right away that it's probably suspicious. So if you see something like that, you might want to avoid it or at least have a trusty application. Security team review it. But there are other things you can do with less obvious examples. You can scan or review the code if it's open source.

A

If it's not, you can use static analysis or take a look at the guarantees from whoever is offering it preferably. There will be some sort of a third-party audit of the code and you can use that depending on whether you trust the organization you can decide based on that organization's past experience and there's probably a lot more of it than the developer of the application you're using.

A

You can also look at the reputation and the response to previous incidents of the maintainers. uh There was a really good talk yesterday about single maintainer dependencies and that's a real problem, not just for security, but for many other things as well so soon. I hope that we can start to look at the components that are going into the supply chain as well. We can expect more from our suppliers.

A

We can start to ask for receipts, ask for an s bomb and ask what your software is made of, and some of the most important things to do for securing your software supply chain have nothing to do with software supply chain security. It's just things like updating your software, keeping it up to date, using least privilege.

A

So if you see something that is massively overprivileged or a tool bar that is asking for administrator privileges, that's probably a sign that something is amiss and once you know exactly what your build environment is supposed to be doing, it makes it much easier to use a threat detection engine. We use falco, and you can write rules for this. That will allow you to detect any anomalous behavior if your build system starts reaching out to third-party servers that it's never connected to before.

A

That may be a sign that something is happening, and you need someone testing this stuff, so you can use external pen testers or a security bug bounty program.

A

That's what we're doing I'd like to talk about what you can do. You can talk to a sig or a tag or a working group about this, I'm not affiliated with any of those organizations. I just think that they're doing good work. You can find good information there. You can find open source tools, open a pr or an issue or just use them and give feedback, follow best practices, cloud providers and cis benchmarks.

A

The cncf released a great white paper that has some good information about this, and the last few years, they've really underscored the grave concerns that we have and the supply chains we all depend on.

A

Cyber attacks are wreaking havoc on our digital supply chains and more and more attackers are aware of these lucrative targets and they're working harder than ever to breach them. So please ask about your organization's supply chain. Think about the software that you're using and where it comes from. When you pull a dependency, we have a chance to improve this process and to protect the physical and virtual infrastructure that depends on it. So please help to make our supply chain secure for everyone.

A