Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Open Source Intrusion Detection for Containers at Shopify - Shane Lawrence & Kris Nóva

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Open Source Intrusion Detection for Containers at Shopify - Shane Lawrence, Senior Security Infrastructure Engineer, Shopify & Kris Nóva, Chief Open Source Advocate, Sysdig

Even well protected workloads may be compromised by 0-days and platform vulnerabilities. Observability is essential for detecting and stopping an attack before infrastructure and information is compromised. Shopify uses open source Falco, a CNCF incubating project, to track syscalls at the kernel level and reveal them to a Kubernetes-aware process in userspace. That uses predefined rules to decide which events to log. Additional tooling filters and aggregates logs, and generates alerts when suspicious activity is detected. In this talk, Shane will describe how Shopify first deployed Falco in 2018 and continues to use it to monitor critical systems, including those that process payment card information. He will share tips and tricks for getting the most out of Falco, areas for improvement, and use cases for detecting compromise or data exfiltration when all else fails.

https://sched.co/ZfCl

A

Hello and welcome uh welcome to our story, uh shane- and I are really pumped about this keynote and we're going to be telling a really cool story about shopify's, pci compliant environment and an open source tool called falco and we're going to be sharing our socially distant and remote hugs with everyone at kubecon who's. Here right now.

A

I wish we could be there in person, and I know shane wishes the same thing but we're doing the best we can, and we just wanted to tell everybody to keep up the good work so anyway, shane's going to get started and he's going to talk about their environment and then we're going to talk about falco and how it works, and then we're going to look at what it takes to run falco in production, uh so check it out in shane. You want to get started.

B

Thanks so much for joining shopify is a commerce platform, we're trying to make commerce better for everyone, and we were able to process 61 billion dollars in value for our merchants last year, surveying 300 million online customers, but we really level the playing field by giving small businesses the chance to have the same reliable platform that much larger companies would have and it's their brand that splashed all over the website, not ours.

B

So that's what our customers see, but we see that this is made possible by 10 000 kubernetes services running in 50 clusters to serve 170 000 requests per second and using kubernetes makes this scalable, which is really important things like black friday, cyber monday or flash sales, where we'll see a shop go from their normal volume, increase exponentially in a matter of minutes and then come back down again.

B

So we want to make sure that we can scale up to meet that demand, but also return to that baseline once the flash sale or the event is over we're doing this at a large scale.

B

But it's not a website that you visit in order to buy things and when you do buy things from our merchants, it doesn't come in a shopify box. We power over a million online storefronts and those storefronts, like the cncf merchandise shop, for example, have their own look and feel all we do is enable their brand. So there's a good chance that you've bought something from a shopify store without even realizing it.

B

uh I'm on the infrastructure, security team at shopify, and so it's my job, along with a few other teams to make sure that all of this happens securely. The storefronts are our merchant's livelihoods and when they accept payments, it's not our information, we're protecting. It's your.

A

B

So it's important to us and I'm sure it's important to you- that we get this right now. The focus of this talk is on falco, which is an open source security tool that we've been using since 2018, but there's no panacea and if there was I'd, probably be here trying to sell you that, instead of talking about this open source thing, so remember that it's not enough to just have detection.

B

It's really important to make sure that you have things like defense in depth, where we're doing everything we can to ensure that no one is able to get into these systems in the first place, then do external audits to make sure that we're proving to the rest of the world that we really are doing all of the things that we say we are, and we also have something that doesn't actually affect our payment card processing.

B

But it is really useful for us. It's security, bug, bounty on hacker one and it allows hundreds of people all over the world to attempt to hack shopify and if they are able to find some sort of a vulnerability in our infrastructure. They use test data, so no real credit cards or personal information is at risk. They're, not using real stores, it's their own test shop, but if they find an issue with our platform, they can report it to us we'll pay them a cash bounty and then we'll release that report.

B

So the rest of the world knows what we found, what we did about it and ensures that they can trust us.

B

When I started at shopify in 2017, uh they had just recently decided that trying to figure out which servers go in which racks is not really a core competency that supports their actual mission. Now I actually like hardware, I, like configuring operating systems, I like giving all the computers cute little pet names, but our use case doesn't justify that and we're not a cloud provider. So we don't really have a use for data centers managed by ourselves anymore.

B

Instead, we've got kubernetes, it lets us scale our workloads to meet the demand that we see and we use cloud providers to provide the resources that we need on demand. This is pretty great, but it brings a lot of new challenges.

B

So the first temptation, I think, when a security team is helping a company move their infrastructure to the cloud is to then just move our data center network security model to the cloud. Please don't do that um those old traditional data centers follow this really old school design. Where there's like a wall, the bad scary world is outside that wall and the stuff inside is soft and squishy, and it's assumed to be trustworthy. You can.

A

Only gain access.

B

To the network, presumably by going through these security things like a firewall or an intrusion detection system and if a malicious actor tries to get in, then this ids lights up, but the web server can connect directly to the storage server without any issue.

B

Now, there's some underlying assumptions here that are just plain wrong and this is a gross simplification, but this is roughly the model that data centers use, there's a clear boundary between what's coming from your own systems and what's coming from the outside, so then you move to the cloud now: there's no home net! There's no save 10.008 that you can just use to designate all of your stuff and say that this is safe and everything else you need to protect against.

B

Even if you use private clusters on their own virtual networks, there are still globally distributed services that are inside the network, and you don't want to page somebody every time your cluster is authorized against any group of services or drop data. Every time an app in your google cloud tries to connect to an s3 at a new address.

B

We need to remove those assumptions about a trust boundary which means using proxies and application level authentication so that we're trusting the user not based on where they are in the world or on a network diagram, but some other metric of trust. Now we also can't just go by a security box and drop it in everything's in the cloud. Where would we put it? We also need monitoring, that's aware of kubernetes infrastructure so that we can define expected behavior based on the container cluster pod and namespace again, not where it is in the world.

B

So this would be hard enough if we were doing that on a platform that processes 61 billion dollars in a year, and we knew that at the beginning. But we didn't know that at the beginning, in fact, when we started, we knew that the the previous year we did about 15 billion dollars, and so, while we're building this, it doubled roughly every 18 months. We can't just drop an ids in normalize it for a week and then switch to an ips mode, because the environment is changing too rapidly and effectively analyzing.

B

These events with the rapid growth and frequent changes means that some rule that was a little bit noisy at the beginning would be completely unmanageable. In a year we found an ally in kubernetes when we needed a platform that could scale with us, but now we needed security tooling. That could keep up with it so again the most important piece preventing anything bad from happening. Well, when kubernetes 1.7 was released. This was just before I started at shopify, and there was a lot of stuff that we take for granted.

B

Now that just didn't exist, then there was abac. Only there wasn't r back, so it was really difficult to define appropriate access control and it was really challenging. So for the most part, people aired on the side of over privileging. There are also things like dashboards being enabled by default and those had admin until 1.7.

B

So there were also missing cloud audit logs, which meant that there was no monitoring unless you were pulling those from the kate's master. It was difficult to secure this. There was no way that we could run our pci environment on this at the time, but so much has improved since then. So if these are familiar to you, if those are still a problem and you're running a production web app in a privileged container on case 1.7, then please get up now. Call your friendly local cloud provider and ask them why?

B

Because things have improved a lot since then now we also need to prove as we're trying to do this once we've solved those problems, we need to be able to show that, like yeah, we've actually done this, you don't need to take our word for it now, if you're familiar with pci, don't worry, I'm not going to spend the rest of my talk reading out all of my own made-up pci controls because well that's the auditor's job, but what the auditor.

A

B

Is to see something that's tried and tested that they know is safe and while we're growing and moving and building, we need to introduce confidence with new tooling that it's just as secure as what came before now. It's easy to complain about regulation when it slows you down or makes things harder, but it's there for a reason. It's how we really prove that we're protecting this.

B

So a whole bunch of stuff has gotten better: we've gotten rid of old apis and unused features, they're disabled by default in a lot of cases, there's a metadata proxy so that you can check headers to prevent ssrf and, if you're, using gk workload, identity, for instance. Then this solves a lot of the problems of metadata access in the first place.

B

There's also our back now, and this makes it much easier to implement least privilege, and I think, one of the most important pieces that we have is network policies. It's really nice to break things into services, but they don't help. If your web front end micro service lets, attackers communicate directly to your database on the back end.

B

So we need to make sure that all of these best practices are being followed and that relies on this, like giant fleet of in-house scripts and cron jobs that scan all of our clusters and make sure we're actually doing these things.

B

So we've got the configuration piece, we kind of know what to do. Now. You can read blogs and watch talks and you'll learn all of the best practices. Most cloud providers will even give you a guide or list of things to follow to help make sure that it's secure, even if we assume that we follow all of those perfectly there's still software supply chain issues to consider. Maybe the upstream vendor that you think you're downloading from has been compromised and now that workload that you think is safe and trusted is not.

B

uh Okay, assume you've even figured that out and you've gotten it perfectly there's still going to be unmitigated vulnerabilities. There are zero days when heartbleed came out. Much of the internet was vulnerable to it. When spectrum meltdown came out, it broke a lot of the assumptions that we had before that that shared compute was safe and, more recently, the local host boundary bypass, exploits assumptions about safety within the node. So, no matter how good a job we do on the left, there's always going to be a problem.

B

There's always going to be the issue that prevention is behind and when we're protecting critical services, we need effective monitoring.

B

So that is what brought us to falco and chris is going to tell you a little bit about what falco is and how it does what it does.

A

Thanks shane, so here at systig, we are deeply aware of all of the concerns that shane expressed and our business is building the secure devops to help with everything from pci compliance to prevention and using falco with detective, tooling and shane did a fantastic job at describing the evolution of cloud native over the past few years, and as we evaluate that, we realize that there is one constant which is change being able to detect anomalies such that you can then craft policy to prevent them.

A

Moving forward is a fundamental part of what we're doing here in cloud native, which is why cisde created the falco project in 2016. We created the falco project in 2018. We donated it to the cncf in 2019, falco became an incubating project, and now in 2020, falco is just awesome. So let's talk about what is falco and how it works.

A

So falco processes system calls at runtime and what that means is every application is going to need to communicate with the kernel and applications communicate with the kernel. Using these things called system calls, and we believe that a system call is the fundamental source of truth for what's happening on a system whether or not that system is behaving in the way you expected it or not.

A

So we parse those system calls at runtime and we read them and we see what arguments were passed to them and we rebuild the state of your system in memory at runtime.

A

We then enrich that with more information, information we're getting from kubernetes information, we're getting from the container runtime layer and we build this hybrid object of what's truly happening on your system. We then assert that object against what's running in your security policy configuration so falco has a concept of rules, and these express concerns about privilege, escalation, cve security policies, known vulnerabilities, and you build a set of rules that you then assert against. What's happening at your system.

A

If one of those rules are violated, we trigger an alert, and you can take that and you can plummet up to whatever the rest of your infrastructure looks like so, let's move forward and let's talk about enabling prevention, so, like I mentioned change, is a fundamental part of cloud native and part of infrastructure in a part of deploying software.

A

So sometimes unexpected things happen in the world. I don't think shane and I were ever expecting to be doing a remote keynote, but we have to respond to that in an accurate way. What falco does is it enables us- and it gives us a set of eyes when we otherwise wouldn't have this such that we can respond and we can take action to create new policy moving forward. So falco is a utility for uncovering future security policy.

A

Again, this is using tools that enable things like cve response exploits response, and we like to think of all of this as your cloud native surveillance system.

A

So let's look at how falco does what it does in the cloud native stack, so in the top left, we have your application. This is where you and your team are going to be spending most of your time. Writing business logic. Now, at some point, your application is going to need to communicate with the underlying kernel. Falco runs on the linux kernel and we understand that your application is going to have to go through many moving pieces before it reaches the kernel.

A

Kubernetes container run time, various layers of your your application and as the complexity of cloud native grows, there's one universal truth, which is everything is ultimately going to need to communicate with the kernel, which is how falco approaches security. So we have a set of drivers, one of which uses ebpf, which is how shane is running a gke, and these drivers allow us to intercept and understand what's happening at the kernel.

A

We bring that up into user space and we run that as a daemon, and we are continually asserting that against our set of rules at runtime, if one of those rules is violated, we trigger an alert. So, let's take us a journey through the path of your application, executing commands against the kernel.

A

So, as your application does things, those are going to be translated into system calls that's going to have to traverse through kubernetes and then ultimately, through the container runtime and then way down to the very bottom. At the kernel level, falco intercepts those using either the ebpf probe or a kernel module, and in some cases we can do that directly from user space.

A

Using the system p trace capability in hard environments like amazon ecs in fargate, we bring those up into user space and we assert those against rules and again, if something is violated, we trigger an alert. Let's see what these alerts can look like.

A

We process all types of information coming from the container mail level, the cluster context, even the kubernetes audit screen, we can pull information out of p-trace and we can even get information out of ebpf in the linux kernel and we put all that together into one hybrid, enriched object that you and your team can use and respond to in meaningful ways to draft your preventative security policy moving forward.

A

So, let's understand what it's like to build with falco, so here at falco, we focus on three main parts of building cloud-native applications in cloud-native infrastructure with falco access assert in action. We believe that we're going to have to access parts of your system, but we're going to want to do that in a safe way. So we need to parse. These system calls using read only techniques like ebpf.

A

We also want to be able to access kubernetes and the api servers so that we can build that container context so that it can be used for things like pci compliance.

A

We bring that in and when you assert that against a state of rules using the state of the system that we've created in memory and then we take action which we allow you to dynamically respond so using grpc and several of our software sdks. You can connect this up to parts of your application and use it as a building block for you and your team.

A

So let's look at what it takes to run falco in production shane. Can you give us a day in the life of running falco in your environment.

B

Yeah, so it's great that we have this kubernetes awareness and we can monitor every sys call, but that's kind of useless if we don't have rules that make use of that information. So we have things like the coin mining rule, which I think ships by default with falco and it's pretty straightforward. It detects any outgoing traffic to a known coin mining website and and that's really sort of straightforward.

B

We know for sure that we don't want any of our workloads at shopify connecting out to coinhive. So we can take a rule like that and we can change it up a little bit and then we can use that to instead of just detecting coin mining, detect any unexpected data egress. So if something happens to a sensitive workload- and we don't expect that to be connecting to some arbitrary place on the internet, then we can generate an alert now.

B

There's also things uh you hear like sort of the boogeyman of the vulnerability world, there's always remote code execution, and these are rare.

A

B

If it does happen, you want to be able to see that it is happening and see what was done. So you can use macros like spawned process to see what an attacker who compromised a container was doing to run that you can also use this as sort of your trip wires. So you can say you know these tools are very commonly used by an attacker. These might be used for network reconnaissance, so you would generate an alert when somebody tries to install or run any of those inside your container there's a whole bunch more.

B

You can see the velcro website for some more examples, but one in particular that I'm interested in is the metadata server abuse and there is a rule that you can use to monitor outgoing traffic from a container to the metadata server. This has been exploited quite a bit. There are lots of talks on it and in fact I gave one at kubecon in 2018 about a security researcher who had taken advantage of a and misconfiguration, and they were able to get some secrets in this second-tier, screenshot environment, that we had now.

B

If we had been running falco in that tier 2 environment at the time, then it would have been possible to detect this unexpected activity going to the metadata service and in fact, if we had the rule above then we would have seen using falco sidekick moving it along to slack something come up just like this, and this alert would tell us exactly which container it was running, what the ip addresses were and exactly what command the attacker had run now.

B

This gives us a lot of flexibility and it's quite powerful, but it doesn't mean that it does all the work for us. So um I wanted to show you in a live demo, how we can do all of this, but unfortunately that just didn't work out, I'm not in holland right now able to show this to you. So I have another talk called intro to falco and I'll do a bunch of recorded demos.

B

If you want to find out how to do this and please check that out, otherwise these are sort of the the main takeaways. uh The first thing is, you need to make sure that you're staying on top of noisy rules, because people are going to get alert fatigue people are going to be drained if you're constantly generating these alerts, and then either people are going to burn.

A

B

Or they're going to stop paying such close attention to them. So if you have false positives, you need to make sure that you're going in all of this flexibility doesn't mean anything if you don't use it to tell falco what is normal in your environment and allow it to tell you when something is abnormal in your environment.

B

Now, when you do get an alert, this is entirely non-technical. This is very much a human problem, so if you're woken up at three o'clock in the morning by a page, you want to know, is this severe enough that I need to get out of bed or can I deal with it in the morning and you need to know if it is severe enough that you need to do something about it? What do I do right now so having a playbook?

B

There is really important, and if you haven't touched whatever service is alerting you in quite some time, then you might not remember all of the intricacies of this product or this workload and so make it very clear make it so that someone who hasn't seen it in a while just woke up can do something about this also stay on top of your infrastructure, make sure falco's running everywhere and stay on top of upgrades, because it's actively being developed and they're making a lot of improvements all the time.

B

So again, the first priority has to be prevention. You can't just run falco and not worry about all of those other things that we've learned over the last few years for securing your clusters. But you know we have learned a lot over that time. Prevention is always going to improve over time as long as we're working on it. Fortunately, it's never guaranteed we're always going to find more. So you need detection. You need to allow human reasoning when something abnormal is detected in your cluster.

B

You need a person to be able to look at that and say this is abnormal and it's bad or this is abnormal, but it's okay and I'm going to adjust my rules accordingly.

B

Falco gives us the ability to have manual intervention. If something happens, you can go in. You can mitigate that risk. In some other way you can disable a workload and when you detect something like this, if it wasn't something that you were aware of before then detecting it with something like falco means, you have the chance to prevent it in the future. Based on your observation of what you saw, so thank you so much again, shopify and systig are both hiring. If you're interested, please check out the links here, there's a lot more information.