Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Hello From the Other Side: Dispatches From a Kubernetes Attacker - Ian Coldwater

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Keynote: Hello From the Other Side: Dispatches From a Kubernetes Attacker - Ian Coldwater, Lead Platform Security Engineer, Heroku

Attackers have user stories too. Are you designing with them in mind? As an attacker, Ian Coldwater would like to help you understand these users and their stories. What do their mindsets, motivations and methodologies look like? What do attackers look for when they look at a Kubernetes context, what do they do when they get in there, and what can you do to protect your clusters and code against them? Being able to understand these perspectives can help you broaden your own. Let’s explore them together, and learn how to build stronger, more secure systems accordingly.

https://sched.co/UdIL

A

Hi everybody thanks to CN CF for having me thank all of you for being here and Nara Alana and Jason hi I love you. This talk is about looking at kubernetes from the eyes of an attacker and what we can learn. From that perspective, my name is Ian cold butter as Brian said, I'm a lead platform, security engineer at Heroku, a Salesforce company I specialized in hacking and hardening kubernetes containers and cloud infrastructure. Before I came to Heroku I was independent penetration testing consultant specializing in the same penetration testing.

A

Is people pay me to hack into their systems and break them? I write reports I tell them what I found, how I got in and how they can fix it. I've been penetration testing kubernetes since the days when you could execute commands as route with an honest indicated, curl call, which was a while ago. It's come a long way. I've watched the project in its security posture evolve and improve over time.

A

I was trying to figure out what I wanted to say in this talk when I was talking to absolutely everybody, and the first thing that came to mind was how community y'all are amazing. Look at this look at all of you. Look at everything we've built here. That's awesome. Kubernetes is a project, that's powered by its people and that same sense of openness, trust and possibility that makes this community such a genuine joy to be around can be seen reflected in some of its design decisions. These design decisions aren't unique to kubernetes.

A

They can be seen in projects all across the CNCs landscape and in a lot of ways, that's gone really well. Clearly, it's resulted in kubernetes huge rate of growth and rapid adoption. It's gotten a lot of people up and running without necessarily having to do it the hard way.

A

So, let's look at this, though, from a different perspective, because it's been said that people tend to build systems that reflect their organizations, their processes and sometimes themselves. They tend to reflect the proof, let the perspectives and contain the biases of their creators. This is why it's so important to have diverse teams and to design for diverse sets of users, because diverse teams, they keep more perspectives in mind, can help to build stronger systems.

A

When we make assumptions that others are like us that they share our biases and our perspectives. Sometimes we can miss things, and sometimes those things are important.

A

Kubernetes is a project that is made by really wonderful people with a lot of trust and a lot of openness and a lot of possibility. That's beautiful and I'm being completely sincere about that and also I think. Sometimes these projects get built, assuming that other people also are as good and as lovely and his trusting as we are and I think that we can see some security implications of that. So, let's talk about this a bit when you're building, who do you build for?

A

Do you build for people who share your perspective for an ideal set of users who use the features that you build in the ways that you intend?

A

If the answer to this is always, yes, you aren't designing for all users, and you need to be so I'd like to talk about a particular set of users who have a very particular set of skills.

A

Let's talk about these users, so we can better understand them and their perspectives and design with them in mind, because attackers are users too, and attackers have user stories. As an attacker, I'd like to tell you some so who are attackers, they could be anyone like this mom right here: they're, not all scary-looking stock photos of white boys wearing balaclavas and basements in the dark attackers. Are people like you they're, not all the same.

A

They come from different places, they might have different skill levels and they have different motivations, maybe they're in it, for their ideals or just because they're curious or for the lulz most are financially motivated on some level, whether for bug, bounties or black hattery and their work lives can look more like yours than you might expect. They have bosses, budgets and deadlines. Advanced persistent threat groups like nation-states have been shown to have roadmaps code repositories and you software like JIRA, to track their progress across their differences.

A

Attackers do share things in common, though they tend to have similar mindsets and similar methodologies. So let's take a look at those and break them down. So how do what a pers think? Well, first, let's talk about youth, but you think when you look at a kubernetes cluster, what do you look for if you're a developer, do you look for features or ease of use? If you're an operator, do you look for scalability or observability? Well, attackers look for those things too, but maybe from a different perspective.

A

What do you see here? I can only see some of you but show of hands. Who here sees two faces? Some of you who here sees a face, so some of you, this picture is called Rubens vase. It's a classic optical illusion. Both of those things are right. It's both of them same thing, different perspective, builders and breakers can both look at the same system and see a different view.

A

So what do you see here? Coop's ETL asked anti list namespace cube system as a penetration. Tester can I just thank the release team responsible for this one, because it is one of the best innovations in years for making my job easier, because a lot of things that administrators our operators might find useful can be useful for us before this command up here came out.

A

We had to figure out what permissions we had in this in the namespace by trying every single call individually, as you might imagine, that took forever and was really annoying to do. This automation makes our enumeration process so much faster and easier for us. A tool like project octant that displays the network and relationships in a cluster context isn't only helpful for developers. It can save us hours of yellow parsing for especially on a black box penetration test.

A

When you have no idea what's going on in there and tools like ephemeral containers and you tating admission control present all kinds of interesting possibilities for it. Imaginative attackers, as han Solo says we can imagine a lot so same thing, different perspectives, one user might run this command to say. Do I have permission to do this, but an attacker might run the same commands to be like what can I do because that's the SFIL to know so what aren't attacker is looking for when they look at a cluster they're.

A

Looking at your cluster, not as a list of resources but they're looking at it as a graph. What is in there? What are the relationships between those components? What are their dependencies are their boundaries anywhere of trust? Where do they get cost? Is there anywhere you can get in certain? Where can you go from there? Is there anything vulnerable? Are there things in there like chiller our IP, with no authentication on it, which I will tell you as a pen, tester I really miss our there's. You know eyes our admission control on there.

A

Is there like at all. What is the admission control if there is any is anything running is privileged. We want to know those kinds of things so that again we can figure out how to get in and where we can go, knowing how a network is laid out what's happening in a cluster context and what those relationships are is hugely helpful because kind of the rough edges and where things cross is often where the attack, surface lives and, frankly, in the wild.

A

A lot of people just aren't using security, much at all in kubernetes contexts, and so, if that's what you finds, then you don't really have to go much farther it's a short day. So what is that methodology? I kind of went through it a little bit, but it kind of has an order to it.

A

So, first, you want to figure out what's happening in there if you're, if you're an external attacker, if you're coming from outside I, want to figure out what, if there's anything, exposed what kinds of services are running if there's wider internet access, that kind of thing, if there's sensitive data, that could be interesting of any kind and once you figure out what's running if there's anything vulnerable, once you figure out. What's in there what's connected, then you can see if there's anything that can be exploited to find an initial foothold.

A

Once you get that foothold you get in, then you kind of look around again figure out what's going on in there, if there's anything vulnerable, if there is any way to escalate privileges, you guys know you're privileged as much as you can. You move as far as you can, and then, where do you go from there? Well, it depends you want to get as far and sometimes the goals can differ. Sometimes it can look like getting through the cluster.

A

Sometimes it can look like getting out of the cluster entirely, because lateral movement from north to south is a thing. If you're leaving cloud meta data in your cluster, where I can read it, which I don't recommend doing.

A

So how do we design to defend against attackers like me? How can you make your systems, something that becomes harder, less legible and more difficult for me to attack and to move through first thing, I'm going to suggest you do is that you threatened at all, because every need and every use case is different.

A

um You know better than I do what you want to protect and maybe who you want to protect it from so figure out those assets, look at them from the perspective of who you want to protect them from, and then design for defense around that.

A

So what is your threat model? What are you trying to protect? Who are you concerned about? How are you going to strategize around that? Do this?

A

First, because once you do, it can allow you to like figure out what's going on and what you need to do in order to protect that stuff, otherwise they're just sort of throwing darts at the wall threat modeling can also involve figuring out the components in a cluster or a context and how they go together, how the data flows, where they move and then figuring out, which of those components could potentially have vulnerabilities by themselves or in relation to each other and basically look at it like an attacker.

A

Would there's a lot of information about this? If you haven't heard of threat, modeling there's a an acronym called stride that a lot of people use there's a lot of there's a really good book on it called threat modeling, and it's it's important to do that, because it can really help you figure out what's going on and what you need to do and again what is in your graph. So it's been said that defenders think enlists an attackers thinking grass at the diversity lunch less or yesterday.

A

Somebody asks us at the security table how we defined a secure cluster and I thought. It was just the most perfect moment, because the defender sitting next to be very enthusiastically responded with a very extensive checklist and I responded. Well, I think that it looks like these things being connected and he's would and in a network- and you know like like it looks like a kind of connected area and I realized. That was exactly what we were doing, which was pretty funny.

A

So, if you're a defender, if you look at things as lists of resources, that's okay there's nothing wrong with that. But you need to know that attackers are going to look at that as a connected list figuring out what those dependencies are, how they connect. What they're doing together so know what you're running know what's in there and understand it. Well, I've not asked a lot this week.

A

How can I learn kubernetes security, it's so big, it's so scary, I, don't even know what to do so, really understand how it works, I would say, is the number one thing kubernetes is made out of containers containers they're just made out of Linux primitives.

A

If you understand how Tech works, you can understand better how to break it and how to secure it, and if your attacker knows that Tec better than you you're gonna, have a bad day so learn how it learn, how it works from the low levels on and figure out again what connects what crosses? Where are those edges and look at it? The way an attacker would what would an autocracy and check your assumptions because things don't always work the way you think they're going to I've found as a penetration tester and as a security professional.

A

Then a whole lot of people assume that kubernetes is secure by default.

A

Spoiler alert kubernetes is not in fact secure by default. It is possible to secure it, but you have to put in work to do that, but that's actually a fairly common.

A

This I would say among end users and that's a things that like we need to be talking to them about and making sure that they have appropriate expectations about, because when people put up those defaults which are open for good reasons, historically, the idea is that you want to be able to have granular control according to your own needs and your own use case, which is going to be different.

A

That's okay, but we need to make clear what actually that looks like to end-users and what they need to do, because your assumptions aren't always going to be correct and expect the unexpected when your threat modeling, because again things aren't always going to go the way you think they are so try to plan for unknown unknowns and go in there and make sure that things are actually working. The way you think they are. Is that thing configured for what you think when you run coop CTLs can't I.

A

When you run the tools that pen testers use against it, does that graph? What's the way you think it does it may or may not? It's worth a try.

A

So again, it's possible to secure kubernetes, although it isn't that way out of the box I'm going to run through a little bit of a laundry list, because I don't have time to cover it, and this isn't really a technical talk. This is to talk about user stories and learning from perspectives, but to cover it, you want to limit your attack surface and you want to reduce your blast. Radius know what you have running again limit things.

A

You don't need follow the principle of least privilege build in defense and depth use admission control, please so many people don't do the basics. It matters a lot as Brian said, kubernetes is complex and that complexity often exists for really good reasons. It has a lot of moving parts and those moving parts can be dangerous and easy to miss, configure understanding how those parts and their Institute intersections can be dangerous, can help you be able to configure them better and understanding.

A

How attackers look at those things when they see them is a good step toward figuring that out. So how can you go about increasing that understanding?

A

Make friends talk to InfoSec people, both outside of and across your organization, see where security folks are coming from we're not all trying to be blockers and make your day sad, I promise a better communication between engineering and security teams can help lead to better processes in general and stronger products.

A

Hire hackers encourage security minded contributors, not just in security organizations or unsafe off, and if you are security minded please go across eggs because everybody clusters insig off that ton is absolutely intended and and all hangs out together and I think that security needs to be happening across the stack security is everyone's responsibility, as we say in the Midwest, silos are for grain.

A

And get some practice thinking like an attacker yourself, you can get in and get your hands dirty. It's not the 80s anymore. You don't have to have war games in order to get some practicing trying out some hacking skills.

A

You can practice and you can practice hacking legally by playing capture the flag games which are not the games that we played on the playground is kids they're, like the flags, aren't physical if they're in a server somewhere, and you have to compromise the server in order to get it a couple of like beginning to you know: intermediate ish resources are right there for capture the flag games, but there you can legally hack them.

A

That's what they're for I also actually genuinely recommend playing untitled use game, which is a game called house house um to that, encourages you to be able to think like an attacker in this, you play a horrible goose who's, terrorizing people in a pissed, aural village, and you go in and there's a shopkeeper and a gardener and various other things who have various objects that you're supposed to chain together.

A

Despite the fact that they're small and relatively innocuous seeming in order to be able to properly exploit them to achieve larger goals, it's actually a great game for understanding how to pentester things and I recommend it.

A

You don't need to be able to think like an attacker all the time, but if you can step into an attacker shoes and be able to think like they would that's a superpower, you can learn from that and you can build around it doing this can help you look at your code and the projects and systems you're building with new eyes.

A

This community is powerful. This project has done amazing things powered by the people in it powered by all of you. Our diversity is our strength, diversity of perspectives of backgrounds of skillsets. We all learn from each other. All the time I have learned so much from all of you. Let's keep growing and learning together from all perspectives and let's try to take those different perspectives into account, even when they're difficult, because I know that everybody here is wonderful and trusting and lovely, and unfortunately, every user isn't necessarily that way.

A

So if we take those user perspectives into and design for them too, we can actually make stronger products without having to sacrifice what makes this community really good.

A

And yeah we don't, we don't have to think alike, we shouldn't think alike. We can keep the things that work well, we can learn from others to make them work better, and if you can step into my shoes well, maybe some of you will find out that it's fun to be bad.

A

Together, we can make things stronger and more secure. We can take each other's perspectives into account. We can learn from one another and we can become stronger together. We can do it, I believe in all of us. Thank you very much.