Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Smarter Kubernetes Access Control: A Simpler Approach to Auth - Rob Scott, ReactiveOps

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Smarter Kubernetes Access Control: A Simpler Approach to Auth - Rob Scott, ReactiveOps

Configuring Kubernetes authorization policies with RBAC is an important part of securing any cluster. Unfortunately implementing effective and maintainable policies can be quite challenging. Many organizations end up struggling with a common set of challenges as they try to implement RBAC across their clusters. These challenges have often resulted in insecure configuration that grants far more access than necessary. In this talk, Rob will highlight open source tools and best practices that address many of these common challenges. This presentation aims to make RBAC more approachable for all, and will provide a straightforward path toward effective and maintainable configuration that can make your cluster more secure.

https://sched.co/MPdi

A

Please join me welcoming Rob's called telling us about a simpler approach to how to discuss control in kubernetes.

B

Thanks for that introduction, my name is Rob Scott, as he said, and I'm gonna be talking today about smarter kubernetes access control, how we can simplify our approach to authorization inside kubernetes I work with a great team at reactive, ops and at reactive ops.

B

We get to see a wide variety of kubernetes clusters from a wide variety of organizations and what that's led to is us learning a whole lot about how to do authorization effectively for a whole different host of organizations and a whole different host of sizes and we're going to try and share those lessons with you today and some tools that we've built to help simplify that process.

B

So the outline of today's talk is really simple: we're gonna start off by going through some challenges of authorization and then go into a quick recap of our back basics. So if you came to the talk and aren't familiar with all the are back terminology, I'll just take a couple minutes and go through the basics of our backs. So the rest of the talk makes sense to everyone here and then. Finally, most of my talk is going to be focused on two distinct areas.

B

One is understanding who has access to what inside your cluster and then maybe, most importantly, managing our back simply and effectively all right, let's get started so first off I think we can all agree that authorization is challenging.

B

Authorization systems often feel either way too simple or way too complex. How many of you have ever been frustrated by an authorization system that was so simple it didn't. Let you do what you wanted to do: anyone, okay, that's plenty of people and on other side how many people have been frustrated that an authorize. This authorization system is just way too complex and just way too overwhelming anyone. Okay,.

C

B

So plenty of the same hands for both and that's the case for me as well. I've seen authorization systems that feel way too simple or way too complex, and it's really hard as a developer of an authorization system to that balance right and that balance is going to be different for every different organization in use case, and, unfortunately, authorization is one of those things. That's only really really noticeable when it's getting in the way. How many times have you heard someone say they really love their authorization system. Anyone hear that ever okay yeah.

B

So it's just not something like I've heard, I, don't think ever because authorization is not something we're thinking about until it's getting in our way. It's not something we're thinking about until we're just annoyed, because it's not letting us do what we want to do, and unfortunately, authorization finds itself in that no-win position where, if you think about it, you're just getting annoyed by it and if you don't notice it you don't give it any credit. So even the best intentions, unfortunately, can still end in failure. Organizations start with these perfect ideas.

B

These perfect goals we're going to have the most granular policies, we're going to do everything by the book. We've gone to Kubb Khan. We've listened to the talk, we're gonna do everything perfectly and then at some point, real life happens, and somebody has this urgent need to get access to the simple resource. And can you just make this one temporary exception just once, I promise? It's only gonna. Be this one time.

B

I really need to get into this and I need it now, and so you make an exception, and you do this one thing and before you know it, your authorization system that was once perfect is just full of temporary solutions on top of temporary solutions, and your best intentions ended up in failure and kubernetes itself comes with its own unique challenges.

B

Users and groups are not actually managed by kubernetes. How many of you have tried to do? Coop cuddle, get users or coop cuddle get groups just to like get a list of users as an example and yeah I mean I've, tried that and you don't get any kind of response, because that's not how kubernetes off works. Your users and groups are coming in from your authentication provider, which is often coming from.

B

Let's say cloud: I am or an OID sea provider or whatever it happens to be, and so users and groups don't actually exist as resources within your cluster, and this can be helpful and really great for your authentication system and the simplicity. But when it comes to authorization, it does complicate things a little bit and then the other bit. If you've tried to scale out, kubernetes are back.

B

You realize that at the beginning it works perfectly and then, as you scale more and more, it gets more and more complicated and harder to manage, and that's something that we found at reactive ops that as we're trying to scale this out across larger organizations, it becomes really challenging.

B

Ok, so now that we've gone through those challenges, let's try and understand some basics of our back. Just if you're not familiar with all the terminology, I just want to go through some basic terms, so the rest of this demo. The rest of this talk all makes sense.

B

So the first term is worth understanding when it comes to authorization and kubernetes our roles and cluster rules. After all, our back stands for role based access control, so roles and cluster rules define specific sets of actions allowed in kubernetes, and maybe the best way to explain this is to show you an example. This is a role inside kubernetes and in this very simple example, it allows us to list deployments and it's available in the dev namespace. So this allows us to list deployments in the dev namespace.

B

You can see the rules down there that grant that now we're going to switch to a cluster rule and you'll. If you blink, you'll miss it, because this is the only difference with the cholesterol. It's almost the same thing. The only difference is that we're role is available in a specific namespace. A cluster role is available across an entire cluster.

B

Now these are all well and good, and there are plenty of use cases like for your own custom applications that need to talk to the kubernetes api, where you want to build out really customized rules that have every single bit of access specified in them, but for user level authorization. What you'll likely want to use is the default roles that are included inside kubernetes and those are pretty self-explanatory. There's a view/edit admin and cluster admin that all grant little bit more access each time and combine these roles are really helpful.

B

When you're trying to figure out your user offer policy combined with a few other things like properly scoping namespaces. This can be a really effective way to keep your authorizations simple, but still effective.

B

Now, where this all comes together is the concept of role bindings and cluster role bindings, and these are what connect service accounts, groups or users to those roles, so whether it's a cluster role or role, one of those default roles that exists in a cluster role bindings are what connect your users to your roles, so they're very important aspect in authorization.

B

So, let's walk through a really simple example: to put it all together, and this example is something like we're going to be referencing throughout the entire rest of the talk. So pay attention to the level of access were granting. So we've got a user we're going to call her Avery and she should be able to edit resources in the web namespace and view resources in the API namespace. So to get for that access, we need to create a role binding.

B

So this is a role binding that connects Avery to the Edit role in the web namespace. So this allows her to edit resources inside the web namespace. Now we're going to do the same thing for the API namespace, except we're going to let her view resources in the API namespace. So if you were paying attention, I did just switch slides and all of two words switched.

B

So you may notice that this can get pretty repetitive. As far as configuration there's, probably some ways. We could potentially improve this and make this a little bit more efficient and less repetitive, and that's what we'll be talking about a little bit later on, but first we need to get into how we can understand.

B

Kubernetes authorization I know a very common question that I've seen within kubernetes is how do I know who has access to what inside my cluster and so in this next section, I really want to spend some time highlighting some tools and practices that can help. You answer a variety of questions along those lines.

B

So, following up with our example, can a 'very list pods and if so, why so can any given user list a resource? And if so, why can they do that? That's a pretty basic question: when you're talking about authorization so built right into coop, cuddle is Kubb, cuddle ah can I, and this is the most awesome tool for understanding access inside kubernetes. So you can ask it can I list pods and you can impersonate in this case Avery.

B

So, in this case, I'm going to ask coop cuddle off, can I list pods now, if you've been paying attention, I've got a question for you: I'm, going to run this command and given access that we've already granted Avery, it was to edit resources and to view resources in a couple different namespaces I can tell you. This response is either going to be yes or no. How many people think B response is going to be? Yes, okay, all right! How many people think it's going to be? No okay?

B

How many people have no idea all right, great, all right? Well, we're gonna find out soon enough, so I run this command and unfortunately, I was hoping to get more people to say yes, because I thought. That would be a reasonable thing. But the answer is no, because this command by default is asking if Avery has access to list pods everywhere in the cluster and Avery has access to list pods in a couple places in the cluster, but not everywhere.

B

So what we really want to ask is: can a 'very list pods in one of those namespaces? So if we run this command again- and we specify say the API name space so we'll just add a dash n and API, the answer we get back is yes. And if we do that again,.

D

Because we can also granular access in this way, we.

B

Namespace specify a namespace, so remember, API, that API namespace we granted her view access. If we run that against the API namespace you starting to see some green checkmarks showing up this show. Okay, Avery can list a lot of resources, not quite everything. She can't create, update or delete anything, but she can list. A lot was really really helpful tool and then remember the web namespace, where we grant granted or edit access.

B

If we run this against the web, namespace we're going to see a lot more green checkmarks, and this shows you what the Edit role grants her okay. So this open source tool is requests. Don't worry if you didn't catch that all these will be linked on the very last slide, so you can take a picture of that. These are really really helpful tools, but sometimes you want to just flip your question around entirely. You have a resource you really care about, and I went too fast.

B

Okay, you have a resource, you really care about inside your cluster and you want to know who can access it? It's rare, but it's going to be pods, you don't really care. Usually, if someone can list pods, but let's imagine that you care that someone can list or read secrets in your cluster. That's a really common use case for a tool like this. An aqua security built a really cool open-source tool.

B

I'll warn that they're still labeling this as a work in progress, but it's an awesome tool and already immediately helpful and it's aptly named cube cuddle. Who can so given an action and a resource? You can get an entire list of users in your cluster that can do that action. So in this case, coop cuddle, who can list pods, is going to show every single user service account or group that can list pods inside my cluster. It's a really cool tool from aqua security.

B

And finally, sometimes you just want to get a top level overview of authorization in your cluster. You just want to see every bit of access that's been granted and for that at reactive Ops we build a tool called our back lookup, and this is a really simple tool that just gives you that top-level overview of what's going on in your cluster as far as authorized ation.

B

So our back lookup in its most basic form just prints out every single bit of access in your cluster. So so it shows you every bit of access that you have granted to anyone any account inside your cluster and that's helpful to get a really wide overview.

B

But sometimes you want to dig a little bit deeper and what our back lookup was real intended for is to look someone up by name: that's the name, our back lookup, and so, if we type our back, look up Avery we're going to see every bit of access that Avery has regardless of scope. So it shows her cluster wide access for API access or web access.

B

Whatever it happens to be, it shows it all in one place and then, if you get the wide output, you can see that the this bit of access is coming from specific role bindings named Avery. So this shows you not only what access she has. It shows you it in every namespace across the cluster and then the source of that access.

B

Well, that's all well and good, but sometimes you want to look a little bit wider. So our back lookup matches any pattern and one pattern that we've seen a lot for user names is email addresses. So in this case all our user names have at example.com.

B

So we're just going to change that around and do our bakit lookup example.com and we're going to get everyone in our organization. All three people that have access to our cluster, and so you can see not only does Avery have access. Joe and Sue also have access inside this cluster.

B

It's a really really simple tool, but there's one more thing because sometimes our back isn't all there is when it comes to authorization, sometimes there's a little bit more and one of the most common use cases that I've referred to before is Google's kubernetes engine where I am is also a form of author authorization inside your kubernetes cluster. So if you're trying to understand not just our back access, but your cloud I am access. We support that too.

B

So you can run our back look up and then add a gke flag and you can see there. I am access as well, so you can get a picture of not just. There are back access in a cluster, but also there I am access so for anyone running gke trying to really understand authorization from two different sources. This can really help make that a little bit more clear, alright, so summary of those tools, coop cuddle off can are, is included by default with coop cuddle. It's amazing.

B

It allows you to see if a user can perform a specific action, and if so, you can dig a little bit deeper and ask why Rick s supercharges that runs that same API call a whole bunch of times for you and gets that same information and presents it with those nice green check, marks and red X's in an access matrix, coupe cuddle who can flips the question around entirely answers? Who can perform a certain action in my cluster and finally, our back look up.

B

It allows you to get an art, an overview of our back and if you're on G ke I am overview.

B

Well, that's all well and good, but maybe the most important part of this is how we can manage kubernetes authorization simply and effectively, and so for the final section I want to walk through some of the lessons we've learned as we've worked with so many different organizations of different sizes for how we can manage kubernetes authorization simply and effectively.

B

So when it comes to effective auerbach I think we need to start with the principle of least privilege and that's don't grant any more access than users actually need. If they ever need it in the future. We can grant it then for a good reason, but if you don't grant any more access than users actually need you're, simplifying life, less access, you're granting is less configure, managing and then use namespaces effectively.

B

It's so often that we see that a whole bunch of applications are deployed into a single namespace, and that makes sense, but when our back starts to come along, you really want to have your namespaces represent your project teams, because namespaces are about as granular as you can configure our back. So you want, if you want to be able to grant teams access to a specific application or namespace. It has to be in a namespace, so create names that line up with your teams and then finally have a clear update process.

B

Ideally, this should include some kind of automation with CI. This is just my own personal opinion, but at the very least, any changes to your authorization should have a clear update process and they should all go through that same floo, whatever works for your organization and then for simpler are back I. Think it's worthwhile to centralize that configuration when there's a question of where does that role binding come from? Where does that bit of access come from it's nice if it's all coming from the same place?

B

What's worked for us is to group all our Auerbach configuration together per cluster, so each cluster has a single place for our back configuration that could be different for each organization, but as long as you have a place to where you know to go for your are back configuration and it makes sense for you, then quite simply give less people access. In many cases, engineers don't actually need access to your kubernetes cluster.

B

If you think about it, you probably have CI CD set up some kind of automated deployments and if they can already just push some code and know that it's going to reach your kubernetes cluster, maybe they don't need to actually be getting into the kubernetes cluster itself and finally use those default rules. Take advantage of them they're there for a reason, view/edit admin and cluster admin combined with namespaces properly scoped. Give you a lot of flexibility with how you setup your authorization.

B

So all of this, hopefully can lead to more effective and simpler are back, but some of these things are still a little bit challenging to get to so we built an open-source operator called are back manager that can help here, and we built this to help accomplish these goals a lot of its surrounding making our back automation a lot simpler to work with.

B

So our back manager allows us to use more concise configuration by grouping resources together. It allows us to automate our back changes. So if you want to put your arm back configuration into some kind of CI CD workflow, this allows for that very straightforwardly and then, as a newer feature, we now support ephemeral, namespaces and a whole lot more with a cool feature: called label selectors and we'll go through each one of those and how they work so deployments, simplify managing pods if you're familiar with the paradigm in kubernetes.

B

If you've created a tormund, it creates a bunch of pods for you, and it just simplifies that workflow, it's a nice Parent Resource that owns it spots. We tried to create that same kind of pattern with our back definitions and that's our custom resource for our back manager and they simplify managing role bindings.

B

So role bindings didn't really have a parent resource before and they were a little bit annoying to work with, especially at any kind of scale, and so we thought well what if we could create a custom resource on top of this, that would simplify the management of role bindings, and so the advantage of our back definition is number one, it's more concise.

B

So if you remember that example from before it was a really simple example where we gave jane access at Avery access, we granted her to role bindings, that same configuration can fit on a single slide. So instead of two slides of Y amyl we're down to a single slide of yahwah and usually, if I had more room on this slide, I could show a more dramatic change in configuration.

B

Usually, we cut configuration by around a third of just vanilla royal bindings with kubernetes, so you can cut down the amount of configuration you're using pretty significantly and I. Think one advantages while you're writing a bit less yamo, but another advantage of using a system like this. Is it to me at least it's a little bit easier to read because in one place you can see that Avery or whatever group?

B

Usually this would be a list of users that belong to a specific team, and you can see all the different bits of access you've granted them. So in this case we can see in one place that we've granted Avery view access in the API namespace and edit access in the web namespace, instead of looking through two different configuration files, so just to prove that this works we'll go ahead and apply this auerbach definition and we can watch the rback manager logs before below and right away.

B

It creates those role bindings for us, so it just does that for us we get the same end result we've just saved some configuration.

B

Well, that's all well and good, but really what I think the key idea behind our back manager was. Is we need a better workflow around our back? We need a better way to work with our back manage back and have this be a little bit more automated.

B

So for our back automation, we knew we needed to ensure that newly defined role bindings were reliably reliably created and the good news is that already worked. That was not an issue so, whatever system we built, we could just recreate that and it was going to be fine, but the next two got a little bit more complicated role, bindings that require changes needed to be reliably updated or replaced. Now, why would I say replaced well role? References and role bindings are immutable, so you can't actually change a role reference in a role binding.

B

So what you would have to do is delete an old rule binding and create a new role binding if you were trying to change someone's access from say edit to admin. So not the end of the world there were was some coop cuddle, tooling, called coop cuddle reconcile that helped with that, but we also wanted to make sure that that process was very smooth with our back manager, but finally, maybe the biggest one was we needed?

B

A way to ensure that we could say safely revoke access without accidentally deleting the wrong role binding because any kind of automation you need to ensure that what you have in your code, whatever whatever you're using with infrastructure as code type thing, actually represents what the state is in the cluster.

B

So you need to trust that revocation of access is going to work as you want it to so, going back to our our back definition, we're going to try out a couple of these changes, and so, in this case, I'm going to go ahead and I'm going to change, Avery's access and API from view to admin, and usually ideally, this should go through some kind of CI in this case, I'm going to edit live because it's a demo- and this was the easiest thing I could do, but we're just going to run with this pretend this is a CI workflow.

B

But when this auerbach definition changes our back manager goes ahead, deletes the old role binding and creates another one automatically. So it's just one bit of pain that you don't have to deal with. It just works, and maybe more importantly, is revocation so in this case we're going to go ahead and remove that access to API altogether. So we'll just remove the the reference from the ARB definition and again, editing live instead of using CI.

B

But if you can imagine this going through a CI workflow, that's that's what I'm aiming for here and when we edit this, we can see that right away. Our back manager sees that our back definition has changed, sees that a specific role binding is no longer desired. That was owned by that specific are back definition, and so it revokes that it only it only touches things that it has created and our own by specific are back definitions.

B

But with that- and with that said, we have a really straightforward path to start automating these changes and using them in some kind of CI workflow, with pull request approval whatever it happens, to be for your organization, but you can now automate your are back configuration now. Maybe the most interesting feature with our back manager is label selectors. How many people here are using ephemeral environments and kubernetes? Okay, a few?

B

How many people want to be a few more okay, so this is for you, and maybe there are some other use cases here too, so if you've ever wanted to have an ephemeral environment system where just a new, a new environment just spins up like nothing for your developers, if they're working on a feature branch, this helps solve the authorization side of that. We now allow that we now support namespace, selectors so anytime, that a namespace has a label team equals API.

B

In this example, Avery is going to get edit access and again you can have a whole list of users here, representing your API team, but to keep with our example. This is just going to be Avery. Getting this access, so we're going to create a new namespace will label it. Demo will create a new namespace called demo and we'll label it team equals API, and when we do that, if you notice below our back manager has automatically created that role binding for us, so Avery has the access we want.

B

And finally, if we change that label to another team, like not API, then her access is immediately revoked and she no longer has access in side of that namespace. So really simple, workflow, but if you've ever wanted to work with ephemeral environments, this can help that out. So to recap, our back manager. It gives us more concise and simpler configuration that groups role bindings together. Our back changes are now a whole lot easier to automate and label selectors allow for simple our back for ephemeral environments thanks so much for your time here.

B

These are all these projects that I've featured today already they're all on github they're, all open source. The last two were developed by us at reactive, ops, we're always open to any kind of feedback ideas, issues pull requests whatever you run into. We would love to make these tools even better, so they can make authorization more approachable and more simple inside kubernetes I think I might have a few minutes. Any questions.

D

E

Maybe you can go a bit into detail. What permissions I need to run these tools, so maybe you can start with you Carol. Can I, and can I impersonate anybody even if I don't have any privileges, great question and.

B

Eat Rosie's.

E

B

That's that's a great question with coop cuddle off. Can I that tool comes with the subject, access review permission, so anyone can run that as themselves if they're authenticated in kubernetes cluster, that's a default permission, but if you're trying to impersonate someone else, that's an additional permission, I'm, not sure which default role grants that by default, but it is an additional permission to be able to run it as someone else.

A

F

If you're, using opa or similar on top of the default or Boxtel for the many tools that help make the laid decision, analysis easier, sorry.

B

Can you say that again, if.

F

You're using OPA, on top of our back to do finer, grained access control or the many tools that make it easier to say what got rejected. Why and where yeah.

B

That's that's a great question so so he's asking about if I understood it correctly, he was asking about if you could use open policy agent to use more granular policy when it comes to our back with. Is that correct? Okay,.

F

B

And if there are any tools that help I don't know of any tools in particular that help there are some open policy agent web hooks which would allow you to create an open policy agent on your own and open open policy agent configuration on your own and prevent any kind of actions from happening. Web books are really really amazing. It's it's basically free form, but you can write OPA policies to do off. I, don't know if any specific tool that helps you do that that's off focused.

C

Thank you for, for these compilation of tools. I keep thinking. Do you know if those tools have a keep CTO plugin because they are yeah like they're, just one step away right, I'm for me, cube CTO, yeah plugins would lean more natural flow, I'm.

B

Not 100 percent percent sure about coop kettle who can I know Rick s is known as access matrix if you use crew as a bug, plugin manager and our back lookup is known as our back lookup within crew, but they are both available as coop kettle plugins as well.

G

Thank you actually, I must not. The question is more like a general comment, just as a security guy cannot stay silent. So first of all, I need to thank you, because this was very impressive and very useful presentation and demo, but I couldn't just it was like a splinter in my mind, why do you grant access to individual users? Why don't you grant access to groups of users? Is it for demo purposes or yeah? This is the way.

B

You do that's that's a great question it. It depends on how the users come into your system. Essentially, so your authentication provider, when someone is authenticating into your user and into your cluster, can provide a username and an optional list of groups. Some some authentication providers, support groups, others don't the one thing that is always provided is a user name. So groups when they're available are awesome but they're not always universally supported, so it just kind of depends. But that's a great point: if you do have groups available to you, definitely use them.

C

Thanks a lot for this presentation now.

A

We know everything about every success to the cluster, but do this tool work with service accounts? Yes,.

B

Absolutely service accounts are listed in output for all of these, and even our back manager I didn't cover that specific component that it was too short of a talk to cover everything, but our back manager does also managed service accounts for those use cases as well. If you want to.

H

So you gave what permissions are required for keep CTL can I. Do you know what the permissions required for the other four tools are yeah.

B

So our back manager is some is an operator, that's deployed into your cluster and because it's creating role bindings and cluster role bindings for you, it needs to be able to create cluster role, bindings role bindings for you, and it also needs to be able to watch namespaces for changes.

B

Coop cuttle, who can I'm not as familiar with that tool. Our back lookup is I. Just read only access on our back resources, such as roll bindings and cholesterol, bindings, yep.

D

These tools are enable multi-tenancy, multi-tenancy.

B

Not necessarily it kind of that they can help in that in that process, but there are other limitations currently in kubernetes, where there's there's a lot of work going on to help make multi-tenancy more feasible, but this would be more helpful towards maybe soft multi-tenancy, but for true multi-tenancy. We need additional, tooling and and ways to restrict access between namespaces that aren't currently perfect. So there's there's work going on, but I wouldn't say this on its own can get us to that perfect multi-tenancy.

D

Point so what are those tools.

B

As far as working on multi-tenancy I I guess, tools is in the right term. It's more work inside kubernetes itself, there's a multi-tenancy working group. That's doing some awesome stuff to try and help simplify that process and help get to a more secure state, but I don't have a great resources beyond that.

A

Hello, yeah amia: do you do you plan to available select our pod level? Yes,.

B

Sorry say that again do.

A

You plan to have low levels rectorate, but level, not namespace level, an.

B

Inspector for a pod, a stable.

C

Selector Oh gotcha.

B

Yes, sorry so label selector is something that we initially only supported for namespaces, it's a little bit trickier when it comes to other kubernetes resources, because roll bindings by default are something that can be granted at a namespace level. There are specific ways to grant specific resource names, but that gets a little stranger than that, something that's not in roll bindings. That's in defining the rules themselves, so we kind of have to change how our back manager works right. Now it's just focused on managing role bindings and we can do namespaces that way.

B

But it's harder to you resources themselves individually. Okay,.

F

We're out of time thanks Rob.