Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How the Basics Of Kubernetes Auth Scale For Organizations - Leigh Capili, VMware

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How the Basics Of Kubernetes Auth Scale For Organizations - Leigh Capili, VMware

Speakers: Leigh Capili
Every day, someone new wants to deploy their cloud-native app... then it ends up running on Kubernetes! Suddenly, all sorts of people have questions about the app and need to make changes. Developers, SREs, Product Folks, and Organization Leaders all have different needs and levels of experience with the myriad of Kubernetes resources needed for an app, so how should platform teams give folks the access they need? This session will show how the basics of Kubernetes Role-Based Access Control (RBAC) compose together to create a flexible system that lets teams work together. Come learn about: - RBAC Basics - The Details of Identity - Common Misunderstandings - RBAC's Limitations - Auditing - Useful Patterns for Real-world Implementations This session will also include some fun, relatable Live Demos! Expect to leave better equipped to support the transparent, collaborative, self-service ownership of a DevOps culture.

A

John, what's up peeps back from break yeah? How is everyone?

A

uh What is this day? Two kubecon? It feels like day four for me, yeah yeah, all right! Well, uh let's go! Let's go! Do some learning! uh If you don't know me, my name's Lee I contribute to kubernetes, sometimes also pretty active, with the flux project. I then you're a proud, Filipino people in the room here: no hey, okay, John! What's up yeah filipino-american I live in Colorado and uh I'm gonna. Kick this talk off with something that's a little bit different, a little weird, uh it's performance, art yeah.

A

So uh let me see what let me just make sure I got the tempo here right.

A

All right, I'm gonna need help like this should be about like right. Here. We just gotta, keep it steady, yeah sit back, deploy apps ship in it relapse trying to update a hump day, but I need a new play, the app's on fire. This is so not dope. I can't debug teens losing hope, I just clicked the button it wasn't. Disabled I was on cloud nine. Then it dropped at the table. A big surprise. Just a simple reflex now production's down and I need some kleenex. It was a wallet text needed my glasses.

A

We on kubernetes, but the app still crashes it worked got the sir took some classes, but I can't help now, because I don't have access, just click the button. Now it wasn't disabled, Amazon, Cloud Nine, then I dropped the table big surprise. Just a simple reflex now production's down and I need some kleenex. While the texts needed my glasses, we on kubernetes, but the app still crashes did the work got the search had the classes I want to help now can I get some access.

A

Things down man, all right, I, don't know man. So why are we talking about pain and working together with people on hard systems right when working together we're using kubernetes?

A

We gotta figure out how to use the tools to have the people in process part of getting work done right. We don't use the infrastructure, we don't use fancy computers because it's fun well, sometimes we do because we're nerds but like the bigger problems, is to solve Society humanity and making money with capitalism are working together to do useful things right. We want to make apps, have people use them, store information, and so we got to figure out how to share and scale things for our organizations right and so we're using a computer.

A

Maybe it's just one computer and you go log into that thing and it gives you the access to the thing, and then you start using that computer to do all this imperative, stuff, you're, writing, Google, Docs or whatever, um and it's just like kind of you who like uses this laptop. Maybe you share it with like your cousin or something too, but this kubernetes thing is the operating system. That turns like lots of these servers or whatever cloud data center into one big computer. It's an operating system.

A

It gives us a distributed computer and you can't program it the normal way that you use like a laptop, or maybe you run like some python on a single machine right. A lot of people say we configure kubernetes, but you configure things that are already doing stuff right. Kubernetes is a computer and it doesn't actually do anything useful unless we program it right. So we give these configs, even though they're declarative, they're source code and this source code is in the form of a declarative promise.

A

So kubernetes like makes promises happen for us, but the kinds of promises that it does are pretty useful right. It deploys apps, it can promise. Us networks can manage lots of servers at different pools with labels and like have namespaces schedule over here over there you can enforce policies and stuff, and- and we talk a lot about these objects- we're super focused on the tools there's like deployments. You know, Services nodes, Network policies, it kind of maps up to those use cases they fit into different API groups.

A

We use the same technique to go and declare these promises into this one computer right. So we have the same programming model. Now. These things are all so different and useful that we tend to have a lot of people involved in using this one big expensive, but super fast, Formula, One car of a promise machine, and so, like you got developers, you got operators, you know it's like a whole Community. It's like a city to run the kubernetes race car around the track and um the API to help us share.

A

The computer is called role-based access, control or our back right. So there's the first bit of jargon in this really API introspective talk.

A

So when we first log in to a computer, a computer wants to know who we are, and it wants to also then start granting us access and there's these two kind of other jargony words that people use called authentication and authorization right.

A

So when we authenticate the computer knows who we are right, it attests that oh I really believe that you are are who you say: you are I, know your identity and then, when we Grant access we're talking about authorization, are you authorized to go and do something right now that I know who you are? Let's talk about what you can do right so there's who and what and then linking that together now role-based access control is the authorization part of this right.

A

So now that we know who people are is completely unconcerned with how we figured that out, we're going to try to figure out how to authorize these people right and again this this problem. The reason that this API is important is because it is going to let us share and block off parts of the computer to this whole city that it takes to run this thing right.

A

So if this question is what are the people or apps able to access, we need to figure out like a more generic term, for this, like people or apps thing, so the rbac API role-based access control, it calls it subjects and um I'm. This is not like a royalty or like a king or queen thing like I. Don't have subjects like unless maybe the top is like the CEO or something but there's like all of these people. So what what's the term of subject?

A

What's the type of subject that these people are, uh they are users and groups right, so kubernetes has some objects. Now these objects are sort of special there's, not a route that you can go into kubernetes and you can make users and groups and we'll talk a little bit more about that later.

A

It happens, sort of indirectly based off of what you integrate kubernetes with so these these words they kind of look like the same as how we talk about the rest of the kubernetes objects, but these things aren't real objects, you sort of have to imagine them and use the API to describe and Target them.

A

So, hopefully, that's making a little bit of sense just like not if you're kind of following um the other type of subject is an app and for the most part, apps don't need to talk to kubernetes, but we actually have a lot of apps that we deploy to kubernetes to extend it and sometimes when you're trying to solve distributed computing problems, it can be helpful to use the distributed computer parts of kubernetes, and so you need to have your apps be able to access the API and and learn things about itself.

A

What's around it in its own namespace or elsewhere. I mentioned namespace. If this idea is weird to you, this is just the way that we section up the computer virtually in our heads, so that teams can like not step on each other and apps cannot step on each other. It's really important concept right and so, uh in this case, I'm just drawing out the mechanism of what the service account actually does. So this is other objects like pods replica sets, deployments and stuff that actually run code inside of the piece of the computer.

A

That kubernetes lends you and um basically the kubelet is the thing that sees the service account and sees the Pod, and it gives a token to every pod so that the Pod can authenticate with the kubernetes API. So that's the auth piece kubernetes does have an identity system to authenticate workloads inside the cluster um yeah just little details.

A

Now we got these three types of subjects: for the people we have users and groups and for the apps we have service accounts. The subjects of this API are who it's the? Who part of who has access, and then we can ask about who has access to what? So now we got to talk about a different object and this is a real object. We don't have to be confused with imagining things anymore. Service accounts are real as well.

A

It's just users and groups that are sort of imaginary, so rolls are an object that describes what should be accessed right. You can think of a role like maybe viewer or editor and the what of how something should be accessed also includes things you can do to that object. So roles have a list of rules, and that list has API. Groups has the kinds of things in those API groups, those resources, and then it has the verbs that you're allowed to do to that thing.

A

There's a couple other stuff too, that I'll talk about so roles list of the allowed apis. They list the verbs and specifically there's an object, called roll, and this is a namespaced thing right. So if you've got a team, that's in there part of the computer inside of their own namespace. They can make roles that have rules about access and they can list stuff. That's inside of that namespace resource types that can be used inside of namespaces there's a different object called the cluster rule.

A

A cluster role is a special type of role that doesn't live inside of a namespace, because sometimes we have to talk about things that affect the entire cluster, whether it's just talking about say all of the configs or Secrets across every single namespace, or if we're talking about things that don't have a namespace necessarily uh things like cluster or custom resource definitions or uh like Network policies and uh there's a there's, a bunch of these kind of cluster level objects that can affect everything.

A

Also extensions can add their own cluster level objects and uh it's important to be able to control access to all of this stuff. Maybe that doesn't make so much sense if you're, just thinking about in your head, so I drew you a fancy picture with something that is really good at making round rectangles. This is a kubernetes cluster simplified.

A

We got the blue cluster, you can have resources at the cluster scope that affect everything, and then we split up little pieces of it, and then we can put all different kinds of things that are namespaceable objects inside of the namespaces right. There's the dev team, one and Dev team. Two um roles can only control access to stuff that points in or that's within namespaces and in fact, the roles themselves live in namespaces, but cluster roles can affect everything.

A

So now that we've answered kind of the question of who and what? How do we describe? Who has access and how do we describe what they have access to and what they can do? We have the missing link, which is we have to map all of the connections right, so we need this object, called a role binding. The role binding binds the subject to a role. It means that hey this Dev team has viewer access to this namespace and they can update this policy with this name or whatever.

A

Similarly, you could say that this app has access to all of the configs in this namespace, so it can go and update interesting fields, and it will use a role to describe what it can do with updating the interesting fields in the conflict map, and the subject would be that service account name from the service account object. So um this is basically authorization in action right now we have the ability to ask the full question and and make full statements hey. These people can go and access this thing.

A

So there's a little bit of uh just like kind of 101. How is this API shaped? How do we use it right? We're not even looking at like yaml, configs and stuff um with role bindings, there's a couple of gotchas here and some interesting variations right, so role bindings live in namespaces and they are fully concerned when a role binding is in a namespace.

A

It only authorizes to the stuff in that namespace and that therefore, like you, can only really reference access to namespaceable objects right, you're, not going to be role binding to something that has cluster scope, but cluster role bindings. Allow you to say hey this subject across the entire cluster scope for all of the namespaces, for any namespaceable objects in all namespaces and for stuff at the cluster scope. If it's listed inside of the cluster role, we can grant access now.

A

um There is some really common misconceptions and it's actually because of this particular shape of API right. We have cluster roles, cluster row, bindings role, bindings roles, it seems like they would just kind of only pair up together right, and so this makes sense any subject any person, any group or application. We can role, bind it and we can, within a namespace, give access to stuff. Similarly, anything any app in the cluster, any person or group that we can attest with and authenticate. We can.

A

Cluster role bind it for like across the entire cluster, and this seems like it's enough to start doing some really good work, but uh if you say want to give somebody access to like five namespaces or 50 namespaces, but you actually have a hundred and you don't actually want to give them access to a hundred. It seems kind of like tempting to just go for that cluster role binding, but it's important to be remembering the principle of least privilege.

A

You know we only really want to give people access to truly what they need, and we definitely don't want to make any missteps and give people access to stuff that they don't did. You know that you could roll bind to a cluster role.

A

This means that we don't have to Define roles 50 times right and you're, saying oh lee clusterables also talk about cluster scoped objects in this case it the cluster, scoped objects are just ignored and this is actually still very precise, and this is actually a very recommended thing that you should be doing right. So maybe, when you looked at these names, uh maybe you were using it and you learned it at some point.

A

If you're a beginner, you actually have an advantage, probably over most practitioners right now, because you're, like learning the basics, instead of just pretending to learn something while you use it I've had this misconception before it's very common. You wouldn't believe just how many people do not know that you can do this, so the behavior here is that the cluster will, whatever it describes as namespaceable objects. You will only get access to those namespaceable objects in the namespace that you roll bound from.

A

So it's just a single namespace rule binding, and this at least, if you have to roll by in 50 different namespaces with the principle of least privilege. At least you only have to make 50 roll bindings. You don't have to also redefine the same exact role. The thing that talks about access policy 50 times you can just find that centrally in the cluster Rule and kubernetes actually has a bunch of default ones that I really recommend you should use. This is a beautiful picture, I'm really proud of this one.

A

uh The best thing about this picture is that it is complicated, but there's only one subject, and so there's a bunch of these purple and pink things. um Those are the the roles and the cluster rules sitting in and out of the namespace at the very end of the arrows right. You have like somebody who can administer policy right or like a rule set about viewing things across the cluster, regardless of what you're talking about or a rule set about, updating quotas that might be just in this namespace and we can use cluster role bindings.

A

We can use roll bindings, we can use roll bindings. I didn't draw any arrows from the roll bindings to the cluster roles, but you could do that. You can do that all on the same subject- and this is a common misconception- that there has to be like a one to one to one mapping of this, but it would actually be encouraged that you're being very precise about what you're doing when you're giving people access right. So you can do that I lowered the volume.

A

um Another thing, that's sometimes not obvious- is we're really used to giving access to apps within their namespace. But how do you give access to an application to stuff outside of its own namespace?

A

Well, the truth is that apps are just users: service accounts, they're, just usernames, there's nothing special really about them, except for that kubelets can give them tokens, and so you can actually have an app with a service account or even a service account group in another namespace, and you can roll bind it from the place that you want access right, so say: I own, a namespace I have some interesting juicy configs or whatever inside my namespace, and some other team has an app that we want to give them access to well.

A

Service accounts in a namespace are actually all part of the same groups, and so you can actually roll bind to the group of that namespace and you don't even have to know what the name of their service account is. Another common underly used feature and sort of a misconception you can roll bind to to apps from other namespaces um service accounts are just users, their usernames. Look like this beautiful templated string over here system service account. This is a reserved prefix.

A

Most authentication systems will not spit this out, uh which means that it only mostly Works inside the cluster and using cluster mechanics, and then it's got the namespace so and then the name and then service accounts also get groups. So there's a group for every service account in the cluster.

A

If you saved wanted to give literally everything access to something you could bind to system service accounts and you can bind to system service accounts from just a couple of different name spaces if you want so underused features now that we've kind of gotten past the basics of the API and we're getting into a little bit esoteric stuff, just make sure I know what the time is here: yeah um there's resource names, so we can get very specific about our rules.

A

Inside of our roles or cluster roles, you can restrict your resources that you get verbs to to particular resource names. This example is from the kubernetes documentation, there's a config map updater. This is something that you maybe might want to do with an application to spit out something into a consistent, FCD storage.

A

um So then, that could be like consumed by something else. uh Here we are having the update and the get verb in the core API Group, which is just what that empty string is for, but sometimes that string is really long and it talks about other API groups. Resources is just the config Maps you'll notice that this is spelled a little differently than normal, it's not capitalized and stuff.

A

This is because it's just like the section of the rest endpoint when you like talk to the HTTP service for the API server uh and then resource names we're actually restricting that we can only update the my config map uh instead of all config maps in this namespace default. So uh also this one is like kind of funny.

A

um You can't filter what people create with resource names, except that you actually really can, but it only works with server side apply. So the reason for that is because, when you're in a server-side apply, it knows about the whole object that you're sending it so they can do diffing.

A

But if you are not using the server side apply like if you're actually doing the create verb uh like just normally, then this is not going to work because the API server doesn't know the name of the thing you made yet so sort of a weird limitation I feel, like that's a pretty useful feature and for it to only work with SSA Machinery. It's kind of odd uh cluster role.

A

Aggregation I can't really explain this on a slide, so I'm just going to show you in the cluster and now my friend Rich is going to be happy because he came for a live demo. So uh in a kubernetes cluster, if you don't actually even modify anything at all and you get all of the cluster roles.

A

So these are the centrally defined roles that can talk about cluster specific things, as well as all of the namespaced things, and you can roll bind to this from just one namespace or you can give access across the entire cluster.

A

With the cluster rule binding, um there's all this stuff and there's actually a bunch of extra stuff in here as well, that got added from other add-ons that I installed to the cluster external DNS, whatever Cayenne, but if we kind of grep for the system prefix, you can see there's all of this cool stuff that kubernetes uses internally to have all of its components talk to each other. So you can see that uh the identity of things is very important.

A

All of these things have various role bindings and cluster role bindings so that they can do like compute problems and make the kubernetes cluster keep promises for us right now. Extending this lets. You really use the power of the distributed computer yourself, and it's also very necessary, then for you to also be able to um what am I saying, also be able to give people access to the information from the computer right. So here's all the cluster rules.

A

We can look at the cluster role bindings.

A

And you can see, then, that every uh sort of bit of access that we've needed to open up for somebody- it's all in here now- I'm not interested in that I'm, mostly interested in how I can open access up for other people, because the cluster works great and if I need to program an app to talk to kubernetes. That's fine, but I really want to enable my team. So let's get rid of all the system stuff and just look at what's in here, there's a bunch of still stuff for add-ons and that kind of thing.

A

But if you look at the ammo I'm gonna pipe this to neat and also just syntax, highlight that for you. Let's, like look at the viewer role, I guess: I mistyped it here View.

A

Double gets. Thank you pair programming uh and I need to also type cluster role. This is why you script demos cool, so you can see. There's if you say like wanna, have the view cluster role, there's like getting and listing, and watching for pretty much a bunch of API groups and resources across the cluster, but there's also these labels and there's also this aggregation rule. That's weird: what is that this aggregation rule has a cluster role, selector that matches the label of other cluster roles and it what it does, is it Aggregates them together?

A

So if there's any other cluster roles that have the label aggregate to view, then we can actually from this default cluster role. If people bind to this- and we add to the cluster and we extend it, we can also let them view other things stuff that didn't Ship by default with kubernetes.

A

Similarly, this viewer cluster role as soon as you give access for something to be viewed in this rules list, it gets aggregated, see how this is labeled aggregate to edit. This is an official kubernetes API and you can use this and it works fast. It's in the API server, and so these aggregated cluster roles, uh they're very precise. There is another.

A

Let me get this label selector here. I. Did this beautiful command earlier.

A

Why am I began? We don't need to do that yeah. Let's just do this.

A

uh And that needs a t, so these are all of the bootstrap cluster roles. It's very simple, everyone's heard of cluster admin because it's notorious, but there's admin, there's edit there's View and you can see how not having to figure out how to tell somebody that you can view every um like API resource or inside of a namespace would be very useful right. So now, if say, I want to give a development team access to view everything that they own in their namespace.

A

A very good default thing to do would just be to roll bind against the view, cluster role and again like if you, if you do a survey, a lot of people do not know that you can do this. um So, let's, let's use our back. uh This is this is really good. Let's talk a little bit.

A

uh Oh there's, there's other uh underused features here: sub resources, there's a bunch of these they're very powerful, mainly the status the proxy and the port forward um ones also attaches and binding, and then there's also these weird virtual verbs. This is how you can do some interesting stuff with rbac. You can, for instance, Implement pseudo access using these verbs. Also impersonate is very useful for like debugging our back.

A

um This is a very underused feature and I will talk about it later. But let's talk about then: okay, we talked about using the API together and about how to Grant access.

A

Kubernetes doesn't actually authenticate people out of the box. Maybe you installed it in a cloud and it's integrated with the Cloud's identity provider just out of the box, but you actually have to do work with default kubernetes to make it know who your people are, so that people can actually log into the thing.

A

There's some great projects out there, kubelogin I'm really a believer of the pinniped project because you don't have to modify the API server just go check out pinaped.dev, but we need to be able to authenticate people, and we want to probably do that with the identities that we already use, because we probably have multiple kubernetes clusters at some point. I, don't really recommend using cert off and service accounts outside the cluster, because those things are for apps and Integrations and cluster internal things, and they will never mean anything else in another cluster.

A

You should use the bootstrap cluster rules like I mentioned. um We should be binding to groups binding to single people is good for like ad hoc things, but we want to enable teams. We want transparency and, ultimately, the tools that we use start to shape. Our habits right, so if we use a declarative computer, we should probably be using declarative configurations, so I would recommend to really get Ops your Access Control. We can store all of these objects in some place where some or maybe even everybody could reach them. Everyone could see them.

A

Maybe not everyone can write to these things. You probably want to really protect them, but you should be thinking about how you can onboard people, how you can give people access when there's an emergency and they accidentally drop the table and didn't have access. You know so um yeah. You want proper props process because we need to be able to work with each other. This is how we share the computer.

A

This is how we get stuff done. We need to empower people with information. Can you can you actually raise your hand if you've tried to do your job before and you didn't have the information to make good decisions yeah right? So let's give people the info. They need to be successful if you're a manager. Maybe you should be able to find out for yourself if some software got released right, if you're a Dev team, maybe you should be able to delete pods so that you don't have to page SRE.

A

If you know something's a simple fix, and you know that, there's a memory link there, you have more context on your app right.

A

Similarly, I mean like if you are, um this is like some of the most important intellectual property in your organization, it's like who has access to what? What do people actually do? If you don't have a good handle of how people get access and and when people had access and what people have access to? Currently, you're, never going to be able to recreate queer clusters, move your clusters somewhere else, you're going to get very tightly coupled in an imperative.

A

State that's difficult to recover, especially because all of this stuff really needs to be commented and annotated with things that are readable by humans. You can't just get this from like a Valero backup of your kubernetes cluster. You need to preserve this intellectual property.

A

um I would say that you should consider using some integration tests. You should run these periodically so check that people and apps have access to the stuff that they need. This is how we can keep stronger promises to each other. um Actually, you know, I'll just show the slide of the command that you need to run. This is Coupe. Cuddle off can I it's a way to probe.

A

If you have the impersonate verb and you can also restrict the impersonate verb with resource names, then you can go and say: does this service account for this application have access to update its config? Do these people have access to these important business? Critical objects like deleting pods, you know, and if you put this in your CI Pipeline and you have a transparent way to update access and to like request platform admins like give you a little bit more permission to be able to solve harder and more intricate problems.

A

As you learn more about kubernetes, then running. These sorts of things in a CI pipeline gives people a very clear indication that they have what they need and that they don't have access to the things that they definitely should not have access to. So there's a bunch of limitations list and watch. You got to be careful with this. If you have list and watch on secrets, you can actually read all of the secrets at either the namespace or, if you cluster role, bind it that's bad. You cannot filter lists with resource names.

A

If you need something like this, like filtering namespaces for secrecy, you should maybe be thinking about using V clusters or API server filtering proxy, like the classics project, which is open source um resource names with the crate verb I mentioned only works with server-side apply, although maybe you could also just put this somewhere else so that you don't have to filter what the name of the thing is and um yeah you know: I've suffered a lot with our back.

A

I've helped run kubernetes and make it more usable for lots of Dev teams, as well as really seeing some interesting stuff from our users in the flux and kubernetes projects. You can definitely reach out to me on Twitter. My DMs are open and yeah I think I'm, probably out of time, but feel free to find me after this cheers. Friends.