Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing Kubernetes With Admission Controllers - Dave Strebel, Microsoft

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Securing Kubernetes With Admission Controllers - Dave Strebel, Microsoft

An admission controller allows you to intercept or mutate request to the Kubernetes API before it's persisted as an object in Kubernetes . By utilizing admission controllers to intercept objects we have to the flexibility to enforce enterprise security policies for developers and operators of Kubernetes. In this demo heavy session, we will review admission controller capabilities and use cases for extending Kubernetes security. We will also demonstrate how to use an admission controller to restrict access to specific service types in a Kubernetes cluster. Attendees will leave understanding how to utilize admission controllers to extend security in their Kubernetes environment. They will also learn about use cases of using admission controllers to provide enterprise grade security policies.

To Learn More: https://sched.co/GrZQ

A

All right we're gonna, go ahead and get started so you're at securing kubernetes with the mission controllers. Is everybody in the right room, because, when I talk about security, usually people run for the door like I was amazed at how many people registered for this I'm sure the Seavey's that came out last week had something to do with it.

A

So Who am I. My name is Dave strable I'm, a global open source architect at Microsoft I'm, also a co-chair of cig Asher and on the kubernetes release team. All sounds really important, but it's not if you want to connect with me. I am at Dave, strable or Dave underscore strable on Twitter. Does anybody know what that keyboard? Looking thing is: there's not many in here that helps me like at least know my demographic that I'm talking to, but it was my first computer and why I got inspired by technology.

A

Also, how many people in here are primarily just doing security? That's their main role at their company Oh, a lot more than I thought. Okay, great. How many people in here struggle with security and kubernetes?

A

How many are in denial.

A

Awesome all right so you're going to deploy kubernetes. That's why we're all here at least I'm guessing and the first hurdle you come into a lot of times is going to be. This is IT security, and it's not that IT security is bad, but there's fundamental problems that we do have. If you look at cloud native architectures are very dynamic in nature. This is something security teams are not used to. You can never count on, say a pod being around 24/7.

A

These things come and go. That's that's a big hurdle for security teams. You can also look at a lot of security tools. They haven't been retrofitted for these cloud native architectures. So we have a lack of tools. Also clusters not immutable this one. You could really argue because a lot of people think hey I, use terraform. Now my clusters immutable right.

A

It's not because what tends to happen in kubernetes is will build a cluster than another cluster, and then somebody will install operator on cluster one cluster to never get set, and we have clusters that are just a bunch of snowflakes there's, no way to really control policy around those different clusters.

A

Some of the different approaches we can take can be manual intervention. This doesn't scale well and not something we really want to do. We could also restrict users from creating any type of objects and become really restrictive around that and when we apply these methodologies that we've always done it. That way, our architecture looks like this. Some security professionals would say this looks great, but if you, if you remember Attica the prisoners rebelled, so this is not a place we want to be because it really leads to this frustration.

A

Not just frustration- and this is so funny- aah-ha-ha- not just frustration from security, not just frustration from developers or operators. Everybody gets frustrated when we build architectures, like that. We start building architectures that are kind of like cloud sort of native and we kind of twist and turn kubernetes and make it into something that was never supposed to be. So we lose a lot of advantages of kubernetes when we build them that way so does this mean we should just get rid of security?

A

Now this is what your clusters look like. When you don't apply security principles to your clusters, you have Bob having a heart attack, John I, don't even know what he's doing he's starting more fires. There's people running out the door I like to call this cluster admin as a service where everybody becomes cluster admin. That's not also not a good way to build and run our clusters Oh.

A

So what can we do to kind of bridge this gap to mode both maker developers, operators, productive, but also our Security's security teams? Happy we can use something called a mission controllers.

A

A mission controllers is a way to extend what kubernetes can do, especially at the object level, so who's, actually using a mission controllers here, come on everybody should be raising their hand. It's a trick. Question. There's default of mission controllers, you may not know you're using a mission controller, but you actually are. These are the default of mission controllers for 1.12 I? Think the same for 1.13 here, I'm not going to cover all these in depth.

A

Here, the two I'm going to focus on is mutating a mission webhook and also of validating a mission webhook and we're gonna learn about how we can extend policy and compliance by using these two web hooks.

A

So what are a mission controllers by the kubernetes websites mission control controllers, a piece of code that intercepts requests to the kubernetes api server prior to persistence of the object, but after the request is authenticated and authorized? So what does that actually look like?

A

So what that looks like we have Jane here Jane's going to make a request to the kubernetes api. It's going to go through authentication with authentication, it's going to make sure she belongs there. It's then gonna go through authorization. What does she actually have access to? Then it's going to go through the mission phase. The mission phase is going to clewd mutating emission mutating emission is when we may mutate objects. So if I deploy a manifest, it may add additional labels to that manifest.

A

For me, oh then, I'll go through object, validation, who's ever getting troubled, object, validation, who's ever like added a tab in there yeah mo okay. That's when you get object, validation errors, then it's going to go through the validating admission before it's ever stored in at CD. So what the mutating webhook and validating webhook allow us to do is really add more logic. So we can, you know, building custom logic that may mutate an object or it may deny an actual object.

A

So validating webhook in mutating webhooks are both perd dynamic emission control. The validating webhook allows you to intercept and validate requests, so example of this may be restricting a resource creation. So, for example, let's say we're creating a load balancer, but we don't want to allow public load balancers. We just want internal load balancers. We could deny that if it doesn't contain an annotation for a internal load balancer, the mutating webhook is a little different, because a mutating webhook is when we're going to add additional field, so we're gonna actually patch that object.

A

So this example of this would be something like Sto, I'm sure everybody's heard of Sto. So I don't need to explain that so when it injects a sidecar also it injects that Envoy proxy as a sidecar. It's actually doing a mutation on that object. You deploy.

A

You need to think of policy enforcement differently than things like Network policy, pod security policy, where those two are really focused on the data plane policy and emission control is really focused on the actual object. So when we go to deploy a pod or a service or ingress we're creating an object, so we want to focus on policy around that.

A

So this is all super awesome. Now we can create custom logic to apply policy to our clusters, but to do that, we're going to read the go code here. Actually, there's nothing important in that code. The only number I want you to remember is the number down there in the left corner 198.

A

So this deployment actually was a mutating webhook to automatically add annotations to a load. Balancer, so it'd be an internal load balancer and to manage this you would have to you know, create a deployment within your cluster or you could actually host this. It just needs HTTP endpoint. You could host this on, say like a function as a service.

A

So how can we get policy enforcement without writing a bus bunch of custom logic like we just saw there, so we can use a general purpose policy engine. The general purpose policy engine is open policy agent. How many have heard of open policy agent?

A

Okay, so about quarter of the room they have a booth in the hall. I would definitely recommend going to visit them. But what open policy agent is it's a CNC F hosted sandbox project, its general purpose and the reason they say its general purpose, because it's just validating JSON data, so this is just for kubernetes. This could be to validate your terraform plans. This could also be to restrict access to SSH, but today, we'll talk more in the context of what it provides for kubernetes.

A

It uses a declarative policy. Language called rego and I'll cover a little more of what that is, but it essentially was built off a query. Language called data log from like way before my time it was like in the 70s sorry, if I called anybody old for being born in 70s.

A

Ok, so how the open policy agent kind of works here, so we're really decoupling policy from our application and those policy decisions were offloading. So a request comes in the enforcement happens at the service layer that service layer refers to could be our kubernetes api could be a custom API.

A

It could be a message queue any of those. Once that request comes into the service, it sends that policy quarry to open. So the open policy agent, the open policy agent, compares its actual policy with the data it has and then based on that, you may have a true or false return on that, so example may be of we create a service, it has a specific annotation. We create a policy to say with that specific annotation here called dev, we're gonna, add cost Center 8000 automatically to that.

A

So when it goes through and makes that policy decision, it's automatically going to annotate that object and send it back to the service.

A

So example, Rago policy. Here, if you look at the Rago, the the policy, it's pretty easy to think about you're, really focusing on the logic and the data and you're not focused on like methods. It's not a programming language. It's a query. Language.

A

So to write this logic in this example I'm doing a deny rule. So you see here it's a deny. Then we say what kind of resource our resource is going to be of any kind of any namespace of any name. Then the resolution for this policy is going to be to patch that object and we go down and look at see how we match these objects. It's going to look at the kind namespace and then it's going to look at where I define matched object, metadata annotations. So any time it sees mr.t.

A

It's automatically going to add this path and patch with a cost center and the value of 18.

A

So who starts to manage all this policy and it's going to be different in every organization, but typically this is kind of the personas. You would see managing the policy, so we have our developer. Ice Cube Ice Cube is huge at cube. Con I guess Mesa sphere made the hem huge here they're going to deploy apps so they're the one creating the objects. This may also be operators too. Then you have your platform operator. Your platform operator is typically the one it's going to create and maintain that policy.

A

Then over arching you may have security security is going to be able to audit the policy, because one of the features of a allows you to actually audit policy and not just apply a true/false to a policy.

A

So where this gets us is to building immutable platforms, because now we can wrap up policy in code now for lack of a better term. You can think of oppa's policy as a service. Here, we've built apps that are immutable. We've built our kubernetes clusters in a mutable way, and now we've built policy around those clusters.

A

So if he thought I was done, there is more to this, and that is the kubernetes policy controller. The kubernetes policy controller was built by a team with in Azure. It will be moving officially under the OPA project, but you can still find its home at the URL below. This is a community project. I think that's one of the most exciting things about OPA in the kubernetes policy controller is now we everybody can build knowledge into this and build policies for this.

A

If you just think of everybody in this room created, one policy think how much of each of your companies could use policies.

A

This is something I think is going to be policy around Cooper knaves will be something talked about a lot over the next year if you haven't already seen it, but we're getting closer and closer to being there.

A

So what the policy controller does is it has both mutating. You can do mutating and validation. You can also audit. You know your policy to make, so we can go out there and see things that don't follow our policies. The deployment kiss consists of three containers: OPA, our queue management controller opens open policy. Agent container then cube management. It's going to load our policy into OPA, and one quick note on OPA is some might think. This could be, you know, add additional latency because it has to go validate policy. Opa, it's policy runs in memory.

A

This is one of the great things to see from the OPA team. They really considered for things like latency, sensitive applications and how to architect OPA. For that it adds very little overhead, so some examples that you may use whip the kubernetes policy controller things like white listing or black listing container registries. So if you only want your users to be able to deploy container images from your corporate image repository, you could believe you could whitelist that and blacklist anything else. You want also not allowing conflicting hosts for ingress.

A

So if team one uses a specific host address for their ingress team, two will be denied to be able to do that. We can also block things with the authorization module. So maybe we don't want to allow anybody to exact into pods that live within Cube system. We can also use it for authorization there.

A

All right so from demos all this is just a quick overview of the demos we're going to show here. So in the first demo we have George Jetson and George never follows policy. That's been documented, unfortunately, in SharePoint and mr. spacely's pissed about this. So mr. Spacely is going to enforce George Jetson to always use an internal load balancer. So in this demo we're going to show how we can easily enforce any type of service deployment that states a type of load balancer to be only a intern announcer.

A

In the second demo, Elroy we're going to he's going to make a request for ingress and then based on that it will check for any conflicting host names here and then Judy here. We're also going to restrict access to specific CR DS. So the CR dsl show is calico CR D, so we're going to restrict what actually Judy can do with those custom resource definitions and if you couldn't tell I, was a 80s kid by all my eighties references.

A

Alright, so to show her first policy here, so IRA have a few policies defined in here, so I'm, just gonna pull back what these policies are. I have a annotate and an ingress conflict here.

A

So, to show you what this annotate object looks like here, it's going to deny it's looking for kind, the kind, the type of the object type we're looking for is services in any namespace and any name. So we could limit this if we wanted to on a particular namespace. Also it doesn't have to be cluster wide. Then we're going to match this object with any labels that say internal app. So if my app has a label or my service has a label that says internal app, it's automatically going to patch that object with this path.

A

So this path is saying, add a annotation of service, step, beta Cooper, naze, dot, IO adder or slash as your load balancer internal, and that value is going to be true.

A

And here it's service I'm going to deploy here, as you can see my kind of service, it has a label of internal app and that's all. We really need to look at there.

A

So I will apply this service.

A

So it already created it. So let's actually check out my service here, I'm just going to refresh it here.

A

We're gonna do this on.

A

Alright, so you saw my actual object: I deployed did not have an annotation. Now it has this annotation of load, balancer internal. So essentially what I did there and what took.

A

Let's say 10 lines of a query: language I've just restricted anybody from deploying a internal or external load balancer within my cluster. If you remember that slide back there I told you to remember one number: how much how many lines of code was that custom logic?

A

Yes, this I did within 10 lines of a query language. That's just matching JSON data. How awesome is that I mean.

A

All right so Oh running out of time here. So let's go to the next one here, so my next service or policy I'm going to enforce here. This policy is a little more complicated. So this policy is actually going to take data from namespaces and also from a object that a that a end user would deploy. So in here. What it's looking for is matching anything in ingress and then it wants a valid host names. So, when I deploy it ingress, it should have valid host names that can be applied to ingress.

A

What its first going to look for is a white list, so this white list would be an annotation on my ingress object and that white list would define the URLs are fully qualified domain names that I can utilize for that ingress host.

A

It's then going to go through and match a pattern here, so it's going to match this exact pattern and what it's looking for is.

A

Is this white list here so if it does not match this white list here, it's going to deny that deployment and I've already created this namespace here. Why that's important is we'll see in just a second, so first I need to create this policy, so the policies just being deployed to the OPA namespace OPA namespace, is where that OPA agent is the cube management which is going to load that policy and also our kubernetes policy controller.

A

So I go ahead and create that service and we're going to hit the audit end point and I'm I guess my connection probably died for that.

A

Okay, so the policy controller enables a audit endpoint here, so in my audit endpoint, even though I haven't rejected any policy or anything, it did go out there and look at things that did not meet this policy. So today you could go out, write a policy, apply it to your environment and see which things don't match up and violate policy.

A

So now, if I want to actually deploy this ingress, so I'm, a new user I've created a ingress I created a hostname I will go through and.

A

I'm going to deploy ingress really bad, so it automatically comes back and tells me that my host name is not a valid host name. So it's not on the actual whitelist here. So if we look at that really bad ingress here, you can see its host name is Acme Corp comm and what our policy says in that namespace it has to be either a wild card, Q a wild card internal and only when I match that will it actually submit that to @cd and persist that actual object.

A

Okay, so we're running out of time on the demos here so I'm just going to show the policy for ingress I won't actually deploy it for or not for ingress. Sorry, the CR DS here this is another very easy policy. That's going to restrict somebody from having access and specific permissions to CR DS, so we can get very granular here and how we allow users to modify a CR DS. The important piece here is that you may have some system accounts that actually do need to update and modify CR DS.

A

So here we're matching up here. We're actually matching the CR D. So this is a calico CR DS down here. Will we always set a resolution so a message, so it kicks back to the end-user, so they do have feedback of what actually failed here. We're going to do a not match. So we want the system, cube, controller manager and the cube scheduler to be able to actually access those CR DS and then allow and kick any other user out from actually creating deleting or patching those actual CR DS.

A

So some of the good, the bad and gotchas about using an open policy agent and describing your policy this way. If you look at the OPA approach, allows you to decouple policy from applications, which is a good thing, especially when we want to talk about immutability its general purpose. I think this is a huge benefit, because it's not just for kubernetes.

A

You can start driving policy in other places within your technology stack so whether that be tariffs or more, if Co, linker D, there's integrations into those because we're just taking plain JSON data, making a quarry on that and returning a true or false, the bad. This one I had a hard time with, because I got so excited about it, but there there can be a learning term or learning curve to Rago. Well, that's the first thing: you'll probably run into and struggle with a little.

A

It's documented very well, and you can also go in the OPA slack channel and they're very, very helpful, and there you have to be considerate of latency. Also, if you have a very highly transactional system, there is some latency OPA can add. They did design it in a performant way. I would say: majority of your applications aren't going to run in latency problems. That's why I had a hard time putting it under the bad I. Think it's more of just a consideration. You should take from it ah got chills mutating objects.

A

You need to really handle with care', because when a end user deploys something they're expecting outcome a and they may get outcome Z. So you need to really think about when you're actually mutating objects.

A

So takeaways from this session is focus. You have to focus on security within kubernetes and just deploying a default cluster is not secure by default. You need to start thinking about the other layers you need to add on from security. This will benefit you, security teams and your company.

A

You also and I think this is probably the most important piece is instead of pushing back against security. Hope them understand how you can extend kubernetes to fit their models also and think of new ways to implement security in your clusters. Don't just settle for that. We used to do it this way or our tool works. This way look at new ideas in this space always treat your crew Bernays clusters as immutable. I can't harp on that enough, and this is just another way to get us there to having an immutable platform.

A

Like I said you can have you can do this multiple different ways build your own logic and use you utilize, the mutating webhook or the validating webhook. You can absolutely do that or you can utilize something like OPA in the kubernetes policy controller that can help automate this for you and reduce the management overhead of building this complex logic, I would also say, definitely get involved in this project. The kubernetes policy controller. It is a community project. This is going to be a big base and kubernetes over the next year.

A

So, if you're looking to get involved, this is a great project to get involved in other sessions. This week, the first ones on Wednesday the intro to open policy agent. There will be a couple customers actually speaking at that one and it will be with the maintainer zuv open policy agent. The second one will do a deep dive. We just kind of touched the surface in this session. The this deep dive for open policy agent will go very deep and how exactly open policy agent works. These are both great sessions.

A

I've seen a couple of met last few cube cons, but definitely check those out. I will be around all week. So if you want to buy me a beer feel free to, but if you have any questions, I love talking about this stuff so feel free to come up and say hi. Thank you.