Cloud Native Computing Foundation Cloud Native + Open Source Virtual Summit China 2020, 11 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Policy Management in Kubernetes - Moving to a Centralised Management.. - Olive Power & Rachel Leekin

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Policy Management in Kubernetes - Moving to a Centralised Management Model - Olive Power & Rachel Leekin, VMware

Policy management of enterprise resources is hard. The scope and the vast number of resources to which policy should be applied makes it so. Current technologies in this area include OPA and OPA Gatekeeper, which bring to table the ability to discreetly define policies around workflows such as access, security, licensing and auditing. We are seeing more offerings around central policy management that allow creation and management of policies spanning multiple Kubernetes clusters and namespaces In this talk we will review the current offering in this space, and how these are being applied in production. We will discuss current development and where the future development areas will need to be. We hope to provide a comprehensive outlook on how to deal with Policy management for your Kubernetes estate, that will satisfy the current policy requirements of current and future workloads.

https://sched.co/cp9a

A

Welcome to the cloud native and open source summit, uh china 2020. um today's session is about policy management in kubernetes and how it's moving towards a centralized management model. um My name is olive power and I'm joined by my colleague, rachel leaken, and we both work at vmware.

A

um We're going to talk a little bit about myself, just really quick. My name is oliver. As I said, I work with kubernetes at vmware, I'm based out of singapore. um I've had previously worked with heptio and red hat, so I have a bit of experience with kubernetes and about policy management they're in over to you, rachel.

B

Hi, I'm rachel leakin, I'm a kubernetes field engineer here at vmware as well, and I previously was at ibm as a cloud architect. So I have a couple years of experience in cloud native development applications. So I'll pass it over back to olive to take us through the agenda and the rest of our session.

A

Thanks rachel, oh so so today we're going to quickly cover well, you know: why do we need policy management in kubernetes and then we're going to talk about how we sort of implement policy management within kubernetes and we're going to specifically today focus on open policy agent, which has quickly become one of the most adopted strategies for implementing security policies within kubernetes and then rachel's going to cover how we centralize this with opa, how it works, and then we hopefully leave time for some q a at the end.

A

So diving straight in you know: why do we need policy management in kubernetes, so for any platform? You're running and kubernetes. Remember is a platform for building platforms and whatever platform you're running um security is always a concern both from the infrastructure side and from the application side and in kubernetes everything is processed through the api. So we have a an option to start implementing security at the api level and secure kubernetes already has a basic implementation of that where it uses admission controllers to help you define and customize what is allowed on your cluster.

A

So, let's just have a look at that in a little bit more detail, and this is the flow of an api request in kubernetes, and so the request comes in and the authorization is checked to whether that request is allowed even to be processed and then there's two there's a the first gatekeeper, if you like um to to deciding in kubernetes whether that request is valid, is putting it through a mutating admission, which means that uh a controller is is, is watching that request and checking the content of that request or the specification of that request and seeing, if there's anything in there, that's not allowed, and if there's something in there that's not allowed.

A

You can implement some logic to change that value to to what is actually allowed. So something is specified as having a limit, let's say, for example, a capacity limit of two gig, and it only should be specified as one gig, then that mutating admission, controller and logic around it can be implemented to change the spec of that part to change it from 2g to one day.

A

So that's a mutating admission, so it checks whether it's right and can edit the specification before it can continue on its sort of uh life cycle and then then, after that, the the schema is then validated again to make sure that that editing of the pod, spec or whatever specification is valid. And then it goes through a validating admission to make sure that that specification is still allowed within kubernetes. Now. The difference here between mutating admission is that it's just a validation. Is this value allowed?

A

So in our previous example, before we've mutated it from one two to one, because that's the implementation that we've specified um that validation admission would would say that it's more gay and allowed, and so it would allow it through to lcd persistence.

A

However, you can you can implement code at the validating admission stage and just like we did at the mutating ambition stage, we can implement code at the validation to see if everything that we've specified is actually valid and, as I said, kubernetes has some um controllers that are implemented out of the box, but we can there's a lot of other use cases in terms in and around security that are not implemented as controllers out of the box, and so with kubernetes.

A

There are no hidden apis, and so that means that any request via the kubernetes api could be seen as valid. If there isn't a policy around it. That says it's not valid, and so that can lead to several sort of bad behaviors. So you can construct. You know, specifications in your yaml files that you.

C

Know are valid.

A

In in their format, but they may not actually be valid in in how you want, uh you know that action to be perceived within your environment.

A

So what I mean by bad behaviors is things like um you know, is the pod requesting too many resources, so you're checking to make sure that all your pods have been configured correctly and and have got the right reservations and limits in place to make sure they're conforming to standards, and so that can be validated at the at the validation admission stage and making sure that all your prospects are constructed correctly. um There's also use cases like uh are the images that are on your services that your containers are pulling. Are they secure?

A

So you can? You can implement things like a trusted registry that um that is specific to your environment, and only images from that trusted registry can be can be used. So if there's a case where a part specification file has been constructed incorrectly with the wrong registry, um you know kubernetes itself doesn't know about that. But you can you? Can you can start applying extra logic in your environment to make sure that only specified registries can be implemented and there's other things like privileges granted to service accounts again to kubernetes?

A

There's no uh right and wrong values for this, but you can start implementing your own code to start imposing the correct behavior and the correct permissions to service accounts again very important from security. Also from services are they you know, created the correct type, whether it's load balancer, you know?

A

Are you allowed to create service type of load balancer, for example, or should they all be cluster ip, for example, and again other things like pod permissions and the correct storage um mounts things that can compromise your security and again so so so by being able to implement custom code to be able to implement your own specific security requirements, your own specific policies that.

C

Are not available.

A

Out of the box, um this is available to you through the webhook code implementation. So this is a whereby, in the in the flow and the life cycle of an api request, we can hook out to external providers that can read some external policies that you've written and they can be validated through the api flow within kubernetes. So this gives the ability to have like a really um a lot of external policies that can be written.

A

So it gives you a lot of flexibility to be able to extend kubernetes to implement the policies that are very specific to your use cases and your security policies, and one of the players in this field is a open source project called open policy agent, and that allows you to start implementing your own custom code and your own custom policies, and it allows it to allows you to to to be able to run um and test these policies before you implement them in your kubernetes cluster, and you can start getting reports on which policies have been allowed and which policies have been denied and.

C

A

Interesting uh uh open source project and rachel, I'm going to hand over to you now to discuss a bit more in depth. What exactly open policy agent is and how it can be used to secure your clusters.

B

Thank you, olive. So, let's go into more detail on what open policy agent is or, as you all see me refer to as oppa some people say, opa uh officially, it's called oppa by their documentation.

B

So most of you, I'm making assumption that have you have gone through some a lot of the documentation already on oppa's site. But let's give you a little background on opa once again, it is an open source. General purpose uh policy agent that unifies, like you, said the public enforcement for the stack. So that's the kind of gist of the gener, the definition um it's a declarative, language, uh rego or rigo you'll, hear some either terminology or uh pronunciation and then also it in certain cases.

B

But the case that we'll focus on today is oppa will be deployed as a validating uh mission web uh sorry mission controller. So that's the case that we'll be going through today, so not just that as well. But what we want to discuss is is that you know with oppa and let me actually go back one more slide with opa you're, not just trying to um block people from doing things right or or just stop the bad behaviors. You have to look at it as your whole strategy behind.

B

How do you take that approach from blocking bad behavior or preventing bad behavior? It's a it's a tactic and a strategy that you have to deploy. So one of our colleagues, duffy cooley always says what problem are you trying to solve and that's the way that when I and oliver working with customers, we always take that approach of what problem are we trying to solve with the customer and, as olive said, a lot of the times they come to us and they say: hey.

B

We have these people who are doing uh bad things or have bad behaviors on the on the on the team. How do we stop them from doing that and we always kind of look at them and say: okay? Well, let's not just stop them right. Let's look at the whole approach in totality.

B

So, in the case like this, these are some examples that I've seen in the real world right rule case uh in customer sites. When we get there is we'll have platform. Operators will say: I want a developer to require approval before they try to create a service type load bouncer right, and this could be for any multiple reasons.

B

This could be because they don't have access to a cloud provider or it just doesn't fit that their needs for the application type or or sometimes they just want to make sure they understand what's happening before they before a developer, can just go and create this service type. That's one example: another example would be having developers only pull from certain uh registry to approve registry. So if you know that you want them to only pull from your approved, github or docker hub, you can do that as well and then the last one.

B

Another examples is just making sure that you have uh every resource. That's created has a label, so this comes up in a case where sometimes you need the label to indicate what business unit this uh developer is deploying application for, for maybe cost back or chargeback things like that. So you start off by just using these simple reasons and logic right. You don't really go straight to code, you just start off what are you trying to solve and we took this with the customer and we started making policies right.

B

So now you say policies, but what kind of policies you some the customer says hey. I think I can do this with psps or network policies and then the customer says: well, maybe we should use opa and they weren't really sure. So we sat down and we said well, what can we accomplish with psps and what can we accomplish with oppa? And then we came to the conclusion that you can use both right: it's not either or they complement each other.

B

So there will be times that it might be simpler for you for your team to just develop uh policies psps. It might just be a simpler, quicker way, and sometimes you need something. That's a little bit more robust that can go into maybe looking at external data or things that pspe very can't do so you would look at both but they're, not either they're, not mutually exclusive.

B

You can use them both equally, both in your cluster without um any conflicts right as long as you develop it correctly, and you look at it from the bigger picture right so like I said why not both so now, I want to take you to how we took those examples of what problems that we were trying to solve into how we turned those into code.

B

So I'm going to make the assumption that you've gone through some of the documentation right with oppa there's different types of opa there's different versions. So there's the cube management version. There's uh the gatekeeper version in.

C

B

Example, I'm going to focus more on cube management because it is, I want to say it's a little more developed and is being used a little more by customers right now than gatekeeper because of gatekeepers still somewhat in beta, so it's a little more stable. So these examples are how we it's more focused on cube management, which is deployed as a as a config map right on when you're in your cluster versus gatekeepers deployed as crds, and that also changes the way that you write your policies.

B

So it's a little different, which we'd probably say for another time another session, but uh but uh for right now, we'll stick with. What's this the simplest way for you to get started right, you want to get something that you you get started, get the process flowing so as you're ready to explore gatekeeper. You already have a solid foundation on how to build uh your policies so we'll take that first example of we wanted someone to. We wanted the developers to require approval before they uh try to provision a loan bouncer service type.

B

So, as you can see here on the slide, this is a very simple policy and that's one thing you should keep in mind is that sometimes people leave customers. They want to write these very complex, uh complex policies and to keep in mind that sometimes the simpler is just better right.

B

It gets their point across it does what you wanted to do and later on it saves you from trying to figure out complications, as maybe teams switch in and out, and someone else wrote the policy and then the next person comes in and they don't know what this person was. The rationale behind the policy, so sometimes just keeping it simple is the best method for for writing these policies.

B

So in this example, uh oppa has a policy, a package statement, and um this is just very generic right now, but you can name it based off of your strategy. So, for example, if you want you have cloud a and team a and team b, you can say this policy belongs to uh cloud a team, and you know they're gonna just write this portion and then, for example, if you have another policy, that's slightly different that might be focused on maybe um cluster api. Oh sorry, cluster ip, you might have team cloud.

B

Sorry cloud team b might have that policy, so you would change the package to that. It's based off of your strategy, uh but you want to keep it for. I said very simple as possible. So then you have operations of where you're trying to create and update a policy. So once again, very simple: uh the kubernetes resources right you're, just I'm sorry methods you just want to either create or update. So this is what that applies to. So if they try to deploy it sorry destroy a service.

B

It's not going to do anything because it's only looking if they're trying to create or update a service and then here's the deny so it's going to deny and in this parentheses here it gives you the reason of why. But it's going to deny if it sees a service come through and if it does one of those operations and if the the spec type of that service is load balancer.

B

So the reason why you see this here the reason is low bouncer and then you see. The next reason is the same reason, but it just says node port now the logic behind this. When we were creating, it was sometimes um you might want it to block both right, but if they have the same error message, they might not understand what what the actual, what they're blocking or what's being blocked. So we kind of we took the same policy and just had it customized a little bit.

B

So this way the user, inter interaction with oppa, is also positive as well, so they can see what's what's actually being blocked. uh So that's a very simple policy, then now what you would do with this? Is you got your policy? You want to test it locally before you do anything in the cluster. You want to really test it locally because it will just save you a lot of headache later down the line. When you know all your policies are working locally. So what you have here on the side here is a test case.

B

So if you're familiar with the traditional development, where you might see they're developing junit testing right, this is the same kind of feel you get. You have your use case here and your test case, and it gives you what would be a kind of like a manifest and yaml right in json here format and it's testing against to see if this this example would uh break the rule or be allowed in oppa or not allowed in opa.

B

So you can- and if you know it's a lot of details here, but you can see that in this case it should deny because it's a little bouncer type now for the next slide.

B

uh It's the same policy on this side, but if the, if the user tries to deploy a cluster ip service, it will allow it. So this is how you build out your test cases. You want to make sure you have coverage for for all the possible scenarios right, the more scenarios you have, the better test coverage you have before you get into your cluster and it really gives you a preventative measures so now that you've, after we've kind of tested locally, you would see you get a pause. You make sure all your test case pass.

B

You would to now deploy this same rego policy into your cluster, um like I said that will be deployed as a config map. Of course, you'll go through all the standard setup that you need to do to get opa deployed onto your kubernetes cluster. uh So that's making that assumption that all that has been done and your policy has been accepted by kubernetes.

B

So once all that's done now, you would start testing it in a possible real world scenario. So here's how we would do that we would have a yaml similar to what we had previously right, but now it's in yamo format, and it's just going to test this in this case. We I like to always test what what opa will let through first.

B

So in this case I know that it's a cluster ip example here, so opus should allow it to go through and and, as you can see from the screenshot, that's what it does it goes through and then, if my next one is what happens now when I want to do load bouncer type, and just so, you could take a look at it. I've tried to pass that through, and it gives me this nice error message, which was what you saw in the previous uh slide of what what my message.

B

My personalized message was so with all that being said, I want to take also a step back and give some pointers on what we've experienced with customers. When doing this is uh once you create the policies you want to look at it long term.

B

So when you create one policy, you know other problems are going to come up, so you want your policies to be somewhat open-ended, that you can update them later, that you can add to them later that you can also modify them. So you want to make sure that your policies are not so rigid that it makes it difficult to update as your circumstances change in the future or as your kubernetes cost clusters mature and your developers mature right with kubernetes.

B

So that's one thing that we've learned also testing with the in cluster right. Something like this. Where you see you want to make sure that you're testing a cluster that no one else is using, because if you block something that we've run into this, where you know you're, you might be testing with someone else and they might try to run something and it blocks it actually open, does a great job and actually blocks them.

B

So you want to make sure you're testing in a singular cluster and um and another another tip that we learned through this process is that that opa is, is always changing. So you want to make sure you're up to date with it. You might also want to consider putting oppa into your ci cd cd pipeline we've seen customers do that where they they have a nice pipeline on how to deploy these clusters.

B

These policies to their cluster through cicd, so things like that, make it a lot easier than manually going in and deploying each policy as a as a config map right, for example, and then. Lastly, uh just thinking about the future of gatekeeper, you want to make sure whatever policy you create now that you can convert them to gatekeeper, templates and constraints in the future. So that's something that we've had customers think about.

B

uh Let's see last part to this is um oops hold on one. Second, there we go. um I want to also mention that, oh, that uh strata has a oh on oliver gotta switch to.

B

One second here.

B

There we go so strata is the company that has the maintainers of opa, but they also do have a nice ui that they've been developing over the last couple of months. I've seen where you can take your centralized policies right and get a nice ui to them, and I think this is something that it's worth checking out for your team. If your team is uh looking to have more of a user interface experience versus command line experience, and then one other piece is how do you get involved?

B

What's next steps, just go straight to github, go to the slack channel, go to the documentation. Oppa is a very great community. I've had a blast working with the people on the team and they're very responsive, and the community is growing looking to grow. Looking for help looking for new ideas, so I definitely suggest if you have time to take a look at all of the community activities and get involved that way.

B

So all that being said, thank you and had a great time presenting today and we'll look forward to the q a.

A

Thank you, bye.

B

B

C

Okay thanks uh oliver and rachel for the great talk. Thank you thank.

A

You no problem.

C

And uh yeah uh do we have the qa session.

B

C

Oh okay: let's go ahead. Let's go ahead and check the qa zone in zoom.

B

C

Yep there are, there is one question.

A

How can I set open best practices in ci, cd.

B

I didn't see the q a session there, um so yeah, so one way that I've seen the customer use it as um with with their jenkins city platform, is that every time a cluster is created, they have opa a installed right through their ci cd pla platform through helm charts, and they also then deploy the policies automatically. You know they deployed the big maps right the policies as configmaps, so that was one way that they they chose to do it.

B

Does that answer your question.

A

I would say so and then I guess you can um you know you can edit those those policies right um and then just think they can just like blend into your ci cd pipeline as those policies change, yeah and grow, because I guess a lot of strategies implementing sort of a minimum set of policies first and and then start building on those policies. Once you kind of get comfortable about where you're going to implement those, whether they're an opa policy or whether they're you know, you know, kubernetes sort of the mission controller type.

C

C

A

Questions last session of the of the conference is a little bit quiet.

C

Yeah well, um we have. We have another like uh another, like a broadcast channel. I mean outside of this zoom, so lots of people are outside uh from this zoom meeting. Yes yeah yeah, so yeah. If you have any questions, you can email to our speakers and then they can answer any of your questions uh and thank you. Oliver and uh and rachel.

A

C

Yeah for the great partner yeah. Thank you.

A

Part of this conference. Thank you.

C

Yeah, thank you.

A