Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OPA Introduction & Community Update - Rita Zhang, Microsoft & Patrick East, Styra

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

OPA Introduction & Community Update - Rita Zhang, Microsoft & Patrick East, Styra

Come to this session to learn about the Open Policy Agent (OPA) project. OPA is a general-purpose policy engine that solves a number of policy-related use cases in Kubernetes and the wider cloud native ecosystem. During this session the OPA maintainers will introduce the project and then provide updates on the latest and greatest features to land in OPA and OPA Gatekeeper.

https://sched.co/Uahw

A

Okay, so this is the open policy agent project, update and introduction.

A

Just kick things off with some intros. We have Rita, saying she's a engineer at Microsoft, beekeeper, maintainer, open contributor as well as me, Patrick yeast, engineer, it's Tyra and OPA maintainer, Oliver, slack and Twitter details. Are there please come track us down with questions so to talk about OPA, I.

A

Think the first thing a little bit is to give some context on what it is, what we do so the goal for OPA the open policy agent is to provide a unified policy enforcement layer that can be used pretty much across the stack, particularly in the cloud native environment, where you start to see dozens hundreds, thousands of services, interacting that probably all have their own authorization engines and their own way of declaring the policies, and you know it becomes a mess.

A

So in comes OPA, it was founded by styro open-source donated to the CN CF, currently incubator project. It's, let's see we're up to 70 plus different contributors. This is a thousand I think we're like 1200 or something people on slack. It's continually growing we're seeing a lot of popularity in a huge number of use cases. I think this is koukin. Most people hear you talk about kubernetes, so mission controller for kubernetes is kind of the main one that a lot of people.

A

Think of, but it's also being used in a lot of sea ice to be pipelines as well as things like micro service, api authorization, there's a ton of users, the you know the list here sort of speaks for itself. It's not just a pet project, it's being used in production in many large.

A

B

Probably also heard touring from star who just.

A

B

This morning and all the places that oppa is been utilized across the NCR projects, exactly.

A

Yeah, thank you so I give some background on oppa, but this is the intro. Some of you are probably wondering what is oppa. Why are we here? Oppa is a declarative, general-purpose policy engine, the very core is Rago, is torn I explained earlier. It's the policy language that you use to define your policies. Oppa is basically an evaluation of engine for them with a collection of services and other features around it. To do policy management. Authoring, editing. You know, logging, all kinds of stuff anything you need basically to build out the full authorization service.

A

Thank you so it the the engine itself is already an NGO, primarily it's being used as a library or as a sidecar. So oppa does have a HTTP service that you can start up. You can. Next, you know evaluate queries. You can push data into it policy, all these things or embedding it with go. You have access to all the same things via the API is there's another one. That's not on here right now, Turin announced at the keynote, which is the Waze and integration.

A

So you can start to evaluate your policies with any webassembly runtime next up the the management API is. This is really where things start to get pretty cool you're evaluating the policies is one thing, but knowing which policies were being evaluated for any particular decision being able to centralize your policy management, pushing updates doing you know all kinds of profiling auditing. You know, there's there's groups that are looking into doing, like formal verification, all kinds of cool stuff for these policies, and that's that's really the icing on the cake.

A

Last but not least tooling, if you can't write the policies, it's not very useful, so there are integrations for vyas code as well as the Larry go playgrounds which hopefully anybody's writing. Borrego policies is seen and the a rich CLI package, because it's that's kind of the primary entry point for any new features. No, but.

B

I'm also curious how many people here have used old party. That's.

A

Good question yeah: it's.

B

Awesome I'm just curious how many people have tried OPA gatekeeper, awesome. Thank you.

A

Okay, so we talked about what open is, but by itself it's not super useful. You have to have things to evaluate policies. The inputs for these policies are basically any JSON. The output also any JSON. Usually this boils down to some API requests. Some admission request. We have some some structured data, that's coming in and says this is the request being made. This is the you know, job that came with the request. It's the llamó for the admission request, the output being a allow.

A

You know it's a boolean, yes or no, or in the case of something like mutating webhook. You have JSON structure that comes back out and defines what the result should be. The service box in here, somewhat ambiguous. It really can be pretty much anything the the queries. These requests, like I, said before. If you're using the send a sidecar model, most people are doing HTTP requests, it's very easy to extend to OPA. So there's also like the the sto envoy, integration uses, G RPC, it's really it's very flexible.

A

In that regard, you just have to open up with transport path, to give JSON to hope, yeah to give JSON to open, and it's pretty much the integrations hopefully did this show of hands. Well, we'll saw the keynote this morning where we probably saw this same slide: okay, cool! So not super new, as you guys know. Basically, the the integrations are becoming like pretty wide and pretty deep.

A

There are I think you'd be hard-pressed to find some category in the CN CF landscape, where you don't have an existing integration with OPA or an opportunity to integrate.

A

The primary use cases- these are things that we see a lot so, if you're, if you're, still kind of within the new to OPA, wondering how you might use it, how do you leverage this in your system? Admission control, like I, said before we're here at KU comm. This is like bread and butter. Everybody who's got a kubernetes clusters should be doing some kind of admission control.

A

It be like next kind of big one API authorization we're seeing a pretty big uptick with, like the Envoy, his tio plugin, basically being able to push these policies to getting enforced outside of some service that you implement, but still very close to the service I you kind of want to avoid having that long jump, because it's a API request, you don't want to sit there and have a 5 millisecond delay or something like that. Just to authorize it. Data protection is kind of an interesting one.

A

You see, there's a I could talk, I, think they're, all recorded from the OPA summit. There was one that interested that's pretty cool about doing their their data protection. I'd recommend you guys go check out. The recording for that, when they're posted I, don't think your post there.

A

Ok, so this is a little bit more on the updates, there's a lot of like since, since the last coop con well, especially since the last North America coup cough since, like Barcelona, there's been a lot of interest in CI, CD pipelines and open policy agent. There's tools, that's that are becoming pretty much sure, like contest and other, like integrations with spinnaker there's. A new project called boomerang that just got released.

A

A IBM publish this not too long ago, as well as things for terraform, and this is all using basically the the inputs for what you're gonna use in your terraform templates for your gamal deployments. For you know any any kind of like CI CD pipeline validating them, using OPA with policies to make sure that you know you're compliant and then checking that pushing it as far left as you can so that your developers get that immediate feedback, the change they're trying to push they. You know it gets caught.

A

First thing on the pull request by the CI system. The next bullet point has been a ton of changes for having OPA support, object, storage for bundles. If you're new to OPA, you may not know what bundles are if you're using already this might be exciting. Basically, the de-facto way to push and manage policies with OPA is using bundles.

A

You right now need an HTTP endpoint to get them from there's support built in now for OPA to do the signing and verification authorization that you need to pull them out of s3 GCS as well as spec and support for OC I. So, basically, in a you know, CI image registry could also contain policies as layers in there.

A

Speaking of integrations, new integrations, there's now an integrations page on the Oppo ducks. It's not complete, I think anybody here who knows of more or works on one. Please do go at it, so I'll just open on the website for OPA, as you can see, it's pretty wide, but largely dominated by kubernetes things. There's astillero on the way the kubernetes admission controller with gatekeeper, which we're gonna talk about next kou-kun Eddie's and OPA. They go hand-in-hand, the admission controller.

A

It gets used by pretty much anybody that I know. If that's doing you know like real production, the deployments in one form or another, it may not be oppa, but oppa gives you an extremely powerful tool to leverage doing external leveraging external data to do contextual policy enforcement for these objects, which leads us to get deeper. Alright,.

B

So for the rest of this talk, we're gonna talk about the gatekeeper project and also some updates for folks who have been using it in awhile for a while. So and we have a demo, it wouldn't be a talk without a demo course. So what is gatekeeper so where I keep come on right, I'm. Sorry, a lot of companies probably have their own company policies, maybe some compliance requirements or legal requirements, and and and so a lot of admins asks how do I actually enforce these policies are my Trinity's clusters control.

B

My in years can actually do echo echo and, and how do I actually do this without writing an admission and webhook every single time? And so that's what gatekeeper is?

B

Is a customizable criminales omission web hook that enforces policies and ensures governance for your all your clusters and what it does is allows you to write your policies as configurations rather than code, and you don't have to recompile every time so some of the motivations behind the project, as I mentioned earlier web with gatekeeper, you should be able to control what end-user can actually do on your clusters and it helps admins and a lot of companies to ensure that their clusters at any given moment is actually in conformance to the company policies.

B

And when the clusters are not in compliance, then we actually present an audit capability that allows you to identify the the resources that are actually out of compliance and, last but not least, is actually something we developed since Barcelona and cube con, which is enable a lot of the cluster admins.

B

The ability to roll out policies in a production environment, as all of us know that a lot of policies are actually created or the thought for new policies are actually the requirement for new policies are actually there after your cluster has been running for a while right. So then, a lot of people came up to came to us and said well how do I actually test my policies and build confidence in my policies in these production clusters without actually impacting existing workloads?

B

And so we developed this concept, eat record dry run which allows you to deploy your policies in cluster, and you can actually look at the effects of the policy on your on your cluster without actually impacting marryin users. And last but not least, we have the cluster admins and we have the developers right. So how do we make sure we have a cluster that conforms to company policies, but without sacrificing developer, agility and autonomy?

B

So how do we get? How do we get here? In fact, in 2017 steyr, released v1 of gatekeeper and for a lot of you, I think you're, probably running the v1 in production, which has started back in December 2017 in the cube corner in Austin, and that is the version that is running acute management and so in December 2018 in Seattle.

B

You know into a capital, one basically started using v1 in production and that's around the same time that Microsoft released v2, which is called kubernetes, kubernetes policy controller and it was donated to OPA and around the same time, Microsoft Google, Styron and others basically started to collaborating on this project and back in May in Barcelona. That's when we released the three, which is build upon the constraint framework, which allows you to deploy your policies as CR DS rather than config, and then now we're here. November cube con San Diego.

B

This is when we actually release 304 beta. If some of you may have been running it for a while and we're gonna talk about some of the new features, so v3.

B

So v3, this is a diagram of what what the gatekeeper architecture looks like. So, as you can see, at the centre of gatekeeper is opa and open is the again the policy agent that actually runs the logic of your policies and the data is then replicated into OPA from gatekeeper and, as you can see, the policies are now created as CRTs as opposed to config Maps, and we created this concept of policy templates and policy instance. So what what do they mean so think of a function?

B

So a template is a function where you write the rules and you write. The Ray goes the rules that describes what kind of rules that your policy should should actually execute, and then the instance. The policy into instance, in this case is, is the call called ler to the function that passes in all the parameters.

B

So what this allows cluster admins to do is be able to reuse the policies over and over and just by passing in different parameters to the to the actual policy instance and we'll see that in a little bit, we'll have a demo that demonstrate that and another thing that gatekeeper does is in the case where you want to have referential policies. So, for instance, if you want to look at do I have unique. Does my ingresses have unique hostname right as an example, so for those type of referential policies?

B

Okay, it is really great for that, because it it's basically a cache of the resources in your cluster and allows you to actually do these comparisons between the object that you're trying to create at a mission time in as part of the mission request and all the resources that already exist in your cluster right. So the so I guess you're. Looking at that way, so yeah, so the the admins.

B

Does it keep CTL, create or apply the resource get submitted, the API server and then the API server offloads, the the calls, the gatekeeper admission web hook and then the gatekeeper.

B

Then takes the request and pass it into oppa for an actual policy execution, and then okhla returns information back to the api server as a deny or allow, and then all of that information is basically how gatekeeper works and, as you can see on the very very left, we also have the audit capability, which basically runs all the policies against all the resources in your cluster every so often, and the auto results are then populated as part of each policy and again we'll see a demo of that in a bit so I'm curious how many people are actually running v1 right now, okay, so this slide is for you.

B

So we got a lot of questions on flak about the difference between v1 and v3. So this is a table basically describes that and again so v1. You have cute management again, it's something that we're continuing to maintain, but not a lot of new development there and then v3 is where we're doing a lot of new development and we'll continue to add advancement there and in terms of policy management in in v1. The the raw ray goes. The raw logic basically are stored in config Maps and in v3.

B

Now there are creative CRTs and the wall. Regular ray goes are stored in conferring templates and again you can basically pass in specific policies that uses those ray goes as constrained as constraints as constraint. Instances and some of the feature differences. As you can see you know, v3 has still v1 v3, both have contextual awareness, referential policy and then v1 has validating a Mission. Control v3 has validating a Mission Control v3 has mutating v3 does not have mutating, yet we're still working on that and then both have more both sources. So you can write motive.

B

You can write. Array goes in a library format and then import those libraries in your rigor and then in v3. We have audit, we have dry run and, as we mentioned before, we there's also CI CD with comp test and then all in all this enables you to reuse the code. You already have right so yeah.

B

So some of the core functionalities of validating and mission control controls where end users can actually do in the cluster so when they actually create something that they're not supposed to, or they might know that hey I should have resource limits on these pots, but they just didn't do it, and so this is really for us to enforce them to make sure that they do the right thing context aware again, for because we have okhla where we're now able to do referential policies that looks that compares one object again to every other objects in the cluster and once again, constraints really allows us to prioritize and make it easy configurable for admins to scale and expand on the policies.

B

That's already written and content templates again is the source code of the constraints, they're easily shared testable and they can be shared across the community. What we evolved also down in v3, is we created this library of all the constraint templates, that's created by users in the community, people who have like similar requests and I. We can show that in a bit and the thought behind that. Currently, it's only a folder on there, the gatekeeper repo, but the thought behind that is.

B

We will eventually have a repo that has a library of all the kubernetes policies that people love and it's tested and it's supported by the community and last but not least, errs audit again. He allows you to look at on going monitoring and health of your cluster and detect any remediation, detect and remediate and existing misconfigurations.

B

So new features so again the new dry run feature allows you to test canary releases in a cluster in stages without actually impacting your users, and this allows admins to gain confidence in the policies that they're deploying and it allows you to have gradual, rollouts and we've also added namespace selector to narrow the scope of resources that are constrained. The actual policy can enforce to sir namespaces. Only and again, these features were basically requested by users of gatekeeper and policy library.

B

Is this concept of community develop policies and we've also developed bunch of alternatives to pot security policies? For those of you who might be interested in seeing hey I wonder if I could use gatekeeper to enforce pasa curative policies, as opposed to you know, turn it on all at once, and so we also added support for multi-source constraint templates to allow you to write your Rago, those libraries and import, the libraries within different constraint, templates and last but not least, we also introduced metrics.

B

So now we have metrics added to gatekeeper, integrated with Prometheus, so I will we have a demo in a little bit. That shows you. What are some some of the data that we're scraping?

B

Alright, so demo, so a fictitious bank that builds that has like high transaction a day to day and it's a very highly regular, regulated industry and both developers are admins, are not happy because admins want agilities and they want to do want quick turnarounds. Meanwhile, admins like well I just want to make sure my cluster is safe. I want to make sure everything that's deployed is actually supportable.

B

So how do we get around that so agile Bank has these policies, and some of these may be familiar to you as well, so I wonder if there's a way for us to make sure that all namespaces have point of contact as a label, so that in the future, admins can actually contact the people that have created the resources. Another one is: all pots should actually have upper bound resource usages, so we can limit what the impact a specific application can have in within the cluster.

B

Also for security reasons, all images should really be from an approved repository. You probably don't want your users to just pull from the internet right and all services should really have unique selectors and the last one. A lot of you probably have already seen, is ingresses really should have globally unique host names, all right so and so we're gonna go quickly into our demo.

B

A

I'm, a developer I work at the bank I create my new advanced transaction system. It's a very advanced project, but five weeks go by and I'm actually working on a new project. Now that that one was left behind long forgotten.

A

B

Our intrepid amman finds in it and says: well what is this a VM transaction system, and what do we do with it and what do we use it for, and so we go on a three-day quest across all the departments and in the end we find that the project was scrapped, never again system in and now the admin says well I've heard of this project called OPA gatekeeper, I, wonder if he can actually help me with this, and so, let's see some of the constraint templates, that I will be deploying.

B

So, as you can see, as we mentioned earlier, those were the policies that add your bank wanted to enforce, and here we have bunch of yeah. Moles basically describes what those policies are and then we'll show that in a little bit- and so first I'm going to apply and deploy these templates into my cluster and once they're deployed these constraint templates can then utilize and- and let's see what a constraint looks like sorry. These are the constraints that actually use the constraint templates and, as you can see, we have container must have limits.

B

Owners must be provided and production repos must be coming from a particular repo and uniques or services must have unique selectors. So let's take a look at what a constraint actually looks like here, as you can see, you using the gates, require labels, constraint, templates that I've already deployed previously up here right and so using this kind, I'm, creating a customer resource of this kind and I'm matching on the namespace resource. So what this constraint is saying is for namespace resources.

B

These are the labels with this Fredericks that I want to make sure that all namespaces must have right and so again imagine you can deploy this in your before your cluster as well, but instead of namespace you could have on services or or pots, and then you can have your own parameters and regex as an example.

B

So I'm gonna go ahead and apply this constraint.

A

So now, when I try to create a new production environment, I'm gonna get an error back because I did not add the label to this. The error is descriptive, it says denied all must have owner and the nice error message that says all namespaces must have an owner label. Knowing this I go back I edit, my file make it file, I didn't have one before and now. I have one with the label saying the owner is me: demo I can attempt to deploy this and it will be allowed through.

B

So now he's able to actually self correct to get an admin to create a namespace and I just forgot that I'm supposed to start.

B

So while this is running continue.

A

So we've created the new namespace everything looks good now: I'm gonna try and create some resource and I'm gonna use a bad repository. I get same kind of error message: I get that immediate feedback without having to ask the admin. What to do it says: there's an invalid image. Repo tells me the repo, and then it tells me which ones are allowed then can go and edit my file and deploy this one using the correct, supported repository is allowed through.

A

Similar issue with the duplicate service I'm, going to attempt to create one it's gonna have the service name is gonna, be a duplicate here with the selector controller manager, we're not allowed to write that without gate keeper. This would be really bad and sure enough gate keeper says no. It says that you're not allowed to do that. It's alright News and.

B

It tells you which namespace and another project is actually using it. Yeah.

A

Alright, so we've seen the key-keeper enforces those policies. All's well in the world like most good clusters, there's an outage. Bank is down for hours.

B

Well, we have no idea where the resources why the resources are without resource limits now they're causing issues in production. We need to get all the resources in the cluster that don't have resource limits and there's not really an easy way to do that right now. So, let's actually see if the auto results with within the constraints have limits constraint actually can give me that information, so I'm gonna go ahead and get the constraint must have limits, constraints and.

B

And, as you can see here, the audit- this is a time stamp that the last audit ran and, as you can see here, total violations is one and the violating resource. Is this pot running in the default namespace and this container does not have any resource limits. So this is super helpful. Now I can just go in and update and make sure this pot has the right resource limits.

A

This week's go by. We now have this new policy. We want to roll it out to production, I'm sure anybody here, who's already using a web hook, has felt the fear when I hit enter. Is the cluster gonna go down? How.

B

Many support tickets am I, gonna get yeah.

A

Yeah, so we need to check. We need to make sure that all the host names are unique, but we don't want to just deploy an enforce.

A

So now we're going to yeah.

B

Introducing new cause is dangerous and it can often introduce any breaking change. So how do we gain confidence right and we can do that with try.

A

B

Dry run future in gatekeeper, so we can actually preview the effect in the cluster without impacting the workload and our users. So let's see what the actual policy looks like first, so actually, first we're gonna go ahead and deploy this policy, the constraint template and then next we're gonna deploy the constraint.

B

So let's actually take a look at what they look like right. We keep talking about regulars, but we haven't actually yeah. How could that be anyway? So here's the kubernetes, unique ingress, host constraint, template and, as you can see here, this describes the CRD spec of the gates, unique ingress house kind, which will be later used by the constraint to pass in parameters right and, as you can see here in within this constraint, template we talked.

B

We have the actual Ray goes here, the source code and within here it we actually have four resources of the ingress kind right and that the host names are identical, but not the same ingress right where the namespace and the name are actually different. So in those scenarios we know the ingresses are different, but the host names aren't the same.

B

So go ahead and just reject this as a violation right as a violation and display display an error message: ingress host conflicts within an existing ingress and return the host name to the user so that they know why it's complaining.

B

So next, let's take a look at what the constraint looks like right. So, as you can see here, the constraint provides parameters, but it also provides hey here. Are the resources I want you to actually enforce this policy on and, as you can see here see here, we have the ingress kind and all the API groups that kubernetes offers, and last but not least, this is the key thing that enables dry run feature which basically says for this particular policy. I want to make sure the enforce action is dry, run, not enforce.

B

So let's take a look at what that looks like so before we do that, let's see if we actually have ingresses in the cluster. Oh looks like we have two ingresses in the cluster both deployed in a default namespace, and the host names are exactly the same. So this is pretty terrible right, I, don't know how they have their applications up. It's.

A

Not working this.

B

Is terrible so next, let's see we now look at the unique ingress host constraint and within this constraint, as you can see, we have total violations and they actually identifies the ingresses that share the same host name, and this is super helpful because, as an amen' who deployed this policy now I know hey this policy actually works and in the developer.

B

At the same time, let's see if what, if I try to deploy another ingress, would it block me right so here we have a bad ingress, as you can see same host, name just different, ingress right and because we're in dry run mode. It should not block me all right, so it was created and everything works, but I felt some confidence in the policy that I just rolled out all right. So.

B

A

Gonna say we just started Prometheus. What was it doing.

B

Good question: so we have: what do you make sure it's up and running? So we add our the metrics feature upon request by yours truly, and so here, as you can see, we have some metrics that is scraped from gatekeeper that really helps admins again for the cycle. We are focusing on admins like how do they get more insights into how gatekeepers running in the cluster? What are some, how many total constraints are there that people that we have in the in the in the cluster? How many violations we're getting per constraint?

B

And then you know in terms of requests, are we actually getting denies or are most of them allowed right? So, let's take a look. What that looks like so, as you can see, these are the allows, and these are the deny.

B

So these are the requests that gatekeeper denied within the cluster and then, let's see constraints, I wonder how many constraints we have and, as you can see as we deploy earlier, we had four constraints that were denying in running in deny mode so enforce mode, and then one constraint that we just deployed the ingress host name, one and now it's in dry, run mode. So this really gives a holistic view to admins hey.

B

These are the gatekeeper, related resources running in your cluster and here's, how they're doing and where we will be adding more more resources here so that we can do a lot more with the Prometheus reports and if you have any and if you have any like thoughts around this and things that you want to see in the project. Please come talk to us after this or in the myth.

A

B

The maintainer stuff yeah all right so with that.

A

So I think we're getting close on time. That's not so.

B

The current status for gatekeeper is were so in beta and for the next release cycle, where we're going to focus on production readiness, so you know think of you know again. This is why we're working on metrics, you know h.a.

B

Things that will that make sure that this is production ready before you actually roll it out to your clusters, and then there are folks who are continue to look at mutation and another feature that we're really excited about, and we really would want more feedback and requests and user stories on is external data right. So imagine when you get a mission request in addition to the request itself, that has like kubernetes information.

B

What, if there's a way where I can call an external data source and get data into open and also use that information as part of my policy, that would be pretty awesome. So this is one area, we're really looking for user feedback and to drive this feature home and then, of course, more other features, more metrics and more policies from the community, such as yourselves, and also more developer tooling around this, and some people also mention authorization, but we haven't. We heard like one request, so if you this is something you're excited about.

B

Please also come and talk to us right.

A

So I think just wrapping up now come join. Us has Reedy's been saying. If you have use cases features you want to see, please don't hesitate. File new issues on github come join us on slack. There are weekly meetings for oppa as well as gatekeeper, they're, all open. Anybody can join and chitchat with us.