Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Secure Policy Distribution With OPA - Ash Narkar, Styra

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Secure Policy Distribution With OPA - Ash Narkar, Styra

OPA can download bundles of policy and data from remote HTTP servers. Once the policies and data have been loaded, they are enforced immediately. But how does OPA know that these bundles are coming from a trusted source ? How does OPA verify the authenticity or integrity of the policies and data included in the bundle ? An attacker can potentially include corrupt policies and data in the bundle and OPA would end-up enforcing those policies, thereby compromising the entire system. In this talk, we will describe how OPA can assist in the secure distribution of policies and data by creating a “Signed Bundle” - a bundle that is digitally signed so that industry-standard cryptographic primitives can verify its authenticity. Our demo will show an end-to-end flow of generating and validating a “signed bundle” and also how this reduces OPA’s attack surface.

https://sched.co/ekEY

A

Hello, thank you for joining me today for a talk on secure policy distribution with the open policy agent. My name is ash narker. I am a software engineer at styra and I'm one of the maintainers of the open policy agent and I care about developing secure systems that can be easily deployed scaled and managed.

A

In today's talk, we will look at a quick overview of oppa and some of its features. Then we'll talk about how to do policy, distribution and, finally, we'll see a demo about secured policy distribution with oppa. So, let's get started, what is oppa oppa, which is the open policy agent, is an open source. General purpose policy engine when you use oper, you are decoupling. The policy enforcement from the policy decision making, so your services can now offload policy decisions to oppa by executing a query.

A

So let's understand this a bit more using this figure. Imagine you have a service and it can be any service at all. It can be a kubernetes api server. It can be your own custom, server kafka terraform any service at all. Whenever your service gets a request, it's going to ask oppa for a policy decision by executing a query.

A

Oppa is going to evaluate this query based on the policies and the data it has access to and send a decision back to your service where it gets enforced.

A

So you can see that we have decoupled the policy enforcement from the policy decision making the policy query itself can be any json value. So, for example, if you are doing ap authorization, this policy query could be the http method, the path and so on, and if you're doing kubernetes admission control, your policy query could include the pod manifest, for example, so as long as you give oppa some kind of structured data and you write policies that make sense for that data, oppa sends a decision back to your service.

A

That's why oppa is called a general purpose policy engine because it's not tied to any particular data format.

A

Even the policy decision, oppa makes can be any json value, so not just through false or allowed deny it can be any other json value as well.

A

Let's quickly, look at some of oppa's features at the core of oppa is a high level declarative language called as rego and with rego you can write policy decisions which are boolean, sets objects, arrays and so on, oppa's written in go and it's designed to be as lightweight as possible. So all the policies and all the data you need for evaluation are stored in memory.

A

You can deploy oppa as a sidecar, a host level daemon, or you can embed it inside your code. If you're writing a go, oppa does not have any runtime dependencies, which means it does not have to pull down any external services or does not have to talk to any external services for making a policy decision.

A

You can, however, extend oppa to do that, but that's completely optional.

A

Oppa does provide you with some management apis that allow oppa to pull policy and data from a remote service or upload its decision logs to an external service or upload its status to an external service and finally, along with the core policy engine, oppa provides you with a rich set of tooling that you can use to generate test.

A

Your policies that you can use to benchmark or profile your policies and also there are integrations with ides like vs core intellij, and there is also an online rego playground which allows you to experiment with regular policies and share your policies with the world.

A

So these were some of oppa's features, a high level, declarative language, multiple deployment models, management apis for control and visibility, and a rich tooling set.

A

So now, let's talk about policy distribution.

A

So today a lot of organizations are using microservices to build out their infrastructure and what they do is they run oppa alongside these microservices to authorize access to these services and oppa's architectural flexibility allows them to do that, because you can simply run op as a sidecar and you can authorize access to these services, and so now imagine if you scale to like tens, hundreds or even thousands of micro services, and you have oppa running next to each of these micro services.

A

The question arises: how do you distribute policies and data to each of these oppas? How do you ensure that each of these operas have the most up-to-date versions of policy and data?

A

And finally, how do you load these different versions of policy and data without having to restart oppa every single time so to help with all these issues? Oppa supports bundles. So what are these bundles?

A

Bundles are basically gzip turbos that contain policy and data.

A

So you can have your bundles stored on an external service and oppa will periodically pull down these bundles from your service load them and start enforcing the policies inside of those bundles immediately.

A

So, for example, you could have your bundle stored on s3 and you could point oppa to that bundle on s3 oppa will pull it down and start using the policies in that bundle immediately.

A

So this is really convenient if you want to distribute policies to all these different operas which are running alongside your micro services.

A

So bundles are great they're awesome, but we do have a problem.

A

How does oppa know that these bundles are coming from a trusted source?

A

Oppa will assume that the policies and the data in those bundles are authentic and not corrupt, but is that always the case and what, if an attacker, corrupts the policies in the bundle and now oppa downloads, this corrupt bundle and start enforcing those corrupt policies, thereby compromising your system?

A

How will you be able to trust the decisions that oppa is making.

A

Introducing the signed bundle, the signed bundle looks and feels exactly like a regular bundle, but it has an extra signatures.json file and the signatures file. It dictates essentially what what files should be included in the bundle what the sha hashes are for those files and is cryptographically secured.

A

So this is how a signatures file looks it's essentially an array of json web tokens which encapsulate the signature of a bundle and if you were to decode a jot, what you would see is that, for example, in this case the bundle includes couple of files like the manifest file and it includes a data.json file and you can also see the hash of the content of each file.

A

So this is simply how the signatures.json file is structured.

A

So now what you'll see so this was a signed bundle and how it looks, but now, let's look into how oppa actually verifies that signed bundle.

A

So, whenever oppa receives a signed bundle, it's going to grab that signatures.json file open it up and the first thing it will do it's going to verify the george signature with the appropriate public key or secret.

A

So you need to provide oppa with this public key or secret out of band, and we will see how that is done during our demo.

A

Next oppa will try to verify that all the files which have been specified in the jot payload actually match the files in the target bundle. So it's going to make sure that there are no extra files or there are no lesser files in the target bundle to what's mentioned in the jot payload.

A

And finally, it's going to take every file in the target payload in the in that, in the bundle um generate a hash for that file and then compare it with the hash recorded in the jot payload and make sure they are equal.

A

So if any of these steps fail, oppa is not going to activate that bundle, and so only if all these steps are successful, oppa is going to activate this bundle and then it's going to start using whatever policies and data are included in that bundle uh and start enforcing those policies immediately.

A

So this is how oppa verifies a signed bundle.

A

So we've also provided some command line tools, some tooling that can help with signed bundles.

A

So we have a brand new opa sign command that you can use to generate signatures for the contents of a given directory or an existing bundle, and all you have to do is provide it with the signing algorithm that you want to use by default. That's rs 256, but there are others like hs256 and much more that you can use and also you need to provide the signing key. So if you're using an asymmetric algorithm, you provided the private key.

A

If you're, using a symmetric algorithm, you provided a secret and the sign command will generate the signature for all the contents of the given directory or the existing bundle.

A

We are all familiar with the oppa build command which is used to generate regular bundles, and now you can use the oppa build command to also generate signed bundles, and it just takes the same parameters as the sign command. You give it the algorithm and you give it the key and it's going to generate a signed bundle for you.

A

And finally, if you have to verify a signed bundle, you have the oppa run command which takes in the signing algorithm and the verification key, and it's going to verify that signed bundle.

A

So, if you're using an asymmetric algorithm like rs256, you provided the public key if you're, using something like hs256, which is a symmetric algorithm. You provided the secret.

A

So these are some of the command line tools and the tooling support that oppa provides for signed bundles.

A

Now, let's see sign bundles in action so to set up our demo. What we have is a healthcare application by acme healthcare which allows its members to see their invoices, and we are going to enforce a very simple policy which says that the claims in the invoices should be hidden from members.

A

We will see the simple policy later, but all it does. It says that we have to hide this claims inside the invoice from the members.

A

The policy will be distributed to oppa using bundles and let's see the scenarios that we are going to address in this demo.

A

So what happens when a malicious user updates a policy they're going to see that in the demo? How does that compromise? The healthcare app and finally can sign bundles, provide a compensating mechanism, so we are going to try and answer all these questions as part of the demo, so, let's jump into it.

A

So what you see here is essentially a dashboard which allows a member alice of acme healthcare to see her invoices.

A

There are two buttons here: the first one, it simulates a scenario without oppa in the picture and second one simulates a scenario with oppa actually enforcing certain policies.

A

So if you were to click the first button, you can see that there are a couple of invoices and pay careful attention to the claims.

A

You can see the claims that are part of this invoice, and now, if you were to click the second button, you would see that the claims are hidden, which means that uh the oppa policy is getting enforced.

A

So, with this uh base in mind, let's see what happens if a malicious user now tries to corrupt the policy.

A

So say you have a malicious user who simply changes the policy.

A

Okay and this malicious user now creates a bundle with this corrupt policy, so this bundle is now going to be pushed to your bundle. Server oppa is going to pull it down and it's going to enforce the policies which are in that bundle which are now corrupt.

A

So if we go back to our sample application and if we click the same button again, look at that, you can now see the claims in the invoices, which means that the corrupt policies which were pushed by the attacker are now being enforced by oppa.

A

So how can you prevent this disastrous scenario from happening, because now you can see your system is compromised. So how can you prevent this scenario from happening and again? How can you trust the decisions that oppa is making because oppa simply trusted the policies which were in that bundle those policies turned out to be corrupt, and now oppa is enforcing decisions which are compromising your system.

A

So how can you prevent all these things from happening will sign bundles help in this case, let's find out so what? Let's? Let's revert back to our original policy.

A

And what we do now is, instead of creating a regular bundle, we'll create a signing assigned bundle, and you can see that we are providing the algorithm.

A

We are providing a private key and we are going to create a signed bundle now so.

A

If you look, the only difference is that you have a signatures.json file which is included in the sign bundle. So now the sign bundle has been pushed to the bundle server. Oppa is going to download this signed bundle and now verify the signed bundle with the given public key that we have to provide.

A

So, let's see how that actually happens. Next, seeing now is the runtime configuration that is provided to oppa and you'll notice that we have provided the public key inside of this runtime configuration and since we're using rs256, we are providing the public key, which is going to be used by oppa to now verify that signed bundle.

A

So, let's see, if that worked out, we'll check out opus logs and see that so you can see that the bundle has been activated, and so now we can go back to our demo and so again, if we simulate a scenario with oppa in the picture, it's hiding the claims again, and so the reason this happened is that we generated the signed bundle, oppa verified that signed bundle and enforced the policies inside that bundle and the policy said, do not show the claims inside the invoice object, which is great, which is what we wanted.

A

So now again, the malicious user comes in and updates the policy. So let's do that again. So now we have the malicious user coming in just like before.

A

Updates the policy and.

A

Builds the bundle and now this bundle has again been pushed to the bundle server and has been downloaded by oppa. So the question now is like before will oppa just activate that bundle and start enforcing those corrupt policies or using signed bundles helped. So let's check it out again by looking at the oppa logs.

A

So if you look at the opa logs now, you'll see that there are a bunch of errors in that log, and these errors in the log are happening because oppa could not verify that bundle which it just downloaded and when oppa cannot verify the bundle which is downloaded it's going to keep on using the last downloaded signed, bundle and enforce the policies from that bundle.

A

So now, if we press this button again, you can see that the claims are still hidden, which means that those corrupt policies which were fed by the attacker were not being enforced by oppa.

A

And so in this way we have prevented an attacker from feeding corrupt policies into oppa by using signed bundles.

A

We used sign bundles in this demo from a security perspective to prevent an attacker from feeding corrupt policies into oper. You can also use signed bundles for other use cases like if you wanted to rotate the keys inside a non-discovery bundle, and there are other applications of sign bundles as well.

A

So this was a talk on secure policy distribution with the open policy agent. I hope you all saw how sign bundles can help in reducing oppa's attack surface. If you want to learn more about sign bundles, please check out the oppa website on openpolicyagent.org and check out the documentation about signed bundles join us on slack. If you have any questions and also check out the project on github, and if you like what you see, please do start the project again. I thank you all for joining me today and I'm happy to answer any questions.