solo.io Hoot Livestream, 13 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Episode 7: Get to know Open Policy Agent (OPA)

Description

Open Policy Agent (OPA) is a general purpose policy engine that enables teams to de-couple policies from the application code to unify policy deployment and enforcement across cloud-native environments.

Watch this hoot to get an overview of what OPA is, its architecture, how it works, and how to use it.
https://www.openpolicyagent.org
https://www.solo.io

Code Samples: https://github.com/solo-io/hoot
Suggest a topic to cover here: https://github.com/solo-io/hoot/issues/new?title=episode+suggestion:

A

And all right, we should be live.

A

Welcome everyone to another episode of hoot with your host me you well and chief arctic. At solo, I o and today we will be talking about the open policy agent, and this will be an introductory level and it's always interactive. If you have any questions, feel free to ask and we're gonna have some examples.

A

Gonna give some brief overview points to some other resources and in the next suit, we'll be demonstrating how to use open policy agent with envoy, and it's seo the more kind of advanced topic how to configure it with uh some more obviously, some more interactive demos, all right. So, as always, um let's get started, don't forget you know if you want to be notified about our next episodes uh subscribe and do that bell thing to get notified.

A

uh We do episodes every two weeks at 1 pm eastern time on tuesdays.

A

uh So also, if you have any more feedback for us, I'm on the solo slack, you can give us me or betty feedback on episodes and episodes. We want to be next. We're still figuring out the topic that we're going to do after we're done with the opa series.

A

So with that with me, my trusty turkish coffee.

A

Let's begin all right, so let's uh talk a little bit the introduction. What is open policy agent, so open policy agent solves a problem where you want to express policy uh but not tied to a specific application application you kind of want to have a general policy layer right. So the way open policy agents solve this by creating a policy definition language they call regul and a runtime to run it and at one time you can run it either as a go library right, but only doing a go.

A

Library is not that helpful, because then all the non-go languages they will not work right. So you can also run it kind of like as a server or as a side, car and use a rest api and something that's currently working progress is web assembly support. Allow you to compile it to your policy into webassembly and run it in any language that has web assembly support all right now. Rego, open policy agent is very general purpose policy, but we're going to focus on talking about apis and requests.

A

So we're going to use this terminology just to have it easier to explain what we're talking about now. Some concepts related to open policy agent as open policy agent evaluates a policy to make a decision right. Essentially, the workflow we're talking about is that a request comes in and we want to decide. Should we allow the request or deny the request right. So the request in the open policy terminology is what they call input right.

A

It's evaluated against the policy which can be composed or of modules for extendability and that's written in riego, so the input is json and that's evaluated through a policy that's written in rego, and if you have more data, that's related to the policy. That is not request specific and you can also represent something. They call data, and you can that's also json represented by json and there's a lot of small details that are irrelevant when you run this in production.

A

But for now we're going to focus just on the intro demo level flow and all right. So input is json and is it is essentially a representation of the request. Data is json and it is stuff that the policy needs. That is not request specific and the policy itself is rego and is evaluated on every request and take into account the input and the data and now to make it a bit easier.

A

Visualization, let's open the opa, docs and I'll go over a few of their dogs and then we're going to go through all the few of them. So looking at their dogs, you can see them very good dogs.

A

Are there slides? We should be seeing and no slides today, but there is a I'm going to review the docs in some of the images they have there and we're going to review some live examples. I prepared that we're going to run together and if you have any questions on those examples you can obviously yes and so the overview in their docs kind of explains what I explained right now right in this model, you have a service running opa is a sidecar.

A

The service gets a request and ask the opa sidecar to make a decision right, whether or not to allow the request and opi processes with the request. As input the policy is regular and the data- that's not request, specific, is json and now jason from this.

A

The slack pointed out this nice blog post. That explains more details about regular than we're going to go on here. We're going to just going to show an introductory example, and another resource you can use is the rego playground where you can run a bunch of regular yourself and see the results live without installing anything and, of course, there's always the reference, which is the full reference of the regular language, and with that, let's switch over to vs code and talk about some examples with rego.

A

Now I've prepared your very basic data that has a list of organizations, it's so loyal that has you well and indeed, and they have fabricam and fake company that has alice and bob and now.

A

Let's I have already downloaded the opa binary that allows me to run this examples, so I'm going to use that so, let's do opa run and give it basic data as a data input.

A

And now this is an interactive rego console that we can use to evaluate regular expressions right. So our first demo we're going to run a few expressions just to get the hang of rego.

A

Then if you have any questions, feel free to ask, then we're going to move to a bit more advanced example and we'll show how that works with the with a custom input and curl, and all that so given this data, how would I query all the members of solio, so the data ends up in a global variable in regular call data, and I can use dot to access its members, so I can do data.orgs I'll get. You know the subtree of the words if I'll go to orgs solo.io, I can get into solo.

A

and if I can go to members, you know I'll see the list of members so far pretty basic. Now, let's ask another question: is there a a member in solo io named yuval right? So how do we express that in rego now the way to do that now? Rego is a language based on data log that is based on prologue. So, if you're familiar with prologue, it will make sense and if not, I will explain it. The way you ask questions in rego is that you essentially provide a variable.

A

That's not bound, then rego will find a value that uh makes that expression, work and- and let me just write the expression that would make a lot more sense when you see it live by the way. Are we seeing the text clear clear this all right, so this query, what it does it answers? Is there a member in solo named view val, and how does it do that? So first thing we go to the orgs and get the members which, if you recall, returns an array.

A

And now here this index underscore is essentially not an index. That is. This is special variable that essentially iterates the members right. So this is kind of like a for loop doing this like this right, so I say: is there and- and then I ask, is there an element that equals your value right? So this query will evaluate to true if inside the members of a solo, I o there is a member name eval. So essentially, this underscoring desk index essentially turns this into a for loop asking.

A

Is there an index that I can put here that will make this expression true right. So when I press that you see, the answer is true right so far. If anybody has any questions, feel free to ask now, a slightly more advanced question we can ask is tell me all the organizations that you val is a member of. So how do we do that?

A

So, let's write the query, so some org, I can say in rego, I define a variable called or, but I don't give it a value.

A

It's essentially going to get the value from the query um when I'm defining a variable with some.

A

I'm essentially saying that this variable is is very similar to the underscore we've seen before it's kind of generates a for loop and used in the query, and then I can go data or so far similar to before then I can use this org variable that I've defined to kind of iterate over all the orgs and then in each org, go to the members and use the underscore to iterate all the members and ask if any of them is you well right, then, when we press that we can see, we get the list of orgs that match this query.

A

All right so far so good, so you can see how I can use these regal expressions very simple expression and we're gonna in the next example. We're gonna build on top of that to ask questions in a concise way. Right. Imagine this was python or c plus plus or go this one liner would be a lot more complicated, it'd be a double loop, essentially right and in rigo I can say kind of declaratively.

A

You know tell me: is there somewhere in the data, something that will make? This query become true right and then it can tell me all the values that do that right. So, for example, I can add myself to the fabricom organization and let me just exit so I it'll reload the data and ask the same question clear the screen here and you can see that now I get the two organizations.

A

All right so with the basic example out of the way: let's talk a little bit about the more advanced policy example that I've prepared. So I'm going with the theme of a request coming in and we want to create a policy that can tell us which, if a user, has an access to a resource right so essentially we're going to curl and based on user attributes. So we need to make a decision whether or not the request is allowed or denied, and we're gonna demonstrate this in a few ways.

A

So first thing: first, let's start with our input. How would our input to such an example would look like so remember: opa text is input as json, so, for example, an input can be a json structure that describes an http request right. A method is both path, contains the different path components.

A

So, for example, this will be posts alice first post, but we break it down to this list with elements. So it's easier to process right. The the micro service that will be interacting uh with the opa sidecar and asking these questions uh will transform the hp requested received to this json and essentially split the pi, the path to its parts now uh sub, I'm using the word sub, because that's how identities in json in jwt look like and that's essentially the username alice in our example and the claims right. What is this user allowed to?

A

Do? I'm going to say, write posts right, so this is something you can do unrelated to opiate all right. We've taken a http request and transformed it into a json representation.

A

If any question up to now again feel free to ask, if not, I will go over to the data and remember the data is something we can use in our policy. That's not request specific, so this input will change for every request, but the data is a consistent across requests right.

A

So the data that I have is essentially mapping the jwt claims into actions right. So I can say the claim right post should only be allowed if a method is posted in patch and if the path prefix has posted the same right review should only be allowed if it's a post or a patch and the path prefix is reviews.

A

Now again, this is uh not anything specific. This is just json and what will give it? The semantics is the policies, the query that I will show in one second right so, but this is essentially, you can think of this data. As my organization policy right, I want to make sure that these claims in the jot are only allowed to do these operations right. So that's why it's in the data. It's not request specific, but rather it's a a business rule right all right.

A

So now, let's talk about the query and that's the last file here, the query, rigo and let's break this line by line. So we talked about expressions- and you can see this here- another construct grigo has is rules and rules are essentially surrounded with braces right. So what is happening here? I'm saying allow will be true if all of these expressions are true and the order doesn't matter right. So this defines a rule that says. Allow is true if all of these are true all right.

A

So let's talk about this file line by line so first thing package, api, a app.api is just namespacing so that different values will not clash with each other next thing over. We define a default value for the variable. Allow right, if no other rule matches it will default to false.

A

Next, we define a rule and we say allow will be true if all of these lines of all of these expressions inside the rule are true and let's examine these expressions and let's do it a bit side by side with the input sorry input data.

A

Here we go so we can get a better understanding of what's going on so all right here we go so first thing: we do. We ask let's look at the claims that are available in the input, so for each claim.

A

So remember this underscore is an iteration right, so for every claim check if the data claims and data claims right, if we descend down to the claim inside the data claims right, is there a method that matches the input method right so we're asking if, in the method we saw in the input, is there a method inside this list of methods and remember? This is also an iteration that matches the input method right.

A

So these two lines essentially has take the claim from the input right go inside the data inside the claims variable inside it, go to the claim that matches the each claim of the input and if a claim from the input has a method that has the method from the input in this list of method, then this will evaluate to true right. So let me just bring this up here. We go.

A

Does that make sense, so these two lines essentially validate that the claim has uh is allowing the the method that is in the input of the request right, the next line. We take the path prefix from the same claim and we check if that prefix is indeed a prefix, and here we kind of slice the input path to match the prefix.

A

So essentially, what we're saying is take. This is this path. Prefix right is really a prefix of this path right. So here we see that this starts with posts, and this starts with posts, so it is indeed a prefix, but if instead of post it'll be cats, then it will no longer be your preference right, because we expect the first element to be the same in this and in here right.

A

So, together these two lines essentially validate that all these attributes inside the claim match the input request right. So this is the first part and then, if all of these are true, then allow will be true. The next part is to allow uh anonymous access to posts, and these are these a few lines here. The way we do, that is, we ask first, we define a a variable, is reader and we is reader will be true if the input method is get or the input method is head right and notice.

A

I have here two of these rules that essentially define an or semantics so inside them it's an end semantic.

A

But if I have multiple of them, it's an or semantic right so is reader will be true if the method is get or the method is head and then we allow, if it's a reader and it's a requesting to read a post. So we allow anonymous access to read all posts right so far so good, and now I'm going to demonstrate three ways to run this policy and get a result and the first way, let me just put those back together so more screen real estate.

A

The first way is the the most basic way you can think of is write unit tests. So in rego you can write unit tests in rego, and I have here the unit test for this query and you can see test posts allowed by post writers. So I'm checking allow true, given that the input is this input right and I can do opa test, and I can see that all my test tests, so this is pretty basic.

A

If you want to write a policy, you probably want to write some unit tests to make sure you've done it correctly, and this is a very easy way to run it and see that you've done what you wanted to do. So this is way number one to to evaluate this policy and see that it's working way, number two you're gonna use the command line.

A

Essentially, opa has an evil sub command that you can use to evaluate a command on the fly.

A

Opa also has that sidecar mode, and that's what we'll also demonstrate next, but let's first go with the opa evaluation, all right. So let me clear the screen now, let's see what's going on here, I'm doing a cat on input json.

A

What I'm doing is I'm just taking the the inner element so cut with jq input. Just takes this part and turns it into its own element, because opa eval doesn't expect this input variable and and the opa server does expect it. So I just use jq to remove it all right, so cut input, json jqueue to remove the the input part oppa eval, take your input from std in that dash capital. I and I give it these data files.

A

One is the policy and one is the data right and if we look at the policy again, it's package app api and I want the value of the allow variable right. So I say data out api. Allow because that's the expression whose result I want, then I just press enter and you can see the value is true right and if I go into input and change it from uh post to reviews the path and I'll run it again, you can see that the result is false right, so it ran uh the the input json.

A

I provided in the cat command it evaluated through the query and returned this uh variable that I escaped to return now.

A

The last way I'm going to show today for processing is using the sidecar method and I'm going to talk a little bit about a production implications of this, but let's first see how this works all right. So what I'm doing here is opa run. I give it the data json and the query json like before, but I don't give it any input. Instead, I do dash s which essentially tells it to start the server right.

A

Let's do that in the background you can see it started port 881 now a little note about pro when running things in prod, and you want to take into account a few things.

A

First, you're, probably not going to attach your data as a file to the server where you run the opa server right, you're, probably going to store it somewhere in some database or some centralized location and opa has a way to fetch data from a remote location. They call it bundles, you can read all about it and when you do that, it might take opa some time to fetch those bundles right.

A

So you also want to make sure that you have some sort of a readiness probe and opia is health checks to tell you when it's ready and when it's got all its bundles. So just before you, you know, run an opa server and you get 503s that are happening only for a second and then disappear. You don't know why. This is probably why you probably want to take all this consideration to a account- and this is just an in an example- we're not it's not production ready. Example right, it's just for a basic understanding.

A

So always before you run things in production.

A

Do your research and make sure you follow all the best practices and opa does have ways to make sure that your uh deployment will succeed with health observability prometheus metrics, all that so it's it was built for this cloud native use cases all right. So what now we run it? It runs in the background here. Let's run it curl and I'll show you the curl, so we curling it it's port, 8181 and similar to the command line. Right we go. The path is data app api, allow it's equivalent.

A

Instead of dots, it slashes, because it's an http request, see data app api, allow that's the path, we're going to query and we're going to provide it as input to the body will be input.json and this time when, when it's through the api, it expects the input to be under this input variable right. So how this would normally work is that your service will receive a request and will prepare the input for opa and will issue a request to its opa, sidecar or okay server, and that's the part I'm simulating here with curl right.

A

It's that service that asking a opa to evaluate the policy. So you can say- and I press this and I get a json response- a result. True, these two lines are logs from the opa server. You can ignore them, and maybe I will run it again with the devnet just so it's more clear and let's do two is going into one and one is going into dev now, love it.

A

All right now run the curl again and I still got the logs. I obviously messed something up but never mind for now so anyway, this is the result from curl. um Oh, you know what I can do I'll just run it in a different terminal.

A

It would be more clear what is what.

A

So, let's take our little friend here, copy open terminal, number, two paste: it run it um to go to the right directory, of course, and we're gonna go back clear, the screen and curl all right. Now you can see we just got back. The result is true great and then your service will get this response result in true. It knows it can pass the request.

A

Now again, let's change the input a little bit and right here, reviews and we'll ask the opa server again, and this time you get result is false right and when your service will get resolved false, it will know to reject the request now in here, because, usually with apis, we care about.

A

You know allow or not allow a boolean response. I've demonstrated boolean responses, but the response doesn't have to be boolean. It could be any json. So you can also say you know. Result is false and here's a message that you can log or return into the client or whatnot. So you can have a more sophisticated policy here.

A

That returns not just a boolean, and we might see some of that and next week in regards to when we fail, we want to give some more information on. What's going on all right, I believe this is all the stuff I wanted to show today, just kind of a basic introduction and again there's a bunch of resources. I didn't go too deep on all the stuff in the language, because there's a ton of reference- and I want to save some stuff for next week and but yeah feel free to go over those resources.

A

I I showed- uh and if you have any questions, feel free to ask now, if not I'll, go ahead and summarize what we've seen today.

A

So we talked about a opa. A opa is essentially a language and a runtime to specify policies or pay has kind of the following courses in the life of the request. A request comes into your service and you know what I'll go and pick and show that diagram from the opa docs.

A

So a request comes in to your service. The service prepares it to a an input for opa by essentially converting it into a json format, similar to what we've just seen sends it to opa. Opa evaluates it evaluates this input, together with the data that he has. That is not a request specific and the policy, the regular policy and gives a response.

A

It can be energized in most likely if you're doing an api, it will be some sort of a boolean response. You know reject or allow, and the service can then do with this. Whatever it wants right, opa does not reject the request for you. It just tells you what to do and then it's the service responsibility to actually reject the request right. This just takes part the part of the policy and as the savior decouples it from the service.

A

So um so opa has this input json the data that is not a request, specific json and policy. That's rego and evaluates them to give you a decision now. In the example, we saw how to use opa to kind of evaluate a policy based on jwt claims right, and in that example, the service receives a http request.

A

Parses the jot takes the claims from the job and the subject from the job and prepares it in the input for uh oppa, together with the http method and the path the organization policy lives in the data json, where we saw you know we, this claim is allowed to use these methods and this path, http path and the query.

A

Essentially, the policy verifies that the claims given in the input and the method and the path match the organizational policy and it returns true if they do and it returns false if they don't and then we can query, we saw three different methods of query: nope using opa eval, using the opa run as server and using uh unit tests, which are probably the most important thing. When writing code is writing some tests and that's it for today in the next hood, we will show how to do this with services using android and zeo.

A

Essentially, the opa have an implementation for an external authorization server using op and we'll give a demonstration of that. And if there are not any other questions, I will say goodbye to everyone, and I will see you in two weeks have a great day.

A