Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Customizing OPA for a "Perfect Fit" Authorization Sidecar - Patrick East, Styra

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Customizing OPA for a "Perfect Fit" Authorization Sidecar - Patrick East, Styra

The Open Policy Agent (OPA) has become widely used in the CNCF ecosystem and is a go-to option for application developers as the standardized decision engine for authorization. Many users rely on the existing integrations with Envoy/Istio, or the OPA REST API Server, but this doesn't always fit in an application cleanly. Common performance questions arise related to using the OPA HTTP API, plus management concerns around collecting decision logs, and requirements for integration with existing security infrastructure. Join Patrick East, an active OPA maintainer, to see how easy it is to use OPA's public Golang API's to create a tailored OPA binary with the following extensions: * Custom High Performance gRPC font-end API * Custom Kafka decision log plugin * Custom OAuth2 Rego builtin functions

https://sched.co/ekEV

A

Hello welcome. This is customizing oppa for a perfect fit authorization sidecar, I'm patrick east, open maintainer. um You know always find me on the open policy agent slack. My username is just patrick east. Nothing exciting and it's my twitter handle feel free to tweet at me. If you have questions or want to get in touch.

A

So let's start this off with a little bit of like base knowledge. Hopefully everyone here knows about the open policy. Agent knows what it is. Otherwise uh the session will be very confusing for you. um If you don't pause now, go look at the url, come back when you're ready um just to make sure we're on the same page, john purpose policy engine.

A

Most people use this as a library or an http server. The use case we're kind of talking about today. A little bit is more in the sidecar model, where you have your application requests come into it and then running on the same host. Whether that's like you know, like literally like a linux host, it's another service or like in kubernetes. It's just another container in your pod. um You have like opa running, but the application makes a local host call through the http.

B

A

To open it comes back, you know, usually it's something like is it allowed or give me reasons why it's denied or something like that? um Good news is for that model. uh Opa is perfect.

A

Usually that's, okay, fine, except when it isn't, um there's a there's, a lot of like requests that come in for for, like the open maintainers for uh custom built-in functions or like like new new functionality, that's not there in the language, um so something that we hear about uh from time to time is like. I have this authorization use case I'm using these jots and I need some special behavior to get my keys um they're. You know using their their own kind of like homegrown system or uh for like the demo.

A

I'm gonna show here I'm using auth z uh to altzero, um but but you know it needs the. The problem is that they've got like they've got their keys somewhere, uh it's remote. They can't just hard code it in their policy; they they rotate whatever. um I need any just the right error handling. Caching for my my special system, um so the solution, obviously, is just add it to opa. uh You know this is like it's a problem. I have it's a problem, other people probably have so, let's make it part of the language.

A

um That's great, uh you can file an issue. We have a lot, though, and you know like the the problem is that you're gonna find some pushback, usually for uh like we don't necessarily want oppa to you know be a kitchen sink, have have everything um it's it's got to be like balanced with the the amount of overhead for like pulling in additional libraries, uh how many people are going to use it who's going to maintain it.

A

So, unfortunately, you'll see a lot of pushback on issues for for new extensions to the language for build-ins, especially if it's somebody's, you know like special service, they have- and it's like you know, it's sort of a one-off thing. um So what's the answer.

B

A

You know screw it, I'm gonna go make my own opa uh with my own, build dents, so the behavior that we want uh for this demo. Whether the example use case that we're going to talk about is that I want to be able to fetch keys from a remote url we're going to look at the open id configuration. It's a well-known path, pull it down, check and see. If there's a key or I specified in there, if there is we're, gonna go download it.

A

Otherwise, we'll fall back to trying this other, um just like well-known, uh keyset url.

A

If you guys are familiar with opa and writing policies, you're, probably like hey patrick, that's dumb, you can just use http, send um two things. uh First of all, with the demo don't read into it too. Much second thing is that, like this is like a legitimate problem for for error handling with http.

A

Send it's not always so easy um in in the regular policy to get kind of like exactly the right behavior for for the error handling, as well as like caching and other things, plus it's a lot of complexity that you're going to put in your regular policy that, like you may or may not want to actually deal with there like.

A

Sometimes it is easier to just write some of this stuff, um as is like built-ins, uh with like a little bit of like a native like general purpose programming, language versus like policy language, because, like frankly like fetching, something from a url is not necessarily what like a policy language is designed to do so.

A

First of all, go read the docs. If you ask me on slack how to add something to oppa, I will point you to this url. Most people, though, read it and are like oh crap. This looks hard like it's it's. You know, wall of text. That leads you right into a bunch of go code and uh go code, isn't always like the easiest to get started.

B

A

um So now you know the fear sets in you're like wait. I have to extend oppa this sucks. I I'm gonna need to make a custom image. I can't just use like the opal one anymore.

A

uh I have to learn gold length because extra like build tools and stuff um plus I have more code like I don't you know to like fork, opa. That sounds like a headache to manage, um but like in practice, good news. It's actually really easy. uh There's there's not that much code. I think it's like three lines. I'll show you in a minute um to to like just get the base like forked, not even forked, like customized opa entry point, um the dependency management's really easy.

A

You can go modules, so you don't have like a separate fork. You don't have to like sync your opa with, like you know, git histories and all those stuff um you really just like when you're ready and you want to update to a different version of opa. You just like you know, go get that version and upgrade it, and as far as like the tools and like having like golang and all that and the custom image, it's really easy to make your own.

A

um These, like multi-stage, like you, use the golang image to build, and then you put in your you know whatever company's blessed base image that uh you know secured team has decided, is okay and just copy the binary into it. um It's it's really like I'll, show an example of one they're super straightforward.

A

So let's get this started all you have to do to get your custom version of opa off the ground start a new project.

B

A

Get in it go mod in it sort of like basic. uh You know starting point for most uh going apps, you need a main.go and um the the like actual implementation of this. All you have to do is call that a command.rootcommand.execute, that's the the like main entry point that opa uses for its own cli.

A

This will give you access to basically everything that open can do and it lets you then control like any other stuff, that you want to do on top of oppa. So, let's, uh let's take a second and go look at some code.

A

I have a project started. This is that main function, we're just looking at um there's, not a lot to this uh I'll show you, the the you know like the actual usage in a minute dockerfile I promised I'd. Show you guys this too.

A

All you need to do is pick like a you know, going version that you're comfortable with I use 115, just it's like the latest, uh as if I was recording this. It's like the latest stable. I think, um but it doesn't really matter. Oppa is pretty compatible with with most versions that are like relatively modern, and then you just go build your your opa in this case, I'm making an open, plus, plus binary and then pick whatever base image.

A

You want and copy open, and this will behave just like the regular open images you can slide it in use it with good management, whatever it doesn't really matter, it'll it'll, you know just work so back to our main.go.

A

Let me get the terminal up and prove that this does in fact work. um So we've got our project here. I'm just gonna go run main.go, it's doing a little compilation step here, so it's yeah there we go. It takes a second when you run with like go run versus like building and running the binary, but um this should look familiar. This is the opa like command entry. Point you've got everything here, um so I want to run like open the server.

A

I do run s and you know there we go so this is our customized opa off the ground running ready to go. If you need to control the oppa version, um you have the the go module. So if you're not familiar with these, essentially in here, you have a requirement for the dependency on opa. Whenever you want to change which oba version you're using uh just just update this and uh you'll pull in the latest code, you get all the changes, features et cetera.

A

So that's that's really pretty easy to manage it's not much harder than you know, setting the the opa version and, like your yaml, manifest somewhere.

A

So we've got our base version. You might be wondering hold on where's my built in. How do I do that? There's three steps, so the first thing we have to do is decide on a signature.

A

This is for compile time for opa to be able to do like type checking ensuring that, like nobody has conflicts, they didn't define some other like rule that has like the name that conflicts or something like that um as well as for the run time, we have to have the implementation of our function.

A

You know have code that actually does what we want to do for the built-in. The last piece is pretty crucial, so you have to register the built-in using one of these functions that I'm showing here on the right.

A

You have to do this before you start oppa so like in in our main function. We do that that execute on the the entry point, um so we need to do is is register and then call the entry point execute. So, let's.

B

A

Look at that a little more code.

A

There we go, I'm going to cheat and jump to a version that already has this implemented. Save us all a little.

A

Time, okay, so our main.go has been updated to now have a built-in stock register function. uh This thing is a function that I added. This built-ins package is new. It's part of the the example code that I'm showing I like.

B

This kind of pattern.

A

uh Basically, just to minimize, what's in the main function, we don't want to do too much in here. um You there's no reason you couldn't have your entire built-in everything in this one file, but it becomes a little unwieldy over time. So I've got another built-ins package where I'm going to put the built-in and the implementation for it um our built-ins package.

A

uh We have the the one function here. This is the one we're calling in main. The call here is the one that we saw on the slide a minute ago. We're calling built in one the the one just means: there's one argument: there's a handful of them or a dynamic one. If you have, you know like a variable list of arguments. um The declaration here um this is that that signature that I was talking about so in here we're going to define what the function is called. So this is the name.

A

This is like literally the string inside of the the rego code. That will, you know like be the function uh in this case, I'm picking custom.fetch.jwks, um the actual like signature portion of this is a little bit cryptic. um Take a look at the the opa like ast package, documentation and there's a ton of examples in there for all the built-ins in the language.

A

If you get confused on how to do one of the more kind of complicated signatures, there's lots of examples in there, this one's pretty straightforward, we're saying that there's arguments of a you know a single string type and a return of a single string type. um The last piece for this registration is pointing to the implementation implementation as a function not too complicated.

A

You get a built-in context. This thing's pretty cool I'll talk about it in a second and your arguments, so in this case our argument single one.

A

We uh translate it from the the ast, like the the regular type to a just, a regular going string um for our implementation. We're gonna have like effectively a base. Url we're gonna prepend, some well-known paths to it. um Like literally, like you know, it's not well known in the um in the urls.

A

Just noticed a bug in this one. Look at that fixing bugs as we go doing it live. um So we have our base url argument, we're going to be returning back a single ast term, so this is going to be our our key, we're going to turn it back as a string.

A

The built-in context is pretty cool, so make sure that you take advantage of this. All the built-ins get called with this context on it. There's a cache that lets you cache uh values, so you don't have to recompute things as much the the nice thing here being that like, if, in some policy evaluation this this function gets executed or evaluated more than once, you don't have to do the work for it. Each time you can have a consistent result.

A

Every time you don't worried about, if you had like um maybe errors or something like that, so what we're going to do first thing is just check and see. If this you know, we already have this in the cache.

A

If we do, then we're going to just return back whatever was cached, don't worry about it like assume it was valid if we don't find something in the cache we're going to go off and fetch the code fetch the key.

A

I'm going to skip over some of this code. I don't think it's super interesting, basically we're just looking for like an open id config, we'll check we'll go and like fetch that if it's there we try to parse out the the uri. If it's not, you know, if any of that fails, we fall back to this. Just like default, well-known jwks thing. um All of this is sort of a hypothetical you know workflow, it doesn't really.

A

You know like usually you kind of have one or the other or something, but you know for the sake of example, uh so we're going to do the last part here is that we're going to fetch the the the uri for the key, whether that's the one that we found out of the config or just the one that we're guessing and then return back the the value as a string.

B

One thing here: it's another.

A

Another bug in the code, um so we need to add it back to our context, cache so catch key, which I defined earlier and our string so.

A

When we do this, what we're going to have is.

A

A new string term which we're gonna save in this in the cache that way, we can just blindly return it later and then we're gonna return. This uh the this is going to be the string of the key, the the actual value that comes back out of the function in in radio when we use this um and then the last bit here is: if any of this failed we're going to fall through you'll notice.

A

There's a lot of like arrogance now, so we're only like half happy path uh like like checking our way through this stuff. So there's any errors. We're gonna fall through and return back, undefined, uh undefined being there's no error and there's a nil value for the asd term result.

A

This makes it really easy in the like policies that are calling this code to do like error, handling and um sort of reason about like when you, when you do like a rule for like validate that the key was okay. If part of that is fetching the the um the key, then if it comes back as undefined, that rules can be undefined, you know your your allow step presumably is like calling to get this key. That will also be undefined so like it should it should bubble up in kind of a pretty like sane way.

A

It's easy to reason about. uh I highly recommend using this as like your kind of default pattern.

B

A

Okay, so anyways we've got this implemented. Let's go look and see when we run this on the terminal.

A

We're gonna run this one in the uh the repel, just for example, purposes, and what we'll be able to see is that our function is recognized and it knows about the syntax of it. So if we try to pass in like one two three, it's telling us hey, you gave us a number, but it's expecting a string, and I want to return back a string like this is this is invalid. You can't do that um so, like there we go. We verified that our signature is being registered correctly.

A

Now, let's try and fetch with an actual string, the the url that'll work uh and sure enough. We get back something that looks good. This is like the the key for my my dev optio account here feel free to steal this url play with it. It'll leave it up for a while, but there's nothing um secret in there. It's just a sort of demo, one um so yeah. So there we go so our built-in's working we've now extended the language we have. This new custom function um we're all set now.

A

Let's say we have a new requirement. um We have decision vlogs for opa people like it, but they don't always like having to have like a special server for it or something. So you know a lot of times: they'll have some other cool infra. Let's say hypothetically you've got kafka running you've got a sweet production cluster.

A

You, you know you'd much prefer to just dump all your decision logs into it so yeah. Let's do it! Why not? uh What we're going to do is make a new plugin, so open plugins are pretty straightforward. You need to define a plugin name and a struct.

A

You implement the plugin interface with your struct and then you have to have a factory that instantiates your plugin.

B

None of these are too.

A

Complicated start, stop reconfigure exactly what they sound like.

A

The factory validate is essentially you're, given like a raw chunk of the oppo config, and it's up to you to parse it and decide if it was valid for your plugin or not so it makes it really flexible. You can have as complicated of config as you want um without having to to like do any any sort of like config registration or anything like that with oppa, and then the last piece here is actually registering your plugin same pattern as the the built-in function.

A

You have to do this before actually like executing the the oba entry point. So let's go look. I have some code set up for this should be easy.

A

We will jump ahead to a version that actually has the plugin save us all a little time.

A

So notice here I've got a new package called plugins. I've got some changes in the main.go plugins.register, I'm using the same pattern as I did with the built-ins where I want to. I want to try and you know, minimize main code. um So let's go look in our new package in here. We have a register.

A

I split out into a separate package for the logger, so we have a plug-in name defined for it uh and a factory to instantiate it. The plug-in name is important. um Let me go look in the logger here uh so that the plug-in name for us right now is cockpit logger.

A

um This thing is, is actually kind of like you can almost think of it as part of like the api for your plugin or something because it's going to be the string that shows up in the configuration and it's going to be as far as like other plugins ability to find your plugin and use it. uh You have to configure them and like like this thing, it's it's more than just an arbitrary string. You know you should pick something and not change it. It's it's like. There are migration uh implications.

A

If, over time, you change the plugin name for something um so anyways, like I said before, we've got our plugin name. uh We have our our actual struct. This thing's got a manager uh reference, you're, gonna wanna hold on to that you get it as part of the the factory initialization step.

A

um We have a mutex that we use to protect our our config during reconfigure, um and then we have our config, which in this case for the kafka plug-in we're gonna have a host and a topic. So this is just telling it where it should publish information um the factory, nothing too exciting. uh You know it returns a new instance. The logger.

A

The only thing that's maybe of interest here are the plugin statuses, so the plugin manager has api calls on it to set the plugin status again this you know the plugin name gets used everywhere, um so you're going to want to use this as extensively as you can in your plugin to give accurate information back. This is exposed via like the health and um status apis for oppa.

A

So you want to make sure that, like if your plugin is in a bad state and you you need to like get opa to stop getting requests for authorization, you have to make sure to set your plug-in to an error state and make sure that your health, probes and other things like that are configured to look at plug-in health. um So don't don't skip these they're important um they're relatively new.

A

So if you have, if you've seen like older plugins or you're like uh you have built an older plugin, these might be a new api for you. I'd recommend going and adding them. um The last piece here is actually like being a decision logger. So there's a little bit more to the plugins um to actually be a decision logger. You have to be a plug-in, but you also have to implement this log api um for us. It's gonna, be you know you get in a context.

A

You get a event object which is like the the decision, and then we wanna basically send that to kafka. So if we go back to the code, I'm gonna jump ahead to a version with this log function implemented to use kafka.

A

And here we go take a second for the id to catch up. Our plug instruct now includes a pointer to a kafka producer. We're using just the uh the like confluent kafka. Go library, nothing to you too, fancy there um when we start up we're using our config. Now we set up uh a config saying that our bootstrap server is going to be that host option that we we provide um when we start and stop, we, you know, start the producer. We close it on stop on reconfigure.

A

We we effectively stop and start it again, setting the um the states appropriately and then for log. Then you know like, as expected, we're going to take the uh the event we're going to use it's built in like json marshalling stuff. um This will this will give us a string. That's basically the same payload that you get with the http uh decision, log service um and we're gonna just do a like produce on our producer.

A

With this we're going to specify the topic that was configured and any partition, so it'll just get you know thrown into that topic. Anybody can receive it and that's pretty much it. The only consideration that you might want to think about when you do these um decision loggers is whether you want to do it asymptotically or not. So if you do a synchronous one, you make sure that that decision is is logged before the decision is like returned back to the client right.

A

So if somebody says is this allowed, you have it's sort of up to you and your internal policies. Most people want to have that decision logged before the client gets back yes or no, the reason being that if they get back yes but they weren't supposed to, uh if you lost that log somewhere, you may not know what happened. So it's like a security, I would say like risk, but um you know typically you're going to want to make this like a synchronous thing.

A

So when we do the produce we we sit here and wait on the delivery channel until we make sure that it worked, and if there was an error, um then we we go ahead and return that back versus returning back no error.

A

If there's an error in the log, um the the server will actually return like a 500 or something back to the client it won't, they won't give back. Am you know like okay status with the result, so there we have it. We have our plugin. We should be ready to go.

A

Let's run this thing. The first thing we have to do is is configure it. So let me set up a conf.yaml real quick.

A

um We have here the decision, log plugin configured to use our kafka plugin and we have kafka, plugin plugin settings. It's a lot of it's plugins, all the way down um the the topic we're going to set to opa the host is going to be localhost 1992., um I'm just using the like kafka. Quick start thing, look running locally.

A

So if you, if you go, google that, like um you, follow the instructions there you'll have the same setup that I do um and now when we run opa, I'm gonna specify to use that config use it in debug mode.

B

A

It as a server, we should see this thing up and running there we go, and now, if I make a request to opa.

A

A

Oh my gosh spell here: we go uh v1 data, so this is going to trigger a decision log event and we can see in our debug log that yeah it delivered the topic or delivered to the opa topic.

A

We can verify that over here. I have a script queued up here. We go.

B

A

Going to read from the the topic.

B

A

A consumer and point it at the same server and we will see that yes, indeed there, it is, there's our decision so success we're now logging into kafka.

A

Let's go back to slides because surprise, surprise. We have another requirement. So we have this thing as a side car. We.

B

A

Be lightning fast, we don't want to sit here and wait house http, json overhead. You know like come on, that's slow! um Let's, why not use grpc uh and the answer is like yeah. Why not? Let's do.

B

A

So how are we gonna add a new api to this? um You might be thinking to yourself like okay, we can add a built-in function, but, like that's weird, this doesn't seem like the right way to do it. uh Well, the answer is a plug-in yeah.

B

A

Have one but you can have as many as you want, so let's keep doing it. um Let me jump to the code and I will skip ahead to our base. Plugin now.

A

What you'll notice I've just added a new directory here um api in our plugins? I'm registering one more same pattern same. You know everything kind of looks about the same and then in here in our api, I'm defining a protobuf. You know like um api definition here for a auth, z or authorizer service with an opt z, api.

A

It takes in a request that just has a string jot and returns back a boolean. Yes, no so super streamlined. You know as minimal as we can get here um generated the the you know like boilerplate code, that comes from that with the regular golang protobuf stuff.

A

Our plugin looks very similar. um We define our plugin.

B

Name, we have a config.

A

Struct uh this time we only need a a like host or like a address to listen on, so this is going to be where we set our like tcp listener. For the grpc api we have our server struct, which is our plug-in same. You know the usual gangs. All here we've got the plug-in manager. We've got our lock, we've got our config. The new parts are the little bit of boilerplate.

A

We need for doing the the implementation of the authorizer api, as well as a pointer to a grpc server that we're going to start so start function. Is you know you probably imagine we start the grpc.

B

A

I'm not going to go too far into this code, but you know the it's all it's all available online so feel free to dig in take a look.

A

The important part here is the actual implementation of that api. So we have this grpc api endpoint for otzi. The thing we get in is going to have uh jblt on our request, we're supposed to return back a boolean, uh but then like wait. How? How do we evaluate the policy? um The good news is that there's there's pretty good, like opa api documentation for how to evaluate policies. So here we go.

A

I jumped the code ahead a little bit fast forward, we're going to use the rego package, we're going to do like a prepare for eval and then we're going to evaluate our input. You know we already got that figured out we're going to expect the jwt as part of our request. So that's you know we're just going to wrap that put it in a map so that openness had to deal with it and evaluate, uh but wait. There's no policies right.

A

When you do this, this rego.new, you have to tell it which policies to use and like say you had external data or something in a bundle suddenly, like you, don't have.

B

Any of that right.

A

um So what do we do? Do we need to like parse bundles in our plugin or something? The answer is no. What you need to do is use the plugin manager, so there's a little snippet here in the docs that I think a lot of people skip over um the the manager. uh Let's see here gives plugins access to engine wide components like storage, uh so the pro tip main takeaway manager.store.

A

That's that's where the policies are that's the good stuff. So let's go back and implement that there we go super fast when you cheat and use uh git, checkout, um okay, so our oxy function now same thing: we've got our input, um there's a little bit of of typing to to work around, but nothing new same query.

A

Now we start a storage transaction. We use the manager store when we construct our regular object, we're passing in the store, the transaction and the compiler. So this is a cool step. If you do this and you're using bundles. In the background, you won't have to recompile it'll use the compiler, that's already available. It's it's done like on the background threads. It's not part of like your evaluation.

A

uh You know like hot path, so like definitely stick to this, give it a whirl. um Once we have this we're ready to go, we can evaluate this thing is going to have all the policies that are loaded from bundles from the cli from the you know, http api, if somebody's pushing policies into opa you'll still have access to them this way.

A

So the last piece here is that we do need to call the decision logger. um What would do there's a log api very similar to the one that we implemented, uh but you just have to get a reference to that plugin. So, let's start what we do is go and look it up. So this goes back to the plugin name, being kind of part of that, like api, you use those names to look up plugins on the manager. Again, you know, manager is kind of the key to being able to do this.

A

Once you have that reference to the plugin, we just stash a copy of it on our server plug-in and use it in an evaluation to do the decision logs. So the end result here is that we're going to have a grpc api that takes in a job it does the evaluation logs it to kafka and then returns it back.

A

So let's go ahead and jump back to the slides here. uh So thank you. All this is available on github. It all works. I sort of skipped over the demo for time but feel free to grab the code and run it. It should have no problems uh yeah. Thank you.