Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Protecting the Data Lake - Ash Narkar, Styra, Inc

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Protecting the Data Lake - Ash Narkar, Styra, Inc

In today’s world, data is at the core of every business decision made. As data grows, companies have started implementing their own Data Lakes to store and run analytics on the data. Ceph is widely used to implement a Data Lake. Securing the data is a priority for every organization and is influenced by the technologies they use, legal regulations, internal conventions, and so on. Enforcing policies to protect the data is difficult because it often affects the entire stack, requires state from multiple locations, and must evolve over time as business needs change. In this talk, we will see how the Open Policy Agent (OPA) can be integrated with Ceph to guard access to sensitive data while satisfying strict latency and availability requirements. In our demo we will deploy Ceph in Kubernetes using Rook and show how to enforce custom policies over the Ceph Storage Cluster.

https://sched.co/MPdu

A

Hi everybody we're gonna, go ahead and get started. My name is Tim Hendricks I'm the track host for this session, I'm from Styron, oppa and so I'm especially excited to host this talk a reminder we'll have five minutes for Q&A at the end. I'll run around and get you the mic, so the recording will pick up your questions also remember to rate the talk, go and login to sched to go ahead and do that so without further ado, I'll give you a shocker he's gonna, be talking a bit about policy problems in the data Lake.

B

Hola, thank you Tim, so my name is ash. Nutter I am a software engineer at styro and we are the makers of the open policy agent project and I'm, also a poor contributor to the project, and so today is special for me, because it's my first time, speaking at Q Khan, so I'd like to thank you all for sharing your time with me.

B

So in today's talk, I'm gonna discuss data leaks, their architectures and the security challenges that they face then introduce the open policy agent to talk about its features, its use cases and then we'll drill down into a data protection use case.

B

So in today's world data is king, and so it can be data from your cell phone. Your laptop your tablet, data is pervasive, its abundant and it just keeps on growing, and so this is an awesome opportunity for businesses, because now they can use all this data and they can analyze this data. They can drop trends from this data and they can use it to drive customer experience and they can also use this data to drive the growth of their companies and so with data being so critical and so sensitive.

B

It is really important that this data be protected, especially in today's world, where cyber attacks are pretty common and the threat of data breaches is a reality for businesses, both big and small, and so with these attacks and breaches, your companies are not only going to get fined but, more importantly, you are to lose the customers trust, and so we need a really good solution to solve this important problem, and so what companies normally do.

B

They are going to build up a data Lake to store their data, and so basically, what is a data Lake and so a data Lake is nothing but a large storage repository which can store massive amounts of raw data in its native format until it is really needed, and some of the features of data lakes are like centralized content.

B

So you can store your data from different sources in this data Lake and once it is stored, you can that later use it for applications like machine learning, for analysis, whatever you choose, and since this data Lake is like a central store for your entire division or for an entire unit, it is capable of significant scalability data. Leaks can support multiple data types, so you can have structured semi-structured, unstructured data can all be put inside your data Lake and finally, you have resource optimization.

B

So by decoupling, the compute from the storage, you can scale both independently, and so now you can store your data which, in the cheap manner in your data Lake as compared to databases, and so you have all this.

B

Data storage is cheap, but data in its raw form is rarely meaningful and cannot be used by applications for image at consumption, and so you do not want to end up in a situation where you have this massive amount of data which you are just holding and you can't use any of it and the one which is even useful for you.

B

You've made it difficult to access and so to solve this problem, companies typically build out a data Lake platform in which you're going to ingest your data you're, going to store it and then make it available for analysis in the future, and so this figure illustrates a typical workflow for a data lake platform in which you use cough cough or ingestion.

B

You can use safe for storage and then you can use elasticsearch for analysis, and so in today's talk we are going to focus on three components: kafka, safe and elastic search and the security challenges that all these face. And so today we are going to provide you a unique and a revolutionary way in which you can enforce custom security policies in each of these systems in a flexible and a powerful manner.

B

So, let's first look at Kafka Kafka, as you guys may know, is a distributed streaming platform. It can handle trillions of events in a day. It's typically used for building streaming data pipelines which can transfer data between different applications and also for building streaming data applications which can transform or which can react to streams of data.

B

So the authorization model in Kafka is access, control lists and a typical Kafka ACL statement says that a principle P is allowed or denied to perform an operation o on a resource, R and so AC ELLs are simple and they are straightforward, but they are limited by their scale, their flexibility, the expressiveness and their management. And so imagine you want to enforce.

B

Policies such as consumers should not be reading from topics which contain PII or your personally identifiable information, or you want to restrict users from writing to topics which have out if you try to enforce these kinds of properties or policies using ACLs at scale, given their limited expressiveness is going to be a management nightmare, and so the question is: how can you enforce such security policies in Kafka and, more importantly, who is going to do it?

B

Sef SEF is a distributed. Storage system that provides block storage, object, storage and file system storage in a unified system.

B

The security protocol in SEF only applies to the safe clients and the safe servers, and so it's the clients responsibility to provide access control for the data that is stored in the object store, and so imagine you want to enforce policies which are based on a user's location such that a user can only be allowed access to buckets which are in the same location or you want to have access control policies based on a user's business unit, his department.

B

How are you going to enforce such policies that you don't even have a security framework in self and again, more importantly, who is going to do it? And, lastly, we have elasticsearch, which is a full-text search and analytics engine. It's capable of storing, searching and analyzing vast amounts of data in almost real-time.

B

So again, the security in elasticsearch is more of a premium feature, at least the authorization, and so the user is an exercise left for the user to implement access, control policies, and so imagine in elasticsearch you have a patient's. You have an individual's ph, I, which is the protected health information, and you want to provide secure access to that information based on, say, a user's role or some attribute based on the user. How are you going to enforce that kind of attribute based policies in elasticsearch and who is going to do it for you?

B

And so now you have these three distinct systems, each in their own different programming languages, each having different security requirements which keep on changing over time and hence become difficult to predict.

B

And so how are you going to enforce policies in each one of them without hard coding that policy in so where or how are you going to enforce policies in each one of them without tightly coupling the underlying system to the policy enforcement and, more importantly, how are you going to achieve these two objectives without compromising on speed and performance and in a really expressive manner?

B

And so my question, for you is: is there a solution that can provide uniform policy enforcement across the stack in all these systems in an expressive and powerful manner?

B

And luckily the answer is yes: it's the open policy agent and so with Kafka. You can take the Kafka's authorization interface and you can plug in OPA with that, and you can enforce policies such as don't allow certain users to write on our topics in self. You can integrate OPA with SEPs object, gateway to provide access, control over the object store, and we will look at this use case later in detail and, finally for elasticsearch. So this is where OPA does something special.

B

This is like an experimental use case for OPA that we have, and so in this use case instead of OPA giving you a decision back, OPA will return to you a set of conditions which you can then translate into elastic, search, queries and then enforce them on the database. And so now, let's look at what is OPA, just a quick show of hands. How many of you have heard about OPA wow, that's a lot and how many of you are actually using OPA.

B

Ok that that's helpful. Thank you. So oppa is an open-source general purpose policy engine you can take oppa and you can use it at any layer of the stack in any system, and so your services can offload policy decisions to oppa by executing queries before we go more into oppa I, just like to mention something around opus.

B

Community oppa was founded in 2016 at styro, and the goal of the project is being to unify policy enforcement across the stack one of the earliest adopters of oppa was Netflix and they use oppa for authorization over their HTTP and G RPC api's, and there are more than 20 companies who are using oppa actively in production for use cases such as admission control, authorization, risk management, data protection and so on.

B

Who powers recently promoted to the incubating stage at C, & C F, we have more than 59 contributors on git. We have a healthy slack community of more than 800 users and just to give you some context. At the beginning of the year, we had around 400 users, so it's great to see oppa grow and people using oppa, and so just to give you a brief of what oppa actually is. Imagine you have a service and the service can be anything. It can be your cough car.

B

It can be your authorization service, your API gateway, your kubernetes api server, absolutely anything, and so, whenever the service gets an incoming request, it's going to ask oppa whether this query is allowed or not, by executing a query, and so this query can contain your input method.

B

Your path, your request, basically any JSON value, and then oppa is going to evaluate this query based on the policies and the data it has access to and send a decision back to your service, and this decision can again be any JSON value, allow' denied through false any JSON value, and so the whole point sure is that we are trying to decouple the policy enforcement from the decision-making and so to oppa.

B

It does not matter if you are doing Kafka HTTP API is SSH pseudo to cough go to oppa, it's all just data, and that's why we say oppa is a general-purpose policy engine.

B

Just to give you a quick recap of oppas features at the core of oppa is a high-level declarative language called as rego and so with Raghu. You can answer questions such as can exit row operation. Why, on reso see or what users is a feel is what Fields is a user allowed to see? So with Rago you can get more than just boolean allowed the night through false kind of decisions.

B

You can get decisions which are collections of values, opai's written in go and you can embed it as a library or run it as a sidecar or a host level daemon, it's designed to be as lightweight as possible. So all the policies and the data that you need are kept in memory and it does not have any runtime dependencies, so it doesn't need anything like an external call to do a policy evaluation.

B

Opa can obviously be expanded to talk to external services for getting policy and data. It can upload its status logs to external servers and also its decision logs the decisions that it made in the rule that was queried and so forth. All this information that can help in auditing and debugging OPA can upload this to your external servers, and so finally, along with the core policy engine, OPA provides rich set of tooling that you can use to build and debug your policies.

B

We provide you a unit test framework which you can use to test your policies. We also provide you an interactive shell to write queries as well as integrations with editors like vs code and wim, and so just how does OPA actually work? Now, we've seen this figure before your service gets a request. Your service asks oppa, whether it is allowed or not.

B

Oppa will evaluate the request based on the policy and data it has access to and send a decision back to your service, and so now imagine you wanted to enforce a policy in English which said that employees can read their own salary and the salary of anyone they manage so now, they're going to take this policy in English and try to implement it oppa by adding some regu, and so like.

B

I said when, when your service asks oppa decision, it needs to provide some kind of input to oppa like your method, your path or the authenticated user. Who makes the request, and so now, let's take this policy and try to write some Rago to implement this policy in oppa.

B

So to do that to help you get started with oppa so that you can write more policies today, I'm going to give you three steps to getting started with oppa I like to call them the three steps to oppa and so step number one. Is you clone the oppa repo? So all you have to do. You have to go to github.com, slash, open slaughter policy, agent, slash oppa and you just clone the oppa repo, simple step 1 once you've cloned the repo you just build the oppa binary so basically go in the directory.

B

Do a make, build or a go build and that will generate for you, a beautiful oppa binary and lastly, you simply run the binary. So once you go to github clone oppa, then you build the binary. And finally, you run the binary. You are going to get an interactive shell which you can use to write queries in which you can use to experiment with oppa policies. So it's that simple. So, just with these three steps, you can get started with oppa, literally right now and start writing some Rago policies. It's that easy seriously.

C

B

So can we do better like this? Is okay like this is good? Three steps are easy, but the question is: can we do better and so answer again is yes, so recently we released a regu playground which you can use to write an experiment with Lego policies. It's available at play, dot, open policy, agent, dot, org, and you can use the playground right now to start experimenting with Lego policies.

B

So this is how the playground looks like you can immediately see some syntax highlighting for Lego core, which makes it easy to read your as well as to debug your Lego policies, and so the example we were taking before is that an employee should be able to read his own salary, and so the way you read this policy is that allow is through.

B

Allow is true if input dot method is get and input path, a salary, employee, ID and input dot user is employee, ID, and so the cool thing about this policy is that the employee, ID value is going to be bound from a value from the input, and so the way to give an input to this policy is simply by clicking the input button.

B

So you have pre-loaded an input to this particular policy, and so the question I'm asking here is: can Bob see his own salary and to evaluate the policy? You simply click the output button and the answer is yes: Bob can see his own salary, which is what we wanted now imagine. Alice is curious about Bob salary and she wants to see Bob salary. So if Alice wants to see Bob salary is that allowed can Alice see Bob's salary?

B

The result is false, so Alice will not be able to see Bob salary and the reason is because the expression on line number 8 is going to fail because the user making the request is Alice, but the employee ID variable gets bound to Bob and now, let's imagine that Alice gets a promotion and she becomes Bob's manager. So the second part of a policy said that a manager should be able to see the employees salary.

B

So you need to tell oppa this new information about Alice being Bob's manager, and so you can imagine this information being stored somewhere in your LDAP server and you can pull that into the power, see and use that as a context to your policies. So in this case just for demo purposes, I'm just going to hard code.

B

This information in the policy itself, so I can say something like managers and, let's say, bob's manager is Alice and Fred and let's say Alice's manager is Fred, and so now we need to simply extend our policy to make use of this new information. And so what we can say is managers.

B

So now the person making the request can also be a manager of the person whose salary is being queried for, and so, if I have typed correctly, if I again ask the same question can Alice who's Bob's manager access his salary? This time the result should be true, so Alice who's Bob's manager can now access his salary. So in this way you can take a simple policy in English and write a simple Rhaego code to implement or enforce that in OPA, and so one of the cool things of the playground it lets.

B

You evaluate certain sections of the policy, so, for example, if I only want to evaluate the allow rule, I can simply select that and just evaluate that, and the result is true and so now that you've written a first Rhaego policy. Let's share this policy with friends and family and the playground allows you to do that using the share button. It gives you a link which you can simply copy and you can share with anybody you want to, but before you do that, let's see what's in the link.

B

So when you go to the link, you see the policy that you wrote, which is what you expect and that's good, but there's more if you click on the input, you'll also get the input that you had provided the policy, and so now again, if you evaluate the same policy, you should get the result as true, which is what we expect and so I hope. You guys now use the regu playground to experiment with Lego policies and share your policies with the whole world.

B

So this figure just emphasizes the fact how oppa can be used with all of these systems and Hawaii. It's a general purpose policy engine, and so the cool thing here is that we have integrations with each one of these systems and you can start using them out of the box without having to write even a single line of code. And so now, let's look at the data protection use case.

B

So this is so like I mentioned before. Safe is a distributed storage system. It provides object, block and file, system, storage and unified manner, and this is a high-level architecture of safe.

B

And so what we are going to focus on is the radwells gateway, which is basically a gateway which allows access to users and applications to safe storage cluster and how opah is integrated with this component and so the way the integration works is that when a user asks for access to a specific bucket, that request will go to the radaway State Way and the Gateway will again ask oppa whether this request is allowed or not.

B

And in this case that request will contain the user, making the request, the method name and the bucket name that the user is trying to access, and so OPA again is going to evaluate this query. Based on the policies and the data it has access to and send a decision back to the Gateway and depending on. If it's an allow or deny the data will then be fetched from the object, storage and so again.

B

The other high level goal is the same that we are trying to decouple the policy enforcement from the policy decision-making, and so now, let's say we have this policy that we want to enforce in self.

B

It's a location-based policy which says that users can access only those buckets belonging to the same geographical region as them, and so now, let's see we are going to go back to the playground and see how we've implemented this policy in Rago.

B

So the first thing you see is that I have hard-coded the user's location in the policy itself, but again this is for demo purposes.

B

You can imagine this information being stored in get your LDAP server or anywhere else in the world, and so what I'm saying here is that Alice belongs to UK and Bob belongs to USA, have also hard-coded the bucket location in the policy itself and I'm saying there is a super-secret bucket which can which is located in the USA, and so let me emphasize again that all this information can be given to open as external context.

B

It doesn't need to be hard-coded in your policy, and so, if I give an input to this policy which says that Bob, who is from the u.s.a, wants to access a bucket for the super-secret bucket, which is in the USA. Is he allowed to do that and again? If you evaluate this policy, if you evaluate this rule, specifically, you are going to get a value through, which means Bob is gonna, be allowed to access the bucket super-secret bucket, which is locate in the USA.

B

But if you try to do this with Alice, so if Alice makes a request to access a bucket which is located in the USA, hopefully opus policy should kick in and prevent this request from going through.

B

And so now, if we evaluate this query, the result should be false, because obviously Alice does not belong to the US, and so she can't access any buckets located in the United States and so again, we've taken that policy in English and we've converted it into ragu code using such a simple Rhaego code, and so what you've done is we've taken this entire integration and we've converted this integration into a step by step, interactive tutorial, which you guys can try on by going to this specific link, and if you go to that link, you will not just find that self tutorial, but you'll also find a cool admission control tutorial with kubernetes, which you can try out as well.

B

So you simply do you start the scenario and you can go step by step to the entire integration and try out how we can authorize request using open and safe, so I encourage you guys to try it out later, and so this is what we started with. We said data is king. We said data is everywhere.

B

It's super critical is super sensitive and it's really important and so data security is of a big concern and so really important thing that you need to address, and then we saw how OPA the open-source general purpose policy engine can provide a framework which can do uniform policy enforcement across the stack in multiple systems and so I hope.

B

At this point we can update this figure a little to replace it with our favorite policy engine, the open policy agent, and so thank you so much for listening and if you guys want to learn more about OPA, please check out the style of booth. You can come there and, more importantly, if you guys want this special edition, really cool. Opa short, please stop by the booth, and if you have any questions so I can take some questions now.

B

D

Hey, so what was the thinking behind using a domain-specific language instead of like a like a subset of Python or something that's sort of well known and I? Guess the follow-up to that is: what's the library support like? Could you call out to like a Kafka queue from a policy so.

B

The first question was why you can't just do this in code not like in a programming language right, so you have so many systems each having different programming languages and then you're trying to express these policies, which are completely based on deeply nested JSON structures for these various systems. And then you have users who have to interact with these system and learn these different languages to enforce policies in all these distinct systems, and so what oppan does is like? It tries to uniformly enforce policies in any system.

B

So you do not need to go into specific languages and learn specific ways to enforce policies which only apply to that specific system. It gives you a generic way. You can do this for anything, and so imagine if you are something like a security operation person, you don't have to go and learn the details of all your services, which will be written in, go Python Java on all these languages.

B

You can simply write your policies using a unified framework like OPA, so that was the motivation behind not using a programming language and developing an old DSL. Second, question was about: can you make a call from the policy itself to Kafka.

B

The library sub, so we do have so like opposite, didn't go so you can embed it into your go code. That's one option and as far as the support for making calls you can make calls from the policy itself, so we have an HTTP send command which you can use to make calls externally from the policy itself. We don't recommend doing that because it's experimental right now, but there are ways you can do that.

E

I've to question the first question: Islam can talk a little bit about how policies being pushed to the agent that you're going to put in the host like? Is there any like when I update my policy right? Obviously, I want to make sure that my policy works in in the production environment. Is there a way to do the canary or dock launch for the policy in agent?

E

B

The question: is you deploying your policy in production? How do you make sure it's at doing? What it's supposed to do is that the gist okay cool so fit OPA? We provide you a unit test framework so like with any programming language. What we always recommend is before you deploy your policies in production or in kubernetes permission control. You can first unit test your policies thoroughly and only then you should go and deploy them in production.

B

So we provide a really cool or unit test framework which allows you to mock out a bunch of things in the policy, and you can you be recommended to do that before going into production, so that would be our recommended approach for that.

A

E

I guess the sanera see if you have like say 10,000 notes right like can I push my policy to just 1% of the notes see you know you always prepare for the worst. So.

B

Can you push the question is: can you push your policies? If you have 10,000 nodes, can you push them to specific nodes? That's that's the question. Okay. So there are multiple ways in which you can push policies into OPA like you can have OPA talk to your external service and it can pull policies from certain endpoints or because OPA does not provide. It provides you of api's for the control plane. It does not implement the control plane yourself.

B

So that is a responsibility for the user and there are commercial options like styrodur that, for you provide you the control plane, and so there are multi and the other option is using REST API. If you, if you develop your own system, you can push policies into specific opus that you want, as per your use case. So there are multiple ways you can do this.

F

I hope, thank you very much. What would be the general approach for using this with CFS I think that example was using the Ceph object store right? Is it quite generic in the sense that you could use the same print suppose to use the open policy agent with a set of s using work, for example?

F

So if you had SAP FS, okay.

B

F

You wanted to do policies around that, instead of objects or the general approach, so.

B

The approach is pretty constant, you're always trying to decouple the enforcement from decision-making so like in the case of SEP, the rad always gateway made a call to oppa oppa made a decision, send it back to the OS dad OS for enforcement. The same principle will apply event force ffs the agent or whatever is the server which is running the agents. Ffs agent is going to make a call out to oppa oppa by Jerry decision back to your FS, and then you can authorize the access to your file system. So the principle remains the same.

B

Irrespective of your system.

G

It's an adapter available for op my D connect or draw tokens to read the identities out of two scopes from access tokens, for example, can.

B

You please give me the question: I couldn't hear ya.

G

So identities, if I have an identity talking from opmod I connect, can I somehow integrated with OTA or with open policy agent or or do I get our identities of an identity token into the system. So in a connector there, because normally I have for authentication solution which provides me up. My t connector essentially so.

B

The question is: how does oppa handle identities in a way? Okay, cool oppa has support for your tokens. So if you give your job to oppa, oppa is going to decode that and take the claims out and you can make a policy based on the jot. We have support for that. Also oppa oppa has support for x.509 certificates. So even if you give think the certificate you can make a decision based on the stand in the certificate as well. So there are ways you can you can handle identities in oppa. Other option could be.

B

You can connect oppa to your LDAP server, get fetch it from there and use that as well as identity for user or whatever. Alright.

A

H

Or two more yeah.

E

H

You for your talk, if you had a policy where someone requests access and they are able to access under two different options, let's say they are both. You have two claims through which they could gain access. Is it possible for audit reasons to see which claim granted them access? Ultimately,.

B

I couldn't understand the question I'm. Sorry. If.

H

I write a policy ah that says: I can access my own salary and I can access the salary of anyone. You know below me in the company or including myself, then maybe there's two ways: I can gain access to the salary. Can you show which policy was triggered or which one causes a result to occur, because in the example you gave you just had resulted true, weren't able to say why that result was true. Okay,.

B

So yes, so the example I gave only showed you a boolean response, but OPA can return to you any kind of JSON value, so you can have you can make policies around that we say: okay, why a certain policy work and return a specific response. As for that match so yeah, depending on who you write your policy, you can return a specific reason why the policy or which policy was applied or you can work through that for sure, because it supports any random JSON value as a decision.

C

I have a couple of questions. One is: is there any way to manage all to identify conflicts, so I can create policies I'm, a human being, I make mistakes, no conflicts? Is there any methods of managing conflicts? Okay and the second question is I'm. Guessing all of the policies are kept on a file right like the it's a file with all the different policies. Is it kept encrypted? What is preventing potential attacker from adding rules, putting a line at the top, which is all allow or something like that? Okay,.

B

Cool, so the first question was: how do you handle conflicts, so the regul anguish itself is where expressive, and it allows you to make a decision on that. So you can do something like your denies are going to override your allows. So you can have those kind of rules which you, as a user, can enforce in the policy and also we provide you like an else keyboard which gives you an ordering for your statements. So it's wholly up to the user, how they want to resolve context in the policy.

B

Opa is very flexible in that way, and so the second question was: if there is policy in OPA, what's stopping somebody from just going in there and changing the policy or deleting the policy right, so that's possible too. You can lock down the OPA agent using TLS and using a start up authorization policy, and you can only allow certain users access to certain paths and certain policies within OPA. So you can. You can enforce such kind of security policies on the agent itself, so yeah. There are options for that too. All.

A

Right, we are out of time, so let's go ahead and thank the speaker thank.

B

You so much for coming and.