Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Policy Matters! The Why, What, and How of Kubernetes Policy Management

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Policy Matters! The Why, What, and How of Kubernetes Policy Management - Jayashree Ramanathan, RedHat; Aradhna Chetal, TIAA; Jim Bugwadia, Nirmata; Robert Ficcaglia, SunStone Secure

Policies help address several critical challenges with managing Kubernetes clusters and workloads. In the panel-style session moderated by Robert Ficcaglia, co-chair of the Kubernetes Policy Working Group, contributors from the working group will discuss why policies are important to enterprises and other organizations using Kubernetes. Attendees will have an opportunity to ask policy engine and compliance experts about real world use cases and effective techniques for managing policy-as-code from a small cluster to enterprise multi-cluster scale. Attendees will see examples of how to map real world threats to their systems to specific policy automation tools and how compliance automation helps defend against emerging risks and provides greater visibility and enforcement of best practices.

A

Hello, everyone: this is the kubecon session policy matters. Why? What and how of kubernetes policy management? We've got uh jaya, ramanathan, radha, chatal and jim bagwadia with us today to talk about all things policy. uh Why don't we start with some brief introductions? Radna.

B

Good afternoon all my name is: I manage the cloud security team at tiaa and also I'm the co-chair for cncf um technical advisory group. Thank you. Yeah.

C

Thanks hi everyone, um my name is jaya ramanathan. I am a distribution engineer at red hat and focused on security, and governance are really excited to be here, because this policy management is one of my uh passion topics. At this point. Oh dear jim,.

D

Hey everyone- this is jim beguardia, co-founder and ceo at nermata, and I'm also a maintainer in caverno.

A

Hi and I'm robert vicalia, I'm a policy work group co-chair also volunteer with the cncf security tag and cto at sunstone, so the policy work group. We are focused on all things policy, but specifically looking at communities policy itself and how kubernetes operators can use policy as code. We have two current projects, uh but before we talk about that, just some infrastructure logistics we meet every other wednesday 8 a.m. Pacific.

A

We have our own slack channel and here's a link to our github repository where we have some of the prototypes uh jim's gonna talk about next.

D

Yeah, so one of the efforts and initiatives we're you know leading in the policy working group is creating a common way of reporting policy reports right. So one thing we found is, as as kubernetes policies become increasingly important for production deployments.

D

There are several policy tools and different tools have different languages, uh perhaps different features, but what seemed to be missing and something which we felt could be common is a way of reporting results from these policy engines. So the policy report, crd, is that effort and working with the community. We have now several tools like kiverno, coop bench, falco and others supporting the policy report.

D

There's more, you know, integrations and adapters in progress, uh and we continue to sort of expand um other. You know the outreach and projects that can work with the policy report.

A

Yeah and we're absolutely looking for for more projects to incorporate into the uh the policy report crd process so jaya- maybe you we we've been working on this policy white paper for now for for a little bit of time. uh Maybe you could give us a brief intro on the what and how is that we've been discussing in this white paper.

C

Yeah, thank you, robert um one of the things uh we we have been discussing within the work group is, you know. As customers are transforming to adopt cloud, they still need to meet security requirements, regulatory compliance requirements etc. Right, so they need to make sure that the cloud is configured properly for various controls and the sres that are managing the clouds are not necessarily the experts in all the aspects.

C

So this is where policy management comes into play, where the best practices are represented as policies and used to make sure that the controls are configured properly. So we wanted to kind of put together overall uh best practices for policy management that covers. You know what the goals, the overall architecture, as well as how a customer can go about implementing this approach. So that's really what this white paper is about. It's conceptual and uh we plan to then have an identity that references some existing open source technologies uh on implementing the concepts of client here.

C

So we definitely uh welcome participation from the broader community to help us progress. This further.

A

Thank you yeah, so well, let's jump right into it then, and have the panel talk about the whys and some of the details behind how you should manage policy and communities so aradna? Let me throw this to you first, why? Why should we care about policies, devops operators, kubernetes cluster operators? Why can't I just deploy containers and forget about policy.

B

So robert, the misnomer in the industry, is that containers are secure by default, which is not the case as we've seen from a number of breaches that have happened in the industry. The attack surface is just too wide infrastructure threats operating system threats. um You know.

D

Network isolation.

B

Threats, uh software supply chain security threats. um So if you look at the cncf landscape, there are a number of tools and technologies which are provided there that are used to build all this um container platforms, as well as the ci cd pipelines.

B

The complexity of integration of all these tools into the deployment pipelines is tremendous so to improve the developer. Experience provide them a frictionless way to go and deploy these containers in runtime. um I think it's really important to start thinking security policy as code, and that is what this working group has been working on. Defining.

C

B

Are those security controls and how can they be built into security policies that can be deployed at different stages of a platform? So developers have a frictionless deployment experience and yet it is secure from the start and in the runtime as well.

A

Great point: I want to come back to the the mapping that to the the how but before I did one more, why question to jim, so why we have some kube bench and we have things like cis. Why is that not enough.

D

Right yeah, so when you think about kubernetes, of course, kubernetes doesn't exist in isolation. It builds on tops of many layers like, of course, the cloud or your infrastructure.

D

If you're, you know running uh things on-prem or within a data center um and, of course, within kubernetes, there's also several components: there's several different roles right, so things like cis benchmarks tools like bench, which you know manage cis benchmarks, address one specific layer of that, but one of the things we detail in the in the white paper and, of course, which is also very important in practice. Is you really need to think about the cluster, the containers and the containerized workloads you need to think about?

D

You know the different roles, interacting with kubernetes and use policies as a contract uh across these roads right, so the yeah, the the sort of key takeaway there is making sure that every layer in this stack is is covered, uh including the containerized workloads application, the declarative configurations in kubernetes, which can also be managed as code.

A

So so, just to jump back quickly right now to what you were saying, how how do you accomplish all that complexity and map it back to security? How can you do that in a real world, you know always changing kubernetes environment. What have you seen out in the real world.

B

So in the real world, um obviously all these security controls that can mitigate the threats. um Obviously you have to do a threat model and based on that threat model, you know what your tax surface is, and you want to minimize your attack surface and define what are those security controls which will mitigate specific threats and then convert them into security policies.

B

Obviously you need a policy administration point where you can define all those policies and then also policy enforcement points at different places, including the cicd pipelines, where you have admission controllers, which can validate various metadata elements from the incoming deployment uh deployment of containers and services and based on their policies, they may allow them to go, deploy or may not.

C

Allow them to deploy.

B

Based on all the policy requirements that have been set up by the security and compliance teams to reduce their tax service, ultimately and same thing can be done in runtime as well. uh There can be runtime security controls which look for continuous. You know, validation of the images running. You can have a totally different image in the cicd pipeline and what is actually running in production could be totally different, so image, programs, um container, breakouts and several other threads in runtime can be enforced through policy as well.

A

Andrea, what do you see out there in the field? What are these kind of the key security implementations that move the needle.

C

Yeah, I think what I'm seeing is: customers are driven by enterprise security requirements, meeting those standards, as well as the relatory compliance requirements right, uh whether it is depending upon the industry, they are in, whether it's pci or hipaa or fisma, etc. Right so and what they want is, and they have to go through so many audits and some of them, you know, have to have to be done annually right. So that is really the pain point, and uh that extends obviously they've been doing this for traditional id and now for cloud.

C

They have to deal with the same issues right. So what I'm finding is that customers are resorting to policy management more as a way to manage configuration, because if you look at the end at the end of the day right, if you, if you take a particular compliance standard and then you look at a control in that standard and then you're implementing that control using a technology, you need to make sure that that technology for the control is configured properly.

C

So it's really a configuration management problem because you have a desired configuration state which is based on best practices. Like, for example, using tls 1.3 or using strong, ciphers, etcetera right and you are essentially defining policies to ensure that those configurations are set up properly right and imagine if you are able to accomplish that desired configuration straight for every control at every layer of your software stack through all the life cycles, ci cd run time, etc. Right, that's kind of the ideal vision right. If I can accomplish that, then I'm continuously security ready.

C

I'm continuously audit ready right. So that's really the eventual goal of policy management and what I'm seeing customers doing is. They are starting to scratch the surface. They are definitely looking at things that they are automating today, using whatever homegrown scripts and so on. They are now converting those into policies and applying policy management techniques to those as the first step and then then taking it to the next level. Saying: okay, now I'm using this particular vendor for implementing this security control. Now.

D

They're telling those.

C

Vendors, can you now implement best practices for your technology as policies and give it to me right? That's what um customers are doing, so I I think I am seeing I'm very hopeful, I'm seeing a lot of progress in this space, so I feel, like the time, has come to kind of realize this mission.

A

And jim just to talk to that vendor perspective, because as a an implementer of a policy engine you're, seeing where customer needs meet the technology needs. So what are you seeing as those those key asks from the community and and the larger kubernetes operator?.

D

Yeah yeah, so so policies, you know like ji, was very well articulating. Certainly it fits within the configuration management realm of kubernetes and other systems. There's also, you know when you think about policies. Typically, people think about governance and compliance, those type of security realms as well right, but when applying that policies can also be used for automation for reducing friction, reducing handoffs across different roles like developers, operators that are concerned with kubernetes right. So one of the things that was especially what we see uh as very important.

D

One of the reasons why we've been built caverno and are looking at you know doing things in a very kubernetes native manner, is to make sure that the operations team is very comfortable, implementing managing extending those policies and using devops best practices to policies as well right, so not treating it as a foreign realm which some somehow some central team does separately, but making it part of the infrastructure s code and other devops best practices itself and narada.

A

What is your experience, and how do you realize that that policy maintenance life cycle? What does that really look like as folks get kind of beyond the simple you know: here's how I define my policy document, what what challenges do they do they see when trying to maintain that over time and over clusters.

B

um Obviously, security controls means that you have to convert them into policies and security controls, change based on the landscape, right threat, landscape, sorry, based on the threat landscape, the security controls will change and the policies are updated, but then the developer community has to be aware of those policies and controls, because their deployments may fail based on a new control. So the communication is very important to the development community, about the new threats and the controls that are going to be implemented.

B

At the same time, enforcement points have to be the integration for the enforcement points and decision points have to be made as well and then obviously validation of the policies that they are not in conflict with each other. At the same time, thinking of policies as hierarchy is important right.

B

There could be some global policies that apply to everything and then there could be regional or you know, deployment a particular application deployment policies, so we can build an inheritance model, so you have global policies that are inherited by all the workloads, and then we can have application. Specific security policies as well based on the classification of the application, could be highly sensitive or moderate. um So but again, there's.

B

This is continuous work right um because imagine in a regulated environment um auditors tomorrow, rather than having these paper data, calls right where you provide spreadsheets and spreadsheets of answers. All they have to get. Is an output of a report from your policy enforce my points, so that makes life for the developers as well as the organizations much easier, and the audit is very easy to see where the violations are and where the focus needs to be in terms of improving security.

A

And actually on that focus point jaya.

A

What I'm hearing is this is a bit of a culture, evolution or a mashup, maybe of kind of old and new culture where you've you've got to be thinking about continuous uh policy implementation policy maintenance. But yet you have to you know, integrate that with existing itn and existing control structures. What what? What do you see and how you approach this.

C

Yeah, that's an excellent point. I think um I think that's why I feel like uh two things. One is: if I look at customers right, they don't just have one cluster that they are managing. Typically, it is a fleet of clusters right and like arana was saying you know some fleets. Some set of clusters are dedicated to certain application teams, etc.

C

Right, so really what we are looking for is you need a policy management that is multi-cluster and it is oriented toward splitting those clusters into different application teams and having a policy specific for those aspects right, and that said, I think we also need to think about existing idea, operational processes and tools that customers are using today, um that they use, for example, to prepare for audits, to make sure that the of their security posture day-to-day, how the action incidents etc right.

C

So this is where I think, combining the policy management or policy-based governance with the existing I.t operational processes. I think really that um I'm using the term automated governance to refer to that, which is to say you know, customers are already automating things now. If we can add the element of governance to it through policies, then we can basically say um here: are your best practices they're represented as policies here are the violations you can detect using this uh system, and now you can trigger your existing automations to fix those violations right.

C

So you close the loop right, so I think that's really where we are headed so multiple multi-cluster, automated governance is the way to go. In my view,.

A

And jim you know again developers and and devops folks who come to you and and who are interacting with caverno in the community. uh They have to kind of build a business case, even if it's in the open source world they have to focus time and attention is that is that argument about automation across governance across controls? Is that the argument they're using? Are there other arguments that are effective in you know? Getting their teams excited about this getting their their management.

D

Yeah, so what we're seeing is a strong push towards policy-based operations right. So if you look at kubernetes today and of course, control planes, um we know that most enterprises tend to be in a hybrid cloud or hybrid multi-cloud control planes are, you know, being run on public cloud, private cloud edge data centers things like that. So, as you know, jaya was also pointing out if you need central management across all of these clusters.

D

What's really needed is autonomy across these devnops roles, but you need strong alignment and policies become the way to do that. So really operational efficiencies becomes the very very much in kubernetes.

D

uh The first reason why policy management becomes uh critical and then, of course, securing kubernetes and making it secure by default, like aradna, was pointing out becomes uh another leading reason which is equally, if not more important, and then you know we we hear about use cases around compliance and uh governance and other mappings right. So those are the sort of hierarchy of needs. If you will, if we see uh is starting with you know, the basic is look. Kubernetes is complex, make it easy all right. We solved that now.

D

Is it secure and how do we make sure all of our clusters are secure by default and we eliminate all these manual steps and then what else can we do with it? Can we get to that push button, compliance reporting that push button?

D

You know kind of governance, integration with that complete feedback loop through policy as code best practices.

A

I'm going to open this up to all of you, rad and I'll start with you, so how.

D

A

Know that I'm doing this correctly that I'm implementing this I'm managing this well over time.

B

So, in my opinion, I think you have to improve this iteratively on the day, one you're not going to have a full repository of all the policies right policies, as you learn I mean, of course there are best practices out there that can be converted into policies as a starting point, but from there it has to be continuous loop of feedback from the intelligence that you get and you continue to improve and harden your policies from there um and maturity is built over time. um Obviously, because um again, the threat landscape is gonna change.

B

C

D

B

Policies over time um and slowly from one cluster to multi-cluster and cross-cloud clusters, obviously, and uh with solutions like anthos and eks on-prem um you're gonna have hybrid clusters out there as well.

A

Same question to you, how do you define success.

C

Yeah, so I think uh one one additional angle I wanted to add here is: uh we talked a lot about policies for security and compliance. There are also policies for resiliency and policies for software engineering standards right. Those are important as well. So I think um the reason I wanted to mention that is as an example right. Let's say that you have a policy for resource limits or a policy for liveness projects. Things like that.

C

These are things that um you want to kind of catch early, that an application developer is thinking about these things up front rather than you know, they're, not thinking about it, and then it becoming a crisis right because you're running out of resources on your cluster, you know, then they are called over the weekend or late in the night saying your application, is you know, failing or whatever right so so I think this is where I feel like um the policy management aspects span, those kind of criteria as well right, and I think um so, that's one point I wanted to convey.

C

The other thing is uh as aaron. I was talking about right um when you first start, you do have to make some investment, because you are going to put this architecture in place right, and so you won't get the benefits right away, but once you have it in place, then you will kind of see that your operations team is kind of having a view of things on a day-to-day basis.

C

Right, they're, not scrambling, then, to get ready for an audit right. It is a continuous process right, so it does like robert. You are saying it does require a change in culture right of how it operates, but it does have benefits because you can kind of manage on how things are configured. How do I action them? You know on a day-to-day basis right and then to the people who are listening to this panel.

C

This is more a call for participation into our community right, which is, we do have to build out a library of policies, so the the concepts that we have in our white paper talks about. You know various control areas and you know aspects, but we need to really build out that library, based on various policy engines that are out there like you, are now gatekeeper, etcetera right.

C

So I think um this is where I think the community participation in that to actually make this real is going to be extremely important uh because, like I said you know, you need this enabled for all controls across all layers of the stack so to get the full benefits.

A

And jim same question of a little twist anything that you think that the community developers need to enable to make this success more efficient or more possible.

D

Yeah, so certainly to me, success would be you know, happy developers and happy security and operations. Team means successful business right. That's.

B

D

Yeah, so getting to that point- and you know making this so seamless so easy, where it really seems like a nice productive experience without um you know worrying about the complexity, worrying about the security posture and things like that, but yeah to your to the specific question in terms of what else can be done in the community, so that the you know the power of kubernetes is its extensibility and that's where using native tools using native you know, policy as code best practices and really embracing kubernetes as a platform for building platforms.

D

I I think, there's more that can be done there, some of the things beyond what jaya just mentioned. What we're also doing um any the community is also starting to look- and this is the broader kubernetes community in general and the cncf um is. How do you make sure these admission controllers themselves are secured as well as protected right? What are the right controls to have in a place as you're looking at the different layers of your cloud security model?

D

So certainly, there's a lot of good ongoing work, and I think this is pushing and supply chain is another area where up admission controllers are also playing a critical role. So all of this is really you know, pushing the awareness, uh as well as the increased sort of improvedness off the security posture overall.

A

And then to close it up before we take questions uh 10 seconds, each aradna make some predictions. What are going to be the biggest challenges or the biggest wins the biggest successes in the space over the next 12 months.

B

I think in the multi-cloud cloud hybrid um deployments, where you have portability of applications from one cluster to another. I think policy management brings a big value because you can deploy at one place and then distribute wherever you wanted.

A

Your predictions.

C

Yeah my prediction is, though, this requires a change in culture, I'm kind of seeing that change happening. So so I'm optimistic that uh we will have like. I can dream big right.

C

So uh I'm hoping that we will get to a point where we have a library that says: okay, if you, if you deploy this library of policies, uh then your cluster is ready for this particular compliance standard right for the technical controls, I think that's that's kind of would be a nice goal to achieve and I feel like uh the time has come to do that.

C

Go big or go home.

D

Do you have your prediction yeah along similar lines, right so just uh automation, automation, automation and being secure by default.

A

Fantastic, my prediction is: we're going to grow the policy work group community, we're going to have more participants on our wednesday calls. Our white paper is going to be a success. So thank you. Thank you all for joining. In this conversation, it's been fantastic and very thought-provoking, and then I think we now have questions on the virtual platform.

A

Thank you, robert. Thank you thanks, robert.