Cloud Native Computing Foundation GitOpsCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Policy-Based GitOps: How Policies Can Help Secure and Automate GitOps... Jim Bugwadia & Avni Sharma

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Policy-Based GitOps: How Policies Can Help Secure and Automate GitOps Workflows - Jim Bugwadia, Nirmata & Avni Sharma, Intuit

GitOps is awesome for workflows such as managing sets of applications across fleets of clusters, and the provisioning of multi-tenant infrastructure for teams or end-users. However, there are often gaps in these workflows that require manual configuration or the creation of custom controllers. Additionally, these workflows remain hard to secure, and security best practices like “the least privilege principle” cannot be easily applied. In this session, Avni and Jim will show how Kubernetes-native policies can be used to secure and automate complex GitOps workflows. First, they will showcase use cases for using GitOps such as managing a consistent set of applications across multiple clusters and delivering multi-tenant “Namespaces-as-a-Service” and “Clusters-as-a-Service”, using ArgoCD. Then they will highlight the current gaps in automation and security. Next, they will demonstrate how Kyverno, a Kubernetes native policy engine, can be used with GitOps to address these critical gaps. Attendees will learn how to successfully use policies and GitOps together and also avoid common pitfalls when multiple controllers are in play.

A

Thank you Scott for the introduction. So, as Scott mentioned, our talk would be around how we can um use policies to have a secured and automated github's workflow hello, everyone. My name is avni Sharma I am currently a graduate student at the Indiana University I also interned as a product manager at Intuit for the Argo CD project and I've also been a software engineer at Red Hat. We also have your gym, who is co-founder and CEO of nirmata.

A

He is also co-chair of policy and multi-tenancy work groups, and he is the maintainer and co-creator of keburno. So this is the agenda. We would be going through the GitHub strands and what are the gaps uh currently in the github's workflows and how policies can help in minimizing those gaps, and then we would be demonstrating that with an example, demo workflow.

A

So let's quickly walk through some of the important GitHub strands. There was a user study that was conducted in 2022 by the Argo project, which showed that more than 80 percent of the respondents were using argosidian production and more than 66 percent of respondents have been using this for more than six months. In production, we also saw from the user study that um Advanced app patterns has led to um you know the impetus to application sets in the project, and it helps in going for multi-cluster deployments.

A

We can also see it from a snippet that is taken from that user survey that how um the Apple 5 patterns has helped in multi-cluster deployments. Also and application sets, is something that's being widely adopted in the community, and we've also seen from the survey that it's not only applications, but now git Ops is going Beyond applications where we are increasingly using it for infrastructure management as well.

A

So what doesn't work well, um security any day in kubernetes itself is very complex and hard, and it becomes even more hard with complex workflows, like multi-cluster deployments.

A

um Self-Service uh with guardrails is something uh that is not really cohesive and it doesn't really work well, so we want a model that has guard rails, which helps in preventing misconfigurations and helping and standardizing workflows. Automation across multiple projects and controllers is also something um that is tricky, and that is something where we can definitely improve.

A

So these are some of the gaps that are currently there and now Jim will go over how we can use policies in minimizing these gaps.

B

Thank you Omni. So let's talk a bit about what policies can do to help with this right, so the way I think about policies, especially in the kubernetes world, is there a contract between the different roles that end up sharing a cluster, so you typically have your developers or deploying the applications. The security teams, of course, who want to secure everything from the cluster configs to the you know, workloads the Pod security Etc, and then you have your operations team, that's managing all of the shared services and the cluster itself.

B

So policies become this digital contract between the dev second Ops roles. One of the main use cases we see with policies is is to prevent misconfigurations to be able to standardize and automate, and that's one of the other things it's like. Typically, when you think about policies, you think of them, as restricting things or enforcing certain types of checks, but policies can also be used for Automation and we've done the Us and other I.T systems. You know in several other domains. Why not do that with kubernetes, especially with the powerful declarative configuration Concepts?

B

We have so here in the demo we'll use kivarno, which is an admission controller, a policy engine designed for kubernetes, so bit. Kevano policies themselves are declarative resources like any other kubernetes resource. So if you're comfortable writing you know a deployment or a pod or any other kubernetes resource pack. Kiverno looks extremely familiar. There's no other language to learn. Nothing else, to kind of step out to kivano runs inside the cluster as an admission controller, but you can also use it as a CLI, a command line tool in your CI CD pipelines.

B

That can also do background scans. So lots of different flexibility in how you apply kuberno policies in terms of the policy rule sets itself you can use it for validation, so verify different configurations either just in audit mode or to block certain configurations. You can use it to mutate, different configurations as they're coming into the cluster or even generate brand new resources. Based on certain triggers that you set up, then those triggers could be existing resources. The triggers could also be new resources that are getting created. A classic example is when a namespace is created.

B

You might want to generate a default Network policy, different role, bindings roles, etc. For that particular namespace itself. Governor also, you know in the last talk we looked at cosine and how it can be used to sign and verify images. Kiverno has built-in integration for verification of image signatures, as well as yaml Manifest, also integrating with six door cosine. So that's a very you know widely adopted or growing use case as well. So here's a quick example of a caverno policy. This is an extremely simple policy.

B

It's just checking for your tag and making sure that the latest tag is not allowed right, so just to make sure that your tags are specifying a version and you don't have a mutable tag. Here's a more complex policy and one question we often get is well cavernos. You know nice and simple. It's so easy to get started with. Can you really do complex policies?

B

And our answer is almost always will show us the use case, and we can, you know, generate a policy for that, so kivano can do extremely complex policies which require API calls other you know kind of lookups. You can even fetch data from oci Registries like signatures layer, information, Etc, so everything you would do you know you would want to um for kubernetes guard rails. You can do in this example.

B

We're making an API call and then applying various James path, Expressions to be able to do certain logic in this case to block duplicate, Ingress source.

B

So one one note before we proceed further, so I mentioned that Kevin can do validation, mutation as well as generation of resources, mutate policies and githubs do end up sometimes conflicting with each other right, because with githubs, the whole idea is get as your source of Truth. You want to make the changes there. You can mutate certain elements, but then you have to make sure your git Ops controller can exclude those and you don't get into a reconciled Loop. You can also you know with some of the newer features like flux as server side apply.

B

Argo CD is also adding that feature in and with support for, manage. Fields policy engines can also claim you know ownership of certain Fields now within a resource manifest itself. So there is there. There are ways to work around this, but just be careful if you're adopting mutate policies all right.

B

So let's take a look at a sample workflow and we'll use this for the demo itself, so the goal that we cannot set out to solve when we were looking at this you know and year we wanted to use a combination of different tools, see how policies can help. We wanted to offer secure self-service clusters to teams uh within, let's say within an organization within an Enterprise.

B

We want to use standard tools wherever possible, like so command line, Cube cuddle, other tools which are already familiar, and we want this to be completely automated end to end right. So just some terminology we'll use during the demo. So let's assume you have a platform team. You know that's managing all of this infrastructure. The policy sets and the shared services. You have a cluster owner who is going to request a cluster and assume that role, so they can manage the life cycle of the cluster and they have full cluster admin permissions.

B

And then you have a management, cluster and a tenant cluster. So the management cluster is what you're going to use to provision new clusters using cluster API tenant clusters are the Clusters that your team- or you know folks within the organization, will end up using. So looking at the scenarios, you know some of the challenges we immediately encountered, as we were trying to build up to this demo, is first of all. Cluster config is complex right.

B

So, if folks, who have looked at cluster API, it's very declarative, very powerful, but there's gobs and gobs of configurations which is not something you want to allow everybody to edit and use. So you want to standardize those you want to automate how cluster configs are managed, Cappy or cluster apis. Also has this concept of templates now, and it has you know, clusters which can leverage those templates so we're going to use those and we're going to use policies to standardize the types of clusters somebody can request.

B

The other challenge is um every tenant. Cluster kubernetes, of course, wants to spun up. You bring up the control plane, the worker nodes that cluster itself is not secure, so you need to you know Harden that cluster. You need to put policy management on it. You need to put maybe other security tools other add-ons on that. So how do you standardize that delivery of add-ons with you know each of these new clusters that get spun up by by a user request itself?

B

uh Another challenge is the management cluster now becomes a shared resource. So, of course you could take the approach that nobody touches the management cluster, but we want it to be a little bit Bolder and say: let's share that cluster. Let's, let's see, if we can, you know enable multi-tenancy on that management. Cluster, allow multiple application teams to make requests and not trip over each other's. You know tenant clusters that they're provisioning itself right. So that's where again we're using policies for multi-tenancy with generate resources, enforce our back and other standard configs.

B

You will find in kubernetes itself. And finally, you know one of the other challenges is every cluster owner. So let's say I request a new cluster I have cluster admin, so there's nothing preventing in a vanilla, kubernetes deployment, there's nothing preventing the cluster owner from going and removing all the policies. Your platform team might have configured, so we wanted to prevent that and we'll show how that can be done as well, and you want to make sure that, even though you have cluster admin, there are certain restrictions and guardrails in place.

B

So with that context, let's dive in into the demo itself and avni will lead us through that.

A

Question so for the demo, there are certain prerequisites that we have established on the management cluster, so we are going to use Argo CD as our continuous delivery tool. This is going to be our GitHub software agent. Then we have cluster API, which we are going to use um for, creating and provisioning our tenant clusters and then, as Jim mentioned, we are going to use kivano as the policy engine, and all of this is already pre-installed in our management. Cluster we've also defined two cluster owner roles, who are gonna request for tenant clusters.

A

Those are Nancy and Ned.

A

So how are we going to create our or request our tenant cluster, so a user Nancy would um create a namespace, let's say its prefix with cluster or you can have any regex for that. We just wanted to distinguish a tenant cluster creation from a normal namespace creation. So let me just um create a cluster real quick here.

A

So I'm gonna create a namespace cluster one as Nancy, and this is going to trigger the tenant cluster creation.

A

So once um the namespace is created, it triggers off this workflow, or this is like cluster as a service concept. uh The policy that's there on the management cluster. It will create relevant gappy templates roles and role bindings for the specified namespace and the cluster API or copy also Provisions, the tenant cluster that is bringing up the worker nodes and the control plane.

A

Once we have our tenant cluster ready, there is a policy that generates secret to register that tenant cluster with Argo CD. For this. uh After after this workflow, we have Argo CD application sets which are already there installed in the management cluster and once um the cluster is registered with Argo CD.

A

um The Argo CD application set will start applying the Calico controller in each of the tenant cluster via app sets and then Argo CD will also apply the kivano controller in each of the Tenon clusters or via app sets, and once the kivano controller is installed up and running.

A

um Argo CD will apply the policies um in each of the tenant cluster with the help of application sets and to achieve all of this, we are using the cluster generator concept of the application set.

A

um So we know that the application set controller also it gleans the secret from which it knows what clusters to be registered with Argo CD, and we can also Target specific clusters with matching labels for this. In each secret we have a label which is cluster type, colon tenant and once the application set controller knows that it, it basically spuns out the Calico controller controller and all the policies for the tenant cluster.

A

um All of this is sequential and it is being enabled by using sync waves, because we want the Calico controller and the cni plug-in basically to be up and running first and then the kivano controller. And then the policies.

A

So y'all, let me just quickly refresh the apps.

A

Here on the UI, if we'll go under the Clusters, it will show us that our tenant cluster was registered and it's taking a while to load right now.

A

Anyway, I'll just show through the CLI.

A

So here we see that our tenant cluster um has been provisioned, we have a worker node and the control plane up and running. So let me now switch to my tenant cluster 1 context.

A

So the kivano Pod is initializing and let's wait for it to run.

A

So here you can see that the 10 in cluster, one that was spun up, is showing, as a listed cluster on the Argo CD UI.

A

I hope the screen is visible, should I also enlarge it: okay, okay, so now we can see that our keyword, no controller, is up and running um thanks to application sets.

A

Now let me try and edit a cluster policy which is disallow latest tags, and um so cluster policies has been set by the platform team and the intent is that nobody is able to actually edit this and once I try to let's say edit, an image regex here it should disallow.

A

So here are the web of the admission controller is kicking in, and it's not allowing us to really edit that cluster policy, which is being managed by the platform team. Now let me try and run a test image which is not signed, and it's not um gonna be validated by the policy engine.

A

It's a basic engine, X image.

A

And I'm trying to run it, but the policy engine is telling that it's not allowed because the image is not signed. So let me try and run again a test or with image that is signed.

A

So here, even though it's signed are we, we had certain policies that had to be validated like. We cannot spin up our deployment in default namespace. um The deployment requires some labels uh like app.cubernetes dot IO. uh The Pod also require probes, um Readiness, probes, Etc, and it requires request limits. So let me show a good deployment. Yaml.

A

So this is a good deployment camera which, which has everything that's um gonna, pass the validation uh from the kevano policy engine. So we we have a signed image. We have all the requests limits, Library list Pro, Readiness probe set. So if now I try to run this.

A

So we do remember that we needed it to be in a different namespace than the default namespace. So let me just create a namespace test.

A

And let's apply this good Channel there.

A

And now we see that the creation of that deployment succeeded, so the the controller acted up it. It helped in validating uh the deployment against all the policies that had been um added on the tenant cluster as a standard add-on. Now let me go to the management cluster again.

A

So we also um elucidated that how the management cluster itself for shared resource, so we need to have some namespace level tenancy as well. So we also have a user Ned who has not really created the tenant cluster, and so he shouldn't be able to list delete or do anything with the tenant cluster. So we can just try that by.

A

Cube cattle get clusters as net, and we see that the policy on the management cluster is showing that that is not allowed for the user net. If we do it as Nancy who had actually requested for the tenant cluster, it shows a success.

A

We also have to elucidate on a concept of creating a bad namespace that is requesting a cluster which is not being validated by the engine. So let's say we don't want to get crazy with. You know, worker nodes going more than three um and we we really want to keep the resources to a limit.

A

So for this example, we've specified that the control plane and worker nodes cannot exceed three, but in this bad namespace example, we see that we are providing something greater than three and these are provided as annotations.

A

So, if I try to apply this bad namespace, if we try to ask for a tenant cluster with something that's not valid, we would have an error.

A

So here you can see that there is a validation error and we can only have a maximum of three control plane and three worker nodes for this demo example. So we saw how we can achieve a policy based get Ops workflow, which will enable in having a standardized, automated and secure github's workflow. There was no manual handoff and it was all secure end to end.

A

In the demo, we went through creating a 10 inch gappy cluster and was provisioned based on configured standards. Githubs was used to configure required add-ons and policies on the tenant cluster for RC plus cluster. One and policies were also deployed to the tenant cluster for workload, security, best practices, image signing and manifest signing.

B

Quite a lot of things we covered in the demo there but, of course, there's still room to improve and do better right. So a few things we were thinking of, we would do. In addition, you know if we were to put this into any any anywhere near production use. So obviously you could also use git to provision the some of the initial workflows in this demo. We use Coop got all to just create a namespace, but that may be done through a pull request.

B

You also, ideally might want to separate roles and service accounts for the cluster admin as well. As you know, the Argo CD itself, which is also provisioning and managing things on the cluster, but we saw policies, can still be used, even if both end up with cluster admin, you can still add additional restrictions to your policies and, finally, you probably want to protect like your Calico and your Coupe system and other namespaces through additional policies as well. So just some key takeaways here I mean policies are not just about enforcement and validation.

B

Although that's one of the key use cases, you can do, end-to-end automation trigger you know and and use policies as that glue, as that handoff between different controllers, git, Ops and policies, work really nicely together and the combination is extremely powerful, as you saw in the demo, uh the one caveat being with mutate policies. There's new features coming in both you know, policy engines as well as github's controllers, to help with that, and you can.

B

You know the whole goal years to get to secure self-service and it's that's an interesting combination of words, because security by itself is difficult, self-service. Yes, you can automate in many different ways, but putting that together, you know. Otherwise, without things like policy engines and git, Ops becomes extremely daunting and a very complex task so feel free to you know. If you want to check out caberno or of course you know also Argo CD or any of the tools go to some of the links.

B

All of these projects are boots uh Wednesday through Friday, so stop by. If you have any other questions, here's some info on the caverno project itself, um and certainly if you want to get in touch with us or ask any questions or give feedback or anything else, you would like to see feel free to do that as well.

B