Cloud Native Computing Foundation GitOpsCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Secure Your GitOps - How to Implement a Robust Security Strategy - Todd Ekenstam, Intuit

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Secure Your GitOps - How to Implement a Robust Security Strategy - Todd Ekenstam, Intuit

Access control and security are especially crucial for managing the deployment of applications and infrastructure. Traditional security processes that rely on human operational knowledge will struggle to scale and meet the needs of enterprises utilizing automated build and release infrastructure. The GitOps methodology can improve the integrity and security of your application deployments while at the same time exposing new attack vectors that must be secured.

This talk will cover the following topics:
- Improving security with GitOps
- Implementing access control in GitOps
- Multi-tenant cluster and namespace management for GitOps
- Configuring access limitations for critical GitOps components
- Common security model patterns and best practices

This talk will teach you how to securely implement a GitOps methodology to deploy applications and cluster components to Kubernetes. You will also learn strategies to securely manage multi-tenant clusters and common security model patterns and best practices.

A

Secure your get ops and implement a robust security strategy.

A

So first a little bit about me, I'm an engineer at intuit on a platform team. That's building, secure, multi-tenant, kubernetes infrastructure, I'm also a co-author of git, ops and kubernetes, and actually this talk covers much of chapter six of that book. If you're interested, please check out the book so into us using kubernetes at a pretty large scale.

A

We have 350 plus clusters and in fact, if you filed your taxes last year with turbotax, you filed your taxes with kubernetes and for all those clusters we're using get ops kind of as a standard mechanism to deploy apps to kubernetes.

A

So first I want to kind of take a high level view. Look at the the get ops operator security pattern.

A

What this diagram is showing is, what are you know? Who are the subjects who are making changes and what are the the things that need to be protected? And what are some of the points in between that need to be secured and the rest of the talk is really going to be drilling down into each of these. Looking at specific things that you can do, but first, let's go through a typical process where you have an engineer, pushing code changes to the the code repo.

A

You would then have a ci pipeline. That's watching that code repo and would check out the code run. The unit test run other kinds of scans, uh depending on the the ci pipeline, that's defined, and then ultimately, it's going to push a new image to your image registry.

A

Then you're going to update your kubernetes deployment repo. This is your git ops, repo and you're, going to update that either with the new image or maybe some other infrastructure changes to your application.

A

Some other kubernetes changes that you want to be pushed out, and then the operator is watching that deployment, repo and and any merges to master will be applied to your kubernetes cluster. But before that, before that happens, you have the opportunity to have code reviews on that on those changes, approvals and that's one of the key kind of inherent security benefits of get ops is to allow you to have a second pair of ides.

A

That audit trail that people have talked about this afternoon as part of your normal process, and so once that is approved and merged to master, then the get ops operator can then deploy that out to your kubernetes cluster.

A

So, given all that, you know, we have some inherent benefits with for security with with get ops, but what are some specific things that we need to look at to secure it even more?

A

So what do you need to secure? Really? The answer is pretty much everything so there's a long list. uh So, on the the left hand, side uh are all the different components that that really need to be looked at closely.

A

The last two application and the cloud provider- those are a little bit outside the scope of git ops per se, so we're not going to be covering it, but the other six we'll be covering in this talk.

A

So here's a typical ci cd pipeline and really the cicd pipeline is defining your policies and your standards for deploying to production right. This is how you enforce your your engineering process, and so, as part of that, you really want to protect it. You don't want that pipeline to be bypassed. You don't want it to be compromised. It's really codifying what you think requires. uh You know quality software to be deployed to production, so the first step is securing that pipeline.

A

You know, obviously you want to restrict who has access to that pipeline? Who can make changes and really that's, because that pipeline is driving all the new software to production, to your different environments, and if that pipeline is compromised, then it can deliver harmful software, whether buggy software or or otherwise, for your ci pipeline in particular, you don't want to use long-lived sort of machine user credentials in the ci pipeline, for either the kubernetes clusters or for the cloud providers. You want to use short-lived credentials.

A

Next is the container registry, and most enterprises will use a private image registry to store their container images, not only for security, but also for reliability, performance and privacy.

A

If you're writing proprietary code, you don't really want to be pushing that to you know docker hub or something like that, so at least at into it we have our own private image registry and once you start managing your own private image right excuse me private image registry.

A

You can have some more configuration and control over that registry, and one thing is you'd want to make it so that image tags are immutable. You don't want to change the meaning of an image tag once it's set.

A

If you've deployed an image to production, you don't want that image tag to be changed out from underneath that running system.

A

You also want to use tools like twist lock, prismacloud to scan those images for vulnerabilities and we'll come back a little bit in a couple slides to talk more about the container registry.

A

Next is securing the git repository itself, so the the get commit log and the revision history of your get ops deployment repository is the audit log. You know it tells you everything, that's going on everything that's being deployed to your kubernetes cluster, and so it tells you what changes were made when they were made by whom, and hopefully, if you have a descriptive, commit message.

A

It'll tell you why right, and so you really want to use pull requests and code reviews for all the changes to the deployment repo so that you have a good audibility, the ability to roll back, and you know exactly what's going into your cluster, it's also desirable to use multiple deployment repos for each team or application.

A

This allows you to have granular security access controls on those repositories.

A

Different teams can be restricted from accessing certain git ops repositories or, if you're, using a single repository, you can take advantage of the code owner's file, that's a feature in get ops to or a github that allows you to control which teams can access and employ merges approved, merges for certain branches of code.

A

Also, you want to closely control master branch, merge and write access to the github, the get ops deployment repos because really, in effect, whenever you merge something to those repositories that those changes are being propagated to your cluster. So it's the same as giving those developers cluster access.

A

So you want to closely control who can approve, merges and and affect those changes.

A

Next for the get repositories you need to enable branch, protection rules for the master branch and the three that I list here require pull requests for before merging. That is basically means that you need code reviews of those pr's before they're merged to master. Now keep in mind these are not. You know. This is your your get ops deployment repo? So it's not traditional programmatic code, but it is your your desired state of your cluster.

A

uh You want to have a second pair of eyes on all those changes, so you definitely want to enable request reviews before merging. Also you want to require review from code owners like mentioned previously, if you you're, using the code owner's file to define who has merge authority, and then you also want to enforce automated checks.

A

This is where you can run unit tests effectively unit tests on your configuration before it's even deployed to the cluster, um and you know at a minimum. You can do some kind of linting or some kind of syntax chain uh checks of your configuration, but then you can even go deeper and do some static code analysis to really implement deep policy enforcement, and then again, this is even before being deployed to the cluster itself.

A

Next is securing the kubernetes cluster.

A

Ideally, you would require that only images that have successfully passed through your ci cd pipeline be allowed to be deployed into your cluster. You really don't want developers to be able to side load images right. You want to ensure that it's gone through your process and and gone through all the steps in your in your pipeline.

A

You also don't want to allow images with known vulnerabilities to be deployed, which is hopefully part of part of that pipeline is to identify those so there's you know this is. This is difficult. This is not necessarily easy, but you need to do some type of signature of those images when it's going through the ci pipeline, and you also need to use some type of admission web book like opa, to require that all images being deployed into the cluster have that kind of signature and validate that signature.

A

Next, don't don't allow the latest image tag, that's just a bad idea and you want to verify compliance with the cis kubernetes benchmark. This is a whole topic. A whole talk in itself, there's probably additional talks over the next few days on this.

A

But you know the cis kubernetes benchmark is a good place to start to make sure that your cluster itself is secure.

A

Then, as we talked about earlier, you really want to have your own private image registry, but as part of your kubernetes cluster, you want to block image pools from any untrusted registries. In other words, you don't want to allow polls from you know: docker dot, docker hub or any other public internet registries.

A

And how do you do that? Well, at intuit, we use opa, and this is a little uh code snippet, which is really the one of the first tutorials for oppa on how to how to block untrusted registries. It's a very common pattern. If you're not already doing this, you know strongly recommend checking out opa.

A

We use it not only for this, but for a lot of other policy enforcement.

A

Next is you need to secure your secrets? You know, as was just mentioned in the previous talk, you can't just commit sensitive information to get right. That's a bad idea.

A

Kubernetes secrets are not encrypted right. You need to think of some other solution for doing that. You also don't want to bake sensitive information into the container images themselves. You might think. Okay, maybe that's a different way. We could do it, but that's also problematic. So it really comes down to three different ways.

A

You could do this uh one is you could use an out-of-band process, you know, in other words, not get ops for deploying the secrets to a cluster, so you would maybe use good ops for all your other deployments, and you know everything else, but then secrets may need to be manually provisioned, but that's kind of a pain and doesn't really follow the whole benefits of get ups that we're trying to achieve in the first place.

A

So the next one is, perhaps you could store encrypted secrets in git.

A

So if you encrypt the secret and then commit it to git, then that's that's okay, but then how do you get the key to unencrypt it right? So there's always some secret that you need at some point, but luckily there's some patterns that allow you to do that. So sealed secrets is one. You know, there's other approaches that allow you to do that as well.

A

The last is to use some type of external secret management system. This is where the application containers retrieve the secrets at runtime and example of this is vault and also at intuit. We have our own proprietary system that does this.

A

Then next, obviously, you need to secure your get ops operator itself and how you do. This might depend a little bit on which operator you're, using whether it's argo, cd or flux, but there's some common themes, and you know at the end of the day you want to use the principle of least privilege so that the get ops operator only has the permissions that it needs.

A

If you're, not, you know, if you're not deploying a certain type of object uh through get ops, don't let the get ops operator uh have privileges to write that type of object.

A

You definitely want to limit the get ops operator, cluster role right privileges, so anything that's a cluster-wide access. You want to be very careful about it, especially if you have sort of end users or developers accessing that your git ops and managing their own git ops pipelines.

A

You want to only give the get ops operator the privileges that the dev teams themselves would have to the cluster. So don't give them don't give gitop's operator cluster admin privileges.

A

Then you want to also enforce team specific security checks in the good ops operator, and this is where the multi-tenant aspect comes in. If you have multiple teams, you don't want one team to deploy to another team's namespace. You don't want other teams to be able to deploy another team software. You want to have them strictly controlled to only the teams that are allowed to do those operations.

A

You also want to maintain the audit log of the get ops operator, because again, this is your.

A

This is telling you everything. That's happening in your cluster, everything that's being deployed, and if, if you need to go back for forensics and see what happened, you can understand what what the flow was that caused a certain operation to happen.

A

And, lastly, you need to have a disaster recovery plan for your good ops operator or your get ops infrastructure, because likely in the event of a disaster or some other type of incident. Other teams are going to need to use that infrastructure to restore their services. So really the get ops infrastructure is a critical foundational piece.

A

Quickly, I want to talk about the get ops, multi-tenant management, so at intuit we have the our cluster platform team that provides get ops infrastructure to each tenant, and then each tenant has one or more name spaces that they're allowed to deploy into, and the git ops is configured to only allow those tenants to deploy to their name spaces and we've specifically prohibited cluster scoped resources, such as demon sets and cluster roles, and other things like that.

A

We also use limit range resource quota, opa and other mechanisms to enforce policy. And again you don't want those to be manipulated incorrectly, and you want to limit that access in the get ops infrastructure now. At the same time, the platform team itself would like to use git ops to manage the clusters, so we have a separate set of get ops infrastructure, that's used by the platform team that has elevated privileges that lets you do those types of things you know deploy cluster scoped add-ons manage the attendant namespaces and do the policy provisioning.

A

That's all I have for this afternoon. Are there any questions.