Continuous Delivery Foundation GitOps Summit 2021, 9 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Self Healing GitOps: Continuous, Secure GitOps using Flux, Helm and OPA - Om Moolchandani, Accurics

Description

Self Healing GitOps: Continuous, Secure GitOps using Flux, Helm and OPA - Om Moolchandani, Accurics

Flux empowers the community to adopt GitOps for K8s without a separate CD tool. Flux triggers automated operations for cluster reconciliation by monitoring changes in git for images and artifacts such as Helm Charts. While Flux enables hyper automation for cluster deployment, how can teams ensure they aren't slowed down by requirements such as security, privacy and compliance? In this talk, Om Moolchandani will discuss how to leverage the power of the Open Policy Agent to automate delivery of secure, compliant deployments. Flux with OPA can ensure that any Helm charts and container images to be deployed, are compliant with the established policies. Om will also demonstrate a new approach of self healing GitOps to the community which leverages OPA's Rego language to remediate risks and violations on the fly.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Hello, everyone and welcome to my talk: self-healing githubs, continuous, secure git tops using flux, hem and opa.

A

My name is omul chandani, I'm co-founder cso and cto at a qrx. My favorite attack technique is waterhole attack and I love cyber security and mechanical keyboards, especially holy banner switches and sa keycaps.

A

Let's take a look into the agenda for today's talk.

A

We're going to see how containers and kubernetes systems are built, especially using github's pipelines, we're going to also talk about what is policy is code and how it can be used for securing your kubernetes clusters, especially when you're using github's pipelines and then how does policy score help secure the overall ecosystem when it comes to kubernetes, git, ops and many other processes associated with it, and we can also see a demo of how we can be how we'll be using policy as code, an open source project called terascan to secure a githubs pipeline built on flux and on gitlab.

A

So let's get started so how containers are built today. Let's take a look when we use githubs how exactly the process looks like so, of course, in your local development environment. Any developer, who writes application code ends up checking in the code into some sort of source code management tool. It could be any kind of git based repository github, bitbucket, gitlab and many others.

A

So once the source code is checked into the git repository now, it's time for that source code to get compiled or get packaged if it is a language which doesn't require compilation into a container image, basically to be deployed on a kubernetes cluster. This process is also called as build process, so this build process. The end result is a container image which lives either in a container registry or it can live in any other storage location.

A

Once the container image is ready, this container image needs to get deployed and for this deployment, what is required is infrastructures code. So you would require to write choice, various different choices available for your infrastructures code. It could be hem, customize, yaml and many other different choices that you have once you've written your infrastructure's code. This code also needs to be living in your gate repositories, and once that happens, that's where your git ops tools come into the play.

A

The gitobs tools, the way they work is they are part of the kubernetes cluster and they actually sync your cluster state with the infrastructures code that you have provided in your git repositories.

A

Essentially, every infrastructure's code repository can be synced with your cluster state by using a github's tool. We're going to take a look into a github store today called as flux, it's a very good, open source project, and it allows you to sync your clusters with infrastructures code and we're going to see the demo for that as well. Let's take a quick look into how flux looks like so. This is of the flux project.

A

This project. um We can quickly see some of the instructions, of course, but let's take a look into it. A high level diagram how flux works, so flux can be deployed on a cluster within its own namespace, and then it can be asked to be syncing with any given registry or even any given uh repository located in in any kind of git base source code management system.

A

Once the developer commits into the repository code, like hem, charts, customize yaml files, those files are then synced with the with the state of the cluster by flux automatically.

A

So essentially, the developers or the devops teams do not necessarily have to uh specifically do cuddle apply on all these different infrastructures code files that are being checked and which are being maintained. This is also going to be the structure for our demo. Today, we're going to be using a gke cluster with flux installed on it and we're going to be syncing it with one of the repositories that we're going to be writing on gitlab, exciting.

A

So that's how your git, ops, pipelines and githubs methodology works. So how do we make sure that everything that is being synced into your clusters is secure, or at least has been validated from security point of view against certain policies? That's where policy as code comes in policy is code. What is that?

A

It's basically also a strategy as well as technology policies code, allows you to codify your security policies and store them as source code. You can version control them as well. It allows you to consistently do the enforcement of security policies throughout the life cycle from development pre-deployment deployment to operations.

A

It's typically used in automated workflows because it is a low effort enforcement and it also is a very good mechanism to do security testing on your github pipeline. Some examples. If you want to ensure your secrets, are not used in environment variables, you can include such kind of checks using policy as code within your githubs pipeline itself.

A

If you want to limit privileged containers from becoming super users or from super capabilities, you can achieve that goal as well writing policies uh and embedding them into your life cycle within githubs. So how do we achieve that? We're going to take a look into a framework called as opa open policy agent, which is going to be allowing us to do policy as code seamlessly and we're going to be using open source project called terascan which uses opa behind the scene?

A

To achieve this, terascan is a project from a qrx has got 500 built-in policies, it's open, extensible and it's based on opa. Let's take a quick look into terascan how it looks like.

A

So terascan basically comes with a lot of different features and capabilities, and it also comes with a lot of different integrations with various different tool chains: ci, cd pipelines as well as github's pipelines.

A

It helps you identify security misconfigurations as well as policy violations within various different types of infrastructures code, including, but not limited to terraform, hem, customize, yaml and few others with this architecture.

A

Terrascan also comes with the flexibility of being integrated in any kind of source code management tool, as well as any kind of pipeline, and we're going to take a look into that. So let's also go ahead and understand how this whole policy is code with opa can help us secure our gitoff's pipeline.

A

In order for us to implement policy as code for a given kubernetes and github space environment, we have opportunities available to do that at several different stages.

A

So we could do this at the time of local code development when the code is being developed, say using terascan cli.

A

We can also do it at the source code management tool layer when the code is actually being checked into your git repository or we can do it at the build layer when the build processes are being triggered or we can do it at the sinclair when the code is about to get synced and the cluster state is also about to get synced with the ise and that's where your github pipelines come in. So today we're going to take a look into that.

A

So, symbolically, you can see on the screen that terrascan can be used at various different stages. We're going to take a look into specifically at the stage 6 and 7 how to scan using opa can be used for enforcing policies on your guitars pipeline.

A

So in the demo we're going to see we're going to be using a gke cluster with flux installed on it, we're going to be seeing a pipeline built on git lab and we're going to see some infrastructures code to deploy some applications on the cluster using flux as well as gitlab ready, let's get into it.

A

So what you're seeing on the screen? It's basically a very you, know simple uh namespace file. It's it's a file which creates a namespace, uh a kind on a cluster called as demo flux2. This is what we had used to create a namespace on a gke cluster, and this is where we have installed flux.

A

So I'm going to show you a gk cluster with flux installed on it, and what we're going to do is we're going to be. We have already gone ahead and configured flux to sync, with the repository called sas integration, which is located at git lab so right now, what we're going to do is we are going to basically deploy a engine. Is application, so nginx is basically a web server. So this is our deployment file for it we're going to name the app as nginx and we're going to do two replicas for it.

A

What we will do is we will push this file into the main repository called sas integration on gitlab, and then we are going to trigger a sync from flux, we're going to do the trigger manually so that we can show in the demo how things work technically. This thing could happen automatically as well, but we're going to do the sync manually with between flux and the repository and then we'll see that the engine application is going to get deployed. So let's take a look into that. So this is our deployment.

A

A

And we are going to commit this deployment file into our repository and gitlab.

A

There we go, we pushed it and now we are on a cloud shell, we're going to do a flux, cuttle sink. Basically, we are going to be syncing it and you can notice here that flux is pre-integrated with the sas integration repository located on git lab and we are, we are just allowing the flux to sync with it.

A

We push the application deployment yaml file, which is our infrastructure's code and we've synced it, and now let's see the namespaces, so we can see, there's a demo flux to namespace, of course, which is where we have deployed our application as well.

A

Now we are going to quickly see the pods in that namespace.

A

Here we go and we see that the nginx is being deployed after the sync, so the developer just made changes into the deployment yaml file or just made changes into the ise file, and that change has now been synced with the cluster state.

A

That means that flux is a gitops mechanism tool which is automatically syncing the state with the isc. So that's why you're able to see engine x being deployed next step? What we're going to do is we're going to create another nginx deployment file and we're going to name it differently. This time, this deployment file, what you're seeing on the screen is called nginx deployment 2.0 and it is going to deploy another nginx application basically, but uh what the name will be 2 this time now why we are doing this?

A

The previous one, when we did the installation we did not perform any kind of security assessment before the cluster was synced this time around now we're going to do a security assessment before the isc is synced with the cluster state by flux.

A

Now you would see that we will be able to detect the security issues before the cluster. Sync happens, but we will be checking in the code again into the main repository, which is sas integration and so flux still will be able to sync up, even though there will be security issues in it. Let's take a look into that scenario and then we're going to implement the scenario how we can prevent flux from synchronizing ise, which has security problems.

A

So what you're seeing now is the gitlab ciaml file? This is basically the pipeline configuration for gitlab. What we have done is that, as the code gets checked in, we want gitlab to kick in a pipeline with the couple of stages in it. The first stage that we are calling here is a scan stage, we're going to be using terascan for that. You can see on the screen how we have configured terascan for performing the scan on the code that gets checked into gitlab.

A

In this particular case, we're going to be pulling down terascan image and then we're gonna be basically issuing a scan command into it. As soon as this eml file, which contains the configuration, gets checked into gitlab, we will see a new pipeline getting triggered. Let's take a look into those and, of course, what we are attempting it. We are also checking in the second deployment file into the main branch.

A

So technically we should be able to also see more applications running so currently, there's only one engines, application running there should be an nginx 2 application that that should also come up, but first, let's see how the pipeline gets triggered, so we're checking in the code we've checked in also the gitlab pipeline configuration file and we are committing it now.

A

We'll punch in our password the commit happens. We go to gitlab. Now we're going to be checking out. If there was a new pipeline created, you can see on the screen. The yaml file scan has been configured and if we go to to merge request but to the pipeline, we will see that a new pipeline would have got triggered.

A

There you go, the new pipeline is triggered and we will see at least one stage there, which is terra scan stage, and if you open it, it is actually failing, and the reason for that is we see that there are multiple issues being detected in the deployment files right, including the app armor profile not being set to default.

A

Let's take a look into terascan how these things work and how they're looking like. So if I go to terascan and if I go to kubernetes folder under the opa regular folders here under the kubernetes pod, I see app armorprofile.rego file.

A

So if I go to kubernetes here, if I go to pod, there are a lot of lego policies here. Gregor is a language that is used by opa for defining the policies. So this is a policy which checks, whether you have app armor enabled or not.

A

Now, terascan has gone ahead and also created templates for regos, so that you can use the same lego for multiple different policy types. If you have different parameters that you want to pass, what we're looking into is right now, a json file which is basically invoking the rego template here. This is the app armor profile, drago template, which has been invoked, which is going to be applied on a resource type called kubernetes pod, and this is going to check whether you have app armor profile enabled or not.

A

This is what is your policy is code now with this policy as code, what you are able to clearly identify is if such a situation is not mitigated or detected. You want to take certain actions on the guitars pipeline, so this is how policy is code internally works, let's go back and take a look further into our demo. How we can now prevent this from happening or how we can at least make sure that flux is not doing any kind of sync so currently, because we checked the code into the main repository.

A

Although the pipeline reported that there are security issues, the code is still present in the main repository, we will be able to see that the application is still deployed and we'll be able to see that in a bit. Let that command finish in the meanwhile, let's take a look into the next part of the demo. What we're going to do here is we are going to basically do a new feature branch.

A

We are going to check the code into a new feature branch and we will make sure that we are checking in a wonderful code and we're going to raise a merge request to merge this feature branch into the main branch, and we will prevent that from happening so that flux does not sync up with the wonderful code. So what we are doing right now is we are just creating a new feature branch and then we are going to add all this code into the feature branch.

A

So, while that is happening, the previous command would have, uh you know would have been finished, so we are going to quickly uh also make sure that we are able to see that nginx.

A

Second, application, which is nginx deployment, 2, is also synced, so here you can see that this is the previous run that we had. Since we were working on the main branch directly, we were not doing any kind of pull request based mechanism.

A

Therefore, we were not able to prevent this one dribble application package to reach through your cluster, but this was for demo purposes and now, let's do this in a pull request. Manner.

A

Okay, so we've created a new branch, we're gonna just push it now, keep in mind as soon as we're gonna push this new branch, it will also have the pipeline configuration in gitlab. A new scan still will be triggered on this new feature branch. You can see the feature branch has been created and we will see the results still being showing that there will be violations policy violations in this new code in the feature branch itself.

A

So, if I go to pipeline, we could see it's on the feature branch and it is saying it is failing, because terascan opa-based policy escort assessment was performed and there were issues that have been discovered here right. That's why the pipeline failed and now when we will try and create a merge request or a pull request against the main branch we will not be able to because of the review restrictions that we're gonna we have put in and therefore now, if we do the flux sync, we will not be able to see the fluxing happening.

A

So even if we create a new branch out here, we will not be able to so. You could see that the merge request would not work against the main branch, because there are issues here so basically, in this integration, what we have achieved is we've made sure that bad code does not reach to the sync branches or the sync repositories which are in connection with flux.

A

That's one way of making sure that you are only giving good code to flux and, of course, there could be other ways of integrations as well, where we can have flux itself, determine whether the scan has happened or not or maybe perform a scan. These are two different strategies both have their own pros and cons. So in this demo we saw today that how we can secure flux based github's pipeline using opa-based mechanisms, which our policy is code mechanism.

A

Thank you all for the time today and enjoy the rest of the conference appreciate.