Cloud Native Computing Foundation Cloud Native Security Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Integrating Security in the Build Pipeline - Anirban Saha, Allianz Direct

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Integrating Security in the Build Pipeline - Anirban Saha, Allianz Direct

With infrastructures and platforms going cloud native, there are greater security risks now than ever before. In the build and release workflow, often the focus on security comes at a very later stage and sometimes unfortunately, it does not happen at all due to lot of constraints. In this session, Anirban Saha will demonstrate security can be integrated in the build and deployment pipeline and address the problems early in the process. He will stress on the factors that can be problems in achieving this objective and how to go about solving them. He will also discuss the different vulnerability points that can and need to be considered when designing the solution. He will also demonstrate situations where deployment decisions can be made depending on the security status of the artifacts thus eliminating the need for post deployment measures to tackle security flaws.

A

Hey everyone. I hope you have been having a great experience at the cloud native security day, europe virtual event and I'm here to try and add to that experience with this session, integrating security in the bill pipeline.

A

Let me take a moment to introduce myself. My name is anirban and I'm a member of the devops engineering team at allianz direct based in munich islands direct is the online insurance platform of allianz operating in a few countries in europe.

A

I have been working with cloud native infrastructures for almost 10 years and tend to automate solutions to make others and my life easier. Also with so many security incidents happening in our daily lives, I sometimes tend to get a bit paranoid about security in my projects.

A

I should also mention here that all the information tools, strategies, opinions and demonstration presented in this session are entirely my own and they do not represent the views of alias direct or any other allianz entity. Although we do a lot of cool stuff at alias direct, the contents of this session are not really aligned with them.

A

This session ends with a live demonstration of a ci cd pipeline with security tests, enabled and involves a cloud native setup I'll explain the tools that have been used here during the session and for the attendees who are really interested in trying it out themselves. I have added the configuration files in a couple of github repositories.

A

The build pipeline repository also has general guidelines about the base components that we are using in the demo and the commands to set them up.

A

I understand that for some beginners it might be a bit much to grasp, but, unfortunately, because of some time constraints I'll not be able to demonstrate setting up the base components such as the kubernetes cluster, the auto scaler and the ingress controllers.

A

So we will first look at why we are here understand the problem analyze. It find out reasons for them and look at ways to mitigate them. We will look at how a vulnerability scanning workflow works, how they plug into the build pipeline, the options that we have in terms of tools, analyze different strategies to integrate security scans into the build pipeline.

A

And finally, we will watch one such pipeline in live action. I know that we all hate staring at powerpoint presentations. Believe me, I know I really know that pain, so I'm gonna finish up with the presentation as fast as possible and we'll dive into the fun stuff.

A

This is required, as I really want to keep this simple and really want you to understand what we are doing and why we are doing it. So why are we really here? What is the problem that we are trying to solve? In general terms, we are trying to implement software security.

A

So how do we do it generally? It starts with the business goals being defined, resulting in products to be launched, which happens at the management level.

A

The architect designs, the software as per requirements, developers get on the same page with the architect and they start creating some cool software with modern life cycle software life cycle processes, testing the software is very essential. We all we all know that once all of this is achieved, the artifact is created and the product is released generally. At this point, a security or penetration test is scheduled.

A

Where vulnerabilities are detected or sometimes, unfortunately, there's really no security test done at all, as the software has never been scanned before this list is quite long now think about a cloud native scenario where we are talking about more than 50 to 60 micro services and the amount of time it would require to fix those problems. That's huge right now with product priorities and feature requests. It is quite difficult to devote dedicated time for such fixes.

A

As a result, it gets postponed or greatly neglected altogether and before we know there is a breach taking place. So how do we approach the problem? You might have come across the phase shifting left on security.

A

This very very simply means that in the left to right process, progress of software development life cycle, we need to move the security testing towards the left that is early on in the development process.

A

The basic idea is to detect and fix security problems during development and not after the service is released. So how do we actually go about addressing the issue?

A

The final objective of any business is to create a trust relationship with the customers, the clients and and the registered users. To do this, you need to be able to convince them that when they provide you with the data, it is not going to be compromised under any circumstances.

A

At present, we are providing our financial insurance, medical and our personal data online to service providers every day. We would definitely not want to be associated with a service who are not able to protect our data.

A

To do this. First and foremost, thing to be done is to make each and every person responsible for security.

A

When you leave your office door open by mistake, you are responsible for the safety of hundreds of your colleagues in the office building, and you can compromise it by letting in an intruder. This person can be an office administrator. He can be an engineer, a manager or even the ceo of the company.

A

Similarly, in the software product lifecycle, each person needs to understand that they need to be responsible for security, whether it is via software design, code creation, software deployment or really site. Reliability developers really understand, features and feature requests very well right, so start treating security fixes as features and integrate them in the development process in large development teams.

A

Instead of asking and expecting everyone to fix their libraries use a pre-built library repository, which only contains pre-approved and vulnerability-free libraries to fix problems, you need to detect them first and know how to make it part of the process, so go for professional help in such cases, or really hire dedicated security personnel train the teams so that they first understand what they are doing and why they are doing it.

A

It's not only important to decide to shift left on security, but also to understand what to scan for vulnerabilities, are of different kinds, and there are different stages when they can be detected and fixed. The first step in this direction is to look for package or dependency vulnerabilities.

A

They include base operating system packages and also application packages such as ruby python, java, node or just any other programming language dependencies.

A

This is a stage where the code is not involved, yet it's just the system on which the code is supposed to run. Next, we need to scan for vulnerabilities in code which has not been compiled. Yet these are situations when problems in the code can lead to unintended code executions.

A

This is called static, application, security, testing or sast, because the code is not running yet. Another type of security scanning is the dasd or dynamic application security testing. In this stage, the application needs to be scanned for vulnerabilities, which exist when the code has been executed.

A

This need needs to be done against a running web application. So we can really see that the attack vector is quite huge.

A

Next, we will look at general vulnerability scanning workflow. The ci cd tool is probably one of the most important component in the setup which orchestrates the entire pipeline.

A

The code repository is used as a source for our code and the image repository for image. Artifact storage. The security tool is really not mandatory. In most cases, there is a central solution, security platform or tool which coordinates the communication between the build pipeline and the vulnerability scanners.

A

It is also used as a visibility tool to visualize vulnerability reports. Believe me, when you look at the command line and see all those vulnerabilities, it's not really that helpful. A dashboard can really help you here.

A

Pipelines can also direct commun directly communicate with the scanners. The vulnerability scanners are the databases against which we compare our image packages. If all goes well, we can deploy our artifact on the container orchestration platform for our demo. We will use a similar setup where we will use github as the code repository techton and argo cd as the cicd tool harbour as a security tool and the image repository and kubernetes as a container orchestration platform.

A

There are few open source vulnerability scanners available, such as clear and trivi, which are really really powerful tools: the default deployment of harbor ships with trivi as the default scanner.

A

We will fetch our application code and dockerfile from the app repository, build it and push it to a test image repository. We will then scan the image from the test repository for vulnerabilities.

A

If there are none to be found, then the image is promoted from the test to a prod repository. Finally, we will deploy the app from the prod repository.

A

We definitely wouldn't want vulnerable images to land up in plot repositories right if the image has vulnerabilities the pipeline will stop at that stage and we can visualize the vulnerabilities in the harvard dashboard for our demo. Our app is a simple nginx container built from a debian or ubuntu based image.

A

Nginx is installed on this image and a test web page is added so really really simple stuff. The debian image has high and critical vulnerabilities, which we don't want to be in fraud, whereas the ubuntu image only has medium and lower vulnerabilities, which we definitely want deployed that that will actually be our simulation.

A

We will use techton to create a build pipeline, an argo cd to run the pipeline and deploy it on kubernetes. Argo cd is a cloud native deployment tool based on the gitops methodology.

A

There are some two specific terminologies used in the session, but if you feel intimidated, please feel free to use any other ci tool of your choice. It is also a cncf incubating project, however, is also a cncf graduated project, which is an image registry and helm repository.

A

It can also add plugable vulnerability scanners to be able to scan the uploaded images, although we will demonstrate an automated bill pipeline in this demo. This is quite far from a production, build setup, advanced features such as triggering pipelines via events and passing parameters are not part of this demo.

A

I have kept the app repo with helm chart and the pipeline repo different to help. You understand this better. However, they can be modified to create a more sophisticated workflow. Okay, so we will now start with our demo. This is the hardware dashboard that we have already installed in our kubernetes cluster.

A

We have created two different projects, one for broad and one for test. Each harbour project acts as a docker image registry and also a help chart repository. We have already gone to the configuration page for both projects and enabled the option to automatically scan images on push.

A

If we go to integration services, we can see that trivi is already installed as the default scanner. If we want, we can add a new scanner like claire. We have already set up claire in the harbour namespace in kubernetes.

A

If we check the parts we can see, we have a clear part running and a clear postgres board, which is a database where it stores the vulnerabilities database.

A

We also have a clear scanner adapter, which helps clear to communicate with hardware. If we add a new scanner, we name it claire.

A

We put in the address, which is the service for glare and the port.

A

There you can see we have added glare as a secondary vulnerability scanner for our demo. We will still use trivi for our demo. We have created couple of help, charts which will help us run our pipeline.

A

The first term chart is called app deploy, which is actually a simple help chart to deploy an application and create an ingress for it to expose it to the internet. The second helm chart is called tecton build pipeline, which contains all the configurations to run a tecton build pipeline.

A

We have already built both of these hunt charts and we have already uploaded them to the health charts repository in the broad project in harbour.

A

If we check the build pipeline repository, we will find a directory called secrets which are some kubernetes secret manifests which are to be applied to the combinator's namespaces. Before our demo can be run. They are just basic authentication credentials as some and some docker registry credentials.

A

Next, if we check our health charts, we will quickly go to the components that we have added here. We have added some pipeline resources which have our git repository and the image registry.

A

We have added some config maps, which has the metadata for our build pipeline like docker tag, the image name, the namespaces and the argo cd server and the argosy application name.

A

We also have a series of tasks that we are going to run in our pipeline, which we have already discussed, which is the docker build task. The security scan task, the image, promotion task and finally, the rocd up sync task.

A

We also have a service account which is attached to one of the docker credentials, which helps us to provide authentication for our pipeline.

A

We have a pipeline definition which integrates all the tasks that we have already created and finally, we have a pipeline run definition which actually runs our pipeline using tecton. Our second repository is called example, app, which has the configurations for the application that we are going to deploy.

A

The first directory is called app and it contains the docker file and the sample web page that we are going to deploy. We are going to start with the debian image, which is supposed to have the high and critical vulnerabilities and will be breaking our pipeline.

A

The second directory is called hell map and it has the helm chart specification to deploy the application and it uses the app deployment chart as a dependency.

A

The third directory is called helm pipeline and it uses the tecton build pipeline, helm, chart as a dependency and will be used to run a tecton pipeline in the build pipeline repository there's another directory called argo cd resources, which has some manifests for our argo cd application.

A

The first one is used to add the hardware registry information into argo cd, so that it can fetch help charts the example app manifest is used to create the applique, the argo cd application to deploy to actually deploy the application, and the third manifest is used to create the argo city application which will run our pill pipeline.

A

We will now go to the command line, go to the argo city, resources directory and we will apply each of this.

A

Manifests if we now go to the argo cd dashboard, we will see both our applications have been created, and if we go to the repositories page, we will see that our hardware, helmet repository, has automatically been added and the connection is successful. The example lab build pipeline gets its configuration from the helm pipeline directory of the example app repository.

A

The values file of the sun chart has the corresponding values which helps the pipeline determine its status.

A

If we go to the argo cd application, we will see that all of these resources are ready to be created and it creates a dynamic name for the pipeline run. With help of the tag that we have added in the values file in a more sophisticated workflow, we will not be adding this tag manually, but it will be created when our automated build pipeline works via triggers and events.

A

So, finally, it's time for us to run our pipeline and see the output. We will run the sync option and we will select the prone parameter here and we will run synchronize.

A

This will take some time, so I will skip the video here.

A

If we now check our pipeline, we will see that it has stopped at the security scan step and we will see the reason why, because it has found high and critical vulnerabilities.

A

So, in the meantime we have gone ahead and we have edited a docker file to change the image from debian to ubuntu, and we have also bumped the tag for the image.

A

If we go to argo cd, we will see that the change has automatically been detected and the zero zero one pipeline run is to be removed and a new pipeline run with 002 will be started when we synchronize the pipeline.

A

So we will go ahead and run synchronize again and we will select the prune option so that the old pipeline's old tasks are removed and the new ones are created.

A

If we now check our pipeline, we will see that it has completed successfully because the security scan step was not able to find high or critical vulnerabilities.

A

If we now quickly check our hardware dashboard and check the test project, we will find our example have image here and if we go into details we will see each of our tags with its corresponding vulnerabilities.

A

Here we can see tag, 001 has critical vulnerabilities and 0 0 2 has medium and lower ones, which is why it allowed us to perform our deployment.

A

If we go further, we can actually get details about all the vulnerabilities that were found if you're curious as to what we did for the security scan step. Just go to the tasks manifest file in the tecton.

A

Build pipeline head chart go to the app docker security scan task and there you will be able to see that we used the trivi docker image and we ran the trivi command to scan our image from the test repository and we checked for high and critical vulnerabilities, and we asked the command to fail if it did find such vulnerabilities.

A

So with that, we have reached the end of this session. I hope the session was able to help you get some understanding of how we can have better and secure, build pipelines and optimize the software development process. I have added some links about this topic, so be sure to check them out. I hope to see you in another awesome event in the future. Thank you.