Cloud Native Computing Foundation Kubernetes Forum Sydney 2019, 12 Dec 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Adventures in Production with GitOps, Secure Pipelines & Compliance - Brandon Lum & Ricardo Aravena

Description

Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Adventures in Production with GitOps, Secure Pipelines and Compliance - Brandon Lum, IBM & Ricardo Aravena, Rakuten

In the last two years, Kubernetes GitOps has become more common in many organizations helping them enhance their software CI/CD by removing manual commands and keeping release versions consistent. However, there are still some gaps when it comes to security and compliance technologies.

We will go over some of the most popular open-source tools for GitOps, container images scanning, encryption and signing tools and how they work together. Among them, Flux, Scaffold, Ignite, Aqua scanner, in-toto, and Grafeas. In addition, we will talk about incorporating compliance into DevSecOps pipelines and explore the importance of application specifications such as CNAB.

In the end, the audience will understand how to create a container software pipeline in a fully automated, encrypted and secure way with Kubernetes in production environments, with compliance built-in.

A

Good afternoon, everyone thank you for coming to this session, we're very very excited to be here at the kubernetes forum to tell you about get-ups, kubernetes security and pipelines with compliance a little bit about ourselves, work Rakuten as a necessary manager.

A

My name is Ricardo and we run a lot of kubernetes clusters in different kinds of environments, production, staging and QA.

B

Research and I look at containers, security and basically creating facilities to help provide security and the content of that problem.

A

Some of what we'll be talking about today, first we'll talk about why you want to have something like get-ups, then we'll talk about why security is important in the get-ups context, then we'll talk about some tools in terms of how to put it all together with CI, CD and security. Then we'll give you a brief demo on a secure pipeline in tacked-on with mini queue.

A

Then we'll talk about what it may look like in production having a full-blown pipeline with depth of sack ups and compliance, then we'll talk about what it may look like in the future and finally, we'll give you some takeaways. So why do people want to have something like get-ups?

A

So we came up with these three reasons, but other people may have others, but basically people want to have a tool that allows them to compare a current state with a desired state, and not only that, but have that tool automatically changed from that current state to that desired state, and then they also want to attract when somebody changed something when or what got changed and who changed it.

A

So, for that get is a great tool and they also want to be able to automate deployments to different environments, so have that transparency across a team of an organization so everybody's on the same page. So this is kind of like what a get-ups infrastructure looks like at the bottom on the left. You have cluster operators and developers and then typically, the developers commit code and then that code kicks off our CI CD process and that actually creates a container image and gets posted to container registry.

A

And then you may have cluster operators managing kubernetes, manifest files in basically changing them and committing those to get repositories and on the right. You have the kubernetes clusters watching all those different changes, changing a container version and automatically doing a deployment to one of these clusters like development production or staging, and then you have tools like cubes, ETL or other tools that allow these folks connect to the clusters to see what's going on. So why do you care about security in all this context?

A

So if you look at the previous diagram, you see that there's so many different endpoints. When you talk about get-ups connecting to your kubernetes clusters, you have your get repository where your developers are committing code to say connecting through HTTP or HTTPS, and then you have also your get off stools: syncing, a container image versions to your kubernetes clusters, so many different parts where an attacker can come in and cause some damage and there's also vulnerabilities. So you have the package dependencies, there's container backdoors.

A

Then you have typosquatting where you may download something from a fake repository. Then there's also the kubernetes vulnerabilities that allows you to do some sort of escalation with kubernetes, where maybe people can do things on your deployments and then also get up stools specific to these tools that we'll talk about in a little bit.

A

People also care about compliance. So there are many different compliances around the world. So a few examples are gdpr if PCI FedRAMP, so many different types of compliances and that's why they care about security. So now, let's talk about some of the tools for get-ups and security.

A

So first we have the developer tools. We have draft careful and flux, and these tools allow developers to create that workflow from the workstation to a deployment in kubernetes draft ins. Careful are very similar. They have their own commands in flux, relies on some other commands like get cube CTO, and they also have their own command to manage deployments remotely called flexi deal.

B

So that's first softball Roberti scanning, which is one of the most common. So there are two types of vulnerability scanning. Really you. What this is about is finding known security issues with our applications.

B

So the first is code scanning and what code scanning does is it looks at the libraries have been used by your applications as well, as sometimes logic like or wasps, and then you have image scanning, which is you know, looking at the underlying operating system and the different packages that are being used by application, so called scanning tools are usually language, specific, so for go along. We have go sack and image scanning tools. Examples are Claire and trivy.

B

Another thing we can do in pipelines best practices, so examples of this is for dr. Faustus Harland, which does best practices, including security for your doctor files, and also some benchmarks released by C, is the Center for Internet Security, and they do things like here. Are the best practices for kubernetes deployments all right? Next up we have image signing. So what does image signing actually give you? So the first thing is the integrity of the image we want to ensure that the image when we are running it in kubernetes has not been tampered with.

B

So this is the first thing that it provides. The second thing it does is: it also gives you Providence of the image, and what this means is that you can range from knowing that the image was created by a particular user. All the way to something like this image has gone through a certain set of compliance checks so, depending on how you set it up, you get different amounts of provenance.

B

So this examples of this is topical content, trust that goes along with notary as well as we have image signing.

B

So we just talked about integrity. How about confidentiality, so image encryption can be used to ensure confidentiality of our images. So this is good if we work with proprietary code or secret on trade secrets and secret algorithms, and things like that. So what this provides is that, if, in the event that your registry or docker hub or something is compromised, the contents of the image will still not be accessible.

B

So this is implemented in container D as well as copy or in the Red Hat stack. So you can start playing around with this, and another thing to not forget is secret management. So every time we have an application, application usually requires several secrets and all access to different services, so in these credentials to access those services, so how secret management is usually done is in two parts.

B

One is the process of creating the secret which we define what this application is successful, while the secrets that we want to create for it and the second part of this, which is as important, is defining our identity for the application. So this really means that we want to ensure that the secrets that we are creating in the pipeline provision to the correct identities, our application, so we ensure that the correct secrets go to the correct workloads, so example: identity in this case could be spiffy and then examples is inspired.

B

That does attestation of the node itself, so examples of secret management, schemas, seen secrets and many more.

A

So another interesting tool, part of the cloud native foundation, it's in toto and basically this tool allows you to verify every single step of your pipeline. So it's say you have a bill step. You have a test step, you have a deploy step. You have a bill, artifact step, all these different different steps. You want to verify.

A

So it allows you to do that with basic asymmetric cryptography in private and public key encryption, and it can be added to any CI CD to which we'll talk about in a little bit and then it makes it possible by creating these links between different steps, meaning if you have a deploy step and then you have or for example, you have a QA step and then you have a deploy step. But if the QA step somebody tampers with that, then it wouldn't actually continue.

A

So now, let's talk about packaging, so there are a couple of tools that are. You know some talks before this one. One of them is scene app and it's basically a way to distribute. You distribute your applications. It has its own command line. You can add security using notary by signing the metadata and then it you can provide that provenance. Meaning know what that package comes from using a tool like in total.

A

They'll pack is another tool and it's basically converting source code directly to OCI container images and it supports multiple languages can be used with this to open-source tool from VMware called Capac and for security. You can add encryption because it creates or CI images. So you can take advantage of that.

B

How do we build a pipeline? So what we've done is we have example here and we've broken down down into four four stages. So, let's start on the Left, which is the bill so in the bill, what we're doing is we're taking the application code and we are doing anything, that's code specific. So this is where we do then thing.

B

This is where we run static analysis, anything that language specific and what we get at the end of this process is a binary right and then we take this binary and we want to continue to rise it. So, in the containerization phase we do dr. bill and at the same time we do things I image scanning image, encryption and also image signing, so the output of that is then this docker image that we have that's encrypted inside, which is great, but an application, is more than just docker image right.

B

We need to define how this application will run. You know the deployment yeah Milles. We need to define what secrets the application only we need to define what is the identity of the application, so this set in the context of how we're going to deploy this application and at the end of it we get to the final phase, which is packaging, so this is Tina helm bill packs that we just talked about.

B

So something to know about this is that the first is that all the ring in this DevOps pipeline actually matters so example of this is goldang, so in golang, if you build a binary, what happens is if it's built statically a lot of the metadata about libraries get stripped away, so, if you're running scanning, if you don't run scanning before your build step, you're actually missing out on certain metadata that you may not find vulnerabilities in an example of this is encryption right.

B

If you're encrypting something before scanning it, the scanner will not be able to work on the encrypted data. Another thing to note here is that security is not something that you can add on after the fact. So over here, we've highlighted all the security and compliance related tools in red, and you can see that it's basically literature out the entire pipeline. So this is something that is integral to the entire process and has to be there from the start.

B

So this is some of these control mappings on songs on the technology we just talked about, so this can be helpful in getting compliance, because a lot of these compliance standards are based on these controls.

A

Now, let's talk about the C ICD tools that you can use to put it all together. So first we have Tecton. So what is Tecton tech tongs, basically Native kubernetes pipelines and it's basically have to burn a DC or DS and there they have multiple CR days, and so you have pipelines, you have tasks, you have resources and resources are specific to CIC D workloads.

A

So you can create images in the resources or git repositories, and then you can have multiple pipelines in these pipelines, so they have multiple tasks and the way it works is similar to kubernetes jobs in a way that when you start a job, it goes to into running state and once it's done it goes to into a completed state. And then, if your pipeline fails, then it goes into an error. State.

A

And if you want to add that extra security, you can use existing kubernetes mechanisms like part security policies, network policies which is a kubernetes firewall and to pull your repositories securely, you can use that get SSL support, so Jenkins X is another complex tool, so it's based on the original Jenkins, but it has a lot of other components. For example, has a chart repository? It has a way to discover those helm charts. It uses some of the other tools that I talked about earlier, less careful and draft.

A

It uses draft to manage, manage the source code and uses scaffold to Bill and push the container images. And if you want to add that extra security you can use auto upgrades we'll get up. Some tacked-on and Jenkins has its own built-in security too, with a lot of plugins. So you can use that as well. For example, integration with LDAP and different types of authorization and managing the users, for example.

A

Another tool is spinnaker is very similar to Jenkins sax I'm, not going to talk about all the details, but basically it also allows you to create a full workflow of your pipelines and interact with cloud providers and define all the steps demo time.

B

So what we're going to do now is we're gonna. Take a look at how to do pipeline is tacked on. So what we're going to look at today is we are going to start with this really really simple: flask Python application and we're going to show you an example of Tecton on mini queue, so over here I can see.

B

Let me make it a little bit bigger, so I have a meaningful cluster here and we're gonna see that we have Tecton running so the first thing we're gonna look at is really what is the inputs and the outputs of my pipeline right? So this intact on is defined by resources.

B

So for the disraeli say, for example, the input of this pipeline is to get repo'd. It I want to build right. So here you can see. This is a very simple pointing to the github URL and the output for this pipeline is gonna, be an image built, and we want this image to be in a local registry. We have in the cluster, so in this case you're gonna create and clip the image.

B

So the tag that's going to be encrypted and finally, we take a look at the actual pipeline itself and so for this demo, we've put everything within a single toss in tacked on so we can see over here. What we do is after clone. The repository the first set we run is Python lip safety, which is a code scanner for Python, and then we do a doc of al dente' using jato event.

B

We use chemical to build the image and we go ahead and we use aqua sex trivy to do an image scan and after that we do an encrypt step and finally, a push that so I've created all the CMOS ready and I'm gonna do a toss to create a tossed around object which will kick off the pipeline.

B

So I can watch this toss description over here and we can see it's pending, which will change to running in a couple of seconds.

B

In a couple seconds, no I think Internet's a bit slow and there we go so the next thing we can do is we can look at the locks of the pipeline here. So let me screw up here, make it a bit smaller. So the first thing that the pipeline did, we can see the lock here is the good sauce Python app. So this is actually just closing the gate, repository and then RAM Python safety, fortunately no vulnerabilities found, and then we hit the talk of event. The bill states, which is majority of this pipeline.

B

You can see it's really just a basic Python application over here and then we're hoping to dr. archive, so it can be scanned by a trophy and we run trivia over here, it's using Alpine, which has no issues with the datas, and then we do the encrypts up. And finally, we push the image alright. So now that this has successfully run what we're gonna do now is we are going to try and download this image to take a look to make sure that so I'm gonna just copy the output image here. I.

B

Have this part that should have the tools to download it.

B

So I'm going to use Scorpio, which is to to manipulate images so I'm, going to say copy.

B

So I'm gonna get it from the repository and I'm gonna, say my image over here. So it looks like it's successfully push. We are downloading it, and now we encrypted the image, so let's make sure that it's actually encrypted.

B

So the first thing that we can do is look for the layer descriptor plus encrypted in my image, and you can see that encrypted is actually part of the image under the immediate idea, but let's be extra sure, just to make sure that it doesn't just say it's encrypted and it actually is so we can go into the image itself. We can go into the blobs and one of this big ones is a data block. So let's take the biggest one and what we're going to do is run.

B

We want to see how random it is right. So if something to encrypt it, it should be pretty random. So we're going to use this to call and Twitter's entropy. So this really detects the entropy of the data, and we can see over here that it says that if we were to do optimum compression of this, it will be compressed by 0%. That means it would be pretty random.

B

So that was the the demo for the pipeline. All these, the the application itself, as well as the pipeline, is available on github. So if you want download play with it, there's instructions on how to set it up on the Mini Cooper cluster.

A

Ok, so what does it look like or may look like in production? So so this is out of the Jenkins X documentation, and you see here on the on the left.

A

You have a developer, committing something to a repository and creating a pull request in that pull request, starting a serial series of steps, including build test package and deploy to a preview environment and in that preview environment you can see a basic functionality of that software and you can have other people in the team review it and if that's successful, basically the PR is merged and another set of processes.

A

Just like the first one gets created and once that's complete, it gets deployed to a staging environment, and then you have other people in your org doing other types of tests. Qa tests, you could have also maybe selenium tests and once that's complete, you can say you know automatically or have somebody push that button and deploy to a production environment, and that could be something like a canary deployment if you want to so having the full pipeline with compliance. Would look something like this.

A

So, as you can see on the first step, you will have something like in todo to verify every single one of these steps. You will have something like trivy to scan your container images. You have something like the open policy agent before you deploy to you to Banaras clusters to make sure you're not running as a root user or you're running with the right permissions. You could have something like spire verify your workloads in kubernetes and your infrastructure, and then on this other step.

A

You could also have the same process in todo with trivia, and then, when you push your cut your to your container registry, you could have something like hardware, another cloud native foundation project and then to add the extra security you can have also container image scan in there. So you have that security and to an all across throughout your pipelines. So what does it look like in terms of the future for some of these configurations and pipelines?

A

So maybe we'll see some new configuration languages, so the word is that llamo is great for smaller type of files, but didn't. If you have this huge, yellow files, it becomes really cumbersome to manage and then becomes really repetitive. So some new languages are starting to show up in open source. A couple of these RQ Lang and Dao, and these languages allow you to embed some code in the configuration to automate some of those repetitive tasks and then at the last cute con.

A

It was announced that the pot security policies will be replaced eventually by the open policy agent and there's an implementation for there include Benares and that's called gatekeeper. It's an open source project and it'll help. You keep some of those configurations, more uniform, an angel cluster, and if you have multiple clusters across your clusters and then many will see more get ups for cluster management. So one of the problems when you upgrade to Banaras or the kubernetes control plane is that your api is constantly keep changing.

A

So if you do an in-place upgrade, you run into the risk of your workloads, not starting the next time, because your definitions are not compatible. So a better approach is maybe to have blue and green deployments or kubernetes clusters, and then you can use something like ghettos for that and then we'll see more integrations and integrations with all these different tools. So, finally, what can you get out of all of this so so think about security?

A

First, a lot of people build these pipelines and they become just really big set of pipelines and software, and then, by the time you want to come back and add that security you have to got it all out and start from the beginning. So if you think from about security from the beginning, then you don't have to you know you just you're, not doubling the work impact.

A

Constantly all your components run see you get ups tools, you know all the different protocols, so whatever you're using in your get ups infrastructure and and then, if you want to add that extra security you care about compliance, think about encrypting, your container images. You may have valuable data. There use a privileged mode to unprivileged mode to build your container images mean and don't allow access to just last product directory use route less.

A

If you, if you can with dock it and build, there's a couple of tools that support that and to put it all together, you can use some of these tools like Jenkins, X, vinegar and tacked on that I talked about so yeah. That's all we have now here are some resources.

A

Thank you very much and then we'll be around. If you have any question so feel free to come and talk to us, if you for anything.

A