Cloud Native Computing Foundation GitOpsCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing GitOps - Andrew Block & Shoubhik Bose, Red Hat

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Securing GitOps - Andrew Block & Shoubhik Bose, Red Hat

Implementing a GitOps based strategy brings a wealth of benefits including automation and conformance, each of which increases the overall security posture of any software and infrastructure delivery lifecycle. But, how do you ensure that you have fully taken advantage of all of the available options to provide the most secure GitOps environment? In this session, attendees will learn the role security plays in a GitOps environment and the areas that can be configured to properly enforce appropriate security controls common to organizations big and small including:

* Git repositories
* Static code analysis of manifests
* Ensuring appropriate rights in a Kubernetes environment
* Taking advantage of GitOps tools configurations
* Operating in a multitenant environment

A

Hey everyone, uh thanks for the opportunity to present to you today a very important topic really fast. There's a lightning talk, 10 minutes. We want to go hit the key points and then get right onto things. So my name is andrew block, I'm a distinguished architect with red hat. My co-presenter couldn't be here: shubik bose. He is obviously very much.

A

You know in spirit, as they say with us here in the room as well as virtually so uh any questions that you have and virtually please go ahead and put it into the questionnaire chat so that you know he might be able to answer too. So a very important topic in get ops is security.

A

Why is security important? Because traditional models of git, ops or just traditional models of managing infrastructure is typically managed via people? And for that standpoint you want. You typically focus on physical security. You want to keep people out of buildings out of systems when you move over towards a get ups model.

A

You have to shift your thinking to where it's more autonomous and changes just occur through the systems and there's a lot of things that you need to think about that you might not have thought about before everything from and we're going to talk about those all these different principles: everything from physical security to securing your git ops engine to repository security, things that you just had not thought about before you now need to think about. So, first of all, we're not talking about anything new. These principles are on security.

A

Are things that you probably should be taking advantage of automatically in your current day, whether you're using get ops or not everything from understanding where source the source of truth? For your content, making sure if you're using a git ops engine, you know that's in the container making sure you're getting container images from certified locations, you're going ahead and scanning for vulnerabilities inside these containers, you want to make sure that you enforce regulatory compliance.

A

You also want to be able to monitor your systems so that things don't go bump in the night. Things that you should have already been doing. We want to make sure we continue to do them in a get ops world.

A

First, let's go ahead and, let's start with, we want to secure the get ops engine. This is going to be an argo cd, a flux. If you're, not even using containers and kubernetes, you can have an automation tool like ansible or terraform. We want to make sure that you secure your get ops engine. You want to make sure that you have proper authentication and authorization on them, so only certain individuals can access your gitops engine.

A

You want to make sure that you regulate the source of content, where your, where your, where is your source of content for your git repositories, are you you're pulling from just some random repository on github or you going from a certified source?

A

You also want to make sure that if you have multi-tenants in your get ops environment, so in a kubernetes environment, it's multi-tenant by default, you can have multiple get ops engines in a single cluster, making sure that you limit the amount of access that you give your get ops engine or if you happen to be leveraging a single git, ops engine. You can have multiple teams inside that get ops engine ensuring that they they are going ahead and limiting the types of access that each one is given.

A

That's according to get ops engine next is with git ops, you're, going to typically go ahead and leverage a repository of some sort, a githouse repository, that's on a server. It could be hosted on a public sas solution like github git lab, but you most likely in an enterprise organization is going to be leveraging one, that's internal! You want to go ahead and embrace common best practices for managing infrastructure. You want to make sure it's highly available. You want to make sure it's monitoring and alerting.

A

So if things go bump in the night, you won't go ahead and be surprised about it and very important. Make backups make backups really regularly access control, making sure that nobody has access to these infrastructures that shouldn't and if you are leveraging git, make sure to only enable the protocols you're going to leverage if you're going to use http you're going to use ssh disable the ones you aren't going to use because they could involve some backdoors or some vulnerabilities.

A

Next we'll go into securing the get out repository, making sure that only in that certain individuals have access to your githubs repository. So if you have a ci cd system, you will make sure that you have separate access controls for that system or your git ops tools or any individual. You want to make sure you limit the amount of access that is given to those tools and then inside the repository itself. Enforcing branch protection, since your git is your source of truth for your entire infrastructure.

A

You don't want to have some individual accidentally pushed to master when they wanted to potentially go to. You know a non-prod environment because that could then get rolled out and affect your entire system. You want to utilize a typical contribution, workflow like like the fork and pull model from github or something along those lines and then finally perform content. Verification of anything you are introducing into your environment.

A

We're going to talk about that in a future slide, restrict the get ops engine restrict exactly what the tool can do in a kubernetes environment if you're going ahead and just managing certain components in a certain name, space don't give an entire cluster admin access, because someone could accidentally push changes that affect the entire cluster.

A

Make sure you lock things down utilize, a non user account a dedicated service account for your get ops engine, as well as, as I mentioned, employ the least print, at least the least policy of least principle pardon me about that, ensuring that only it has access to what it needs to do and make sure that you are able to audit the amount of permissions that you are given to your get ops engine.

A

Finally, secure your get off content. The content is the brains behind your entire operations. If you give it bad configuration, it's going to spit out bad configuration. So if you in a kubernetes environment, give it bad yaml.

A

Well, if you had good yammer when you first started and you gave it bad gamble, that's going to be downtime potentially, depending on how you have your rolling updates, et cetera and the kubernetes environment, but make sure you enforce good content in your git ops, environment, so utilize code, linting, schema validation, so in kubernetes you can validate different schemas.

A

So if your deployment has a field that it doesn't recognize, it'll fail validation, integrate these type of processes into your typical git ops, workflow, your ci cd process, as well as leveraging policy enforcement engines like oppa kyverno.

A

You can go ahead and enforce state and compliance before you even get anywhere near to merging to master or your target branch for your individual deployment. And finally, the last thing, which is a bane of everyone's distance, is how the heck am I going to manage my secrets. Every environment is going to have sensitive data. How do I manage that in a get ops world? Well, number one never store ever store secrets in plain text, plain and simple: we're not plain and simple um number two kubrick's kubernetes secrets themselves are not encrypted.

A

If you didn't know that that's a big big thing, it's base64 encoded anyone with any tool on their on their laptop, we'll, probably be able to go ahead and decode that utilize an encryption strategy. If you're going to store content inside your git repository or just don't even store it inside get use a secrets, management tool like a hashtag or vault a cyber. You know cyberark et cetera, or you know you know aws or azure, depending on what cloud provider you're using utilize secret storage.

A

Is there and then reference that inside your applications, certain plugins are available to help with your get ops tool like argo cd can integrate with with vault and flux, has similar tools as well and then a lot and then once again, utilize code scanning tools to be able to be detected and notified if it detects any type of content that might be a password or anything around those lines.

A

And finally, the last thing is security is always continuous. What may be good today isn't going to be good tomorrow, make sure you go ahead and design appropriately implement it when you are beginning always start with security. First, don't go ahead and bolt it on after the covers, because at that point you're already too late in your process, you're gonna have to go ahead and rework a lot of things, make sure you enforce it and then monitor and then adapt accordingly be able to refer to it, adapt and be very agile.

A

These are a lot of different things and a lot of different topics that I went through in this quick lightning talk. I hope that I was able to provide some thoughts, some insights in how you can enforce proper security inside your git ops workflow, I will be around the entire conference- feel free to go ahead and come up to me also for those of you in the virtual audience, uh feel free to hit me up on my social media accounts. No twitter, facebook linkedin happy to go ahead and have a conversation there.

A

Thank you for the time today. Thank you.