Cloud Native Computing Foundation GitOpsCon EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Exploring HashiCorp Vault and ArgoCD - the GitOps Way - Tracy P Holmes, Codefresh

Description

Lightning Talk: Exploring HashiCorp Vault and ArgoCD - the GitOps Way - Tracy P Holmes, Codefresh

A big topic in GitOps that isn't really discussed enough is proper GitOps secrets management and just how serious of an issue it can be in a cloud-native aspect. While normalized usage of Bitnami Sealed Secrets is typically explored, but not much is seen around using HashiCorp Vault. Especially as it pertains to using Vault with Argo Project's continuous delivery tool Argo CD. What exactly is HashiCorp Vault? HashiCorp Vault is a secrets management tool created primarily to control access to sensitive credentials in a low-trust environment. It can be used to manage secrets, encryption as a service, and privileged access. This talk will explore integrating HashiCorp Vault with Argo CD, any pros and cons, and (hopefully) what ended up working for Tracy and her sensitive credentials. Note: A shorter version of this talk was presented at HashiTalks 2022. However, the intent is to go more in-depth with exploration and findings while attempting the integration.

A

Hi and welcome to exploring history, court vault and argo cd, the githubs way, I'm tracy p holmes and I'm a developer advocate at code fresh. So let's get started first off if you're here, you probably already know about get ops, but you may not really know about vault I'll get back to vault layer. That said, my title mentioned: git ops and argo cd. So what exactly are these things?

A

Well, if you aren't familiar with get ops head over to opengetops.dev, which is the official page of the gitops working group in short, get ops is a set of standards and best practices that can be applied to application, configuration and delivery all the way to infrastructure management. The principles of git ops are as follows: the system is described in a declarative manner, so in practice this typically means kubernetes manifests. The definition of the system is version and audited.

A

In practice it's typically stored in git a software agent automatically pulls the get state and matches the platform state. In practice, this would mean argo city or flux, and the state is continuously reconciled. That means that any changes happening in git should also be reflected in the system. Now you will usually see get ops paired with kubernetes, though kubernetes is not necessarily required.

A

Get ops can absolutely integrate with other deployment pipelines or infrastructure, including containers and vms.

A

Additionally, some benefits of get ops are deploying faster and more often easier and quicker error handling and recovery and self-documented deployments, and, most importantly, you also get the complete elimination of configuration drift. These benefits can make it easier to handle the applications and allow teams to deliver quality software faster now, a githop's tool that follows this approach of a git based workflow is argo cd. It's a continuous delivery tool for kubernetes. That is essentially a git ops controller that does two-way synchronization.

A

The argo cd deployment process is the embodiment of the core ideas behind git ups that we discussed at the beginning. Remember argo dude. On my shirt.

A

All application configuration is stored in git, so that's usually in a separate repository. Then the source code deployments are happening in a pull manner where the cluster is fetching manifest from git. So that's. Instead of traditional solutions where updates are pushed to the cluster, a deployment is a unidirectional reconciliation process between the two states, so what's described in git versus what is deployed in the cluster and even though the sync process is vital for performing the initial deployment of the application.

A

One of the true strengths of argo cd is the continuous monitoring of both states clustering it after the deployment takes place. This continuous monitoring is essential for solving configuration drift, a common issue in organizations with a considerable number of deployment targets.

A

Adopting the get apps methodology provides transparency between the application configuration deployed in the cluster and the one residing in git.

A

All right, so, let's talk about secrets: why are secrets important and can I use them with git ops? Well, a secret is anything you want to tightly control access to such as api, encryption, keys, passwords or certificates and well secrets are important because you want to keep your secrets well secret and, yes, you can use them with get ops, but the truth is there is no single accepted practice on how secrets are managed with get ops. We can all agree.

A

You probably don't want to keep any secrets and get where they're left in the open to anyone who has access in your repo- and you definitely don't want any unencrypted secrets in there now. I could also argue, and some security people out there may also agree. You don't really want to commit secrets to get now. There's a lot of hesitance in encrypting and committing secrets managers are get get ops friendly. You want to commit stuff to get and make it the source of truth. But honestly, do you want to commit everything?

A

Wouldn't you want your secrets outside of your orchestrator, so in this instance kubernetes and before we go any further, let me get argo cd and secret management out of the way. Here's their stance straight from the website. Argo cd is unopinionated about how secrets are managed. There are many ways to do it and there is no one-size-fits-all solution. So, as you can see, they have more than a few options for you to look at on the docs page.

A

So, let's talk about some tools that I typically hear about or see in virtu, a variety of blogs. uh Kubernetes includes a native secret resource that you can use in your application by default. These secrets are not encrypted in any way, and the base64 encoding used should never be seen as a security feature.

A

While there are many ways to encrypt the kubernetes secrets within the cluster itself, we're more interested in encrypting them outside the cluster, so that we can store them externally and get another one is bitnami, sealed secrets. So the bitnami sealed secrets controller is a kubernetes controller that you can install in the cluster and it performs a single task. It converts sealed secrets that can be committed to get to plane secrets that can be used in your application.

A

Sealed secrets solves the following problem: I can manage all of the kubernetes configs and get except secrets so encrypt your secret into a seal secret which is safe to store even to the public repository and the still seeker can be decrypted only by the controller running in the target cluster, and nobody else is able to obtain the original secret from the sealed secret.

A

Oh, and we also have sops and that's sops and paul s for those of you that want to search for it. But sops stands for mozilla secrets. Operations.

A

Sops is an editor of encrypted files that supports yaml, json, env, binary formats and crypts, with aws kmsg. All of the acronyms age and pgp tldr is a cli that you can use to encrypt kubernetes and secrets, and they also have a section on encrypting using hashicorp vault.

A

Now there are a few disadvantages: if you use these kubernetes secret encryption tools, then your secrets are encrypted and committed to version control, which is what github says: thou shall not use a secrets manager that doesn't commit a secret in version control, but I repeat as a pattern and for the other side, it's actually important to not do this right. The brief disadvantages or questions I've heard from a few people about committing to version control.

A

How do you know who used the secret if it did get used? What happens if you need to rotate it? Do you need? Do you need to find and replace it across all of your repos? What happens if someone else or someone period commits a decrypted secret?

A

These are all valid questions.

A

So let me give you a brief overview of vault and get ops, so we finally made it to vault if you're not familiar with vault vault is an identity based secrets and encryption management system, and I don't know if I said this yet so for you all to know this talk was inspired by discussions. I've had with two people I respect. One is adamant, vault is get ops friendly, natively and the other says it's not. So that's why you see both sides in this talk.

A

So what makes vaults not get ops friendly but like? Is it actually not getups friendly, though vault is its own controller? It has an awareness of the diff for you from a security standpoint. You just need to know that you need to change the secret. You don't need to know what the secret actually is. It's trying to reconcile the state of what it thinks is going on versus the actual secret, it's declarative and it will find something isn't right and overwrite and renew the secret.

A

Also, remember principle: 2 versioned in immutable. The desired state is stored in a way that enforces immutability versioning and retains a complete version. History now there's an argument to be made that while people automatically think about git when referencing virgin and immutable, you know git ops does it have to be in version control. Vault does this because it does have access, control and ways to lessen the blast radius.

A

Your application and version control history associated with the app don't actually need to know the secret and, if you're, using the vault agent injector with kubernetes you're using vault state store in its record reconciliation mechanism. Apologies, so your app doesn't need to know what the state of the secret is.

A

See here I deployed this app to argo cd without having to commit any secrets to get. I let vault handle the rotating issuing manager etc for those secrets and it's versioned, I use kvv2 and it's still storing secrets, you'll also notice. There are no secrets related to the database name or password, and the only token you see is the one related to the service account.

A

And yes, I realized getting things to work was never my issue in my talk, but I was happy that I made it work doggone it and I do realize some people would still rather use environment variables. So if you like using uh environment variables and committed to using them, you can use a template file. While you source the env.

A

And for those of you who don't want to change anything you're currently doing like your app add another sidecar, you can either one use argo vault plug-in, which works with kubernetes secrets or two use secret store, csi driver. I swear I sound like I'm speaking in parcel tongue and the vault provider for it. It will provide a volume mount to use your secrets from vault without a side car container.

A

If you have a secret store, you don't need to commit your secrets to a version control, but vault will automatically detect any differences for you of git ops says configuration must be declarative. You can't configure vault declarity declaratively. Well, you can use terraform for your configs or the vault config operator, which is a community tool. There are two links on this page.

A

If you want to see what the vault config operator looks like in practice, or if you want to see what argo sync wave uses or how it works to sync things that need to reconcile. So conclusion, why are secrets important and can I use them with git ops? Well, at the end of the day, my question is: do we really need to commit our secrets in version control? Is that the only thing keeping you from get ops? Let us know in the chat either way, I'm not telling you where I sit, stand or fall thanks.