Cloud Native Computing Foundation / GitOpsCon EU 2022

100,000 Different Ways to Manage Secrets in GitOps - Andrew Block, Red Hat

Any GitOps implementation involves managing a wide range of resources. Sooner rather than later, there will become a need to manage sensitive assets, such as passwords or tokens. So, how can these types of assets be handled appropriately so that they are not visible in plain text when stored in a Git repository? Secrets management is a prevalent topic in the cloud native ecosystem and given its importance, tools and frameworks have been developed that can be applied to not only the content itself, but also within GitOps engines. Understanding where, how and when they can be used could make all the difference when employing proper security measures while implementing a GitOps solution. Attendees will learn: * Tools for detecting the presence of sensitive assets within Git repositories * Strategies for encrypting data at rest * Integrations with purpose built secrets management engines * How sensitive assets can be stored and used when working with public cloud providers * The mechanisms for which GitOps engines can aid in the management of sensitive resources Upon completion, attendees will no longer have an excuse to leave sensitive assets unprotected again!

Applied GitOps with Argo CD Autopilot Using Multiple Clusters with an ApplicationSet - Hannah Grace Seligson, Codefresh

Using a GitOps controller, such as ArgoCD, to deploy applications allows you to deploy more often, execute rollbacks, and avoid configuration drift. However, what if you are an organization beginning to adopt GitOps and are new to Argo? What if you need to deploy hundreds of applications? Manage several clusters? How do you structure your directory? Or apply GitOps at scale for your organization? You can do all of this with Argo CD Autopilot, which provides an opinionated directory structure, allowing you to promote changes across environments with an ApplicationSet Controller for Argo CD applied by the Autopilot bootstrap. Autopilot enables organizations to simplify disaster recovery and quickly set up one-off environments or easily promote changes by leveraging GitOps. This tool works well for teams that are new to Argo CD but want to enable the ApplicationSet's automation and templating abilities to create, modify, and manage multiple applications simultaneously while also targeting numerous clusters with a no-brainer directory structure.

CTA | How to Get involved + Project Update - What's Next- Dan Garfield, Codefresh; Scott Rigby, Weaveworks & Chris Short, AWS

Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin, Docker

Supporting multi-tenant environments in Kubernetes is easy, right? (insert laugh here) Well, it can be. But, it takes organization, structure, and proper policy enforcement.

At Virginia Tech, I helped build a "Common Application Platform" that gives each tenant its own manifest repo and deploys those manifests into isolated namespaces using Flux. By leveraging Gatekeeper and Karpenter, we can properly isolate workloads into node pools and ensure tenants don't step on each other's toes. And best of all, our tenant config is in a simple Helm chart that we call "the landlord."

In this talk, we'll dive into how we've built the landlord, the various policies and mutations we're using, and how it works... all with the intent that you can build your own platform too! We'll have live demos and even try to break a thing or two!

Crossing the Divide: How GitOps Brought AppDev & Platform Teams Together! - Priyanka "Pinky" Ravi, Weaveworks

Priyanka, now at Weaveworks, and Russ, her former colleague at State Farm will share their varied experiences of application change management and how they updated their process to follow the GitOps methodology. Together, they will emphasize the benefits of adopting GitOps by commiserating on the elaborate difficulties and contorted solutions of earlier alternative deployment strategies. Priyanka and Russ come from different sides of the Application / Platform development team divide, and each bring their own historical context and separate understanding of how a deployment system should function! Be prepared to hear horror stories and tales about those systems that should never be told, about the worst solutions that should never have been built, and problems that should never have needed to be solved - and what it looks like on the other side.

What is GitOps and How to Get It Right - Dan Garfield, Codefresh; Scott Rigby, Weaveworks & Chris Short, AWS

The term "GitOps" goes back to Alexis Richardson's initial blogpost on operations by pull request but over time it became clear that GitOps was poorly understood. At the end of 2020, Amazon, Azure, Codefresh, Github, Redhat, and Weaveworks came together to start the GitOps Working Group. The goal was to work together and invite the community to collaborate on clear principles and definitions around GitOps. In this talk, two Open GitOps Co-Chairs will share how the principles were created, what they mean, and some common misunderstanding about what is and isn't GitOps.

GitOps Based Infrastructure as Code with Rancher Fleet and Crossplane - Hossein Salahi, Liquid Reply

With more demand for deploying at scale, we need a new concept for multi-cluster management. The majority of current multi-cluster and multi-cloud management solutions are missing a unified control plane not only to manage Kubernetes cluster life cycle (vanilla or managed), but also application deployment, security policies enforcement (e.g., RBAC, network policies, etc.). By using GitOps patterns we can solve most of the above-mentioned challenges. Fleet is the next generation of cluster management tooling and uses Kubernetes Custom Resource Definitions (CRDs) to manage GitOps at scale up to deployments of thousands of clusters. Crossplane, on the other hand is a cloud control plane that transforms Kubernetes API into an interface to the public or private cloud providers. The Crossplane fits really well into the Gitops realm, since it allows declarative specifications of cloud services that are stored in a git repository for GitOps tools to pull from it.

GitOps Everything!? We Sure Can!, Ayelet de-Roos, AppsFlyer

Usually when developers mention GitOps, it is to describe yet another continuous delivery mechanism for their (micro-)services. There are developers that to some extent apply it to their infrastructure, but can it describe both? Can you also apply it to your SaaS solutions? At Appsflyer, with an architecture of over 850 micro-services, thousands of cloud resources and dozens of SaaS integrations, we strive to automate them all using GitOps workflow. In this talk, Ayelet de-Roos will present how AppsFlyer applies Terraform with GitOps to automate everything, how AppsFlyer keeps its deployables safe and easy to manage with Flux CD, how AppsFlyer identifies drifts in non-Kubernetes environments, and how to enforce company standards and policies using Open Policy Agent.

Implementing Preview Environments with GitOps in Kubernetes - François Le Pape, Remazing

Implementing Preview Environments with GitOps in Kubernetes: how to spin up ephemeral environments on the fly at each Pull-requests for the delight of your team. Have you ever experienced a delayed-release because your Staging environment was broken by another feature? Ending up merging a non-production tested Hotfix branch to your Main branch? Preview environments are an answer to increasing velocity inside your team. Kubernetes combined with GitOps allow you to quickly create and destroy resources along with keeping a clear code history and infrastructure changes in Git. Using Bitbucket Pipelines for Continuous Integration and ArgoCD for declarative Continuous Delivery, we will go through different challenges you can encounter to deploy Preview environments such as Secrets management with Sealed Secrets, splitting repositories, and clearing resources to prevent further costs.

Infra-like-apps - GitOpsifying Cloud Natively Managed Infrastructure with Crossplane and Argo CD - Al-Hussein Hameed Jasim

Infrastructure as Software with GitOps - Justin Garrison, Amazon

The cloud has enabled abstractions and automation, but Infrastructure as Code (IaC) doesn't scale. You can use declarative YAML or imperative scripts and still lose control. Infrastructure as Software (IaS) allows you to control and scale infrastructure with the same practices as applications. GitOps is an implementation of IaS with lots of benefits over IaC. We'll look at how it's different, when you should use it, and where it potentially breaks down.

Intuitive Progressive Delivery Across Microservices in a Dependency Graph Using Argo Rollouts. - Hari Kumar Rongali, Intuit & Rohit Agrawal, DataBricks

Progressive Delivery as you all know is controlled deployments to minimize the risk associated with service updates. Organizations use tools like ArgoRollouts to achieve Progressive delivery for individual services. Progressive delivery across microservices with dependencies is a challenge that many companies are working to solve. In this session, We will demonstrate how to use Argo Rollouts in achieving progressive rollouts & automated rollbacks across microservices in a dependency graph using capabilities such as automated analysis & custom webhook notifications. We will also discuss how to use your current complex multi-service monitoring queries in analysis and automate the entire deployment flow across microservices. Another important capability that we will present is ‘dry runs’ that will enable users to identify and validate possible automated rollbacks in production without actually doing rollbacks.

Keynote: Everything as Code: Declarative Application Delivery with GitOps Workflows- Christian Hernandez, Red Hat

Modern tools are a key component to building a successful application delivery framework based on the GitOps principles. Combining the advantages of Tekton, ArgoCD, and now StackRox open source projects, deliver security without compromising the quality or speed and encourage security, SRE, and application development teams to get declarative. Join us to see an example framework for what it means to really deliver everything as code.

Lightning Talk: A Practitioners Guide to GitOps - Introduction, Principles, and Implementation in Keptn - Thomas Schuetz, Dynatrace

Not so long ago, we had to remember lots of commands and their execution order to configure systems and infrastructure. With the rise of Kubernetes and Infrastructure-as-Code, we learned that it's easier to declare the desired state of systems and let other tools bring our intentions to the system. Finally, Git can be our best friend when it comes to storing and versioning our configuration.

To put it to practice we look at one specific GitOps implementation approach from the CNCF Sandbox Project Keptn. Together we walk through declaring the desired state for application delivery in Git, see how the GitOps operator translates that definition into tasks and how those tasks get executed by various tools to bring the desired state to life.

Lightning Talk: Day 2 Has Arrived - How Carvel Suite and Cluster API Can Bring GitOps to Your Kubernetes Infrastructure - Pietro Terrizzi, CLASTIX srl

Kubernetes has emerged as the de-facto standard for cloud and container orchestration, thanks to its composability and extensibility; however, these positives don't come without a cost. While setting up a cluster as compositions of YAML manifests may seem like a challenge at hand, managing and upgrading a fleet across multiple environments and distributions could seriously become hell. In this talk, we will discover how to maintain and upgrade the state of an infrastructure deployed through the Cluster API, a Kubernetes SIG project based on kubeadm that aims to extend k8s core capabilities through a configuration-as-code. Then, we’ll see how this methodology can be empowered by the Carvel suite, a composable toolchain that makes use of a declarative and layered approach for application building, configuration and deployment, in order to reduce the toil of cluster day 2 operations.

Lightning Talk: Exploring HashiCorp Vault and ArgoCD - the GitOps Way - Tracy P Holmes, Codefresh

A big topic in GitOps that isn't really discussed enough is proper GitOps secrets management and just how serious of an issue it can be in a cloud-native aspect. While normalized usage of Bitnami Sealed Secrets is typically explored, but not much is seen around using HashiCorp Vault. Especially as it pertains to using Vault with Argo Project's continuous delivery tool Argo CD. What exactly is HashiCorp Vault? HashiCorp Vault is a secrets management tool created primarily to control access to sensitive credentials in a low-trust environment. It can be used to manage secrets, encryption as a service, and privileged access. This talk will explore integrating HashiCorp Vault with Argo CD, any pros and cons, and (hopefully) what ended up working for Tracy and her sensitive credentials. Note: A shorter version of this talk was presented at HashiTalks 2022. However, the intent is to go more in-depth with exploration and findings while attempting the integration.

Lightning Talk: GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador, Anova

Organizations that use progressive delivery are able to ship new code faster, reduce risk, and continuously improve customer experience. Progressive delivery is an essential component of DevOps, and feature management is the primary way it works. In this talk, Marco Amador (Anova) will describe their journey into progressive delivery with some hands-on demos and explain why they've chosen progressive delivery on their multi-cluster and multi-region Kubernetes cluster.

Lightning Talk: GitOps, A Slightly Realistic Situation on Kubernetes with Flux - Laurent Grangeau, Google & Ludovic Piot, theGarageBandOfIT

You're tired of talks that deploy hello-worlds to demonstrate the relevance of the younameit tool. That's good news: what we're interested in is trying out a slightly realistic DevSecOps situation. So we're going to build a step-by-step enterprise scenario where devs and ops collaborate on a daily basis around a GitOps workflow based on Kubernetes and Flux. The dev teams deploy / update / rollback Pokémon WebApps using Kustomize and/or Helm charts. On the Ops side, we take care of the platform's security issues by implementing Kyverno: segregation of team rights, WebApps network flows and control of activities on the cluster. And we monitor everything via Prometheus and Grafana. Finally, we will see how to articulate upgrade and configuration while respecting the blue/green pattern and canary deployment, thanks to Istio. At the end of this hands-on, you may have discovered some technologies. But above all you will have seen how to implement them in a dev-to-prod process that resembles a real case.

Lightning Talk: Hiding in Plain Sight - How Flux Decrypts Secrets - Somtochi Onyekwere, Weaveworks

GitOps has been all the rage of late and it requires you to store all your YAML files in Git. This works great for YAMLs containing non-sensitive information but it gets trickier for files that contain secrets even if the repository is secret. Anyone who has or gains access to your repository can access your secret, which could include database passwords and API keys. This talk explores how Flux, with the help of projects such as Mozilla SOPS and sealed secrets, lets you encrypt your secrets and then store them as files in Git. Then, it decrypts them for you and applies them to the cluster. Flux also lets you use key management systems (KMS) in the major cloud providers so you don’t have to create the secret containing your private key in the cluster. The talk will end with a showcase of this feature in Flux using live demos and practical examples. These tools and techniques will help users to benefit from GitOps with the added security that Flux brings.

Lightning Talk: How Intuit Enables GitOps at Scale For All Its Developers - Omer Azmon, Intuit

This is the story of how we at Intuit learned what it really takes to enable our front-end, back-end, and AI developers – all our developers – to rapidly create, update, and dispose of applications. Our learning from building such a GitOps on-boarding/asset-management system include: * What are the independent personas whose needs must be addressed by any such system, and how to resolve their disparate needs. * What are and how to handle the overlapping needs of the assortment of application types: service, serverless, UI, ML, etc.. * Why is updating and cleanup harder than onboarding * Why workflows, orchestration, and even traditional choreography can't handle the variability of such a system, and what we did about it. * How to avoid flurries of PRs. * What happens when a developer can rapidly standup an asset and abandon when not needed like cattle, including build, deploy, persistence, everything * Why is it impossible for one team to develop this alone, and how to make it possible to deliver such a system. We hope that you will find the needs common and join us in building an open source community based on our proven tools.

Lightning Talk: Taming the Thundering Gitops Herd with Update Policies - Joaquim Rocha & Iago López Galeiras, Microsoft

gitops in Kubernetes is a simple but powerful workflow: declare the deployments' desired state in git and an agent (often Flux) should pick it up and reflect the state in the cluster automatically. However, this approach allows for the propagation of issues by "broken" versions of software, which could be avoided with a progressive rollout and enforcement of policies around those. In this talk we propose the use of an update and policy manager – Nebraska – as a complement to gitops. Nebraska allows to set up policies to be met for granting updates, and aggregates the data about update statuses. The integration is accomplished using a new Nebraska Update Agent (NUA), which controls Flux itself, automatically reports statuses, and has a minimal impact to the gitops users’ workflows. With NUA and flux, users can manage new deployment rollouts in a more controlled way, by defining policies for updates, for example: update just one cluster at a time and halt all updates if one cluster fails to update; update cluster only during certain hours; see a global view of the updates’ statuses and drill down to any error reports. Hence, tying gitops and policy-based updates in Kubernetes.

Managing Thousands of Clusters and Their Workloads with Flux - Max Jonas Werner, D2iQ

With Kubernetes becoming more and more popular, so is managing clusters at scale. Applying GitOps principles with Flux simplifies provisioning clusters and managing workloads deployed onto them, including tenant and RBAC management. In this session I will demonstre a best practice approach towards GitOps with a management cluster handling the provisioning and further maintenance of clusters, tenants and workloads, employing the CNCF projects Flux, OPA Gatekeeper and the Kubernetes Cluster API sub-project. The benefits of such an approach are: * It creates a simplified way to declaratively define thousands of clusters and perform operations on those clusters * Makes it easy to have a multi-tenancy approach where each team or group of applications gets their own cluster or individual namespace on a certain cluster * Operations against clusters are fully audited and attributable, as reverting changes is hard.

Organizing Teams for GitOps and Cloud Native Deployments - Sandeep Parikh, Google Cloud

Large scale Cloud Native deployments typically include multiple teams running multiple applications across multiple environments - but how should teams be organized to enable efficient software delivery? How should responsibilities be split between platform, DevOps, and application teams? In this talk we’ll walk through the different approaches teams can adopt for organizing Git repos, handling upstream dependencies, and managing software rollouts. This talk will go in-depth about repo structure and strategies for managing the release process, as well as how to enforce policies across configs and manifests.

Pipelines and the Multiverse of Madness - Christian Hernandez & Hilliary Lipsig, Red Hat

Gone are the days of daisy chaining Jenkins Jobs together. With modern CI/CD tools pipelines are the new high ways. But unlike a driving a car, which you can only take down one road at a time. CI/CD pipleines have the power to run concurrent processes, across multiple lanes, to get to the same destination: Code in production, fast, reliably, and automatically. Join us for a review of Pipelines, CI/CD, and how to leverage tools to get jobs done with your GitOps workflows.

Solving Environment Promotion with Flux - Sam Tavakoli & Adelina Simion, Form3

Without a doubt, Flux is now one of the most popular tools for GitOps. Form3 have been using Flux extensively for PR based operations of our Kubernetes clusters, which has resulted in a great developer experience for their growing engineering teams. However, Flux Kustomize overlays have proven insufficient for Form3's complex business needs, which involve multi-cloud workloads and controlled releases between environments. As a result, the Tooling team at Form3 have written their own tool, k8s-promoter. It automates the promotion of workloads by creating pull requests which copy manifests to the target cluster. Then, Flux reconciles manifests from the directory structure and correctly promotes workloads. This talk will cover: - How the typical commit/deploy flow at Form3 looks, as well as how they use GitOps - Why the tooling Team have decided to use promotion via duplication - Architecture and design details of k8s-promoter - A demo of the deploy flow using k8s-promoter - Lessons learned and future improvements for the project Join this talk to learn from the journey of solving the problem of environment promotion at Form3!

We Have Always Done It This Way! Now Let’s Try Something Completely Different -Eliran Bivas, AppsFlyer

At Appsflyer, we have 400 engineers that write software in several programming languages and with an architecture of over 850 micro-services, thousands of cloud resources and dozens of SaaS integrations. We felt a change was needed. Can GitOps be that change? In this talk, Eliran Bivas, AppsFlyer's Cloud Native Leader, will present the challenges the department faced and still faces when adopting GitOps practices. How AppsFlyer’s Engineering Platform organization changed, how AppsFlyer uncovered the unknowns, and how AppsFlyer educated its Platform Group, and later the entire R&D organization to practice GitOps.

When GitOps Meets UX - Cansu Kavili Örnek & Angels Dimitri Gutierrez, Red Hat

We had the ambition to create a sustainable platform to support development, lower the cognitive load of onboarding new applications and teams, and increase the products' visibility while giving developers the freedom to experiment. Sounds like a lot of work, right? Yet GitOps provided us fast and fine solutions for processes like self-service or onboarding! But that required some design considerations and a strong relationship between platform and developers. And that's where UX comes in!

Join us to talk about how to combine tech and UX practices to bring the best out of GitOps and create a state-of-the-art platform focusing on DevEx! Together we’ll address questions like:
*What are the benefits of working techies and UX together?
*How can we validate the ideas and merge them with devs feedback to create platform features and implement them through GitOps?
*How to simplify GitOps and make it accessible?
*How can we empower devs to own the approach and encourage them to contribute?