Cloud Native Computing Foundation / ArgoCon 2022

A Distributed Job Scheduler Story: CronWorkflows integration with RBAC infrastructure for Over 100+ Teams - Ismail Bülbül & Doğukan Tuna, Trendyol

As a Trendyol Tech Team, we are managing our applications on 200+ k8s clusters which include 7K+ applications on 5 DC. Many development teams in Trendyol encounter a scheduled task solution requirement. There is no common product for such use cases. So with the CronWorkflow capability of ArgoWorkflows, while we are providing a unified solution for distributed job scheduler needs, we aim to deal with scalability and disaster recovery scenarios with its k8s native nature. In this direction, we are combining Argo CD’s manifesting and repository as code capabilities with the workflows templates. Teams are able to create their custom workflows and apply them over Argo CD. On the other hand, to provide an auth mechanism for users, we have combined Trendyol’s Argo CD Dex Server and its LDAP structure. Also, we needed permission separation according to LDAP groups. To provide this structure, we used the namespace delegation feature which comes with Argo WF v3.2.4. As a result, teams can manage their workflows & cron jobs via their allocated namespaces. In this presentation, you will learn how to integrate Argo CD Dex Server with Argo WF which provides service account/namespace delegation.

Argo CD Observability with AIOps - Detect Incident Fast - Amit Kalamkar & Vigith Maurice, Intuit

Intuit runs 2000+ services on Kubernetes and uses Argo CD to implement GitOps workflows and progressive delivery with Argo Rollouts to minimize risks associated with service changes. However, about one third of all incidents at Intuit are caused by rolling out those changes. We have a need to surface operational insights in real-time to make it easier for service operators to detect and resolve change-induced incidents. In this session, we will discuss how we extended Argo CD with in-cluster observability and how we integrated an open source, Kubernetes-native operational analytics platform (Numaproj) with Argo CD to provide AI-driven anomaly detection. We will demonstrate the ease of configuring in-cluster observability with AIOps in Argo CD, how these new capabilities help service operators detect incidents and take corrective actions faster, and how this effort is helping Intuit reduce incident MTTD/MTTR.

Argo Roadmap Updates For CD, Rollouts, Workflows, and Events - Jesse Suen & Alex Matyushentsev, Akuity; Saravanan Balasubramanian, Derek Wang, Intuit

Hear the latest updates and upcoming features from the Argo project maintainers. The roadmap will cover all the Argo subprojects including CD, Rollouts, Workflows, and Events. You’ll get the opportunity to have a discussion through Q&A and provide feedback in session.

Argo Rollouts at Scale: How We Brought Automated Rollback to 2100+ Micro-services at Monzo - Joseph Pallamidessi, Monzo Bank ltd.

At Monzo, we massively invested in the micro-service philosophy and shipping small and often: we have 2100+ services and deploy over 100 times a day. We have great tooling and a fast deployment pipeline, yet our deployment strategy was still trailing some of the modern best practices. We were too reliant on testing, alerting, and engineers discipline: as a regulated bank this exposed us to a lot of risk, fines and potentially having the UK financial regulator taking action. This talk describes why we chose and integrated Argo Rollouts with a complex and established backend. You will see how we migrated 2100+ micro-services safely with minimal change despite the heavy customisation of our cluster. We will address how we improved our tooling to make the experience seamless to the backend engineers, and moved to a world where automated rollback became the default strategy (and not canarying !) By making deployment safer, we guarantee a better outcome for our customers: they will see fewer issues when using our platform. For our engineers, that means less time to deal with incidents, more time for delivering value, and reduces the anxiety and cognitive load of deploying.

Argo Welcome + Schedule Announcement - Pratik Wadher, Senior Vice President, Product Development, Intuit

We will open up the day with welcomes, brief announcements, and an overview of the schedule for the day.

Attacking Argo CD with Argo CD (and then Defending) - Michael Crenshaw, Intuit

Argo CD manages Kubernetes resources, and Argo CD is itself a set of Kubernetes resources. This talk will show how a lax RBAC configuration could allow users to escalate their privileges by using Argo CD to modify Argo CD. We’ll start with a trivial attack and then incrementally restrict Argo CD RBAC and Project restrictions until no attack is possible. This talk will demonstrate the process that every Argo CD admin should follow when setting up their Argo CD RBAC and Project settings.

From Code to Docs to Marketing: How to Contribute - Jesse Suen, Alex Matyushentsev, and Remington Breeze, Akuity; Dan Garfield, CodeFresh; Regina Voloshin- Bank Hapoalim, Michael Crenshaw & Julie Vogelman, Intuit

Have you contributed to the Argo community yet? Interested but not sure where to start? Get an overview of the Argo community and the key areas where your contribution is needed. Covering SIGs and sub projects, identify your next move in the community.

From PR to Artifactory in 60 Secs! Adobe Service Runtime Java Libraries Release Pipelines Using Argo Workflows - Anirudh Mathad, Adobe

- Adobe Service Runtime (ASR) Libraries are used by more than 60% of Adobe's Java Services. A strong community of contributors maintain the libraries across the company. Instant availability of a release once a PR is merged is mission critical for teams across Adobe. - Our legacy process used Jenkins jobs with Maven Artifactory plugins. The cross dependency of libraries, stale libraries in Jenkins VMs, lack of Graph/DAG view, scaling issues and container upgrades were some of the many problems which slowed our release process. - In this talk we discuss how we migrated our manual Jenkins release pipelines to Argo Workflows. How the visual representation of release pipelines through DAGs helped us troubleshoot build issues. Embracing Gitops has dramatically reduced our maintenance efforts and updates to pipelines have become seamless.

Harder, Better, Faster, Stronger Pipelines @ Adobe - Larisa Andreea Danaila & Ionut Maxim Margelatu, Adobe

At Adobe, Larisa and Ionut have been building and exercising fully automated CD pipelines with Spinnaker since 2018. Could you improve on that? Over the years, they've asked themselves: • how robust and hands-off are deployment tools when encountering problems? • how responsive to problems are these tools? • how do you optimize the deployment pipelines for speed? • how fast do you onboard a new codebase? Join them to hear how they've addressed these problems by making a seamless switch to GitOps with Argo. Find out how they managed to provide the same levels of speed and quality for high frequency deployments of dozens of multi-region microservices. They also explore in detail the challenges to move existing patterns to GitOps workflows, including: • isolated test environments • automated promotion throughout environments • automated rollback • automated canary analysis • wave deployments

Keynote: Adobe’s Internal Developer Platform Journey and Lessons - Srinivas Peri, Director of Ethos + Cloud Platform Engineering, Adobe

In this session, Srinivas Peri of the Adobe Cloud Foundation (Ethos) team will share the story of Adobe's journey to the cloud. Lessons learned, mistakes made, and the software built to run thousands of services from hundreds of teams on shared infrastructure in a secure and compliant fashion. Then we will look at how adoption of CNCF in general and Argo in particular is allowing the Ethos team to streamline and optimize offerings and operations while providing increased flexibility and control to our service teams, all without compromising the guardrails that make our platform successful.

Keynote: Challenging the Status Quo with Kubernetes and Argo - Hong Wang, Co-Founder and CEO, Akuity

Keynote: Full Stack GitOps with the DRAG Stack - Dave Nielsen, Senior Director of Community & Developer Advocacy, Harness

As Argo CD has gained traction, it has expanded to use cases well beyond deploying Kubernetes with configuration details stored in Git to deploying applications on top of Kubernetes. But applications developed and deployed continuously with Argo CD also need to be built and tested continuously. In the Drone community, there are many companies using Drone CI with Argo CD to build an end-to-end CI/CD GitOps solution. After interviewing yet another such company, we decided it would be helpful to others if we created a software stack based on Drone CI and Argo CD. We call it the DRAG Stack. It is the first open-source, Full Stack GitOps delivery platform that combines the most popular open-source CI & CD projects (based on GitHub Stars) into one stack that can be easily installed on your laptop or a cloud-scale Kubernetes environment. The DRAG Stack includes Drone CI, a container Registry of your choice, Argo CD, and the Git repository of your choice. We are introducing the DRAG Stack for the first time at ArgoCon. We hope you will find it interesting.. and if you do, we hope you will join us at our sponsored workshop in downtown Mountain View on Wednesday, Sept 21st.

Keynote: Scaling Argo for the Enterprise - Raziel Tabib, Founder and CEO, Codefresh & Thansha Sadacharam, Tech Learning & Insights, Peloton

The DevOps movement was all about removing boundaries and accelerating application development. Now more than ever, it's paramount to embrace automation that lets your developers do what they do best and let them focus on the things that are important to them. In this Keynote, Thansha Sadacharam from Peloton will talk about how the Argo Project helped them be more productive. Also, Raziel Tabib will talk about how Codefresh is democratizing GitOps to enable all teams reap the same benefits.

Keynote: Smarter Progressive Delivery at Intuit - Ed Lee, Fellow & Chief Architect, Development Platforms, Intuit

Intuit has been using progressive delivery to increase operational excellence by simplifying, and in many cases automating, the process of rolling out and rolling back new releases. Based on our experiences, we are contributing new observability features to Argo CD to make it easy to compare key metrics between new and old deployments and quickly make the correct rollout decisions. In the future, we plan to enhance the system to make it easy to compute more complex metrics including anomaly scores for deployments to make our application deployments smarter, faster, and safer!

Keynote: The Future of GitOps - Evan Shortiss, Developer Advocate, Red Hat

The GitOps practice continues evolving and becoming more accepted and integrated into teams daily. Now that we have a more clear path to GitOps, how can we continue to evolve the practice to make it as easy as possible to integrate across the application development life cycle? In this session, we’ll explore possibilities to integrate across the application development lifecycle to further coordinate releases.

Keynote: Welcome + Opening Remarks - Priyanka Sharma, Executive Director, Cloud Native Computing Foundation & Pratik Wadher, Senior Vice President, Product Development, Intuit

Large Scale Batch Processing with Argo Workflows and Events - Rakesh Subramanian Suresh & Saravanan Balasubramanian, Intuit

Argo Workflows is Kubernetes native workflow engine, it is a good fit for batch processing jobs running in Kubernetes. In Intuit, our Batch Processing Platform manages ~40,000 data processing pipelines by using Argo Workflows, 10% of these pipelines are running concurrently each every day. All of these pipelines are scheduled or triggered in an event-driven fashion by using Argo Events, that the workflow executions are based on calendar events, or a logical combination of calendar events, status of upstream pipeline workflows, or events in data like file/database update, REST calls, etc. In this session, we will discuss how we run 40K pipeline workflows in Kubernetes clusters, and share how we use Argo Events to manage ~400,000 different sorts of events for pipeline workflows, and achieve exactly-once triggering.

Leveraging Argo Workflows and Argo Events for Cloud Agnostic Enterprise Machine Learning Deployments and Model Management - Charles Adetiloye, MavenCode & Alex Lerma, MavenCode

The goal of every organization integrating Machine Learning into their product pipeline is to get it into production as fast as possible using standard software engineering best practices, however, in most cases, this is not often the reality. Even when the models are successfully deployed there is always a need for an efficient workflow to monitor for things like model drift, model overfitting, or constantly changing datasets. To solve this recurring problem in our MLOps practice and create a reusable pipeline and solution framework, we have explored and implemented best practices around Argo Workflows and Argo Events. In this talk, we will discuss how we have implemented a continuous deployment environment stack to containerize and deploy Argo workflows for data ingestion, transformation, and feature store curation. We will also talk about how we trigger model retraining and redeployment with the Argo workflow pipeline. This includes scheduled events, message queue updates, and external storage data changes. We will go through the challenges we encountered and lessons learned with recommended best practices for any MLOps team considering the Argo workflow approach.

Lightning Talk: Automating Map-building Pipelines for Safe Autonomous Driving with Argo Workflows at TuSimple - Yuan Tang, Akuity & Ning Xu, TuSimple.ai

Autonomous driving has made tremendous progress in recent years. One of the fundamental challenges to achieving fully autonomous driving is map-building. In order to ensure that autonomous driving is as safe as possible, the map data has to be accurate, precise, and updated in real-time. In addition, the map-building process can be complex due to some steps that will need human involvement and making decisions based on how the workflow behaves. Argo Workflow is a cloud-native workflow orchestration engine that is critical to achieving these goals and grants us fine-grained control over the execution of workflows. In this talk, we will walk through the rich set of features and best practices of Argo Workflows that helped us to set up and execute complex map-building pipelines at TuSimple.

Lightning Talk: Best Practices on Organizing GitOps Repositories - Konstantinos Kapelonis, Codefresh

One of the first questions that must be answered when adopting GitOps, is how to organize the various Git repositories. Environment-per-branch? Environment-per-Repo? Environment-per-folder?

This talk examines some common patterns and practices on how to organize the Git repositories along with their advantages and disadvantages

Lightning Talk: CI/CD for Data Pipelines with Argo Workflows - J.P. Zivalich, Pipekit

Making sure that data pipelines don't break in production is no small feat. To combat this, we are seeing a new paradigm emerging in the data space influenced by modern software engineering discipline. This paradigm borrows heavily from CI/CD concepts, but applies them to data pipelines. In this talk, we'll go over how to implement CI/CD for data pipelines with Argo Workflows as the data pipeline orchestrator. We'll cover storing Workflows and WorkflowTemplates in git, validating them on pull requests, and syncing them to a cluster or clusters when releasing new versions of the data pipeline. Additionally we will cover how to test WorkflowTemplates using sample data, allowing teams to be sure that all base, corner, and edge cases are validated every time that they make a change to their production workflows.

Lightning Talk: Creating A Scalable Foundation for Kubernetes Multi-Tenancy - Lukonde Mwila, SUSE & Kelvin Piroddi, Yieldstreet

One of the biggest challenges when operating Kubernetes at scale is managing unrelated workloads and different teams that share a cluster. In some cases, you might have one big cluster, and in other cases, you might have multiple clusters housing different teams and workloads. Kelvin Piroddi and Lukonde Mwila worked as DevOps leads on a project that used Argo CD to manage multiple EKS clusters, as well as the dedicated Argo cluster itself. As the project grew, more people needed access to the clusters, such as additional operators, solution architects, QA testers, and different development teams that were responsible for microservices. In this talk, Kelvin and Lukonde will elaborate on how a GitOps strategy with Argo CD optimized their continuous delivery, how Argo CD's projects and RBAC enabled secure access control for different teams, and how Argo's SSO feature complemented the organization AuthN model with Azure AD. Furthermore, they'll discuss how Argo CD solves multi-tenancy challenges for GitOps, but can be further complemented by a cluster lifecycle management tool that also supports features like projects and RBAC for other cluster operations outside the scope of GitOps.

Lightning Talk: Don’t Let a Bot In! Protect Fortune 500 Companies & Users - Sudhamsh Bachu, ArkoseLabs & Remington Breeze, Akuity

Context: ArkoseLabs is in the security space. Focused on Fraud Detection space. It protects its customers from Bots and human bad actors. Most of its customers are Fortune 500 companies like PayPal, Microsoft, Amazon, etc. Serving Fortune 500 customers is a double-edged sword for a start-up. Enterprise customers can detect start-up service issues quickly. How would I know? - I built some of the monitoring and detection tools for one of our largest customers. Arkose Labs improved Developer Experience, Increased Product Releases, Improved quality, and reduced outages with ArgoCD & Rollouts implementation. Application availability & stability are paramount as we provide warranty on Credential Stuffing use cases. ArgoCD’sGitOps approach & Argo Rollouts deployment Strategies like blue/green & Canary were loved by the application teams. We were plagued with multi-day releases, rollback, toil, outages, etc. So we implemented ArgoCD with our existing GitHub actions for CI to automate manual tasks, metric analysis, and blue/green deployment strategy to address our pain points.

Lightning Talk: Open Source Canary Deployments with Application Metric Analysis - Hannah Troisi, New Relic

Deploying new code to your production cluster can be stressful. A canary release can reduce risk and give you more control over the progression of your application updates. However, canary releases are more of an art than a science. How much traffic should the canary get? What type of analysis should be run? Which metrics should I analyze? How long should the analysis run? A carefully designed canary configuration can maximize your chance of catching an issue before the new application version is fully rolled out. Luckily, Argo Rollouts provides advanced deployment capabilities that can be customized to your application’s needs. This session will cover the following topics: - How to do canary deployments with Argo Rollouts - The different ways to perform analysis to drive progressive delivery - Selecting the best metrics for analysis (this is very application dependent) - Practical demo: Canary deployment with application metrics from Pixie

Lightning Talk: Running Cloud-native Spark Jobs with Argo Workflows - Caelan Urquhart & Darko Janjić, Pipekit

Companies with large computational workloads often use Apache Spark combined with numerous Python packages such as PySpark, NumPy, MLlib, XGBoost, and more. Unfortunately, as teams add the number of jobs running on a single Spark cluster managing dependencies becomes a nightmare. Kubernetes makes it easy to use numerous packages for large data jobs in distributed environments, and Argo Workflows is the best way to run pipelines on Kubernetes. This talk demonstrates how to orchestrate common Spark jobs with Argo Workflows, from the architecture to resource and workflow definitions. We'll show how to provision Spark and Argo Workflows on Kubernetes to process large data jobs. We'll also show how Argo Workflows and Kubernetes provide distinct scaling and stability advantages for Spark users by running some example jobs. We hope that listeners of this talk will learn the pros and cons of orchestrating their Spark job on Kubernetes with Argo Workflows, instead of traditional local or cloud environments.

Lightning Talk: Using Argo Workflows to Curate Chaos Engineering with LitmusChaos - Amit Kumar Das & Saranya Jena, Harness.io

As resilience use-cases proliferate, Chaos Engineering has become a compelling practice for enhancing your application resilience in production. As a large number of organizations and individuals are moving towards the Kubernetes-Native approach, an important problem arises, testing these applications the right way to prevent outages. In this talk, we will talk about how the LitmusChaos CNCF project leveraged the Argo project to curate Chaos Engineering Workflows as set of actions strung together to achieve desired chaos impact on a Kubernetes cluster. While simulating real-world failures via chaos experiment injection on developer/staging environments as part of a left-shifted, continuous validation strategy, it was essential to create a sequence instead executing standalone chaos injection actions. They are an effective mechanism to simulate real world conditions & gauge application behaviour in an effective manner. Further, we will cover the role of Argo in the evolution LitmusChaos 1.x to 2.x by constructing and executing ChaosWorkflows with the help of Argo and observe the changes which are introduced when the system/application is under Chaos.

Managing Thousands of Automatic Machine Learning Experiments with Argo and Katib - Yuan Tang, Akuity & Andrey Velichkevich, Apple

The fairly recent field of Automated Machine Learning (AutoML) provides the richness of powerful algorithms for model selection and hyperparameter (HP) tuning – one of the most important steps of the MLOps lifecycle. However, it’s non-trivial to advance these AutoML algorithms from research papers to production. ML engineers have to learn DevOps and cloud-native technologies to achieve that, but the main focus should be on inventing new ML algorithms. Katib and Argo open source projects provide an infrastructure to use and develop AutoML algorithms easily and fast in a cloud-native environment. In this talk, we will walk through the best practices (such as Argo caching and synchronization) for managing thousands of complex HP tuning experiments that bring the optimal performance. We will demonstrate how Argo Workflows and Katib bring the best of both worlds of Kubernetes-native workflow orchestration and HP tuning at scale.

Secure by Default with GitOps - A Guide to OPA with Argo CD - Dan Garfield, Codefresh

Argo CD has become the defacto entry for engineering teams to deploy and manage their applications while Open Policy Agent has become one of the most reliable security policy enforcement engines. These two tools work perfectly together and yet there are very few articles and videos on the subject! In this talk, we look at the best strategies for getting these tools to work together to improve the software delivery supply chain. We’ll look at admission hooks, sync waves, and other ways to ensure your applications and clusters are always running in a secure and safe manner. Including example files and reference repos.

Presentation repo: https://github.com/todaywasawesome/osssecure

Securing GitOps Supply Chain with Sigstore and Kyverno - Roberto Carratala & Faz Sadeghi, Red Hat

In this session, we will present how to secure GitOps supply chain by signing and verifying container images within the Argo Workflows pipelines, using open source projects such as Sigstore and Kyverno. We will demonstrate how to eliminate security risks on the software supply chain by signing all container images in public or private registries and ensuring no malicious images are deployed in Kubernetes clusters.

Unleash ArgoCD Observability Superpowers - Leonardo Luz Almeida, Intuit & Deng Zhou, ByteDance

At both Intuit and Tiktok, several instances of ArgoCD are responsible for managing thousands of applications. At large scale, ArgoCD encounters more challenges and performance bottlenecks. While ArgoCD’s UI is a great tool to visualize deployment issues,the nature of problems can go beyond your application boundaries and may actually be related to external factors (e.g. github api is down, argocd controller is overloaded, etc). In those cases, it is much harder to identify the real culprit. Digging in logs can be very useful in identifying internal issues however it requires precious effort and deeper ArgoCD knowledge. ArgoCD’s built-in observability capabilities take away the needle in a haystack feeling for troubleshooting deployment failures. Understanding the root cause is the key element to maintaining healthy SLOs. Come learn from 2 ArgoCD contributors about how to leverage distributed tracing, metrics and continuous profiling to quickly identify and react to the hard problems.

Using Argo Project to Help Elastic infoSec Team in Securing Elastic - Christopher Cutajar & Angel Rios, Elastic

InfoSec’s Security Engineering (SecEng) team at Elastic builds and maintains a collection of K8s clusters using Elastic Cloud on Kubernetes and Helm. Such clusters are used by the InfoSec team for various security use-cases such as vulnerability management, security assurance, threat intelligence, security detection and incident response amongst others. To be able to provide a reliable service and keep up-to-date, the SecEng team leverages various tool sets within the Argo Project that makes it easy and efficient for the team to manage. ArgoCD is used to deploy various ElasticSearch clusters that ingest logs from various different sources such as Qualys, Endpoints, Okta, AWS, GCP, Azure, GitHub and others. Clusters need continuous attention, Argo Notifications was configured together with ArgoCD to be able to keep abreast of the environments. Being a GitOps driven team, Argo Workflows and Argo Events are being leveraged to manage a number of microservices. In this session we’re planning to showcase how easy it is for a team to go from a small proof-of-concept to production in a very short period of time! Not only that, showing how to start evangelizing Argo within the organization.