Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: A Distributed Job Scheduler Story: CronWorkflows integration with RB... Ismail Bülbül & Doğukan Tuna

Description

A Distributed Job Scheduler Story: CronWorkflows integration with RBAC infrastructure for Over 100+ Teams - Ismail Bülbül & Doğukan Tuna, Trendyol

As a Trendyol Tech Team, we are managing our applications on 200+ k8s clusters which include 7K+ applications on 5 DC. Many development teams in Trendyol encounter a scheduled task solution requirement. There is no common product for such use cases. So with the CronWorkflow capability of ArgoWorkflows, while we are providing a unified solution for distributed job scheduler needs, we aim to deal with scalability and disaster recovery scenarios with its k8s native nature. In this direction, we are combining Argo CD’s manifesting and repository as code capabilities with the workflows templates. Teams are able to create their custom workflows and apply them over Argo CD. On the other hand, to provide an auth mechanism for users, we have combined Trendyol’s Argo CD Dex Server and its LDAP structure. Also, we needed permission separation according to LDAP groups. To provide this structure, we used the namespace delegation feature which comes with Argo WF v3.2.4. As a result, teams can manage their workflows & cron jobs via their allocated namespaces. In this presentation, you will learn how to integrate Argo CD Dex Server with Argo WF which provides service account/namespace delegation.

A

Hello, everyone Welcome to our presentation. We are happy to see you in this session today we will talk about our job education story with the Argo, workflows and transfer. We will talk about why we need a job schedule and how we came up with a solution on our workflows. They are part of the developer experience. Our main focus is increasing the auto productivity and our customers are generally developed and data teams. We try to provide unified solutions to common problems into radio techno strips. Let's start with introducing ourselves.

A

I am a software engineer at durability. There are main focuses, developer productivity.

A

Some of my personal interests are distributed systems, application platforms and sometimes I, enjoy writing mobile apps from this point to account will continue and he will talk about who are we our current Tech status at the Journey of job scheduler, with other workflows, interaction.

B

Thanks Ismail, hello, everyone, I'm loka I'm, a software engineer at tranvio, I am focusing on production delivery tools such as the arbor products, and also I, have a special research interest for kernel engineers, GPU platforms and its applications on metaverse plus.

A

B

I constantly do developments with C, plus plus I hope you find this session interesting and also insightful.

B

So, let's head into our agenda, we will talk a bit about teranjo and then we will head over the story of our job, scheduler needs and solutions. Finally, we will pass over what's coming next and all the equation. So as a reminder, any questions are welcomed. Please don't hesitate to ask your questions.

B

Let's start with who is Tradio is a Ecom and technology company, and also first decagon in turkey and one of the top e-commerce Platforms in the world. The annual mission is to make a positive impact by offering a seamless e-commerce experience to customers and sellers here on the screen. You can see some of the important facts about trundle.

B

As you can see, we saw multiple platforms to millions of people and also actively operating in turkey and Germany, and a few European region. Countries too, that's why our platforms are always in some kind of pursuit of innovation. If you want to learn more about perennial, please visit our LinkedIn page. Therefore we can look at our current text. Status.

B

I will talk a little bit about our application, infra and platform status. Our current status heavily relies on the kubernetes ecosystem in this journey. There are more than 250 clusters and we are one of the faster adapters of Argo and maintaining all Argo products family internally, such as rocd power, workflows, cargo, logs and algo events.

B

Now, let's look at our metrics. In this part, infrastructure metrics are shown up. Service count is more than 8200 and 7500 registered on Largo City. All of our infrastructure, metrics are publicly reachable foreign.

B

Please feel free to inspect. There is lots of info in real time about regions, memory and CPU usage, virtual machines clusters and many more systematics to square the info in here shows to register job info are the workflows currently by the team, assembling and processing data from online spaces via our service catalog data collector for Chrome workflow guides and calculating service adaptation, a general team by T. We definitely love Argo products.

B

We care about the current way is this like in general, but how did we get here and how did we reach the arbor workflow paths?

B

This is the second section in our agenda where we try to explain why we needed a distributed job scheduler and why we choose our lower flows. We all know that maintaining and scheduling service shops in Big technology companies could be pretty challenging and this could make developers uncomfortable when you have such conditions. Innovation is surely indeed in our situation at first team teams were keeping their jobs on different platforms and environments, failures and doubts where constant arising Runners were blocking. Cpu utilization was Rising. The also there was a nuclear of visibility.

B

Jobs had an issue of lack of stability, manageability and effectiveness. We have more than 750 scheduled jobs, distributed on different tools and solutions, so handling Chrome jobs in such way plus overhead. For us, it was increasing our error rates and eventual occasional audience in time developers requirements dropped into our backlog because developer experience should have been improved and eventually our search for a new platform started.

B

The first Criterion was to handle and trackshops without issues and complexity of configurations. The invite solution for distributed job scheduler needs ease of dealing with scalability and Disaster Recovery scenarios compared and scared various platforms for days, then analyze different platforms to manage workflows without pay for more than 100 teams and more than 750 jobs.

B

So our first choice was to find a kubernetes native platform, as our entire infrastructure is kubernetes based. Finally, we decided to head into Aguilar thought because we were already part of the Arlo ecosystem with Argo CD, so the mindset is the same with this platform. We want all of these and the rest isability. Thanks to the Ardo community.

A

Starting point of our journey, why we need it and what we were looking for and now I will try to explain why we choose our workflows and here in the slide, we Mark some of advantages that argue workflow cell does I will believe Passover.

A

It natively supports kubernetes, okay, guys it is our first criteria while searching the right tool for our needs. So this is a big plus for us, it's product of our government. We will be using rocd for more than a year, so it was close to our ecosystem and we had a know-how of it.

A

Ease of deployment and maintenance in Toronto is a Tech Deck keep growing. We want to keep our Footprints smaller every day. We want to minimize effort to maintain these tools. So the argument floss is based on custom resource definitions on kubernetes. We can keep all the conflicts, eczema, yamas and it stays stored on kubernetes, it's containerized. We want to keep the columns or containers to use our resources more effectively.

A

Its fault tolerance argument flows can discharge schedules with the backup mechanism. That's really important to us. So far we talk about our adventure and right now we can jump right into our architecture overview.

A

Now we can start to talk about the actors. This architect should always shows us how we make it possible for this kind of platform. Delivery, I'll, try to explain in two parts, one for installation and backup processes, then another one for Arabic and manifest of workflows.

A

Our accelerating is maintaining our installation and backup repositories over pipelines, the backup configs every night. The conflicts are all the workflow kinds, namespaces and Arabic configurations.

A

All of them can be up and running in just minutes before diving into details. I would like to mention that kubernetes native nature of our workflows takes away all the stables and Disaster Recovery complexity from us. He can just export and apply Amos in software. Besides that we separate team Resources by kubernetes, namespaces Argo, workflows, support this feature with SSO Arabic namespace delegation that comes with version 3.3, which is currently in beta.

A

This feature help us to Define Arabic for each team's. Namespace with this feature are isolated. Kubernetes namespaces help us to prevents closed team modifications on workflows.

A

Here you can see we are using workflows closely with argosity, which helps manifest our infra escort. Let's move to the next slide, we will explore the Arabic mechanism deeply and the integration of teams, let's dive into installation and backup processes.

A

Sls, are maintained, infrared to reposters one for installation where they can also keep it simple to latest versions. On the other side, we provide Dex core equipment in the installation repository to use argosity deck server for SSO.

A

Another one is backup Repository, which is just another Chrome that back up all the workflow clients and copies from our production cluster. Every day. In the case of emergency, we can move our workflow to any data center and cluster in just minutes.

A

B

A

Connection of using customized with Argosy, they keep all the configuration and workflows as customized, manifests and apply through argosity to argue workflow clusters, as you mentioned in the earlier steps, we separate teams by kubernetes namespace. Therefore, we provide unified sales accounts to all team members over Elder groups, since the obviously the Dexter is already integrated with our ldap server. We can reach lap groups from other workflows for that. We just needed to integrate two products of alcohol.

A

Actually, we had three service accounts for an airspace. These are admission seconds default style. Second and read: write service accounts.

A

Do you think one of the service accounts with the Arabic annotations as block Soviet president's highest? One is admin the lowest one is default default. One is just a return name. Is we use that just for three weeks?

A

So how do we integrate teams? You just add an email, space and arbit to our manifest. Then I'll go see the text job. This is an example of how we add Hardware, cruel annotation with their elder group. We just modified this over customized and actually our base status accounts teams are promoting themselves or our integration guidelines, so they can move their jobs within the following two documentaries: that's how we Empower all the workflows with smooth authorization and platform delivery, and here you can see our roadmap for the Argo workflow platform inside scenario.

A

They are aware of the power of Argo workflows. So that's why we want to distribute this platform to the most of our developing things. Besides that we mentioned, we have five different data centers, so another Focus will be a multi-dc, stable team distribution and job networking by iterating. All this, you will continue to follow the new features of our workflow coming to office. We appreciate all contributions from the community.

A

That's all we have for today, thanks for having us here, I hope you enjoyed this session. If you have further questions on our approach, please drop your questions.