Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Automating Map-building Pipelines for Safe Autonomous Driving... Yuan Tang & Ning Xu

Description

Lightning Talk: Automating Map-building Pipelines for Safe Autonomous Driving with Argo Workflows at TuSimple - Yuan Tang, Akuity & Ning Xu, TuSimple.ai

Autonomous driving has made tremendous progress in recent years. One of the fundamental challenges to achieving fully autonomous driving is map-building. In order to ensure that autonomous driving is as safe as possible, the map data has to be accurate, precise, and updated in real-time. In addition, the map-building process can be complex due to some steps that will need human involvement and making decisions based on how the workflow behaves. Argo Workflow is a cloud-native workflow orchestration engine that is critical to achieving these goals and grants us fine-grained control over the execution of workflows. In this talk, we will walk through the rich set of features and best practices of Argo Workflows that helped us to set up and execute complex map-building pipelines at TuSimple.

A

Welcome to our talk, this is yang from acuity, I'm one of the maintainers of of argo workflows, I'm presenting today with my co-speaker nin who's, the tech lead manager of map infrastructure. At too simple today we are going to talk about how to automate map building pipelines for safe, autonomous driving with argo workflows are too simple. With that, I'm going to hand over to nin to talk about our workflows for autonomous driving are too simple.

B

Okay, thank you again uh glad to be here today. I'm going to talk about a little bit of about how we use argo workflow in our map building process.

B

Workflow engine is one of the fundamental tools in our map building process. When we compare the different workflow tools, we choose argo workflows for these reasons. First is open source. It gives us the opportunity to read the source code and make necessary changes if needed, and the second is kubernetes native.

B

All our services are hosted on kubernetes and it makes perfect sense to have a kubernetes native workflow engine for easier management and maintenance, three availability and scalability.

B

We value availability and scalability a lot, and that is what our goal workflow can bring us uh four or five uh rich features. There are a lot of features needed in map building process, especially when human efforts are needed in the middle of a workflow.

B

We need to pause the workflow and wait for the human input and the collaboration collaborations and the support we have received very good collaboration that support from equity and help us with some critical features.

B

Next, slightly yeah- and here let me name some examples of collaborations. We had one example is during our process. Sometimes before we finish the workflow, we don't know which step can cause issues or needed needs redo.

B

So if, once that failed, we might need to retry a node a few steps before the failed failed one. If we can have a feature to retry an arbitrary node that will make it make it more flexible to implement another example, is we listen on workflow events or notify other microservices the status changes?

B

We need event aggregation customization to better meet the requirements. Both of these features are critical for us and equity helped us a lot on these. We work together on issues and the tests.

B

We had a lot of pipelines in our whole map building process like a single back processing pipeline and the patch pipeline.

B

On the other hand, we use argo workflow not only for the map building process, but also in our ci cd system for application deployment deployment, and the deployment pipeline here are some screenshots of our workflows and the data processing intermediate results.

B

As you can see in the picture we have some complex workflows and on the right side, we have some data map building intermediate data, I'm going to hand it over to you.

A

Besides, the specific use cases are too simple. That name mentioned previously: machine learning, pipelines, cicd and infrastructure. Automation, as well as data processing, are among the popular use cases we found in the community members. For example, trip advisor, uses other workflows for cicd automation of their machine learning models, intuit uses it for distributed load testing. You can find more use cases and past presentations at various conferences and meetups from the community.

A

In the awesome, argo repository linked at the bottom.

A

Next, I'd like to talk about community contributions, this is a latest diagram from cncf that provides a project rankings for developer velocity based on project activities such as activities on pre-requests and issues. The number of commits, and so on. Argo is one of the fastest growing cncf projects. Argo currently has contributions from over 800 contributors.

A

We also provide mentoring for new contributors, as well as regular contributors meetings to provide an opportunity for the community to participate and in design design discussions or decisions. There are 40 core maintainers to order argo projects from over 10 organizations.

A

Besides active contributions from the community, argo is also widely adopted. It is used and trusted by more than 200 end user companies more than 14k slack members, more than 25k github stars and 6k forks on guitar.

A

We also have active feedback loop from user surveys that we send out every year, as well as in-app surveys that users figure out when using the auger workflows ui below are some example screenshots of the in-app survey. We categorized use cases and actively collect additional use cases from our users on github. We have different issue templates for bug. Reports, enhancement, requests, mentoring, requests and so on. We also open github discussions for q and a's and showcases that are easily searchable on github.

A

There are different slack channels for different argo sub projects in the cncf slackhawk workspace with over 14 000 members. Last but not least, as I mentioned earlier, we host regular contribution contributors meetings to help new or existing contributors.

A

That's a wrap. If you have any questions you can find us on the cncf slack and on various social media. Thank you for listening and see you around at algocam.