Argo ArgoCon 2021, 10 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ArgoCon '21 - Dynamic, Event-Driven Machine Learning Pipelines with Argo Workflows (Ian McGraw)

Description

Most people would say Argo is a YAML-based orchestration framework. But really, it's a highly expressive YAML-based API. Recently at Arthur, we've harnessed Argo to build highly dynamic, event-driven workflows using the Golang Argo Client and the Kubernetes API. Using this setup, we can create 1000's of variations of the same workflow in seconds to configure multiple machine learning pipelines unique to each of our customers. These pipelines stand up dynamic infrastructure, build and deploy containers, and set up autoscaling resources to implement flexible, scalable, and cost-effective big data processing. In this talk, we will describe the journey that lead us to Argo and some of the road bumps we had with competitors along the way. In addition, we will describe how we use GO to build and run event-driven, dynamic workflows that power our machine learning as a service capability. By the end of this talk, listeners will learn how Argo not only stands above the competition for static, scheduled workloads but continues to power the most cutting-edge, event-driven machine learning capabilities.

A

Up next, we're joined by ian ian is a platform engineer at arthur where he led the effort to assess and implement our go workflows. The team uses argo workflows to dynamically, orchestrate the data pipelines and infrastructure that powers arthur's machine learning as a service capabilities.

A

Ian has over five years of data platform engineering experience previously at zillow capital, one and qualcomm. He holds a bs in computer science from the university of michigan. Welcome ian.

B

Hi everyone, my name, is ian mcgraw, I'm a lead platform engineer with arthur aai, and today I'm going to be telling you a little bit about our dynamic event-driven machine learning pipelines that we've built with argo workflows and go to start the talk, I'm just going to give you a quick intro on me and a little bit about my background and then I'm actually hoping to tell you a little bit about arthur the product and sort of how we use argo under the hood and then would actually like to dive into like our architecture with argo and then show you a little bit about how we use it with go to build these event, driven dynamic pipelines.

B

So with that um a little bit about me, as you can see there in the center, I'm mission congrats so go blue. If anyone else is on the call in michigan grad we're having an incredible fall. So hopefully we can keep this going before that I was actually from michigan originally and in that upper right corner, you can see a photo near my hometown in northern michigan. You can see the lakes and the beautiful sand dunes. I definitely recommend it if anyone ever gets up there to check it out.

B

uh Most recently, though, I actually live in new york city, which you can see the skyline there in the middle and when I'm not hands on keyboard building, something I usually like to get outside climb rocks. I also love water sports and things like that, as well in terms of professional career. Originally, I was working actually at capital, one in their center for machine learning, where we were working on model training, infrastructure and building some machine learning apis for customers.

B

After that I actually went on to zillow and for anyone on the call in new york city. I was actually the lead engineer for the brand street, easy which a lot of new yorkers know for apartment hunting after that and most recently, I've been at arthur now for a few years and I'm leading their data platform and engineering in that regard.

B

So with that, what I'd actually like to tell you a little bit more about arthur and for those of you who haven't heard of arthur, we are the ai performance company, and that means essentially anything monitoring observation related to deploying machine learning in production. We provide a platform to do those things and our product kind of breaks down into three categories.

B

The first is performance, so we do data drift detection, we check for model accuracy, we track the performance and we do things like logging, all of your predictions and doing alerting and things like that out of the box.

B

The second pillar, then, is explainability and for people not as familiar with machine learning explainability, you can think of as a reason why a model made the prediction it did and we'll actually show an example of that later and then our third pillar is fairness and binds. So we provide tools both to analyze bias, as well as mitigate bias in your predictions and algorithms, and so that's kind of arthur from a high level, and today I'm actually going to be talking about one of our platform capabilities.

B

We call enrichments, which argo is at the core of and to start that conversation I'd like to actually ground it with an example here and so the way that arthur product works is customers have models in production.

B

Those customers are feeding data into their models and getting predictions out of their models, and then they send us both that input and that output. So as a hypothetical example here, let's take this customer data.

B

You know this customer might be trying to decide if people are worthy of a loan, for example. So let's say this customer feeds in this record into their model and their model hypothetically did not exit this person for credit, so the customer then, would take both of these things, send them to arthur and then in the arthur product. We apply machine learning on top of these data streams and add what we call enrichments.

B

So the first example of an enrichment is an explanation and, like I said before, we think of explanations as a reason why a model made a decision it did so. In this case, we see the explanation of income is the highest, so it in the model paid the most attention to that column.

B

Another enrichment we do is like I mentioned before bias mitigation, so using statistics we can actually analyze all of the predictions your model has made and make sure across those different groups, you're making fair predictions. So in this case, maybe we identified this record as someone who had been discriminated in the past. We might actually change that prediction to a yes.

B

The final example of an enrichment, then is anomaly. Detection and anomaly. Detection is a number that tells us how similar is this customer data to the data they've sent us in the past so collectively, then we call this whole bottom section arthur enrichments and I just kind of to ground this from a high level.

B

Again we can think of enrichments as arthur's machine learning deployed on top of customers, machine learning, so a little bit of machine learning to monitor machine learning there, but that's over simplification and customers can use these machine learning capabilities sort of as machine learning as a service. So they can say I want anomaly detection and push a button and get anomaly detection.

B

All of our enrichments follow a similar pattern where something is trained saved and then deployed against their customer data stream and at scale, obviously with a lot of customers and all of our customers having a lot of models.

B

We have a ton of data streams running in our back end and orchestrating all of these machine learning pipelines and plugging them into those data streams is a huge, huge, complex, orchestration task.

B

So this is where argo comes into play and we actually started down the road with argo about a year ago and when we first were looking for orchestrators, we evaluated three main ones. Obviously, airflow everyone hears about a lot and airflow had a number of drawbacks in this use case, specifically because it's so tied to a schedule, we needed something that could be event driven and dynamic, which airflow did not do well.

B

Prefix, then, is a little bit more in that direction, but prefix is so heavy and runs like a variety of services on the back end that it's actually a pretty heavy application to install, which is important for us, because we actually ship our product to customers, so they can install it in their on-prem data centers or in their private clouds.

B

So we need to be very cognizant of every container we ship, so prefect was too heavy for our solution, which then we found argo which ended up being a great fit because it's kubernetes native, which so is the rest of our platform. So that was perfect and it's a single container. So it's super lightweight and we don't have to worry about. You know maintaining a suite of micro services on a customer on-prem deployment.

B

One of the things we actually didn't like with argo at first was that it was a yaml based definition, and we spent a bit of time iterating on this, because we wanted our back ends to be able to communicate with argo sort of natively and our backend is actually all written and go. At least our api layer is, and we realize, because it's a go, we'd actually import.

B

The argo source, client libraries to manipulate those workflow objects natively, and this is sort of an unlocking point for all of us, and we realize we could then sort of do much more with workflows in a codified way. We can make them event driven. We can assemble them dynamically. We can even unit test them, and this is all sort of been a huge huge win for argo.

B

In our specific use case and if anyone's interested, I have a link to a blog post where we dive in a little bit more on the comparisons between these different orchestrators.

B

If anyone would like to check that out next, I'd actually like love to dive into the architecture here of how sort of argo fits in this machine learning, dynamic, pipeline event, driven ecosystem we've built here in the upper left corner, you can see a sample customer and that customer is making a simple api call. That says you know I would like anomaly: detection or some sort of enrichment.

B

Our enrichments api then receives that request and looks up information about the customer, their model and the type of data that their model is so, for example, it could be computer vision, processing images, it could be tabular, it could be nlp where it's looking at text, so it combines all the information and then on the fly, constructs a workflow to stand up and train the correct algorithms for that model and that data and then hook those deployed algorithms into that customer data pipeline and configure all the infrastructure needed and the data flow needed to make that a unified stream processing pipeline.

B

So once that workflow is then assembled, it submits that workflow to kubernetes, at which point, of course, argo picks it up and starts executing it, and we can see it's sort of a representation of the argo workflow in that big hexagon on the lower left hand, side and within the hexagon, we can see three different enrichment streams, those enrichment streams. Then you can see the common steps, so they validate data. They update, you know, statuses, to say things like processing.

B

It then trains and saves the algorithm and then there's a step in the workflow that actually calls out to deploy all of the infrastructure needed to serve that algorithm and hook it in to that data ingestion pipeline. We can see sort of on the right hand, side there.

B

So looking at the data ingestion pipeline, then we can see how those enrichments piece in and the data flows from one to the next to the next before finally ending up in our olap database, where customers query it, we do alerting and all of those things.

B

One of also the huge wins of using argo and kubernetes. To do these data processing and serving data streams. Is we've actually been able to build a super scalable stream processing engine here, we've instrumented our kafka cluster with metrics. That report back to prometheus about the lag of all the topics and we've actually built an in-house custom hpa that queries prometheus for those lag, metrics and we'll scale up and down our enrichment processing based on the lag in those topics, and this actually goes as far as when a topic is empty.

B

Like all the data has been processed, it will shut down the enrichment completely, so we're not paying for any compute, and this has been huge because some of these enrichments are extremely compute intensive.

B

So we've been able to save a ton of costs and uh you know optimize our infrastructure, so it grows when we need it with this setup and we've been actually really happy with it and we're continuing to build and evolve and work with so up. Next, then, I'd like to show you actually a quick demo and the source code for this demo is available publicly on github.

B

So if you're ever trying to experiment with dynamic workflows in argo or see how to import the argos source code into go feel free to check it out, I definitely would recommend it so quick snippet of code here to show you in this demo. I've created a chef, argo. We can think about it.

B

Where argo can cook you, a variety of meals based on what you like to eat and the core of the source code comes down to these few functions here, where we're actually importing the argo types for the templates and the dags, and you can see here that workflow itself.

B

So this code then assembles a dag of steps to assemble each meal and then combines those into a workflow and submits it to argo. So I'd like to actually show you what that looks like behind the scenes now so to invoke the script you can just run. You know feed me and then let's say I'm in the mood for steak and cake for dinner, so I'll go ahead and submit that.

B

Then, if we flip over to argo, we see that newly created workflow and we can see it's a little small right now, but there's two main branches at the beginning, one to cook my steak and one to cook my cake and as we can see, this will evolve and those steps will finish- and um let's see, let's give it a second here so yeah we see the steaks finished and now the cake is finishing up as well and we can even extend this then to say you know, I'm hungry for a lot of things.

B

Let's make all the meals we possibly can so I'll go ahead and submit that as well, and we can see now we have a second workflow running and this one's much bigger for creating all these different meals and really these workflows are just an example of things you can do.

B

But you know the point here is to illustrate that it's very easy to set up code to build and construct and submit these workflows to do an arbitrary number of tasks based on user demand.

B

So this could be understanding of infrastructure, training, ml pipelines, much managing that infrastructure or different things like that, so super powerful and the expressive way to use argo almost as an api behind the scenes to implement a lot of these features that you know, customers would like to see in long-running workflow tasks so that I'd like to flip back to the slides- and I just like to finish up by saying you know if anyone has any questions or comments, please connect I'd love to hear from you guys is anyone doing something similar?

B

Have you run into roadblocks? What have you found has worked? Well, I would love to connect, feel free to hit me on linkedin or shoot me an email and also just a shameless plug. We are hiring so you know engineers, business sides, sales marketing, if anyone's interested in working on sort of the next generation of ai and the tools and the ecosystem around it feel free to reach us out there we'd love to speak with you with that. Thank you.