Dagster Dagster Talks, Panels, & Interviews, 18 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building ML Pipelines with Dagster: The role of the orchestrator in machine learning, Sandy Ryza

Description

Building ML Pipelines with Dagster: The role of the orchestrator in machine learning
Dagster is an orchestrator that puts data at the center. While orchestrators typically focus on sequencing computations in production, Dagster brings orchestration to the entire ML development lifecycle
Slides: https://yadi.sk/i/DuZkASxS7ZbvDA
Sandy Ryza, Software Engineer @ Elementl, working on Dagster

All talks & tutorials from Machine Learning REPA Week 2021:
- All talks at Track 1 - Machine Learning Product and Team Management: https://youtube.com/playlist?list=PLlxErbAvYYLBa2uQkROxn4OZJvGQACn8i
- All talks at Track 2 - ML pipelines automation. Code and Data version control. Reproducibility. MLOps: https://youtube.com/playlist?list=PLlxErbAvYYLDRP6cHtVP76f2g5Yoh6c5R

Links:
- Learn more about ML REPA: https://mlrepa.com/
- Learn more about LeanDS: https://leands.ru/
- Learn more about DataTalks.Club: https://datatalks.club/
- Machine Learning REPA Week 2021 Online Conference: https://mlrepa.com/mlrepa-week-2021

Join us:
DataTalks.Club #mlrepa - https://datatalks-club.slack.com/archives/C01Q6698JTV
Slack ODS.ai #mlrepa -https://opendatascience.slack.com/archives/C019A9H5V0X

#mlrepa #machinelearning #datascience #reproducibility #mlops #artificialintelligence #ai #python #deeplearning #technology #programming #coding #bigdata #computerscience #data #dataanalytics #tech #datascientist #pythonprogramming #ml #developer #software #robotics #innovation #coder #datavisualization #analytics #neuralnetworks #leands #automation

A

Okay, so yeah, my name is sandy rizza, I'm an engineer at elemental, which is the company behind dagstr.

B

um There we go.

A

So, first of all, a little bit on where I'm personally coming from my career has been kind of a loop. I started at cloudera where I was building software to process. Big data sets. uh I worked directly on hadoop and then spark and co-authored a book about spark. It got some popularity at the time distributed systems seemed cool, but machine learning seemed even cooler, so I spent the next few years working as a data scientist.

A

First I was at cloverhealth, then managed a team at keeptruckin. The experience I had with tooling in those roles finally led me back to the world of open source data software. So now I'm working on dagster, which I'll talk about soon before we talk about dexter, though, let's talk about data flow, a little bit abstractly when many of us visualize how data flows into machine learning models. It looks something like this. First, we pull a bunch of data from different sources and land it in a warehouse or data lake.

A

Then we run a bunch of transformations to clean it up and derive features, and then finally, we direct we pull data sets from the lake to build machine learning models.

A

If we work at a sophisticated organization, um we might have internal platforms for each of these components so for ingest. You might have some nice system that lets you specify data sources that get replicated into the warehouse um you might want to simplify transformations, maybe use dbt or something to build derived tables, and then maybe you've got your ml system. That makes it easy to target some data train a model on it and wrap that model on a web server like uber's, michelangelo or sagemaker, or something so. This is all hunky-dory right.

A

I'm going to argue that this vision is actually a little bit counterproductive and it's kind of reductive, because it views these as separate steps when in fact they all sit on top of the same data structure, which is the dag.

A

So here's what a more realistic machine learning tag might look like. It starts with a set of source tables that we ingest and extract some features from. We use those features to build a model. um We back test that model and also maybe use it for batch inference um and then likely. We use the inferences from our back test or production set to derive even more tables.

A

Maybe they do reporting on the model, um some sort of analysis, or maybe even like a production email campaign to our customers, that's based on the model, so I want to point out a couple of questions um that become important to us when we think about our ml pipeline.

A

In this way, the first one is what happens if this changes, so if you make any changes to the base tables they're going to affect what happens in the modeling phase, um so a change to how one of your columns is defined is just as likely to hurt or boost your model's precision, as a tweak to a hyper parameter is similarly there's often many transformation steps that come after the modeling. So if you're tweaking a hyper parameter in your model, often you're curious about more than just the effect of that change.

A

On your model's precision, um for example, if you're using the model to decide who gets a loan, you might want to know how it's going to affect the total number of loans that you make um and determining that might take many steps and data transformations after the step that trains the model itself.

A

So a change anywhere in your dag can affect anything that's downstream of it. um It might hurt or help performance um it might simply make it so that some step can't complete at all if it um changes a column or breaks a table or something like that. um So if you separate how you manage your modeling steps from how you manage your data transformation steps, it becomes very difficult to answer these what-if questions. You really want to be easy to experiment across the entire chain, not just at the machine learning step on the chain.

A

The second important question is: where did this come from so anytime you're trying to understand the behavior of a data set or ml model? You want to know the process that created that object.

A

For example, if you notice your website is giving funny predictions, and you want to debug why it's doing that the problem could stem from something as early as a bad record and a base table to figure that out. You need to find what version of your model produced the predictions.

A

What features were used to train that model and what data was used to generate that features when that data was ingested so again to understand this question um effectively, you really need to understand the entire dag, not just the transformation phase and the modeling phase zooming out a little bit. These questions correspond to stages in a fuller data development lifecycle.

A

B

A

Wants to uh that's weird: oh there we go. um Anybody who wants to use a machine learning model needs to do these three things they need to develop and test it to get a working model in the first place. They need to deploy and execute it to keep it updated and actually generate predictions, and they need to monitor and observe it to catch problems and find the sources of those problems.

A

Each of these lifecycle stages corresponds to a key question that the dag is crucial for answering.

A

So when we're working on changes to our model, we want to understand what will happen if we make a change we're working on keeping our model updated, we need to know what order we actually should execute steps in when we're monitoring and debugging our model. We need to understand how that model was generated in the first place.

A

What tooling, is there to support this lifecycle? um So there's traditional orchestrators like airflow and luigi, but they focus exclusively on the deploy and execute stage.

A

It's typical for developers to combine a set of ingestion transformation, machine learning steps into an airflow dag. However, this deck only gets used to order the computation, so it's not available in developing a pipeline or when monitoring an asset. That's produced by that pipeline.

A

Dexter is an organization, so you define a single dag and you can use that dag to answer these key questions at all three of the stages. um So this is the same dag uh the dagger you use to determine what runs first can also help you um uh locally develop understand what happens if something changes as well as help you understand um where your data comes from in production, so I'm going to spend the rest of this talk going through each of these stages in detail.

A

What does it look like concretely to develop and test using a dag? What does it look like to monitor the output of a data pipeline? Make it a little bit more concrete, we'll use a particular dag, and so this is a pipeline that builds a recommendation model for hacker news stories.

A

For those of you who aren't familiar hacker news is a social news website where people can post stories and other people can comment on them popular in the tech world.

A

Hacker news provides public data, sets for all their stories and comments.

A

Based on the stories that the user has commented on the past, we can make guesses about which stories will be relevant to them in the future. So what this pipeline does? Is it aggregates the story and comment data? It uses it to build a matrix which has been the training set for a recommendation model.

A

uh Then it takes that recommendation model and does two things one. It builds a table of recommendations that um we could theoretically use to serve recommendations on the site um and two it reports on the model uh itself and helps us understand what the model is doing.

A

So, let's dive into the first phase of the data lifecycle, um development and testing development in machine learning usually starts with a question, and the question you know is often something like what, if we change, this hyper parameter value. What if we change the set of features in our model? um What if we change the data sources, that our features are derived from on any input to a model, training process and anything that input depends on can affect the ultimate model that comes out, um and so you know, once we've made a change.

A

We also want to understand the impact of that change. How does this change affect the performance of our model? um How does it affect the performance of the things we care about that are downstream of our model? You know the ultimate business metrics that we're trying to drive by building this model in the first place, um and then.

B

You know, I think more.

A

Prosaically will our model run it all.

B

A

There we go um so you can catch problems at different points in your process.

A

um Some errors get caught during local development when you're, iterating and coding your laptop other errors, get caught in production when a pipeline breaks or someone notices a metric as dipped and calls you and gets mad at you and, of course, there's uh sort of intermediate stages as well like ci staging this little plot shows percent of errors caught versus versus development stage.

A

um I would posit that catching regressions in most data systems look kind of like this, meaning that reactions are caught in staging and production a lot more than in development in testing.

A

This is a big problem for a couple reasons. First of all, it slows down development. So if you need to push a change to production every time, you want to try out that change. It's going to take you a long time to get that change right, because the iteration cycle is really big.

A

Second of all, it hurts your users because they actually need to deal with all the bugs and regressions that you push a much better place to be is here. That is, we want to detect problems as early as possible. We can't detect every problem. um You know in a unit test, but many of them can be um uh so you know. Eventually, we need more realistic environments if we want to catch problems with actually performance like our how long it takes from all to execute or data quality.

A

um There's a lot of opportunity to push these errors earlier and ultimately sort of uh increase our development time and reduce the burden on our users.

A

Okay, so enough diagrams, let's get to some code. This is what it looks like to define a basic pipeline indexer, the code that we're looking at on the top of the screen defines a pipeline that has the structure we're seeing on the bottom. um It looks a lot like we're just running code here, but what's actually happening is that we're building up a pipeline definition? um The pipeline definition is explaining how data flows not just the dependencies between tasks, so we actually have these variables um like comment stories.

A

Recommender model that represent the output of particular uh particular pipeline stages, um we'll cover how that works. In a little bit more detail later, once we've defined the pipeline, we can directly execute this pipeline in a few different ways.

A

So, first of all, we can execute it via python. This is especially useful for unit testing um or running our pipeline from inside a jupiter notebook. We can also execute it from the command line. This is useful for local development loops, um especially if you like to work out of a terminal um and last of all, we can launch it from inside the dag's web ui um web ui is called daggett and that's useful both for local development and for launching pipelines manually in production, which you sometimes need to do.

A

So how do we define one of the nodes inside this pipeline?

A

It's basically a simple python function, so we refer to a node of computation in our graph or in our pipeline as a solid here's, a solid that actually trains, the recommendation model we construct it by taking a python function and annotating it with this solid decorator.

A

There are a few things to notice here. The body of the function is an arbitrary, arbitrary python compute and you can do anything in here- um invoke any tool in this case we're calling the truncated svd from scikit-learn.

A

Second, the solid is structured with data dependencies. That means that it declares a set of inputs which are the arguments of the function and declares an output, which is what the function returns and, third of all, we believe in cleanly separating io and compute. So the output of this transform is a data frame. It's not a file I'll get into uh how that works a little more later.

A

So, let's load this into the daggett ui for viewing um daggett is the extras web ui. um All you need to do to actually run dagger is to type tag it on the command line um and it'll pick up pipelines that are in the surrounding directory.

A

um So if we take a look at daggett, here's a representation of the pipeline that we defined in code a couple slides ago, you can see that every piece of metadata is exposed in the tooling. So for each solid you can see its inputs, its outputs, as well as the python types of those inputs and outputs. So the goal is for the graph to be sort of a rich self-describing unit of documentation.

A

If we want to launch our pipeline, we can go here and kick off a run.

A

That'll take a second.

A

As the pipeline runs, we can see logs and events stream in. We also have this gantt chart up here. That shows the progression of the execution.

A

um So you can see you know here are events. We have filtering blah blah blah.

B

Oh, that's: uh filter.

A

So here's what a completed execution of the pipeline looks like um uh imagine. I just made a change to my code that alters how I built the matrix they used to train the model. um Maybe I want to know whether that will affect my model performance. I can select just those steps, um so that's what I've done here.

A

Let's see those two steps are the steps that build the user story matrix as well as the step that builds the recommended render model, and I can re-execute just those steps instead of the entire pipeline, which allows me to get into sort of a type dev loop, where I make changes um to one part of the pipeline, um execute the subset of steps to the particular other part of the pipeline, where I'm trying to understand the impact of those changes and rinse and repeat until I get the outcome that I'm looking for.

B

um So let's go back here.

A

As I alluded to briefly before, you can also execute pipelines locally. It's just a function call, um it invokes the entire thing and you can examine the results of the computation in process, so here's an example of executing our pipeline from inside a unit test. It just requires this simple invocation.

A

Something interesting is happening inside this unit test that I want to point out um when we execute our pipeline locally. We pass in this argument called mode. So, what's a mode to back up a little bit, it's likely that when we run our pipeline in production, we want to store our tables in a data. Warehouse store, a model in some kind of model, store, um etc.

A

We want to use our pipeline in a unit test, though all those external services kind of become a liability.

A

Ideally we'd like to be able to test the business logic in our pipeline without needing to stand up snowflake without needing to stand up, our model store, etc. So this is where the concept of dexter modes becomes very.

A

Handy, the idea is that you can set up separate modes corresponding to separate environments, so each mode supplies a set of resources to the pipeline that represent the services that live inside that environment.

A

The business logic in the pipeline stays the same across all the environments, but the resources change.

A

Here's a demonstration of what it looks like to vary how data is stored between different modes. So what we're looking at now is a solid from the hacker news pipeline that we talked about earlier, and this solid produces the final table of recommendations that are used for the website in our production mode. We'd like to be able to write the table to snowflake in our local mode, we'd like to be able to write it to our local file system, where we can inspect it. So.

B

By declaring an I o manager, key.

A

When we define our solid output, we're saying that we want that output to be handled by whatever python object is supplied for that key um in the mode we use when we launch the pipeline. So if we use the um uh as we look over here, we have two modes: we have a development mode and a production mode in the development mode.

A

We supply a file system, io manager um for the I o manager, key, which is warehouse um and that will store results in a local uh on the local file system and our you know. When we're in development, we can use this development mode in the production mode, we're providing an I o manager resource that stores outputs in snowflake, and when we launch our pipeline in production, um we can use the production mode um that writes to snowflake here's our pipeline definition down here.

B

Which? Basically.

A

Just bakes in these two modes as the two options for executing the pipeline, the resource itself is just a python object with a function for handling the solids output. um So you can see this handle output function. um I haven't included the entire thing here, but it basically takes the um the output that's produced by the solid. In this case a data frame um and writes it out to snowflake so modes and resources are a set of abstractions that make it possible to structure your code for testability.

A

That was a little window into what it looks like to use dexter for involvement testing. What about running pipelines in production dexter is a scheduler that can launch pipelines at regular intervals. It also has a full ui for monitoring those schedules. Re-Executing steps within a pipeline and diving into their logs in a more productive environment, as I mentioned before, running pipelines in production is the part of the data lifecycle that most other orchestrators actually do pretty well.

A

So I'm going to cover this part pretty quickly, because it's less than novel here's what it looks like to define a schedule for a pipeline in dagstar. We have this convenience api called daily schedule. We have similar convenience apis for weekly, daily hourly schedules.

A

You can also use chronic expressions or insert any kind of customized logic for launching your schedules.

A

You can then monitor a schedule from inside daggett. So here's the timeline view- and it shows you um when each of the scheduling takes occurred and when runs, were requested and failed as well as um links to the sort of like particular runs of all your previous pipelines. Related to schedules are a dagger concept called sensors. A sensor like a schedule is something that um is responsible for launching a pipeline, um but it's a little bit more flexible.

A

So a sensor is basically a function that gets called inside of a loop and when it detects that something has changed, it can instigate a pipeline run. For example, this sensor is listening for new objects inside an s3 bucket when it notices a new one. It launches a pipeline run to process that particular object, link schedule. Sensors can be monitored inside daggett.

A

So now we have our pipeline up and running, and it's regularly training our ml model and producing a table of recommendations. Imagine we start getting angry emails from users telling us that our recommendations are total garbage debugging. This problem is tough because the pipeline, the problem, could originate from anywhere in our dag. um You know it could be.

B

A problem with.

A

The model, but it could also be some um bug in our code that derives the features for the model, or maybe some bad data that came in um uh at the base of our pipeline.

A

The fundamental question you have is: where did this data come from? um So we have our recommendation data. We want to understand what code was used to generate it. What data was used to generate it um and these sort of interleave all the way back? So what code was used to generate this data? What data was used when we ran this code?

A

Something to note here is that, while answering this question is especially useful for production debugging, it's also very helpful for just getting a handle on. What's going on with your data from dagster's perspective, this data that you're trying to learn about is called an asset.

A

An asset is our term for anything that is produced by a computation but outlives the scope of that computation. So, for example, it could be a table in a data warehouse, a par k, file or a data lake, um or you know, maybe a pickled machine learning model at the end of the day. Assets are why we build these pipelines in the first place um from the perspective of another team in your organization. They might know about the table or model that your pipeline has produced but not know about the pipeline itself.

A

You know they might care about the asset, but not even care about the pipeline. They might understand they might want to understand when that asset was updated, how it changed over time, maybe even launch their pipeline whenever the asset changes.

A

um So what you essentially want is the ability to um to publish your assets, to publish um your machine learning model, to publish your um table of comments to publish your uh your recommendations so that people who care about those tables can understand what's going on with them.

A

Dexter allows you to do this um through some like fairly simple function. Calls we've looked at this class definition briefly before, and it's the resource that we use for storing our solid outputs in snowflake.

A

We can add a small function to it that basically gives the name of the asset that we're storing after we do this anytime, that this code is invoked to store a model or rather a table in snowflake. It will also tell dagster that it's done that and and record this asset.

A

We can then record metadata that gets attached to the asset. So, for example, um we decide to um store you know one of our tables in snowflake. We might actually want to know at the time we started. How big was that table? um What were the columns in that table? So we can then go in later and understand um how these change over time.

A

So now that we've recorded all this metadata, what can we do with it? um The first thing is that it shows up in dagger, so this here is our daget asset catalog. It shows the list of all the assets in our system. Each of these was produced by um by one of our pipelines.

A

In this case, um remember, we were trying to debug something that was wrong with the recommendations table. So let's go to the asset that corresponds to that recommendation. Table.

B

There are a few things we see here so.

A

First, we have a link to the most recent pipeline run that actually was responsible for uh regenerating this table. Most recently, we can easily navigate to logs, for that run, to see if anything was fishy, so um you can do that here. This takes us back to this log view and helps us understand what was going on inside that pipeline run.

A

um Second, we have structured metadata that we've recorded about the table, so in this case we've logged the number of rows in the table each time we run it, and we've also recorded the columns of the data frame that are written to the table. We don't.

B

Have a ton of data here.

A

You can actually see how the number of rows changes over time if we have a set of scheduled, runs.

A

um Third, we have the upstream assets that were actually used to generate the table, so in this case the table was generated using the recommender model, as well as the matrix of users and stories. You can see that over here, where um there's this latest parent asset keys section.

A

If we want to learn about the recommender model, we can click into it and get taken to that asset page. um So oops, that's the wrong.

B

B

A

Click into the recommender.

B

Model so again, this.

A

Shows us the um the recommender model itself. We have a set of different things. We care about here in this case we're actually looking at um metrics that correspond to the performance of the model, um and we can keep going back. We can see um the transformation that was responsible for generating this model. uh We can see the parents of this model itself, in this case the user's story matrix and basically we can go back and back and back until the beginning of any sort of data that we've recorded.

A

One thing: that's cool about asset lineage is that I can actually span across pipelines, so a different pipeline might be responsible for creating this table, we're still able to track the lineage.

A

uh If we want to know how an asset was generated, that a particular pipeline relies on, lineage can actually help us find the upstream pipeline that generated the asset.

B

So let's go back over here.

A

So the second main use for that asset metadata that we're publishing is building sensors off of it. As we talked about before you can write, sensors that launch pipeline runs whenever an event happens. A common type of sensor to write is an asset sensor um which launches a pipeline run whenever a particular asset proceeds receives an update.

A

um This is particularly useful for across team dependencies. um The interfaces between teams become these assets so kind of similar to how, um in a more um web-based environment, the interfaces between teams become microservices in the world of data and dags interfaces between teams are data, assets um and asset sensors. Allow you to basically listen on an asset, that's produced by an upstream team and take actions when that asset changes.

A

um This is a quote from one of our users. um You can basically agnostic to the upstream pipeline, that's generating the data that you depend on. You just need to know about the data itself.

A

So in conclusion, um there we go um machine learning is all about dags um um dexter embraces dags across the entire uh machine learning and data development life cycle, not just the deployment and execute phase, um so the dag is relevant in the world of development and testing, because you want to understand, what's going to happen, if you make a change and catch errors before you deploy them to production, the dag is useful, as we've seen from existing orchestrators um for deploying executing and then the dag is useful um for monitoring and observing, because it helps us understand the provenance of our data.

A

So um if you want to learn more, here's our github, um here's our docs, um join us in slack. um This talk ran a little bit under, but I'd be happy to answer any questions. You have thanks a.

A

A