Dagster Dagster Hot Takes, 30 Dec 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Dagster Hot Takes: Retries and Alerts

Description

Getting an on-call page is the worst. Unfortunately most task-based orchestrators page teams frequently, whenever jobs fail. With Dagster you can reduce this alert fatigue by using retry strategies and only getting notified when SLAs are violated.

Resources:
- *Star Us*: https://github.com/dagster-io/dagster
- *Example*: https://github.com/dagster-io/hooli-data-eng-pipelines

A

Hey everyone: it's Sean here with another dagster hot take today, I want to talk about on-call Pages they're, the worst, especially over the holidays and, unfortunately, I think for a lot of data engineering teams. The orchestrator is actually causing too many interruptions and on-call pages.

A

So today, I want to look at how dagster is built to be robust so that, even if you have flaky Upstream data sources like API aestheticationally return errors as long as things are recoverable. The orchestrator, in this case diagster, can actually move on with its life without interrupting or paging on call and only alert you when there's actual violations of the slas that your stakeholders care about. So let's go ahead and look at a asset graph.

A

Here we have a set of assets that start by querying an API for some user data in some orders data and then a number of DBT models live Downstream of those two AP Raw data sets that are processed and then further Downstream. We have some forecasting assets that maybe represent ml models, as well as some marketing assets that maybe represent in different reports.

A

So, let's look at the timeline for these assets. We can see that there's definitely some flakiness here. Typically, jobs are running every hour, but sometimes those jobs fail and if we go and look at one of these runs say the one that occurred at 12. I want to talk about some of the ways that Daxter helps. You be robust to these types of failures. So here we have two.

A

Those two assets we're looking at are Upstream call to an API to get orders information an upstream call to the API to get users information and so in dagster we can set up various retry strategies that these two assets demonstrate.

A

So the first is our orders asset, so this asset is calling out to an API and retrieving some information about orders, and you can see that in this run we actually saw this API timeout twice, so we added timeout and then a subsequent timeout, then finally a success, and so what dagster is doing here is it's retrying at the asset level, so we've specified that our orders asset should have a retry policy and dagster is automatically implementing that policy and re-attempting to to get that order's information.

A

Until it's successful our second asset, though users we can see failed in a different way. The API timed out again, but in this case we haven't added a retry policy to users, and so instead, what's happening is we're falling back on dagster's global run, retry policy.

A

So if we look at the very end here, it's saying that our users step failed, and so, as a result, this entire run is going to fail, but dagster is going to prepare a rerun. So it's going to try a new run.

A

If we go look at this new run, we can see that diagster was actually smart enough to know that the order step from earlier was successful, and so it only needed to re-execute those steps from our original run that had failed, and this is because dagster is aware of the fact that we're working within a dag and so it's able to do partial re-execution.

A

So as a result, if you, if you our final run, we have a success of our DBT processing. Job I was able to use the orders information that had been retried at the asset level and then this users table that had been retried at the global job level.

A

So as a result uh where we land is that we're able to successfully run our model and avoid getting undue Pages now that was at 12 o'clock. If we look at our runs table, we can see that uh subsequently, we've had some failures, and maybe these are failures now that are actually impacting the team, so as opposed to 12 o'clock, where we were able to use our various retry strategies to succeed. Now we have runs that have failed and so what's happening. Are we getting an on-call page?

A

Well, if we look at our Global asset View, we can see that Daxter has tracked the fact that some of our runs have now failed, but we're not getting on call Pages yet because in dagster we've specified whether or not an asset is on time based on its SLA and so, for example, daily order summary here is a data set that only needs to be available by 9 am every day, and so, even though this last run failed and wasn't able to recover through our various retry strategies, the team is still not going to get paged about daily order summary because it's not violating the SLA that stakeholders actually care about yet and it's possible that sometime between now and 9 A.M the system will recover and our API will return successfully and we'll be back in a happy state.

A

In contrast, if we look at average orders, this is a data set that has a more strict SLA. This is saying that at any point in time the asset should be up to date within the last 90 minutes of data, and so right now the team hasn't gotten paged.

A

Yet because it's one o'clock and the data was last updated at 12 30., so we haven't violated the SLA yet, but we're closer to violating this SLA and if that were to happen, What dagster would do is send me an alert notification that looks like this, so the team can basically worry only about data sets that are actually violating their SLA and not get paged based on spurious errors that are either recoverable automatically within the orchestrator through our various retry policies, the asset or run level or might just recover throughout the course of the day.

A

So overall I hope this is interesting and exciting. For you, I'd love to see your team stop getting paged I, don't want anyone's holidays to be interrupted. So if this looks interesting, star, dagster on GitHub or join our slack community and send me a message thanks so much.