Dagster / Airflow Migration event

Many data engineering teams are frustrated with the limitations of Apache Airflow and are looking to switch to a more contemporary solution. While Dagster offers many advantages vs. Airflow, making a migration can seem like a daunting effort.

Luckily, with the release of recent tooling - namely the dagster-airflow library - making the switch from Airflow to Dagster can take just a few days, and a proof of concept can take just one afternoon.

Join the Elementl team along with data practitioners from Dagsters Open Source and Cloud userbase, including some of Dagster's implementation partners, as we discuss the best ways to make the transition from Airflow to Dagster.

Many data engineers are looking to get past the limitations of Apache Airflow, the incumbent in the data orchestration layer. Dagster proposes a new paradigm centered on Data Assets and the tools to support a full development lifecycle that radically boosts the productivity of data engineering teams.

Group 1001 is a technology-driven financial services company that provides insurance and annuity products.

Group 1001 migrated 73 Airflow DAGs within 2 weeks of evaluating Dagster with just 2 engineers thanks to the dagster-airflow library. In doing so, they achieved much faster development velocity to help meet the organization's data requirements.

Gu Xie, Head of Data Engineering at Group 1001 led the effort.

INFOSTRUX is a Dagster technology partner. They help organizations select and implement the best possible technology to meet their business goals. In this video, Nasko Grozdanov (Manager, Engineering) explains the business benefits of using Dagster over Airflow.

Many teams want to migrate off legacy orchestrators and onto newer best-of-breed orchestration platforms, but the idea can be daunting.

Well, it's now easier to make the switch. Odette Harary shares details on the new dagster-airflow library and perspectives on how to successfully migrate off Airflow by converting Airflow Ops to Dagster Jobs.

Nick Schrock, Founder and CTO at Elementl explains the fundamental challenge for data teams: they are mission-critical to modern organizations and yet face deep problems related to their ability to manage their data platforms and achieve high levels of productivity.

Dagster was designed to help engineers shift from being stuck in a 'scarcity mindset 'to being highly productive and being able to adopt an 'abundance mindset'.

whatnot is the fastest-growing e-commerce marketplace in the USA.
It specializes in livestream auctions for collectors and enthusiasts.
Stephen Bailey led whatnot’s migration from Airflow to Dagster to boost the Data Engineering team’s productivity. Starting with 50 Airflow DAGs running on mostly-daily cadences, the team now runs one hundred thousand individual Dagster jobs a month on myriad schedules.
Here is his advice to other data practitioners.