9 Aug 2023
In this month's Dagster Community Call, Rex Ledesma provided an overview of the new dbt™ integration and Georg Heiler shared his dbt™ use case: Unlocking Advanced Metadata Extraction with the New dbt™ API.
- 3 participants
- 28 minutes
16 Jun 2023
Arun Kumar shares how the team at DoorDash uses Dagster to power metrics layer and scale experimentation.
- 2 participants
- 31 minutes
8 Feb 2023
whatnot is the fastest-growing e-commerce marketplace in the USA.
It specializes in livestream auctions for collectors and enthusiasts.
Stephen Bailey led whatnot’s migration from Airflow to Dagster to boost the Data Engineering team’s productivity. Starting with 50 Airflow DAGs running on mostly-daily cadences, the team now runs one hundred thousand individual Dagster jobs a month on myriad schedules.
Here is his advice to other data practitioners.
It specializes in livestream auctions for collectors and enthusiasts.
Stephen Bailey led whatnot’s migration from Airflow to Dagster to boost the Data Engineering team’s productivity. Starting with 50 Airflow DAGs running on mostly-daily cadences, the team now runs one hundred thousand individual Dagster jobs a month on myriad schedules.
Here is his advice to other data practitioners.
- 1 participant
- 6 minutes
8 Dec 2022
In his opening remarks for Dagster Community Day, incoming CEO Pete Hunt provides a recap of 2022 progress, covers HIPAA and SOC 2 Type II, implementation partners, community growth, user growth, and Elementl Staff changes. The session wrapped up with a quick introduction by Bosmat Eldar, Elementl's new Head of Engineering.
- 2 participants
- 9 minutes
11 Oct 2022
In our fourteeth Dagster Community Meeting, the Dagster team showed new updates and assets.
👨🏫 Agenda 👩🏫
Introduction/Announcements: 0:00
Community Spotlight: 4:18
Re-introducing Dagster-Airflow: 7:17
Asset updates - Loading asset values: 18:05
Asset updates - Multi-asset sensors and asset reconciliation: 20:35
Dagit updates: 25:40
Orchestrating Airbyte connections with Dagster: 35:11
Q&A: 40:27
🗞 Related blog posts 🗞
Dagster 0.15.0: Cool for the Summer: https://dagster.io/blog/dagster-0-15-0-cool-for-the-summer
Orchestrating python and dbt with Dagster: https://dagster.io/blog/dagster-0-15-0-dbt-python
🌟 Socials 🌟
Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Agenda 👩🏫
Introduction/Announcements: 0:00
Community Spotlight: 4:18
Re-introducing Dagster-Airflow: 7:17
Asset updates - Loading asset values: 18:05
Asset updates - Multi-asset sensors and asset reconciliation: 20:35
Dagit updates: 25:40
Orchestrating Airbyte connections with Dagster: 35:11
Q&A: 40:27
🗞 Related blog posts 🗞
Dagster 0.15.0: Cool for the Summer: https://dagster.io/blog/dagster-0-15-0-cool-for-the-summer
Orchestrating python and dbt with Dagster: https://dagster.io/blog/dagster-0-15-0-dbt-python
🌟 Socials 🌟
Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 7 participants
- 43 minutes
21 Jun 2022
In our thirteenth Dagster Community Meeting, the Dagster team showed new 0.15.0 and Dagster Cloud updates.
👨🏫 Agenda 👩🏫
Introduction: 0:00
Data Engineering with Dagster Course: 1:45
Community Spotlight: 3:00
Software-defined assets update: 4:29
Ops and Graphs with Software-Defined Assets: 13:13
Orchestrating dbt and python using Dagster: 26:13
Dagit Improvements: 38:36
Dagster Cloud Update: 48:03
Q&A: 52:41
🗞 Related blog posts 🗞
Dagster 0.15.0: Cool for the Summer: https://dagster.io/blog/dagster-0-15-0-cool-for-the-summer
Orchestrating python and dbt with Dagster: https://dagster.io/blog/dagster-0-15-0-dbt-python
🌟 Socials 🌟
Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Agenda 👩🏫
Introduction: 0:00
Data Engineering with Dagster Course: 1:45
Community Spotlight: 3:00
Software-defined assets update: 4:29
Ops and Graphs with Software-Defined Assets: 13:13
Orchestrating dbt and python using Dagster: 26:13
Dagit Improvements: 38:36
Dagster Cloud Update: 48:03
Q&A: 52:41
🗞 Related blog posts 🗞
Dagster 0.15.0: Cool for the Summer: https://dagster.io/blog/dagster-0-15-0-cool-for-the-summer
Orchestrating python and dbt with Dagster: https://dagster.io/blog/dagster-0-15-0-dbt-python
🌟 Socials 🌟
Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 7 participants
- 56 minutes
12 Apr 2022
In our twelfth Dagster Community Meeting, the Dagster team provided some hiring, community, and Github discussion and 0.15.0 updates. Daniel Blinick—Software Engineer at Immunai—presented on Immunai’s use of Dagster to tackle bio-tech data engineering challenges. We also heard from Marcos Alcozer—Analytics Engineer at Alcozer Consulting LLC—who presented on how K12 education leverages analytics engineering using Dagster.
👨🏫 Agenda 👩🏫
Introduction: 0:00
Github Discussions Demo: 3:36
0.15.0: 6:10
Dagster at Immunai - Daniel Blinick: 8:16
Analytics Engineering | K12 Education: 27:11
Q&A: 40:10
🌟 Socials 🌟
Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Agenda 👩🏫
Introduction: 0:00
Github Discussions Demo: 3:36
0.15.0: 6:10
Dagster at Immunai - Daniel Blinick: 8:16
Analytics Engineering | K12 Education: 27:11
Q&A: 40:10
🌟 Socials 🌟
Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 7 participants
- 48 minutes
8 Feb 2022
In our eleventh Dagster Community Meeting, we are happy to introduce Mollie Pettit, who is our new Developer Relations Lead at Elementl! We also hear from David Laing, a Staff Data Engineer at VMware on monitoring-driven asset development. Owen from the Elementl team covered recent work on software-defined assets and dbt.
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
0.14.0 Update: 1:55
Monitoring-driven Asset Development - David Laing from VMware: 6:08
Software-Defined Assets & the Modern Data Stack: 29:38
Q&A: 44:42
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
0.14.0 Update: 1:55
Monitoring-driven Asset Development - David Laing from VMware: 6:08
Software-Defined Assets & the Modern Data Stack: 29:38
Q&A: 44:42
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 6 participants
- 55 minutes
14 Sep 2021
In our ninth Dagster Community Meeting, we heard from Alessandro Marrella at Earnest Research on using Dagster to power their machine learning pipelines. Owen and Max from the Elementl team presented our Dagster logging improvements and on our Dagster Cloud Private Beta.
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Alessandro Marella at Earnest Research: 7:26
Dagster Logging Improvements: 28:50
Dagster Cloud Private Beta: 47:07
Q&A: 49:22
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Alessandro Marella at Earnest Research: 7:26
Dagster Logging Improvements: 28:50
Dagster Cloud Private Beta: 47:07
Q&A: 49:22
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 6 participants
- 50 minutes
13 Jul 2021
In our eighth Dagster Community Meeting, we heard from Carlson Cheng at Thinking Machines on running Dagster for ML pipelines in production. With our new 0.12.0 release, our team presented our new features and the road to 1.0.
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Carlson Cheng at Thinking Machines: 1:26
0.12.0 and Road to 1.0: 19:37
Q&A: 43:08
Special Announcement 45:11
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Carlson Cheng at Thinking Machines: 1:26
0.12.0 and Road to 1.0: 19:37
Q&A: 43:08
Special Announcement 45:11
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 5 participants
- 49 minutes
11 May 2021
In our seventh Dagster Community Meeting, Simon Späti from Rohde & Schwarz MNT discusses the production Dagster setup at R&S, and the core team presents our recent API "approachability" improvements, as well as upcoming 0.12.0 features.
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Dagster at Rohde & Schwarz MNT: 1:26
Dagster Approachability Improvements: 26:40
Upcoming Dagster 0.12.0 Features: 35:30
Q&A: 39:25
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Dagster at Rohde & Schwarz MNT: 1:26
Dagster Approachability Improvements: 26:40
Upcoming Dagster 0.12.0 Features: 35:30
Q&A: 39:25
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 4 participants
- 51 minutes
9 Mar 2021
In our sixth Dagster Community Meeting, the core team presents upcoming 0.11.0 features, including Dynamic Orchestration, a system to establish Asset Lineage, and a new documentation site.
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Dynamic Orchestration and the new `collect` step: 1:48
Dynamic Orchestration Demo: 6:40
Dynamic Orchestration tl;dr: 10:05
An introduction to Asset and Asset Lineage: 11:05
An example pipeline producing assets: 14:11
Inferring Asset Lineage from the Execution Graph: 20:40
Dagster Features using Asset Lineage: 22:31
Dagster Asset Lineage tl;dr: 24:23
New Backfill and Partition Features Demo: 26:03
New documentation infrastructure and design: 33:31
Q&A: 36:56
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
👨🏫 Today's Agenda 👩🏫
Introduction: 0:00
Dynamic Orchestration and the new `collect` step: 1:48
Dynamic Orchestration Demo: 6:40
Dynamic Orchestration tl;dr: 10:05
An introduction to Asset and Asset Lineage: 11:05
An example pipeline producing assets: 14:11
Inferring Asset Lineage from the Execution Graph: 20:40
Dagster Features using Asset Lineage: 22:31
Dagster Asset Lineage tl;dr: 24:23
New Backfill and Partition Features Demo: 26:03
New documentation infrastructure and design: 33:31
Q&A: 36:56
🌟 Socials 🌟
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
- 6 participants
- 47 minutes
9 Feb 2021
In our fifth Dagster Community Meeting, Dennis Hume from Drizly and Noah Kantrowitz from Geomagical Labs discuss about their production Dagster setups.
Register for the next community meeting ➡️ http://bit.ly/march-cm
👨🏫 Today's Agenda 👩🏫
00:00 - Introduction
01:02 - Dennis Hume demonstrates how Drizly configures its Dagster deployments to support a heterogeneous team.
16:32 - Noah Kantrowitz discusses Geomagical’s approach to building customer-facing applications.
🎞 Slides 🎞
Drizly & Dagster (Dennis Hume) ➡️ https://drive.google.com/file/d/1sF_eIjwzVutPxjUwtayObgYgkUoquvE4/view?usp=sharing
Geomagical Labs & Dagster (Noah Kantrowitz) ➡️ https://drive.google.com/file/d/1sFBrqX8qQEgOKGTKxCqMBwDbdPdPWkHd/view?usp=sharing
🌟 Socials 🌟
Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/
Register for the next community meeting ➡️ http://bit.ly/march-cm
👨🏫 Today's Agenda 👩🏫
00:00 - Introduction
01:02 - Dennis Hume demonstrates how Drizly configures its Dagster deployments to support a heterogeneous team.
16:32 - Noah Kantrowitz discusses Geomagical’s approach to building customer-facing applications.
🎞 Slides 🎞
Drizly & Dagster (Dennis Hume) ➡️ https://drive.google.com/file/d/1sF_eIjwzVutPxjUwtayObgYgkUoquvE4/view?usp=sharing
Geomagical Labs & Dagster (Noah Kantrowitz) ➡️ https://drive.google.com/file/d/1sFBrqX8qQEgOKGTKxCqMBwDbdPdPWkHd/view?usp=sharing
🌟 Socials 🌟
Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/
- 3 participants
- 29 minutes
8 Nov 2020
Register for the next community meeting: https://bit.ly/38snrNr
00:00 - Introduction
01:40 - Tobias Macy on running Dagster in Production at MIT Open Learning
20:52 - Dmitry Matasov on building a platform for data analysts using Dagster
35:47 - Daniel Gibson on new features coming out in Dagster 0.10.0
46:35 - Summary by Nick Schrock
49:36 - Note about Python 2 Support: https://github.com/dagster-io/dagster/discussions/3208
50:04 - Questions
Slides:
- Tobias Macey: https://docs.google.com/presentation/d/1TKL9kem6SDyPr0MADOQIRqwvFgHOF7_gJ9Hqdubiuhs/edit
- Dmitry Matasov: https://drive.google.com/file/d/1BSaQmSc9szcKTT16-B_HzwPIYUKuxe81/view?usp=sharing
- Daniel Gibson: https://drive.google.com/file/d/1ixj8sKMszTkPwgZ1TLG5mq_iZB9CoV8M/view
00:00 - Introduction
01:40 - Tobias Macy on running Dagster in Production at MIT Open Learning
20:52 - Dmitry Matasov on building a platform for data analysts using Dagster
35:47 - Daniel Gibson on new features coming out in Dagster 0.10.0
46:35 - Summary by Nick Schrock
49:36 - Note about Python 2 Support: https://github.com/dagster-io/dagster/discussions/3208
50:04 - Questions
Slides:
- Tobias Macey: https://docs.google.com/presentation/d/1TKL9kem6SDyPr0MADOQIRqwvFgHOF7_gJ9Hqdubiuhs/edit
- Dmitry Matasov: https://drive.google.com/file/d/1BSaQmSc9szcKTT16-B_HzwPIYUKuxe81/view?usp=sharing
- Daniel Gibson: https://drive.google.com/file/d/1ixj8sKMszTkPwgZ1TLG5mq_iZB9CoV8M/view
- 5 participants
- 53 minutes
8 Sep 2020
Slides:
0.9.0 Slides: https://docs.google.com/presentation/d/17ts4HfRUEqLbBF52DZ66G1QHI1PutK2MnY_cnlTWbug/edit#slide=id.g9380d0b962_0_0
0.10.0 Slides: https://docs.google.com/presentation/d/1qwV2i_4wp-72HsCeQza1ZEP02QPoz8KLIC4czcNzw5s/edit#slide=id.g94347a5a1e_0_14
Prezi Slides: https://prezi.com/view/kveaLi8KasReSs4pyP5l/
0.9.0 Slides: https://docs.google.com/presentation/d/17ts4HfRUEqLbBF52DZ66G1QHI1PutK2MnY_cnlTWbug/edit#slide=id.g9380d0b962_0_0
0.10.0 Slides: https://docs.google.com/presentation/d/1qwV2i_4wp-72HsCeQza1ZEP02QPoz8KLIC4czcNzw5s/edit#slide=id.g94347a5a1e_0_14
Prezi Slides: https://prezi.com/view/kveaLi8KasReSs4pyP5l/
- 5 participants
- 56 minutes
16 Jun 2020
This is a kickoff meeting for the Dagster community announcing Dagster 0.8.0. This is the biggest release for the project since inception.
- 4 participants
- 56 minutes