Dagster / Dagster Community Demos

In this month's Dagster Community Call, Rex Ledesma provided an overview of the new dbt™ integration and Georg Heiler shared his dbt™ use case: Unlocking Advanced Metadata Extraction with the New dbt™ API.

Arun Kumar shares how the team at DoorDash uses Dagster to power metrics layer and scale experimentation.

Group 1001 is a technology-driven financial services company that provides insurance and annuity products.

Group 1001 migrated 73 Airflow DAGs within 2 weeks of evaluating Dagster with just 2 engineers thanks to the dagster-airflow library. In doing so, they achieved much faster development velocity to help meet the organization's data requirements.

Gu Xie, Head of Data Engineering at Group 1001 led the effort.

Marcos Alcozer—Analytics Engineer at Alcozer Consulting LLC—who presented on how K12 education leverages analytics engineering using Dagster

🌟 Socials 🌟

Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio

Daniel Blinick—Software Engineer at Immunai—presented on Immunai’s use of Dagster to tackle bio-tech data engineering challenges.

🌟 Socials 🌟

Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio

As organizations grow so does the complexity and number of tools. In small data teams it is not uncommon to be responsible for more systems than there are team members. Without a strong orchestration layer to hold these systems together, many workflows are held together with one-off jobs and shaky deployments.

Dagster is a modern framework for defining jobs that treats the data platform as an application. Jobs can be parameterized and iterated on locally before moving into the cloud and interacting with production services. Using Dagster as the foundation for a modern data platform ensures that all data stakeholders can easily interact with their data and build out specific pipelines to maximize value without having to worry about underlying infrastructure.

Presenter: Dennis Hume @Dutchie

David Laing—Staff Data Engineer at VMware—present on monitoring-driven asset development

See full Feb 8, 2022 Community Meeting here: https://www.youtube.com/watch?v=fYJBN6MAtbE

Alessandro Marrella—Staff Software Engineer at Earnest Research—discusses using Dagster to power their machine learning pipelines

See the full September 14, 2021 Community Meeting here: https://www.youtube.com/watch?v=oCakb_tB_dU&t=1643s

Carlson Cheng—Machine Learning Engineer at Thinking Machines—demos running Dagster for pipelines in production

See full July 13, 2021 Community Meeting here: https://www.youtube.com/watch?v=tjDnyE7Xcvo&t=1148s

Warning: This video is fairly out of date due to the rapid development of Dagster.

This is supposedly a quick introduction to the Dagster tooling for Python.
Dagster is useful for data engineering, and more generally any parallelizable expensive python script.

https://dagster.io/
https://airflow.apache.org/

Simon Späti—Lead Data Engineer at Rohde & Schwarz—discusses the production Dagster setup at Rohde and Schwarz Mobile Network Testing

See the full May 11, 2021 Community Meeting: https://www.youtube.com/watch?v=HRd6rEU33XM

#dataOps #dataengineering #dagster #dbt #bigQuery #SPARK
En este video vamos a hablar explorar los principios para el diseño de una plataforma de Data Engineering moderna y construiremos un pipeline de proceso de datos con dagster (https://dagster.io/) - un orquestador de pipelines de datos utilizando:
* BigQuery como DWH
* dbt como herramienta de transformación de datos SQL
* dataproc/pySpark para procesar datos a escala con SPARK
* Jupyter notebook, para explorar y visualizar el resultado
Código del video: https://github.com/velascoluis/dagster_gcp
Sígueme en:
👉twitter: @luisvelasco
👉medium: https://medium.com/@velascoluis
👉github: https://github.com/velascoluis

Dennis Hume, a staff data engineer at Drizly, demonstrates how Drizly utilizes the Dagster framework to build a data platform that diverse data practitioners enjoy using. Dennis covers how he leverages Dagster abstractions to setup local, staging, and production environments.


🎞 Slides 🎞

Drizly & Dagster (Dennis Hume) ➡️ https://drive.google.com/file/d/1sF_eIjwzVutPxjUwtayObgYgkUoquvE4/view?usp=sharing


🌟 Socials 🌟

Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/

Noah Kantrowitz, a principal site reliability engineer at Geomagical Labs (part of the IKEA family), discusses how he used Dagster to power a customer-facing applications. In this talk, Noah talks about their data orchestration system, cost management via scale-to-zero, and the Dagster features he uses.


🎞 Slides 🎞

Geomagical Labs & Dagster (Noah Kantrowitz) ➡️ https://drive.google.com/file/d/1sFBrqX8qQEgOKGTKxCqMBwDbdPdPWkHd/view?usp=sharing


🌟 Socials 🌟

Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/

Dmitry Matasov from Bestplace talks about the business problem the company solves, the evolution of their data platform, how they leverage Dagster in their workflows across Gitlab, Jupyter, even Google Sheets!

🎞 Slides 🎞
Bestplace & Dagster (Dmitry Matas)➡️ :
https://drive.google.com/file/d/1BSaQmSc9szcKTT16-B_HzwPIYUKuxe81/view?usp=sharing

🌟 Socials 🌟

Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/

Tobias Macey from Data Engineering Podcast discusses migrating from Cron job to Dagster including the resulting tech stack (Pulumi, Packer, SaltStack, Hashicorp Vault, Consul, Vdist/FPM) and the Dagster features he used (scheduler, resources, hooks, assets, etc).


🎞 Slides 🎞
MIT Open Learning & Dagster (Tobias Macey)➡️ :
https://docs.google.com/presentation/d/1TKL9kem6SDyPr0MADOQIRqwvFgHOF7_gJ9Hqdubiuhs/edit


🌟 Socials 🌟

Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/

Tamas Nemeth presents why and how Prezi migrated their production data pipelines into Dagster from a homegrown orchestration solution.


🎞 Slides 🎞
Prezi & Dagster (Tamas Nemeth)➡️ :
https://prezi.com/view/kveaLi8KasReSs4pyP5l/


🌟 Socials 🌟

Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/

Do you want to know if DAGSTER.IO, an open source python data pipeline orchestration library is the right ETL data pipeline orchestration tool for you? Then watch this video.

I decided to demo open source python data orchestration tool for scientists, dagster.io because I wanted to know, if it is worth to migrate my python data flows into DAG based data pipeline tool.

I was after a tool with in-build scheduler, a nice dashboard and a easy to install documentation. So is dagster.io superior to Apache Airflow or prefect.io? We will see. So far my impressions are positive.

Buy me a coffee
○ https://buymeacoffee.com/vietphananh

Also make sure you subscribe to my channel!

MY DEVELOPMENT STACK
○ Linode VPS - https://www.linode.com/?r=cb6b809841dbb52fd72839e010bc42e417b3ff81

VIDEO PRODUCTION GEAR USED:
○ My camera - Sony A6400 - https://amzn.to/3ikLQbc
○ My main lens - Sigma 16mm f1.4 - https://amzn.to/36xHKtZ
○ My B camera - Sony ZV-1 - https://amzn.to/3IoHxpV
○ My C camera - iPhone 12 Pro - https://amzn.to/3KU0l1W
○ My lav mic - Rode SmartLav+ - https://amzn.to/3Jvr05c
○ Joby GripTight tripod - https://amzn.to/3wnxLlx
○ 13’ MacBook Pro - https://amzn.to/3tlZNfr
○ Elgato Green Screen - https://amzn.to/3imGrAB
○ Elgato Cam Link 4K - https://amzn.to/36eOmNV
○ Neewer RGB Lights - https://amzn.to/3IkzSct

COME SAY HI!
Blog:
○ https://vphventures.com/
Instagram:
○ https://www.instagram.com/chillifunk/
Twitter:
○ https://twitter.com/vphventures

MY PRODUCTIVITY APP
Join TheHustle:
○ https://thehustle.app/

MY MUSIC CHANNEL
Good House Music:
○ https://www.youtube.com/channel/UC4Dv6nLAdVPWhmKSthMYMXQ


DISCLAIMER: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!