Dagster / Dagster Community Day - Dec 2022

In this series of demos, Elementl engineers Shalabh Chaturvedi and Johann Miller provide an overview of three recent improvements to Dagster Cloud Serverless:
1) Fast Deploys: By using prebuilt Docker images, Dagster Cloud now deploys changes radically faster than it used to by bypassing the image rebuild. Dependencies get managed seamlessly.
2) Secrets Managements: Users can now set environment variables straight in the Dagit UI in Dagster Serverless.
3) Non-isolated runs: By bypassing the isolation of runs, developers can speed up their development and testing cycles.

Dagster 1.1 introduces a system for scheduling data pipelines that allows you to escape writing workflows entirely. Instead, you directly specify how up-to-date you expect each asset to be, as well as how to determine whether source data has changed. Dagster then automatically schedules asset materializations to ensure that data arrives on time while avoiding unnecessary computation.

Rex Ledesma provides an overview of the newer Dagster Integrations, while Pedram Navid gives us a quick tour of the templates he used in building the Hex and the Hightouch integrations for Dagster.

In his opening remarks for Dagster Community Day, incoming CEO Pete Hunt provides a recap of 2022 progress, covers HIPAA and SOC 2 Type II, implementation partners, community growth, user growth, and Elementl Staff changes. The session wrapped up with a quick introduction by Bosmat Eldar, Elementl's new Head of Engineering.

In this session, Ben Pankow provides a demo of 'Data Ingestion as Code', and Nick Schrock explains the broader context of 'managing managed services' from within the context of the data orchestration layer.

In this demo, Elementl engineer Shalabh Chaturvedi provides an overview of a recent improvements to Dagster Cloud Serverless: Fast Deploys: By using prebuilt Docker images, Dagster Cloud now deploys changes radically faster than it used to by bypassing the image rebuild. Dependencies get managed seamlessly.

In this demo, Elementl engineer Johann Miller provide an overview of three recent improvements to Dagster Cloud Serverless: Non-isolated runs.
By bypassing the isolation of runs, developers can speed up their development and testing cycles.

In this demo, Elementl engineer Shalabh Chaturvedi provides an overview of a recent improvement to Dagster Cloud Serverless: Secrets Management. Dagster Cloud users can now set environment variables straight in the Dagit UI in Dagster Serverless.