Dagster Dagster Day 2022, 9 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Introducing Dagster Cloud

Description

A managed orchestration service built on top of Dagster open-source with two hosting options:
Serverless (we host everything) and Hybrid (we host the control plane, and you bring the compute).

Get started with Dagster Cloud here: https://dagster.io/cloud

For more information on getting started using Dagster, check out this demo: https://www.youtube.com/watch?v=lRwpcyd6w8k&t=0s

A

Companies, big and small, across the world, have adopted dagster open source. We are committed to developing and maintaining our open source product. It is critical to our success and supporting our community is core to our values. But feedback is clear. Tons of users want a managed service with enterprise features and support.

A

So, first and foremost, dagster cloud is a managed orchestration service whose goal is to minimize your operational burden with a platform that is both easy to use and that scales to the most demanding workloads.

A

Today we are launching two hosted options: serverless and hybrid serverless, dagster cloud frees, the user from all operational burden hosting all computation spin up in serverless, is totally effortless and provides an unmatched, fully integrated experience just write a python file and we do the rest, no kubernetes, no docker files, no provisioning, but not all users can offload computation to a hosted service.

A

For this we offer hybrid dexter cloud with hybrid. We host all the stateful system, infrastructure necessary to run dexter's control plane, and you bring your compute platform of choice. Dijkstra cloud has no ingress into your infrastructure, does not see your code and does not touch your data.

A

You run a simple agent in your existing infrastructure, which in turn manages a stateless elastic cluster that is easy to maintain hybrid dexter cloud, maximizes flexibility and security, while offloading the vast majority of operational burden to a managed service.

A

After you have set up dexter cloud, you are ready to write and deploy code, and this is where you see the daxter framework and daxter cloud combined to create a breakthrough engineering workflow for data teams.

A

The current state of developer, workflow and orchestration is highly unsatisfactory.

A

You can structure your code for local development, but you are still limited in what you can test, as data pipelines depend on secrets, services and data that are not accessible or runnable on your laptop teams either have to test in production or to set up heavyweight, inflexible, staging environments that are clumsy to use and costly to manage.

A

Dagster and daxter cloud combine to provide a novel, new type of environment, a branch deployment. It is a lightweight staging environment, created with every pull request that becomes a focal point of development, testing and collaboration.

A

Let's talk about how this works.

A

For most users, they have a main branch that contains their code and a production deployment where that code is deployed with branch deployments, you follow a familiar development workflow. You create a new branch edit your code and create a pull request. Then our out-of-the-box infrastructure takes over on every push. It creates a lightweight staging environment from that pr, where you can run and test your code.

A

The dagster framework is designed with this in mind and allows you to parameterize your pipelines and assets to point to test data in the branch deployment. In effect, you can branch your entire environment, providing a safe collaborative engineering workflow on top of your data platform.

A

When you have completed your work in your pr and validated your changes in the branch deployment, you merge your changes back to main and then new code is deployed into production and your branch deployment goes dormant, so you've set up dexa cloud, you're, developing and deploying code with newfound speed and confidence and you're executing that code on a robust, managed orchestration engine, but daxter's value doesn't stop there.

A

Dagster is the single pane of glass for your data operations. Dagstr cloud's run timeline provides a full overview with real-time status on a single screen. You can monitor all your data runs. This is the command center for your data platform operations and you can quickly diagnose problems with surgical precision with dagster cloud's asset graph. You not only have full asset lineage and observability, but you can execute operations directly in the orchestrator, so you can tame the sprawling complexity of your data platform with dagster's launchpad.

A

You can launch, runs with a powerful but easy to use, auto, completing editor usable by all stakeholders. You can kick off runs as defined or easily experiment with their configurations and observe the result. This barely scratches the surface of daxter's ui and operational capabilities, debugging runs managing backfills monitoring schedules and sensors. Inspecting your assets are all done in easy to use powerful uis that are a joy to use.

A

Security and access management are built in with soc2 compliance and sso for all dags or cloud users for enterprise users, octa and active directory integrations are available, along with role-based access control and the ability to create multiple deployments under a single account with daxter. Your team gets to regain full visibility and control over your data processes.