Dagster Dagster Day 2022, 9 Aug 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Dagster Cloud Demo

Description

Dagster Cloud is the easiest way to get started using Dagster. This video explores how Dagster Cloud lets developers build, test, and deploy their code with confidence.

Get started with Dagster Cloud here: https://dagster.io/cloud

For more information on getting started using Dagster, check out this demo: https://www.youtube.com/watch?v=lRwpcyd6w8k&t=0s

A

Dijkstra cloud makes adopting dagster easier than ever. It builds on top of dagster's orchestration platform to enhance the collaboration and productivity of your data team.

A

Let's start by seeing what it looks like to get up and running on dagstr cloud, which we can do in a matter of a few clicks as a user, I have the choice to use dagger cloud as a hybrid model or as a fully serverless solution. The hybrid model ensures your code runs entirely in your infrastructure.

A

Dagstr cloud manages the operational burden, but never sees your code or touches your data.

A

The fully hosted serverless option lets us productionize our code without having to set up any infrastructure, we'd like to get up and running as soon as possible, so we'll choose serverless next, we'll connect our github account.

A

We could load some pre-existing dexter code, but we'll instead have daxter cloud template some example code in our account for us automatically set up with ci cd.

A

Here we have our example: repo. Our code is automatically being deployed to our production environment from our main branch.

A

Once the deployment is complete, we can view our code in dagster cloud from here. We can re-materialize our assets kicking off a run in the cloud.

A

With our code successfully running, let's go ahead and make a change we'll open up the source file in our github repo. Here we have a few software-defined assets. The first pulls a list of serials from an api into a data frame. The second filters down to serials manufactured by nabisco, we'll add another asset which filters down to serials manufactured by quaker.

A

Now that we've made our change, we can commit it back to our main branch, and this will kick off another deployment to production.

A

After a few moments, our new asset is live in production and ready to be materialized.

A

Dexter cloud is built to scale with the size of your data team and your data workflows. Let's take a look at what the development process on a more mature data platform looks like here: we've loaded a set of data pipelines, our code fetches and processes, some population data producing population rankings.

A

From this data we build some dvt models and our ml team also generates some forecasts.

A

These pipelines are built using software-defined assets to see more about how this codebase was built check out our intro video on software-defined assets. By deploying our code to dagstarcloud, we can ensure our assets are reliably up to date.

A

But dexter cloud isn't just a platform to run your code. It's built to supercharge the entire developer, workflow, helping your team build, modify test and deploy their data workflows with confidence.

A

A change begins in the local environment, where developers write their code, let's go ahead and expand on our existing set of assets by adding some assets representing country gdp, we'll start by adding some imports.

A

Next, we'll add a country gdp asset, we'll pull some data into a data frame, then we'll add the per capita gdp asset we need to depend on our existing gdp and population assets. Dagster's programming model lets us specify them as inputs to the asset function, we'll join the input tables compute the per capita gdp and return it we'll create a branch, commit our changes and push up to our github repo.

A

Next, we'll create a pull request.

A

The extra cloud's github integration will automatically create a branch deployment, an ephemeral staging environment where we can run our work against real cloud services and test our changes.

A

After a few moments, our branch deployment is ready.

A

Branch deployments run against a forked environment that we can test against to gain confidence in our work. Here, we've configured our branch deployments to create a temporary copy of our cloud environment. This job creates a duplicate of our snowflake schema. Our pipelines and assets are parameterized to write to this copy, so we can materialize assets in our branch deployment. Without interfering with production, I can now go ahead and kick off a run to ensure that my changes work and a context nearly identical to production.

A

Of course, these branch appointments scale with the size of your team. Every pull request spawns a new branch deployment. It's easy for me to get context into a team member's change as a reviewer or another stakeholder by viewing their branch deployment where I can explore their jobs, their assets and their runs.

A

Once I'm ready to merge, my change deploying to production is as easy as simply merging the pull request.

A

Once the deploy is complete, I can go view. My new assets live in the production environment, ready to be materialized for the first time in production.

A

Dexter cloud is built to not only be the easiest way to productionize your data platform. It's designed to empower your team to develop faster and with more confidence. We want dexter cloud to be the right choice for teams with large and small.