Dagster Dagster Community Day - Dec 2022, 8 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dagster Integrations - Dagster Community Day - Dec 2022

Description

Rex Ledesma provides an overview of the newer Dagster Integrations, while Pedram Navid gives us a quick tour of the templates he used in building the Hex and the Hightouch integrations for Dagster.

A

Thank you, hey everyone. My name is Rex and I'm an engineer working on Daxter since 1.0 the Dax report team has been cooking up new features and robust documentation for integration. Libraries, we've revamped our existing libraries and added on new ones. So here's some of them that we're highlighting in our 1.1 release our airflow integration now supports out of the box utilities to let users incrementally migrate, their airflow dies to use a dagger framework, our duct, DB and snowflake Integrations.

A

Allow you to use them as databases so that you can automatically load and store your asset inputs and outputs.

A

Our notable integration allows you to orchestrate notebooks in the cloud and manage them as often defined assets.

A

Our earbud and five range Integrations can now automatically generate software-defined assets for your connectors and, finally, our DBT integration is expanding to cover DBT cloud similar to the existing support that we offer DBT core. We can now automatically generate software-defined assets from your projects, managed by DBT Cloud jobs.

A

So from all the Integrations that we've highlighted there are common themes that we want to mention. These themes will inform how we're thinking about future Integrations to our framework, as well as the type of future content that you should expect from the Dax record to you.

A

First, we're committed to out-of-the-box support for software-defined assets for external services that manage your data. You can expect our corresponding Integrations to always include an API that generate the corresponding software-defined assets.

A

Second, we're we'll be providing more than just an integration API for you, we're committed to providing step-by-step guides so that you can easily set up the integration for its common use cases just copy and paste some code and you'll be off to the races and finally, we're focused on helping you achieve end-to-end business value with dagster.

A

This means that for prioritizing Suites of Integrations for overarching business use cases so for this round of Integrations, we've prioritized Integrations that will help you manage your ETL and elt workflows and in the future, we'll be accommodating Suite of Integrations that help you manage your bi dashboards and visualizations your ML workflows and operations, data, quality, alerting and monitoring, and much much more so stay tuned with dagster you'll be able to manage your data platform's complexity in one single pane of glass. So now, let's see our work in action.

B

Hey everybody, my name is and I'm going to talk to you about building a Dax or integration if you're not familiar with dixter Integrations they're, a great way to connect Dexter with.

A

Your outside world Baxter already comes with a bunch.

B

Of uh really well written libraries, these are things like airflow and 5, Tran and DBT, but if the particular integration you want isn't in here or you want to write your own for whatever reason well, I can talk to you a little bit about how I do them through this demo. Whoops come back screen, okay and.

A

B

Is the Dax your integration demo, repo, which I have I've, also created uh some for high touch and hex in the past? These were kind of the same pattern and so I've copied that in some ways for the demo, both the Dexter High touch and Daxter hex all essentially do the same thing. They hit an API, they request some type of activity.

B

They pull that API until that activity is complete and then they return the status, whether it's a success or an error, and so I copied that same basic formula for this demo, which you can find under westmarine data, diagster integration, demo and really there's just a small weed meet and a couple folders I back to your project. Folder is the implementation of the integration. So we won't go into that one too much, but the demo integration itself contains the code that you might write.

B

If you wanted to write your own integration and so we'll dive in real quick. If I can find my code there, it is so really. If you want to get started, all you have to do is type make, run and that'll install the requirements and start dagger in the background for you. So we'll do that and while that's running we will take a quick look at some of the files here.

B

The first one is a resource and let's just close that and that and that a resource is sort of that API to that outside world and we've defined. You know a class called my resource and a make request, helper function that sets the user agent. That could be nice, for example.

B

If you want to do analytics in the future uh and then from there we have a run project and a run status, so run project will make some requests to a run, endpoint and then run status will get the status from that endpoint, so fairly straightforward, API calls, and then we have the run and pull, which is a wrapper around those two functions. It'll run the project, it'll wait some time.

B

It'll, pull for status and it'll continue to do that until either a timeout happens or the project is complete and then it'll return that error or status back to the user.

B

And if you scroll down all the way, we can see the actual implementation of the Daxter side. So we create this resource. We use the resource helper here and you can specify a schema. So in our case, maybe we need an API key, maybe there's a base URL. You can see here I'm using HTTP bit and then we have macview, tries and delay as well, so different type of user configurable things for that resource and then that resource is consumed by go away.

B

uh An OP and op is what the user would typically use to run a project. In our example, you could imagine they might want to run several projects, they would create several Ops for one particular resource, and so here we have a project ID, that is user, configurable and some other attributes as well that the user can provide the op just takes that resource and it runs that run and pull command or function. We saw earlier so fairly straightforward, it passes the project ID and then it returns that result and.

A

It materializes.

B

The asset, so that often resource is really the main thing you want to set up. We have some constants, which is always nice. We have some types, so we can specify. Oh, my gosh stop some types for the data that comes back. This is what the response looks like from HTTP bin. So that's kind of what we've typed here and then we combine that into an output that could be read by dagster other than that. That's there's not much else. So, let's take a look at it.

B

Live in action, we'll open up our little browser here we will go to the multiple Dexter and you can see we have this asset. We can materialize it and click on run and you can see it ran successfully. We can run it again and there it goes running again and that's the little asset that we just materialized, how you can see the status here as well, and if you go to assets you can see here, we've materialized, it twice right and uh that's pretty much it.

B

So if you found this useful I hope you go ahead and create your own dagster integration and uh have fun bye.