Dagster Dagster Shorts, 21 Apr 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Dagster Shorts: Setting up multiple and single Code Locations in Dagster

Description

Dagster Code locations are a collection of Dagster code and accessible by Dagster's tools, such as the CLI, Dagit, and Dagster Cloud.
A code location can be comprised of:
- A single python file
- A collection of python files

In this short tutorial, Odette Harary walks us through how to define both a single code location and multiple code locations using both the `workspace.yaml` file and `dagster_cloud.yaml` file for Dagster Open Source and Dagster cloud, respectively.

A

A

Hi everyone, my name, is Odette today, I'll be walking you through how to set up code locations for your diagster project. A code location is either a single python file or multiple python files that contain all of your different dagster code. So everything from your assets to your schedules to your sensors and what I'll be walking you through today is showing you how to set up first, a single code location, but then showing you how to add other code locations so that you have multiple code locations for your dagster project.

A

I'll show you how to do this for both open source, as well as how to set it up for dagster cloud. Let's get started foreign.

A

So, let's go over how to set up my first single code location.

A

So what I'm going to do here is use the dagster project, scaffold and call it my new project, my data pipeline, in order to create a new folder which contains everything I need in order to set up my first code location, so you'll see my data pipeline is in here as well as an assets folder, so I'm just going to paste in a dummy asset over here, so that we could see how it looks in dagster, and we have a couple other files here.

A

So the first one is this Pi Project toml file, and this is used for a single code location and it's going to say my module name is my data pipeline. So it will actually tell dagster where my code lives and that's within this my data pipeline folder.

A

Then we also have a setup.config file, which also has my data pipeline in there, as well as my setup.py file, which is saying, hey, look at this module, my data Pipelines.

A

So now that we've set that up. That's actually all you need to do to set up that single code location and tell dagster where to look I'm going to change directory into that folder and now I'm, going to run the extra Dev and spin up the dagster UI.

A

And if we take a look at that.

A

So if I look at that store here, you'll see that we have this new random asset and if I look at my code locations, there's that single code location there so that's great. But what if we have multiple code locations? So what I'm going to do is in addition to this, my data pipeline folder over here I'm, going to actually add a new folder, which has some machine learning pipelines that we're going to add in so I've just cleared my terminal and I'm, going to paste in a new file or actually a new folder.

A

And what I'm going to do is paste it over here. So here we have. Our ml pipeline you'll see that there's this ml assets.py file, which has different machine learning assets and the next thing I'm going to do, is actually create a new file called the workspace.yaml file and you'll be able to see a bunch of different examples on the docs for this, and what I'm going to do is actually.

A

um Walk you through what this means in order to set this up so just to show you different flavors of what this would look like. I'll, also paste in a new py file which just has a single asset and is not a module. So what this has is load from, and the first thing I'm doing is loading in my data pipeline code, location and you'll see that this has both the module name, but also working directory, which tells it where to look.

A

So it's actually looking within this folder here to know that I need to change my working directory in order to find this new module. The next thing, I'm going to do, is add in this new machine learning pipeline module. So that's this line of code here and here you'll see that there's no subfolder, so I just need to say that this is the code, location and it'll know to look in this module for my assets.

A

The last thing I'm going to do is show this is actually how you load in a single python file, so you could have any mix of different code locations. You want, and here I'm just saying that this is the python file that I want for my third location and you'll, see here. I went back into this subdirectory called code location, so I'm no longer in my data pipeline directory.

A

So let's save that and now from this code locations, which has this workspace.yaml file in it all I need to do is tell dagster that I want it to spin up.

A

So, if I look at my code, locations under deployment here, you'll see that we have not one but three different code locations and if I look at any one of those I'll be able to see my different assets as part of each of those code locations. So congratulations. You've been able to set up multiple code locations with your decks with your workspace.yaml file in your dagster project, for open source.

A

So next I wanted to show you how to set up multiple code locations when you're using dagster Cloud so you'll see here that we have all of the same types of files, but instead of a workspace.yaml file, we actually have our dagster Cloud diamo file. So if I open that up, I have already added in my second code location here so same as before, I have my data Pipeline and I. Have my random assets up py file. I could also add in that same machine learning, Pipeline and that will work similarly.

A

So the first thing I have here is my first code location, which is my data Pipeline and that's I'm, actually calling it data pipelines and then the second here is the python file called random assets. Dot py so you'll see here that here we have the package name, and here we have the python file.

A

You'll only have a single dagster, cloudy ammo file for all your different code locations, so no need to create multiple files, and let's see what this looks like in dagster Cloud, so I have this here.

A

This is my dagster cloud UI and you'll notice. The same type of pattern, so I have my Dax data pipelines over here, as well as this random asset and you'll, see here that this has a package name, and this has my py file and I'm able to see all of my different Assets in here. We hope you enjoyed learning about code locations within dagster and looking forward to seeing all the different projects you'll be creating. Thank you.