Dagster Dagster Talks, Panels, & Interviews, 13 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dagster: Modern Data Platform Orchestration

Description

As organizations grow so does the complexity and number of tools. In small data teams it is not uncommon to be responsible for more systems than there are team members. Without a strong orchestration layer to hold these systems together, many workflows are held together with one-off jobs and shaky deployments.

Dagster is a modern framework for defining jobs that treats the data platform as an application. Jobs can be parameterized and iterated on locally before moving into the cloud and interacting with production services. Using Dagster as the foundation for a modern data platform ensures that all data stakeholders can easily interact with their data and build out specific pipelines to maximize value without having to worry about underlying infrastructure.

Presenter: Dennis Hume @Dutchie

A

Hi, I'm dennis, and I'm just rehashing a presentation I did earlier in the year for big data madison, just giving a quick overview of dagster. So a little bit about myself, I'm a data engineer at duchy. I've been there just under six months now, and I've been using dagster for the last two years so before I get really into the presentation, just want to say up front that I don't work for dagster, I'm just a fan.

A

So all of these thoughts and opinions are just my own and but I am still a proud owner of a dagster sweater.

A

So for this talk I wanted to give a very quick overview of data orchestration where it fits in with most data stacks and then really just dive into dagster, its various components and abstractions and kind of build from the smallest units and dagstr up into how you really create an entire dagster application.

A

But really this is going to be just a simple, quick overview of the framework. It's going to be just a nice midwestern introduction to it and with everything in this presentation is in github, so feel free to try anything out yourself.

A

So for data orchestration, the main responsibility of data engineers is to coordinate work across various systems.

A

With the rise of the modern data stack and data teams, relying on so many different sas offerings to manage data, you really need a good orchestration system to be able to tie all these various components together and a good orchestration framework allows you to kind of treat and maintain the creation of these orchestration pipelines in a way that you would maintain regular application code. So you can take your pipelines through the full development life cycle, so this is where dagstr is different than some previous orchestration tools in that one.

A

It knows a lot about your pipelines. It knows a lot about uh what the nodes that make up a graph rely on what resources and external dependencies they reach out to so this lets them have a lot of very rich metadata about what's happening, which really lets. You know a lot about your pipelines and the infrastructure. 4 dagstr is very customizable, so you can separate out compute and storage very easily and really configure pipelines to what that pipeline needs and not just have to give.

A

All the effort just to like maintaining the orchestration framework, you can maintain isolated workspaces, so you can have pipelines that have specific dependencies to that pipeline. You can also, then group pipelines together based on them sharing dependencies or you can include pipelines together if they're all used by the same business unit and taken all together. This really allows for really easy local development and testing, which is historically very difficult in data engineering and data pipeline maintenance.

A

But you can really set up environments that match production outside of production and really get it, so you can spot errors early and really catch things before they hit production.

A

If you'd, like a more detailed explanation of how dexter is different from previous generations of orchestrators, there's a nice slightly opinionated blog post on their site, so starting at the most basic unit in dagster is an op and an op is just a single unit of computation.

A

You can see that an opt looks very similar to just a regular python function. Writing dagster is very pythonic. You tend to just write functions and then use special dagster decorators.

A

So for the pipeline, we're building out is kind of a convoluted bmi calculator, but for this first op in our graph is going to just convert pounds to kilograms. So you can see that there's this basic basic function that takes in weight in pounds and converts it into kilograms.

A

So it takes in an int and returns a float, and this is what we were talking about before with the metadata that dagster attaches to the components of a graph.

A

So this is just a screenshot of what uh this will look like in daggett and daggett is the ui for dagster, so we'll get into that when we actually jump out of the slideshow and into dagster. But this is just again our one node in what will eventually be our pipeline.

A

So a graph is just a connect or a collection of ops arranged to produce a dag. So here you can see that we have a couple more ops. We have one that converts inches to meters. We have an op that calculates bmi, so this takes in two inputs and then a final op that uh converts uh b uh bmi into just a into uh value for its range. So you can see that again.

A

This is pretty easy to understand of you're just hit defining your ops and then your graph is just um the output of each of those op functions and chained together. So it really encourages a very functional style of programming, your pipelines, so in a daggett. This is what our graph looks like. You can see how the various ops interact with each other. So we have our two input: ops of converting meters and kilograms and those go into our bmi calculation.

A

So a job is a graph that has been configured for a certain use case. So if we go back to our our graph in order to run this, we need starting input values for the pounds and inches that we're going to be converting ultimately into our bmi calculation.

A

So a job is just configured with specific values for our graph, so in this case the job we're configuring is configured specifically for someone who weighs 190 pounds and is 75 inches tall.

A

So the nice thing with dagster is it really encourages parameterized pipelines so compared to some other frameworks where you really had to duplicate code in order to have multiple runs of the same pipeline with different values? Dagster, you keep the same core logic and set different values at time of execution. So this is really nice in that we can have multiple configurations of the same graph for different jobs, for different values and also for ad hoc runs, which you'll see when we get into daggett.

A

You can, at time of execution change these to whatever we want.

A

So the last thing I'm going to mention before we kind of dive into it is just a repo and workspace.

A

So a repo is just a collection of our jobs and schedules and sensors and we'll talk about schedules and sensors in a second and a workspace is just our collection of repos that all rely on the same underlying dependencies all right. So, let's see what this looks like, so I actually have oops.

A

I have um dagster running uh just locally within docker and we'll talk about that more in a bit, but so here is our interface for dagster, which is our dagget ui, so this pipeline that we were talking about this is our bmi pipeline that we saw in our slide deck. If we go to running launch, you can see that if I ran this with nothing, this would use our underlying job parameters of.

A

I think it was 75 inches and 190 pounds, but we can also change these to whatever we want. So that is some odd.

A

That's an odd pairing of height and weight so we'll see how healthy this person is.

A

So when you launch a job it goes into this interactive gantt chart so there'll be some cueing of the job, but then you can see as the jobs running the individual tasks that make up the graph and.

A

My computer is being a little slow, so this is all just running locally, um so you can see our pounds and inches both calculated they're, passing that off to the next node, which is calculating the bmi and finally,.

A

A

All right so there's our job. We can see that we can see the specific logging and metadata for each node in our graph, and then we can see that, based on the final output, that person is underweight given their height of well into just under seven feet and weighing 160 pounds, which makes sense.

A

A

So architecture of dagster, um going to keep this mostly to the docker compose, like I said, is very flexible in it can pretty much align to whatever infrastructure you're using at your organization. I've now set up dagster twice and have had two different infrastructure setups for it, and um it's it's very nice in that way of that it's entirely.

A

The abstractions are very clean and you can change things out pretty easily. So what we used is we had one workspace that we were using to run that pipeline called data analytics, which contained the bmi pipeline.

A

We had a different service daggett, which was maintaining the ui, and then we had a postgres instance, which was just storing the metadata about our pipeline, so things about run execution and things like that.

A

um So there's still more going on under the hood, with the stalker compose of there's a daemon there's a second workspace and we're using local stack to emulate aws, but we'll dive into those. Now, as we talk more about schedules and sensors, so talking about all the customizable uh infrastructure for dagster uh the main place where you can figure, this is within the dagster yaml, and so the dexter gamble determines things like your run. Storage, your events, storage, schedulers, run launchers, so pretty much. This is just a way.

A

This is just a way of saying that you can configure all the different components of the metadata storage where jobs execute where artifacts are stored. All of this is configurable within the dagstr yaml, but we're keeping this pretty simple for the most part of keeping it to.

A

Most things being stored just within our postgres metadata, um and not doing anything too too tricky for this docker compose setup.

A

So we kind of talked about all this before, but this is just how the various objects in dagster for kind of a pipeline roll up into each other of how a graph can contain multiple ops, a job can be configured or multiple jobs can be configured off of a graph. A repo can contain many jobs and a workspace can contain many repos.

A

So what we haven't talked about yet is how you can automatically schedule. Dagsta runs like we just uh set off an ad hoc run when we went into the ui and went to launchpad to set our values and kick off a run, but there's two other main ways to interact with dexter pipelines. The first is a schedule and a schedule is pretty simple of just giving it some crown syntax and a job and telling it to run at a specified schedule. So that's nothing too crazy.

A

What is interesting are sensors. A sensor is a way to execute a job based on a change in in an external resource.

A

So an example of this would be to set up a pipeline that triggers every time there's a new file in an s3 bucket, so you can have dagster, be pulling this external resource this s3 bucket, and when there's a new file uploaded to that bucket, it will kick off a run of the pipeline.

A

So let's go back into daggett and kind of look at these. So with our bmi pipeline, you can see that we have. Oh, I didn't even know that I had turned on the sensor.

A

So if we look at our schedule, you can just. This is just easy enough to turn on and turn off, so you can see when our schedule is set to run and for our sensor uh I turned it off now, but you can see previous runs that it's been pinging, uh our local stack, s3 bucket and looking for changes uh in it. So it's been skipping the last.

A

uh It's triggers every 30 seconds, checking for new files and you can see it's been just marking these as skipped in case there was a new file, but since nothing was added, they just skipped and now that the sensor is turned off, it's no longer polling that s3 bucket, but you can see that I did have some previous.

A

Runs where I've uploaded a file to our local stack s3 bucket and that triggered a run of dagster.

A

So zooming out again into our architecture, you can kind of see where the daemon fits in. So the damon works with dexter to maintain things like schedules and sensors kind of these external processes outside of the pipelines themselves and pipeline executions that happen within the workspaces, and you can see. Local stack is just an external dependency that we've taken on uh for dagster, and this is something that the damon is pinging to look for.

A

New runs of a job, so how schedules and sensors kind of roll up into a job and that a job can have many schedules or many sensors.

A

So, as I was talking about um with external dependencies, this is where dagster is really useful in how it views external dependencies.

A

So for dagster, that's an external dependency can be defined as a resource, and a resource is just something that exists outside of the dagstar framework, so we're going to create a pipeline with a resource now, so this is should be pretty familiar to most people who do data engineering of having to do a pipeline that relies on a database. So this pipeline very quickly creates a table if it doesn't exist and inserts a record into it.

A

So pretty simple in our op and graph abstractions of how you would define this, the only thing that's different now is you can see that that this pipeline relies on a required resource, key name database, so this is taking on an external resource name database.

A

So we're going to define uh our resource and dexter has a lot of resources pre-built as modules so you're, not always going to have to build a custom resource for everything you're going to interact with, but this is kind of just showing what a resource looks like. So we're building out a resource that connects to postgres, so it takes in a host a username, a password database and then allows you to interact with the database.

A

So what's interesting about dagster is you can define your resources at different uh environments and set them for different resources so for our same graph, we're configuring two different jobs, we're doing one? That's called etl local and one that's called etl docker, so our etl local uh job when we say database we're actually using a mock resource.

A

So this can be useful for things like unit tests and just doing very quick experimentation for our etl docker, we're actually using our that postgres resource we defined and we're going to be using just the postgres.

A

Docker resource that we've been using um for running this project locally, but this can be helpful because if we had additional environments where maybe in production database meant snowflake, we could just use the snowflake resource. But this again is where dagster kind of puts a lot of emphasis on parameterization and that you can not only parameterize the values that go into a pipeline. You can parameterize out the resources that make up a make up a job, so you can have the same job used across multiple environments, with multiple different configurations of external resources to match.

A

Whatever your infrastructure is so this again is just nice that you can test things between local dev stage and prod without having to change core logic. You can just have these configured for the different environments.

A

So here kind of, like I was just saying, is how we configure our resource and it's very similar to how we configure our op. So in our bmi op we are configuring, the values for height and weight in this case we're configuring, the op value that requires a table name to a table called postgres table, but we're also configuring our resource for um to connect to our postgres docker instance.

A

All right. So let's try this. I'm worrying slightly about my computer that this will take a little bit of time, but we'll be optimistic. Okay, so this is our just local version of our pipeline. So this is using just the mock resource.

A

So this is going to execute and it's going to create a table or it's going to pretend to create a table and then pretend to insert a row into the table.

A

All right, this actually produces multiple rows and then for our docker version of the same pipeline. Here you can see that we have the additional configuration for connecting to the postgres resource.

A

And this time, even though it's connecting to an actual resource this time, it's actually going to be connecting to uh postgres uh the pipeline will behave the exact same so I'm creating our table.

A

And inserting some rows.

A

All right, another thing to talk about are assets and an asset is an entity, that's created or changed by an op. So this can refer to anything. This can be a database table an ml model or an or an aws resource.

A

So with those pipelines we just ran, we actually had asset materializations associated with them. So when we run those pipelines as well as just running the pipeline itself, we're saving this historic information about the run that individual run. So if you go into daggett, you can see in your asset catalog kind of the history of the individual artifacts that are being created by your pipeline.

A

So this shows things like execution time. So how long.

A

How long the assets were taking to create, so you can get a kind of sense of historically how your pipeline is behaving and you can also connect uh whatever metadata. You want to your asset. So in this case I was including our table name and since the pipeline just adds a random number of rows.

A

You can see just the number of rows that get added with various runs of this.

A

So let's go back in and look at our asset, catalog oops, so here's our micro batch all right so here is our runs of our etl local and etl docker. So you can see that etl local inserted seven in its run versus the docker, which only took three. You can kind of see.

A

Other things like we're, recording the table name and you can just see previous runs of our of our pipeline.

A

So you can see both that we record all of the um individual runs of the pipeline and the assets that are created by our pipeline.

A

All right so two more things to add to our list of dagstr abstractions are just resource and assets. So again, resources are associated with with an op and it's just an external dependency, and an asset is also associated with an op or a job, and it is created by created and maintained by our extra pipelines.

A

So, like I said, this is just more of a quick introduction to dagster and just getting your first pipelines up and running but wanted to give a give a quick overview of some other highlights of the framework, um the other main yaml for configuration and there's a lot of different ways to maintain workspaces in dagster. But your workspace yaml lets you set up multiple workspaces that all fall into the same day, extra application.

A

So I mentioned earlier that we were using our data analytics pipeline or workspace, and that contained.

A

Our bmi and etl pipelines, but there's also this additional workspace that I created called data science, which just has its own pipeline and what's nice about this- is that these workspaces have their own dependencies. So I think for data science I put pandas in it, but so the pipelines that exist in that workspace can use pandas, but for data analytics I didn't need to do any pandas so that dependencies not doesn't exist within it.

A

If you look in the den or in the repo, you can see that each workspace just has its own requirements file, so yeah data science does indeed have pandas.

A

So that's talking about multiple workspaces, uh launchers and executors. This is more about the infrastructure of running dagster and just how you want to uh be setting it up and provisioning jobs. So, like I mentioned, dagster is very flexible. You can do what we're doing here, where you're just running your pipelines on the same node. That holds your workspace or you can set up configure a launcher, so launcher just runs pipelines outside of the workspace node. So you can.

A

This can be configured for things like kubernetes or ecs, um or you can create your own launcher like everything in dagster, there's kind of these core abstractions that are that can be configured for whatever you need.

A

Executors just allow you to how you run the various um nodes in your graph in parallel.

A

So this is kind of an overview of how you can some some of the different launchers that are available um tests. This is something that I really appreciate. Just since testing data pipelines is historically awful um with dagster. They make this relatively straightforward and you can be testing your individual ops. You can test your entire graphs and jobs and, as you saw with being able to sub in resources and do just like dependency injection for unit tests, you can really run entire pipelines as if they were production just locally, which is a huge plus.

A

um We've now been at dutchy, been running dagster for a couple of months now and with the way we've configured our environments, we have yet to really have a pipeline fail in production. That, outside of like an aws error, we've been able to really front load our testing and make sure that everything is running as we expect. We don't have to do that. Data thing of kind of deploying and testing in production, which is always terrifying.

A

um So what we didn't even cover were backfills partitions, io management, uh graphql layer, um the it's very cool with dagster, since the core team was the creators of graphql at facebook, everything in dagster is backed by a graphql, so you can be anything you see and dag it. uh You can be running with graphql.

A

The other thing we didn't talk about yet, but is very interesting- and I can say this now that I'm redoing this presentation in the future, our software defined assets and dagster gave a great talk at 2022 data console about software-defined assets and they have a blog post about it. But this is really, I think, the direction that the framework's going in just defining your pipelines, not as just kind of a collection of pipelines but as the assets that get produced.

A

So you can see this in their talk or their blog post, but really starting to build out your data platform layer based around these shared software assets rather than just building it around pipelines. So that's kind of its own talk, but it's very interesting as well and the other nice thing with dagster is they now have a cloud offering so, if you're interested in kind of getting into dagster but really want to just focus on the pipelines and not as much the underlying infrastructure.

A

The cloud is really a great solution for that uh at dutchy were a cloud customer and it was very nice to just be able to get up and running with dexter day one and just be able to quickly build out and iterate on our pipelines themselves.

A

So this is a rehash talk, so can't take questions. So thank you.