Dagster Dagster Hot Takes, 16 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dagster Hot Takes: Upgrades

Description

No more dependency hell or Orchestrators for your orchestrator

Prefer to read: https://gist.github.com/slopp/7c64973c70b12ffbd506d4e64d80d65e

Resources posted below, or star us on GitHub to get started.

- Star Us: https://gitub.com/dagster-io/dagster
- Example Project: https://github.com/slopp/dagteam

A

Hey it's Sean here with another Daxter hot take and today I do want to talk about upgrades now, unfortunately upgrades whether it's changing a new version of a python package or updating code. It's something that can cause a lot of challenges for data engineering teams. You might have heard of people that are stuck in dependency, hell or production systems breaking, because airflow has some python dependency that conflicts with a package that you want to use in your code.

A

So it can be a challenge to think about how to upgrade a system, how to add new packages, how to make changes, and that challenge is compounded. If you have a large team, you might be in an organization that has different groups working on different projects and often you're forced to create separate orchestrators for those different projects, just to avoid dependency conflicts, which is way less than an ideal, because really an orchestrator is supposed to or straight everything.

A

You don't want to be in a world where you have to have an orchestrator managing your orchestrators, and so today, I want to show you how dagster resolves all these concerns, and so we have a GitHub repository here for a data engineering team called dag team, and the way that this is set up is that, within this GitHub repository, there's actually two dagster projects. One project called analytics one project called ml, and these projects are separate python packages, so they can have their own separate dependencies here.

A

In this diagram, we can see we might have an ml project with a new version of dagster and sklearn, where it's an analytics project might have an older version of dagster. The way that texture works is that each of these individual projects is totally isolated from the orchestrator and the dependencies that the orchestrator might have. So if we go and look at this project inside of dagster, what we'll see is two deployments so one for the ml project, one for the analytics project in our workspace.

A

We can see that each of these has a couple of different things defined inside of it, but what's great about dagster is even though we have these separate projects, we can still have cross project dependencies. So if we go to our Global asset view, we can see that the machine learning team might have this asset called penguin clusters.

A

That depends on an asset that the analytics team has created called penguins, and so, even though these assets have separate dependencies uh and run in totally separate environments, there's still a global awareness of how different assets relate to each other. Okay. So what happens now if a team were to make some breaking changes?

A

So if we go to our overview, what we'll see is that, right now we have a job, that's running every minute from the analytics team to update a asset called penguins, and then we've defined a sensor that anytime, our Penguins asset is updated, triggers the code to update our ml project penguin cluster. So in other words we have a schedule, that's executing our analytics team's work and then our machine learning team, which has that Upstream dependency, is automatically being run through a sensor.

A

In fact, if we go and look at these two things we can see, our schedule is running here once a minute and then our sensor is triggering every 30 seconds, but it's only finding new work to do every minute, and so we have this nice pattern where automatically our both of our assets are in sync.

A

But now let's go ahead and make a change that maybe breaks one of these projects, so I'm going to go inside of Visual Studio here I'm in the machine learning project and what, if I, do something that will make this a non-loadable diagster project?

A

Well, the first thing is that I can totally prevent any changes, breaking changes, making their way into dagster with a really simple test, and so we've written this test that will check if our repository can load and so in cicd or even locally I can just execute this test, uh and it's going to tell me right away that I've made a change. This makes this dicer project unloadable, so that's an easy way to to prevent any errors.

A

But let's say for whatever reason: I didn't run that test and instead I go into diagster and I just try to reload this project. What you'll see is this error message and it's telling us here what's going on, but if I dismiss this for a second, let's look at this Global State of Affairs.

A

So the first thing to note is that if we go back to our overview page, our analytics team is unaffected. Their job to update the Penguin's asset is going to continue to run. In fact, if we look at the runs, we can see our update. Penguins task continues to March forward, even though our machine learning task updating the cluster asset has failed. So the system is robust, so it changes that one team might make, even though it's unlikely that those changes will actually make it into production.

A

So hopefully that alleviates any concerns that you might have and gives you a great example of how dagsters different projects can be separated and isolated from each other. The other thing that I'll mention is neither of these teams are associated directly with dagster the orchestrator, and so we could break both of these deployments and dagster itself would continue to run, and that in turn means, if you go back to this picture here, that you don't ever have to worry about upgrading the diagster orchestrator in sync, with upgrades that you might make to your projects.

A

So if we wanted to upgrade the version of dagster and that the orchestrator is using, that would not impact or require an update of either of our projects.