GitLab Incubation Engineering, 19 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What can you do with Server Runtime - Part 4 - Airflow

Description

I wanted to do something new this week. I wanted to take the existing Server Runtime / Remote Dev architecture and see what we can do with it. I will mostly be making changes to Devfiles in these updates and therefore these changes should work both on DWO as well as Server Runtime.

This is a continuation of the "What can you do with Server Runtime Series?". This is **Part 4 - Airflow**. Airflow is one of the most common tools used by Data Engineers. Typically you orchestrate ETL/ELT pipelines via Airflow. Airflow can get hard to work with when you're working in large teams. There is always the fear of overwriting each others changes. Server Runtime can create a dedicated Airflow instance from a branch or a MR that can then be used to test your changes so that you don't accidentally interfere with the work that someone else might be doing at the same time.

This idea needs to be credited to Fred who is working on making Airflow better on GitLab in the Airflow SEG.
You can find out more about the SEG at:
https://about.gitlab.com/handbook/engineering/incubation/airflow/

0:00 - Introduction
0:57 - Demo

A

Hi, everyone welcome to another update from the server runtime site for the 19th of January 2023. This is another part in the series. What can you do?

A

It's our runtime today I'm focused on Apache airflow, so a budget airflow, if you've not heard of it, is an extremely common tool used by uh most data engineering teams today to do elt and ETL uh another tons of issues with working with airflow in a CI CD caravana, because you know you typically are working a shared cluster and you tend to override each other's changes, and so it'll be great if you could use server runtime to spin up a dedicated airflow instance where you can work on now. This is not an original idea.

A

I can install this idea from Fred who's working on this airflow seg, which has a goal of making development of airflow much easier, and so this is just my take on how server runtime can help in that space.

A

um So I'm gonna start the journey here um in in a in a repository in the GDK, um where I've got an airflow demo uh with a bunch of dags um and uh I've got a Dev file as well here, which contains um an airflow image which I used to run airflow and just something to hold a runtime. So I'm going to go ahead and create a workspace I'm going to specify an ID to inject I, choose vs code, this time, I'm going to go ahead and and create a new workspace.

A

We'll just give a few seconds for this workspace to start up. Okay, so the workspace is running now. I can go ahead and open the workspace sign in and I'm actually inside. My IDE I can go ahead and open up the Repository home workspaces.

A

And within the repository it show you that I have a few dags and I'll open up. One of them. I have a few dags here, an auto calculator and then also the shipping tag. What I can also do, because in my Dev file, I've exposed airflow on a on a board here, I can go ahead and just open that up by appending the port number, so that put the port number, and here I have a dedicated airflow instance. Just for me, you can see.

A

The dags have already been loaded in I can enable the order calculated tag and then um I can have a look at the graph of what are the steps are in the tag in the ETL workflow see, there's three very simple steps here: I can go ahead and Trigger that, and that will start running so you'll see as it as a scheduler sort of picks up the various steps you can start doing the various phases so extract, transform and load uh just take a few seconds to complete.

A

And now you can see it's completed, I could go and see the status of your jobs Etc. Now, let's see how the editing experience works right. So we had this simple workflow. I can actually open this in my IDE, the dag itself um and I can make a change to the stack. Maybe I want to add an additional step. So I can create a new task here.

A

um And change some parameters.

A

um And just make sure I call my task as well. At the end, so I go ahead and I refresh my uh my bag. You can see immediately, you can see. My task has come in here.

A

Okay, we have the task in now. I can just go ahead and go ahead and Trigger the tag, and you can see the dag would execute as we expect it to.

A

The steps are complete now. I can actually have a look at the logs as well clicking on that scrolling down and seeing that that log generated. So that's it. It's airflow development mail, easy with server on time. Thank you for listening.