DataHub Tech Deep Dives, 13 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: NEW! DataHub Actions Framework

Description

John Joyce (Acryl Data) shares the new Actions Framework for developing & deploying real-time outbound integrations with DataHub during the April 2022 Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

All right so um just one disclaimer, my apartment, complex, just mentioned they're gonna, be running fire, drills or fire alarms, so just disclaimer. Hopefully that doesn't happen during the demo. um All right, I'm gonna talk about data datahub actions framework, which is a project that we've been working on for the past few months at acryl and I'm going to get right into it by starting with a brief history of getting data into datahub.

A

Now, if you were in the data hub community around january 2021, you would have probably noticed there were a lot of questions like this in our chat. How do I get data into data hub and at that time our response was pretty much this I don't know write some scripts. We have some example python scripts you can use. We have some example java, but you're kind of on your own uh come february. 2021 we've decided to actually answer build an answer to that question, which was the metadata ingestion framework.

A

What this allowed us to do is to ingest metadata using a simple data hub ingest command.

A

Since then, we've seen how powerful it is to kind of break things down into simple standardized abstractions uh we've gotten myriad of different contributions to the ingestion framework, since we rolled this out even to some sources that the team probably hadn't ever heard of before um fast forward to today and we're getting a lot of new types of questions. For example, how do I create a jira ticket when something happens on data hub?

A

How do I ping slack or ping teams? When something happens? How do I receive notifications when something that I care about has changed on datahub.

A

Sorry guys, I don't know if you can hear this, but my compartment's coming back over the loudspeaker, I'm gonna try to continue.

A

um So what we did is we kind of leaned into these questions and we found that there was a theme which was getting data out of datahub, particularly in real time, and we narrowed down to a few different categories that we saw. People asking about. The first was around workflow integration.

A

So how can I integrate data hub into my organization's? Unique data workflows, for example, creating a jira ticket when a certain glossary term is added to a data set? How do I notify the right person when important things happen on datahub?

A

How do I synchronize metadata changes from datahub into some other third-party system? Finally, how do I actually audit the usage of datahub itself so just restating the problem? Clearly, we found that there was no easy way to get data out of datahub in real time and we needed a way to build.

A

Outbound integrations similar to how we have a way to build inbound integrations via the ingestion framework and one particularly insightful member of our community on the feature request board suggested that we could build an api for this, so that packages could be written for publishing to different systems, and we thought that was a pretty good idea.

A

So we started thinking about what that api would look like, and we came up with a set of characteristics for a solution. We wanted the solution to be extensible to support.

A

You know new use cases, new third-party systems or destinations new types of data or events flowing through configurable by design, so that you can configure outbound connections without coding, similar to the ingestion framework scalable, so that you can scale out processing of events that are happening on data hub in the case that you have a large load or a large deployment and then finally robust so a system, that's able to provide strong guarantees and handle event, processing failures, gracefully.

A

All of this, so that starting in may 22 2022, we can respond to this question of how do I respond to changes happening on datahub, not with the shrug, but instead with a framework, and so we've built the actions framework which allows you to take action when things happen, on data hub using a simple cli command data hub actions.

A

So what is it well in a sentence? It's a framework for developing and deploying real-time outbound integrations with datahub. So on the left side of the picture.

A

Here we have a lot of activity, ingestion, ui changes and on the right side we have a new actions framework which is able to respond to those things in real time and then take certain actions like sending notifications, audit logging integrating into your company's workflows, and so now, I'm going to pop out of the presentation into a quick demo where I'll show you what it looks like to work with the framework.

A

So let me start with.

A

So I'm going to start by just walking through a hello world of using the actions framework. uh It all starts with installing the datahub actions cli.

A

You can do that by just tip installing acro data hub actions very similar to how you install actual data hub. I've already done that, so I'm going to skip this step. The second step is to define what we're calling an action configuration file so similar to an ingestion recipe. This is a way to tell datahub what you want to do.

A

There's a few kind of critical pieces. The first is a name for your action that will be stable across executions. We'll talk about why this is important in a little while.

A

The second thing is an event source, so this is basically telling data hub where the events are coming from, in our case, we're just tuning into the kafka streams that come out of datahub and then finally, an action which is saying what code do I invoke or what process do I invoke when I receive an event from the event source and so you'll see here that we've configured a hello world action and all this does is basically it prints out every event that comes to it from the event source.

A

It also takes some configuration which allows you to print the events in uppercase or lowercase um just for fun, let's, let's start with uppercase, so once we've defined this, we can now move on to the cli and actually run the datahub action.

A

So we're just going to run. Datahub actions see hello world.

A

And what this will do is start an action pipeline which is basically a persistent process. That's going to react to changes happening on data hub and you can see that it has the name hello world. Now. What I'm going to do is go back to datahub and I'm going to start playing around with datahub.

A

So I'm on this dataset sample kafka dataset. Maybe I'm going to add a tag called pii.

A

Maybe I'm going to add a term called customer account and maybe I'll add an owner called john doe, and what you'll see on the right side is that something is happening and actually what I'll do is restart this framework to print in lowercase, so that it's a little bit easier to read.

A

Maybe I'll set the marketing domain as one last thing, and so if we expand this, what we'll see is that we're receiving different types of events which detail what has happened on datahub, for example? We just added a domain to this dataset and above we added owners, tags, etc.

A

You'll see that there are actually two types of events that we're getting and I'll talk about these a little bit more in depth once we get back to the presentation, but we have a metadata change log event and we have an entity change event coming in so right now. This action is just printing, pretty much everything that's happening on data hub. What we can do inside of this configuration file is actually filter down. To only invoke that hello world action on events that we care about.

A

For example, we can filter only when a tag is added to a data set, let's say, and we can do that using the event filter block we'll say I want to filter for when tags are added and maybe actually care only about the pii tag, because that's particularly sensitive for my organization, so I'm going to save this and then I'm going to restart the actions framework.

A

With my new configuration and now you'll see that if I remove this pii tag, nothing comes up, but if I add this back in, of course, we get the event you can also do or conditions. So maybe I care about a tag being added or removed, so maybe anytime, a pii tag is changed. It's really important for me, so I'm gonna, you know, do an or condition, and what this will allow me to do is now capture events that represent removals and additions.

A

Awesome. Okay, so this is super interesting, um maybe not so useful, though, because all we're doing is uh printing hello world.

A

So what I'm gonna show you quickly is just what it looks like to build a custom action to do something that you want it to do, maybe send an email or audit log or whatever else, and this all starts with a simple interface. That's called action to implement a custom action is really a matter of just extending this base interface and implementing a simple act method. This act method is invoked by the framework whenever an event comes in, in this case we're going to just print, but in reality you would probably do something really important.

A

The second step is to just configure that actions file to point to your custom action and then, finally, we have to make this action kind of available to the python runtime environment, and so I'm doing that via pip, a pip module that I've installed.

A

Once we do that, we can run the custom action and you'll see that it will print out a bunch of events. This is because this action is stateful. It actually tracks where it left off in the audit log, and so it has some catching up to do once it starts up. That's the first thing: it'll do all right so that pretty much rounds out the demo, I'm going to go back to the slides.

A

All right, quick recap of the quick start for folks who maybe just want to reference it or who couldn't make the talk today. Install data hub actions, configure an action and run it you'll see action pipeline with name x is now running. If it's successful custom actions is a simple matter of implementing the action interface and then running it now. The events that you saw coming in were of two types, and these will both be included in the first release of actions. The first is entity change event.

A

This is a high level event which is emitted when important changes are made to a data hub entity, for example, tags or added or removed glossary terms are added or removed. Even data set, schema fields are added or removed, and finally domains, there's also many more descriptions and and the list kind of goes on, but these are probably the most critical and then number two is the metadata change log event.

A

This is a lower level event, which really mirrors the structure of the metadata change proposal that we use during ingestion, which represents changes to the metadata graph. It's a little bit less easy to consume because you have to understand the kind of the details of the internal model.

A

There will be more details in the framework. Docs that'll describe exactly what these events will look like what the structure will be and what you can expect in your action code.

A

Now, as usual, I like to take a look under the hood to talk about how this actually works and really it starts with a few fundamental concepts, an event which is a data object which should be processed by the framework, an event source which is a source of events. In our case, the event source is coming from kafka, a transformer which is a transformer of events or a filter of events. That filter block is actually a transformer, an action which takes action on events, a pipeline which manages the coordination among these different components.

A

Essentially, it manages the life cycle of an event and then a pipeline manager which allows you to manage multiple pipelines running in parallel in the same process. So at a glance, this is kind of how things look. You have a source which produces events. You have a set of transformers, including filtering, and then you have an action which takes action on the event.

A

Finally, that's wrapped by a pipeline and that's wrapped by a pipeline manager, so in terms of availability, we're targeting release for may 3rd of 2022 that's early next week and the deliverables will include the data hub action cli, you saw sample actions, sample, transformers framework, docs and the two events we described the entity change event and the metadata change log event.

A

Some notable capabilities I wanted to call out is that we will support distributed actions from the very beginning. This means you'll be able to load balance among actions instances as long as the configuration is the same. We achieve this because we use kafka consumer groups under the hood, which allows you to load balance among a single stream.

A

We also have robust error handling a configurable failure, mode or continuation policy, which describes what to do if an event fails to process I.e.

A

Do we shut down the pipeline, or do we continue to make progress a failed message, log which will keep track of those events which fail to be processed by the action, a configurable retry policy, so the ability to rerun an action if it fails and then finally, single process pipeline, parallelization so basically running multiple configuration files in the same data hub actions command in parallel all right now, I'll just quickly talk about the road ahead, so where we want this to go, is to become kind of an ecosystem of general purpose actions and transformers that the community can use and share processing guidelines around how to contribute how to document and how to review new components and then, finally, just some framework improvements that we already know we'll probably have to make runtime event time validation a little bit stronger.

A

What this means is making sure that the inputs and the outputs of each transformer and each action are what it's expected failed event replay. So currently we log the events to a failed events file, but we don't have a mechanism to yet load that file and replay it through the entire action pipeline asynchronous event commits. Currently we support synchronous acting after an action has actually processed an event which is conducive to kafka and more filter types so being able to filter by a regex pattern or something more dynamic than a simple exact match.

A

This is a call to action. We definitely need your help as the community to make this framework a success, so we're going to be accepting contributions on pretty much everything from the core framework to the actions and transformers, and with that I think, that's the presentation. So thank you very much and I'll hand it back to maggie.