DataHub Product Demos, 22 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Supercharge Data Mesh with Domains in DataHub

Description

John Joyce (Acryl Data) gives an overview and demo of Domains in DataHub during the January Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

I'm going to start by talking about what domains are on datahub now domains are basically a collection of data assets that is sort of logical in nature. You can think of it as a top level, folder or curated folder of related assets that can be managed on datahub, as opposed to managed on some external platform.

A

It's also managed sort of centrally managed and governed centrally, or can be distributed to owners of a particular domain which are two key requirements. We heard from the community um now just talking about this. I know a lot of you are probably already like what the heck is this like. What are? I don't even know what tags and terms difference between those two are like now: we're adding something new, so I'm gonna just quickly talk about the difference between tags terms and the new concept of domains, um at least how we think about them.

A

So, in our perspective, tags are sort of informal, loosely controlled pieces of metadata that you can attach to assets. You can attach multiple of them to an asset and there's not really any central management or curation. Today, glossary terms are more like a controlled vocabulary, with the option to add some hierarchy to a graph structure between them.

A

Typically, we've seen this used to kind of standardize the types of leaf level attributes like fields in a schema say to define you know: email id, the standard concepts that your business cares about and then associate that standard, controlled vocabulary with real columns in the wild that you've ingested into data hub and domains are really these top level categories.

A

Maybe finance marketing that align with the nature of your business, there's central management and for now there's a single domain for asset limit, so one asset can be in one domain and soon enough we intend to add support for sort of managing access management for metadata via via domains as well, so they're kind of a way to govern metadata management.

A

um So now I'm going to jump into a demo of the domains feature in the ui, I'm going to be lazy about it today, because I have two different branches going for these features and just share a loom. You can see that we've added a new tab up in the right called domains.

A

I'm just going to start by going and clicking on that, and this is basically a place where you can manage the domains on datahub, which again are just sort of logical collections of assets. They can span across different entity types but they're managed centrally here on data hub, and you can see, we've already created a few domains. One may be for our sales assets, marketing and maybe data platform engineering.

A

So I'm going to go ahead and create a new domain just to show you the process. Let's say that we have. You know a product engineering team that we'd like to associate some assets with at the top level.

A

You can go ahead and create that should show up here in a second and then what we can do is actually start to associate that domain with um with entities, so we can actually add entities to the domain, and so maybe we will go ahead and add user creations to this domain. So we'll come in here and you can see that, in addition to all the existing metadata about an asset, we've added a new domain section here on the right and we can come into here and actually just search product engineering.

A

Add it set that domain and now what that means is. We can actually can actually filter by the new domain and see that we have just one asset in there user creations and you can see this is actually represented as a new tag which takes precedence over the existing tags and terms. So if you have multiple, this will take the precedence, but you can also click into a domain and search a domain.

A

In fact, as well um and that'll, take you to the domains sort of entity page where you can do things like see all of the assets that are actually inside of that domain, and so currently we only have one asset, so we can only see like airflow tasks, but you can imagine a domain may have hundreds or thousands of assets, and this allows you to browse and filter down using this filter slide out to see exactly what's inside of the domain.

A

So, for example, you can say user filter it down by text or you can filter it down by these faceted filters. Much in the same way, you would search you can also edit documentation for a domain. You can see that it inherited that documentation we had created. Initially we can add owners, we can add links, etc. So pretty much everything you can do with the normal entity. You can actually do with a domain as well.

A

Now I'm just going to quickly cover the policies part of this, so there's a new platform privilege that we've added called believe manage domains which will allow you to view this. This domains tab list. All of those domains um create domains etc, but in addition to that, there's a new metadata policy as well, which allows you to assign a domain to an asset or basically put put a nasa into a domain, and that is going to be called edit domain here.

A

So that's how the the privileges will work pretty much the same way that you know tags terms are managed with the one addition that they're sort of centralized management control for domains themselves.

A

um Final thing I'll show you is that on the home page, we have a new sort of set of set of recommendations here around domains where you can actually see you know the assets inside of a domain pretty easily as a kind of a launch point for discovery. So pretty excited about um about the feature. Let us know if you have feedback thanks, so that's uh domains in a nutshell. What will be available in the first release, at least so coming in eight to five will be the ability to create domains via the ui edit.

A

Their descriptions and owners browse assets inside of the domains attach them via the ui as long as well as attaching them at ingestion time, so you'll actually be able to take a reference to that domain, earn or id and place it in your ingestion recipe so that every asset from that ingestion source is automatically placed into that domain.

A

Searching and filtering by domain and finally, home page domain recommendations as well on the horizon, I think, is a few interesting things we'd like to add bulk, adding domains via the ui so being able to go to the domain profile page and actually just search and select a large group of assets to add all in one go as opposed to going and finding the asset and then adding it to the domain.

A

Second thing is domain based metadata policies, so this is something we've heard directly from the community. Is that you'd like to be able to sort of manage those policies or kind of set predicates for those policies based on domains?

A

And then finally, we've heard a couple use cases for nested domains or hierarchical domains, but we're not sure if this will be on the roadmap so waiting to get some more signal on that. One.