DataHub Product Demos, 8 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Add Ownership, Tags, Terms, and more to DataHub via CSV!

Description

Aditya Radhakrishnan (Acryl Data) demos how you can enrich your metadata using DataHub's new CSV Enrichment Source .

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

Hey everyone: it's aditya good, to see you all again, um the great town hall so far great job, everyone um cool. So I'm here to talk about the csv uh ingestion, plug-in that we recently added um it's actually already available in the uh datahub cli. So why would I use the csv ingestion plugin? So this is really great if, like you want to bulk, apply things like tags terms and owners to um various data assets that you have so um you know if you're like.

A

Oh, I want to apply a bunch of glossary terms that are um for like various pii things uh across like different data sets. But, like you know, you want to click through each of your data sets individually on the ui. You can use this. The csv ingestion plug-in to you know put in which data sets you want to apply this on and which terms- and it's just easy to do- that on the oil like one command.

A

This is also really great because we can easily extend this to other aspects of the future. So right now we only support tags terms and owners, but descriptions is coming very soon in the next cli release. um You know you can imagine this is easily extendable to things like domains and like whatever um aspects like we will add to data hub in the future.

A

um This is also really great, if, like you already have some other mechanism, like maybe other tools or like, maybe you just manually, keep track of things like oh um on some spreadsheet, you say like I: have these owners attached to these data sets and these owners attach these data sets and, like you, want all of that data to just be synced into your data hub instance. um This csv ingestion plug-in is uh really great for allowing your data up to be updated over here.

A

You can see like a little example of what that csv will look like awesome. So, let's get straight into a demo.

A

A

Great, uh can you guys see my data instance over here? Yes, okay, cool, so this is the data set that I'm going to apply um new tags firms and owners too. You can see right now. I don't have any owners, um we're missing some tags here at the entity level, and um maybe we want to add some new glossary terms, and one thing that I don't know if I called it out is that um you can uh apply tags terms and, like all of these aspects, both at um the data asset level.

A

So, like you know, uh data sets and charts and those sorts of things and also at the column level. um So let's go a little bit into like what defining that would look like.

A

So here is what the um the recipe looks like. So there's a couple of flags that you can put here so um there's this should overwrite flag that allows you to decide whether you want to either append um you know like uh any metadata, that's um or any aspects, that's defined in the csv um on your metadata or whether you want to replace um you know like replace like new owners or something or just append to the existing ones.

A

uh You can also, depending on how your csv is defined, put a delimiter or define a delimiter and then a rarity delimiter for array fields like tax terms and owners um awesome, and then let's take a look at what that csv looks like so, okay, so right now for just for this example, I'm just doing this on one data set sample hive data that I just showed you, but um you can do this on as many data sets as you want, but for the purpose of this demo I think it's a little easier so that you can just see the changes applied on one screen.

A

So um we're going to add some owners here. um uh I don't think there's any existing tags and um there's only one existing glossary term on the data set, so we're gonna add another one and we're also gonna make some changes to the column level. So, let's go back over here.

A

As you can see, I already ran a run, but um I'll just run so also I'm just using the existing like sample recipe and the demo data that's already available on our github repo great.

A

Let's run this awesome, you see that it produced for work units great.

A

Cool, so now you can see that uh we had some new owners that were applied. um Those tags and terms were added and yeah your data's. All in sync, great, let me go back here awesome.

A

um So what's next for this plugin, so obviously um you know, as I said, like descriptions on ownership type domains. Those are all a pretty low hanging fruit um if you have any other descriptions and ownership types to be clear, are going to be available in the next python release. um If you have any other ideas for how you want to extend this to other aspects that you think would be useful like. Please feel free to dm me on slack, and I would love to collaborate and help you.

A

um You know push any contributions and towards improving the plug-in cool.