DataHub Product Demos, 23 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Data Containers are Here!

Description

John Joyce (Acryl Data) gives an overview and demo of Data Containers in DataHub during the January Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

I'm going to quickly go through asset containers as well. um I think you'll see that there's a lot of synergy between these two concepts, there's a lot of reuse between them, which is why we're doing them together.

A

But basically a container is a general purpose entity on data hub that represents kind of a physical collection of data assets, as opposed to a logical collection in the domain's case, and this physical collection typically will come directly from that third-party source, like snowflake mysql bigquery, and is you know the common ones you think of? Are databases schemas, maybe projects in the case of of bigquery yeah? So that's pretty much the examples here and just a quick overview of containers versus domains again.

A

Containers are those physical collections, they're also going to be immutable once you've ingested them onto data, so you can't kind of freely, add or remove entities from a domain, it's kind of taken as part of that assets, entity or identity. Rather the domain that it's a part of or the container that's part of rather and then domains are logical collections. Curated on data.

B

Hub so the source of truth is data.

A

All right, one more quick demo.

B

A

Demo, the new containers feature which will be available in version 0.8.25 of datahub, so containers are collections of data assets that are ingested from a third-party system like snowflake, redshift, bigquery, postgres and more some common containers. You would think about our databases, schemas projects and bigquery.

A

Currently, on data hub, we only ingest leaf level nodes, so things like data sets, tables, etc, and so the goal of this project is to really bring in those uh containers, as first class citizens on the data hub metadata graph. So what I'm going to show?

A

You is what it would look like to ingest containers from my sql and I'm going to start by just searching my sql if we scroll down a bit you'll notice that, for this datahub metadata aspect table that lives in my sql there's a new piece of information that appears in the the preview here, and that is a reference to the kind of container in which this table lives. In this case, metadata aspect table lives in a schema, called datahub schema.

A

If you scroll down a bit more you'll, see that the schema is actually appearing in the search results. As a separate node in itself- and it actually has a parent container as well- we can also see that it has three entities inside of it, so three tables inside of it.

A

Probably we can go ahead and click into that and you'll notice that what you get is a profile page very similar to you know: data set pages where you can do things like editing, documentation, viewing properties, adding links, tags terms and owners, even the new domains concept as well.

A

You can also easily view all of the entities that are inside of this container in this case. Inside of this schema, you can also slide out this filters bar to slice and dice the list by some secondary filters, or you can just use a free text search to search down the entities.

A

I think this is very very handy, especially in the case where you have a large number of entities inside of a container, and what you can do from this page is navigate both down to the children so using this search, but you can also navigate up to the parent, and so, if you see at the schema level, we're actually inside of another container, which is called datahub db, which exists in my mysql instance.

A

So if I click on that, you'll see that this also gets its own page will appear in search results as a separate, node, etc.

A

And so this just makes things a little bit better, because you can add that additional context that additional metadata at that parent level, you can also easily see how data is organized physically in these external systems via data hub. So the last thing I'll show is just filtering by container you'll notice if you scroll down on the left here that there's a a new container filter, and this allows you to easily search for those assets that are inside of a particular container.

A

Of course, you can combine search filters, as you normally would um to to whittle it down even more so this is basically the the first version of the containers feature.

B

All right back to the slides here.

A

Okay, so availability uh for the containers feature in zero. Eight two five will be support for ingesting containers like schemas and databases and projects from bigquery, as well uh from all of the sql sources. So.

B

Snowflake redshift bigquery, plus my sequel, ms sql, etc, um searching for filtering by container as demo browsing assets inside the container, adding metadata to a container.

A

On the horizon, for this feature is kind of based on the feedback we get from the community support for additional sources and container types. So now that we have that general purpose model, we can actually go through the other connectors that weren't listed and try to add them in as necessary as well. For things like blue s3, mongodb is another obvious one. We want to get to container-based metadata policies, so this is another direct piece of feedback we've received.

A

Is that folks want to manage metadata policies via container as well as domain, and I think yeah I think that's pretty much it. um Let me know if you guys have feedback, I'm really excited to get these out to the community zero. Eight two five will be released next friday, um so a week from today, at which point you should be able to start using these. That being said, domains is already on head, so quick start will already have the domains feature. Containers is uh following shortly behind.