DataHub Product Demos, 19 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Looker Search Improvements

Description

Gabe Lyons (Acryl Data) provides a walkthrough of improvements around indexing, looks, and dashboards within DataHub.

From DataHub August 2022 Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

So I'm going to talk about some improvements that we've made to the integration between data hub and looker today.

A

um So the main thrust of this improvement is around indexing, looks and dashboards from looker. So the main feedback that we keep getting from the community is that when people are going and searching for things like looks and dashboards, they really want to discover these visualizations based on what they contain inside of them. So things like what metrics do they visualize?

A

What are different dimensions that you're slicing the data by, um but historically we've only indexed other types of metadata around these, so things like title and description, but not actually the content, that's being visualized, and we keep hearing that this presents a challenge for discovery, and so sometimes you might feel like this hamster mma is trying to find the content instead of it just being presented to you.

A

So um what we did is we index the metrics and dimensions that power these looks and dashboards for uh for the purposes of search and discovery. So we added these indexes and now, when you search for something like a metric or a dimension, looks and dashboards that contain those are going to be surfaced immediately. So, instead of having to go through that maze, you'll be like this other hamster here and find your snack.

A

Immediately um and as a side note, uh looker has this concept of labels for fields, and this is associated with fields that are chart and dashboard specific, but also fields that are part of looker views or looker explorers. And historically, we haven't been bringing those into data hub.

A

We did make an extension to the metadata model of schema fields so that we can include the label property and that's now going to be brought in for both looker charts and dashboards for indexing purposes and search discovery, but it will also be brought in for looker views and look for explorers. The data set entities so that you can discover them those entities based on label as well. It's not being shown right now, but soon.

A

Just like this answer, it will break through and appear on the ui down the line, all right and it would not be a data hub town hall session without a live demo. So let me show you what it looks like in practice.

A

So, for example, the first thing I'll search for is a dimension breed that we use in looker so searching by breed. Well, what you'll see is I'll come up with results that we might have seen before, where we see dashboards that contain breed in the title or a chart that contains breed in the title.

A

But as you go down pages, what you'll see is here are charts and dashboards that do not have reading title but use breed as a dimension, and this is these are the results that we would not have been able to surface previously.

A

Now, if we go into one of these looker charts, you can see that, although it doesn't have breed in the metadata that we previously indexed once we load up and look at what metrics and dimensions are in use, you can see that read was in use, so you're going to be able to recall many more data sets than you could previously.

A

um One other use case that I wanted to call out was not just searching metrics and dimensions that came from looker explorers, but also calculations that were done on the chart, specifically so say. For example, um I wanted to uh search for a metric. I wanted to see who is displaying this concept of average agent years.

A

So searching by average 18 years, I'm getting only charts and dashboards that use this particular label and then going in you can see that this label is not not necessarily a dimension. That's coming from the underlying explorer, but actually this was a custom field built as a calculation specifically for this chart, so we're also able to extract and index those.

A

So hopefully, these changes make it a lot easier for you, folks to discover the charts and dashboards that are relevant to you in terms of what's coming down the line. We want to show these labels that we're now extracting on the schema tab for views and explorers.

A

We also want to not just index the inputs on, looks and dashboards, but also give you a way to access them, so that we'll have that tab where you can see what metrics and dimensions a given looker dashboard includes and then finally, as part of this, we're also integrating column level in each so we're able to pull in column level lineage as metadata for uh charts and dashboards, um and once we do uh the column level lineage push that we want to do across the whole.

A

App you'll be able to see this column level image in the lineage ui.

A

So that wraps it up for my presentation- and hopefully you folks find this useful and we really look forward to hearing your feedback on this, and this will all be included in the next release. That's coming up.