DataHub Product Demos, 15 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: BIG Improvements to UI-based Metadata Ingestion!

Description

Chris Collins (Acryl Data) gives a tour of recent improvements to setting up metadata ingestion via the DataHub UI during the July 2022 Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

All right agreed cool, uh so hey everyone uh today, I'm gonna be talking about our ui based ingestion and some of the improvements that we're making to it all right. So today, if you want to set up a new ingestion source through the ui, you first have to go to your ingestion, tab and click new source. You can then select from a grouping of provided sources, as you can see here in this screenshot, and then you need to fill out your recipe using a ui based yaml editor.

A

However, as many of you can imagine, this isn't necessarily the most user-friendly experience. How do I know what fields are needed for my ingestion source? How do I know what these fields even do to my ingestion runs?

A

What if I make a typo or a spelling mistake or even a formatting mistake, what, if you're not used to working with yaml files and it's confusing and intimidating in reality, users are often struggling with this step for these reasons having to constantly reference our doc site or ask questions to the community or peers that might have more experience in this area.

A

Therefore, it's very common for users to make mistakes in their recipes. However, the only way to know if there's even an issue in your recipe is to try running it and maybe even waiting a little while setting up the whole flow and just hoping that it works uh also. So if it does fail, then you have to have the fun of sifting through some monster logs, maybe to simply learn that you must misspelled one of the fields, not very much a good time.

A

If you ask me- and I'm sure also many of you have been in this situation, so we got to thinking. How can we make this process easier for our users? Can we help prevent basic mistakes or even answer some? Maybe uh questions that people might have along the way, so in order to make ui-based ingestion more approachable for the average user, we've started rolling out some new changes that I'm going to be talking about and showing you here. So in order to show these new changes, I'm actually just going to hop straight into a demo.

A

So let me pull this guy up all right. So now, when you go to your ingestion tab and choose to create a new source, uh we are now offering ui-based forms in order to help the average user create a recipe that they want and they need. So, let's check out a common connector redshift.

A

So now, uh instead of tossing you in the deep end with a yaml file with some comments, we now walk you through the steps of creating recipe in a much more helpful way. We break this down into three different steps. First is the connection section which handles, as you guessed it all the fields necessary for connecting to your source of choice, and then, after that we have the filter section, and then this section is going to allow you to write patterns for allowing or denying certain assets.

A

uh You can see the broken down into different sections for the asset type that you care about and what's specific to this source and then finally, we have this advanced section, and this is going to be some of the more common fields uh used in recipes but aren't necessarily required in order for you to get things going.

A

Not only do you not need to reference our doc site to figure out what fields are needed for your source, but you now have access to tool, tips that are going to be providing you some information about what these fields actually are and what they're doing so now you can be a little bit more confident in what you're doing when you're checking stuff or adding things together.

A

Also, I'm here to say: don't you worry, though, yaml file lovers, because we still provide the same functionalities. We always have with managed congestion, and we can go up here to this little toggle and check out the yaml file.

A

So here you can actually edit your yama file directly and I'll, say it's not specifically one way or the other form or yaml. It is both. So let's change this host port, for example. I'm just gonna change it to something silly like test123, that should update and be reflected in our form which, yes, you can see it here, and you can actually edit the form and therefore see changes here right in your yaml file.

A

You can also add more specific fields to configure end configurations for your gamma file that might not be added in the form but are available for this connector. You can check those out on our dock sites whatever. So, if I wanted to add a limit, for example- and I can set that I can go to my form- change other stuff and whatnot and then my limit's still going to be there and I'm going to be able to submit that, and it will work just nicely all right.

A

So with this next release, we plan on providing forms for several different connectors, uh the ones that, with this release currently are bigquery and redshift, which I just showed you guys snowflake as well, but I'm going to be getting to that in one second uh looker and tableau.

A

Our plan is to continue rolling. These forms out for different sources over time, um they're pretty quick to get going. So our hope is to get them out as soon as possible to you guys for all the major sources that people care about all right now onto the final piece of what I wanted to show you guys.

A

So this feature aims to solve the problem of where you need to fill out a recipe schedule it all that sort of good stuff and then wait what could be a long while just to determine if you've even set things up properly and you can even establish a connection with this source.

A

So I'm going to go to snowflake. Like I talked about, we are now rolling out the ability to test your connection with different ingestion sources, so you can check that box before you even have to worry about more specifics like what assets you want to include or not include whether you care about profiling or lineage or whatever.

A

So now, I'm going to demo. This feature live to see. If I can establish a connection with our snowflake source and for demo purposes, I prepared a recipe that I'm just going to copy paste in here. I could fill it out through the form, but it's going to be quicker. You can see it all updates in the form pretty fun. So, let's test test this connection, uh so this the first time you do it. It could take a little bit longer.

A

But here it shouldn't take too long, maybe five to ten seconds or so and boom there we go. You officially tested your connection with the credentials that you provided, and now we can know for certainty that our credentials are all set.

A

So not only does this help with debugging issues later on, but you're also going to know straight away all the different capabilities that afford to do with your connection credentials. So, as you can see here, we list these capabilities and give you the thumbs up. If you have access to it, uh we'll obviously also let you know if there's any issues such as like you can't connect, because your credentials are wrong.

A

Maybe you misspelled something or maybe you didn't, fill out all the required fields um and then also, depending on your connection, configuration you might not have access to all the different capabilities depending on roles or whatever. uh So in order to show you guys what that might look like, I'm gonna actually update this. uh If I get rid of our role here, I should still be able to establish a connection and have access to most of my sources or excuse me my capabilities, but not necessarily all of them.

A

So let's try it again see what happens.

A

It's gonna wait, it's a good time all right and there you go so you can see once I removed that I have access to most things, but I don't have access to table level lineage, for example, and also we provide you with a nice helpful reason as to why you don't have access to this, so should help you in debugging and figuring out uh what you need to do in order to get that capability. If that is a capability that you desire all right, let me go back to the presentation.

A

So luckily, you don't have to wait long for this feature as it's going out with our next release and, as I said over time, we're going to be introducing more and more forms for the different sources, more test connection functionality for different sources and actually even more fun features that were going to make configuring your sources and your recipes even easier. So keep an eye out, and that's it for me.